2024-01-24T17:28:02.447 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f 2024-01-24T17:28:02.447 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972 --owner scheduled_vshankar@teuthology --verbose --name vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} -- /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/orig.config.yaml 2024-01-24T17:28:02.493 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-24T17:28:02.577 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972 branch: wip-vshankar-testing1-reef-2024-01-24-1150 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} email: vshankar@redhat.com first_in_suite: false job_id: '7530972' kernel: flavor: default kdb: true sha1: f2d3460408951339ac0933340ad449487c9efddc last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing1-reef-2024-01-24-1150 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - client session with non-allowable root sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing1-reef-2024-01-24-1150 sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4668 sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 sleep_before_teardown: 0 subset: 346/1024 suite: fs suite_branch: wip-vshankar-testing1-reef-2024-01-24-1150 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKIzEkcl8WMnQW1kTCFzMXo6NA5rOiCIVmprT9OVTY5g4soVADD3sd9U5Sy2pPHYpjgZjbhWpvpAbWGKYxXKpMg= smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6sBC4h2uBb88W06XRSHJhI8OMNG9o8ip/9uHl7OoVnW/PfJCu9Q38OVllMnilIDdnnCzFgqRJ8AZ8nKvxECl4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_sessionmap teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: c856e0f9244dd29a50697123b480d63f16a4ed81 timestamp: 2024-01-24_12:12:30 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-24T17:28:02.577 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa; will attempt to use it 2024-01-24T17:28:02.578 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks 2024-01-24T17:28:02.578 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-24T17:28:02.580 INFO:teuthology.task.internal:Checking packages... 2024-01-24T17:28:02.609 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '4e3182a65800aed6d4aa2f6b86da3cfd72188134' 2024-01-24T17:28:02.609 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-24T17:28:02.609 INFO:teuthology.packaging:ref: None 2024-01-24T17:28:02.609 INFO:teuthology.packaging:tag: None 2024-01-24T17:28:02.609 INFO:teuthology.packaging:branch: wip-vshankar-testing1-reef-2024-01-24-1150 2024-01-24T17:28:02.609 INFO:teuthology.packaging:sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T17:28:02.609 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing1-reef-2024-01-24-1150 2024-01-24T17:28:02.934 INFO:teuthology.task.internal:Found packages for ceph version 18.2.1-341.g4e3182a6 2024-01-24T17:28:02.936 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-24T17:28:02.943 INFO:teuthology.task.internal:no buildpackages task found 2024-01-24T17:28:02.943 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-24T17:28:02.983 INFO:teuthology.task.internal:Saving configuration 2024-01-24T17:28:03.001 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-24T17:28:03.008 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T17:28:03.033 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 17:22:05.655884', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKIzEkcl8WMnQW1kTCFzMXo6NA5rOiCIVmprT9OVTY5g4soVADD3sd9U5Sy2pPHYpjgZjbhWpvpAbWGKYxXKpMg='} 2024-01-24T17:28:03.054 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 17:22:05.657644', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6sBC4h2uBb88W06XRSHJhI8OMNG9o8ip/9uHl7OoVnW/PfJCu9Q38OVllMnilIDdnnCzFgqRJ8AZ8nKvxECl4='} 2024-01-24T17:28:03.054 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-24T17:28:03.061 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-24T17:28:03.061 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-24T17:28:03.062 INFO:teuthology.run_tasks:Running task console_log... 2024-01-24T17:28:03.148 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fec916c50d0>, signals=[15]) 2024-01-24T17:28:03.149 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-24T17:28:03.155 INFO:teuthology.task.internal:Opening connections... 2024-01-24T17:28:03.155 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:28:03.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:28:03.269 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:28:03.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:28:03.355 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-24T17:28:03.362 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-01-24T17:28:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-01-24T17:28:03.380 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-01-24T17:28:03.435 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2024-01-24T17:28:03.435 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2024-01-24T17:28:03.435 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-24T17:28:03.436 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-24T17:28:03.437 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-01-24T17:28:03.460 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-01-24T17:28:03.478 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-01-24T17:28:03.478 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-01-24T17:28:03.534 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-24T17:28:03.535 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-24T17:28:03.536 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-01-24T17:28:03.560 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-24T17:28:03.570 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-24T17:28:03.593 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-24T17:28:03.593 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-01-24T17:28:03.595 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-01-24T17:28:03.611 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-24T17:28:03.619 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-24T17:28:03.620 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-01-24T17:28:03.650 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-01-24T17:28:03.697 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-24T17:28:03.747 INFO:teuthology.run_tasks:Running task kernel... 2024-01-24T17:28:03.763 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': 'f2d3460408951339ac0933340ad449487c9efddc'} 2024-01-24T17:28:03.763 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': 'f2d3460408951339ac0933340ad449487c9efddc'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': 'f2d3460408951339ac0933340ad449487c9efddc'}}, timeout 300 2024-01-24T17:28:03.763 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-24T17:28:03.764 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-24T17:28:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:28:03.780 DEBUG:teuthology.task.kernel:sha1 for mon.b is f2d3460408951339ac0933340ad449487c9efddc 2024-01-24T17:28:03.780 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "f2d3460408951339ac0933340ad449487c9efddc"... 2024-01-24T17:28:03.781 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-01-24T17:28:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:28:03.782 DEBUG:teuthology.task.kernel:sha1 for mon.a is f2d3460408951339ac0933340ad449487c9efddc 2024-01-24T17:28:03.782 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "f2d3460408951339ac0933340ad449487c9efddc"... 2024-01-24T17:28:03.782 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-01-24T17:28:03.837 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-361.el9.x86_64 2024-01-24T17:28:03.838 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs f2d3460408951339ac0933340ad449487c9efddc 2024-01-24T17:28:03.838 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-24T17:28:03.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f2d3460408951339ac0933340ad449487c9efddc 2024-01-24T17:28:03.842 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-361.el9.x86_64 2024-01-24T17:28:03.842 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs f2d3460408951339ac0933340ad449487c9efddc 2024-01-24T17:28:03.842 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-24T17:28:03.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f2d3460408951339ac0933340ad449487c9efddc 2024-01-24T17:28:03.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum upgrade -y linux-firmware 2024-01-24T17:28:03.984 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum upgrade -y linux-firmware 2024-01-24T17:28:06.217 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2024-01-24T17:28:06.223 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2024-01-24T17:28:07.826 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-01-24T17:28:07.839 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-01-24T17:28:09.458 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-01-24T17:28:09.864 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-01-24T17:28:11.259 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-01-24T17:28:15.510 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 5.9 MB 00:00 2024-01-24T17:28:16.993 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2024-01-24T17:28:18.756 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2024-01-24T17:28:23.936 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-01-24T17:28:24.149 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:01 ago on Wed 24 Jan 2024 05:28:23 PM UTC. 2024-01-24T17:28:26.546 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-01-24T17:28:26.550 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-01-24T17:28:26.550 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 1 Package 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 375 M 2024-01-24T17:28:26.551 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-01-24T17:28:35.437 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 784 kB/s | 19 MB 00:24 2024-01-24T17:28:40.113 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 5.9 MB 00:01 2024-01-24T17:28:40.228 INFO:teuthology.orchestra.run.smithi039.stdout:linux-firmware-20240109-142.el9.noarch.rpm 27 MB/s | 375 MB 00:13 2024-01-24T17:28:40.229 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-01-24T17:28:40.230 INFO:teuthology.orchestra.run.smithi039.stdout:Total 27 MB/s | 375 MB 00:13 2024-01-24T17:28:40.230 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-01-24T17:28:40.318 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-01-24T17:28:40.318 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-01-24T17:28:41.571 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2024-01-24T17:28:42.830 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-01-24T17:28:42.831 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-01-24T17:28:43.373 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-01-24T17:28:48.176 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-01-24T17:28:49.209 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-01-24T17:28:49.432 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-24T17:28:49.528 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-24T17:28:50.566 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-01-24T17:28:50.571 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 1 Package 2024-01-24T17:28:50.572 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:28:50.572 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 375 M 2024-01-24T17:28:50.572 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-01-24T17:28:50.675 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-24T17:28:50.675 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-24T17:28:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-24T17:28:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:28:51.089 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-01-24T17:28:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-24T17:28:51.089 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:28:51.089 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-01-24T17:28:51.374 INFO:teuthology.task.kernel:Downloading kernel f2d3460408951339ac0933340ad449487c9efddc on mon.a... 2024-01-24T17:28:51.374 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f2d3460408951339ac0933340ad449487c9efddc 2024-01-24T17:28:51.515 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/f2d3460408951339ac0933340ad449487c9efddc/centos/9/flavors/default/x86_64/ 2024-01-24T17:28:51.515 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_gf2d346040895-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/f2d3460408951339ac0933340ad449487c9efddc/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-24T17:28:51.518 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi039.front.sepia.ceph.com to complete... 2024-01-24T17:29:05.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24 17:29:05 URL:https://1.chacra.ceph.com/r/kernel/testing/f2d3460408951339ac0933340ad449487c9efddc/centos/9/flavors/default/x86_64/kernel-6.7.0_gf2d346040895-1.x86_64.rpm [330270367/330270367] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-24T17:29:05.449 INFO:teuthology.orchestra.run.smithi039.stderr:FINISHED --2024-01-24 17:29:05-- 2024-01-24T17:29:05.449 INFO:teuthology.orchestra.run.smithi039.stderr:Total wall clock time: 14s 2024-01-24T17:29:05.449 INFO:teuthology.orchestra.run.smithi039.stderr:Downloaded: 1 files, 315M in 13s (23.6 MB/s) 2024-01-24T17:29:05.451 INFO:teuthology.task.kernel:Installing kernel f2d3460408951339ac0933340ad449487c9efddc on mon.a... 2024-01-24T17:29:05.452 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-24T17:29:07.441 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying... ######################################## 2024-01-24T17:29:07.499 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing... ######################################## 2024-01-24T17:29:07.586 INFO:teuthology.orchestra.run.smithi039.stdout:Updating / installing... 2024-01-24T17:29:08.538 INFO:teuthology.orchestra.run.smithi100.stdout:linux-firmware-20240109-142.el9.noarch.rpm 21 MB/s | 375 MB 00:17 2024-01-24T17:29:08.539 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-01-24T17:29:08.540 INFO:teuthology.orchestra.run.smithi100.stdout:Total 21 MB/s | 375 MB 00:17 2024-01-24T17:29:08.540 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-01-24T17:29:08.623 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-01-24T17:29:08.623 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-01-24T17:29:11.008 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-01-24T17:29:11.009 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-01-24T17:29:15.356 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-6.7.0_gf2d346040895-1 ######################################## 2024-01-24T17:29:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-01-24T17:29:17.673 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-24T17:29:17.746 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-24T17:29:18.884 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-24T17:29:18.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-24T17:29:19.246 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-24T17:29:19.247 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:29:19.247 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-01-24T17:29:19.247 INFO:teuthology.orchestra.run.smithi100.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-24T17:29:19.247 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:29:19.247 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-01-24T17:29:19.546 INFO:teuthology.task.kernel:Downloading kernel f2d3460408951339ac0933340ad449487c9efddc on mon.b... 2024-01-24T17:29:19.546 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f2d3460408951339ac0933340ad449487c9efddc 2024-01-24T17:29:19.696 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/f2d3460408951339ac0933340ad449487c9efddc/centos/9/flavors/default/x86_64/ 2024-01-24T17:29:19.697 DEBUG:teuthology.orchestra.run.smithi100:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_gf2d346040895-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/f2d3460408951339ac0933340ad449487c9efddc/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-24T17:29:19.699 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi100.front.sepia.ceph.com to complete... 2024-01-24T17:29:31.235 INFO:teuthology.orchestra.run.smithi039.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-24T17:29:31.708 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24 17:29:31 URL:https://1.chacra.ceph.com/r/kernel/testing/f2d3460408951339ac0933340ad449487c9efddc/centos/9/flavors/default/x86_64/kernel-6.7.0_gf2d346040895-1.x86_64.rpm [330270367/330270367] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-24T17:29:31.708 INFO:teuthology.orchestra.run.smithi100.stderr:FINISHED --2024-01-24 17:29:31-- 2024-01-24T17:29:31.708 INFO:teuthology.orchestra.run.smithi100.stderr:Total wall clock time: 12s 2024-01-24T17:29:31.708 INFO:teuthology.orchestra.run.smithi100.stderr:Downloaded: 1 files, 315M in 12s (27.1 MB/s) 2024-01-24T17:29:31.711 INFO:teuthology.task.kernel:Installing kernel f2d3460408951339ac0933340ad449487c9efddc on mon.b... 2024-01-24T17:29:31.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-24T17:29:33.601 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying... ######################################## 2024-01-24T17:29:33.724 INFO:teuthology.orchestra.run.smithi100.stdout:Preparing... ######################################## 2024-01-24T17:29:33.804 INFO:teuthology.orchestra.run.smithi100.stdout:Updating / installing... 2024-01-24T17:29:41.188 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-6.7.0_gf2d346040895-1 ######################################## 2024-01-24T17:29:54.004 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-24T17:29:54.004 DEBUG:teuthology.orchestra.run.smithi039:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-24T17:29:54.074 INFO:teuthology.orchestra.run.smithi039.stdout:/boot/System.map-6.7.0-gf2d346040895 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/boot/config-6.7.0-gf2d346040895 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/boot/vmlinuz-6.7.0-gf2d346040895 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-24T17:29:54.075 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-24T17:29:54.076 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kernel 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kernel/cpu 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kernel/cpu/mce 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kvm 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kvm/kvm.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/ansi_cprng.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/anubis.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/arc4.ko 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx 2024-01-24T17:29:54.077 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_memcpy.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_pq.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_tx.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_xor.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/raid6test.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/authenc.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/authencesn.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/blake2b_generic.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/blowfish_common.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/blowfish_generic.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/camellia_generic.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cast5_generic.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cast6_generic.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cast_common.ko 2024-01-24T17:29:54.078 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/ccm.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cmac.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/crc32_generic.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/crypto_engine.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/crypto_user.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cts.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/des_generic.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/dh_generic.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/echainiv.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/essiv.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/fcrypt.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/khazad.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/lz4.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/lz4hc.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/md4.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/michael_mic.ko 2024-01-24T17:29:54.079 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/pcbc.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/pcrypt.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/rmd160.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/seed.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/serpent_generic.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/tcrypt.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/tea.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/twofish_common.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/twofish_generic.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/vmac.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/wp512.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/xcbc.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/xor.ko 2024-01-24T17:29:54.080 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/xxhash_generic.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/acpi_pad.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/custom_method.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/ec_sys.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/platform_profile.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/sbs.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/sbshc.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/video.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/acard-ahci.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/ahci_platform.ko 2024-01-24T17:29:54.081 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/ata_generic.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/libahci_platform.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_acpi.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_ali.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_amd.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_artop.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_atiixp.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_atp867x.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_cmd640.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_cmd64x.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_cypress.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_efar.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_hpt366.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_hpt37x.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-24T17:29:54.082 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_it8213.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_it821x.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_jmicron.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_marvell.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_mpiix.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_netcell.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_ninja32.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_ns87410.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_ns87415.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_oldpiix.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_opti.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_optidma.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_pcmcia.ko 2024-01-24T17:29:54.083 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_piccolo.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_rdc.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_sch.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_serverworks.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_sil680.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_sis.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_sl82c105.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_triflex.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_via.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pdc_adma.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_inic162x.ko 2024-01-24T17:29:54.084 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_mv.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_nv.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_promise.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_qstor.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_sil.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_sil24.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_sis.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_svw.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_sx4.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_uli.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_via.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_vsc.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/auxdisplay 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/auxdisplay/ks0108.ko 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/base 2024-01-24T17:29:54.085 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/base/regmap 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/bcma 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/bcma/bcma.ko 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/aoe 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/aoe/aoe.ko 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/brd.ko 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/drbd 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/drbd/drbd.ko 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/floppy.ko 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/loop.ko 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/mtip32xx 2024-01-24T17:29:54.086 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/nbd.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/null_blk 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/null_blk/null_blk.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/pktcdvd.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/rbd.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/virtio_blk.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/xen-blkback 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/xen-blkfront.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/zram 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/zram/zram.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hangcheck-timer.ko 2024-01-24T17:29:54.087 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/via-rng.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/lp.ko 2024-01-24T17:29:54.088 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/mwave 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/mwave/mwave.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ppdev.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tlclk.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/uv_mmtimer.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/virtio_console.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-24T17:29:54.089 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/ccp 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/ccp/ccp.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat/qat_common 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-24T17:29:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/padlock-aes.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/padlock-sha.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/virtio 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dca 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dca/dca.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/devfreq 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma/dw 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma/ioat 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/amd64_edac.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/e752x_edac.ko 2024-01-24T17:29:54.091 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/edac_mce_amd.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i3000_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i3200_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i5100_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i5400_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i7300_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i7core_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i82975x_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/ie31200_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/sb_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/x38_edac.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/firmware 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/firmware/edd.ko 2024-01-24T17:29:54.092 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/gpio 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/gpio/gpio-ich.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-appleir.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-aureal.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-axff.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-dr.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-elecom.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-elo.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-emsff.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-gaff.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-gt683r.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-gyration.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-24T17:29:54.093 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-holtekff.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-hyperv.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-icade.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-keytouch.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-kye.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-lcpower.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-led.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-lenovo.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-multitouch.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-ortek.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-penmount.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-petalynx.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-picolcd.ko 2024-01-24T17:29:54.094 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-pl.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-plantronics.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-primax.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-rmi.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-common.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-24T17:29:54.095 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-saitek.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-samsung.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-sjoy.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-sony.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-speedlink.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-steelseries.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-sunplus.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-tivo.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-tmff.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-topseed.ko 2024-01-24T17:29:54.096 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-twinhan.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-uclogic.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-waltop.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-wiimote.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-xinmo.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-zpff.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-zydacron.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/uhid.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/wacom.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hv 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hv/hv_balloon.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hv/hv_utils.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hv/hv_vmbus.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/abituguru.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/abituguru3.ko 2024-01-24T17:29:54.097 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ad7414.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ad7418.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adc128d818.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm1025.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm1026.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm1029.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm1031.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm9240.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ads7828.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7410.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7411.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7462.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7470.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7475.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7x10.ko 2024-01-24T17:29:54.098 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/amc6821.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/applesmc.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/asb100.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/asc7621.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/atxp1.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/coretemp.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/dme1737.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ds1621.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ds620.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/emc1403.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/emc6w201.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/f71805f.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/f71882fg.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/f75375s.ko 2024-01-24T17:29:54.099 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/fam15h_power.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/fschmd.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/g760a.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/g762.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/gl518sm.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/gl520sm.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/i5500_temp.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/i5k_amb.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ibmaem.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ibmpex.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ina209.ko 2024-01-24T17:29:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ina2xx.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/it87.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/k10temp.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/k8temp.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lineage-pem.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm63.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm73.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm75.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm77.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm78.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm80.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm83.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm85.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm87.ko 2024-01-24T17:29:54.101 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm90.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm92.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm93.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm95234.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm95241.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm95245.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc2945.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4151.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4215.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4222.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4245.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4260.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4261.ko 2024-01-24T17:29:54.102 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max16065.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max1619.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max1668.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max197.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max6639.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max6650.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max6697.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/mcp3021.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/nct6683.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/nct6775-core.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/nct6775.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/nct7802.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pc87360.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pc87427.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pcf8591.ko 2024-01-24T17:29:54.103 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/powr1220.ko 2024-01-24T17:29:54.104 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sch5627.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sch5636.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sht15.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sht21.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/shtc1.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sis5595.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/smsc47b397.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/smsc47m1.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/smsc47m192.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/thmc50.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/tmp102.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/tmp103.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/tmp401.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/tmp421.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/via-cputemp.ko 2024-01-24T17:29:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/via686a.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/vt1211.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/vt8231.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83627ehf.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83627hf.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83781d.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83791d.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83792d.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83793.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83795.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83l785ts.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83l786ng.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/algos 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-24T17:29:54.106 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-24T17:29:54.107 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-dev.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-mux.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-smbus.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-stub.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-24T17:29:54.108 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/ib_core.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/cxgb4 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mlx4 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mlx5 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mthca 2024-01-24T17:29:54.109 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/ipoib 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/iser 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/isert 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/srp 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/srpt 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/gameport 2024-01-24T17:29:54.110 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/gameport/gameport.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/joydev.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/keyboard 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/appletouch.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/bcm5974.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/cyapatp.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/sermouse.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-24T17:29:54.111 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/rmi4 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio/altera_ps2.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio/arc_ps2.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio/serio_raw.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/sparse-keymap.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/leds-blinkm.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/leds-lm3530.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/leds-lp3944.ko 2024-01-24T17:29:54.112 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/leds-ss4200.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/bcache 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/bcache/bcache.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-bio-prison.ko 2024-01-24T17:29:54.113 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-cache-smq.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-cache.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-crypt.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-delay.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-flakey.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-log-userspace.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-multipath.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-queue-length.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-raid.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-round-robin.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-service-time.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-switch.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-thin-pool.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-verity.ko 2024-01-24T17:29:54.114 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/faulty.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/linear.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/multipath.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/persistent-data 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/raid0.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/raid1.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/raid10.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/raid456.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptbase.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptctl.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptfc.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptlan.ko 2024-01-24T17:29:54.115 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptsas.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptscsih.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptspi.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/lpc_ich.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/lpc_sch.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/mfd-core.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/sm501.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/viperboard.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/vx855.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/wl1273-core.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/altera-stapl 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-24T17:29:54.116 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/apds9802als.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/apds990x.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/bh1770glc.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/cb710 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/cb710/cb710.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/eeprom 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/eeprom/at24.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/eeprom/max6875.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/enclosure.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/hpilo.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/isl29003.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/isl29020.ko 2024-01-24T17:29:54.117 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/lis3lv02d 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/mei 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/mei/mei-me.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/mei/mei-txe.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/mei/mei.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-gru 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-xp 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/tifm_7xx1.ko 2024-01-24T17:29:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/tifm_core.ko 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/tsl2550.ko 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/vmw_balloon.ko 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/vmw_vmci 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/bonding 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/bonding/bonding.ko 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/b53 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/mv88e6xxx 2024-01-24T17:29:54.119 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dummy.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/eql.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-24T17:29:54.120 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/adaptec 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/agere 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/alteon 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/altera 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/amd 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-24T17:29:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/alx 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atlx 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-24T17:29:54.122 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/genet 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/brocade 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/brocade/bna 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-24T17:29:54.123 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/cisco 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/cisco/enic 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-24T17:29:54.124 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dlink 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dnet.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/emulex 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/emulex/benet 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/ethoc.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/fealnx.ko 2024-01-24T17:29:54.125 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e1000 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e1000e 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/fm10k 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/i40e 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/iavf 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/igb 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-24T17:29:54.126 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/igbvf 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/jme.ko 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/marvell 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-24T17:29:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/micrel 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/myricom 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/natsemi 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-24T17:29:54.128 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/neterion 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/nvidia 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/packetengines 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/rdc 2024-01-24T17:29:54.129 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/rocker 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sfc 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/silan 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sis 2024-01-24T17:29:54.130 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/stmicro 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-24T17:29:54.131 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/tehuti 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/ti 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/via 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/wiznet 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/xircom 2024-01-24T17:29:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/geneve.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/hyperv 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ieee802154 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ifb.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ipvlan 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/macvlan.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/macvtap.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-24T17:29:54.133 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mii.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/net_failover.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/netconsole.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/nlmon.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/pcs 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/amd.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/bcm87xx.ko 2024-01-24T17:29:54.134 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/broadcom.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/cicada.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/davicom.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/dp83640.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/et1011c.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/icplus.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/lxt.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/marvell.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/micrel.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/national.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/phylink.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/qsemi.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/realtek.ko 2024-01-24T17:29:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/smsc.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/ste10Xp.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/vitesse.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_async.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/pppoe.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/pppox.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/pptp.ko 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/slip 2024-01-24T17:29:54.136 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/slip/slhc.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/slip/slip.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/sungem_phy.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/tap.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_random.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/tun.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/veth.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/virtio_net.ko 2024-01-24T17:29:54.137 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/vmxnet3 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/vxlan 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/vxlan/vxlan.ko 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/xen-netback 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/xen-netfront.ko 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ntb 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ntb/ntb.ko 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/nvme 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/nvme/host 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/nvme/host/nvme-core.ko 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/nvme/host/nvme.ko 2024-01-24T17:29:54.138 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport/parport.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport/parport_cs.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport/parport_pc.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport/parport_serial.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/hotplug 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/pcie 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/xen-pcifront.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pcmcia 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pcmcia/i82092.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pcmcia/pd6729.ko 2024-01-24T17:29:54.139 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pinctrl 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pinctrl/intel 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/acerhdf.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-24T17:29:54.140 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/hdaps.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/intel 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/intel_ips.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-24T17:29:54.141 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/wmi.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/powercap 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/clients 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/clients/pps_parport.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/pps_core.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ptp 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ptp/ptp.ko 2024-01-24T17:29:54.142 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ptp/ptp_kvm.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-em3027.ko 2024-01-24T17:29:54.143 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-max6900.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-24T17:29:54.144 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-x1205.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/3w-9xxx.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/3w-sas.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/3w-xxxx.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/BusLogic.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/a100u2w.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aacraid 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-24T17:29:54.145 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/advansys.ko 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aic7xxx 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/am53c974.ko 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/arcmsr 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/atp870u.ko 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/be2iscsi 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bfa 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bfa/bfa.ko 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bnx2fc 2024-01-24T17:29:54.146 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bnx2i 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/ch.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/csiostor 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/dc395x.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-24T17:29:54.147 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/dmx3191d.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/esas2r 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/esp_scsi.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fcoe 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fnic 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fnic/fnic.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/hpsa.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/hptiop.ko 2024-01-24T17:29:54.148 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/hv_storvsc.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/initio.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/ipr.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/ips.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/isci 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/isci/isci.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libfc 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libfc/libfc.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libiscsi.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libsas 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libsas/libsas.ko 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/lpfc 2024-01-24T17:29:54.149 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mpt3sas 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mvsas 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mvumi.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/pm8001 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/pmcraid.ko 2024-01-24T17:29:54.150 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla1280.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla2xxx 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla4xxx 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/raid_class.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_debug.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/ses.ko 2024-01-24T17:29:54.151 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/st.ko 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/stex.ko 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/sym53c8xx_2 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/virtio_scsi.ko 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/wd719x.ko 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ssb 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ssb/ssb.ko 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/iscsi 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/loopback 2024-01-24T17:29:54.152 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_file.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_iblock.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_mod.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_pscsi.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_user.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/tcm_fc 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-24T17:29:54.153 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-24T17:29:54.154 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/ipwireless 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/n_gsm.ko 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/n_hdlc.ko 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/nozomi.ko 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/8250 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/arc_uart.ko 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/jsm 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs/core 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-24T17:29:54.155 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs/host 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio_aec.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio_cif.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio_pci_generic.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio_sercos3.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class/cdc-acm.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class/usblp.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class/usbtmc.ko 2024-01-24T17:29:54.156 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/host 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/image 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/image/mdc800.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/image/microtek.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/adutux.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/appledisplay.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/emi26.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/emi62.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/ezusb.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/idmouse.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/iowarrior.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-24T17:29:54.157 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/ldusb.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/legousbtower.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/sisusbvga 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/usb3503.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/usblcd.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/uss720.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/yurex.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/phy 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/phy/phy-generic.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/aircable.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ark3116.ko 2024-01-24T17:29:54.158 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ch341.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/cp210x.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/cyberjack.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/empeg.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/io_ti.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ipaq.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ipw.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ir-usb.ko 2024-01-24T17:29:54.159 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/keyspan.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/mct_u232.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/mos7720.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/mos7840.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/navman.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/omninet.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/opticon.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/option.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/oti6858.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/pl2303.ko 2024-01-24T17:29:54.160 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/qcaux.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/qcserial.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/quatech2.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/safe_serial.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/sierra.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ssu100.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/symbolserial.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/usb_debug.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/visor.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/whiteheat.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-24T17:29:54.161 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/uas.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-karma.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-24T17:29:54.162 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/usb-storage.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/usbip 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/pci 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/vfio.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost/vhost.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost/vhost_net.ko 2024-01-24T17:29:54.163 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost/vhost_scsi.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video/backlight 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video/backlight/apple_bl.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video/backlight/lcd.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_balloon.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_mmio.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_pci.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_ring.ko 2024-01-24T17:29:54.164 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/masters 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/masters/ds2482.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/masters/ds2490.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-24T17:29:54.165 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/wire.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/hpwdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/i6300esb.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/ib700wdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/ibmasr.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/it87_wdt.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/machzwd.ko 2024-01-24T17:29:54.166 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/nv_tco.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/softdog.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/via_wdt.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/wdt_pci.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/xen_wdt.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-evtchn.ko 2024-01-24T17:29:54.167 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-gntalloc.ko 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-gntdev.ko 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-pciback 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-privcmd.ko 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xenfs 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/9p 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/9p/9p.ko 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/affs 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/affs/affs.ko 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/befs 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/befs/befs.ko 2024-01-24T17:29:54.168 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/binfmt_misc.ko 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/btrfs 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/btrfs/btrfs.ko 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/cachefiles 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/cachefiles/cachefiles.ko 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ceph 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ceph/ceph.ko 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/coda 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/coda/coda.ko 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/cramfs 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/cramfs/cramfs.ko 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/dlm 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/dlm/dlm.ko 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ecryptfs 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-24T17:29:54.169 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/f2fs 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/f2fs/f2fs.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fat 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fat/fat.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fat/msdos.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fat/vfat.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fscache 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fscache/fscache.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fuse 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fuse/cuse.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fuse/fuse.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/gfs2 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/gfs2/gfs2.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/hfs 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/hfs/hfs.ko 2024-01-24T17:29:54.170 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/hfsplus 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/hfsplus/hfsplus.ko 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/isofs 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/isofs/isofs.ko 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/jfs 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/jfs/jfs.ko 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/lockd 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/lockd/lockd.ko 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/minix 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/minix/minix.ko 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/netfs 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/netfs/netfs.ko 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/blocklayout 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/filelayout 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-24T17:29:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/flexfilelayout 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/nfs.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/nfsv3.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/nfsv4.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs_common 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs_common/grace.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs_common/nfs_acl.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfsd 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfsd/nfsd.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nilfs2 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nilfs2/nilfs2.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-celtic.ko 2024-01-24T17:29:54.172 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-centeuro.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-croatian.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-cyrillic.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-gaelic.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-greek.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-iceland.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-inuit.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-roman.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-romanian.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-turkish.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp1250.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp1251.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp1255.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp737.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp775.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp850.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp852.ko 2024-01-24T17:29:54.173 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp855.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp857.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp860.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp861.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp862.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp863.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp864.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp865.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp866.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp869.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp874.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp932.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp936.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp949.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp950.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_euc-jp.ko 2024-01-24T17:29:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-1.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-13.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-14.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-15.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-2.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-3.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-4.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-5.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-6.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-7.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-9.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_koi8-r.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_koi8-ru.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_koi8-u.ko 2024-01-24T17:29:54.175 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_utf8.ko 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/cluster 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/dlm 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/dlmfs 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/ocfs2.ko 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-24T17:29:54.176 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/overlayfs 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/overlayfs/overlay.ko 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/pstore 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/pstore/ramoops.ko 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/reiserfs 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/reiserfs/reiserfs.ko 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/romfs 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/romfs/romfs.ko 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/client 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/client/cifs.ko 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/common 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/common/cifs_arc4.ko 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/common/cifs_md4.ko 2024-01-24T17:29:54.177 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/squashfs 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/squashfs/squashfs.ko 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/sysv 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/sysv/sysv.ko 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/udf 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/udf/udf.ko 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ufs 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ufs/ufs.ko 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/xfs 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/xfs/xfs.ko 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/rcu 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/rcu/rcutorture.ko 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/torture.ko 2024-01-24T17:29:54.178 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/trace 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/asn1_encoder.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crc-itu-t.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crc8.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crypto 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crypto/libarc4.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crypto/libdes.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/libcrc32c.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/lru_cache.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/lz4 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/lz4/lz4_compress.ko 2024-01-24T17:29:54.179 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/lz4/lz4hc_compress.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/math 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/math/cordic.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/raid6 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/raid6/raid6_pq.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/reed_solomon 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/ts_bm.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/ts_fsm.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/ts_kmp.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/zstd 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/zstd/zstd_compress.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/mm 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/mm/hwpoison-inject.ko 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net 2024-01-24T17:29:54.180 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/6lowpan.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_dest.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_fragment.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_hop.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_mobility.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_routing.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_udp.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/garp.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/mrp.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/p8022.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/psnap.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/stp.ko 2024-01-24T17:29:54.181 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/8021q 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/8021q/8021q.ko 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p/9pnet.ko 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p/9pnet_fd.ko 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p/9pnet_rdma.ko 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p/9pnet_virtio.ko 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/batman-adv 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/batman-adv/batman-adv.ko 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/br_netfilter.ko 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/bridge.ko 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter 2024-01-24T17:29:54.182 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-24T17:29:54.183 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebtables.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ceph 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ceph/libceph.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/core 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/core/failover.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/core/pktgen.ko 2024-01-24T17:29:54.184 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp/dccp.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp/dccp_diag.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp/dccp_ipv4.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp/dccp_ipv6.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dns_resolver 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dns_resolver/dns_resolver.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/dsa_core.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/tag_brcm.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/tag_dsa.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/tag_none.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/tag_trailer.ko 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154/6lowpan 2024-01-24T17:29:54.185 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154/ieee802154.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ah4.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/esp4.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/fou.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/gre.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/inet_diag.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ip_gre.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ip_tunnel.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ip_vti.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ipcomp.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ipip.ko 2024-01-24T17:29:54.186 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-24T17:29:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_bic.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_dctcp.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_diag.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_highspeed.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_htcp.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_hybla.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_illinois.ko 2024-01-24T17:29:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_lp.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_scalable.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_vegas.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_veno.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_westwood.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_yeah.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tunnel4.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/udp_diag.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/udp_tunnel.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ah6.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/esp6.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/fou6.ko 2024-01-24T17:29:54.189 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ip6_tunnel.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ip6_vti.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ipcomp6.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-24T17:29:54.190 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-24T17:29:54.191 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/sit.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/tunnel6.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/key 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/key/af_key.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_core.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_eth.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_ip.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_ip6.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_netlink.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_ppp.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/llc 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/llc/llc.ko 2024-01-24T17:29:54.192 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/mac802154 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/mac802154/mac802154.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/mpls 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/mpls/mpls_gso.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-24T17:29:54.193 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-24T17:29:54.194 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conncount.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-24T17:29:54.195 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_flow_table.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_log_syslog.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_irc.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_sip.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-24T17:29:54.196 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_tables.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink_log.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_chain_nat.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_compat.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_ct.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_fib.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_hash.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_limit.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_log.ko 2024-01-24T17:29:54.197 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_masq.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_nat.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_numgen.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_queue.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_quota.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_redir.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_reject.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_reject_inet.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_socket.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_tunnel.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_AUDIT.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_CT.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_DSCP.ko 2024-01-24T17:29:54.198 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_HL.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_HMARK.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_LED.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_LOG.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_NETMAP.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_NFLOG.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_RATEEST.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_SECMARK.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TEE.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TPROXY.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TRACE.ko 2024-01-24T17:29:54.199 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_addrtype.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_bpf.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_cgroup.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_cluster.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_comment.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_connbytes.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_connlabel.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_connlimit.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_connmark.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_conntrack.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_cpu.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_dccp.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_devgroup.ko 2024-01-24T17:29:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_dscp.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_ecn.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_esp.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_hashlimit.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_helper.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_hl.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_ipcomp.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_iprange.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_ipvs.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_l2tp.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_length.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_limit.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_mac.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_mark.ko 2024-01-24T17:29:54.201 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_multiport.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_nat.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_nfacct.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_osf.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_owner.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_physdev.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_pkttype.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_policy.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_quota.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_rateest.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_realm.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_recent.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_sctp.ko 2024-01-24T17:29:54.202 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_set.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_socket.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_state.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_statistic.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_string.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_tcpmss.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_time.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_u32.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netlink 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netlink/netlink_diag.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/nsh 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/nsh/nsh.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch/openvswitch.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch/vport-geneve.ko 2024-01-24T17:29:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch/vport-gre.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch/vport-vxlan.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/packet 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/packet/af_packet_diag.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/rds 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/rds/rds.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/rds/rds_rdma.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/rds/rds_tcp.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_csum.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_gact.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_ipt.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_mirred.ko 2024-01-24T17:29:54.204 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_nat.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_pedit.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_police.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_simple.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_skbedit.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_vlan.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_basic.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_bpf.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_flow.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_fw.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_route.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_u32.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_cmp.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_ipset.ko 2024-01-24T17:29:54.205 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_meta.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_nbyte.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_text.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_u32.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_choke.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_codel.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_drr.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_fq.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_gred.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_hfsc.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_hhf.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_htb.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_ingress.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_mqprio.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_mqprio_lib.ko 2024-01-24T17:29:54.206 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_multiq.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_netem.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_pie.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_plug.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_prio.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_qfq.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_red.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_sfb.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_sfq.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_tbf.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_teql.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sctp 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sctp/sctp.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sctp/sctp_diag.ko 2024-01-24T17:29:54.207 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/auth_gss 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/sunrpc.ko 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/xprtrdma 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/tipc 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/tipc/diag.ko 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/tipc/tipc.ko 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/unix 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/unix/unix_diag.ko 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock 2024-01-24T17:29:54.208 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vsock.ko 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/xfrm 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys/encrypted-keys 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys/trusted-keys 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys/trusted-keys/trusted.ko 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/virt 2024-01-24T17:29:54.209 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/virt/lib 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/virt/lib/irqbypass.ko 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.alias 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.alias.bin 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.builtin 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.builtin.alias.bin 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.builtin.bin 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.builtin.modinfo 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.dep 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.dep.bin 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.devname 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.order 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.softdep 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.symbols 2024-01-24T17:29:54.210 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.7.0-gf2d346040895/modules.symbols.bin 2024-01-24T17:29:54.211 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-gf2d346040895 2024-01-24T17:29:54.211 DEBUG:teuthology.orchestra.run.smithi039:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-24T17:29:54.235 INFO:teuthology.orchestra.run.smithi039.stdout:postinstall scriptlet (using /bin/sh): 2024-01-24T17:29:54.235 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-gf2d346040895 -a -r /boot/System.map-6.7.0-gf2d346040895 ]; then 2024-01-24T17:29:54.235 INFO:teuthology.orchestra.run.smithi039.stdout:cp /boot/vmlinuz-6.7.0-gf2d346040895 /boot/.vmlinuz-6.7.0-gf2d346040895-rpm 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:cp /boot/System.map-6.7.0-gf2d346040895 /boot/.System.map-6.7.0-gf2d346040895-rpm 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:rm -f /boot/vmlinuz-6.7.0-gf2d346040895 /boot/System.map-6.7.0-gf2d346040895 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/installkernel 6.7.0-gf2d346040895 /boot/.vmlinuz-6.7.0-gf2d346040895-rpm /boot/.System.map-6.7.0-gf2d346040895-rpm 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:rm -f /boot/.vmlinuz-6.7.0-gf2d346040895-rpm /boot/.System.map-6.7.0-gf2d346040895-rpm 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:new-kernel-pkg --remove 6.7.0-gf2d346040895 --rminitrd --initrdfile=/boot/initramfs-6.7.0-gf2d346040895.img 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-install remove 6.7.0-gf2d346040895 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-24T17:29:54.236 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-24T17:29:54.237 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/update-bootloader --remove 6.7.0-gf2d346040895 2024-01-24T17:29:54.237 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-01-24T17:29:54.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 68.el9 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8264254 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-24T17:29:54.321 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:29:54.322 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2024-01-24T17:29:54.323 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-24T17:29:54.323 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 6.7.0-gf2d346040895 2024-01-24T17:29:54.323 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-24T17:29:54.903 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-01-24T17:29:56.283 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-24T17:29:56.312 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-01-24T17:29:56.314 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-01-24T17:29:56.329 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.PDDW1siHBx 2024-01-24T17:29:56.330 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PDDW1siHBx 2024-01-24T17:29:56.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.PDDW1siHBx 2024-01-24T17:29:56.556 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.PDDW1siHBx is 6KB 2024-01-24T17:29:56.605 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.PDDW1siHBx 2024-01-24T17:29:56.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2024-01-24T17:29:56.689 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-24T17:29:56.689 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-24T17:29:56.689 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-24T17:29:56.689 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-6.7.0-gf2d346040895.conf 2024-01-24T17:29:56.690 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-gf2d346040895 2024-01-24T17:29:56.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-01-24T17:29:57.286 INFO:teuthology.orchestra.run.smithi100.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-24T17:30:19.520 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-24T17:30:19.521 DEBUG:teuthology.orchestra.run.smithi100:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/boot/System.map-6.7.0-gf2d346040895 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/boot/config-6.7.0-gf2d346040895 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/boot/vmlinuz-6.7.0-gf2d346040895 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-24T17:30:19.594 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-24T17:30:19.595 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kernel 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kernel/cpu 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kernel/cpu/mce 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kvm 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/arch/x86/kvm/kvm.ko 2024-01-24T17:30:19.596 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/ansi_cprng.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/anubis.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/arc4.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_memcpy.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_pq.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_tx.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/async_xor.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/async_tx/raid6test.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/authenc.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/authencesn.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/blake2b_generic.ko 2024-01-24T17:30:19.597 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/blowfish_common.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/blowfish_generic.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/camellia_generic.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cast5_generic.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cast6_generic.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cast_common.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/ccm.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cmac.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/crc32_generic.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/crypto_engine.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/crypto_user.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/cts.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/des_generic.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/dh_generic.ko 2024-01-24T17:30:19.598 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/echainiv.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/essiv.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/fcrypt.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/khazad.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/lz4.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/lz4hc.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/md4.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/michael_mic.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/pcbc.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/pcrypt.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/rmd160.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/seed.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/serpent_generic.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/tcrypt.ko 2024-01-24T17:30:19.599 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/tea.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/twofish_common.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/twofish_generic.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/vmac.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/wp512.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/xcbc.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/xor.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/crypto/xxhash_generic.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/acpi_pad.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/custom_method.ko 2024-01-24T17:30:19.600 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/ec_sys.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/platform_profile.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/sbs.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/sbshc.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/acpi/video.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/acard-ahci.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/ahci_platform.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/ata_generic.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/libahci_platform.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_acpi.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_ali.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_amd.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_artop.ko 2024-01-24T17:30:19.601 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_atiixp.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_atp867x.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_cmd640.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_cmd64x.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_cypress.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_efar.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_hpt366.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_hpt37x.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_it8213.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_it821x.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_jmicron.ko 2024-01-24T17:30:19.602 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_marvell.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_mpiix.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_netcell.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_ninja32.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_ns87410.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_ns87415.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_oldpiix.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_opti.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_optidma.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_pcmcia.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-24T17:30:19.603 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_piccolo.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_rdc.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_sch.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_serverworks.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_sil680.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_sis.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_sl82c105.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_triflex.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pata_via.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/pdc_adma.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_inic162x.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_mv.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_nv.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_promise.ko 2024-01-24T17:30:19.604 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_qstor.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_sil.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_sil24.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_sis.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_svw.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_sx4.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_uli.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_via.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ata/sata_vsc.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/auxdisplay 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/auxdisplay/ks0108.ko 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/base 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/base/regmap 2024-01-24T17:30:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/bcma 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/bcma/bcma.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/aoe 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/aoe/aoe.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/brd.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/drbd 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/drbd/drbd.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/floppy.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/loop.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/mtip32xx 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-24T17:30:19.606 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/nbd.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/null_blk 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/null_blk/null_blk.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/pktcdvd.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/rbd.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/virtio_blk.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/xen-blkback 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/xen-blkfront.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/zram 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/block/zram/zram.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hangcheck-timer.ko 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random 2024-01-24T17:30:19.607 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/via-rng.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/lp.ko 2024-01-24T17:30:19.608 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/mwave 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/mwave/mwave.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/ppdev.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tlclk.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/uv_mmtimer.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/char/virtio_console.ko 2024-01-24T17:30:19.609 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/ccp 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/ccp/ccp.ko 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat/qat_common 2024-01-24T17:30:19.610 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/padlock-aes.ko 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/padlock-sha.ko 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/virtio 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dca 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dca/dca.ko 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/devfreq 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma/dw 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-24T17:30:19.611 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma/ioat 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/amd64_edac.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/e752x_edac.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/edac_mce_amd.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i3000_edac.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i3200_edac.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i5100_edac.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i5400_edac.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i7300_edac.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i7core_edac.ko 2024-01-24T17:30:19.612 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/i82975x_edac.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/ie31200_edac.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/sb_edac.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/edac/x38_edac.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/firmware 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/firmware/edd.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/gpio 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/gpio/gpio-ich.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-appleir.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-aureal.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-axff.ko 2024-01-24T17:30:19.613 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-dr.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-elecom.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-elo.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-emsff.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-gaff.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-gt683r.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-gyration.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-holtekff.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-hyperv.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-icade.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-keytouch.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-kye.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-lcpower.ko 2024-01-24T17:30:19.614 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-led.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-lenovo.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-multitouch.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-ortek.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-penmount.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-petalynx.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-picolcd.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-pl.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-plantronics.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-primax.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-rmi.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-24T17:30:19.615 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-common.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-roccat.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-saitek.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-samsung.ko 2024-01-24T17:30:19.616 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-sjoy.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-sony.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-speedlink.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-steelseries.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-sunplus.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-tivo.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-tmff.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-topseed.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-twinhan.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-uclogic.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-waltop.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-wiimote.ko 2024-01-24T17:30:19.617 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-xinmo.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-zpff.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/hid-zydacron.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/uhid.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hid/wacom.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hv 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hv/hv_balloon.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hv/hv_utils.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hv/hv_vmbus.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/abituguru.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/abituguru3.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ad7414.ko 2024-01-24T17:30:19.618 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ad7418.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adc128d818.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm1025.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm1026.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm1029.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm1031.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adm9240.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ads7828.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7410.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7411.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7462.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7470.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7475.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/adt7x10.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/amc6821.ko 2024-01-24T17:30:19.619 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/applesmc.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/asb100.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/asc7621.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/atxp1.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/coretemp.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/dme1737.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ds1621.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ds620.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/emc1403.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/emc6w201.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/f71805f.ko 2024-01-24T17:30:19.620 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/f71882fg.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/f75375s.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/fam15h_power.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/fschmd.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/g760a.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/g762.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/gl518sm.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/gl520sm.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/i5500_temp.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/i5k_amb.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ibmaem.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ibmpex.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ina209.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ina2xx.ko 2024-01-24T17:30:19.621 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/it87.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/k10temp.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/k8temp.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lineage-pem.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm63.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm73.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm75.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm77.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm78.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm80.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm83.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm85.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm87.ko 2024-01-24T17:30:19.622 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm90.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm92.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm93.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm95234.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm95241.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/lm95245.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc2945.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4151.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4215.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4222.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4245.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4260.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/ltc4261.ko 2024-01-24T17:30:19.623 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max16065.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max1619.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max1668.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max197.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max6639.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max6650.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/max6697.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/mcp3021.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/nct6683.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/nct6775-core.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/nct6775.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/nct7802.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pc87360.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pc87427.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pcf8591.ko 2024-01-24T17:30:19.624 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/powr1220.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sch5627.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sch5636.ko 2024-01-24T17:30:19.625 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sht15.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sht21.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/shtc1.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/sis5595.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/smsc47b397.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/smsc47m1.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/smsc47m192.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/thmc50.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/tmp102.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/tmp103.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/tmp401.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/tmp421.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/via-cputemp.ko 2024-01-24T17:30:19.626 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/via686a.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/vt1211.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/vt8231.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83627ehf.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83627hf.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83781d.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83791d.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83792d.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83793.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83795.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83l785ts.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/hwmon/w83l786ng.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/algos 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-24T17:30:19.627 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-24T17:30:19.628 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-dev.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-mux.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-smbus.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/i2c/i2c-stub.ko 2024-01-24T17:30:19.629 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/ib_core.ko 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/cxgb4 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mlx4 2024-01-24T17:30:19.630 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mlx5 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mthca 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/ipoib 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/iser 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/isert 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/srp 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-24T17:30:19.631 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/srpt 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/gameport 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/gameport/gameport.ko 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/joydev.ko 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/keyboard 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/appletouch.ko 2024-01-24T17:30:19.632 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/bcm5974.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/cyapatp.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/sermouse.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/rmi4 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio/altera_ps2.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio/arc_ps2.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/serio/serio_raw.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/input/sparse-keymap.ko 2024-01-24T17:30:19.633 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/leds-blinkm.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/leds-lm3530.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/leds-lp3944.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/leds-ss4200.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-24T17:30:19.634 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/bcache 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/bcache/bcache.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-bio-prison.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-cache-smq.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-cache.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-crypt.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-delay.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-flakey.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-log-userspace.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-multipath.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-queue-length.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-raid.ko 2024-01-24T17:30:19.635 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-round-robin.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-service-time.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-switch.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-thin-pool.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/dm-verity.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/faulty.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/linear.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/multipath.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/persistent-data 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/raid0.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/raid1.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/raid10.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/md/raid456.ko 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message 2024-01-24T17:30:19.636 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptbase.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptctl.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptfc.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptlan.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptsas.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptscsih.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/message/fusion/mptspi.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/lpc_ich.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/lpc_sch.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/mfd-core.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/sm501.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/viperboard.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/vx855.ko 2024-01-24T17:30:19.637 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/mfd/wl1273-core.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/altera-stapl 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/apds9802als.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/apds990x.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/bh1770glc.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/cb710 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/cb710/cb710.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/eeprom 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/eeprom/at24.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/eeprom/max6875.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/enclosure.ko 2024-01-24T17:30:19.638 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/hpilo.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/isl29003.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/isl29020.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/lis3lv02d 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/mei 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/mei/mei-me.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/mei/mei-txe.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/mei/mei.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-gru 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-xp 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-24T17:30:19.639 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/tifm_7xx1.ko 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/tifm_core.ko 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/tsl2550.ko 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/vmw_balloon.ko 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/vmw_vmci 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/bonding 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/bonding/bonding.ko 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/b53 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-24T17:30:19.640 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/mv88e6xxx 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/dummy.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/eql.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-24T17:30:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/adaptec 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/agere 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/alteon 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/altera 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/amd 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-24T17:30:19.642 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/alx 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atlx 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-24T17:30:19.643 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/genet 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/brocade 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/brocade/bna 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-24T17:30:19.644 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/cisco 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/cisco/enic 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip 2024-01-24T17:30:19.645 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dlink 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/dnet.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/emulex 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/emulex/benet 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/ethoc.ko 2024-01-24T17:30:19.646 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/fealnx.ko 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e1000 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e1000e 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/fm10k 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/i40e 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/iavf 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-24T17:30:19.647 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/igb 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/igbvf 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/jme.ko 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/marvell 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-24T17:30:19.648 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/micrel 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/myricom 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/natsemi 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-24T17:30:19.649 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/neterion 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/nvidia 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/packetengines 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/rdc 2024-01-24T17:30:19.650 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/rocker 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sfc 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/silan 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sis 2024-01-24T17:30:19.651 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/stmicro 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-24T17:30:19.652 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/tehuti 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/ti 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/via 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/wiznet 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/xircom 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-24T17:30:19.653 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/geneve.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/hyperv 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ieee802154 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ifb.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ipvlan 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/macvlan.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/macvtap.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-24T17:30:19.654 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/mii.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/net_failover.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/netconsole.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/nlmon.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/pcs 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/amd.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/bcm87xx.ko 2024-01-24T17:30:19.655 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/broadcom.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/cicada.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/davicom.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/dp83640.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/et1011c.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/icplus.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/lxt.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/marvell.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/micrel.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/national.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/phylink.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/qsemi.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/realtek.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/smsc.ko 2024-01-24T17:30:19.656 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/ste10Xp.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/phy/vitesse.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_async.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/pppoe.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/pppox.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/ppp/pptp.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/slip 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/slip/slhc.ko 2024-01-24T17:30:19.657 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/slip/slip.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/sungem_phy.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/tap.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_random.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/tun.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/veth.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/virtio_net.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/vmxnet3 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/vxlan 2024-01-24T17:30:19.658 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/vxlan/vxlan.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/xen-netback 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/net/xen-netfront.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ntb 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ntb/ntb.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/nvme 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/nvme/host 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/nvme/host/nvme-core.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/nvme/host/nvme.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport/parport.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport/parport_cs.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport/parport_pc.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/parport/parport_serial.ko 2024-01-24T17:30:19.659 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/hotplug 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/pcie 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pci/xen-pcifront.ko 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pcmcia 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pcmcia/i82092.ko 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pcmcia/pd6729.ko 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pinctrl 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pinctrl/intel 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform 2024-01-24T17:30:19.660 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/acerhdf.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/hdaps.ko 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/intel 2024-01-24T17:30:19.661 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/intel_ips.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/platform/x86/wmi.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/powercap 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-24T17:30:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/clients 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/clients/pps_parport.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/pps/pps_core.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ptp 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ptp/ptp.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ptp/ptp_kvm.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-24T17:30:19.663 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-em3027.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-24T17:30:19.664 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-max6900.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/rtc/rtc-x1205.ko 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi 2024-01-24T17:30:19.665 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/3w-9xxx.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/3w-sas.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/3w-xxxx.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/BusLogic.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/a100u2w.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aacraid 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/advansys.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aic7xxx 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/am53c974.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/arcmsr 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/atp870u.ko 2024-01-24T17:30:19.666 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/be2iscsi 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bfa 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bfa/bfa.ko 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bnx2fc 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bnx2i 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/ch.ko 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/csiostor 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-24T17:30:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/dc395x.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/dmx3191d.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/esas2r 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/esp_scsi.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fcoe 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-24T17:30:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fnic 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/fnic/fnic.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/hpsa.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/hptiop.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/hv_storvsc.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/initio.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/ipr.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/ips.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/isci 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/isci/isci.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libfc 2024-01-24T17:30:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libfc/libfc.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libiscsi.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libsas 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/libsas/libsas.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/lpfc 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mpt3sas 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mvsas 2024-01-24T17:30:19.670 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/mvumi.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/pm8001 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/pmcraid.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla1280.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla2xxx 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla4xxx 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/raid_class.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_debug.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-24T17:30:19.671 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/ses.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/st.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/stex.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/sym53c8xx_2 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/virtio_scsi.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/scsi/wd719x.ko 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ssb 2024-01-24T17:30:19.672 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ssb/ssb.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/iscsi 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/loopback 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_file.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_iblock.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_mod.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_pscsi.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/target_core_user.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/tcm_fc 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel 2024-01-24T17:30:19.673 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-24T17:30:19.674 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/ipwireless 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/n_gsm.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/n_hdlc.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/nozomi.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/8250 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/arc_uart.ko 2024-01-24T17:30:19.675 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/jsm 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs/core 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs/host 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio.ko 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio_aec.ko 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio_cif.ko 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio_pci_generic.ko 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/uio/uio_sercos3.ko 2024-01-24T17:30:19.676 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class/cdc-acm.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class/usblp.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/class/usbtmc.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/host 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/image 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/image/mdc800.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/image/microtek.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/adutux.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/appledisplay.ko 2024-01-24T17:30:19.677 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/emi26.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/emi62.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/ezusb.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/idmouse.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/iowarrior.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/ldusb.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/legousbtower.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/sisusbvga 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/usb3503.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/usblcd.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/uss720.ko 2024-01-24T17:30:19.678 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/misc/yurex.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/phy 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/phy/phy-generic.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/aircable.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ark3116.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ch341.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/cp210x.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/cyberjack.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/empeg.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-24T17:30:19.679 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/io_ti.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ipaq.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ipw.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ir-usb.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/keyspan.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/mct_u232.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/mos7720.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/mos7840.ko 2024-01-24T17:30:19.680 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/navman.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/omninet.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/opticon.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/option.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/oti6858.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/pl2303.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/qcaux.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/qcserial.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/quatech2.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/safe_serial.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/sierra.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ssu100.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/symbolserial.ko 2024-01-24T17:30:19.681 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/usb_debug.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/visor.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/whiteheat.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/uas.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-24T17:30:19.682 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-karma.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/storage/usb-storage.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/usbip 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio 2024-01-24T17:30:19.683 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/pci 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/vfio.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost/vhost.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost/vhost_net.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/vhost/vhost_scsi.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video/backlight 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video/backlight/apple_bl.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video/backlight/lcd.ko 2024-01-24T17:30:19.684 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_balloon.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_mmio.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_pci.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/virtio/virtio_ring.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/masters 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/masters/ds2482.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/masters/ds2490.ko 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves 2024-01-24T17:30:19.685 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/w1/wire.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-24T17:30:19.686 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/hpwdt.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/i6300esb.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/ib700wdt.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/ibmasr.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/it87_wdt.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/machzwd.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/nv_tco.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-24T17:30:19.687 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/softdog.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/via_wdt.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/wdt_pci.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/watchdog/xen_wdt.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-evtchn.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-gntalloc.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-gntdev.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-pciback 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-24T17:30:19.688 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xen-privcmd.ko 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xenfs 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/9p 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/9p/9p.ko 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/affs 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/affs/affs.ko 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/befs 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/befs/befs.ko 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/binfmt_misc.ko 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/btrfs 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/btrfs/btrfs.ko 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/cachefiles 2024-01-24T17:30:19.689 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/cachefiles/cachefiles.ko 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ceph 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ceph/ceph.ko 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/coda 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/coda/coda.ko 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/cramfs 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/cramfs/cramfs.ko 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/dlm 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/dlm/dlm.ko 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ecryptfs 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/f2fs 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/f2fs/f2fs.ko 2024-01-24T17:30:19.690 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fat 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fat/fat.ko 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fat/msdos.ko 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fat/vfat.ko 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fscache 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fscache/fscache.ko 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fuse 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fuse/cuse.ko 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/fuse/fuse.ko 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/gfs2 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/gfs2/gfs2.ko 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/hfs 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/hfs/hfs.ko 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/hfsplus 2024-01-24T17:30:19.691 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/hfsplus/hfsplus.ko 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/isofs 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/isofs/isofs.ko 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/jfs 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/jfs/jfs.ko 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/lockd 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/lockd/lockd.ko 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/minix 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/minix/minix.ko 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/netfs 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/netfs/netfs.ko 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs 2024-01-24T17:30:19.692 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/blocklayout 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/filelayout 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/flexfilelayout 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/nfs.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/nfsv3.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs/nfsv4.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs_common 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs_common/grace.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfs_common/nfs_acl.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfsd 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nfsd/nfsd.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nilfs2 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nilfs2/nilfs2.ko 2024-01-24T17:30:19.693 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-celtic.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-centeuro.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-croatian.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-cyrillic.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-gaelic.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-greek.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-iceland.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-inuit.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-roman.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-romanian.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/mac-turkish.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp1250.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp1251.ko 2024-01-24T17:30:19.694 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp1255.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp737.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp775.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp850.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp852.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp855.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp857.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp860.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp861.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp862.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp863.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp864.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp865.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp866.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp869.ko 2024-01-24T17:30:19.695 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp874.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp932.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp936.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp949.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_cp950.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_euc-jp.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-1.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-13.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-14.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-15.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-2.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-3.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-4.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-5.ko 2024-01-24T17:30:19.696 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-6.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-7.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_iso8859-9.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_koi8-r.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_koi8-ru.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_koi8-u.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/nls/nls_utf8.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/cluster 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/dlm 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-24T17:30:19.697 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/dlmfs 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/ocfs2.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/overlayfs 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/overlayfs/overlay.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/pstore 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/pstore/ramoops.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/reiserfs 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/reiserfs/reiserfs.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/romfs 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/romfs/romfs.ko 2024-01-24T17:30:19.698 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/client 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/client/cifs.ko 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/common 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/common/cifs_arc4.ko 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/smb/common/cifs_md4.ko 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/squashfs 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/squashfs/squashfs.ko 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/sysv 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/sysv/sysv.ko 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/udf 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/udf/udf.ko 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ufs 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/ufs/ufs.ko 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/xfs 2024-01-24T17:30:19.699 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/fs/xfs/xfs.ko 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/rcu 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/rcu/rcutorture.ko 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/torture.ko 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/trace 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/asn1_encoder.ko 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crc-itu-t.ko 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crc8.ko 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crypto 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crypto/libarc4.ko 2024-01-24T17:30:19.700 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/crypto/libdes.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/libcrc32c.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/lru_cache.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/lz4 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/lz4/lz4_compress.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/lz4/lz4hc_compress.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/math 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/math/cordic.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/raid6 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/raid6/raid6_pq.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/reed_solomon 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/ts_bm.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/ts_fsm.ko 2024-01-24T17:30:19.701 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/ts_kmp.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/zstd 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/lib/zstd/zstd_compress.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/mm 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/mm/hwpoison-inject.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/6lowpan.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_dest.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_fragment.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_hop.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_mobility.ko 2024-01-24T17:30:19.702 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_routing.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/6lowpan/nhc_udp.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/garp.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/mrp.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/p8022.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/psnap.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/802/stp.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/8021q 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/8021q/8021q.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p/9pnet.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p/9pnet_fd.ko 2024-01-24T17:30:19.703 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p/9pnet_rdma.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/9p/9pnet_virtio.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/batman-adv 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/batman-adv/batman-adv.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/br_netfilter.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/bridge.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-24T17:30:19.704 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/ebtables.ko 2024-01-24T17:30:19.705 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ceph 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ceph/libceph.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/core 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/core/failover.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/core/pktgen.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp/dccp.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp/dccp_diag.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp/dccp_ipv4.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dccp/dccp_ipv6.ko 2024-01-24T17:30:19.706 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dns_resolver 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dns_resolver/dns_resolver.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/dsa_core.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/tag_brcm.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/tag_dsa.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/tag_none.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/dsa/tag_trailer.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154/6lowpan 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154/ieee802154.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4 2024-01-24T17:30:19.707 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ah4.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/esp4.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/fou.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/gre.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/inet_diag.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ip_gre.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ip_tunnel.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ip_vti.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ipcomp.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/ipip.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-24T17:30:19.708 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-24T17:30:19.709 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_bic.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_dctcp.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_diag.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_highspeed.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_htcp.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_hybla.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_illinois.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_lp.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_scalable.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_vegas.ko 2024-01-24T17:30:19.710 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_veno.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_westwood.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tcp_yeah.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/tunnel4.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/udp_diag.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/udp_tunnel.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ah6.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/esp6.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/fou6.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ip6_tunnel.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ip6_vti.ko 2024-01-24T17:30:19.711 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/ipcomp6.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-24T17:30:19.712 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/sit.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/tunnel6.ko 2024-01-24T17:30:19.713 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/key 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/key/af_key.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_core.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_eth.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_ip.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_ip6.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_netlink.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/l2tp/l2tp_ppp.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/llc 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/llc/llc.ko 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/mac802154 2024-01-24T17:30:19.714 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/mac802154/mac802154.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/mpls 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/mpls/mpls_gso.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-24T17:30:19.715 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-24T17:30:19.716 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conncount.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-24T17:30:19.717 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_flow_table.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_log_syslog.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-24T17:30:19.718 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_irc.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_sip.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nf_tables.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink_log.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_chain_nat.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_compat.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_ct.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-24T17:30:19.719 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_fib.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_hash.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_limit.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_log.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_masq.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_nat.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_numgen.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_queue.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_quota.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_redir.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_reject.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_reject_inet.ko 2024-01-24T17:30:19.720 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_socket.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/nft_tunnel.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_AUDIT.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_CT.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_DSCP.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_HL.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_HMARK.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_LED.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_LOG.ko 2024-01-24T17:30:19.721 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_NETMAP.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_NFLOG.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_RATEEST.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_SECMARK.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TEE.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TPROXY.ko 2024-01-24T17:30:19.722 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_TRACE.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_addrtype.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_bpf.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_cgroup.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_cluster.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_comment.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_connbytes.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_connlabel.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_connlimit.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_connmark.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_conntrack.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_cpu.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_dccp.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_devgroup.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_dscp.ko 2024-01-24T17:30:19.723 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_ecn.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_esp.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_hashlimit.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_helper.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_hl.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_ipcomp.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_iprange.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_ipvs.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_l2tp.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_length.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_limit.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_mac.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_mark.ko 2024-01-24T17:30:19.724 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_multiport.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_nat.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_nfacct.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_osf.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_owner.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_physdev.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_pkttype.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_policy.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_quota.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_rateest.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_realm.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_recent.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_sctp.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_set.ko 2024-01-24T17:30:19.725 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_socket.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_state.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_statistic.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_string.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_tcpmss.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_time.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netfilter/xt_u32.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netlink 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/netlink/netlink_diag.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/nsh 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/nsh/nsh.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch/openvswitch.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch/vport-geneve.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch/vport-gre.ko 2024-01-24T17:30:19.726 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/openvswitch/vport-vxlan.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/packet 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/packet/af_packet_diag.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/rds 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/rds/rds.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/rds/rds_rdma.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/rds/rds_tcp.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_csum.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_gact.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_ipt.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_mirred.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_nat.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_pedit.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_police.ko 2024-01-24T17:30:19.727 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_simple.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_skbedit.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/act_vlan.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_basic.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_bpf.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_flow.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_fw.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_route.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/cls_u32.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_cmp.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_ipset.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_meta.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_nbyte.ko 2024-01-24T17:30:19.728 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_text.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/em_u32.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_choke.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_codel.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_drr.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_fq.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_gred.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_hfsc.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_hhf.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_htb.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_ingress.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_mqprio.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_mqprio_lib.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_multiq.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_netem.ko 2024-01-24T17:30:19.729 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_pie.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_plug.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_prio.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_qfq.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_red.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_sfb.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_sfq.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_tbf.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sched/sch_teql.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sctp 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sctp/sctp.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sctp/sctp_diag.ko 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/auth_gss 2024-01-24T17:30:19.730 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/sunrpc.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/xprtrdma 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/tipc 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/tipc/diag.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/tipc/tipc.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/unix 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/unix/unix_diag.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vsock.ko 2024-01-24T17:30:19.731 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/xfrm 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys/encrypted-keys 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys/trusted-keys 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/security/keys/trusted-keys/trusted.ko 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/virt 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/virt/lib 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/kernel/virt/lib/irqbypass.ko 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.alias 2024-01-24T17:30:19.732 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.alias.bin 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.builtin 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.builtin.alias.bin 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.builtin.bin 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.builtin.modinfo 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.dep 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.dep.bin 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.devname 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.order 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.softdep 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.symbols 2024-01-24T17:30:19.733 INFO:teuthology.orchestra.run.smithi100.stdout:/lib/modules/6.7.0-gf2d346040895/modules.symbols.bin 2024-01-24T17:30:19.734 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-gf2d346040895 2024-01-24T17:30:19.734 DEBUG:teuthology.orchestra.run.smithi100:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:postinstall scriptlet (using /bin/sh): 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-gf2d346040895 -a -r /boot/System.map-6.7.0-gf2d346040895 ]; then 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:cp /boot/vmlinuz-6.7.0-gf2d346040895 /boot/.vmlinuz-6.7.0-gf2d346040895-rpm 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:cp /boot/System.map-6.7.0-gf2d346040895 /boot/.System.map-6.7.0-gf2d346040895-rpm 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:rm -f /boot/vmlinuz-6.7.0-gf2d346040895 /boot/System.map-6.7.0-gf2d346040895 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/installkernel 6.7.0-gf2d346040895 /boot/.vmlinuz-6.7.0-gf2d346040895-rpm /boot/.System.map-6.7.0-gf2d346040895-rpm 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:rm -f /boot/.vmlinuz-6.7.0-gf2d346040895-rpm /boot/.System.map-6.7.0-gf2d346040895-rpm 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:fi 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:new-kernel-pkg --remove 6.7.0-gf2d346040895 --rminitrd --initrdfile=/boot/initramfs-6.7.0-gf2d346040895.img 2024-01-24T17:30:19.756 INFO:teuthology.orchestra.run.smithi100.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-24T17:30:19.757 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-install remove 6.7.0-gf2d346040895 2024-01-24T17:30:19.757 INFO:teuthology.orchestra.run.smithi100.stdout:fi 2024-01-24T17:30:19.757 INFO:teuthology.orchestra.run.smithi100.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-24T17:30:19.757 INFO:teuthology.orchestra.run.smithi100.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-24T17:30:19.757 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/update-bootloader --remove 6.7.0-gf2d346040895 2024-01-24T17:30:19.757 INFO:teuthology.orchestra.run.smithi100.stdout:fi 2024-01-24T17:30:19.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-01-24T17:30:19.831 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-01-24T17:30:19.831 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-01-24T17:30:19.831 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 68.el9 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8264254 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-24T17:30:19.832 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-01-24T17:30:19.833 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-01-24T17:30:19.833 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:30:19.833 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-24T17:30:19.833 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-24T17:30:19.833 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-24T17:30:19.833 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-01-24T17:30:19.833 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:30:19.833 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-01-24T17:30:19.834 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-24T17:30:19.834 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 6.7.0-gf2d346040895 2024-01-24T17:30:19.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-24T17:30:20.435 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-01-24T17:30:21.857 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-24T17:30:21.880 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-01-24T17:30:21.883 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-01-24T17:30:21.897 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.Lh3VvVBQdZ 2024-01-24T17:30:21.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Lh3VvVBQdZ 2024-01-24T17:30:21.966 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.Lh3VvVBQdZ 2024-01-24T17:30:22.102 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.Lh3VvVBQdZ is 6KB 2024-01-24T17:30:22.150 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.Lh3VvVBQdZ 2024-01-24T17:30:22.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-01-24T17:30:22.228 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-24T17:30:22.228 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-24T17:30:22.228 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-24T17:30:22.228 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-6.7.0-gf2d346040895.conf 2024-01-24T17:30:22.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-gf2d346040895 2024-01-24T17:30:22.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-01-24T17:30:26.849 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-24T17:30:26.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-01-24T17:30:26.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:30:52.403 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-24T17:30:52.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-01-24T17:30:52.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:31:26.884 DEBUG:teuthology.orchestra.remote:timed out 2024-01-24T17:31:26.908 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-01-24T17:31:35.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-01-24T17:31:35.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:31:35.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-01-24T17:31:35.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:31:54.300 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-01-24T17:31:54.300 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-01-24T17:32:06.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-01-24T17:32:06.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:32:06.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-01-24T17:32:06.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:32:08.872 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-01-24T17:32:09.372 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-01-24T17:32:09.510 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-01-24T17:32:09.511 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.7.0-gf2d346040895"... 2024-01-24T17:32:09.511 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-01-24T17:32:09.554 INFO:teuthology.orchestra.run.smithi100.stdout:6.7.0-gf2d346040895 2024-01-24T17:32:09.554 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-gf2d346040895 vs 6.7.0-gf2d346040895 2024-01-24T17:32:09.554 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-24T17:32:09.554 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-24T17:32:10.555 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-24T17:32:10.555 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-24T17:32:10.730 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-01-24T17:32:10.794 DEBUG:teuthology.parallel:result is None 2024-01-24T17:32:24.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-01-24T17:32:24.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:32:24.838 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-01-24T17:32:25.367 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-01-24T17:32:25.368 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.7.0-gf2d346040895"... 2024-01-24T17:32:25.368 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-01-24T17:32:25.406 INFO:teuthology.orchestra.run.smithi039.stdout:6.7.0-gf2d346040895 2024-01-24T17:32:25.406 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-gf2d346040895 vs 6.7.0-gf2d346040895 2024-01-24T17:32:25.406 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-24T17:32:25.406 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-24T17:32:26.407 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-24T17:32:26.408 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-24T17:32:26.563 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-01-24T17:32:26.601 DEBUG:teuthology.parallel:result is None 2024-01-24T17:32:26.602 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-24T17:32:26.608 INFO:teuthology.task.internal:Creating test directory... 2024-01-24T17:32:26.608 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-24T17:32:26.615 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-24T17:32:26.668 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-24T17:32:26.678 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-24T17:32:26.711 INFO:teuthology.task.internal:Creating archive directory... 2024-01-24T17:32:26.712 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-24T17:32:26.718 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-24T17:32:26.779 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-24T17:32:26.787 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-24T17:32:26.787 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-24T17:32:26.792 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-24T17:32:26.864 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T17:32:26.876 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T17:32:26.901 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T17:32:26.914 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T17:32:26.918 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-24T17:32:26.925 INFO:teuthology.task.internal:Configuring sudo... 2024-01-24T17:32:26.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-24T17:32:26.931 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-24T17:32:27.023 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-24T17:32:27.034 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-24T17:32:27.034 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-24T17:32:27.039 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-24T17:32:27.085 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T17:32:27.158 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T17:32:27.260 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T17:32:27.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T17:32:27.405 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:32:27.405 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-24T17:32:27.472 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T17:32:27.555 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T17:32:27.720 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T17:32:27.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T17:32:27.867 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:32:27.867 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-24T17:32:27.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-01-24T17:32:27.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-01-24T17:32:28.099 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T17:32:28.155 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T17:32:28.468 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-24T17:32:28.477 INFO:teuthology.task.internal:Starting timer... 2024-01-24T17:32:28.477 INFO:teuthology.run_tasks:Running task pcp... 2024-01-24T17:32:28.525 INFO:teuthology.run_tasks:Running task selinux... 2024-01-24T17:32:28.559 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2024-01-24T17:32:28.742 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2024-01-24T17:32:28.748 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-01-24T17:32:28.884 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-01-24T17:32:28.889 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-24T17:32:28.889 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2024-01-24T17:32:28.964 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2024-01-24T17:32:28.965 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-01-24T17:32:29.028 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-01-24T17:32:29.028 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi100.front.sepia.ceph.com': 'permissive'} 2024-01-24T17:32:29.028 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T17:32:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:32:29.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T17:32:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:32:29.261 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-24T17:32:29.261 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2024-01-24T17:32:29.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-01-24T17:32:29.525 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-24T17:32:29.535 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-24T17:32:29.552 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-24T17:32:29.553 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com,smithi100.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-24T17:38:22.891 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi100.front.sepia.ceph.com')] 2024-01-24T17:38:22.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-01-24T17:38:22.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:38:23.067 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-01-24T17:38:23.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-01-24T17:38:23.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-01-24T17:38:23.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T17:38:23.365 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-01-24T17:38:23.511 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-01-24T17:38:23.512 INFO:teuthology.run_tasks:Running task clock... 2024-01-24T17:38:23.524 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-24T17:38:23.525 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-24T17:38:23.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T17:38:23.533 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-24T17:38:23.533 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T17:38:23.633 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-24T17:38:23.638 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-24T17:38:23.682 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-24T17:38:23.697 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-24T17:38:23.774 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-01-24T17:38:23.797 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2024-01-24T17:38:23.815 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-01-24T17:38:23.841 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2024-01-24T17:38:23.865 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-24T17:38:23.897 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-24T17:38:23.917 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-24T17:38:23.953 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-24T17:38:24.041 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-01-24T17:38:24.049 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T17:38:24.049 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-01-24T17:38:24.049 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:38:24.050 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:38:24.050 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:38:24.050 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:38:24.076 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-01-24T17:38:24.085 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T17:38:24.086 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-01-24T17:38:24.086 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:38:24.086 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:38:24.086 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:38:24.086 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:38:24.087 INFO:teuthology.run_tasks:Running task install... 2024-01-24T17:38:24.096 DEBUG:teuthology.task.install:project ceph 2024-01-24T17:38:24.096 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'}} 2024-01-24T17:38:24.096 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'} 2024-01-24T17:38:24.096 INFO:teuthology.task.install:Using flavor: default 2024-01-24T17:38:24.102 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-24T17:38:24.102 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-24T17:38:24.103 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'tag': None, 'wait_for_package': False} 2024-01-24T17:38:24.103 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T17:38:24.105 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'tag': None, 'wait_for_package': False} 2024-01-24T17:38:24.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T17:38:24.249 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/9/flavors/default/ 2024-01-24T17:38:24.249 INFO:teuthology.task.install.rpm:Package version is 18.2.1-341.g4e3182a6 2024-01-24T17:38:24.376 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-24T17:38:24.376 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:38:24.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-24T17:38:24.405 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/9/flavors/default/ 2024-01-24T17:38:24.405 INFO:teuthology.task.install.rpm:Package version is 18.2.1-341.g4e3182a6 2024-01-24T17:38:24.483 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-24T17:38:24.483 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3182a65800aed6d4aa2f6b86da3cfd72188134/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-24T17:38:24.629 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-24T17:38:24.664 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-24T17:38:24.664 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:38:24.665 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-24T17:38:24.766 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-24T17:38:24.766 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3182a65800aed6d4aa2f6b86da3cfd72188134/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-24T17:38:24.774 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-24T17:38:24.868 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2024-01-24T17:38:24.872 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2024-01-24T17:38:24.905 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-24T17:38:25.032 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-24T17:38:25.126 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2024-01-24T17:38:25.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2024-01-24T17:38:25.242 INFO:teuthology.orchestra.run.smithi039.stdout:74 files removed 2024-01-24T17:38:25.282 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-24T17:38:25.484 INFO:teuthology.orchestra.run.smithi100.stdout:74 files removed 2024-01-24T17:38:25.524 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-24T17:38:26.417 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 127 kB/s | 77 kB 00:00 2024-01-24T17:38:26.494 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 169 kB/s | 77 kB 00:00 2024-01-24T17:38:26.892 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2024-01-24T17:38:27.011 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2024-01-24T17:38:27.459 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-01-24T17:38:27.570 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-01-24T17:38:28.157 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2024-01-24T17:38:28.229 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 9.2 MB/s | 6.4 MB 00:00 2024-01-24T17:38:29.808 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-01-24T17:38:29.830 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-01-24T17:38:31.221 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 8.0 MB/s | 8.0 MB 00:00 2024-01-24T17:38:31.759 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-01-24T17:38:34.093 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-01-24T17:38:37.304 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 19 MB 00:04 2024-01-24T17:38:38.528 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.9 MB 00:00 2024-01-24T17:38:40.553 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-01-24T17:38:42.227 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-01-24T17:38:42.438 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 5.9 MB 00:01 2024-01-24T17:38:44.089 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 15 kB 00:00 2024-01-24T17:38:45.124 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2024-01-24T17:38:48.246 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-01-24T17:38:50.824 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-01-24T17:38:51.103 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-24T17:38:51.105 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-24T17:38:51.123 INFO:teuthology.orchestra.run.smithi039.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-24T17:38:51.124 INFO:teuthology.orchestra.run.smithi039.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-24T17:38:51.126 INFO:teuthology.orchestra.run.smithi039.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-24T17:38:51.127 INFO:teuthology.orchestra.run.smithi039.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-24T17:38:51.130 INFO:teuthology.orchestra.run.smithi039.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-24T17:38:51.133 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-24T17:38:51.134 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-24T17:38:51.220 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-01-24T17:38:51.245 INFO:teuthology.orchestra.run.smithi039.stdout:=========================================================================================== 2024-01-24T17:38:51.245 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-01-24T17:38:51.245 INFO:teuthology.orchestra.run.smithi039.stdout:=========================================================================================== 2024-01-24T17:38:51.245 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-01-24T17:38:51.245 INFO:teuthology.orchestra.run.smithi039.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-24T17:38:51.245 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 6.5 k 2024-01-24T17:38:51.245 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 5.2 M 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 840 k 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 142 k 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 1.4 M 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 132 k 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 1.8 M 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 7.4 M 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 50 k 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 7.7 M 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 40 M 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 262 k 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 221 k 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 219 k 2024-01-24T17:38:51.246 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 18 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 31 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 659 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 127 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 162 k 2024-01-24T17:38:51.247 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 319 k 2024-01-24T17:38:51.248 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 298 k 2024-01-24T17:38:51.248 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 99 k 2024-01-24T17:38:51.248 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 86 k 2024-01-24T17:38:51.248 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 3.0 M 2024-01-24T17:38:51.248 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 171 k 2024-01-24T17:38:51.248 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-24T17:38:51.248 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-01-24T17:38:51.248 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager x86_64 1:1.45.10-1.el9 CentOS-BaseOS 2.2 M 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-libnm x86_64 1:1.45.10-1.el9 CentOS-BaseOS 1.8 M 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-team x86_64 1:1.45.10-1.el9 CentOS-BaseOS 42 k 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-tui x86_64 1:1.45.10-1.el9 CentOS-BaseOS 246 k 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 3.3 M 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 3.0 M 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-24T17:38:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 18 M 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 23 k 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 2.1 M 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 242 k 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 4.5 M 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 18 M 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 15 k 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 25 k 2024-01-24T17:38:51.250 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 165 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-24T17:38:51.251 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 475 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 4.5 M 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-24T17:38:51.252 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 45 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 124 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.26.2-1.el9 epel 202 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-24T17:38:51.253 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-24T17:38:51.254 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-24T17:38:51.255 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-24T17:38:51.256 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout:=========================================================================================== 2024-01-24T17:38:51.257 INFO:teuthology.orchestra.run.smithi039.stdout:Install 133 Packages 2024-01-24T17:38:51.258 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 16 Packages 2024-01-24T17:38:51.258 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:38:51.258 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 200 M 2024-01-24T17:38:51.258 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-01-24T17:38:51.898 INFO:teuthology.orchestra.run.smithi039.stdout:(1/149): ceph-18.2.1-341.g4e3182a6.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2024-01-24T17:38:52.440 INFO:teuthology.orchestra.run.smithi039.stdout:(2/149): ceph-fuse-18.2.1-341.g4e3182a6.el9.x86 1.5 MB/s | 840 kB 00:00 2024-01-24T17:38:52.540 INFO:teuthology.orchestra.run.smithi039.stdout:(3/149): ceph-immutable-object-cache-18.2.1-341 1.4 MB/s | 142 kB 00:00 2024-01-24T17:38:52.957 INFO:teuthology.orchestra.run.smithi039.stdout:(4/149): ceph-common-18.2.1-341.g4e3182a6.el9.x 14 MB/s | 18 MB 00:01 2024-01-24T17:38:53.082 INFO:teuthology.orchestra.run.smithi039.stdout:(5/149): ceph-mgr-18.2.1-341.g4e3182a6.el9.x86_ 12 MB/s | 1.4 MB 00:00 2024-01-24T17:38:53.290 INFO:teuthology.orchestra.run.smithi039.stdout:(6/149): ceph-mds-18.2.1-341.g4e3182a6.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2024-01-24T17:38:53.416 INFO:teuthology.orchestra.run.smithi039.stdout:(7/149): ceph-base-18.2.1-341.g4e3182a6.el9.x86 2.9 MB/s | 5.2 MB 00:01 2024-01-24T17:38:53.549 INFO:teuthology.orchestra.run.smithi039.stdout:(8/149): ceph-mon-18.2.1-341.g4e3182a6.el9.x86_ 9.6 MB/s | 4.5 MB 00:00 2024-01-24T17:38:53.591 INFO:teuthology.orchestra.run.smithi039.stdout:(9/149): ceph-selinux-18.2.1-341.g4e3182a6.el9. 621 kB/s | 25 kB 00:00 2024-01-24T17:38:53.610 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-24T17:38:53.612 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-24T17:38:53.630 INFO:teuthology.orchestra.run.smithi100.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-24T17:38:53.632 INFO:teuthology.orchestra.run.smithi100.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-24T17:38:53.633 INFO:teuthology.orchestra.run.smithi100.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-24T17:38:53.634 INFO:teuthology.orchestra.run.smithi100.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-24T17:38:53.637 INFO:teuthology.orchestra.run.smithi100.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-24T17:38:53.640 INFO:teuthology.orchestra.run.smithi100.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-24T17:38:53.641 INFO:teuthology.orchestra.run.smithi100.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-24T17:38:53.723 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout:=========================================================================================== 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout:=========================================================================================== 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 6.5 k 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 5.2 M 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 840 k 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 142 k 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 1.4 M 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 132 k 2024-01-24T17:38:53.751 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 1.8 M 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 7.4 M 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 50 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 7.7 M 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 40 M 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 262 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 221 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: cephfs-mirror x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 219 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: cephfs-top noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 18 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 31 k 2024-01-24T17:38:53.752 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 659 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 127 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 162 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 319 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 298 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 99 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 86 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 3.0 M 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 171 k 2024-01-24T17:38:53.753 INFO:teuthology.orchestra.run.smithi100.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager x86_64 1:1.45.10-1.el9 CentOS-BaseOS 2.2 M 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-libnm x86_64 1:1.45.10-1.el9 CentOS-BaseOS 1.8 M 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-team x86_64 1:1.45.10-1.el9 CentOS-BaseOS 42 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-tui x86_64 1:1.45.10-1.el9 CentOS-BaseOS 246 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 3.3 M 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 3.0 M 2024-01-24T17:38:53.754 INFO:teuthology.orchestra.run.smithi100.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 18 M 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 23 k 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 2.1 M 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 242 k 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 4.5 M 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 18 M 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:18.2.1-341.g4e3182a6.el9 ceph-noarch 15 k 2024-01-24T17:38:53.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 25 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 165 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-24T17:38:53.756 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 475 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 4.5 M 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-24T17:38:53.757 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 45 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:18.2.1-341.g4e3182a6.el9 ceph 124 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:2.26.2-1.el9 epel 202 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-24T17:38:53.758 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-24T17:38:53.759 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-24T17:38:53.760 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-24T17:38:53.761 INFO:teuthology.orchestra.run.smithi100.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout:Installing weak dependencies: 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout:=========================================================================================== 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout:Install 133 Packages 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 16 Packages 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:38:53.762 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 200 M 2024-01-24T17:38:53.763 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-01-24T17:38:54.650 INFO:teuthology.orchestra.run.smithi039.stdout:(10/149): ceph-radosgw-18.2.1-341.g4e3182a6.el9 6.3 MB/s | 7.7 MB 00:01 2024-01-24T17:38:54.659 INFO:teuthology.orchestra.run.smithi100.stdout:(1/149): ceph-18.2.1-341.g4e3182a6.el9.x86_64.r 15 kB/s | 6.5 kB 00:00 2024-01-24T17:38:54.800 INFO:teuthology.orchestra.run.smithi039.stdout:(11/149): cephfs-mirror-18.2.1-341.g4e3182a6.el 1.4 MB/s | 219 kB 00:00 2024-01-24T17:38:54.900 INFO:teuthology.orchestra.run.smithi039.stdout:(12/149): libcephfs-devel-18.2.1-341.g4e3182a6. 314 kB/s | 31 kB 00:00 2024-01-24T17:38:55.542 INFO:teuthology.orchestra.run.smithi100.stdout:(2/149): ceph-fuse-18.2.1-341.g4e3182a6.el9.x86 952 kB/s | 840 kB 00:00 2024-01-24T17:38:55.567 INFO:teuthology.orchestra.run.smithi039.stdout:(13/149): libcephfs2-18.2.1-341.g4e3182a6.el9.x 990 kB/s | 659 kB 00:00 2024-01-24T17:38:55.667 INFO:teuthology.orchestra.run.smithi039.stdout:(14/149): libcephsqlite-18.2.1-341.g4e3182a6.el 1.6 MB/s | 165 kB 00:00 2024-01-24T17:38:55.692 INFO:teuthology.orchestra.run.smithi100.stdout:(3/149): ceph-immutable-object-cache-18.2.1-341 957 kB/s | 142 kB 00:00 2024-01-24T17:38:56.176 INFO:teuthology.orchestra.run.smithi039.stdout:(15/149): ceph-test-18.2.1-341.g4e3182a6.el9.x8 15 MB/s | 40 MB 00:02 2024-01-24T17:38:56.209 INFO:teuthology.orchestra.run.smithi039.stdout:(16/149): librados-devel-18.2.1-341.g4e3182a6.e 234 kB/s | 127 kB 00:00 2024-01-24T17:38:56.260 INFO:teuthology.orchestra.run.smithi039.stdout:(17/149): libradosstriper1-18.2.1-341.g4e3182a6 5.6 MB/s | 475 kB 00:00 2024-01-24T17:38:56.292 INFO:teuthology.orchestra.run.smithi100.stdout:(4/149): ceph-base-18.2.1-341.g4e3182a6.el9.x86 2.5 MB/s | 5.2 MB 00:02 2024-01-24T17:38:56.318 INFO:teuthology.orchestra.run.smithi039.stdout:(18/149): python3-ceph-argparse-18.2.1-341.g4e3 784 kB/s | 45 kB 00:00 2024-01-24T17:38:56.359 INFO:teuthology.orchestra.run.smithi100.stdout:(5/149): ceph-mds-18.2.1-341.g4e3182a6.el9.x86_ 3.2 MB/s | 2.1 MB 00:00 2024-01-24T17:38:56.385 INFO:teuthology.orchestra.run.smithi039.stdout:(19/149): python3-ceph-common-18.2.1-341.g4e318 1.9 MB/s | 124 kB 00:00 2024-01-24T17:38:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:(6/149): ceph-mgr-18.2.1-341.g4e3182a6.el9.x86_ 5.0 MB/s | 1.4 MB 00:00 2024-01-24T17:38:56.619 INFO:teuthology.orchestra.run.smithi039.stdout:(20/149): ceph-osd-18.2.1-341.g4e3182a6.el9.x86 5.3 MB/s | 18 MB 00:03 2024-01-24T17:38:56.653 INFO:teuthology.orchestra.run.smithi039.stdout:(21/149): python3-cephfs-18.2.1-341.g4e3182a6.e 610 kB/s | 162 kB 00:00 2024-01-24T17:38:56.719 INFO:teuthology.orchestra.run.smithi039.stdout:(22/149): python3-rbd-18.2.1-341.g4e3182a6.el9. 4.4 MB/s | 298 kB 00:00 2024-01-24T17:38:56.761 INFO:teuthology.orchestra.run.smithi039.stdout:(23/149): python3-rados-18.2.1-341.g4e3182a6.el 2.2 MB/s | 319 kB 00:00 2024-01-24T17:38:56.795 INFO:teuthology.orchestra.run.smithi039.stdout:(24/149): python3-rgw-18.2.1-341.g4e3182a6.el9. 1.3 MB/s | 99 kB 00:00 2024-01-24T17:38:56.828 INFO:teuthology.orchestra.run.smithi039.stdout:(25/149): rbd-fuse-18.2.1-341.g4e3182a6.el9.x86 1.3 MB/s | 86 kB 00:00 2024-01-24T17:38:56.937 INFO:teuthology.orchestra.run.smithi039.stdout:(26/149): librgw2-18.2.1-341.g4e3182a6.el9.x86_ 6.1 MB/s | 4.5 MB 00:00 2024-01-24T17:38:56.971 INFO:teuthology.orchestra.run.smithi039.stdout:(27/149): rbd-nbd-18.2.1-341.g4e3182a6.el9.x86_ 1.2 MB/s | 171 kB 00:00 2024-01-24T17:38:57.037 INFO:teuthology.orchestra.run.smithi039.stdout:(28/149): ceph-grafana-dashboards-18.2.1-341.g4 233 kB/s | 23 kB 00:00 2024-01-24T17:38:57.121 INFO:teuthology.orchestra.run.smithi039.stdout:(29/149): rbd-mirror-18.2.1-341.g4e3182a6.el9.x 9.2 MB/s | 3.0 MB 00:00 2024-01-24T17:38:57.157 INFO:teuthology.orchestra.run.smithi039.stdout:(30/149): ceph-mgr-cephadm-18.2.1-341.g4e3182a6 712 kB/s | 132 kB 00:00 2024-01-24T17:38:57.543 INFO:teuthology.orchestra.run.smithi100.stdout:(7/149): ceph-mon-18.2.1-341.g4e3182a6.el9.x86_ 3.8 MB/s | 4.5 MB 00:01 2024-01-24T17:38:57.691 INFO:teuthology.orchestra.run.smithi039.stdout:(31/149): ceph-mgr-dashboard-18.2.1-341.g4e3182 2.7 MB/s | 1.8 MB 00:00 2024-01-24T17:38:57.824 INFO:teuthology.orchestra.run.smithi039.stdout:(32/149): ceph-mgr-diskprediction-local-18.2.1- 11 MB/s | 7.4 MB 00:00 2024-01-24T17:38:57.850 INFO:teuthology.orchestra.run.smithi039.stdout:(33/149): ceph-mgr-rook-18.2.1-341.g4e3182a6.el 320 kB/s | 50 kB 00:00 2024-01-24T17:38:57.875 INFO:teuthology.orchestra.run.smithi039.stdout:(34/149): ceph-prometheus-alerts-18.2.1-341.g4e 298 kB/s | 15 kB 00:00 2024-01-24T17:38:57.900 INFO:teuthology.orchestra.run.smithi039.stdout:(35/149): ceph-mgr-modules-core-18.2.1-341.g4e3 327 kB/s | 242 kB 00:00 2024-01-24T17:38:57.925 INFO:teuthology.orchestra.run.smithi039.stdout:(36/149): ceph-volume-18.2.1-341.g4e3182a6.el9. 3.4 MB/s | 262 kB 00:00 2024-01-24T17:38:57.959 INFO:teuthology.orchestra.run.smithi039.stdout:(37/149): cephadm-18.2.1-341.g4e3182a6.el9.noar 2.6 MB/s | 221 kB 00:00 2024-01-24T17:38:58.051 INFO:teuthology.orchestra.run.smithi039.stdout:(38/149): cephfs-top-18.2.1-341.g4e3182a6.el9.n 121 kB/s | 18 kB 00:00 2024-01-24T17:38:58.226 INFO:teuthology.orchestra.run.smithi039.stdout:(39/149): bison-3.7.4-5.el9.x86_64.rpm 4.6 MB/s | 920 kB 00:00 2024-01-24T17:38:58.277 INFO:teuthology.orchestra.run.smithi039.stdout:(40/149): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 662 kB 00:00 2024-01-24T17:38:58.302 INFO:teuthology.orchestra.run.smithi039.stdout:(41/149): boost-program-options-1.75.0-8.el9.x8 543 kB/s | 106 kB 00:00 2024-01-24T17:38:58.327 INFO:teuthology.orchestra.run.smithi039.stdout:(42/149): elfutils-libelf-devel-0.190-2.el9.x86 209 kB/s | 20 kB 00:00 2024-01-24T17:38:58.361 INFO:teuthology.orchestra.run.smithi039.stdout:(43/149): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 307 kB 00:00 2024-01-24T17:38:58.394 INFO:teuthology.orchestra.run.smithi039.stdout:(44/149): flexiblas-3.0.4-8.el9.x86_64.rpm 501 kB/s | 32 kB 00:00 2024-01-24T17:38:58.410 INFO:teuthology.orchestra.run.smithi100.stdout:(8/149): ceph-osd-18.2.1-341.g4e3182a6.el9.x86_ 9.7 MB/s | 18 MB 00:01 2024-01-24T17:38:58.477 INFO:teuthology.orchestra.run.smithi100.stdout:(9/149): ceph-selinux-18.2.1-341.g4e3182a6.el9. 384 kB/s | 25 kB 00:00 2024-01-24T17:38:58.478 INFO:teuthology.orchestra.run.smithi039.stdout:(45/149): flexiblas-openblas-openmp-3.0.4-8.el9 148 kB/s | 17 kB 00:00 2024-01-24T17:38:58.586 INFO:teuthology.orchestra.run.smithi039.stdout:(46/149): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-01-24T17:38:58.611 INFO:teuthology.orchestra.run.smithi039.stdout:(47/149): libaio-devel-0.3.111-13.el9.x86_64.rp 48 kB/s | 10 kB 00:00 2024-01-24T17:38:58.645 INFO:teuthology.orchestra.run.smithi039.stdout:(48/149): libpmemobj-1.12.1-1.el9.x86_64.rpm 985 kB/s | 159 kB 00:00 2024-01-24T17:38:58.671 INFO:teuthology.orchestra.run.smithi039.stdout:(49/149): librabbitmq-0.11.0-7.el9.x86_64.rpm 788 kB/s | 44 kB 00:00 2024-01-24T17:38:58.694 INFO:teuthology.orchestra.run.smithi100.stdout:(10/149): ceph-radosgw-18.2.1-341.g4e3182a6.el9 6.7 MB/s | 7.7 MB 00:01 2024-01-24T17:38:58.738 INFO:teuthology.orchestra.run.smithi039.stdout:(50/149): librdkafka-1.6.1-102.el9.x86_64.rpm 5.3 MB/s | 661 kB 00:00 2024-01-24T17:38:58.777 INFO:teuthology.orchestra.run.smithi100.stdout:(11/149): cephfs-mirror-18.2.1-341.g4e3182a6.el 2.6 MB/s | 219 kB 00:00 2024-01-24T17:38:58.779 INFO:teuthology.orchestra.run.smithi039.stdout:(51/149): libtool-2.4.6-45.el9.x86_64.rpm 4.3 MB/s | 578 kB 00:00 2024-01-24T17:38:58.796 INFO:teuthology.orchestra.run.smithi039.stdout:(52/149): libuuid-devel-2.37.4-15.el9.x86_64.rp 465 kB/s | 24 kB 00:00 2024-01-24T17:38:58.813 INFO:teuthology.orchestra.run.smithi039.stdout:(53/149): libzstd-devel-1.5.1-2.el9.x86_64.rpm 671 kB/s | 46 kB 00:00 2024-01-24T17:38:58.853 INFO:teuthology.orchestra.run.smithi100.stdout:(12/149): libcephfs-devel-18.2.1-341.g4e3182a6. 419 kB/s | 31 kB 00:00 2024-01-24T17:38:58.855 INFO:teuthology.orchestra.run.smithi039.stdout:(54/149): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-01-24T17:38:58.889 INFO:teuthology.orchestra.run.smithi039.stdout:(55/149): openblas-0.3.21-2.el9.x86_64.rpm 476 kB/s | 34 kB 00:00 2024-01-24T17:38:58.986 INFO:teuthology.orchestra.run.smithi100.stdout:(13/149): libcephfs2-18.2.1-341.g4e3182a6.el9.x 4.9 MB/s | 659 kB 00:00 2024-01-24T17:38:59.047 INFO:teuthology.orchestra.run.smithi039.stdout:(56/149): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-01-24T17:38:59.062 INFO:teuthology.orchestra.run.smithi100.stdout:(14/149): libcephsqlite-18.2.1-341.g4e3182a6.el 2.2 MB/s | 165 kB 00:00 2024-01-24T17:38:59.145 INFO:teuthology.orchestra.run.smithi100.stdout:(15/149): librados-devel-18.2.1-341.g4e3182a6.e 1.5 MB/s | 127 kB 00:00 2024-01-24T17:38:59.156 INFO:teuthology.orchestra.run.smithi039.stdout:(57/149): openssl-devel-3.0.7-25.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-01-24T17:38:59.215 INFO:teuthology.orchestra.run.smithi039.stdout:(58/149): postgresql-13.11-1.el9.x86_64.rpm 5.0 MB/s | 1.5 MB 00:00 2024-01-24T17:38:59.256 INFO:teuthology.orchestra.run.smithi039.stdout:(59/149): postgresql-contrib-13.11-1.el9.x86_64 4.2 MB/s | 811 kB 00:00 2024-01-24T17:38:59.270 INFO:teuthology.orchestra.run.smithi100.stdout:(16/149): libradosstriper1-18.2.1-341.g4e3182a6 3.8 MB/s | 475 kB 00:00 2024-01-24T17:38:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:(60/149): postgresql-private-libs-13.11-1.el9.x 934 kB/s | 134 kB 00:00 2024-01-24T17:38:59.582 INFO:teuthology.orchestra.run.smithi039.stdout:(61/149): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-24T17:38:59.824 INFO:teuthology.orchestra.run.smithi039.stdout:(62/149): postgresql-server-13.11-1.el9.x86_64. 10 MB/s | 5.7 MB 00:00 2024-01-24T17:38:59.849 INFO:teuthology.orchestra.run.smithi039.stdout:(63/149): python3-jinja2-2.11.3-4.el9.noarch.rp 422 kB/s | 228 kB 00:00 2024-01-24T17:38:59.891 INFO:teuthology.orchestra.run.smithi039.stdout:(64/149): python3-jsonpatch-1.21-16.el9.noarch. 81 kB/s | 24 kB 00:00 2024-01-24T17:38:59.916 INFO:teuthology.orchestra.run.smithi039.stdout:(65/149): python3-jsonpointer-2.0-4.el9.noarch. 276 kB/s | 18 kB 00:00 2024-01-24T17:38:59.937 INFO:teuthology.orchestra.run.smithi100.stdout:(17/149): librgw2-18.2.1-341.g4e3182a6.el9.x86_ 6.7 MB/s | 4.5 MB 00:00 2024-01-24T17:38:59.941 INFO:teuthology.orchestra.run.smithi039.stdout:(66/149): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-01-24T17:38:59.984 INFO:teuthology.orchestra.run.smithi039.stdout:(67/149): python3-markupsafe-1.1.1-12.el9.x86_6 351 kB/s | 32 kB 00:00 2024-01-24T17:39:00.013 INFO:teuthology.orchestra.run.smithi100.stdout:(18/149): python3-ceph-argparse-18.2.1-341.g4e3 608 kB/s | 45 kB 00:00 2024-01-24T17:39:00.017 INFO:teuthology.orchestra.run.smithi039.stdout:(68/149): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 217 kB 00:00 2024-01-24T17:39:00.088 INFO:teuthology.orchestra.run.smithi100.stdout:(19/149): python3-ceph-common-18.2.1-341.g4e318 1.6 MB/s | 124 kB 00:00 2024-01-24T17:39:00.151 INFO:teuthology.orchestra.run.smithi039.stdout:(69/149): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-01-24T17:39:00.172 INFO:teuthology.orchestra.run.smithi100.stdout:(20/149): python3-cephfs-18.2.1-341.g4e3182a6.e 1.9 MB/s | 162 kB 00:00 2024-01-24T17:39:00.176 INFO:teuthology.orchestra.run.smithi039.stdout:(70/149): python3-oauthlib-3.1.1-5.el9.noarch.r 915 kB/s | 174 kB 00:00 2024-01-24T17:39:00.201 INFO:teuthology.orchestra.run.smithi039.stdout:(71/149): python3-prettytable-0.7.2-27.el9.noar 251 kB/s | 41 kB 00:00 2024-01-24T17:39:00.243 INFO:teuthology.orchestra.run.smithi039.stdout:(72/149): python3-pyasn1-modules-0.4.8-6.el9.no 5.2 MB/s | 211 kB 00:00 2024-01-24T17:39:00.264 INFO:teuthology.orchestra.run.smithi100.stdout:(21/149): python3-rados-18.2.1-341.g4e3182a6.el 3.4 MB/s | 319 kB 00:00 2024-01-24T17:39:00.268 INFO:teuthology.orchestra.run.smithi039.stdout:(73/149): python3-pytz-2021.1-5.el9.noarch.rpm 729 kB/s | 47 kB 00:00 2024-01-24T17:39:00.294 INFO:teuthology.orchestra.run.smithi039.stdout:(74/149): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-01-24T17:39:00.344 INFO:teuthology.orchestra.run.smithi039.stdout:(75/149): python3-requests-oauthlib-1.3.0-12.el 583 kB/s | 43 kB 00:00 2024-01-24T17:39:00.356 INFO:teuthology.orchestra.run.smithi100.stdout:(22/149): python3-rbd-18.2.1-341.g4e3182a6.el9. 3.2 MB/s | 298 kB 00:00 2024-01-24T17:39:00.431 INFO:teuthology.orchestra.run.smithi100.stdout:(23/149): python3-rgw-18.2.1-341.g4e3182a6.el9. 1.3 MB/s | 99 kB 00:00 2024-01-24T17:39:00.490 INFO:teuthology.orchestra.run.smithi100.stdout:(24/149): rbd-fuse-18.2.1-341.g4e3182a6.el9.x86 1.5 MB/s | 86 kB 00:00 2024-01-24T17:39:00.511 INFO:teuthology.orchestra.run.smithi039.stdout:(76/149): python3-toml-0.10.2-6.el9.noarch.rpm 179 kB/s | 39 kB 00:00 2024-01-24T17:39:00.711 INFO:teuthology.orchestra.run.smithi039.stdout:(77/149): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-01-24T17:39:00.736 INFO:teuthology.orchestra.run.smithi039.stdout:(78/149): socat-1.7.4.1-5.el9.x86_64.rpm 821 kB/s | 300 kB 00:00 2024-01-24T17:39:00.761 INFO:teuthology.orchestra.run.smithi039.stdout:(79/149): uuid-1.6.2-55.el9.x86_64.rpm 232 kB/s | 56 kB 00:00 2024-01-24T17:39:00.804 INFO:teuthology.orchestra.run.smithi039.stdout:(80/149): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-01-24T17:39:00.829 INFO:teuthology.orchestra.run.smithi039.stdout:(81/149): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.3 MB/s | 87 kB 00:00 2024-01-24T17:39:00.854 INFO:teuthology.orchestra.run.smithi039.stdout:(82/149): zlib-devel-1.2.11-40.el9.x86_64.rpm 488 kB/s | 44 kB 00:00 2024-01-24T17:39:00.879 INFO:teuthology.orchestra.run.smithi039.stdout:(83/149): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-01-24T17:39:00.913 INFO:teuthology.orchestra.run.smithi039.stdout:(84/149): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-01-24T17:39:00.946 INFO:teuthology.orchestra.run.smithi039.stdout:(85/149): libgfortran-11.4.1-3.el9.x86_64.rpm 8.7 MB/s | 802 kB 00:00 2024-01-24T17:39:00.980 INFO:teuthology.orchestra.run.smithi039.stdout:(86/149): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 194 kB 00:00 2024-01-24T17:39:01.042 INFO:teuthology.orchestra.run.smithi039.stdout:(87/149): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-01-24T17:39:01.072 INFO:teuthology.orchestra.run.smithi039.stdout:(88/149): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 241 kB 00:00 2024-01-24T17:39:01.073 INFO:teuthology.orchestra.run.smithi100.stdout:(25/149): ceph-common-18.2.1-341.g4e3182a6.el9. 2.7 MB/s | 18 MB 00:06 2024-01-24T17:39:01.131 INFO:teuthology.orchestra.run.smithi039.stdout:(89/149): python3-cryptography-36.0.1-4.el9.x86 7.7 MB/s | 1.1 MB 00:00 2024-01-24T17:39:01.156 INFO:teuthology.orchestra.run.smithi039.stdout:(90/149): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-01-24T17:39:01.157 INFO:teuthology.orchestra.run.smithi100.stdout:(26/149): rbd-mirror-18.2.1-341.g4e3182a6.el9.x 4.5 MB/s | 3.0 MB 00:00 2024-01-24T17:39:01.181 INFO:teuthology.orchestra.run.smithi039.stdout:(91/149): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2024-01-24T17:39:01.191 INFO:teuthology.orchestra.run.smithi100.stdout:(27/149): rbd-nbd-18.2.1-341.g4e3182a6.el9.x86_ 1.4 MB/s | 171 kB 00:00 2024-01-24T17:39:01.207 INFO:teuthology.orchestra.run.smithi039.stdout:(92/149): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2024-01-24T17:39:01.224 INFO:teuthology.orchestra.run.smithi100.stdout:(28/149): ceph-grafana-dashboards-18.2.1-341.g4 352 kB/s | 23 kB 00:00 2024-01-24T17:39:01.240 INFO:teuthology.orchestra.run.smithi039.stdout:(93/149): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-01-24T17:39:01.258 INFO:teuthology.orchestra.run.smithi100.stdout:(29/149): ceph-mgr-cephadm-18.2.1-341.g4e3182a6 1.9 MB/s | 132 kB 00:00 2024-01-24T17:39:01.474 INFO:teuthology.orchestra.run.smithi039.stdout:(94/149): python3-devel-3.9.17-2.el9.x86_64.rpm 846 kB/s | 245 kB 00:00 2024-01-24T17:39:01.809 INFO:teuthology.orchestra.run.smithi100.stdout:(30/149): ceph-test-18.2.1-341.g4e3182a6.el9.x8 12 MB/s | 40 MB 00:03 2024-01-24T17:39:01.850 INFO:teuthology.orchestra.run.smithi039.stdout:(95/149): fmt-8.1.1-5.el9.x86_64.rpm 173 kB/s | 111 kB 00:00 2024-01-24T17:39:01.900 INFO:teuthology.orchestra.run.smithi100.stdout:(31/149): ceph-mgr-dashboard-18.2.1-341.g4e3182 2.6 MB/s | 1.8 MB 00:00 2024-01-24T17:39:01.934 INFO:teuthology.orchestra.run.smithi100.stdout:(32/149): ceph-mgr-modules-core-18.2.1-341.g4e3 1.9 MB/s | 242 kB 00:00 2024-01-24T17:39:01.958 INFO:teuthology.orchestra.run.smithi039.stdout:(96/149): libarrow-doc-9.0.0-10.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-01-24T17:39:01.967 INFO:teuthology.orchestra.run.smithi100.stdout:(33/149): ceph-mgr-rook-18.2.1-341.g4e3182a6.el 766 kB/s | 50 kB 00:00 2024-01-24T17:39:02.001 INFO:teuthology.orchestra.run.smithi100.stdout:(34/149): ceph-prometheus-alerts-18.2.1-341.g4e 220 kB/s | 15 kB 00:00 2024-01-24T17:39:02.017 INFO:teuthology.orchestra.run.smithi039.stdout:(97/149): gperftools-libs-2.9.1-2.el9.x86_64.rp 398 kB/s | 309 kB 00:00 2024-01-24T17:39:02.076 INFO:teuthology.orchestra.run.smithi100.stdout:(35/149): ceph-volume-18.2.1-341.g4e3182a6.el9. 2.4 MB/s | 262 kB 00:00 2024-01-24T17:39:02.083 INFO:teuthology.orchestra.run.smithi039.stdout:(98/149): liboath-2.6.7-2.el9.x86_64.rpm 390 kB/s | 49 kB 00:00 2024-01-24T17:39:02.101 INFO:teuthology.orchestra.run.smithi100.stdout:(36/149): cephadm-18.2.1-341.g4e3182a6.el9.noar 2.2 MB/s | 221 kB 00:00 2024-01-24T17:39:02.134 INFO:teuthology.orchestra.run.smithi039.stdout:(99/149): libunwind-1.6.2-1.el9.x86_64.rpm 580 kB/s | 67 kB 00:00 2024-01-24T17:39:02.210 INFO:teuthology.orchestra.run.smithi100.stdout:(37/149): cephfs-top-18.2.1-341.g4e3182a6.el9.n 136 kB/s | 18 kB 00:00 2024-01-24T17:39:02.409 INFO:teuthology.orchestra.run.smithi039.stdout:(100/149): python3-asyncssh-2.13.2-3.el9.noarch 2.0 MB/s | 548 kB 00:00 2024-01-24T17:39:02.461 INFO:teuthology.orchestra.run.smithi100.stdout:(38/149): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 662 kB 00:00 2024-01-24T17:39:02.510 INFO:teuthology.orchestra.run.smithi100.stdout:(39/149): bison-3.7.4-5.el9.x86_64.rpm 3.0 MB/s | 920 kB 00:00 2024-01-24T17:39:02.536 INFO:teuthology.orchestra.run.smithi100.stdout:(40/149): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-01-24T17:39:02.561 INFO:teuthology.orchestra.run.smithi100.stdout:(41/149): elfutils-libelf-devel-0.190-2.el9.x86 417 kB/s | 20 kB 00:00 2024-01-24T17:39:02.568 INFO:teuthology.orchestra.run.smithi039.stdout:(101/149): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.7 MB/s | 838 kB 00:00 2024-01-24T17:39:02.593 INFO:teuthology.orchestra.run.smithi039.stdout:(102/149): python3-bcrypt-3.2.2-1.el9.x86_64.rp 238 kB/s | 43 kB 00:00 2024-01-24T17:39:02.603 INFO:teuthology.orchestra.run.smithi100.stdout:(42/149): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-01-24T17:39:02.628 INFO:teuthology.orchestra.run.smithi100.stdout:(43/149): flexiblas-3.0.4-8.el9.x86_64.rpm 478 kB/s | 32 kB 00:00 2024-01-24T17:39:02.710 INFO:teuthology.orchestra.run.smithi039.stdout:(103/149): libarrow-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-01-24T17:39:02.735 INFO:teuthology.orchestra.run.smithi039.stdout:(104/149): python3-certifi-2021.10.8-2.el9.noar 106 kB/s | 15 kB 00:00 2024-01-24T17:39:02.761 INFO:teuthology.orchestra.run.smithi039.stdout:(105/149): python3-cachetools-4.2.4-1.el9.noarc 168 kB/s | 32 kB 00:00 2024-01-24T17:39:02.770 INFO:teuthology.orchestra.run.smithi100.stdout:(44/149): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-01-24T17:39:02.796 INFO:teuthology.orchestra.run.smithi100.stdout:(45/149): flexiblas-openblas-openmp-3.0.4-8.el9 117 kB/s | 17 kB 00:00 2024-01-24T17:39:02.838 INFO:teuthology.orchestra.run.smithi100.stdout:(46/149): libaio-devel-0.3.111-13.el9.x86_64.rp 248 kB/s | 10 kB 00:00 2024-01-24T17:39:02.861 INFO:teuthology.orchestra.run.smithi039.stdout:(106/149): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-01-24T17:39:02.863 INFO:teuthology.orchestra.run.smithi100.stdout:(47/149): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-01-24T17:39:02.894 INFO:teuthology.orchestra.run.smithi039.stdout:(107/149): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-01-24T17:39:02.899 INFO:teuthology.orchestra.run.smithi100.stdout:(48/149): librabbitmq-0.11.0-7.el9.x86_64.rpm 751 kB/s | 44 kB 00:00 2024-01-24T17:39:02.936 INFO:teuthology.orchestra.run.smithi039.stdout:(108/149): python3-google-auth-2.26.2-1.el9.noa 1.1 MB/s | 202 kB 00:00 2024-01-24T17:39:02.942 INFO:teuthology.orchestra.run.smithi100.stdout:(49/149): librdkafka-1.6.1-102.el9.x86_64.rpm 8.3 MB/s | 661 kB 00:00 2024-01-24T17:39:02.982 INFO:teuthology.orchestra.run.smithi100.stdout:(50/149): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-01-24T17:39:02.986 INFO:teuthology.orchestra.run.smithi039.stdout:(109/149): python3-jaraco-8.2.1-3.el9.noarch.rp 86 kB/s | 11 kB 00:00 2024-01-24T17:39:03.008 INFO:teuthology.orchestra.run.smithi100.stdout:(51/149): libuuid-devel-2.37.4-15.el9.x86_64.rp 415 kB/s | 24 kB 00:00 2024-01-24T17:39:03.012 INFO:teuthology.orchestra.run.smithi039.stdout:(110/149): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-01-24T17:39:03.033 INFO:teuthology.orchestra.run.smithi100.stdout:(52/149): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-01-24T17:39:03.045 INFO:teuthology.orchestra.run.smithi039.stdout:(111/149): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-01-24T17:39:03.067 INFO:teuthology.orchestra.run.smithi100.stdout:(53/149): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-01-24T17:39:03.092 INFO:teuthology.orchestra.run.smithi100.stdout:(54/149): openblas-0.3.21-2.el9.x86_64.rpm 614 kB/s | 34 kB 00:00 2024-01-24T17:39:03.121 INFO:teuthology.orchestra.run.smithi039.stdout:(112/149): python3-jaraco-functools-3.5.0-2.el9 146 kB/s | 19 kB 00:00 2024-01-24T17:39:03.146 INFO:teuthology.orchestra.run.smithi039.stdout:(113/149): python3-jaraco-text-3.2.0-6.el9.noar 147 kB/s | 20 kB 00:00 2024-01-24T17:39:03.180 INFO:teuthology.orchestra.run.smithi039.stdout:(114/149): python3-jwt+crypto-2.4.0-1.el9.noarc 68 kB/s | 9.0 kB 00:00 2024-01-24T17:39:03.230 INFO:teuthology.orchestra.run.smithi039.stdout:(115/149): python3-jwt-2.4.0-1.el9.noarch.rpm 378 kB/s | 41 kB 00:00 2024-01-24T17:39:03.242 INFO:teuthology.orchestra.run.smithi100.stdout:(55/149): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-01-24T17:39:03.297 INFO:teuthology.orchestra.run.smithi039.stdout:(116/149): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-01-24T17:39:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:(117/149): python3-logutils-0.3.5-21.el9.noarch 327 kB/s | 46 kB 00:00 2024-01-24T17:39:03.334 INFO:teuthology.orchestra.run.smithi100.stdout:(56/149): openssl-devel-3.0.7-25.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-01-24T17:39:03.364 INFO:teuthology.orchestra.run.smithi039.stdout:(118/149): python3-more-itertools-8.12.0-2.el9. 594 kB/s | 79 kB 00:00 2024-01-24T17:39:03.393 INFO:teuthology.orchestra.run.smithi100.stdout:(57/149): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-01-24T17:39:03.431 INFO:teuthology.orchestra.run.smithi039.stdout:(119/149): python3-natsort-7.1.1-5.el9.noarch.r 433 kB/s | 58 kB 00:00 2024-01-24T17:39:03.443 INFO:teuthology.orchestra.run.smithi100.stdout:(58/149): postgresql-contrib-13.11-1.el9.x86_64 8.3 MB/s | 811 kB 00:00 2024-01-24T17:39:03.468 INFO:teuthology.orchestra.run.smithi100.stdout:(59/149): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2024-01-24T17:39:03.490 INFO:teuthology.orchestra.run.smithi039.stdout:(120/149): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-01-24T17:39:03.516 INFO:teuthology.orchestra.run.smithi039.stdout:(121/149): python3-portend-3.1.0-2.el9.noarch.r 109 kB/s | 16 kB 00:00 2024-01-24T17:39:03.574 INFO:teuthology.orchestra.run.smithi039.stdout:(122/149): python3-pyOpenSSL-21.0.0-1.el9.noarc 634 kB/s | 90 kB 00:00 2024-01-24T17:39:03.608 INFO:teuthology.orchestra.run.smithi039.stdout:(123/149): python3-repoze-lru-0.7-16.el9.noarch 263 kB/s | 31 kB 00:00 2024-01-24T17:39:03.641 INFO:teuthology.orchestra.run.smithi039.stdout:(124/149): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-01-24T17:39:03.644 INFO:teuthology.orchestra.run.smithi100.stdout:(60/149): ceph-mgr-diskprediction-local-18.2.1- 3.1 MB/s | 7.4 MB 00:02 2024-01-24T17:39:03.683 INFO:teuthology.orchestra.run.smithi039.stdout:(125/149): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-01-24T17:39:03.717 INFO:teuthology.orchestra.run.smithi039.stdout:(126/149): python3-tempora-5.0.0-2.el9.noarch.r 332 kB/s | 36 kB 00:00 2024-01-24T17:39:03.742 INFO:teuthology.orchestra.run.smithi039.stdout:(127/149): python3-typing-extensions-4.4.0-2.el 510 kB/s | 51 kB 00:00 2024-01-24T17:39:03.777 INFO:teuthology.orchestra.run.smithi100.stdout:(61/149): postgresql-server-13.11-1.el9.x86_64. 17 MB/s | 5.7 MB 00:00 2024-01-24T17:39:03.801 INFO:teuthology.orchestra.run.smithi039.stdout:(128/149): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-01-24T17:39:03.834 INFO:teuthology.orchestra.run.smithi039.stdout:(129/149): python3-websocket-client-1.2.3-2.el9 768 kB/s | 90 kB 00:00 2024-01-24T17:39:03.901 INFO:teuthology.orchestra.run.smithi039.stdout:(130/149): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-01-24T17:39:03.911 INFO:teuthology.orchestra.run.smithi100.stdout:(62/149): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-01-24T17:39:03.927 INFO:teuthology.orchestra.run.smithi039.stdout:(131/149): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2024-01-24T17:39:03.936 INFO:teuthology.orchestra.run.smithi100.stdout:(63/149): python3-jinja2-2.11.3-4.el9.noarch.rp 878 kB/s | 228 kB 00:00 2024-01-24T17:39:03.961 INFO:teuthology.orchestra.run.smithi100.stdout:(64/149): python3-jsonpatch-1.21-16.el9.noarch. 144 kB/s | 24 kB 00:00 2024-01-24T17:39:03.993 INFO:teuthology.orchestra.run.smithi039.stdout:(132/149): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-01-24T17:39:04.003 INFO:teuthology.orchestra.run.smithi100.stdout:(65/149): python3-markupsafe-1.1.1-12.el9.x86_6 793 kB/s | 32 kB 00:00 2024-01-24T17:39:04.028 INFO:teuthology.orchestra.run.smithi100.stdout:(66/149): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-01-24T17:39:04.054 INFO:teuthology.orchestra.run.smithi100.stdout:(67/149): python3-jsonpointer-2.0-4.el9.noarch. 199 kB/s | 18 kB 00:00 2024-01-24T17:39:04.211 INFO:teuthology.orchestra.run.smithi039.stdout:(133/149): thrift-0.15.0-2.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-01-24T17:39:04.212 INFO:teuthology.orchestra.run.smithi100.stdout:(68/149): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-01-24T17:39:04.237 INFO:teuthology.orchestra.run.smithi100.stdout:(69/149): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-01-24T17:39:04.252 INFO:teuthology.orchestra.run.smithi039.stdout:(134/149): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-01-24T17:39:04.262 INFO:teuthology.orchestra.run.smithi100.stdout:(70/149): python3-oauthlib-3.1.1-5.el9.noarch.r 927 kB/s | 174 kB 00:00 2024-01-24T17:39:04.304 INFO:teuthology.orchestra.run.smithi100.stdout:(71/149): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-01-24T17:39:04.329 INFO:teuthology.orchestra.run.smithi100.stdout:(72/149): python3-prettytable-0.7.2-27.el9.noar 629 kB/s | 41 kB 00:00 2024-01-24T17:39:04.361 INFO:teuthology.orchestra.run.smithi039.stdout:(135/149): NetworkManager-1.45.10-1.el9.x86_64. 21 MB/s | 2.2 MB 00:00 2024-01-24T17:39:04.375 INFO:teuthology.orchestra.run.smithi100.stdout:(73/149): python3-pyasn1-modules-0.4.8-6.el9.no 1.9 MB/s | 211 kB 00:00 2024-01-24T17:39:04.405 INFO:teuthology.orchestra.run.smithi100.stdout:(74/149): python3-pytz-2021.1-5.el9.noarch.rpm 483 kB/s | 47 kB 00:00 2024-01-24T17:39:04.447 INFO:teuthology.orchestra.run.smithi100.stdout:(75/149): python3-requests-oauthlib-1.3.0-12.el 614 kB/s | 43 kB 00:00 2024-01-24T17:39:04.453 INFO:teuthology.orchestra.run.smithi039.stdout:(136/149): NetworkManager-libnm-1.45.10-1.el9.x 20 MB/s | 1.8 MB 00:00 2024-01-24T17:39:04.495 INFO:teuthology.orchestra.run.smithi039.stdout:(137/149): NetworkManager-team-1.45.10-1.el9.x8 1.0 MB/s | 42 kB 00:00 2024-01-24T17:39:04.545 INFO:teuthology.orchestra.run.smithi039.stdout:(138/149): NetworkManager-tui-1.45.10-1.el9.x86 4.9 MB/s | 246 kB 00:00 2024-01-24T17:39:04.621 INFO:teuthology.orchestra.run.smithi039.stdout:(139/149): acl-2.3.1-4.el9.x86_64.rpm 932 kB/s | 69 kB 00:00 2024-01-24T17:39:04.622 INFO:teuthology.orchestra.run.smithi100.stdout:(76/149): python3-toml-0.10.2-6.el9.noarch.rpm 179 kB/s | 39 kB 00:00 2024-01-24T17:39:04.704 INFO:teuthology.orchestra.run.smithi039.stdout:(140/149): librbd1-18.2.1-341.g4e3182a6.el9.x86 4.2 MB/s | 3.0 MB 00:00 2024-01-24T17:39:04.729 INFO:teuthology.orchestra.run.smithi039.stdout:(141/149): elfutils-debuginfod-client-0.190-2.e 328 kB/s | 35 kB 00:00 2024-01-24T17:39:04.763 INFO:teuthology.orchestra.run.smithi039.stdout:(142/149): elfutils-libelf-0.190-2.el9.x86_64.r 3.3 MB/s | 191 kB 00:00 2024-01-24T17:39:04.797 INFO:teuthology.orchestra.run.smithi039.stdout:(143/149): elfutils-libs-0.190-2.el9.x86_64.rpm 3.9 MB/s | 254 kB 00:00 2024-01-24T17:39:04.822 INFO:teuthology.orchestra.run.smithi039.stdout:(144/149): libacl-2.3.1-4.el9.x86_64.rpm 393 kB/s | 22 kB 00:00 2024-01-24T17:39:04.839 INFO:teuthology.orchestra.run.smithi100.stdout:(77/149): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-01-24T17:39:04.872 INFO:teuthology.orchestra.run.smithi100.stdout:(78/149): socat-1.7.4.1-5.el9.x86_64.rpm 727 kB/s | 300 kB 00:00 2024-01-24T17:39:04.881 INFO:teuthology.orchestra.run.smithi039.stdout:(145/149): openssl-3.0.7-25.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2024-01-24T17:39:04.898 INFO:teuthology.orchestra.run.smithi100.stdout:(79/149): uuid-1.6.2-55.el9.x86_64.rpm 214 kB/s | 56 kB 00:00 2024-01-24T17:39:04.940 INFO:teuthology.orchestra.run.smithi100.stdout:(80/149): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.1 MB/s | 87 kB 00:00 2024-01-24T17:39:04.964 INFO:teuthology.orchestra.run.smithi039.stdout:(146/149): openssl-libs-3.0.7-25.el9.x86_64.rpm 17 MB/s | 2.1 MB 00:00 2024-01-24T17:39:04.965 INFO:teuthology.orchestra.run.smithi100.stdout:(81/149): xmlstarlet-1.6.1-20.el9.x86_64.rpm 958 kB/s | 63 kB 00:00 2024-01-24T17:39:04.990 INFO:teuthology.orchestra.run.smithi100.stdout:(82/149): zlib-devel-1.2.11-40.el9.x86_64.rpm 488 kB/s | 44 kB 00:00 2024-01-24T17:39:04.998 INFO:teuthology.orchestra.run.smithi039.stdout:(147/149): xfsdump-3.1.12-4.el9.x86_64.rpm 3.1 MB/s | 321 kB 00:00 2024-01-24T17:39:05.032 INFO:teuthology.orchestra.run.smithi100.stdout:(83/149): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-01-24T17:39:05.057 INFO:teuthology.orchestra.run.smithi039.stdout:(148/149): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-01-24T17:39:05.066 INFO:teuthology.orchestra.run.smithi100.stdout:(84/149): jq-1.6-16.el9.x86_64.rpm 2.5 MB/s | 186 kB 00:00 2024-01-24T17:39:05.108 INFO:teuthology.orchestra.run.smithi100.stdout:(85/149): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 802 kB 00:00 2024-01-24T17:39:05.141 INFO:teuthology.orchestra.run.smithi100.stdout:(86/149): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 194 kB 00:00 2024-01-24T17:39:05.148 INFO:teuthology.orchestra.run.smithi039.stdout:(149/149): librados2-18.2.1-341.g4e3182a6.el9.x 2.7 MB/s | 3.3 MB 00:01 2024-01-24T17:39:05.164 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-01-24T17:39:05.164 INFO:teuthology.orchestra.run.smithi039.stdout:Total 14 MB/s | 200 MB 00:13 2024-01-24T17:39:05.203 INFO:teuthology.orchestra.run.smithi100.stdout:(87/149): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-01-24T17:39:05.229 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-01-24T17:39:05.233 INFO:teuthology.orchestra.run.smithi100.stdout:(88/149): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 241 kB 00:00 2024-01-24T17:39:05.284 INFO:teuthology.orchestra.run.smithi100.stdout:(89/149): python3-cryptography-36.0.1-4.el9.x86 8.1 MB/s | 1.1 MB 00:00 2024-01-24T17:39:05.309 INFO:teuthology.orchestra.run.smithi100.stdout:(90/149): python3-pycparser-2.20-6.el9.noarch.r 1.9 MB/s | 124 kB 00:00 2024-01-24T17:39:05.334 INFO:teuthology.orchestra.run.smithi100.stdout:(91/149): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-01-24T17:39:05.359 INFO:teuthology.orchestra.run.smithi100.stdout:(92/149): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2024-01-24T17:39:05.393 INFO:teuthology.orchestra.run.smithi100.stdout:(93/149): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-01-24T17:39:05.597 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-01-24T17:39:05.597 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-01-24T17:39:05.643 INFO:teuthology.orchestra.run.smithi100.stdout:(94/149): fmt-8.1.1-5.el9.x86_64.rpm 394 kB/s | 111 kB 00:00 2024-01-24T17:39:05.735 INFO:teuthology.orchestra.run.smithi100.stdout:(95/149): gperftools-libs-2.9.1-2.el9.x86_64.rp 906 kB/s | 309 kB 00:00 2024-01-24T17:39:05.802 INFO:teuthology.orchestra.run.smithi100.stdout:(96/149): libarrow-doc-9.0.0-10.el9.noarch.rpm 394 kB/s | 26 kB 00:00 2024-01-24T17:39:05.877 INFO:teuthology.orchestra.run.smithi100.stdout:(97/149): liboath-2.6.7-2.el9.x86_64.rpm 657 kB/s | 49 kB 00:00 2024-01-24T17:39:06.007 INFO:teuthology.orchestra.run.smithi100.stdout:(98/149): libunwind-1.6.2-1.el9.x86_64.rpm 521 kB/s | 67 kB 00:00 2024-01-24T17:39:06.053 INFO:teuthology.orchestra.run.smithi100.stdout:(99/149): python3-devel-3.9.17-2.el9.x86_64.rpm 355 kB/s | 245 kB 00:00 2024-01-24T17:39:06.162 INFO:teuthology.orchestra.run.smithi100.stdout:(100/149): libarrow-9.0.0-10.el9.x86_64.rpm 8.5 MB/s | 4.4 MB 00:00 2024-01-24T17:39:06.203 INFO:teuthology.orchestra.run.smithi100.stdout:(101/149): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 838 kB 00:00 2024-01-24T17:39:06.237 INFO:teuthology.orchestra.run.smithi100.stdout:(102/149): python3-bcrypt-3.2.2-1.el9.x86_64.rp 584 kB/s | 43 kB 00:00 2024-01-24T17:39:06.270 INFO:teuthology.orchestra.run.smithi100.stdout:(103/149): python3-cachetools-4.2.4-1.el9.noarc 485 kB/s | 32 kB 00:00 2024-01-24T17:39:06.304 INFO:teuthology.orchestra.run.smithi100.stdout:(104/149): python3-certifi-2021.10.8-2.el9.noar 226 kB/s | 15 kB 00:00 2024-01-24T17:39:06.346 INFO:teuthology.orchestra.run.smithi100.stdout:(105/149): python3-cheroot-8.6.0-4.el9.noarch.r 2.3 MB/s | 172 kB 00:00 2024-01-24T17:39:06.396 INFO:teuthology.orchestra.run.smithi100.stdout:(106/149): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-01-24T17:39:06.430 INFO:teuthology.orchestra.run.smithi100.stdout:(107/149): python3-google-auth-2.26.2-1.el9.noa 2.4 MB/s | 202 kB 00:00 2024-01-24T17:39:06.472 INFO:teuthology.orchestra.run.smithi100.stdout:(108/149): python3-jaraco-8.2.1-3.el9.noarch.rp 144 kB/s | 11 kB 00:00 2024-01-24T17:39:06.505 INFO:teuthology.orchestra.run.smithi100.stdout:(109/149): python3-asyncssh-2.13.2-3.el9.noarch 1.2 MB/s | 548 kB 00:00 2024-01-24T17:39:06.531 INFO:teuthology.orchestra.run.smithi100.stdout:(110/149): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-01-24T17:39:06.556 INFO:teuthology.orchestra.run.smithi100.stdout:(111/149): python3-jaraco-collections-3.0.0-8.e 278 kB/s | 23 kB 00:00 2024-01-24T17:39:06.581 INFO:teuthology.orchestra.run.smithi100.stdout:(112/149): python3-jaraco-functools-3.5.0-2.el9 260 kB/s | 19 kB 00:00 2024-01-24T17:39:06.598 INFO:teuthology.orchestra.run.smithi100.stdout:(113/149): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-01-24T17:39:06.615 INFO:teuthology.orchestra.run.smithi100.stdout:(114/149): python3-jwt+crypto-2.4.0-1.el9.noarc 155 kB/s | 9.0 kB 00:00 2024-01-24T17:39:06.641 INFO:teuthology.orchestra.run.smithi100.stdout:(115/149): python3-jwt-2.4.0-1.el9.noarch.rpm 701 kB/s | 41 kB 00:00 2024-01-24T17:39:06.674 INFO:teuthology.orchestra.run.smithi100.stdout:(116/149): python3-logutils-0.3.5-21.el9.noarch 798 kB/s | 46 kB 00:00 2024-01-24T17:39:06.716 INFO:teuthology.orchestra.run.smithi100.stdout:(117/149): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-01-24T17:39:06.750 INFO:teuthology.orchestra.run.smithi100.stdout:(118/149): python3-natsort-7.1.1-5.el9.noarch.r 773 kB/s | 58 kB 00:00 2024-01-24T17:39:06.808 INFO:teuthology.orchestra.run.smithi100.stdout:(119/149): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-01-24T17:39:06.834 INFO:teuthology.orchestra.run.smithi100.stdout:(120/149): python3-portend-3.1.0-2.el9.noarch.r 198 kB/s | 16 kB 00:00 2024-01-24T17:39:06.884 INFO:teuthology.orchestra.run.smithi100.stdout:(121/149): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-01-24T17:39:06.909 INFO:teuthology.orchestra.run.smithi100.stdout:(122/149): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2024-01-24T17:39:06.934 INFO:teuthology.orchestra.run.smithi100.stdout:(123/149): python3-pyOpenSSL-21.0.0-1.el9.noarc 721 kB/s | 90 kB 00:00 2024-01-24T17:39:06.968 INFO:teuthology.orchestra.run.smithi100.stdout:(124/149): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-01-24T17:39:06.996 INFO:teuthology.orchestra.run.smithi100.stdout:(125/149): python3-rsa-4.9-2.el9.noarch.rpm 711 kB/s | 59 kB 00:00 2024-01-24T17:39:07.019 INFO:teuthology.orchestra.run.smithi100.stdout:(126/149): python3-tempora-5.0.0-2.el9.noarch.r 429 kB/s | 36 kB 00:00 2024-01-24T17:39:07.044 INFO:teuthology.orchestra.run.smithi100.stdout:(127/149): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2024-01-24T17:39:07.077 INFO:teuthology.orchestra.run.smithi100.stdout:(128/149): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-01-24T17:39:07.103 INFO:teuthology.orchestra.run.smithi100.stdout:(129/149): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-01-24T17:39:07.136 INFO:teuthology.orchestra.run.smithi100.stdout:(130/149): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-01-24T17:39:07.162 INFO:teuthology.orchestra.run.smithi100.stdout:(131/149): python3-zc-lockfile-2.0-10.el9.noarc 240 kB/s | 20 kB 00:00 2024-01-24T17:39:07.195 INFO:teuthology.orchestra.run.smithi100.stdout:(132/149): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-01-24T17:39:07.295 INFO:teuthology.orchestra.run.smithi100.stdout:(133/149): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-01-24T17:39:07.337 INFO:teuthology.orchestra.run.smithi100.stdout:(134/149): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-01-24T17:39:07.429 INFO:teuthology.orchestra.run.smithi100.stdout:(135/149): NetworkManager-1.45.10-1.el9.x86_64. 25 MB/s | 2.2 MB 00:00 2024-01-24T17:39:07.530 INFO:teuthology.orchestra.run.smithi100.stdout:(136/149): NetworkManager-libnm-1.45.10-1.el9.x 18 MB/s | 1.8 MB 00:00 2024-01-24T17:39:07.563 INFO:teuthology.orchestra.run.smithi100.stdout:(137/149): NetworkManager-team-1.45.10-1.el9.x8 1.3 MB/s | 42 kB 00:00 2024-01-24T17:39:07.605 INFO:teuthology.orchestra.run.smithi100.stdout:(138/149): NetworkManager-tui-1.45.10-1.el9.x86 5.9 MB/s | 246 kB 00:00 2024-01-24T17:39:07.639 INFO:teuthology.orchestra.run.smithi100.stdout:(139/149): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-01-24T17:39:07.695 INFO:teuthology.orchestra.run.smithi100.stdout:(140/149): elfutils-debuginfod-client-0.190-2.e 640 kB/s | 35 kB 00:00 2024-01-24T17:39:07.739 INFO:teuthology.orchestra.run.smithi100.stdout:(141/149): elfutils-libelf-0.190-2.el9.x86_64.r 4.3 MB/s | 191 kB 00:00 2024-01-24T17:39:07.781 INFO:teuthology.orchestra.run.smithi100.stdout:(142/149): elfutils-libs-0.190-2.el9.x86_64.rpm 6.1 MB/s | 254 kB 00:00 2024-01-24T17:39:07.816 INFO:teuthology.orchestra.run.smithi100.stdout:(143/149): libacl-2.3.1-4.el9.x86_64.rpm 671 kB/s | 22 kB 00:00 2024-01-24T17:39:07.848 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-01-24T17:39:07.849 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-01-24T17:39:07.881 INFO:teuthology.orchestra.run.smithi100.stdout:(144/149): openssl-3.0.7-25.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-01-24T17:39:07.982 INFO:teuthology.orchestra.run.smithi100.stdout:(145/149): openssl-libs-3.0.7-25.el9.x86_64.rpm 22 MB/s | 2.1 MB 00:00 2024-01-24T17:39:08.032 INFO:teuthology.orchestra.run.smithi100.stdout:(146/149): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-01-24T17:39:08.099 INFO:teuthology.orchestra.run.smithi100.stdout:(147/149): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-01-24T17:39:13.082 INFO:teuthology.orchestra.run.smithi100.stdout:(148/149): librbd1-18.2.1-341.g4e3182a6.el9.x86 522 kB/s | 3.0 MB 00:05 2024-01-24T17:39:13.274 INFO:teuthology.orchestra.run.smithi100.stdout:(149/149): librados2-18.2.1-341.g4e3182a6.el9.x 548 kB/s | 3.3 MB 00:06 2024-01-24T17:39:13.282 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-01-24T17:39:13.282 INFO:teuthology.orchestra.run.smithi100.stdout:Total 10 MB/s | 200 MB 00:19 2024-01-24T17:39:13.338 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-01-24T17:39:13.677 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-01-24T17:39:13.677 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-01-24T17:39:14.329 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-01-24T17:39:14.456 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/165 2024-01-24T17:39:14.489 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/165 2024-01-24T17:39:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/165 2024-01-24T17:39:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/165 2024-01-24T17:39:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-01-24T17:39:14.786 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-01-24T17:39:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-01-24T17:39:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:18.2.1-341.g4e3182a6.el9.x86_64 8/165 2024-01-24T17:39:15.426 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:18.2.1-341.g4e3182a6.el9.x86_64 8/165 2024-01-24T17:39:15.456 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:18.2.1-341.g4e3182a6.el9.x86_64 9/165 2024-01-24T17:39:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:18.2.1-341.g4e3182a6.el9.x86_64 9/165 2024-01-24T17:39:15.616 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:18.2.1-341.g4e3182a6.el9.x86_64 10/165 2024-01-24T17:39:15.752 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-01-24T17:39:15.752 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-01-24T17:39:15.773 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/165 2024-01-24T17:39:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/165 2024-01-24T17:39:16.212 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/165 2024-01-24T17:39:16.271 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/165 2024-01-24T17:39:16.297 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:18.2.1-341.g4e3182a6.el9.x86_64 15/165 2024-01-24T17:39:16.453 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:18.2.1-341.g4e3182a6.el9.x86_64 15/165 2024-01-24T17:39:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:18.2.1-341.g4e3182a6.el9.x86_ 16/165 2024-01-24T17:39:16.565 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:18.2.1-341.g4e3182a6.el9.x86_ 16/165 2024-01-24T17:39:16.908 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/165 2024-01-24T17:39:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-libnm-1:1.45.10-1.el9.x86_64 18/165 2024-01-24T17:39:17.183 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.10-1.el9.x86_64 18/165 2024-01-24T17:39:17.413 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.45.10-1.el9.x86_64 19/165 2024-01-24T17:39:17.490 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-1:1.45.10-1.el9.x86_64 19/165 2024-01-24T17:39:17.973 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.45.10-1.el9.x86_64 19/165 2024-01-24T17:39:18.021 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/165 2024-01-24T17:39:18.141 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/165 2024-01-24T17:39:18.208 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/165 2024-01-24T17:39:18.284 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/165 2024-01-24T17:39:18.347 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/165 2024-01-24T17:39:18.406 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/165 2024-01-24T17:39:18.522 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/165 2024-01-24T17:39:18.570 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/165 2024-01-24T17:39:18.603 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/165 2024-01-24T17:39:18.718 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-01-24T17:39:18.757 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:18.2.1-341.g4e3182a6.el9.x 30/165 2024-01-24T17:39:18.809 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el9 31/165 2024-01-24T17:39:18.916 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:18.2.1-341.g4e3182a6.el9.x86_64 32/165 2024-01-24T17:39:19.100 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 33/165 2024-01-24T17:39:19.167 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 34/165 2024-01-24T17:39:19.892 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 35/165 2024-01-24T17:39:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 36/165 2024-01-24T17:39:20.225 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 37/165 2024-01-24T17:39:20.296 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 38/165 2024-01-24T17:39:20.346 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:18.2.1-341.g4e3182a6.el9.x86_64 39/165 2024-01-24T17:39:20.432 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 40/165 2024-01-24T17:39:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 41/165 2024-01-24T17:39:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 42/165 2024-01-24T17:39:20.849 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-3.el9.x86_64 43/165 2024-01-24T17:39:20.888 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 44/165 2024-01-24T17:39:20.952 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 45/165 2024-01-24T17:39:21.040 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/165 2024-01-24T17:39:21.097 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 47/165 2024-01-24T17:39:21.137 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 48/165 2024-01-24T17:39:21.207 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/165 2024-01-24T17:39:21.305 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/165 2024-01-24T17:39:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/165 2024-01-24T17:39:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/165 2024-01-24T17:39:21.635 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/165 2024-01-24T17:39:21.692 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 54/165 2024-01-24T17:39:21.848 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 55/165 2024-01-24T17:39:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.26.2-1.el9.noarch 56/165 2024-01-24T17:39:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 57/165 2024-01-24T17:39:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 58/165 2024-01-24T17:39:22.060 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 59/165 2024-01-24T17:39:22.121 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-01-24T17:39:22.234 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fmt-8.1.1-5.el9.x86_64 60/165 2024-01-24T17:39:22.269 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/165 2024-01-24T17:39:22.292 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 61/165 2024-01-24T17:39:22.305 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/165 2024-01-24T17:39:22.377 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 62/165 2024-01-24T17:39:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/165 2024-01-24T17:39:22.444 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/165 2024-01-24T17:39:22.473 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-01-24T17:39:22.573 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-01-24T17:39:22.922 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 63/165 2024-01-24T17:39:22.970 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-01-24T17:39:23.004 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:18.2.1-341.g4e3182a6.el9.x86_64 8/165 2024-01-24T17:39:23.157 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:18.2.1-341.g4e3182a6.el9.x86_64 8/165 2024-01-24T17:39:23.181 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:18.2.1-341.g4e3182a6.el9.x86_64 9/165 2024-01-24T17:39:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 64/165 2024-01-24T17:39:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 65/165 2024-01-24T17:39:23.296 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:18.2.1-341.g4e3182a6.el9.x86_64 9/165 2024-01-24T17:39:23.329 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:18.2.1-341.g4e3182a6.el9.x86_64 10/165 2024-01-24T17:39:23.463 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 66/165 2024-01-24T17:39:23.514 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 67/165 2024-01-24T17:39:23.520 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/165 2024-01-24T17:39:23.538 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 68/165 2024-01-24T17:39:23.583 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 69/165 2024-01-24T17:39:23.631 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 70/165 2024-01-24T17:39:23.650 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/165 2024-01-24T17:39:23.674 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 71/165 2024-01-24T17:39:23.726 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.6-16.el9.x86_64 72/165 2024-01-24T17:39:23.766 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-15.el9.x86_64 73/165 2024-01-24T17:39:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 74/165 2024-01-24T17:39:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 75/165 2024-01-24T17:39:23.941 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : uuid-1.6.2-55.el9.x86_64 76/165 2024-01-24T17:39:23.949 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/165 2024-01-24T17:39:23.993 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 77/165 2024-01-24T17:39:24.003 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/165 2024-01-24T17:39:24.028 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:18.2.1-341.g4e3182a6.el9.x86_64 15/165 2024-01-24T17:39:24.136 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 78/165 2024-01-24T17:39:24.171 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:18.2.1-341.g4e3182a6.el9.x86_64 15/165 2024-01-24T17:39:24.186 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 79/165 2024-01-24T17:39:24.197 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:18.2.1-341.g4e3182a6.el9.x86_ 16/165 2024-01-24T17:39:24.230 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 80/165 2024-01-24T17:39:24.278 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:18.2.1-341.g4e3182a6.el9.x86_ 16/165 2024-01-24T17:39:24.306 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 81/165 2024-01-24T17:39:24.350 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 82/165 2024-01-24T17:39:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 83/165 2024-01-24T17:39:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 84/165 2024-01-24T17:39:24.619 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/165 2024-01-24T17:39:24.737 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 85/165 2024-01-24T17:39:24.802 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : NetworkManager-libnm-1:1.45.10-1.el9.x86_64 18/165 2024-01-24T17:39:24.804 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-01-24T17:39:24.881 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.10-1.el9.x86_64 18/165 2024-01-24T17:39:25.093 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: NetworkManager-1:1.45.10-1.el9.x86_64 19/165 2024-01-24T17:39:25.171 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : NetworkManager-1:1.45.10-1.el9.x86_64 19/165 2024-01-24T17:39:25.822 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 87/165 2024-01-24T17:39:25.841 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: NetworkManager-1:1.45.10-1.el9.x86_64 19/165 2024-01-24T17:39:25.878 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 88/165 2024-01-24T17:39:25.901 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/165 2024-01-24T17:39:25.911 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 89/165 2024-01-24T17:39:25.944 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 90/165 2024-01-24T17:39:26.008 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/165 2024-01-24T17:39:26.078 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/165 2024-01-24T17:39:26.125 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/165 2024-01-24T17:39:26.184 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/165 2024-01-24T17:39:26.193 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/165 2024-01-24T17:39:26.225 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/165 2024-01-24T17:39:26.244 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/165 2024-01-24T17:39:26.320 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/165 2024-01-24T17:39:26.359 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/165 2024-01-24T17:39:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/165 2024-01-24T17:39:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/165 2024-01-24T17:39:26.552 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-01-24T17:39:26.588 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:18.2.1-341.g4e3182a6.el9.x 30/165 2024-01-24T17:39:26.637 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el9 31/165 2024-01-24T17:39:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:18.2.1-341.g4e3182a6.el9.x86_64 32/165 2024-01-24T17:39:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 33/165 2024-01-24T17:39:26.980 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 34/165 2024-01-24T17:39:27.157 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/165 2024-01-24T17:39:27.299 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/165 2024-01-24T17:39:27.668 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 35/165 2024-01-24T17:39:27.760 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 36/165 2024-01-24T17:39:27.803 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 37/165 2024-01-24T17:39:27.858 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 38/165 2024-01-24T17:39:27.901 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:18.2.1-341.g4e3182a6.el9.x86_64 39/165 2024-01-24T17:39:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 40/165 2024-01-24T17:39:28.160 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 41/165 2024-01-24T17:39:28.205 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 42/165 2024-01-24T17:39:28.344 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : re2-1:20211101-3.el9.x86_64 43/165 2024-01-24T17:39:28.382 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 44/165 2024-01-24T17:39:28.445 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 45/165 2024-01-24T17:39:28.556 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 46/165 2024-01-24T17:39:28.635 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 47/165 2024-01-24T17:39:28.676 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 48/165 2024-01-24T17:39:28.747 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 49/165 2024-01-24T17:39:28.808 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 50/165 2024-01-24T17:39:28.867 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/165 2024-01-24T17:39:28.943 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/165 2024-01-24T17:39:29.022 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/165 2024-01-24T17:39:29.058 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/165 2024-01-24T17:39:29.085 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/165 2024-01-24T17:39:29.103 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/165 2024-01-24T17:39:29.127 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 99/165 2024-01-24T17:39:29.147 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 54/165 2024-01-24T17:39:29.272 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 55/165 2024-01-24T17:39:29.341 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:2.26.2-1.el9.noarch 56/165 2024-01-24T17:39:29.401 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 57/165 2024-01-24T17:39:29.435 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 58/165 2024-01-24T17:39:29.474 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 59/165 2024-01-24T17:39:29.511 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/165 2024-01-24T17:39:29.543 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:18.2.1-341.g4e3182a6.el9.x86_64 101/165 2024-01-24T17:39:29.639 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fmt-8.1.1-5.el9.x86_64 60/165 2024-01-24T17:39:29.664 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:18.2.1-341.g4e3182a6.el9.x86_64 101/165 2024-01-24T17:39:29.690 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 61/165 2024-01-24T17:39:29.723 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:18.2.1-341.g4e3182a6.el9.x86_64 102/165 2024-01-24T17:39:29.758 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ply-3.11-14.el9.noarch 62/165 2024-01-24T17:39:29.781 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/165 2024-01-24T17:39:29.840 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 63/165 2024-01-24T17:39:30.087 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 64/165 2024-01-24T17:39:30.171 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 65/165 2024-01-24T17:39:30.276 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 104/165 2024-01-24T17:39:30.304 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:18.2.1-341.g4e3182a6.el9.x86_64 105/165 2024-01-24T17:39:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 66/165 2024-01-24T17:39:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 67/165 2024-01-24T17:39:30.412 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:18.2.1-341.g4e3182a6.el9.x86_64 105/165 2024-01-24T17:39:30.429 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 68/165 2024-01-24T17:39:30.439 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:18.2.1-341.g4e3182a6.el9.x86_64 106/165 2024-01-24T17:39:30.469 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 69/165 2024-01-24T17:39:30.513 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 70/165 2024-01-24T17:39:30.562 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 71/165 2024-01-24T17:39:30.618 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : jq-1.6-16.el9.x86_64 72/165 2024-01-24T17:39:30.661 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.9-15.el9.x86_64 73/165 2024-01-24T17:39:30.712 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 74/165 2024-01-24T17:39:30.771 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 75/165 2024-01-24T17:39:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : uuid-1.6.2-55.el9.x86_64 76/165 2024-01-24T17:39:30.885 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 77/165 2024-01-24T17:39:31.014 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 78/165 2024-01-24T17:39:31.063 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 79/165 2024-01-24T17:39:31.098 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 80/165 2024-01-24T17:39:31.164 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 81/165 2024-01-24T17:39:31.201 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 82/165 2024-01-24T17:39:31.345 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 83/165 2024-01-24T17:39:31.390 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 84/165 2024-01-24T17:39:31.534 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 85/165 2024-01-24T17:39:31.604 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-01-24T17:39:32.573 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 87/165 2024-01-24T17:39:32.637 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 88/165 2024-01-24T17:39:32.687 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 89/165 2024-01-24T17:39:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 90/165 2024-01-24T17:39:32.976 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/165 2024-01-24T17:39:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 107/165 2024-01-24T17:39:33.008 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/165 2024-01-24T17:39:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 107/165 2024-01-24T17:39:33.109 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/165 2024-01-24T17:39:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 107/165 2024-01-24T17:39:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:18.2.1-341.g4e3182a6.el9.x86_64 108/165 2024-01-24T17:39:33.913 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:18.2.1-341.g4e3182a6.el9.x86_64 108/165 2024-01-24T17:39:33.913 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-24T17:39:33.913 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-24T17:39:33.913 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:33.914 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/165 2024-01-24T17:39:33.956 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:18.2.1-341.g4e3182a6.el9.x86_64 109/165 2024-01-24T17:39:34.043 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/165 2024-01-24T17:39:35.570 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/165 2024-01-24T17:39:35.657 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/165 2024-01-24T17:39:35.685 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/165 2024-01-24T17:39:35.720 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 99/165 2024-01-24T17:39:36.063 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/165 2024-01-24T17:39:36.095 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:18.2.1-341.g4e3182a6.el9.x86_64 101/165 2024-01-24T17:39:36.197 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:18.2.1-341.g4e3182a6.el9.x86_64 101/165 2024-01-24T17:39:36.226 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:18.2.1-341.g4e3182a6.el9.x86_64 102/165 2024-01-24T17:39:36.281 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/165 2024-01-24T17:39:36.750 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 104/165 2024-01-24T17:39:36.781 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:18.2.1-341.g4e3182a6.el9.x86_64 105/165 2024-01-24T17:39:36.891 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:18.2.1-341.g4e3182a6.el9.x86_64 105/165 2024-01-24T17:39:36.916 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:18.2.1-341.g4e3182a6.el9.x86_64 106/165 2024-01-24T17:39:39.207 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 107/165 2024-01-24T17:39:39.255 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 107/165 2024-01-24T17:39:39.885 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 107/165 2024-01-24T17:39:39.925 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:18.2.1-341.g4e3182a6.el9.x86_64 108/165 2024-01-24T17:39:40.082 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:18.2.1-341.g4e3182a6.el9.x86_64 108/165 2024-01-24T17:39:40.082 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-24T17:39:40.082 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-24T17:39:40.082 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:40.168 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:18.2.1-341.g4e3182a6.el9.x86_64 109/165 2024-01-24T17:39:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:18.2.1-341.g4e3182a6.el9.x86_64 109/165 2024-01-24T17:39:47.252 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2024-01-24T17:39:47.253 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2024-01-24T17:39:47.253 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2024-01-24T17:39:47.253 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2024-01-24T17:39:47.253 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2024-01-24T17:39:47.253 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2024-01-24T17:39:47.253 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2024-01-24T17:39:47.253 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:47.486 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:18.2.1-341.g4e3182a6.el9.noarch 110/165 2024-01-24T17:39:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:18.2.1-341.g4e3182a6.el9.noarch 110/165 2024-01-24T17:39:47.537 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:47.537 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-24T17:39:47.537 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:49.288 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:18.2.1-341.g4e3182a6.el9.x86_64 111/165 2024-01-24T17:39:49.373 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:18.2.1-341.g4e3182a6.el9.x86_64 111/165 2024-01-24T17:39:49.374 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:49.374 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-24T17:39:49.374 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-24T17:39:49.374 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-24T17:39:49.374 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:49.605 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:18.2.1-341.g4e3182a6.el9.x86_64 112/165 2024-01-24T17:39:49.676 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:18.2.1-341.g4e3182a6.el9.x86_64 112/165 2024-01-24T17:39:49.676 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:49.676 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-24T17:39:49.676 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-24T17:39:49.676 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-24T17:39:49.676 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:50.197 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:18.2.1-341.g4e3182a6.el9.x86_64 113/165 2024-01-24T17:39:50.268 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:18.2.1-341.g4e3182a6.el9.x86_64 113/165 2024-01-24T17:39:50.268 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:50.268 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-24T17:39:50.268 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-24T17:39:50.268 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-24T17:39:50.268 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:50.391 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : automake-1.16.2-8.el9.noarch 114/165 2024-01-24T17:39:51.042 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 115/165 2024-01-24T17:39:51.087 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 115/165 2024-01-24T17:39:51.132 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 116/165 2024-01-24T17:39:51.249 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 117/165 2024-01-24T17:39:51.279 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el9.noar 118/165 2024-01-24T17:39:51.617 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el9.noar 118/165 2024-01-24T17:39:51.663 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el9.no 119/165 2024-01-24T17:39:52.473 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:18.2.1-341.g4e3182a6.el9.x86_64 109/165 2024-01-24T17:39:52.473 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2024-01-24T17:39:52.473 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2024-01-24T17:39:52.473 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2024-01-24T17:39:52.473 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2024-01-24T17:39:52.473 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2024-01-24T17:39:52.474 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /root 2024-01-24T17:39:52.474 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2024-01-24T17:39:52.474 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:52.700 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-volume-2:18.2.1-341.g4e3182a6.el9.noarch 110/165 2024-01-24T17:39:52.749 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-volume-2:18.2.1-341.g4e3182a6.el9.noarch 110/165 2024-01-24T17:39:52.749 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:52.749 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-24T17:39:52.749 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:52.847 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el9.no 119/165 2024-01-24T17:39:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 120/165 2024-01-24T17:39:53.019 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 120/165 2024-01-24T17:39:53.211 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el9 121/165 2024-01-24T17:39:53.249 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:18.2.1-341.g4e3182a6.el9.x86_64 122/165 2024-01-24T17:39:53.319 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:18.2.1-341.g4e3182a6.el9.x86_64 122/165 2024-01-24T17:39:53.320 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:53.320 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-24T17:39:53.320 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-24T17:39:53.320 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-24T17:39:53.320 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:53.374 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el9.noarch 123/165 2024-01-24T17:39:53.414 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el9.noarch 123/165 2024-01-24T17:39:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:18.2.1-341.g4e3182a6.el9.x86_64 124/165 2024-01-24T17:39:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libtool-2.4.6-45.el9.x86_64 125/165 2024-01-24T17:39:54.401 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:18.2.1-341.g4e3182a6.el9.x86_64 126/165 2024-01-24T17:39:54.402 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:18.2.1-341.g4e3182a6.el9.x86_64 111/165 2024-01-24T17:39:54.483 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:18.2.1-341.g4e3182a6.el9.x86_64 111/165 2024-01-24T17:39:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-24T17:39:54.483 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-24T17:39:54.484 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-24T17:39:54.484 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:54.484 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-341.g4e3182a6.el9.x86_64 126/165 2024-01-24T17:39:54.484 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:54.484 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-24T17:39:54.484 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-24T17:39:54.484 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-24T17:39:54.484 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:54.539 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 127/165 2024-01-24T17:39:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-341.g4e3182 127/165 2024-01-24T17:39:54.637 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:54.637 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-24T17:39:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:54.703 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 128/165 2024-01-24T17:39:54.710 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:18.2.1-341.g4e3182a6.el9.x86_64 112/165 2024-01-24T17:39:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:18.2.1-341.g4e3182a6.el9.x86_64 112/165 2024-01-24T17:39:54.787 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:54.787 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-24T17:39:54.787 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-24T17:39:54.787 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-24T17:39:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 128/165 2024-01-24T17:39:54.805 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:54.805 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-24T17:39:54.805 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-24T17:39:54.805 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-24T17:39:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:55.178 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 129/165 2024-01-24T17:39:55.261 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 129/165 2024-01-24T17:39:55.261 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:55.261 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-24T17:39:55.261 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-24T17:39:55.261 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-24T17:39:55.261 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:39:55.278 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:18.2.1-341.g4e3182a6.el9.x86_64 113/165 2024-01-24T17:39:55.359 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:18.2.1-341.g4e3182a6.el9.x86_64 113/165 2024-01-24T17:39:55.359 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:55.359 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-24T17:39:55.359 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-24T17:39:55.359 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-24T17:39:55.359 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:55.472 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : automake-1.16.2-8.el9.noarch 114/165 2024-01-24T17:39:56.091 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 115/165 2024-01-24T17:39:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 115/165 2024-01-24T17:39:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 116/165 2024-01-24T17:39:56.289 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 117/165 2024-01-24T17:39:56.318 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el9.noar 118/165 2024-01-24T17:39:56.636 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el9.noar 118/165 2024-01-24T17:39:56.668 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el9.no 119/165 2024-01-24T17:39:57.786 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el9.no 119/165 2024-01-24T17:39:57.810 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 120/165 2024-01-24T17:39:57.951 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 120/165 2024-01-24T17:39:58.116 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el9 121/165 2024-01-24T17:39:58.154 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:18.2.1-341.g4e3182a6.el9.x86_64 122/165 2024-01-24T17:39:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:18.2.1-341.g4e3182a6.el9.x86_64 122/165 2024-01-24T17:39:58.274 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:58.274 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-24T17:39:58.274 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-24T17:39:58.274 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-24T17:39:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el9.noarch 123/165 2024-01-24T17:39:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el9.noarch 123/165 2024-01-24T17:39:58.423 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:18.2.1-341.g4e3182a6.el9.x86_64 124/165 2024-01-24T17:39:59.223 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libtool-2.4.6-45.el9.x86_64 125/165 2024-01-24T17:39:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:18.2.1-341.g4e3182a6.el9.x86_64 126/165 2024-01-24T17:39:59.361 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-341.g4e3182a6.el9.x86_64 126/165 2024-01-24T17:39:59.362 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:59.362 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-24T17:39:59.362 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-24T17:39:59.362 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-24T17:39:59.362 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:59.416 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 127/165 2024-01-24T17:39:59.485 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:18.2.1-341.g4e3182a6.el9.x86_64 130/165 2024-01-24T17:39:59.492 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-341.g4e3182 127/165 2024-01-24T17:39:59.492 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:59.492 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-24T17:39:59.492 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:59.567 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephfs-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 128/165 2024-01-24T17:39:59.570 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 131/165 2024-01-24T17:39:59.639 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:18.2.1-341.g4e3182a6.el9.x86_64 132/165 2024-01-24T17:39:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 128/165 2024-01-24T17:39:59.642 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:39:59.642 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-24T17:39:59.642 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-24T17:39:59.642 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-24T17:39:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:39:59.676 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 133/165 2024-01-24T17:39:59.976 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 134/165 2024-01-24T17:40:00.026 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 129/165 2024-01-24T17:40:00.102 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 135/165 2024-01-24T17:40:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 129/165 2024-01-24T17:40:00.130 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-24T17:40:00.130 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-24T17:40:00.130 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-24T17:40:00.131 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-24T17:40:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:00.137 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 136/165 2024-01-24T17:40:00.187 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:18.2.1-341.g4e3182a6.el9.x86_6 137/165 2024-01-24T17:40:00.227 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 138/165 2024-01-24T17:40:00.272 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-team-1:1.45.10-1.el9.x86_64 139/165 2024-01-24T17:40:00.384 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-tui-1:1.45.10-1.el9.x86_64 140/165 2024-01-24T17:40:00.439 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 141/165 2024-01-24T17:40:00.480 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 142/165 2024-01-24T17:40:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 143/165 2024-01-24T17:40:01.453 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 143/165 2024-01-24T17:40:01.541 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 143/165 2024-01-24T17:40:01.605 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 144/165 2024-01-24T17:40:03.014 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-top-2:18.2.1-341.g4e3182a6.el9.noarch 145/165 2024-01-24T17:40:03.111 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 146/165 2024-01-24T17:40:03.167 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 147/165 2024-01-24T17:40:03.305 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flex-2.6.4-9.el9.x86_64 148/165 2024-01-24T17:40:03.345 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : bison-3.7.4-5.el9.x86_64 149/165 2024-01-24T17:40:03.378 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 150/165 2024-01-24T17:40:03.413 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 151/165 2024-01-24T17:40:03.455 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 152/165 2024-01-24T17:40:03.456 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-01-24T17:40:03.604 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-01-24T17:40:03.655 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 154/165 2024-01-24T17:40:03.717 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 155/165 2024-01-24T17:40:03.751 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 156/165 2024-01-24T17:40:03.783 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 157/165 2024-01-24T17:40:03.783 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 158/165 2024-01-24T17:40:03.963 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 158/165 2024-01-24T17:40:03.993 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2024-01-24T17:40:04.032 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2024-01-24T17:40:04.032 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2024-01-24T17:40:04.116 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:18.2.1-341.g4e3182a6.el9.x86_64 130/165 2024-01-24T17:40:04.155 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 131/165 2024-01-24T17:40:04.215 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:18.2.1-341.g4e3182a6.el9.x86_64 132/165 2024-01-24T17:40:04.247 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 133/165 2024-01-24T17:40:04.375 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2024-01-24T17:40:04.376 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2024-01-24T17:40:04.528 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 134/165 2024-01-24T17:40:04.543 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2024-01-24T17:40:04.579 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 162/165 2024-01-24T17:40:04.619 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 163/165 2024-01-24T17:40:04.653 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 135/165 2024-01-24T17:40:04.672 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 164/165 2024-01-24T17:40:04.684 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 136/165 2024-01-24T17:40:04.718 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:18.2.1-341.g4e3182a6.el9.x86_6 137/165 2024-01-24T17:40:04.749 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 138/165 2024-01-24T17:40:04.788 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : NetworkManager-team-1:1.45.10-1.el9.x86_64 139/165 2024-01-24T17:40:04.797 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 165/165 2024-01-24T17:40:04.881 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : NetworkManager-tui-1:1.45.10-1.el9.x86_64 140/165 2024-01-24T17:40:04.941 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 141/165 2024-01-24T17:40:04.978 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 142/165 2024-01-24T17:40:05.153 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 165/165 2024-01-24T17:40:05.153 INFO:teuthology.orchestra.run.smithi039.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-24T17:40:05.153 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:05.903 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 143/165 2024-01-24T17:40:05.968 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 143/165 2024-01-24T17:40:06.064 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 143/165 2024-01-24T17:40:06.138 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 144/165 2024-01-24T17:40:07.467 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephfs-top-2:18.2.1-341.g4e3182a6.el9.noarch 145/165 2024-01-24T17:40:07.540 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 146/165 2024-01-24T17:40:07.586 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 147/165 2024-01-24T17:40:07.728 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flex-2.6.4-9.el9.x86_64 148/165 2024-01-24T17:40:07.768 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : bison-3.7.4-5.el9.x86_64 149/165 2024-01-24T17:40:07.801 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 150/165 2024-01-24T17:40:07.835 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 151/165 2024-01-24T17:40:07.877 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 152/165 2024-01-24T17:40:07.877 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-01-24T17:40:08.001 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-01-24T17:40:08.044 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 154/165 2024-01-24T17:40:08.098 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 155/165 2024-01-24T17:40:08.132 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 156/165 2024-01-24T17:40:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 157/165 2024-01-24T17:40:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 158/165 2024-01-24T17:40:08.285 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 158/165 2024-01-24T17:40:08.315 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2024-01-24T17:40:08.353 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2024-01-24T17:40:08.354 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2024-01-24T17:40:08.681 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2024-01-24T17:40:08.681 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2024-01-24T17:40:08.816 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2024-01-24T17:40:08.861 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 162/165 2024-01-24T17:40:08.899 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 163/165 2024-01-24T17:40:08.945 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 164/165 2024-01-24T17:40:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 165/165 2024-01-24T17:40:09.319 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 165/165 2024-01-24T17:40:09.319 INFO:teuthology.orchestra.run.smithi100.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-24T17:40:09.319 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:09.692 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:18.2.1-341.g4e3182a6.el9.x86_64 1/165 2024-01-24T17:40:09.692 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:18.2.1-341.g4e3182a6.el9.x86_64 2/165 2024-01-24T17:40:09.692 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 3/165 2024-01-24T17:40:09.692 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 4/165 2024-01-24T17:40:09.692 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 5/165 2024-01-24T17:40:09.692 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:18.2.1-341.g4e3182a6.el9.x86_64 6/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:18.2.1-341.g4e3182a6.el9.x86_64 7/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:18.2.1-341.g4e3182a6.el9.x86_64 8/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:18.2.1-341.g4e3182a6.el9.x86_64 9/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:18.2.1-341.g4e3182a6.el9.x86_64 10/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:18.2.1-341.g4e3182a6.el9.x86_64 11/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:18.2.1-341.g4e3182a6.el9.x86_64 12/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 13/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:18.2.1-341.g4e3182a6.el9.x86_6 14/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:18.2.1-341.g4e3182a6.el9.x86_64 15/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:18.2.1-341.g4e3182a6.el9.x86_64 16/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:18.2.1-341.g4e3182a6.el9.x86_64 17/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:18.2.1-341.g4e3182a6.el9.x86_ 18/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:18.2.1-341.g4e3182a6.el9.x86_64 19/165 2024-01-24T17:40:09.693 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el9 20/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:18.2.1-341.g4e3182a6.el9.x 21/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:18.2.1-341.g4e3182a6.el9.x86_64 22/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:18.2.1-341.g4e3182a6.el9.x86_64 23/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:18.2.1-341.g4e3182a6.el9.x86_64 24/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:18.2.1-341.g4e3182a6.el9.x86_64 25/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 26/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 27/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:18.2.1-341.g4e3182a6.el9.x86_64 28/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 29/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el9.noar 30/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el9.no 31/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 32/165 2024-01-24T17:40:09.694 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el9 33/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el9.noarch 34/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 35/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:18.2.1-341.g4e3182a6.el9.noarch 36/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 37/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-top-2:18.2.1-341.g4e3182a6.el9.noarch 38/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/165 2024-01-24T17:40:09.695 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/165 2024-01-24T17:40:09.696 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 55/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 56/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 57/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 62/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 63/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 64/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 65/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 66/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 67/165 2024-01-24T17:40:09.697 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 68/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 69/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 70/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 71/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 72/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 73/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 74/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 75/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 76/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 77/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/165 2024-01-24T17:40:09.698 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 80/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 81/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 82/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 83/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.6-16.el9.x86_64 84/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 85/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 86/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 87/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 88/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 89/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 90/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 91/165 2024-01-24T17:40:09.699 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 92/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 93/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 94/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 97/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 98/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 101/165 2024-01-24T17:40:09.700 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.26.2-1.el9.noarch 108/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/165 2024-01-24T17:40:09.701 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/165 2024-01-24T17:40:09.702 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/165 2024-01-24T17:40:09.703 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/165 2024-01-24T17:40:09.703 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/165 2024-01-24T17:40:09.703 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/165 2024-01-24T17:40:09.703 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/165 2024-01-24T17:40:09.703 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/165 2024-01-24T17:40:09.703 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/165 2024-01-24T17:40:09.704 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/165 2024-01-24T17:40:09.704 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/165 2024-01-24T17:40:09.704 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/165 2024-01-24T17:40:09.704 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/165 2024-01-24T17:40:09.704 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:18.2.1-341.g4e3182a6.el9.x86_64 134/165 2024-01-24T17:40:09.704 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/165 2024-01-24T17:40:09.705 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:18.2.1-341.g4e3182a6.el9.x86_64 136/165 2024-01-24T17:40:09.705 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/165 2024-01-24T17:40:09.705 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/165 2024-01-24T17:40:09.705 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/165 2024-01-24T17:40:09.705 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-1:1.45.10-1.el9.x86_64 140/165 2024-01-24T17:40:09.705 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/165 2024-01-24T17:40:09.706 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-libnm-1:1.45.10-1.el9.x86_64 142/165 2024-01-24T17:40:09.706 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/165 2024-01-24T17:40:09.706 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-team-1:1.45.10-1.el9.x86_64 144/165 2024-01-24T17:40:09.706 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/165 2024-01-24T17:40:09.706 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-tui-1:1.45.10-1.el9.x86_64 146/165 2024-01-24T17:40:09.706 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/165 2024-01-24T17:40:09.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/165 2024-01-24T17:40:09.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/165 2024-01-24T17:40:09.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 150/165 2024-01-24T17:40:09.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 151/165 2024-01-24T17:40:09.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 152/165 2024-01-24T17:40:09.707 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 153/165 2024-01-24T17:40:09.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 154/165 2024-01-24T17:40:09.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 155/165 2024-01-24T17:40:09.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 156/165 2024-01-24T17:40:09.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 157/165 2024-01-24T17:40:09.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 158/165 2024-01-24T17:40:09.708 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 159/165 2024-01-24T17:40:09.709 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 160/165 2024-01-24T17:40:09.709 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 161/165 2024-01-24T17:40:09.709 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 162/165 2024-01-24T17:40:09.709 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 163/165 2024-01-24T17:40:09.709 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 164/165 2024-01-24T17:40:11.888 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 165/165 2024-01-24T17:40:11.888 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-1:1.45.10-1.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-libnm-1:1.45.10-1.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-team-1:1.45.10-1.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-tui-1:1.45.10-1.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: acl-2.3.1-4.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.889 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: automake-1.16.2-8.el9.noarch 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: bison-3.7.4-5.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.890 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.891 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: flex-2.6.4-9.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: fmt-8.1.1-5.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.6-16.el9.x86_64 2024-01-24T17:40:11.892 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-24T17:40:11.893 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-13.11-1.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-24T17:40:11.894 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.26.2-1.el9.noarch 2024-01-24T17:40:11.895 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-24T17:40:11.896 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-24T17:40:11.897 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-24T17:40:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-3.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-24T17:40:11.899 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-24T17:40:11.900 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:11.900 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-01-24T17:40:12.478 DEBUG:teuthology.parallel:result is None 2024-01-24T17:40:13.875 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:18.2.1-341.g4e3182a6.el9.x86_64 1/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:18.2.1-341.g4e3182a6.el9.x86_64 2/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 3/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 4/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 5/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:18.2.1-341.g4e3182a6.el9.x86_64 6/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:18.2.1-341.g4e3182a6.el9.x86_64 7/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:18.2.1-341.g4e3182a6.el9.x86_64 8/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:18.2.1-341.g4e3182a6.el9.x86_64 9/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:18.2.1-341.g4e3182a6.el9.x86_64 10/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:18.2.1-341.g4e3182a6.el9.x86_64 11/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:18.2.1-341.g4e3182a6.el9.x86_64 12/165 2024-01-24T17:40:13.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephfs-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 13/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:18.2.1-341.g4e3182a6.el9.x86_6 14/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:18.2.1-341.g4e3182a6.el9.x86_64 15/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:18.2.1-341.g4e3182a6.el9.x86_64 16/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:18.2.1-341.g4e3182a6.el9.x86_64 17/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:18.2.1-341.g4e3182a6.el9.x86_ 18/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:18.2.1-341.g4e3182a6.el9.x86_64 19/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el9 20/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:18.2.1-341.g4e3182a6.el9.x 21/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:18.2.1-341.g4e3182a6.el9.x86_64 22/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:18.2.1-341.g4e3182a6.el9.x86_64 23/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:18.2.1-341.g4e3182a6.el9.x86_64 24/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:18.2.1-341.g4e3182a6.el9.x86_64 25/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 26/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 27/165 2024-01-24T17:40:13.877 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:18.2.1-341.g4e3182a6.el9.x86_64 28/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 29/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el9.noar 30/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el9.no 31/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 32/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el9 33/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el9.noarch 34/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 35/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-volume-2:18.2.1-341.g4e3182a6.el9.noarch 36/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 37/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephfs-top-2:18.2.1-341.g4e3182a6.el9.noarch 38/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/165 2024-01-24T17:40:13.878 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 55/165 2024-01-24T17:40:13.879 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 56/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 57/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 62/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 63/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 64/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 65/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 66/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 67/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 68/165 2024-01-24T17:40:13.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 69/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 70/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 71/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 72/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 73/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 74/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 75/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 76/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 77/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 80/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 81/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 82/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 83/165 2024-01-24T17:40:13.881 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : jq-1.6-16.el9.x86_64 84/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 85/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 86/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 87/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 88/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 89/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 90/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 91/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 92/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 93/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 94/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 97/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 98/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/165 2024-01-24T17:40:13.882 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 101/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:2.26.2-1.el9.noarch 108/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/165 2024-01-24T17:40:13.883 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/165 2024-01-24T17:40:13.884 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:18.2.1-341.g4e3182a6.el9.x86_64 134/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:18.2.1-341.g4e3182a6.el9.x86_64 136/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/165 2024-01-24T17:40:13.885 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-1:1.45.10-1.el9.x86_64 140/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-libnm-1:1.45.10-1.el9.x86_64 142/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-team-1:1.45.10-1.el9.x86_64 144/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-tui-1:1.45.10-1.el9.x86_64 146/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 150/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 151/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 152/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 153/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 154/165 2024-01-24T17:40:13.886 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 155/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 156/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 157/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 158/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 159/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 160/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 161/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 162/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 163/165 2024-01-24T17:40:13.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 164/165 2024-01-24T17:40:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 165/165 2024-01-24T17:40:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-1:1.45.10-1.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-libnm-1:1.45.10-1.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-team-1:1.45.10-1.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: NetworkManager-tui-1:1.45.10-1.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: acl-2.3.1-4.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-24T17:40:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: automake-1.16.2-8.el9.noarch 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: bison-3.7.4-5.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: cephfs-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: cephfs-top-2:18.2.1-341.g4e3182a6.el9.noarch 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: flex-2.6.4-9.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: fmt-8.1.1-5.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-24T17:40:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: jq-1.6-16.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-13.11-1.el9.x86_64 2024-01-24T17:40:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-24T17:40:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-24T17:40:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-24T17:40:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:2.26.2-1.el9.noarch 2024-01-24T17:40:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-24T17:40:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-24T17:40:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-24T17:40:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-24T17:40:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-24T17:40:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply-3.11-14.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-24T17:40:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-24T17:40:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:18.2.1-341.g4e3182a6.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: re2-1:20211101-3.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:16.007 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-01-24T17:40:16.582 DEBUG:teuthology.parallel:result is None 2024-01-24T17:40:16.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T17:40:16.727 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-24T17:40:16.784 INFO:teuthology.orchestra.run.smithi039.stdout:18.2.1-341.g4e3182a6.el9 2024-01-24T17:40:16.785 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-341.g4e3182a6.el9 2024-01-24T17:40:16.785 INFO:teuthology.task.install:The correct ceph version 18.2.1-341.g4e3182a6 is installed. 2024-01-24T17:40:16.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T17:40:16.922 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-24T17:40:16.976 INFO:teuthology.orchestra.run.smithi100.stdout:18.2.1-341.g4e3182a6.el9 2024-01-24T17:40:16.977 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-341.g4e3182a6.el9 2024-01-24T17:40:16.977 INFO:teuthology.task.install:The correct ceph version 18.2.1-341.g4e3182a6 is installed. 2024-01-24T17:40:16.979 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-24T17:40:16.979 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:16.979 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-24T17:40:17.063 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:40:17.063 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-24T17:40:17.142 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-24T17:40:17.143 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:17.143 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-01-24T17:40:17.220 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-24T17:40:17.339 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:40:17.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2024-01-24T17:40:17.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-24T17:40:17.528 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-24T17:40:17.528 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:17.528 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-24T17:40:17.597 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-24T17:40:17.710 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:40:17.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-24T17:40:17.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-24T17:40:17.913 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-24T17:40:17.914 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:17.914 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-01-24T17:40:17.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-24T17:40:18.092 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:40:18.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/stdin-killer 2024-01-24T17:40:18.163 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-24T17:40:18.300 INFO:teuthology.run_tasks:Running task ceph... 2024-01-24T17:40:18.370 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-24T17:40:18.370 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2024-01-24T17:40:18.374 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /var/log/ceph 2024-01-24T17:40:18.454 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-24T17:40:18.455 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-24T17:40:18.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-24T17:40:18.535 INFO:tasks.ceph:Creating extra log directories... 2024-01-24T17:40:18.535 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-24T17:40:18.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-24T17:40:18.636 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-24T17:40:18.637 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-24T17:40:18.637 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-24T17:40:18.637 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:18.637 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-24T17:40:18.686 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-24T17:40:18.958 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-24T17:40:19.049 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:40:19.049 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-24T17:40:19.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-24T17:40:19.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-24T17:40:19.384 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-24T17:40:19.384 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'client session with non-allowable root'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-24T17:40:19.385 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972', 'branch': 'wip-vshankar-testing1-reef-2024-01-24-1150', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7530972', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': 'f2d3460408951339ac0933340ad449487c9efddc'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing1-reef-2024-01-24-1150'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'client session with non-allowable root'], 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing1-reef-2024-01-24-1150', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4668, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'sleep_before_teardown': 0, 'subset': '346/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing1-reef-2024-01-24-1150', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKIzEkcl8WMnQW1kTCFzMXo6NA5rOiCIVmprT9OVTY5g4soVADD3sd9U5Sy2pPHYpjgZjbhWpvpAbWGKYxXKpMg=', 'smithi100.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6sBC4h2uBb88W06XRSHJhI8OMNG9o8ip/9uHl7OoVnW/PfJCu9Q38OVllMnilIDdnnCzFgqRJ8AZ8nKvxECl4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': 'f2d3460408951339ac0933340ad449487c9efddc'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'client session with non-allowable root'], 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_sessionmap']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c856e0f9244dd29a50697123b480d63f16a4ed81', 'timestamp': '2024-01-24_12:12:30', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-24T17:40:19.385 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-24T17:40:19.393 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-24T17:40:19.445 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2024-01-24T17:40:19.450 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/run/ceph 2024-01-24T17:40:19.550 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:19.550 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-01-24T17:40:19.605 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-24T17:40:19.605 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-01-24T17:40:19.659 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-24T17:40:19.659 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T17:40:19.659 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-01-24T17:40:19.660 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T17:40:19.660 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T17:40:19.660 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-01-24 17:40:05.646438473 +0000 2024-01-24T17:40:19.660 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-01-24 17:37:50.848471312 +0000 2024-01-24T17:40:19.660 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-01-24 17:37:50.848471312 +0000 2024-01-24T17:40:19.660 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-01-24 17:37:50.848471312 +0000 2024-01-24T17:40:19.660 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-24T17:40:19.744 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-01-24T17:40:19.744 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-01-24T17:40:19.744 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000500316 s, 1.0 MB/s 2024-01-24T17:40:19.747 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-24T17:40:19.803 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-01-24T17:40:19.846 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-24T17:40:19.846 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T17:40:19.846 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-01-24T17:40:19.846 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T17:40:19.846 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T17:40:19.846 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-01-24 17:40:05.647438453 +0000 2024-01-24T17:40:19.846 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-01-24 17:37:51.453459180 +0000 2024-01-24T17:40:19.846 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-01-24 17:37:51.453459180 +0000 2024-01-24T17:40:19.847 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-01-24 17:37:51.453459180 +0000 2024-01-24T17:40:19.847 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-24T17:40:19.980 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-01-24T17:40:19.980 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-01-24T17:40:19.980 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000425085 s, 1.2 MB/s 2024-01-24T17:40:19.985 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-24T17:40:20.046 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-01-24T17:40:20.100 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-24T17:40:20.100 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T17:40:20.100 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-01-24T17:40:20.100 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T17:40:20.100 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T17:40:20.100 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-01-24 17:40:05.648438434 +0000 2024-01-24T17:40:20.100 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-01-24 17:37:52.069446827 +0000 2024-01-24T17:40:20.101 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-01-24 17:37:52.069446827 +0000 2024-01-24T17:40:20.101 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-01-24 17:37:52.069446827 +0000 2024-01-24T17:40:20.101 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-24T17:40:20.190 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-01-24T17:40:20.190 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-01-24T17:40:20.190 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000406012 s, 1.3 MB/s 2024-01-24T17:40:20.194 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-24T17:40:20.250 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-01-24T17:40:20.300 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-24T17:40:20.301 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T17:40:20.301 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-01-24T17:40:20.301 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T17:40:20.301 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T17:40:20.301 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-01-24 17:40:05.648438434 +0000 2024-01-24T17:40:20.301 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-01-24 17:37:52.683434515 +0000 2024-01-24T17:40:20.301 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-01-24 17:37:52.683434515 +0000 2024-01-24T17:40:20.301 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-01-24 17:37:52.683434515 +0000 2024-01-24T17:40:20.301 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-24T17:40:20.385 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-01-24T17:40:20.386 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-01-24T17:40:20.386 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000426472 s, 1.2 MB/s 2024-01-24T17:40:20.389 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-24T17:40:20.445 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:20.446 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:40:20.446 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-01-24T17:40:20.493 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-24T17:40:20.493 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-01-24 17:40:09.788634547 +0000 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-01-24 17:37:39.560272295 +0000 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-01-24 17:37:39.560272295 +0000 2024-01-24T17:40:20.577 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-01-24 17:37:39.560272295 +0000 2024-01-24T17:40:20.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-24T17:40:20.682 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-01-24T17:40:20.683 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-01-24T17:40:20.683 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000343623 s, 1.5 MB/s 2024-01-24T17:40:20.685 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-24T17:40:20.727 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-01-24T17:40:20.808 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-24T17:40:20.808 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T17:40:20.808 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-01-24T17:40:20.808 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T17:40:20.808 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T17:40:20.808 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-01-24 17:40:09.789634522 +0000 2024-01-24T17:40:20.809 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-01-24 17:37:40.125257764 +0000 2024-01-24T17:40:20.809 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-01-24 17:37:40.125257764 +0000 2024-01-24T17:40:20.809 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-01-24 17:37:40.125257764 +0000 2024-01-24T17:40:20.809 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-24T17:40:20.920 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-01-24T17:40:20.920 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-01-24T17:40:20.921 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000272281 s, 1.9 MB/s 2024-01-24T17:40:20.924 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-24T17:40:20.965 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-01-24T17:40:21.049 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-24T17:40:21.049 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T17:40:21.049 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-01-24T17:40:21.049 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T17:40:21.049 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T17:40:21.049 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-01-24 17:40:09.789634522 +0000 2024-01-24T17:40:21.049 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-01-24 17:37:40.686243336 +0000 2024-01-24T17:40:21.049 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-01-24 17:37:40.686243336 +0000 2024-01-24T17:40:21.050 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-01-24 17:37:40.686243336 +0000 2024-01-24T17:40:21.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-24T17:40:21.175 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-01-24T17:40:21.175 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-01-24T17:40:21.175 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000328205 s, 1.6 MB/s 2024-01-24T17:40:21.179 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-24T17:40:21.233 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-01-24T17:40:21.280 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-24T17:40:21.280 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T17:40:21.280 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-01-24T17:40:21.280 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T17:40:21.280 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T17:40:21.280 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-01-24 17:40:09.790634496 +0000 2024-01-24T17:40:21.280 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-01-24 17:37:41.254228728 +0000 2024-01-24T17:40:21.281 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-01-24 17:37:41.254228728 +0000 2024-01-24T17:40:21.281 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-01-24 17:37:41.254228728 +0000 2024-01-24T17:40:21.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-24T17:40:21.413 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-01-24T17:40:21.413 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-01-24T17:40:21.413 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000348467 s, 1.5 MB/s 2024-01-24T17:40:21.417 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-24T17:40:21.471 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:21.472 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-24T17:40:21.472 INFO:tasks.ceph:Generating config... 2024-01-24T17:40:21.473 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-24T17:40:21.473 INFO:tasks.ceph:[client] debug client = 20 2024-01-24T17:40:21.473 INFO:tasks.ceph:[client] debug ms = 1 2024-01-24T17:40:21.473 INFO:tasks.ceph:[client] pid file = 2024-01-24T17:40:21.473 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-24T17:40:21.473 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-24T17:40:21.473 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-24T17:40:21.473 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-24T17:40:21.474 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-24T17:40:21.474 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-24T17:40:21.474 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-24T17:40:21.475 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-24T17:40:21.476 INFO:tasks.ceph:Setting up mon.a... 2024-01-24T17:40:21.476 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-24T17:40:21.625 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.keyring 2024-01-24T17:40:21.635 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-24T17:40:21.779 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-24T17:40:21.878 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.39'), ('mon.b', '172.21.15.100'), ('mon.c', '[v2:172.21.15.100:3301,v1:172.21.15.100:6790]')] 2024-01-24T17:40:21.879 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-24T17:40:21.879 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:21.880 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-24T17:40:21.936 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.39 --add b 172.21.15.100 --addv c '[v2:172.21.15.100:3301,v1:172.21.15.100:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:22.092 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:22.092 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: generated fsid b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:40:22.092 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = pacific 2024-01-24T17:40:22.092 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 0 2024-01-24T17:40:22.092 INFO:teuthology.orchestra.run.smithi039.stdout:fsid b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:40:22.092 INFO:teuthology.orchestra.run.smithi039.stdout:last_changed 2024-01-24T17:40:22.092434+0000 2024-01-24T17:40:22.093 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-01-24T17:40:22.092434+0000 2024-01-24T17:40:22.093 INFO:teuthology.orchestra.run.smithi039.stdout:min_mon_release 16 (pacific) 2024-01-24T17:40:22.093 INFO:teuthology.orchestra.run.smithi039.stdout:election_strategy: 1 2024-01-24T17:40:22.093 INFO:teuthology.orchestra.run.smithi039.stdout:0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-01-24T17:40:22.093 INFO:teuthology.orchestra.run.smithi039.stdout:1: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.b 2024-01-24T17:40:22.093 INFO:teuthology.orchestra.run.smithi039.stdout:2: [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] mon.c 2024-01-24T17:40:22.093 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-24T17:40:22.096 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-24T17:40:22.148 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b13477e8-9bd1-460b-97e1-2729dfdebd46... 2024-01-24T17:40:22.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:40:22.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:40:22.320 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-01-24T17:40:22.320 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-01-24T17:40:22.320 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.321 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:40:22.322 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-01-24T17:40:22.323 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-01-24T17:40:22.324 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-01-24T17:40:22.325 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:40:22.326 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-01-24T17:40:22.327 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:40:22.328 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-01-24T17:40:22.329 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-01-24T17:40:22.330 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:40:22.331 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-01-24T17:40:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-01-24T17:40:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-01-24T17:40:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-01-24T17:40:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:40:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-01-24T17:40:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:40:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-01-24T17:40:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-01-24T17:40:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-01-24T17:40:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = bitmap 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.338 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-01-24T17:40:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-01-24T17:40:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: mon op complaint time = 120 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: client mount timeout = 600 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:40:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = "" 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout:[mds] 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds = 20 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds balancer = 20 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug frag = True 2024-01-24T17:40:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug scatterstat = True 2024-01-24T17:40:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: mds op complaint time = 180 2024-01-24T17:40:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: mds verify scatter = True 2024-01-24T17:40:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:40:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:40:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:40:22.368 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-24T17:40:22.369 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-24T17:40:22.515 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-24T17:40:22.515 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:22.515 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-24T17:40:22.567 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:22.567 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-24T17:40:22.666 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:40:22.667 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:22.667 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-24T17:40:22.667 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-24T17:40:22.828 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:40:22.828 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:22.928 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:40:22.928 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:40:22.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-24T17:40:22.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-24T17:40:23.077 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:40:23.077 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:23.168 INFO:tasks.ceph:Setting up mon nodes... 2024-01-24T17:40:23.169 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-24T17:40:23.169 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-24T17:40:23.362 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-24T17:40:23.373 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-24T17:40:23.527 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-24T17:40:23.535 INFO:tasks.ceph:Setting up mds nodes... 2024-01-24T17:40:23.535 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-24T17:40:23.740 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-24T17:40:23.750 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-24T17:40:23.837 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-24T17:40:24.025 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-24T17:40:24.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-24T17:40:24.138 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-24T17:40:24.313 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-24T17:40:24.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-24T17:40:24.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-24T17:40:24.557 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-24T17:40:24.565 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-24T17:40:24.651 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-24T17:40:24.651 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-24T17:40:24.787 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-24T17:40:24.851 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-24T17:40:24.983 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-24T17:40:25.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-24T17:40:25.171 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-24T17:40:25.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-24T17:40:25.339 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-24T17:40:25.383 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-24T17:40:25.383 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-24T17:40:25.383 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-24T17:40:25.489 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:25.489 INFO:tasks.ceph:role: osd.0 2024-01-24T17:40:25.489 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:40:25.490 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-24T17:40:25.610 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-24T17:40:25.611 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-24T17:40:26.939 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-01-24T17:40:26.946 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-01-24T17:40:26.946 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-24T17:40:27.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-24T17:40:27.170 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-24T17:40:27.268 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:27.269 INFO:tasks.ceph:role: osd.1 2024-01-24T17:40:27.269 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:40:27.269 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-24T17:40:27.396 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-24T17:40:28.754 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-01-24T17:40:28.761 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-01-24T17:40:28.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-24T17:40:28.879 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-24T17:40:29.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-24T17:40:29.140 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:29.141 INFO:tasks.ceph:role: osd.2 2024-01-24T17:40:29.141 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:40:29.141 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-24T17:40:29.268 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-24T17:40:30.595 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-01-24T17:40:30.602 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-01-24T17:40:30.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-24T17:40:30.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-24T17:40:30.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-24T17:40:30.909 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:30.909 INFO:tasks.ceph:role: osd.3 2024-01-24T17:40:30.909 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:40:30.909 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-24T17:40:31.034 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-24T17:40:31.034 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-24T17:40:31.035 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-24T17:40:31.035 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-24T17:40:31.035 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-24T17:40:31.035 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-01-24T17:40:31.035 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-24T17:40:31.035 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-24T17:40:31.035 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-24T17:40:31.035 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-24T17:40:32.367 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-01-24T17:40:32.374 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-01-24T17:40:32.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-24T17:40:32.494 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-24T17:40:32.582 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:32.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:32.751+0000 7fc7a0e10500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-24T17:40:32.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:32.752+0000 7fc7a0e10500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-24T17:40:32.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:32.753+0000 7fc7a0e10500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-24T17:40:32.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:32.753+0000 7fc7a0e10500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-24T17:40:32.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:32.753+0000 7fc7a0e10500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-24T17:40:32.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:32.756+0000 7fc7a0e10500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-24T17:40:35.966 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-24T17:40:36.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:36.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:36.216+0000 7f0152e14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-24T17:40:36.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:36.217+0000 7f0152e14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-24T17:40:36.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:36.217+0000 7f0152e14500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-24T17:40:36.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:36.217+0000 7f0152e14500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-24T17:40:36.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:36.217+0000 7f0152e14500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-24T17:40:36.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:36.220+0000 7f0152e14500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-24T17:40:39.177 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-24T17:40:39.278 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:39.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:39.427+0000 7ff304194500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-24T17:40:39.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:39.427+0000 7ff304194500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-24T17:40:39.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:39.427+0000 7ff304194500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-24T17:40:39.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:39.427+0000 7ff304194500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-24T17:40:39.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:39.427+0000 7ff304194500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-24T17:40:39.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:39.430+0000 7ff304194500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-24T17:40:42.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-24T17:40:42.720 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:42.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:42.867+0000 7f0e2c9a9500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-24T17:40:42.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:42.868+0000 7f0e2c9a9500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-24T17:40:42.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:42.868+0000 7f0e2c9a9500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-24T17:40:42.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:42.868+0000 7f0e2c9a9500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-24T17:40:42.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:42.868+0000 7f0e2c9a9500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-24T17:40:42.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:40:42.871+0000 7f0e2c9a9500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-24T17:40:46.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-24T17:40:46.174 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-24T17:40:46.271 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:46.272 INFO:tasks.ceph:role: osd.4 2024-01-24T17:40:46.272 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:40:46.272 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-24T17:40:46.370 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-24T17:40:46.370 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-24T17:40:46.370 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-24T17:40:46.370 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-24T17:40:46.370 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-24T17:40:46.370 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-01-24T17:40:46.371 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-24T17:40:46.371 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-24T17:40:46.371 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-24T17:40:46.371 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-24T17:40:46.874 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-01-24T17:40:46.880 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-01-24T17:40:46.880 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-24T17:40:47.053 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-24T17:40:47.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-24T17:40:47.244 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:47.245 INFO:tasks.ceph:role: osd.5 2024-01-24T17:40:47.245 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:40:47.245 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-24T17:40:47.333 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-24T17:40:47.333 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-24T17:40:47.333 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-24T17:40:47.333 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-24T17:40:47.333 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-24T17:40:47.333 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-01-24T17:40:47.333 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-24T17:40:47.333 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-24T17:40:47.334 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-24T17:40:47.334 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-24T17:40:47.813 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-01-24T17:40:47.819 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-01-24T17:40:47.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-24T17:40:47.956 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-24T17:40:48.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-24T17:40:48.141 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:48.141 INFO:tasks.ceph:role: osd.6 2024-01-24T17:40:48.141 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:40:48.141 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-24T17:40:48.239 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-24T17:40:48.239 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-24T17:40:48.240 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-24T17:40:48.240 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-24T17:40:48.240 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-24T17:40:48.240 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-01-24T17:40:48.240 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-24T17:40:48.240 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-24T17:40:48.240 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-24T17:40:48.240 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-24T17:40:48.745 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-01-24T17:40:48.753 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-01-24T17:40:48.753 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-24T17:40:48.939 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-24T17:40:49.040 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-24T17:40:49.127 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-24T17:40:49.127 INFO:tasks.ceph:role: osd.7 2024-01-24T17:40:49.127 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:40:49.128 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-24T17:40:49.223 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-24T17:40:49.223 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-24T17:40:49.223 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-24T17:40:49.223 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-24T17:40:49.223 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-24T17:40:49.223 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-01-24T17:40:49.223 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-24T17:40:49.223 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-24T17:40:49.224 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-24T17:40:49.224 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-24T17:40:49.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:40:49.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:40:49.688 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-01-24T17:40:49.694 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-01-24T17:40:49.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-24T17:40:49.841 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-24T17:40:49.937 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:50.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:50.087+0000 7f89e4a49500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-24T17:40:50.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:50.089+0000 7f89e4a49500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-24T17:40:50.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:50.089+0000 7f89e4a49500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-24T17:40:50.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:50.089+0000 7f89e4a49500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-24T17:40:50.090 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:50.089+0000 7f89e4a49500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-24T17:40:50.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:50.095+0000 7f89e4a49500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-24T17:40:53.281 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-24T17:40:53.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:53.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:53.514+0000 7ffbe3869500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-24T17:40:53.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:53.515+0000 7ffbe3869500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-24T17:40:53.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:53.515+0000 7ffbe3869500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-24T17:40:53.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:53.515+0000 7ffbe3869500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-24T17:40:53.517 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:53.515+0000 7ffbe3869500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-24T17:40:53.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:53.517+0000 7ffbe3869500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-24T17:40:56.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-24T17:40:56.785 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:40:56.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:56.910+0000 7fc775739500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-24T17:40:56.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:56.911+0000 7fc775739500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-24T17:40:56.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:56.911+0000 7fc775739500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-24T17:40:56.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:56.912+0000 7fc775739500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-24T17:40:56.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:56.912+0000 7fc775739500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-24T17:40:56.914 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:40:56.913+0000 7fc775739500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-24T17:41:00.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-24T17:41:00.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:41:00.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:00.304+0000 7f6bf5425500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-24T17:41:00.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:00.304+0000 7f6bf5425500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-24T17:41:00.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:00.304+0000 7f6bf5425500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-24T17:41:00.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:00.304+0000 7f6bf5425500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-24T17:41:00.305 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:00.304+0000 7f6bf5425500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-24T17:41:00.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:00.307+0000 7f6bf5425500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-24T17:41:03.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-24T17:41:03.322 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-24T17:41:03.322 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:03.322 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-24T17:41:03.430 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:03.430 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-24T17:41:03.531 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:03.531 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-24T17:41:03.632 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:03.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-24T17:41:03.728 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:03.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-24T17:41:03.827 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:03.827 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-24T17:41:03.930 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:03.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-24T17:41:04.034 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.035 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-24T17:41:04.128 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-24T17:41:04.212 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-24T17:41:04.303 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.303 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-24T17:41:04.441 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-24T17:41:04.525 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-24T17:41:04.663 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-24T17:41:04.760 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:04.760 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-24T17:41:04.820 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:04.820 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-24T17:41:04.876 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.876 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-24T17:41:04.930 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:04.930 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-24T17:41:04.981 INFO:tasks.ceph:Adding keys to all mons... 2024-01-24T17:41:04.981 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-24T17:41:04.989 DEBUG:teuthology.orchestra.run.smithi100:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-24T17:41:05.077 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.y] 2024-01-24T17:41:05.077 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCHS7Fl512kFRAAdCpU+QjRmGJmMD5uTdFmHQ== 2024-01-24T17:41:05.077 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCHS7FlZZEtLBAAUxf4chL3tgy3zHlZoraXlg== 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCIS7Flo8iGARAAf+29RXWr4RHkILIz7em3mA== 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCQS7FlOj/pLBAAzi89tAacOfDKdlU0BUJvvA== 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCUS7FlbRAKDRAAAuAy47MDTtvmh8qNA3ZLTg== 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCXS7Fl7QmRGRAAVECH7oYTdvDPSHBy9WmHwA== 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.3] 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCaS7FlCQHIMxAA9OchsMzsKFxs110SeJ3Atw== 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2024-01-24T17:41:05.078 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCHS7FlRbF6HxAAoFjlyWb5oqIkC3L1iPRmHw== 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCIS7Fln4utEhAA52ngJFqrdfa9bx56mVhRVA== 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCIS7Flh+08IRAARhZxy//j8c9b3Hg9uV9ajg== 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.4] 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCiS7FlDsBKBRAAXF2HafraT5nRbB5tBRUgWQ== 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.5] 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQClS7FlfrXBHhAAy8GP3TLX/iok3OYPGTP5Mw== 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.6] 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCoS7Fla11hNhAA/3oMP7cFtPSTdJl7eSnPTA== 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.7] 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCsS7FlKFMtEhAAPmaFzr5XHVQaKZYkJiIarA== 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCIS7FlTi/3LhAAc8FwXtLH2XjlFMO7wDMcSQ== 2024-01-24T17:41:05.079 INFO:teuthology.orchestra.run.smithi039.stdout:[client.2] 2024-01-24T17:41:05.080 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCIS7FlIISlOhAAoVxxez7UjXDxyHQNNPnn2g== 2024-01-24T17:41:05.080 INFO:teuthology.orchestra.run.smithi039.stdout:[client.1] 2024-01-24T17:41:05.080 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCJS7FlH5s7ChAAlFfFDziTgfv4ibWO08N+4Q== 2024-01-24T17:41:05.080 INFO:teuthology.orchestra.run.smithi039.stdout:[client.3] 2024-01-24T17:41:05.080 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCJS7FlO/U/FBAA2pI9aGSS2cZ3AMnn+W1fQA== 2024-01-24T17:41:05.114 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.y] 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCHS7Fl512kFRAAdCpU+QjRmGJmMD5uTdFmHQ== 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCHS7FlZZEtLBAAUxf4chL3tgy3zHlZoraXlg== 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCIS7Flo8iGARAAf+29RXWr4RHkILIz7em3mA== 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.0] 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCQS7FlOj/pLBAAzi89tAacOfDKdlU0BUJvvA== 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.1] 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCUS7FlbRAKDRAAAuAy47MDTtvmh8qNA3ZLTg== 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.2] 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCXS7Fl7QmRGRAAVECH7oYTdvDPSHBy9WmHwA== 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.3] 2024-01-24T17:41:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCaS7FlCQHIMxAA9OchsMzsKFxs110SeJ3Atw== 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.x] 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCHS7FlRbF6HxAAoFjlyWb5oqIkC3L1iPRmHw== 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCIS7Fln4utEhAA52ngJFqrdfa9bx56mVhRVA== 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCIS7Flh+08IRAARhZxy//j8c9b3Hg9uV9ajg== 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.4] 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCiS7FlDsBKBRAAXF2HafraT5nRbB5tBRUgWQ== 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.5] 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQClS7FlfrXBHhAAy8GP3TLX/iok3OYPGTP5Mw== 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.6] 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCoS7Fla11hNhAA/3oMP7cFtPSTdJl7eSnPTA== 2024-01-24T17:41:05.116 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.7] 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCsS7FlKFMtEhAAPmaFzr5XHVQaKZYkJiIarA== 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCIS7FlTi/3LhAAc8FwXtLH2XjlFMO7wDMcSQ== 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout:[client.2] 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCIS7FlIISlOhAAoVxxez7UjXDxyHQNNPnn2g== 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout:[client.1] 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCJS7FlH5s7ChAAlFfFDziTgfv4ibWO08N+4Q== 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout:[client.3] 2024-01-24T17:41:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCJS7FlO/U/FBAA2pI9aGSS2cZ3AMnn+W1fQA== 2024-01-24T17:41:05.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-24T17:41:05.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-24T17:41:05.278 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-24T17:41:05.283 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-24T17:41:05.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-24T17:41:05.434 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-24T17:41:05.577 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:05.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:05.726 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:05.734 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:05.877 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:05.884 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:06.031 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:06.038 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:06.182 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-24T17:41:06.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-24T17:41:06.326 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-24T17:41:06.331 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-24T17:41:06.471 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-24T17:41:06.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-24T17:41:06.626 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:06.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:06.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:06.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:06.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:06.932 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:07.075 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:07.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-24T17:41:07.222 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-24T17:41:07.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-24T17:41:07.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-24T17:41:07.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-24T17:41:07.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-24T17:41:07.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-24T17:41:07.685 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-24T17:41:07.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-24T17:41:07.839 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-24T17:41:07.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-24T17:41:07.935 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-24T17:41:08.103 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-24T17:41:08.188 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-24T17:41:08.288 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-24T17:41:08.441 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-24T17:41:08.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-24T17:41:08.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-24T17:41:08.773 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-24T17:41:08.863 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:41:08.871 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-24T17:41:08.924 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-24T17:41:08.924 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-24T17:41:08.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-24T17:41:08.929 INFO:tasks.ceph.mon.a:Started 2024-01-24T17:41:08.929 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-24T17:41:08.930 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-24T17:41:08.935 INFO:tasks.ceph.mon.b:Started 2024-01-24T17:41:08.935 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-24T17:41:08.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-24T17:41:08.939 INFO:tasks.ceph.mon.c:Started 2024-01-24T17:41:08.940 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-24T17:41:08.940 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-24T17:41:08.940 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-24T17:41:08.948 INFO:tasks.ceph.mgr.y:Started 2024-01-24T17:41:08.948 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-24T17:41:08.948 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-24T17:41:08.953 INFO:tasks.ceph.mgr.x:Started 2024-01-24T17:41:08.953 INFO:tasks.ceph:Setting crush tunables to default 2024-01-24T17:41:08.953 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd crush tunables default 2024-01-24T17:41:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.142+0000 7f8ab4400640 1 Processor -- start 2024-01-24T17:41:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.142+0000 7f8ab4400640 1 -- start start 2024-01-24T17:41:09.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab4400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 0x7f8aac070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:09.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab4400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8aac0b07f0 0x7f8aac160c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab4400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac070660 con 0x7f8aac1607d0 2024-01-24T17:41:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab4400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac1612d0 con 0x7f8aac0b07f0 2024-01-24T17:41:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab4400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1615b0 con 0x7f8aac1600d0 2024-01-24T17:41:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab1000640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f8aac1607d0 0x7f8aac0b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:54662/0 (socket says 172.21.15.39:54662) 2024-01-24T17:41:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab1000640 1 -- 172.21.15.39:0/2753476375 learned_addr learned my addr 172.21.15.39:0/2753476375 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2753476375 >> v2:172.21.15.39:3300/0 conn(0x7f8aac0b07f0 0x7f8aac160c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.144+0000 7f8ab1a00640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 0x7f8aac070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.145+0000 7f8ab1a00640 1 -- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 msgr2=0x7f8aac070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.145+0000 7f8ab1a00640 1 -- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 msgr2=0x7f8aac070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.145+0000 7f8ab1a00640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 0x7f8aac070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.145+0000 7f8ab1a00640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 0x7f8aac070af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.145+0000 7f8ab2400640 1 -- 172.21.15.39:0/2753476375 >> v2:172.21.15.39:3300/0 conn(0x7f8aac0b07f0 msgr2=0x7f8aac160c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.145+0000 7f8ab2400640 1 -- 172.21.15.39:0/2753476375 >> v2:172.21.15.39:3300/0 conn(0x7f8aac0b07f0 msgr2=0x7f8aac160c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.145+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2753476375 >> v2:172.21.15.39:3300/0 conn(0x7f8aac0b07f0 0x7f8aac160c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-24T17:41:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.145+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2753476375 >> v2:172.21.15.39:3300/0 conn(0x7f8aac0b07f0 0x7f8aac160c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T17:41:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.154+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746356231 0 0) 0x7f8aac070660 con 0x7f8aac1607d0 2024-01-24T17:41:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.154+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a8c003520 con 0x7f8aac1607d0 2024-01-24T17:41:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 <== mon.1 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1020423789 0 0) 0x7f8a8c003520 con 0x7f8aac1607d0 2024-01-24T17:41:09.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 msgr2=0x7f8aac070af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 0x7f8aac070af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 >> v2:172.21.15.39:3300/0 conn(0x7f8aac0b07f0 msgr2=0x7f8aac160c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 --2- 172.21.15.39:0/2753476375 >> v2:172.21.15.39:3300/0 conn(0x7f8aac0b07f0 0x7f8aac160c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac070010 con 0x7f8aac1607d0 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 <== mon.1 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8a94001670 con 0x7f8aac1607d0 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 >> v1:172.21.15.100:6789/0 conn(0x7f8aac1607d0 legacy=0x7f8aac0b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 0x7f8a8c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a8c003ff0 0x7f8a8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8a8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1612d0 con 0x7f8a8c004d00 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1615b0 con 0x7f8a8c003ff0 2024-01-24T17:41:09.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c00bb70 con 0x7f8aac1600d0 2024-01-24T17:41:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8a8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8ab1000640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a8c003ff0 0x7f8a8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8ab1a00640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 0x7f8a8c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8ab2400640 1 -- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 msgr2=0x7f8a8c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac1600d0 0x7f8a8c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8ab2400640 1 -- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a8c003ff0 msgr2=0x7f8a8c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a8c003ff0 0x7f8a8c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.155+0000 7f8ab2400640 1 -- 172.21.15.39:0/2753476375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c003520 con 0x7f8a8c004d00 2024-01-24T17:41:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.157+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8a8c005180 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0adcc0 tx=0x7f8a8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6441c972a51d0ba server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.158+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa402cc70 con 0x7f8a8c004d00 2024-01-24T17:41:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.158+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa402cdd0 con 0x7f8a8c004d00 2024-01-24T17:41:09.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.158+0000 7f8aabe00640 1 -- 172.21.15.39:0/2753476375 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4034680 con 0x7f8a8c004d00 2024-01-24T17:41:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.159+0000 7f8ab4400640 1 -- 172.21.15.39:0/2753476375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 msgr2=0x7f8a8c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.159+0000 7f8ab4400640 1 --2- 172.21.15.39:0/2753476375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8a8c005180 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0adcc0 tx=0x7f8a8c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.159+0000 7f8ab2400640 1 -- 172.21.15.39:0/2753476375 reap_dead start 2024-01-24T17:41:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.159+0000 7f8ab4400640 1 -- 172.21.15.39:0/2753476375 shutdown_connections 2024-01-24T17:41:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.159+0000 7f8ab4400640 1 -- 172.21.15.39:0/2753476375 >> 172.21.15.39:0/2753476375 conn(0x7f8aac0cb0e0 msgr2=0x7f8aac0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.159+0000 7f8ab4400640 1 -- 172.21.15.39:0/2753476375 shutdown_connections 2024-01-24T17:41:09.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.159+0000 7f8ab4400640 1 -- 172.21.15.39:0/2753476375 wait complete. 2024-01-24T17:41:09.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.161+0000 7f8ab4400640 1 Processor -- start 2024-01-24T17:41:09.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.161+0000 7f8ab4400640 1 -- start start 2024-01-24T17:41:09.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab4400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8aac0c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:09.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab4400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8aac1f5b30 0x7f8aac0ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:09.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab4400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac0aeca0 0x7f8aac151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:09.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab4400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0ae340 con 0x7f8a8c004d00 2024-01-24T17:41:09.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab4400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0c5550 con 0x7f8aac1f5b30 2024-01-24T17:41:09.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab4400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac1f6280 con 0x7f8aac0aeca0 2024-01-24T17:41:09.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab1000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8aac1f5b30 0x7f8aac0ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:09.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab1000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8aac1f5b30 0x7f8aac0ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:59900/0 (socket says 172.21.15.39:59900) 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab1000640 1 -- 172.21.15.39:0/2526790661 learned_addr learned my addr 172.21.15.39:0/2526790661 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.163+0000 7f8ab1a00640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8aac0c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab1000640 1 -- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac0aeca0 msgr2=0x7f8aac151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac0aeca0 0x7f8aac151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab1000640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac0aeca0 0x7f8aac151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab1000640 1 -- 172.21.15.39:0/2526790661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 msgr2=0x7f8aac0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab1000640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8aac0c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab1000640 1 -- 172.21.15.39:0/2526790661 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac1f6560 con 0x7f8aac1f5b30 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab1a00640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8aac0c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab2400640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac0aeca0 0x7f8aac151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.164+0000 7f8ab1000640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8aac1f5b30 0x7f8aac0ae760 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8a94000fb0 tx=0x7f8a940077a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.165+0000 7f8aa9600640 1 -- 172.21.15.39:0/2526790661 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400f960 con 0x7f8aac1f5b30 2024-01-24T17:41:09.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.165+0000 7f8aa9600640 1 -- 172.21.15.39:0/2526790661 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9400fae0 con 0x7f8aac1f5b30 2024-01-24T17:41:09.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.166+0000 7f8aa9600640 1 -- 172.21.15.39:0/2526790661 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400fe90 con 0x7f8aac1f5b30 2024-01-24T17:41:09.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.166+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac1f6840 con 0x7f8aac1f5b30 2024-01-24T17:41:09.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.166+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac1f2300 con 0x7f8aac1f5b30 2024-01-24T17:41:09.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.166+0000 7f8aa9600640 1 -- 172.21.15.39:0/2526790661 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8a9401d760 con 0x7f8aac1f5b30 2024-01-24T17:41:09.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.166+0000 7f8aa9600640 1 -- 172.21.15.39:0/2526790661 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8a94013820 con 0x7f8aac1f5b30 2024-01-24T17:41:09.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.167+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac0af080 con 0x7f8aac1f5b30 2024-01-24T17:41:09.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.170+0000 7f8aa9600640 1 -- 172.21.15.39:0/2526790661 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f8a9401dba0 con 0x7f8aac1f5b30 2024-01-24T17:41:09.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.241+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8aac0ae0a0 con 0x7f8aac1f5b30 2024-01-24T17:41:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.251+0000 7f8aa9600640 1 -- 172.21.15.39:0/2526790661 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8aac0ae0a0 con 0x7f8aac1f5b30 2024-01-24T17:41:09.252 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2024-01-24T17:41:09.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.252+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8aac1f5b30 msgr2=0x7f8aac0ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:09.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.252+0000 7f8ab4400640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8aac1f5b30 0x7f8aac0ae760 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8a94000fb0 tx=0x7f8a940077a0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.252+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 shutdown_connections 2024-01-24T17:41:09.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.252+0000 7f8ab4400640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8aac0aeca0 0x7f8aac151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.252+0000 7f8ab4400640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8aac1f5b30 0x7f8aac0ae760 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.252+0000 7f8ab4400640 1 --2- 172.21.15.39:0/2526790661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a8c004d00 0x7f8aac0c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.253+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 >> 172.21.15.39:0/2526790661 conn(0x7f8aac0cb0e0 msgr2=0x7f8aac0ab950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.253+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 shutdown_connections 2024-01-24T17:41:09.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:09.253+0000 7f8ab4400640 1 -- 172.21.15.39:0/2526790661 wait complete. 2024-01-24T17:41:09.271 INFO:tasks.ceph:check_enable_crimson: False 2024-01-24T17:41:09.271 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-24T17:41:09.271 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:09.271 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-24T17:41:09.352 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:09.352 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-24T17:41:09.469 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:09.470 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-24T17:41:09.588 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:41:09.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-24T17:41:09.722 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:09.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-24T17:41:09.812 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:09.812 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-24T17:41:09.890 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:09.889+0000 7fd73f014040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T17:41:09.898 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:09.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-24T17:41:09.909 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:09.908+0000 7fad26433040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T17:41:09.955 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:09.954+0000 7fd73f014040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T17:41:09.975 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:09.975+0000 7fad26433040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T17:41:10.028 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:41:10.028 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-24T17:41:10.162 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new c62a7326-0fe4-4a6b-93ec-6ac162fe8358 0 2024-01-24T17:41:10.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.349+0000 7fbac9000640 1 Processor -- start 2024-01-24T17:41:10.349 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.349+0000 7fbac9000640 1 -- start start 2024-01-24T17:41:10.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.350+0000 7fbac9000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbac40d14c0 0x7fbac40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.350+0000 7fbac9000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac40b2a10 0x7fbac40cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.350+0000 7fbac9000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac40cda90 con 0x7fbac415fff0 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.350+0000 7fbac9000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac40b2ec0 con 0x7fbac40d14c0 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.350+0000 7fbac9000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac40d32c0 con 0x7fbac40b2a10 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac1e00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fbac415fff0 0x7fbac40d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:50060/0 (socket says 172.21.15.100:50060) 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac1e00640 1 -- 172.21.15.100:0/1678856957 learned_addr learned my addr 172.21.15.100:0/1678856957 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac2800640 1 --2- 172.21.15.100:0/1678856957 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac40b2a10 0x7fbac40cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac1400640 1 --2- 172.21.15.100:0/1678856957 >> v2:172.21.15.39:3300/0 conn(0x7fbac40d14c0 0x7fbac40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac0a00640 1 -- 172.21.15.100:0/1678856957 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333481014 0 0) 0x7fbac40cda90 con 0x7fbac415fff0 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac0a00640 1 -- 172.21.15.100:0/1678856957 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa4003520 con 0x7fbac415fff0 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac2800640 1 -- 172.21.15.100:0/1678856957 >> v2:172.21.15.39:3300/0 conn(0x7fbac40d14c0 msgr2=0x7fbac40b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac2800640 1 --2- 172.21.15.100:0/1678856957 >> v2:172.21.15.39:3300/0 conn(0x7fbac40d14c0 0x7fbac40b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac2800640 1 -- 172.21.15.100:0/1678856957 >> v1:172.21.15.100:6789/0 conn(0x7fbac415fff0 legacy=0x7fbac40d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.352 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.351+0000 7fbac2800640 1 -- 172.21.15.100:0/1678856957 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac40b30a0 con 0x7fbac40b2a10 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.352+0000 7fbac2800640 1 --2- 172.21.15.100:0/1678856957 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac40b2a10 0x7fbac40cdf20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009d90 tx=0x7fbabc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1b7a319d78e3700 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac0a00640 1 -- 172.21.15.100:0/1678856957 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc02cc50 con 0x7fbac40b2a10 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac0a00640 1 -- 172.21.15.100:0/1678856957 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc02cdb0 con 0x7fbac40b2a10 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac0a00640 1 -- 172.21.15.100:0/1678856957 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc034640 con 0x7fbac40b2a10 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac9000640 1 -- 172.21.15.100:0/1678856957 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac40b2a10 msgr2=0x7fbac40cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac9000640 1 --2- 172.21.15.100:0/1678856957 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac40b2a10 0x7fbac40cdf20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbabc009d90 tx=0x7fbabc02bea0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac9000640 1 -- 172.21.15.100:0/1678856957 shutdown_connections 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac9000640 1 --2- 172.21.15.100:0/1678856957 >> v2:172.21.15.39:3300/0 conn(0x7fbac40d14c0 0x7fbac40b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac9000640 1 --2- 172.21.15.100:0/1678856957 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac40b2a10 0x7fbac40cdf20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac9000640 1 -- 172.21.15.100:0/1678856957 >> 172.21.15.100:0/1678856957 conn(0x7fbac409bcc0 msgr2=0x7fbac40d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac9000640 1 -- 172.21.15.100:0/1678856957 shutdown_connections 2024-01-24T17:41:10.354 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.353+0000 7fbac9000640 1 -- 172.21.15.100:0/1678856957 wait complete. 2024-01-24T17:41:10.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.355+0000 7fbac9000640 1 Processor -- start 2024-01-24T17:41:10.355 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.355+0000 7fbac9000640 1 -- start start 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac9000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbac40b2a10 0x7fbac415d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbac40d14c0 0x7fbac41f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac9000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac415fff0 0x7fbac4155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac9000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac4153cb0 con 0x7fbac40d14c0 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac9000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41f6090 con 0x7fbac40b2a10 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac9000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41f6370 con 0x7fbac415fff0 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac2800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac415fff0 0x7fbac4155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac1e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbac40b2a10 0x7fbac415d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac2800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac415fff0 0x7fbac4155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51936/0 (socket says 172.21.15.100:51936) 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac2800640 1 -- 172.21.15.100:0/719156925 learned_addr learned my addr 172.21.15.100:0/719156925 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.356+0000 7fbac1400640 1 --2- 172.21.15.100:0/719156925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbac40d14c0 0x7fbac41f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbac1e00640 1 -- 172.21.15.100:0/719156925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac415fff0 msgr2=0x7fbac4155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.358 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbac1e00640 1 --2- 172.21.15.100:0/719156925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac415fff0 0x7fbac4155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbac1e00640 1 -- 172.21.15.100:0/719156925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbac40d14c0 msgr2=0x7fbac41f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbac1e00640 1 --2- 172.21.15.100:0/719156925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbac40d14c0 0x7fbac41f5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbac1e00640 1 -- 172.21.15.100:0/719156925 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc009a70 con 0x7fbac40b2a10 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbac1e00640 1 --2- 172.21.15.100:0/719156925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbac40b2a10 0x7fbac415d640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00c390 tx=0x7fbaac001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbaba000640 1 -- 172.21.15.100:0/719156925 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac005ce0 con 0x7fbac40b2a10 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbaba000640 1 -- 172.21.15.100:0/719156925 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac005e40 con 0x7fbac40b2a10 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbaba000640 1 -- 172.21.15.100:0/719156925 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0072f0 con 0x7fbac40b2a10 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac4153f00 con 0x7fbac40b2a10 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.357+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac4158730 con 0x7fbac40b2a10 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.358+0000 7fbaba000640 1 -- 172.21.15.100:0/719156925 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbaac01c440 con 0x7fbac40b2a10 2024-01-24T17:41:10.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.358+0000 7fbaba000640 1 -- 172.21.15.100:0/719156925 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbaac012800 con 0x7fbac40b2a10 2024-01-24T17:41:10.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.358+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac4154420 con 0x7fbac40b2a10 2024-01-24T17:41:10.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.361+0000 7fbaba000640 1 -- 172.21.15.100:0/719156925 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fbaac017070 con 0x7fbac40b2a10 2024-01-24T17:41:10.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.439+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c62a7326-0fe4-4a6b-93ec-6ac162fe8358", "id": 0} v 0) v1 -- 0x7fbac4073ef0 con 0x7fbac40b2a10 2024-01-24T17:41:10.447 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.447+0000 7fbaba000640 1 -- 172.21.15.100:0/719156925 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c62a7326-0fe4-4a6b-93ec-6ac162fe8358", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbaac012ae0 con 0x7fbac40b2a10 2024-01-24T17:41:10.448 INFO:teuthology.orchestra.run.smithi100.stdout:0 2024-01-24T17:41:10.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.448+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbac40b2a10 msgr2=0x7fbac415d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.448+0000 7fbac9000640 1 --2- 172.21.15.100:0/719156925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbac40b2a10 0x7fbac415d640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbaac00c390 tx=0x7fbaac001e70 comp rx=0 tx=0).stop 2024-01-24T17:41:10.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.448+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 shutdown_connections 2024-01-24T17:41:10.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.448+0000 7fbac9000640 1 --2- 172.21.15.100:0/719156925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbac415fff0 0x7fbac4155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.448+0000 7fbac9000640 1 --2- 172.21.15.100:0/719156925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbac40d14c0 0x7fbac41f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.449+0000 7fbac9000640 1 --2- 172.21.15.100:0/719156925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbac40b2a10 0x7fbac415d640 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.449+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 >> 172.21.15.100:0/719156925 conn(0x7fbac409bcc0 msgr2=0x7fbac40af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:10.449 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.449+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 shutdown_connections 2024-01-24T17:41:10.450 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.449+0000 7fbac9000640 1 -- 172.21.15.100:0/719156925 wait complete. 2024-01-24T17:41:10.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 25fca6a0-197d-4518-bdaf-9bf4eec56bc9 1 2024-01-24T17:41:10.583 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:10.583+0000 7fd73f014040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T17:41:10.633 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:10.633+0000 7fad26433040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T17:41:10.647 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:10.647+0000 7fd73f014040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T17:41:10.654 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.653+0000 7f14c8e00640 1 Processor -- start 2024-01-24T17:41:10.654 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.654+0000 7f14c8e00640 1 -- start start 2024-01-24T17:41:10.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.655+0000 7f14c8e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f14c415d000 0x7f14c40cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.655+0000 7f14c8e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cad40 0x7f14c40ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.655+0000 7f14c8e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c40cb2b0 con 0x7f14c40cbd80 2024-01-24T17:41:10.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.655+0000 7f14c8e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c415d3e0 con 0x7f14c415d000 2024-01-24T17:41:10.656 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.655+0000 7f14c8e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b0a50 con 0x7f14c40cad40 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.655+0000 7f14c1e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f14c415d000 0x7f14c40cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.656+0000 7f14c2800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cad40 0x7f14c40ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.656+0000 7f14c1e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f14c415d000 0x7f14c40cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57354/0 (socket says 172.21.15.100:57354) 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.656+0000 7f14c1e00640 1 -- 172.21.15.100:0/3620585157 learned_addr learned my addr 172.21.15.100:0/3620585157 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.656+0000 7f14c1e00640 1 -- 172.21.15.100:0/3620585157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cad40 msgr2=0x7f14c40ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.656+0000 7f14c1e00640 1 --2- 172.21.15.100:0/3620585157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cad40 0x7f14c40ceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.656+0000 7f14c1e00640 1 -- 172.21.15.100:0/3620585157 >> v1:172.21.15.100:6789/0 conn(0x7f14c40cbd80 legacy=0x7f14c40b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.656+0000 7f14c1e00640 1 -- 172.21.15.100:0/3620585157 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40b0d30 con 0x7f14c415d000 2024-01-24T17:41:10.657 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.656+0000 7f14c2800640 1 --2- 172.21.15.100:0/3620585157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cad40 0x7f14c40ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:10.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.657+0000 7f14c1e00640 1 --2- 172.21.15.100:0/3620585157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c415d000 0x7f14c40cd2b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009870 tx=0x7f14b402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa87b765de18089a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:10.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.657+0000 7f14c1400640 1 -- 172.21.15.100:0/3620585157 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4030840 con 0x7f14c415d000 2024-01-24T17:41:10.658 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.657+0000 7f14c1400640 1 -- 172.21.15.100:0/3620585157 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b40309a0 con 0x7f14c415d000 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.657+0000 7f14c1400640 1 -- 172.21.15.100:0/3620585157 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4030d30 con 0x7f14c415d000 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.658+0000 7f14c8e00640 1 -- 172.21.15.100:0/3620585157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c415d000 msgr2=0x7f14c40cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.658+0000 7f14c8e00640 1 --2- 172.21.15.100:0/3620585157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c415d000 0x7f14c40cd2b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009870 tx=0x7f14b402b980 comp rx=0 tx=0).stop 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.658+0000 7f14c8e00640 1 -- 172.21.15.100:0/3620585157 shutdown_connections 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.658+0000 7f14c8e00640 1 --2- 172.21.15.100:0/3620585157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cad40 0x7f14c40ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.658+0000 7f14c8e00640 1 --2- 172.21.15.100:0/3620585157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c415d000 0x7f14c40cd2b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.658+0000 7f14c8e00640 1 -- 172.21.15.100:0/3620585157 >> 172.21.15.100:0/3620585157 conn(0x7f14c4098a40 msgr2=0x7f14c40cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.658+0000 7f14c8e00640 1 -- 172.21.15.100:0/3620585157 shutdown_connections 2024-01-24T17:41:10.659 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.658+0000 7f14c8e00640 1 -- 172.21.15.100:0/3620585157 wait complete. 2024-01-24T17:41:10.660 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.660+0000 7f14c8e00640 1 Processor -- start 2024-01-24T17:41:10.660 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.660+0000 7f14c8e00640 1 -- start start 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c8e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c40cad40 0x7f14c414f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c8e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cbd80 0x7f14c4150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c415d000 0x7f14c40b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c8e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c4151670 con 0x7f14c415d000 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c8e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b26c0 con 0x7f14c40cad40 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c8e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c40b29a0 con 0x7f14c40cbd80 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c1e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c40cad40 0x7f14c414f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c1e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c40cad40 0x7f14c414f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57360/0 (socket says 172.21.15.100:57360) 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.661+0000 7f14c1e00640 1 -- 172.21.15.100:0/1622338282 learned_addr learned my addr 172.21.15.100:0/1622338282 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:10.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14c1e00640 1 -- 172.21.15.100:0/1622338282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cbd80 msgr2=0x7f14c4150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14c1e00640 1 --2- 172.21.15.100:0/1622338282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cbd80 0x7f14c4150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14c1e00640 1 -- 172.21.15.100:0/1622338282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c415d000 msgr2=0x7f14c40b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14c1e00640 1 --2- 172.21.15.100:0/1622338282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c415d000 0x7f14c40b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14c1e00640 1 -- 172.21.15.100:0/1622338282 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc009a70 con 0x7f14c40cad40 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14c1e00640 1 --2- 172.21.15.100:0/1622338282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c40cad40 0x7f14c414f7c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000c00 tx=0x7f14b4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14ba000640 1 -- 172.21.15.100:0/1622338282 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4039a10 con 0x7f14c40cad40 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14ba000640 1 -- 172.21.15.100:0/1622338282 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b4039b70 con 0x7f14c40cad40 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b40094d0 con 0x7f14c40cad40 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.662+0000 7f14ba000640 1 -- 172.21.15.100:0/1622338282 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b4043410 con 0x7f14c40cad40 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.663+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c40b2f60 con 0x7f14c40cad40 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.663+0000 7f14ba000640 1 -- 172.21.15.100:0/1622338282 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f14b4043870 con 0x7f14c40cad40 2024-01-24T17:41:10.664 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.663+0000 7f14ba000640 1 -- 172.21.15.100:0/1622338282 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f14b404dc40 con 0x7f14c40cad40 2024-01-24T17:41:10.665 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.664+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c4073ef0 con 0x7f14c40cad40 2024-01-24T17:41:10.666 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.666+0000 7f14ba000640 1 -- 172.21.15.100:0/1622338282 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f14b4048070 con 0x7f14c40cad40 2024-01-24T17:41:10.698 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:10.697+0000 7fad26433040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T17:41:10.702 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:10.702+0000 7fd73f014040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T17:41:10.743 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.743+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "25fca6a0-197d-4518-bdaf-9bf4eec56bc9", "id": 1} v 0) v1 -- 0x7f14c40c5ff0 con 0x7f14c40cad40 2024-01-24T17:41:10.751 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.751+0000 7f14ba000640 1 -- 172.21.15.100:0/1622338282 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "25fca6a0-197d-4518-bdaf-9bf4eec56bc9", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f14b4034070 con 0x7f14c40cad40 2024-01-24T17:41:10.751 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:41:10.751 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:10.751+0000 7fad26433040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c40cad40 msgr2=0x7f14c414f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 --2- 172.21.15.100:0/1622338282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c40cad40 0x7f14c414f7c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000c00 tx=0x7f14b4004730 comp rx=0 tx=0).stop 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 shutdown_connections 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 --2- 172.21.15.100:0/1622338282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c415d000 0x7f14c40b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 --2- 172.21.15.100:0/1622338282 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c40cbd80 0x7f14c4150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 --2- 172.21.15.100:0/1622338282 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c40cad40 0x7f14c414f7c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 >> 172.21.15.100:0/1622338282 conn(0x7f14c4098a40 msgr2=0x7f14c406a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 shutdown_connections 2024-01-24T17:41:10.753 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.752+0000 7f14c8e00640 1 -- 172.21.15.100:0/1622338282 wait complete. 2024-01-24T17:41:10.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new cd1f3734-5bd2-4183-a0e5-1c5f519b42d6 2 2024-01-24T17:41:10.775 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:10.771+0000 7fd73f014040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T17:41:10.817 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:10.816+0000 7fad26433040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T17:41:10.824 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:10.824+0000 7fd73f014040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T17:41:10.868 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:10.868+0000 7fad26433040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T17:41:10.942 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:10.942+0000 7fd73f014040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T17:41:10.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.953+0000 7f1ae8800640 1 Processor -- start 2024-01-24T17:41:10.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.953+0000 7f1ae8800640 1 -- start start 2024-01-24T17:41:10.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae8800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1ae015dd70 0x7f1ae00cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae8800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015e150 0x7f1ae0069c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae8800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae00cbef0 con 0x7f1ae00cb950 2024-01-24T17:41:10.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae8800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae00b11e0 con 0x7f1ae015dd70 2024-01-24T17:41:10.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae8800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00b14c0 con 0x7f1ae015e150 2024-01-24T17:41:10.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015e150 0x7f1ae0069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.956 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015e150 0x7f1ae0069c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51962/0 (socket says 172.21.15.100:51962) 2024-01-24T17:41:10.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae5400640 1 -- 172.21.15.100:0/3875542330 learned_addr learned my addr 172.21.15.100:0/3875542330 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:10.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae5400640 1 -- 172.21.15.100:0/3875542330 >> v2:172.21.15.39:3300/0 conn(0x7f1ae015dd70 msgr2=0x7f1ae00cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:10.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.955+0000 7f1ae5e00640 1 --2- 172.21.15.100:0/3875542330 >> v2:172.21.15.39:3300/0 conn(0x7f1ae015dd70 0x7f1ae00cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.956+0000 7f1ae5400640 1 --2- 172.21.15.100:0/3875542330 >> v2:172.21.15.39:3300/0 conn(0x7f1ae015dd70 0x7f1ae00cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.956+0000 7f1ae5400640 1 -- 172.21.15.100:0/3875542330 >> v1:172.21.15.39:6789/0 conn(0x7f1ae00cb950 legacy=0x7f1ae006cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:10.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.956+0000 7f1ae5400640 1 -- 172.21.15.100:0/3875542330 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae00b17a0 con 0x7f1ae015e150 2024-01-24T17:41:10.957 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.956+0000 7f1ae5e00640 1 --2- 172.21.15.100:0/3875542330 >> v2:172.21.15.39:3300/0 conn(0x7f1ae015dd70 0x7f1ae00cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.957+0000 7f1ae5400640 1 --2- 172.21.15.100:0/3875542330 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015e150 0x7f1ae0069c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009870 tx=0x7f1ac802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8d8e233ca7477c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.957+0000 7f1ae4a00640 1 -- 172.21.15.100:0/3875542330 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80317c0 con 0x7f1ae015e150 2024-01-24T17:41:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.957+0000 7f1ae4a00640 1 -- 172.21.15.100:0/3875542330 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac8031920 con 0x7f1ae015e150 2024-01-24T17:41:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.957+0000 7f1ae4a00640 1 -- 172.21.15.100:0/3875542330 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8031cb0 con 0x7f1ae015e150 2024-01-24T17:41:10.958 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.958+0000 7f1ae8800640 1 -- 172.21.15.100:0/3875542330 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015e150 msgr2=0x7f1ae0069c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.958+0000 7f1ae8800640 1 --2- 172.21.15.100:0/3875542330 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015e150 0x7f1ae0069c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009870 tx=0x7f1ac802b860 comp rx=0 tx=0).stop 2024-01-24T17:41:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.958+0000 7f1ae8800640 1 -- 172.21.15.100:0/3875542330 shutdown_connections 2024-01-24T17:41:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.958+0000 7f1ae8800640 1 --2- 172.21.15.100:0/3875542330 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015e150 0x7f1ae0069c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.958+0000 7f1ae8800640 1 --2- 172.21.15.100:0/3875542330 >> v2:172.21.15.39:3300/0 conn(0x7f1ae015dd70 0x7f1ae00cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.958+0000 7f1ae8800640 1 -- 172.21.15.100:0/3875542330 >> 172.21.15.100:0/3875542330 conn(0x7f1ae0098a40 msgr2=0x7f1ae00cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.958+0000 7f1ae8800640 1 -- 172.21.15.100:0/3875542330 shutdown_connections 2024-01-24T17:41:10.959 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.958+0000 7f1ae8800640 1 -- 172.21.15.100:0/3875542330 wait complete. 2024-01-24T17:41:10.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.960+0000 7f1ae8800640 1 Processor -- start 2024-01-24T17:41:10.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.960+0000 7f1ae8800640 1 -- start start 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae8800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ae00cb950 0x7f1ae00c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae8800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015dd70 0x7f1ae014b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae8800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ae015e150 0x7f1ae0147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae8800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00c89e0 con 0x7f1ae015e150 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae8800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae014b690 con 0x7f1ae00cb950 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae8800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00c7dd0 con 0x7f1ae015dd70 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015dd70 0x7f1ae014b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ae00cb950 0x7f1ae00c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015dd70 0x7f1ae014b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51966/0 (socket says 172.21.15.100:51966) 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5400640 1 -- 172.21.15.100:0/4118269119 learned_addr learned my addr 172.21.15.100:0/4118269119 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae6800640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ae015e150 0x7f1ae0147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5400640 1 -- 172.21.15.100:0/4118269119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ae00cb950 msgr2=0x7f1ae00c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5400640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ae00cb950 0x7f1ae00c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5400640 1 -- 172.21.15.100:0/4118269119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ae015e150 msgr2=0x7f1ae0147d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5400640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ae015e150 0x7f1ae0147d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5400640 1 -- 172.21.15.100:0/4118269119 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac80094d0 con 0x7f1ae015dd70 2024-01-24T17:41:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.961+0000 7f1ae5e00640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ae00cb950 0x7f1ae00c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:10.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.962+0000 7f1ae5400640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015dd70 0x7f1ae014b150 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ac802b8a0 tx=0x7f1ac80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.962+0000 7f1ade000640 1 -- 172.21.15.100:0/4118269119 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80329a0 con 0x7f1ae015dd70 2024-01-24T17:41:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.962+0000 7f1ade000640 1 -- 172.21.15.100:0/4118269119 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac8032b40 con 0x7f1ae015dd70 2024-01-24T17:41:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.962+0000 7f1ade000640 1 -- 172.21.15.100:0/4118269119 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80453f0 con 0x7f1ae015dd70 2024-01-24T17:41:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.962+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae00c80b0 con 0x7f1ae015dd70 2024-01-24T17:41:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.962+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae00c8670 con 0x7f1ae015dd70 2024-01-24T17:41:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.963+0000 7f1ade000640 1 -- 172.21.15.100:0/4118269119 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1ac8045850 con 0x7f1ae015dd70 2024-01-24T17:41:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.963+0000 7f1ade000640 1 -- 172.21.15.100:0/4118269119 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f1ac803b3f0 con 0x7f1ae015dd70 2024-01-24T17:41:10.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.963+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae00b1a30 con 0x7f1ae015dd70 2024-01-24T17:41:10.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:10.966+0000 7f1ade000640 1 -- 172.21.15.100:0/4118269119 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f1ac8045b00 con 0x7f1ae015dd70 2024-01-24T17:41:10.987 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:10.986+0000 7fad26433040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T17:41:11.026 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:11.026+0000 7fd73f014040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T17:41:11.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.043+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cd1f3734-5bd2-4183-a0e5-1c5f519b42d6", "id": 2} v 0) v1 -- 0x7f1ae0067040 con 0x7f1ae015dd70 2024-01-24T17:41:11.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.052+0000 7f1ade000640 1 -- 172.21.15.100:0/4118269119 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cd1f3734-5bd2-4183-a0e5-1c5f519b42d6", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1ac8045d80 con 0x7f1ae015dd70 2024-01-24T17:41:11.052 INFO:teuthology.orchestra.run.smithi100.stdout:2 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.053+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015dd70 msgr2=0x7f1ae014b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.053+0000 7f1ae8800640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015dd70 0x7f1ae014b150 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ac802b8a0 tx=0x7f1ac80046c0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.054+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 shutdown_connections 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.054+0000 7f1ae8800640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ae015e150 0x7f1ae0147d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.054+0000 7f1ae8800640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ae015dd70 0x7f1ae014b150 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.054+0000 7f1ae8800640 1 --2- 172.21.15.100:0/4118269119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ae00cb950 0x7f1ae00c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.054+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 >> 172.21.15.100:0/4118269119 conn(0x7f1ae0098a40 msgr2=0x7f1ae0069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.054+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 shutdown_connections 2024-01-24T17:41:11.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.054+0000 7f1ae8800640 1 -- 172.21.15.100:0/4118269119 wait complete. 2024-01-24T17:41:11.070 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 840d0d11-bf09-4c4a-aff4-23345809b0d3 3 2024-01-24T17:41:11.071 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:11.070+0000 7fad26433040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T17:41:11.083 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:11.083+0000 7fd73f014040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T17:41:11.127 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:11.126+0000 7fad26433040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T17:41:11.147 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:11.147+0000 7fd73f014040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T17:41:11.192 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:11.190+0000 7fad26433040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T17:41:11.204 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:11.204+0000 7fd73f014040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T17:41:11.248 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:11.247+0000 7fad26433040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T17:41:11.261 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.261+0000 7f008e600640 1 Processor -- start 2024-01-24T17:41:11.262 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.261+0000 7f008e600640 1 -- start start 2024-01-24T17:41:11.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.263+0000 7f008e600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f00880b0350 0x7f008806c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.263+0000 7f008e600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008806d4c0 con 0x7f008806c9f0 2024-01-24T17:41:11.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.263+0000 7f008e600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008806cf00 con 0x7f008815e190 2024-01-24T17:41:11.263 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.263+0000 7f008e600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008806d1e0 con 0x7f00880b0350 2024-01-24T17:41:11.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.263+0000 7f0087400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f00880b0350 0x7f008806c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.263+0000 7f0087400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f00880b0350 0x7f008806c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57370/0 (socket says 172.21.15.100:57370) 2024-01-24T17:41:11.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.263+0000 7f0087400640 1 -- 172.21.15.100:0/800579094 learned_addr learned my addr 172.21.15.100:0/800579094 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:11.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.264+0000 7f0087400640 1 -- 172.21.15.100:0/800579094 >> v1:172.21.15.100:6789/0 conn(0x7f008815e190 legacy=0x7f00880b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.264+0000 7f0087400640 1 -- 172.21.15.100:0/800579094 >> v1:172.21.15.39:6789/0 conn(0x7f008806c9f0 legacy=0x7f008815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:11.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.264+0000 7f0087400640 1 -- 172.21.15.100:0/800579094 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00880b15e0 con 0x7f00880b0350 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.265+0000 7f0087400640 1 --2- 172.21.15.100:0/800579094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f008806c4b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0078009870 tx=0x7f007802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33ae47f61f34676b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.265+0000 7f0086000640 1 -- 172.21.15.100:0/800579094 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0078031870 con 0x7f00880b0350 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.265+0000 7f0086000640 1 -- 172.21.15.100:0/800579094 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00780319d0 con 0x7f00880b0350 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.265+0000 7f0086000640 1 -- 172.21.15.100:0/800579094 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0078031d60 con 0x7f00880b0350 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.265+0000 7f008e600640 1 -- 172.21.15.100:0/800579094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 msgr2=0x7f008806c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.265+0000 7f008e600640 1 --2- 172.21.15.100:0/800579094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f008806c4b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0078009870 tx=0x7f007802b910 comp rx=0 tx=0).stop 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.266+0000 7f008e600640 1 -- 172.21.15.100:0/800579094 shutdown_connections 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.266+0000 7f008e600640 1 --2- 172.21.15.100:0/800579094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f008806c4b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.266+0000 7f008e600640 1 -- 172.21.15.100:0/800579094 >> 172.21.15.100:0/800579094 conn(0x7f00880cb0e0 msgr2=0x7f00880cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.266+0000 7f008e600640 1 -- 172.21.15.100:0/800579094 shutdown_connections 2024-01-24T17:41:11.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.266+0000 7f008e600640 1 -- 172.21.15.100:0/800579094 wait complete. 2024-01-24T17:41:11.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.267+0000 7f008e600640 1 Processor -- start 2024-01-24T17:41:11.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.268+0000 7f008e600640 1 -- start start 2024-01-24T17:41:11.269 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.268+0000 7f008e600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008806c9f0 0x7f0088159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.268+0000 7f008e600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f0088153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.268+0000 7f008e600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008815e190 0x7f00880cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.268+0000 7f008e600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881f2480 con 0x7f008806c9f0 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.268+0000 7f008e600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00881f2740 con 0x7f00880b0350 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.268+0000 7f008e600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00881f2a20 con 0x7f008815e190 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0086a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f0088153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0086a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f0088153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57382/0 (socket says 172.21.15.100:57382) 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0086a00640 1 -- 172.21.15.100:0/1617628896 learned_addr learned my addr 172.21.15.100:0/1617628896 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0087e00640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008815e190 0x7f00880cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0087400640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008806c9f0 0x7f0088159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0086a00640 1 -- 172.21.15.100:0/1617628896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008815e190 msgr2=0x7f00880cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0086a00640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008815e190 0x7f00880cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0086a00640 1 -- 172.21.15.100:0/1617628896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008806c9f0 msgr2=0x7f0088159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0086a00640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008806c9f0 0x7f0088159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.269+0000 7f0086a00640 1 -- 172.21.15.100:0/1617628896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00780094d0 con 0x7f00880b0350 2024-01-24T17:41:11.271 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.270+0000 7f0086a00640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f0088153440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f007000c430 tx=0x7f007000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:11.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.270+0000 7f007f400640 1 -- 172.21.15.100:0/1617628896 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070011790 con 0x7f00880b0350 2024-01-24T17:41:11.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.270+0000 7f007f400640 1 -- 172.21.15.100:0/1617628896 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00700118f0 con 0x7f00880b0350 2024-01-24T17:41:11.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.270+0000 7f007f400640 1 -- 172.21.15.100:0/1617628896 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070011ce0 con 0x7f00880b0350 2024-01-24T17:41:11.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.270+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00881f2d00 con 0x7f00880b0350 2024-01-24T17:41:11.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.270+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008815a350 con 0x7f00880b0350 2024-01-24T17:41:11.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.270+0000 7f007f400640 1 -- 172.21.15.100:0/1617628896 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f007001c760 con 0x7f00880b0350 2024-01-24T17:41:11.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.270+0000 7f007f400640 1 -- 172.21.15.100:0/1617628896 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f0070011e40 con 0x7f00880b0350 2024-01-24T17:41:11.272 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.271+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00881f3220 con 0x7f00880b0350 2024-01-24T17:41:11.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.273+0000 7f007f400640 1 -- 172.21.15.100:0/1617628896 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f00881f3220 con 0x7f00880b0350 2024-01-24T17:41:11.351 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.351+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "840d0d11-bf09-4c4a-aff4-23345809b0d3", "id": 3} v 0) v1 -- 0x7f00880b1870 con 0x7f00880b0350 2024-01-24T17:41:11.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.359+0000 7f007f400640 1 -- 172.21.15.100:0/1617628896 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "840d0d11-bf09-4c4a-aff4-23345809b0d3", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f007001ca10 con 0x7f00880b0350 2024-01-24T17:41:11.359 INFO:teuthology.orchestra.run.smithi100.stdout:3 2024-01-24T17:41:11.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 msgr2=0x7f0088153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f0088153440 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f007000c430 tx=0x7f007000c8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 shutdown_connections 2024-01-24T17:41:11.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008815e190 0x7f00880cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f00880b0350 0x7f0088153440 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 --2- 172.21.15.100:0/1617628896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008806c9f0 0x7f0088159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 >> 172.21.15.100:0/1617628896 conn(0x7f00880cb0e0 msgr2=0x7f00880cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:11.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 shutdown_connections 2024-01-24T17:41:11.361 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.360+0000 7f008e600640 1 -- 172.21.15.100:0/1617628896 wait complete. 2024-01-24T17:41:11.376 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 98c2f848-b979-4d9f-87c7-9a510fa337a2 4 2024-01-24T17:41:11.606 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.606+0000 7f1cc6e00640 1 Processor -- start 2024-01-24T17:41:11.606 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.606+0000 7f1cc6e00640 1 -- start start 2024-01-24T17:41:11.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.607+0000 7f1cc6e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1cc015cd20 0x7f1cc00cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.607+0000 7f1cc6e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1cc00d21f0 0x7f1cc00b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.607+0000 7f1cc6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc00cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.607+0000 7f1cc6e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc00b8a50 con 0x7f1cc015cd20 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.607+0000 7f1cc6e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc00cfdf0 con 0x7f1cc00d21f0 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.607+0000 7f1cc6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc015d100 con 0x7f1cc00cf420 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cc4e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc00cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cb7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1cc00d21f0 0x7f1cc00b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cc4e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc00cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51978/0 (socket says 172.21.15.100:51978) 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cc4e00640 1 -- 172.21.15.100:0/2898899118 learned_addr learned my addr 172.21.15.100:0/2898899118 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:11.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cb7e00640 1 --2- 172.21.15.100:0/2898899118 >> v2:172.21.15.100:3300/0 conn(0x7f1cc00d21f0 0x7f1cc00b5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57386/0 (socket says 172.21.15.100:57386) 2024-01-24T17:41:11.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cbfe00640 1 --2- 172.21.15.100:0/2898899118 >> v2:172.21.15.39:3300/0 conn(0x7f1cc015cd20 0x7f1cc00cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cc4e00640 1 -- 172.21.15.100:0/2898899118 >> v2:172.21.15.100:3300/0 conn(0x7f1cc00d21f0 msgr2=0x7f1cc00b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:11.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cc4e00640 1 --2- 172.21.15.100:0/2898899118 >> v2:172.21.15.100:3300/0 conn(0x7f1cc00d21f0 0x7f1cc00b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cc4e00640 1 -- 172.21.15.100:0/2898899118 >> v2:172.21.15.39:3300/0 conn(0x7f1cc015cd20 msgr2=0x7f1cc00cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:11.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cc4e00640 1 --2- 172.21.15.100:0/2898899118 >> v2:172.21.15.39:3300/0 conn(0x7f1cc015cd20 0x7f1cc00cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.608+0000 7f1cc4e00640 1 -- 172.21.15.100:0/2898899118 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc00b6080 con 0x7f1cc00cf420 2024-01-24T17:41:11.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.609+0000 7f1cc4e00640 1 --2- 172.21.15.100:0/2898899118 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc00cf8b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00b55a0 tx=0x7f1cb802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ab13d9829c29dae server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:11.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.609+0000 7f1cbf400640 1 -- 172.21.15.100:0/2898899118 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb802c9e0 con 0x7f1cc00cf420 2024-01-24T17:41:11.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cbf400640 1 -- 172.21.15.100:0/2898899118 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb802cb40 con 0x7f1cc00cf420 2024-01-24T17:41:11.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cbf400640 1 -- 172.21.15.100:0/2898899118 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80343f0 con 0x7f1cc00cf420 2024-01-24T17:41:11.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 -- 172.21.15.100:0/2898899118 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 msgr2=0x7f1cc00cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 --2- 172.21.15.100:0/2898899118 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc00cf8b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00b55a0 tx=0x7f1cb802be40 comp rx=0 tx=0).stop 2024-01-24T17:41:11.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 -- 172.21.15.100:0/2898899118 shutdown_connections 2024-01-24T17:41:11.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 --2- 172.21.15.100:0/2898899118 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc00cf8b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 --2- 172.21.15.100:0/2898899118 >> v2:172.21.15.100:3300/0 conn(0x7f1cc00d21f0 0x7f1cc00b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 --2- 172.21.15.100:0/2898899118 >> v2:172.21.15.39:3300/0 conn(0x7f1cc015cd20 0x7f1cc00cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 -- 172.21.15.100:0/2898899118 >> 172.21.15.100:0/2898899118 conn(0x7f1cc006ae50 msgr2=0x7f1cc006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:11.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 -- 172.21.15.100:0/2898899118 shutdown_connections 2024-01-24T17:41:11.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.610+0000 7f1cc6e00640 1 -- 172.21.15.100:0/2898899118 wait complete. 2024-01-24T17:41:11.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.612+0000 7f1cc6e00640 1 Processor -- start 2024-01-24T17:41:11.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.612+0000 7f1cc6e00640 1 -- start start 2024-01-24T17:41:11.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cc6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc01f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cc6e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cc00d21f0 0x7f1cc00c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cc6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc015cd20 0x7f1cc00c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cc6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc015e330 con 0x7f1cc015cd20 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cc6e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc015e610 con 0x7f1cc00d21f0 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cc6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc015e8f0 con 0x7f1cc00cf420 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cb7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cc00d21f0 0x7f1cc00c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cb7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cc00d21f0 0x7f1cc00c8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57392/0 (socket says 172.21.15.100:57392) 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cb7e00640 1 -- 172.21.15.100:0/3818334146 learned_addr learned my addr 172.21.15.100:0/3818334146 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.613+0000 7f1cbfe00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc01f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cc4e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc015cd20 0x7f1cc00c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cb7e00640 1 -- 172.21.15.100:0/3818334146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 msgr2=0x7f1cc01f1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cb7e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc01f1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cb7e00640 1 -- 172.21.15.100:0/3818334146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc015cd20 msgr2=0x7f1cc00c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cb7e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc015cd20 0x7f1cc00c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cb7e00640 1 -- 172.21.15.100:0/3818334146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8009b30 con 0x7f1cc00d21f0 2024-01-24T17:41:11.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cc4e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc015cd20 0x7f1cc00c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:11.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cb7e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cc00d21f0 0x7f1cc00c8550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800e890 tx=0x7f1ca800ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:11.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cbcc00640 1 -- 172.21.15.100:0/3818334146 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca800c940 con 0x7f1cc00d21f0 2024-01-24T17:41:11.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cbcc00640 1 -- 172.21.15.100:0/3818334146 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca800caa0 con 0x7f1cc00d21f0 2024-01-24T17:41:11.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cbcc00640 1 -- 172.21.15.100:0/3818334146 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca800ce30 con 0x7f1cc00d21f0 2024-01-24T17:41:11.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc015ebd0 con 0x7f1cc00d21f0 2024-01-24T17:41:11.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.614+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc01f2040 con 0x7f1cc00d21f0 2024-01-24T17:41:11.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.615+0000 7f1cbcc00640 1 -- 172.21.15.100:0/3818334146 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1ca8005c90 con 0x7f1cc00d21f0 2024-01-24T17:41:11.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.615+0000 7f1cbcc00640 1 -- 172.21.15.100:0/3818334146 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f1ca8013470 con 0x7f1cc00d21f0 2024-01-24T17:41:11.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.615+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc00c8fd0 con 0x7f1cc00d21f0 2024-01-24T17:41:11.618 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.618+0000 7f1cbcc00640 1 -- 172.21.15.100:0/3818334146 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f1ca8021060 con 0x7f1cc00d21f0 2024-01-24T17:41:11.676 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:11.676+0000 7fd73f014040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T17:41:11.696 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.696+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "98c2f848-b979-4d9f-87c7-9a510fa337a2", "id": 4} v 0) v1 -- 0x7f1cc0067040 con 0x7f1cc00d21f0 2024-01-24T17:41:11.704 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.704+0000 7f1cbcc00640 1 -- 172.21.15.100:0/3818334146 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98c2f848-b979-4d9f-87c7-9a510fa337a2", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1ca8029020 con 0x7f1cc00d21f0 2024-01-24T17:41:11.704 INFO:teuthology.orchestra.run.smithi100.stdout:4 2024-01-24T17:41:11.705 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cc00d21f0 msgr2=0x7f1cc00c8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cc00d21f0 0x7f1cc00c8550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800e890 tx=0x7f1ca800ed50 comp rx=0 tx=0).stop 2024-01-24T17:41:11.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 shutdown_connections 2024-01-24T17:41:11.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cc015cd20 0x7f1cc00c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cc00d21f0 0x7f1cc00c8550 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 --2- 172.21.15.100:0/3818334146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1cc00cf420 0x7f1cc01f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 >> 172.21.15.100:0/3818334146 conn(0x7f1cc006ae50 msgr2=0x7f1cc00b6260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:11.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 shutdown_connections 2024-01-24T17:41:11.706 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.705+0000 7f1cc6e00640 1 -- 172.21.15.100:0/3818334146 wait complete. 2024-01-24T17:41:11.720 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:11.719+0000 7fad26433040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T17:41:11.722 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new f131d23e-1a25-48e6-8fc9-071732abd22b 5 2024-01-24T17:41:11.845 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:11.845+0000 7fd73f014040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T17:41:11.888 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:11.888+0000 7fad26433040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T17:41:11.896 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.896+0000 7fa2eee00640 1 Processor -- start 2024-01-24T17:41:11.896 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.896+0000 7fa2eee00640 1 -- start start 2024-01-24T17:41:11.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.897+0000 7fa2eee00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e80b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2eee00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e80b7e30 con 0x7fa2e80b8250 2024-01-24T17:41:11.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2eee00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e80b6d90 con 0x7fa2e80cdd60 2024-01-24T17:41:11.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2eee00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80b7070 con 0x7fa2e815c9e0 2024-01-24T17:41:11.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2eda00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e80b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2ee400640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fa2e80cdd60 0x7fa2e80d09c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:50098/0 (socket says 172.21.15.100:50098) 2024-01-24T17:41:11.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2ee400640 1 -- 172.21.15.100:0/3151386183 learned_addr learned my addr 172.21.15.100:0/3151386183 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-01-24T17:41:11.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2e7e00640 1 -- 172.21.15.100:0/3151386183 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609011093 0 0) 0x7fa2e80b6d90 con 0x7fa2e80cdd60 2024-01-24T17:41:11.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2eda00640 1 -- 172.21.15.100:0/3151386183 >> v1:172.21.15.100:6789/0 conn(0x7fa2e80cdd60 legacy=0x7fa2e80d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2eda00640 1 -- 172.21.15.100:0/3151386183 >> v1:172.21.15.39:6789/0 conn(0x7fa2e80b8250 legacy=0x7fa2e80cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.898+0000 7fa2eda00640 1 -- 172.21.15.100:0/3151386183 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80b7350 con 0x7fa2e815c9e0 2024-01-24T17:41:11.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.899+0000 7fa2eda00640 1 --2- 172.21.15.100:0/3151386183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e80b6500 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80097d0 tx=0x7fa2d802b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=906af472fa03e1ec server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:11.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.900+0000 7fa2e7e00640 1 -- 172.21.15.100:0/3151386183 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8030890 con 0x7fa2e815c9e0 2024-01-24T17:41:11.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.900+0000 7fa2e7e00640 1 -- 172.21.15.100:0/3151386183 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d80309f0 con 0x7fa2e815c9e0 2024-01-24T17:41:11.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.900+0000 7fa2e7e00640 1 -- 172.21.15.100:0/3151386183 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8030d80 con 0x7fa2e815c9e0 2024-01-24T17:41:11.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.900+0000 7fa2eee00640 1 -- 172.21.15.100:0/3151386183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 msgr2=0x7fa2e80b6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.900+0000 7fa2eee00640 1 --2- 172.21.15.100:0/3151386183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e80b6500 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80097d0 tx=0x7fa2d802b9d0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.900+0000 7fa2eee00640 1 -- 172.21.15.100:0/3151386183 shutdown_connections 2024-01-24T17:41:11.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.901+0000 7fa2eee00640 1 --2- 172.21.15.100:0/3151386183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e80b6500 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.901+0000 7fa2eee00640 1 -- 172.21.15.100:0/3151386183 >> 172.21.15.100:0/3151386183 conn(0x7fa2e8071790 msgr2=0x7fa2e8071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:11.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.901+0000 7fa2eee00640 1 -- 172.21.15.100:0/3151386183 shutdown_connections 2024-01-24T17:41:11.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.901+0000 7fa2eee00640 1 -- 172.21.15.100:0/3151386183 wait complete. 2024-01-24T17:41:11.902 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.902+0000 7fa2eee00640 1 Processor -- start 2024-01-24T17:41:11.903 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.902+0000 7fa2eee00640 1 -- start start 2024-01-24T17:41:11.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eee00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2e80b8250 0x7fa2e80a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2e80cdd60 0x7fa2e8157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eee00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e815fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:11.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80a9950 con 0x7fa2e80cdd60 2024-01-24T17:41:11.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eee00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8158280 con 0x7fa2e80b8250 2024-01-24T17:41:11.905 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eee00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e815ff60 con 0x7fa2e815c9e0 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2ee400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e815fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eda00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2e80b8250 0x7fa2e80a92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eda00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2e80b8250 0x7fa2e80a92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57394/0 (socket says 172.21.15.100:57394) 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2eda00640 1 -- 172.21.15.100:0/4079950446 learned_addr learned my addr 172.21.15.100:0/4079950446 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.903+0000 7fa2ee400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e815fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:51994/0 (socket says 172.21.15.100:51994) 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2ed000640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2e80cdd60 0x7fa2e8157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2ee400640 1 -- 172.21.15.100:0/4079950446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2e80b8250 msgr2=0x7fa2e80a92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2ee400640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2e80b8250 0x7fa2e80a92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2ee400640 1 -- 172.21.15.100:0/4079950446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2e80cdd60 msgr2=0x7fa2e8157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2ee400640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2e80cdd60 0x7fa2e8157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2ee400640 1 -- 172.21.15.100:0/4079950446 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d8009400 con 0x7fa2e815c9e0 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2ed000640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2e80cdd60 0x7fa2e8157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:11.906 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2eda00640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2e80b8250 0x7fa2e80a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.904+0000 7fa2ee400640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e815fa20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c780 tx=0x7fa2e000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.905+0000 7fa2e5600640 1 -- 172.21.15.100:0/4079950446 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000ce30 con 0x7fa2e815c9e0 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.905+0000 7fa2e5600640 1 -- 172.21.15.100:0/4079950446 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e000a640 con 0x7fa2e815c9e0 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.905+0000 7fa2e5600640 1 -- 172.21.15.100:0/4079950446 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000a9f0 con 0x7fa2e815c9e0 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.905+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e80a9da0 con 0x7fa2e815c9e0 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.905+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e815db90 con 0x7fa2e815c9e0 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.905+0000 7fa2e5600640 1 -- 172.21.15.100:0/4079950446 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa2e00070d0 con 0x7fa2e815c9e0 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.905+0000 7fa2e5600640 1 -- 172.21.15.100:0/4079950446 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fa2e0007900 con 0x7fa2e815c9e0 2024-01-24T17:41:11.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.906+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e815dda0 con 0x7fa2e815c9e0 2024-01-24T17:41:11.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.908+0000 7fa2e5600640 1 -- 172.21.15.100:0/4079950446 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fa2e0018070 con 0x7fa2e815c9e0 2024-01-24T17:41:11.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.986+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f131d23e-1a25-48e6-8fc9-071732abd22b", "id": 5} v 0) v1 -- 0x7fa2e8067040 con 0x7fa2e815c9e0 2024-01-24T17:41:11.990 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:11.990+0000 7fd73f014040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T17:41:11.994 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.994+0000 7fa2e5600640 1 -- 172.21.15.100:0/4079950446 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f131d23e-1a25-48e6-8fc9-071732abd22b", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa2e0007450 con 0x7fa2e815c9e0 2024-01-24T17:41:11.994 INFO:teuthology.orchestra.run.smithi100.stdout:5 2024-01-24T17:41:11.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 msgr2=0x7fa2e815fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:11.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e815fa20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c780 tx=0x7fa2e000cc40 comp rx=0 tx=0).stop 2024-01-24T17:41:11.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 shutdown_connections 2024-01-24T17:41:11.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2e815c9e0 0x7fa2e815fa20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2e80cdd60 0x7fa2e8157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 --2- 172.21.15.100:0/4079950446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa2e80b8250 0x7fa2e80a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:11.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 >> 172.21.15.100:0/4079950446 conn(0x7fa2e8071790 msgr2=0x7fa2e80d23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:11.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 shutdown_connections 2024-01-24T17:41:11.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:11.995+0000 7fa2eee00640 1 -- 172.21.15.100:0/4079950446 wait complete. 2024-01-24T17:41:12.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 3b0f4907-1db1-4d56-92b9-9dedceff5179 6 2024-01-24T17:41:12.020 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:12.019+0000 7fad26433040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T17:41:12.046 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:12.046+0000 7fd73f014040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T17:41:12.075 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:12.074+0000 7fad26433040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T17:41:12.163 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:12.163+0000 7fd73f014040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T17:41:12.189 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:12.188+0000 7fad26433040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T17:41:12.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.208+0000 7f3ebde00640 1 Processor -- start 2024-01-24T17:41:12.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.208+0000 7f3ebde00640 1 -- start start 2024-01-24T17:41:12.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.209+0000 7f3ebde00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb80cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.209+0000 7f3ebde00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3eb80cdb90 0x7f3eb80d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.209+0000 7f3ebde00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb80b7e30 con 0x7f3eb815c9e0 2024-01-24T17:41:12.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.209+0000 7f3ebde00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb80ce020 con 0x7f3eb80cdb90 2024-01-24T17:41:12.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.209+0000 7f3ebde00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80b6d90 con 0x7f3eb80b8250 2024-01-24T17:41:12.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.210+0000 7f3ebca00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f3eb815c9e0 0x7f3eb80b6500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:50106/0 (socket says 172.21.15.100:50106) 2024-01-24T17:41:12.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.210+0000 7f3ebca00640 1 -- 172.21.15.100:0/743478382 learned_addr learned my addr 172.21.15.100:0/743478382 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-01-24T17:41:12.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.210+0000 7f3eafe00640 1 --2- 172.21.15.100:0/743478382 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb80cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.210+0000 7f3ebd400640 1 --2- 172.21.15.100:0/743478382 >> v2:172.21.15.39:3300/0 conn(0x7f3eb80cdb90 0x7f3eb80d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.210+0000 7f3eafe00640 1 -- 172.21.15.100:0/743478382 >> v2:172.21.15.39:3300/0 conn(0x7f3eb80cdb90 msgr2=0x7f3eb80d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:12.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.210+0000 7f3eafe00640 1 --2- 172.21.15.100:0/743478382 >> v2:172.21.15.39:3300/0 conn(0x7f3eb80cdb90 0x7f3eb80d09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.210+0000 7f3eafe00640 1 -- 172.21.15.100:0/743478382 >> v1:172.21.15.100:6789/0 conn(0x7f3eb815c9e0 legacy=0x7f3eb80b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.210+0000 7f3eafe00640 1 -- 172.21.15.100:0/743478382 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80b7070 con 0x7f3eb80b8250 2024-01-24T17:41:12.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.212+0000 7f3eafe00640 1 --2- 172.21.15.100:0/743478382 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb80cd650 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00097a0 tx=0x7f3ea002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af0d53ca6bc4855 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:12.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.212+0000 7f3eb7e00640 1 -- 172.21.15.100:0/743478382 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0030800 con 0x7f3eb80b8250 2024-01-24T17:41:12.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.212+0000 7f3eb7e00640 1 -- 172.21.15.100:0/743478382 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea0030960 con 0x7f3eb80b8250 2024-01-24T17:41:12.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.212+0000 7f3eb7e00640 1 -- 172.21.15.100:0/743478382 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0030cf0 con 0x7f3eb80b8250 2024-01-24T17:41:12.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.213+0000 7f3ebde00640 1 -- 172.21.15.100:0/743478382 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 msgr2=0x7f3eb80cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.213+0000 7f3ebde00640 1 --2- 172.21.15.100:0/743478382 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb80cd650 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00097a0 tx=0x7f3ea002b940 comp rx=0 tx=0).stop 2024-01-24T17:41:12.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.213+0000 7f3ebde00640 1 -- 172.21.15.100:0/743478382 shutdown_connections 2024-01-24T17:41:12.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.213+0000 7f3ebde00640 1 --2- 172.21.15.100:0/743478382 >> v2:172.21.15.39:3300/0 conn(0x7f3eb80cdb90 0x7f3eb80d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.213+0000 7f3ebde00640 1 --2- 172.21.15.100:0/743478382 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb80cd650 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.213+0000 7f3ebde00640 1 -- 172.21.15.100:0/743478382 >> 172.21.15.100:0/743478382 conn(0x7f3eb8071790 msgr2=0x7f3eb8071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:12.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.213+0000 7f3ebde00640 1 -- 172.21.15.100:0/743478382 shutdown_connections 2024-01-24T17:41:12.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.213+0000 7f3ebde00640 1 -- 172.21.15.100:0/743478382 wait complete. 2024-01-24T17:41:12.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.215+0000 7f3ebde00640 1 Processor -- start 2024-01-24T17:41:12.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.215+0000 7f3ebde00640 1 -- start start 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebde00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb81f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebde00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3eb80cdb90 0x7f3eb8147de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3eb815c9e0 0x7f3eb8152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebde00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8148320 con 0x7f3eb815c9e0 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebde00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb81f6780 con 0x7f3eb80cdb90 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebde00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8152b60 con 0x7f3eb80b8250 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb81f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3eafe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3eb80cdb90 0x7f3eb8147de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb81f58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.100:52022/0 (socket says 172.21.15.100:52022) 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebca00640 1 -- 172.21.15.100:0/1761148712 learned_addr learned my addr 172.21.15.100:0/1761148712 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.216+0000 7f3ebd400640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3eb815c9e0 0x7f3eb8152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.218 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebca00640 1 -- 172.21.15.100:0/1761148712 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3eb80cdb90 msgr2=0x7f3eb8147de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebca00640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3eb80cdb90 0x7f3eb8147de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebca00640 1 -- 172.21.15.100:0/1761148712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3eb815c9e0 msgr2=0x7f3eb8152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebca00640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3eb815c9e0 0x7f3eb8152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebca00640 1 -- 172.21.15.100:0/1761148712 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea0009400 con 0x7f3eb80b8250 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebd400640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3eb815c9e0 0x7f3eb8152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebca00640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb81f58e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800c480 tx=0x7f3ea800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3eb5600640 1 -- 172.21.15.100:0/1761148712 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8005ce0 con 0x7f3eb80b8250 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3eb5600640 1 -- 172.21.15.100:0/1761148712 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea8005e40 con 0x7f3eb80b8250 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3eb5600640 1 -- 172.21.15.100:0/1761148712 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80072f0 con 0x7f3eb80b8250 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb81f5ee0 con 0x7f3eb80b8250 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.217+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb81f64a0 con 0x7f3eb80b8250 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.218+0000 7f3eb5600640 1 -- 172.21.15.100:0/1761148712 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3ea801c440 con 0x7f3eb80b8250 2024-01-24T17:41:12.219 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.218+0000 7f3eb5600640 1 -- 172.21.15.100:0/1761148712 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f3ea801cce0 con 0x7f3eb80b8250 2024-01-24T17:41:12.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.219+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb8067040 con 0x7f3eb80b8250 2024-01-24T17:41:12.221 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.221+0000 7f3eb5600640 1 -- 172.21.15.100:0/1761148712 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f3ea8021060 con 0x7f3eb80b8250 2024-01-24T17:41:12.280 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:12.280+0000 7fd73f014040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T17:41:12.299 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.299+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3b0f4907-1db1-4d56-92b9-9dedceff5179", "id": 6} v 0) v1 -- 0x7f3eb80b7320 con 0x7f3eb80b8250 2024-01-24T17:41:12.305 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:12.304+0000 7fad26433040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T17:41:12.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.307+0000 7f3eb5600640 1 -- 172.21.15.100:0/1761148712 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b0f4907-1db1-4d56-92b9-9dedceff5179", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3ea8017070 con 0x7f3eb80b8250 2024-01-24T17:41:12.307 INFO:teuthology.orchestra.run.smithi100.stdout:6 2024-01-24T17:41:12.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 msgr2=0x7f3eb81f58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb81f58e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800c480 tx=0x7f3ea800c940 comp rx=0 tx=0).stop 2024-01-24T17:41:12.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 shutdown_connections 2024-01-24T17:41:12.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3eb815c9e0 0x7f3eb8152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3eb80cdb90 0x7f3eb8147de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 --2- 172.21.15.100:0/1761148712 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3eb80b8250 0x7f3eb81f58e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 >> 172.21.15.100:0/1761148712 conn(0x7f3eb8071790 msgr2=0x7f3eb80d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:12.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 shutdown_connections 2024-01-24T17:41:12.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.308+0000 7f3ebde00640 1 -- 172.21.15.100:0/1761148712 wait complete. 2024-01-24T17:41:12.327 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563 7 2024-01-24T17:41:12.447 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:12.446+0000 7fd73f014040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T17:41:12.471 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:12.470+0000 7fad26433040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T17:41:12.498 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:12.498+0000 7fd73f014040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T17:41:12.523 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:12.522+0000 7fad26433040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T17:41:12.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.561+0000 7f28a9200640 1 Processor -- start 2024-01-24T17:41:12.562 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.562+0000 7f28a9200640 1 -- start start 2024-01-24T17:41:12.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.563+0000 7f28a9200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a40cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a9200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f28a40b2230 0x7f28a40b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a9200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a40cebf0 con 0x7f28a40ce760 2024-01-24T17:41:12.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a9200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a40b6770 con 0x7f28a40b2230 2024-01-24T17:41:12.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a9200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a40b2c00 con 0x7f28a415ab30 2024-01-24T17:41:12.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f289be00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f28a40ce760 0x7f28a40b6060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:50108/0 (socket says 172.21.15.100:50108) 2024-01-24T17:41:12.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a40cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.564 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f289be00640 1 -- 172.21.15.100:0/4245267269 learned_addr learned my addr 172.21.15.100:0/4245267269 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-01-24T17:41:12.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a2800640 1 --2- 172.21.15.100:0/4245267269 >> v2:172.21.15.39:3300/0 conn(0x7f28a40b2230 0x7f28a40b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a1400640 1 -- 172.21.15.100:0/4245267269 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588661275 0 0) 0x7f28a40cebf0 con 0x7f28a40ce760 2024-01-24T17:41:12.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a1e00640 1 -- 172.21.15.100:0/4245267269 >> v2:172.21.15.39:3300/0 conn(0x7f28a40b2230 msgr2=0x7f28a40b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:12.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a1e00640 1 --2- 172.21.15.100:0/4245267269 >> v2:172.21.15.39:3300/0 conn(0x7f28a40b2230 0x7f28a40b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a1e00640 1 -- 172.21.15.100:0/4245267269 >> v1:172.21.15.100:6789/0 conn(0x7f28a40ce760 legacy=0x7f28a40b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.564+0000 7f28a1e00640 1 -- 172.21.15.100:0/4245267269 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a40ceed0 con 0x7f28a415ab30 2024-01-24T17:41:12.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.566+0000 7f28a1e00640 1 --2- 172.21.15.100:0/4245267269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a40cfc90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f28940097a0 tx=0x7f289402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1380470647aa396 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:12.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.566+0000 7f28a1400640 1 -- 172.21.15.100:0/4245267269 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2894031830 con 0x7f28a415ab30 2024-01-24T17:41:12.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.566+0000 7f28a1400640 1 -- 172.21.15.100:0/4245267269 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2894031990 con 0x7f28a415ab30 2024-01-24T17:41:12.566 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.566+0000 7f28a1400640 1 -- 172.21.15.100:0/4245267269 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2894031d20 con 0x7f28a415ab30 2024-01-24T17:41:12.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.567+0000 7f28a9200640 1 -- 172.21.15.100:0/4245267269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 msgr2=0x7f28a40cfc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.567+0000 7f28a9200640 1 --2- 172.21.15.100:0/4245267269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a40cfc90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f28940097a0 tx=0x7f289402b8d0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.567+0000 7f28a9200640 1 -- 172.21.15.100:0/4245267269 shutdown_connections 2024-01-24T17:41:12.567 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.567+0000 7f28a9200640 1 --2- 172.21.15.100:0/4245267269 >> v2:172.21.15.39:3300/0 conn(0x7f28a40b2230 0x7f28a40b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.567+0000 7f28a9200640 1 --2- 172.21.15.100:0/4245267269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a40cfc90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.567+0000 7f28a9200640 1 -- 172.21.15.100:0/4245267269 >> 172.21.15.100:0/4245267269 conn(0x7f28a40cd230 msgr2=0x7f28a40cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:12.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.567+0000 7f28a9200640 1 -- 172.21.15.100:0/4245267269 shutdown_connections 2024-01-24T17:41:12.568 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.567+0000 7f28a9200640 1 -- 172.21.15.100:0/4245267269 wait complete. 2024-01-24T17:41:12.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.568+0000 7f28a9200640 1 Processor -- start 2024-01-24T17:41:12.569 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.569+0000 7f28a9200640 1 -- start start 2024-01-24T17:41:12.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.569+0000 7f28a9200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28a40b2230 0x7f28a415d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a9200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a414fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a9200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a41f36a0 0x7f28a41f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.571 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a9200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a41f4090 con 0x7f28a40b2230 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a9200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a415d200 con 0x7f28a41f36a0 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a9200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4150280 con 0x7f28a415ab30 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a2800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a41f36a0 0x7f28a41f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a2800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a41f36a0 0x7f28a41f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57414/0 (socket says 172.21.15.100:57414) 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a2800640 1 -- 172.21.15.100:0/1826145596 learned_addr learned my addr 172.21.15.100:0/1826145596 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f289be00640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a414fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a1e00640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28a40b2230 0x7f28a415d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a2800640 1 -- 172.21.15.100:0/1826145596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 msgr2=0x7f28a414fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a2800640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a414fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a2800640 1 -- 172.21.15.100:0/1826145596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28a40b2230 msgr2=0x7f28a415d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a2800640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28a40b2230 0x7f28a415d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.570+0000 7f28a2800640 1 -- 172.21.15.100:0/1826145596 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2894009400 con 0x7f28a41f36a0 2024-01-24T17:41:12.572 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.571+0000 7f28a2800640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a41f36a0 0x7f28a41f3b50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f289c007ea0 tx=0x7f289c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:12.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.571+0000 7f289a000640 1 -- 172.21.15.100:0/1826145596 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c014620 con 0x7f28a41f36a0 2024-01-24T17:41:12.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.571+0000 7f289a000640 1 -- 172.21.15.100:0/1826145596 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289c0147a0 con 0x7f28a41f36a0 2024-01-24T17:41:12.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.571+0000 7f289a000640 1 -- 172.21.15.100:0/1826145596 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c014b50 con 0x7f28a41f36a0 2024-01-24T17:41:12.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.571+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a40c7ed0 con 0x7f28a41f36a0 2024-01-24T17:41:12.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.571+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a415cbb0 con 0x7f28a41f36a0 2024-01-24T17:41:12.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.571+0000 7f289a000640 1 -- 172.21.15.100:0/1826145596 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f289c01e460 con 0x7f28a41f36a0 2024-01-24T17:41:12.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.571+0000 7f289a000640 1 -- 172.21.15.100:0/1826145596 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f289c01ed70 con 0x7f28a41f36a0 2024-01-24T17:41:12.573 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.572+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a415d3b0 con 0x7f28a41f36a0 2024-01-24T17:41:12.574 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.574+0000 7f289a000640 1 -- 172.21.15.100:0/1826145596 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f28a415d3b0 con 0x7f28a41f36a0 2024-01-24T17:41:12.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.651+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563", "id": 7} v 0) v1 -- 0x7f28a4067040 con 0x7f28a41f36a0 2024-01-24T17:41:12.661 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.661+0000 7f289a000640 1 -- 172.21.15.100:0/1826145596 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f289c02c0b0 con 0x7f28a41f36a0 2024-01-24T17:41:12.661 INFO:teuthology.orchestra.run.smithi100.stdout:7 2024-01-24T17:41:12.662 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a41f36a0 msgr2=0x7f28a41f3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a41f36a0 0x7f28a41f3b50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f289c007ea0 tx=0x7f289c007f90 comp rx=0 tx=0).stop 2024-01-24T17:41:12.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 shutdown_connections 2024-01-24T17:41:12.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f28a41f36a0 0x7f28a41f3b50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28a415ab30 0x7f28a414fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 --2- 172.21.15.100:0/1826145596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28a40b2230 0x7f28a415d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 >> 172.21.15.100:0/1826145596 conn(0x7f28a40cd230 msgr2=0x7f28a40b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:12.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 shutdown_connections 2024-01-24T17:41:12.663 INFO:teuthology.orchestra.run.smithi100.stderr:2024-01-24T17:41:12.662+0000 7f28a9200640 1 -- 172.21.15.100:0/1826145596 wait complete. 2024-01-24T17:41:12.664 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:12.664+0000 7fd73f014040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T17:41:12.678 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-24T17:41:12.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-24T17:41:12.685 INFO:tasks.ceph.osd.0:Started 2024-01-24T17:41:12.686 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-24T17:41:12.686 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-24T17:41:12.693 INFO:tasks.ceph.osd.1:Started 2024-01-24T17:41:12.693 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-24T17:41:12.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-24T17:41:12.694 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:12.687+0000 7fad26433040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T17:41:12.701 INFO:tasks.ceph.osd.2:Started 2024-01-24T17:41:12.701 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-24T17:41:12.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-24T17:41:12.705 INFO:tasks.ceph.osd.3:Started 2024-01-24T17:41:12.705 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-24T17:41:12.705 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-24T17:41:12.713 INFO:tasks.ceph.osd.4:Started 2024-01-24T17:41:12.713 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-24T17:41:12.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-24T17:41:12.720 INFO:tasks.ceph.osd.5:Started 2024-01-24T17:41:12.720 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-24T17:41:12.720 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-24T17:41:12.727 INFO:tasks.ceph.osd.6:Started 2024-01-24T17:41:12.727 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-24T17:41:12.727 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-24T17:41:12.735 INFO:tasks.ceph.osd.7:Started 2024-01-24T17:41:12.735 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.957+0000 7f9e64800640 1 Processor -- start 2024-01-24T17:41:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.957+0000 7f9e64800640 1 -- start start 2024-01-24T17:41:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.958+0000 7f9e64800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9e5c15ca40 0x7f9e5c0b8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.958+0000 7f9e64800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9e5c0b4480 0x7f9e5c0b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.958+0000 7f9e64800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c0d0b20 0x7f9e5c0d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.958+0000 7f9e64800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c0b4e30 con 0x7f9e5c0b4480 2024-01-24T17:41:12.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.958+0000 7f9e64800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c0b7ce0 con 0x7f9e5c15ca40 2024-01-24T17:41:12.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.958+0000 7f9e64800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0b70d0 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9e5c0b4480 0x7f9e5c0b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9e5c0b4480 0x7f9e5c0b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33030/0 (socket says 172.21.15.39:33030) 2024-01-24T17:41:12.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61400640 1 -- 172.21.15.39:0/1060679029 learned_addr learned my addr 172.21.15.39:0/1060679029 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:12.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e62800640 1 --2- 172.21.15.39:0/1060679029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c0d0b20 0x7f9e5c0d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61e00640 1 --2- 172.21.15.39:0/1060679029 >> v2:172.21.15.100:3300/0 conn(0x7f9e5c15ca40 0x7f9e5c0b8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61400640 1 -- 172.21.15.39:0/1060679029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c0d0b20 msgr2=0x7f9e5c0d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61400640 1 --2- 172.21.15.39:0/1060679029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c0d0b20 0x7f9e5c0d0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61400640 1 -- 172.21.15.39:0/1060679029 >> v2:172.21.15.100:3300/0 conn(0x7f9e5c15ca40 msgr2=0x7f9e5c0b8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61400640 1 --2- 172.21.15.39:0/1060679029 >> v2:172.21.15.100:3300/0 conn(0x7f9e5c15ca40 0x7f9e5c0b8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.959+0000 7f9e61400640 1 -- 172.21.15.39:0/1060679029 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0b73b0 con 0x7f9e5c0b4480 2024-01-24T17:41:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.960+0000 7f9e61400640 1 --2- 172.21.15.39:0/1060679029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0b4480 0x7f9e5c0b48f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0d0760 tx=0x7f9e4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99894ed42509eff6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.960+0000 7f9e60a00640 1 -- 172.21.15.39:0/1060679029 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44031870 con 0x7f9e5c0b4480 2024-01-24T17:41:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.960+0000 7f9e60a00640 1 -- 172.21.15.39:0/1060679029 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e440319d0 con 0x7f9e5c0b4480 2024-01-24T17:41:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.960+0000 7f9e60a00640 1 -- 172.21.15.39:0/1060679029 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44031d60 con 0x7f9e5c0b4480 2024-01-24T17:41:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 -- 172.21.15.39:0/1060679029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0b4480 msgr2=0x7f9e5c0b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 --2- 172.21.15.39:0/1060679029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0b4480 0x7f9e5c0b48f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0d0760 tx=0x7f9e4402b910 comp rx=0 tx=0).stop 2024-01-24T17:41:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 -- 172.21.15.39:0/1060679029 shutdown_connections 2024-01-24T17:41:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 --2- 172.21.15.39:0/1060679029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c0d0b20 0x7f9e5c0d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 --2- 172.21.15.39:0/1060679029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0b4480 0x7f9e5c0b48f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 --2- 172.21.15.39:0/1060679029 >> v2:172.21.15.100:3300/0 conn(0x7f9e5c15ca40 0x7f9e5c0b8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 -- 172.21.15.39:0/1060679029 >> 172.21.15.39:0/1060679029 conn(0x7f9e5c071790 msgr2=0x7f9e5c071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 -- 172.21.15.39:0/1060679029 shutdown_connections 2024-01-24T17:41:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.961+0000 7f9e64800640 1 -- 172.21.15.39:0/1060679029 wait complete. 2024-01-24T17:41:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.964+0000 7f9e64800640 1 Processor -- start 2024-01-24T17:41:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.964+0000 7f9e64800640 1 -- start start 2024-01-24T17:41:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.965+0000 7f9e64800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e5c0b4480 0x7f9e5c0cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.965+0000 7f9e64800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0d0b20 0x7f9e5c15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e64800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c15ca40 0x7f9e5c15de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e64800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15d3e0 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e64800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15ce20 con 0x7f9e5c0b4480 2024-01-24T17:41:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e64800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c15d100 con 0x7f9e5c15ca40 2024-01-24T17:41:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0d0b20 0x7f9e5c15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0d0b20 0x7f9e5c15f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33044/0 (socket says 172.21.15.39:33044) 2024-01-24T17:41:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61400640 1 -- 172.21.15.39:0/1497772505 learned_addr learned my addr 172.21.15.39:0/1497772505 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e62800640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c15ca40 0x7f9e5c15de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61e00640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e5c0b4480 0x7f9e5c0cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61400640 1 -- 172.21.15.39:0/1497772505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c15ca40 msgr2=0x7f9e5c15de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61400640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c15ca40 0x7f9e5c15de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61400640 1 -- 172.21.15.39:0/1497772505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e5c0b4480 msgr2=0x7f9e5c0cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61400640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e5c0b4480 0x7f9e5c0cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.966+0000 7f9e61400640 1 -- 172.21.15.39:0/1497772505 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e440094d0 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.967+0000 7f9e61400640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0d0b20 0x7f9e5c15f8d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9e4402b950 tx=0x7f9e440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.967+0000 7f9e5a000640 1 -- 172.21.15.39:0/1497772505 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4403ad30 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.967+0000 7f9e5a000640 1 -- 172.21.15.39:0/1497772505 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e440443f0 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.967+0000 7f9e5a000640 1 -- 172.21.15.39:0/1497772505 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e440447a0 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.967+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c0c6670 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.967+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c0c6c30 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.968+0000 7f9e5a000640 1 -- 172.21.15.39:0/1497772505 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9e44044c00 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.968+0000 7f9e5a000640 1 -- 172.21.15.39:0/1497772505 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9e4402d040 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.968+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c0c64d0 con 0x7f9e5c0d0b20 2024-01-24T17:41:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:12.971+0000 7f9e5a000640 1 -- 172.21.15.39:0/1497772505 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f9e5c0c64d0 con 0x7f9e5c0d0b20 2024-01-24T17:41:13.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.036+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e5c0c6e20 con 0x7f9e5c0d0b20 2024-01-24T17:41:13.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.036+0000 7f9e5a000640 1 -- 172.21.15.39:0/1497772505 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9e44035020 con 0x7f9e5c0d0b20 2024-01-24T17:41:13.037 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:13.037 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":10,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:12.654149+0000","last_up_change":"0.000000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0d0b20 msgr2=0x7f9e5c15f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0d0b20 0x7f9e5c15f8d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9e4402b950 tx=0x7f9e440046c0 comp rx=0 tx=0).stop 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 shutdown_connections 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e5c15ca40 0x7f9e5c15de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e5c0d0b20 0x7f9e5c15f8d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 --2- 172.21.15.39:0/1497772505 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e5c0b4480 0x7f9e5c0cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 >> 172.21.15.39:0/1497772505 conn(0x7f9e5c071790 msgr2=0x7f9e5c0b8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 shutdown_connections 2024-01-24T17:41:13.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:13.038+0000 7f9e64800640 1 -- 172.21.15.39:0/1497772505 wait complete. 2024-01-24T17:41:13.056 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-24T17:41:13.056 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-24T17:41:13.391 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-01-24T17:41:13.391+0000 7fe0ad60e500 -1 Falling back to public interface 2024-01-24T17:41:13.397 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-01-24T17:41:13.397+0000 7f3e8a8b4500 -1 Falling back to public interface 2024-01-24T17:41:13.412 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-01-24T17:41:13.412+0000 7ff908813500 -1 Falling back to public interface 2024-01-24T17:41:13.416 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-01-24T17:41:13.415+0000 7f23b5517500 -1 Falling back to public interface 2024-01-24T17:41:13.426 INFO:tasks.ceph.osd.3.smithi039.stderr:2024-01-24T17:41:13.426+0000 7f38cde10500 -1 Falling back to public interface 2024-01-24T17:41:13.430 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-01-24T17:41:13.430+0000 7f7039ffb500 -1 Falling back to public interface 2024-01-24T17:41:13.438 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-01-24T17:41:13.438+0000 7fc0a9a1b500 -1 Falling back to public interface 2024-01-24T17:41:13.442 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-01-24T17:41:13.442+0000 7f34c1415500 -1 Falling back to public interface 2024-01-24T17:41:13.703 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:13.703+0000 7fd73f014040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T17:41:13.821 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:13.821+0000 7fad26433040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T17:41:13.862 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:14.008 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:14.007+0000 7fd73f014040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T17:41:14.033 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.032+0000 7f43e6e00640 1 Processor -- start 2024-01-24T17:41:14.033 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.032+0000 7f43e6e00640 1 -- start start 2024-01-24T17:41:14.038 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.033+0000 7f43e6e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f43e015cc50 0x7f43e01606c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:14.038 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.033+0000 7f43e6e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f43e00b84d0 0x7f43e00cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:14.038 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.033+0000 7f43e6e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e015d030 con 0x7f43e00cf970 2024-01-24T17:41:14.038 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.033+0000 7f43e6e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e00d1000 con 0x7f43e00b84d0 2024-01-24T17:41:14.038 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.033+0000 7f43e6e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e00cfda0 con 0x7f43e015cc50 2024-01-24T17:41:14.039 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.033+0000 7f43df400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f43e00b84d0 0x7f43e00cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:14.039 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.034+0000 7f43df400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f43e00b84d0 0x7f43e00cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51286/0 (socket says 172.21.15.39:51286) 2024-01-24T17:41:14.039 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.034+0000 7f43df400640 1 -- 172.21.15.39:0/449498159 learned_addr learned my addr 172.21.15.39:0/449498159 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:14.039 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.034+0000 7f43dfe00640 1 --2- 172.21.15.39:0/449498159 >> v2:172.21.15.100:3300/0 conn(0x7f43e015cc50 0x7f43e01606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:14.039 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.034+0000 7f43df400640 1 -- 172.21.15.39:0/449498159 >> v2:172.21.15.100:3300/0 conn(0x7f43e015cc50 msgr2=0x7f43e01606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:14.039 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.034+0000 7f43df400640 1 --2- 172.21.15.39:0/449498159 >> v2:172.21.15.100:3300/0 conn(0x7f43e015cc50 0x7f43e01606c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:14.039 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.034+0000 7f43df400640 1 -- 172.21.15.39:0/449498159 >> v1:172.21.15.100:6789/0 conn(0x7f43e00cf970 legacy=0x7f43e00b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.034+0000 7f43df400640 1 -- 172.21.15.39:0/449498159 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e00b5a20 con 0x7f43e00b84d0 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.035+0000 7f43df400640 1 --2- 172.21.15.39:0/449498159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00b84d0 0x7f43e00cf430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43e006c2b0 tx=0x7f43c802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb815e6fe6c3e05 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.035+0000 7f43dea00640 1 -- 172.21.15.39:0/449498159 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c802f850 con 0x7f43e00b84d0 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.035+0000 7f43dea00640 1 -- 172.21.15.39:0/449498159 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c802f9b0 con 0x7f43e00b84d0 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.035+0000 7f43dea00640 1 -- 172.21.15.39:0/449498159 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c802fd40 con 0x7f43e00b84d0 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.036+0000 7f43e6e00640 1 -- 172.21.15.39:0/449498159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00b84d0 msgr2=0x7f43e00cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.036+0000 7f43e6e00640 1 --2- 172.21.15.39:0/449498159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00b84d0 0x7f43e00cf430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43e006c2b0 tx=0x7f43c802ba30 comp rx=0 tx=0).stop 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.036+0000 7f43e6e00640 1 -- 172.21.15.39:0/449498159 shutdown_connections 2024-01-24T17:41:14.040 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.036+0000 7f43e6e00640 1 --2- 172.21.15.39:0/449498159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00b84d0 0x7f43e00cf430 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:14.041 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.036+0000 7f43e6e00640 1 --2- 172.21.15.39:0/449498159 >> v2:172.21.15.100:3300/0 conn(0x7f43e015cc50 0x7f43e01606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:14.041 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.036+0000 7f43e6e00640 1 -- 172.21.15.39:0/449498159 >> 172.21.15.39:0/449498159 conn(0x7f43e006ae50 msgr2=0x7f43e006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:14.041 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.036+0000 7f43e6e00640 1 -- 172.21.15.39:0/449498159 shutdown_connections 2024-01-24T17:41:14.041 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.036+0000 7f43e6e00640 1 -- 172.21.15.39:0/449498159 wait complete. 2024-01-24T17:41:14.044 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.038+0000 7f43e6e00640 1 Processor -- start 2024-01-24T17:41:14.044 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.038+0000 7f43e6e00640 1 -- start start 2024-01-24T17:41:14.044 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43e6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43e00b84d0 0x7f43e00cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43e6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00cf970 0x7f43e0151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43e6e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43e015cc50 0x7f43e015f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43e6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e0152440 con 0x7f43e00cf970 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43e6e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e015f910 con 0x7f43e015cc50 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43e6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e015fbe0 con 0x7f43e00b84d0 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00cf970 0x7f43e0151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00cf970 0x7f43e0151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51296/0 (socket says 172.21.15.39:51296) 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43df400640 1 -- 172.21.15.39:0/4152255565 learned_addr learned my addr 172.21.15.39:0/4152255565 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:14.045 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43dfe00640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43e00b84d0 0x7f43e00cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43e4e00640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43e015cc50 0x7f43e015f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43df400640 1 -- 172.21.15.39:0/4152255565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43e00b84d0 msgr2=0x7f43e00cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.039+0000 7f43df400640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43e00b84d0 0x7f43e00cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43df400640 1 -- 172.21.15.39:0/4152255565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43e015cc50 msgr2=0x7f43e015f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43df400640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43e015cc50 0x7f43e015f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43df400640 1 -- 172.21.15.39:0/4152255565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c80094d0 con 0x7f43e00cf970 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43df400640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00cf970 0x7f43e0151f00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f43c802ba70 tx=0x7f43c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43d7e00640 1 -- 172.21.15.39:0/4152255565 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8037c40 con 0x7f43e00cf970 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43d7e00640 1 -- 172.21.15.39:0/4152255565 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8037dc0 con 0x7f43e00cf970 2024-01-24T17:41:14.046 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43d7e00640 1 -- 172.21.15.39:0/4152255565 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8041640 con 0x7f43e00cf970 2024-01-24T17:41:14.047 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e015d260 con 0x7f43e00cf970 2024-01-24T17:41:14.047 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.040+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e015d820 con 0x7f43e00cf970 2024-01-24T17:41:14.047 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.041+0000 7f43d7e00640 1 -- 172.21.15.39:0/4152255565 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f43c8041aa0 con 0x7f43e00cf970 2024-01-24T17:41:14.047 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.041+0000 7f43d7e00640 1 -- 172.21.15.39:0/4152255565 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f43c803c070 con 0x7f43e00cf970 2024-01-24T17:41:14.047 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.042+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e004dff0 con 0x7f43e00cf970 2024-01-24T17:41:14.048 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.046+0000 7f43d7e00640 1 -- 172.21.15.39:0/4152255565 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f43c804b460 con 0x7f43e00cf970 2024-01-24T17:41:14.069 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:14.067+0000 7fd73f014040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T17:41:14.114 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.113+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43e00c93c0 con 0x7f43e00cf970 2024-01-24T17:41:14.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.115+0000 7f43d7e00640 1 -- 172.21.15.39:0/4152255565 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f43c804b9b0 con 0x7f43e00cf970 2024-01-24T17:41:14.116 INFO:teuthology.misc.health.smithi039.stdout: 2024-01-24T17:41:14.116 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":10,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:12.654149+0000","last_up_change":"0.000000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00cf970 msgr2=0x7f43e0151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00cf970 0x7f43e0151f00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f43c802ba70 tx=0x7f43c8004730 comp rx=0 tx=0).stop 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 shutdown_connections 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43e015cc50 0x7f43e015f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43e00cf970 0x7f43e0151f00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 --2- 172.21.15.39:0/4152255565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43e00b84d0 0x7f43e00cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 >> 172.21.15.39:0/4152255565 conn(0x7f43e006ae50 msgr2=0x7f43e00c6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 shutdown_connections 2024-01-24T17:41:14.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:14.116+0000 7f43e6e00640 1 -- 172.21.15.39:0/4152255565 wait complete. 2024-01-24T17:41:14.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-24T17:41:14.167 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:14.166+0000 7fad26433040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T17:41:14.181 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:14.179+0000 7fd73f014040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T17:41:14.231 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:14.230+0000 7fad26433040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T17:41:14.351 INFO:tasks.ceph.mgr.y.smithi039.stderr:2024-01-24T17:41:14.351+0000 7fad26433040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T17:41:15.231 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:15.230+0000 7fd6dfc00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-24T17:41:15.232 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-01-24T17:41:15.231+0000 7fd6dfc00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-24T17:41:16.935 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-01-24T17:41:16.933+0000 7f3e8a8b4500 -1 osd.4 0 log_to_monitors true 2024-01-24T17:41:16.964 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-01-24T17:41:16.963+0000 7ff908813500 -1 osd.6 0 log_to_monitors true 2024-01-24T17:41:16.986 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-01-24T17:41:16.985+0000 7fc0a9a1b500 -1 osd.5 0 log_to_monitors true 2024-01-24T17:41:16.993 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-01-24T17:41:16.992+0000 7f34c1415500 -1 osd.7 0 log_to_monitors true 2024-01-24T17:41:16.995 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-01-24T17:41:16.995+0000 7f23b5517500 -1 osd.2 0 log_to_monitors true 2024-01-24T17:41:17.008 INFO:tasks.ceph.osd.3.smithi039.stderr:2024-01-24T17:41:17.007+0000 7f38cde10500 -1 osd.3 0 log_to_monitors true 2024-01-24T17:41:17.056 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-01-24T17:41:17.055+0000 7f7039ffb500 -1 osd.1 0 log_to_monitors true 2024-01-24T17:41:17.173 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-01-24T17:41:17.173+0000 7fe0ad60e500 -1 osd.0 0 log_to_monitors true 2024-01-24T17:41:19.547 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:41:19.552 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:41:20.610 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-01-24T17:41:20.609+0000 7f34bb200640 -1 osd.7 0 waiting for initial osdmap 2024-01-24T17:41:20.610 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-01-24T17:41:20.610+0000 7fc0a5600640 -1 osd.5 0 waiting for initial osdmap 2024-01-24T17:41:20.622 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-01-24T17:41:20.621+0000 7f34b1c00640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T17:41:20.623 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-01-24T17:41:20.622+0000 7ff902600640 -1 osd.6 0 waiting for initial osdmap 2024-01-24T17:41:20.626 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-01-24T17:41:20.625+0000 7fc09a200640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T17:41:20.638 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-01-24T17:41:20.637+0000 7ff8f9000640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T17:41:20.642 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-01-24T17:41:20.641+0000 7f3e84e00640 -1 osd.4 0 waiting for initial osdmap 2024-01-24T17:41:20.656 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-01-24T17:41:20.655+0000 7f3e7b800640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T17:41:20.935 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:20.975 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-01-24T17:41:20.974+0000 7fe0a9200640 -1 osd.0 0 waiting for initial osdmap 2024-01-24T17:41:20.992 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-01-24T17:41:20.990+0000 7f23afa00640 -1 osd.2 0 waiting for initial osdmap 2024-01-24T17:41:20.996 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-01-24T17:41:20.994+0000 7fe09de00640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T17:41:21.000 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-01-24T17:41:20.999+0000 7f7034400640 -1 osd.1 0 waiting for initial osdmap 2024-01-24T17:41:21.008 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-01-24T17:41:21.007+0000 7f23a6400640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T17:41:21.022 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-01-24T17:41:21.020+0000 7f702ae00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T17:41:21.027 INFO:tasks.ceph.osd.3.smithi039.stderr:2024-01-24T17:41:21.025+0000 7f38c7c00640 -1 osd.3 0 waiting for initial osdmap 2024-01-24T17:41:21.047 INFO:tasks.ceph.osd.3.smithi039.stderr:2024-01-24T17:41:21.045+0000 7f38be600640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T17:41:21.110 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.108+0000 7f6327400640 1 Processor -- start 2024-01-24T17:41:21.110 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.109+0000 7f6327400640 1 -- start start 2024-01-24T17:41:21.112 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.110+0000 7f6327400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f632015a950 0x7f63200ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:21.112 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.110+0000 7f6327400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f63200ceae0 0x7f63200cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.110+0000 7f6327400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63200cef40 con 0x7f63200cd7c0 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.110+0000 7f6327400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632006c970 con 0x7f632015a950 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.110+0000 7f6327400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632006cbe0 con 0x7f63200ceae0 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.110+0000 7f6326000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f632015a950 0x7f63200ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.110+0000 7f6326000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f632015a950 0x7f63200ce5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51324/0 (socket says 172.21.15.39:51324) 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.110+0000 7f6326000640 1 -- 172.21.15.39:0/629460805 learned_addr learned my addr 172.21.15.39:0/629460805 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.111+0000 7f6325600640 1 --2- 172.21.15.39:0/629460805 >> v2:172.21.15.100:3300/0 conn(0x7f63200ceae0 0x7f63200cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.111+0000 7f6326000640 1 -- 172.21.15.39:0/629460805 >> v2:172.21.15.100:3300/0 conn(0x7f63200ceae0 msgr2=0x7f63200cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.111+0000 7f6326000640 1 --2- 172.21.15.39:0/629460805 >> v2:172.21.15.100:3300/0 conn(0x7f63200ceae0 0x7f63200cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.111+0000 7f6326000640 1 -- 172.21.15.39:0/629460805 >> v1:172.21.15.100:6789/0 conn(0x7f63200cd7c0 legacy=0x7f63200cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.111+0000 7f6326000640 1 -- 172.21.15.39:0/629460805 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63200b21f0 con 0x7f632015a950 2024-01-24T17:41:21.113 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.111+0000 7f6325600640 1 --2- 172.21.15.39:0/629460805 >> v2:172.21.15.100:3300/0 conn(0x7f63200ceae0 0x7f63200cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.112+0000 7f6326000640 1 --2- 172.21.15.39:0/629460805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63200ce5a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f63200cf280 tx=0x7f631002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85a7323b313d0c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.112+0000 7f6324c00640 1 -- 172.21.15.39:0/629460805 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6310031870 con 0x7f632015a950 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.112+0000 7f6324c00640 1 -- 172.21.15.39:0/629460805 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63100319d0 con 0x7f632015a950 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.112+0000 7f6324c00640 1 -- 172.21.15.39:0/629460805 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6310031d60 con 0x7f632015a950 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.113+0000 7f6327400640 1 -- 172.21.15.39:0/629460805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 msgr2=0x7f63200ce5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.113+0000 7f6327400640 1 --2- 172.21.15.39:0/629460805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63200ce5a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f63200cf280 tx=0x7f631002b910 comp rx=0 tx=0).stop 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.113+0000 7f6327400640 1 -- 172.21.15.39:0/629460805 shutdown_connections 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.113+0000 7f6327400640 1 --2- 172.21.15.39:0/629460805 >> v2:172.21.15.100:3300/0 conn(0x7f63200ceae0 0x7f63200cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.113+0000 7f6327400640 1 --2- 172.21.15.39:0/629460805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63200ce5a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.113+0000 7f6327400640 1 -- 172.21.15.39:0/629460805 >> 172.21.15.39:0/629460805 conn(0x7f6320069df0 msgr2=0x7f632006a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:21.115 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.113+0000 7f6327400640 1 -- 172.21.15.39:0/629460805 shutdown_connections 2024-01-24T17:41:21.116 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.113+0000 7f6327400640 1 -- 172.21.15.39:0/629460805 wait complete. 2024-01-24T17:41:21.116 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.114+0000 7f6327400640 1 Processor -- start 2024-01-24T17:41:21.116 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.115+0000 7f6327400640 1 -- start start 2024-01-24T17:41:21.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.115+0000 7f6327400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63200cd7c0 0x7f632015af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:21.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.115+0000 7f6327400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63200ceae0 0x7f632015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:21.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6327400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63201f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:21.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6327400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63201f5e00 con 0x7f632015a950 2024-01-24T17:41:21.117 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6327400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6320097ed0 con 0x7f63200cd7c0 2024-01-24T17:41:21.118 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6327400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632015bb70 con 0x7f63200ceae0 2024-01-24T17:41:21.118 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63201f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:21.118 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63201f17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51336/0 (socket says 172.21.15.39:51336) 2024-01-24T17:41:21.118 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 -- 172.21.15.39:0/1702300576 learned_addr learned my addr 172.21.15.39:0/1702300576 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:21.121 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 -- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63200ceae0 msgr2=0x7f632015b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:21.121 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326000640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63200cd7c0 0x7f632015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:21.121 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63200ceae0 0x7f632015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.121 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 -- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63200cd7c0 msgr2=0x7f632015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:21.121 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63200cd7c0 0x7f632015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 -- 172.21.15.39:0/1702300576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63100094d0 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.116+0000 7f6326a00640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63201f17b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f631800b9e0 tx=0x7f631800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.117+0000 7f631e000640 1 -- 172.21.15.39:0/1702300576 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631800caa0 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.117+0000 7f631e000640 1 -- 172.21.15.39:0/1702300576 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631800cc00 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.117+0000 7f631e000640 1 -- 172.21.15.39:0/1702300576 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318014440 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.117+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63201f5740 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.117+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632015be50 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.118+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63201f5f40 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.119+0000 7f631e000640 1 -- 172.21.15.39:0/1702300576 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) v1 ==== 88551+0+0 (secure 0 0 0) 0x7f631801d3b0 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.119+0000 7f631e000640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62f806a3d0 0x7f62f806c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.120+0000 7f631e000640 1 -- 172.21.15.39:0/1702300576 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f63180878e0 con 0x7f632015a950 2024-01-24T17:41:21.122 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.120+0000 7f6326000640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62f806a3d0 0x7f62f806c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:21.123 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.121+0000 7f6326000640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62f806a3d0 0x7f62f806c890 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6310036000 tx=0x7f631003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:21.126 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.124+0000 7f631e000640 1 -- 172.21.15.39:0/1702300576 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6318053d30 con 0x7f632015a950 2024-01-24T17:41:21.312 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.311+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f632004e840 con 0x7f632015a950 2024-01-24T17:41:21.314 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.312+0000 7f631e000640 1 -- 172.21.15.39:0/1702300576 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6318019070 con 0x7f632015a950 2024-01-24T17:41:21.314 INFO:teuthology.misc.health.smithi039.stdout: 2024-01-24T17:41:21.314 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":12,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:18.231372+0000","last_up_change":"0.000000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:21.317 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62f806a3d0 msgr2=0x7f62f806c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:21.317 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62f806a3d0 0x7f62f806c890 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6310036000 tx=0x7f631003c040 comp rx=0 tx=0).stop 2024-01-24T17:41:21.317 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 msgr2=0x7f63201f17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:21.317 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63201f17b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f631800b9e0 tx=0x7f631800bea0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.317 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 shutdown_connections 2024-01-24T17:41:21.317 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62f806a3d0 0x7f62f806c890 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.317 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f632015a950 0x7f63201f17b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.318 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63200ceae0 0x7f632015b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.318 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 --2- 172.21.15.39:0/1702300576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63200cd7c0 0x7f632015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:21.318 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 >> 172.21.15.39:0/1702300576 conn(0x7f6320069df0 msgr2=0x7f63200af640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:21.318 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 shutdown_connections 2024-01-24T17:41:21.318 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:21.315+0000 7f6327400640 1 -- 172.21.15.39:0/1702300576 wait complete. 2024-01-24T17:41:21.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-24T17:41:28.136 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:28.331 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.329+0000 7fe088e00640 1 Processor -- start 2024-01-24T17:41:28.331 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.330+0000 7fe088e00640 1 -- start start 2024-01-24T17:41:28.333 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.331+0000 7fe088e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840c3610 0x7fe0840c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.333 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.332+0000 7fe088e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe0840cac90 0x7fe0840cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.333 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.332+0000 7fe088e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0840c3fb0 con 0x7fe0840cc800 2024-01-24T17:41:28.333 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.332+0000 7fe088e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08406aec0 con 0x7fe0840cac90 2024-01-24T17:41:28.333 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.332+0000 7fe088e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840b1f20 con 0x7fe0840c3610 2024-01-24T17:41:28.334 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.332+0000 7fe07be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840c3610 0x7fe0840c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.334 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.332+0000 7fe083e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe0840cac90 0x7fe0840cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.334 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.332+0000 7fe07be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840c3610 0x7fe0840c3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:46056/0 (socket says 172.21.15.39:46056) 2024-01-24T17:41:28.334 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.332+0000 7fe07be00640 1 -- 172.21.15.39:0/3511787876 learned_addr learned my addr 172.21.15.39:0/3511787876 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:28.335 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.333+0000 7fe07be00640 1 -- 172.21.15.39:0/3511787876 >> v2:172.21.15.100:3300/0 conn(0x7fe0840cac90 msgr2=0x7fe0840cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:28.335 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.333+0000 7fe07be00640 1 --2- 172.21.15.39:0/3511787876 >> v2:172.21.15.100:3300/0 conn(0x7fe0840cac90 0x7fe0840cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.335 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.333+0000 7fe07be00640 1 -- 172.21.15.39:0/3511787876 >> v1:172.21.15.100:6789/0 conn(0x7fe0840cc800 legacy=0x7fe0840b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.335 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.333+0000 7fe07be00640 1 -- 172.21.15.39:0/3511787876 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08406b160 con 0x7fe0840c3610 2024-01-24T17:41:28.335 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.334+0000 7fe07be00640 1 --2- 172.21.15.39:0/3511787876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840c3610 0x7fe0840c3a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0098a0 tx=0x7fe06c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9285d62cd77c4cc server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:28.336 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe082a00640 1 -- 172.21.15.39:0/3511787876 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c030840 con 0x7fe0840c3610 2024-01-24T17:41:28.336 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe082a00640 1 -- 172.21.15.39:0/3511787876 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06c0309a0 con 0x7fe0840c3610 2024-01-24T17:41:28.336 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe082a00640 1 -- 172.21.15.39:0/3511787876 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c030d30 con 0x7fe0840c3610 2024-01-24T17:41:28.337 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe088e00640 1 -- 172.21.15.39:0/3511787876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840c3610 msgr2=0x7fe0840c3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.337 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe088e00640 1 --2- 172.21.15.39:0/3511787876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840c3610 0x7fe0840c3a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe06c0098a0 tx=0x7fe06c02b980 comp rx=0 tx=0).stop 2024-01-24T17:41:28.337 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe088e00640 1 -- 172.21.15.39:0/3511787876 shutdown_connections 2024-01-24T17:41:28.337 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe088e00640 1 --2- 172.21.15.39:0/3511787876 >> v2:172.21.15.100:3300/0 conn(0x7fe0840cac90 0x7fe0840cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.337 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe088e00640 1 --2- 172.21.15.39:0/3511787876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840c3610 0x7fe0840c3a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.337 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe088e00640 1 -- 172.21.15.39:0/3511787876 >> 172.21.15.39:0/3511787876 conn(0x7fe0840d0640 msgr2=0x7fe0840d0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:28.337 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe088e00640 1 -- 172.21.15.39:0/3511787876 shutdown_connections 2024-01-24T17:41:28.338 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.335+0000 7fe088e00640 1 -- 172.21.15.39:0/3511787876 wait complete. 2024-01-24T17:41:28.339 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.337+0000 7fe088e00640 1 Processor -- start 2024-01-24T17:41:28.339 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.337+0000 7fe088e00640 1 -- start start 2024-01-24T17:41:28.341 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.338+0000 7fe088e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0840c3610 0x7fe0840c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.341 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.338+0000 7fe088e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0840cac90 0x7fe0840ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.341 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.338+0000 7fe088e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840cc800 0x7fe0840c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.338+0000 7fe088e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840ca5c0 con 0x7fe0840cac90 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.338+0000 7fe088e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08415eb30 con 0x7fe0840c3610 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.338+0000 7fe088e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08415e240 con 0x7fe0840cc800 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0840cac90 0x7fe0840ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0840cac90 0x7fe0840ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34854/0 (socket says 172.21.15.39:34854) 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 -- 172.21.15.39:0/1321407170 learned_addr learned my addr 172.21.15.39:0/1321407170 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe083400640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0840c3610 0x7fe0840c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe083e00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840cc800 0x7fe0840c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 -- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840cc800 msgr2=0x7fe0840c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840cc800 0x7fe0840c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 -- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0840c3610 msgr2=0x7fe0840c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0840c3610 0x7fe0840c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 -- 172.21.15.39:0/1321407170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c0094d0 con 0x7fe0840cac90 2024-01-24T17:41:28.342 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe083e00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840cc800 0x7fe0840c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:28.343 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe083400640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0840c3610 0x7fe0840c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:28.343 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.339+0000 7fe07be00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0840cac90 0x7fe0840ce580 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe06c000c00 tx=0x7fe06c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:28.343 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.340+0000 7fe07b400640 1 -- 172.21.15.39:0/1321407170 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c039a90 con 0x7fe0840cac90 2024-01-24T17:41:28.343 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.340+0000 7fe07b400640 1 -- 172.21.15.39:0/1321407170 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06c039bf0 con 0x7fe0840cac90 2024-01-24T17:41:28.343 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.340+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08415e4e0 con 0x7fe0840cac90 2024-01-24T17:41:28.343 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.340+0000 7fe07b400640 1 -- 172.21.15.39:0/1321407170 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c043410 con 0x7fe0840cac90 2024-01-24T17:41:28.343 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.340+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0840ca2d0 con 0x7fe0840cac90 2024-01-24T17:41:28.343 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.341+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08404e840 con 0x7fe0840cac90 2024-01-24T17:41:28.344 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.342+0000 7fe07b400640 1 -- 172.21.15.39:0/1321407170 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe06c04c3b0 con 0x7fe0840cac90 2024-01-24T17:41:28.344 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.343+0000 7fe07b400640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe05c06a430 0x7fe05c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.344 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.343+0000 7fe07b400640 1 -- 172.21.15.39:0/1321407170 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe06c0aab60 con 0x7fe0840cac90 2024-01-24T17:41:28.345 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.344+0000 7fe083400640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe05c06a430 0x7fe05c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.346 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.345+0000 7fe083400640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe05c06a430 0x7fe05c06c8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe074000fb0 tx=0x7fe074009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:28.349 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.348+0000 7fe07b400640 1 -- 172.21.15.39:0/1321407170 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe06c076410 con 0x7fe0840cac90 2024-01-24T17:41:28.542 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.540+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0840b13e0 con 0x7fe0840cac90 2024-01-24T17:41:28.543 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.542+0000 7fe07b400640 1 -- 172.21.15.39:0/1321407170 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7fe06c079a30 con 0x7fe0840cac90 2024-01-24T17:41:28.544 INFO:teuthology.misc.health.smithi039.stdout: 2024-01-24T17:41:28.544 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":16,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:24.668408+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe05c06a430 msgr2=0x7fe05c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe05c06a430 0x7fe05c06c8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe074000fb0 tx=0x7fe074009040 comp rx=0 tx=0).stop 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0840cac90 msgr2=0x7fe0840ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0840cac90 0x7fe0840ce580 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe06c000c00 tx=0x7fe06c004730 comp rx=0 tx=0).stop 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 shutdown_connections 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0840cc800 0x7fe0840c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe05c06a430 0x7fe05c06c8f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0840cac90 0x7fe0840ce580 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 --2- 172.21.15.39:0/1321407170 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0840c3610 0x7fe0840c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 >> 172.21.15.39:0/1321407170 conn(0x7fe0840d0640 msgr2=0x7fe0840b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:28.547 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 shutdown_connections 2024-01-24T17:41:28.548 INFO:teuthology.misc.health.smithi039.stderr:2024-01-24T17:41:28.545+0000 7fe088e00640 1 -- 172.21.15.39:0/1321407170 wait complete. 2024-01-24T17:41:28.562 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-24T17:41:28.562 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-24T17:41:28.562 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-24T17:41:28.562 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-24T17:41:28.567 INFO:tasks.ceph.mds.a:Started 2024-01-24T17:41:28.567 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-24T17:41:28.567 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-24T17:41:28.572 INFO:tasks.ceph.mds.c:Started 2024-01-24T17:41:28.572 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-24T17:41:28.572 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-24T17:41:28.580 INFO:tasks.ceph.mds.b:Started 2024-01-24T17:41:28.580 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-24T17:41:28.580 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-24T17:41:28.588 INFO:tasks.ceph.mds.d:Started 2024-01-24T17:41:28.588 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-24T17:41:28.588 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:28.718 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:41:28.720 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:41:28.747 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:41:28.753 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:41:28.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.773+0000 7fc379400640 1 Processor -- start 2024-01-24T17:41:28.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.773+0000 7fc379400640 1 -- start start 2024-01-24T17:41:28.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.774+0000 7fc379400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc37415cbf0 0x7fc3740d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.774+0000 7fc379400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37415e010 con 0x7fc37415db60 2024-01-24T17:41:28.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.774+0000 7fc379400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740d0a90 con 0x7fc3740d0da0 2024-01-24T17:41:28.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.774+0000 7fc379400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3740cd760 con 0x7fc37415cbf0 2024-01-24T17:41:28.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.774+0000 7fc373400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc37415db60 0x7fc3740ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39146/0 (socket says 172.21.15.39:39146) 2024-01-24T17:41:28.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.774+0000 7fc373400640 1 -- 172.21.15.39:0/93554750 learned_addr learned my addr 172.21.15.39:0/93554750 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:28.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc373e00640 1 --2- 172.21.15.39:0/93554750 >> v2:172.21.15.100:3300/0 conn(0x7fc37415cbf0 0x7fc3740d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc372a00640 1 -- 172.21.15.39:0/93554750 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169069712 0 0) 0x7fc37415e010 con 0x7fc37415db60 2024-01-24T17:41:28.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc372a00640 1 -- 172.21.15.39:0/93554750 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc354003520 con 0x7fc37415db60 2024-01-24T17:41:28.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc372a00640 1 -- 172.21.15.39:0/93554750 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189815242 0 0) 0x7fc3740d0a90 con 0x7fc3740d0da0 2024-01-24T17:41:28.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc372a00640 1 -- 172.21.15.39:0/93554750 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37415e010 con 0x7fc3740d0da0 2024-01-24T17:41:28.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc373e00640 1 -- 172.21.15.39:0/93554750 >> v1:172.21.15.100:6789/0 conn(0x7fc3740d0da0 legacy=0x7fc37415e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc372a00640 1 -- 172.21.15.39:0/93554750 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1891522991 0 0) 0x7fc354003520 con 0x7fc37415db60 2024-01-24T17:41:28.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc373e00640 1 -- 172.21.15.39:0/93554750 >> v1:172.21.15.39:6789/0 conn(0x7fc37415db60 legacy=0x7fc3740ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.775+0000 7fc373e00640 1 -- 172.21.15.39:0/93554750 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3740cda40 con 0x7fc37415cbf0 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.776+0000 7fc373e00640 1 --2- 172.21.15.39:0/93554750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 0x7fc3740d03b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc3640097c0 tx=0x7fc36402b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bfce8fe0257bef8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.776+0000 7fc372a00640 1 -- 172.21.15.39:0/93554750 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c001690 con 0x7fc37415cbf0 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc372a00640 1 -- 172.21.15.39:0/93554750 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36402e050 con 0x7fc37415cbf0 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc372a00640 1 -- 172.21.15.39:0/93554750 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364030730 con 0x7fc37415cbf0 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc379400640 1 -- 172.21.15.39:0/93554750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 msgr2=0x7fc3740d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc379400640 1 --2- 172.21.15.39:0/93554750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 0x7fc3740d03b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc3640097c0 tx=0x7fc36402b800 comp rx=0 tx=0).stop 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc379400640 1 -- 172.21.15.39:0/93554750 shutdown_connections 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc379400640 1 --2- 172.21.15.39:0/93554750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 0x7fc3740d03b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc379400640 1 -- 172.21.15.39:0/93554750 >> 172.21.15.39:0/93554750 conn(0x7fc37409bb90 msgr2=0x7fc3740d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:28.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc379400640 1 -- 172.21.15.39:0/93554750 shutdown_connections 2024-01-24T17:41:28.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.777+0000 7fc379400640 1 -- 172.21.15.39:0/93554750 wait complete. 2024-01-24T17:41:28.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.779+0000 7fc379400640 1 Processor -- start 2024-01-24T17:41:28.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.779+0000 7fc379400640 1 -- start start 2024-01-24T17:41:28.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc379400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3740d0da0 0x7fc3740c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc379400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 0x7fc374145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc379400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc37415db60 0x7fc3741460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc379400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374158750 con 0x7fc3740d0da0 2024-01-24T17:41:28.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc379400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740c86f0 con 0x7fc37415cbf0 2024-01-24T17:41:28.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc379400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740c89d0 con 0x7fc37415db60 2024-01-24T17:41:28.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc373e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3740d0da0 0x7fc3740c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc373e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3740d0da0 0x7fc3740c5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34886/0 (socket says 172.21.15.39:34886) 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc373e00640 1 -- 172.21.15.39:0/3088406353 learned_addr learned my addr 172.21.15.39:0/3088406353 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc378a00640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc37415db60 0x7fc3741460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.780+0000 7fc373400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 0x7fc374145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc373e00640 1 -- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc37415db60 msgr2=0x7fc3741460f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc373e00640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc37415db60 0x7fc3741460f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc373e00640 1 -- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 msgr2=0x7fc374145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc373e00640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 0x7fc374145bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc373e00640 1 -- 172.21.15.39:0/3088406353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3640093a0 con 0x7fc3740d0da0 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc378a00640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc37415db60 0x7fc3741460f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc373e00640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3740d0da0 0x7fc3740c5f00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc3640097c0 tx=0x7fc3640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc36be00640 1 -- 172.21.15.39:0/3088406353 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364005b90 con 0x7fc3740d0da0 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc36be00640 1 -- 172.21.15.39:0/3088406353 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc364005cf0 con 0x7fc3740d0da0 2024-01-24T17:41:28.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc36be00640 1 -- 172.21.15.39:0/3088406353 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36403c8c0 con 0x7fc3740d0da0 2024-01-24T17:41:28.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3740c8cb0 con 0x7fc3740d0da0 2024-01-24T17:41:28.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.781+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3740c9270 con 0x7fc3740d0da0 2024-01-24T17:41:28.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.783+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3741588c0 con 0x7fc3740d0da0 2024-01-24T17:41:28.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.783+0000 7fc36be00640 1 -- 172.21.15.39:0/3088406353 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc364039040 con 0x7fc3740d0da0 2024-01-24T17:41:28.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.784+0000 7fc36be00640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc34c06a430 0x7fc34c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:28.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.784+0000 7fc36be00640 1 -- 172.21.15.39:0/3088406353 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc3640a9f40 con 0x7fc3740d0da0 2024-01-24T17:41:28.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.785+0000 7fc373400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc34c06a430 0x7fc34c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:28.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.785+0000 7fc373400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc34c06a430 0x7fc34c06c8f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0017c0 tx=0x7fc35c001630 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:28.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.789+0000 7fc36be00640 1 -- 172.21.15.39:0/3088406353 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc364075910 con 0x7fc3740d0da0 2024-01-24T17:41:28.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.980+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc37404dff0 con 0x7fc3740d0da0 2024-01-24T17:41:28.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.981+0000 7fc36be00640 1 -- 172.21.15.39:0/3088406353 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7fc36404c020 con 0x7fc3740d0da0 2024-01-24T17:41:28.983 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:28.983 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":16,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:24.668408+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:28.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc34c06a430 msgr2=0x7fc34c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc34c06a430 0x7fc34c06c8f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0017c0 tx=0x7fc35c001630 comp rx=0 tx=0).stop 2024-01-24T17:41:28.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3740d0da0 msgr2=0x7fc3740c5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:28.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3740d0da0 0x7fc3740c5f00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc3640097c0 tx=0x7fc3640046c0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 shutdown_connections 2024-01-24T17:41:28.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc37415db60 0x7fc3741460f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc37415cbf0 0x7fc374145bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc34c06a430 0x7fc34c06c8f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 --2- 172.21.15.39:0/3088406353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3740d0da0 0x7fc3740c5f00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:28.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 >> 172.21.15.39:0/3088406353 conn(0x7fc37409bb90 msgr2=0x7fc3740d2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:28.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 shutdown_connections 2024-01-24T17:41:28.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:28.983+0000 7fc379400640 1 -- 172.21.15.39:0/3088406353 wait complete. 2024-01-24T17:41:29.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:41:29.003 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:41:29.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.216+0000 7f5144200640 1 Processor -- start 2024-01-24T17:41:29.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.216+0000 7f5144200640 1 -- start start 2024-01-24T17:41:29.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.218+0000 7f5144200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f513c15fff0 0x7f513c0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.218+0000 7f5144200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c0cda90 con 0x7f513c0d14c0 2024-01-24T17:41:29.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.218+0000 7f5144200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c0d32c0 con 0x7f513c0b2a10 2024-01-24T17:41:29.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.218+0000 7f5144200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513c0b2e90 con 0x7f513c15fff0 2024-01-24T17:41:29.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.218+0000 7f5142200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f513c0d14c0 0x7f513c0cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39156/0 (socket says 172.21.15.39:39156) 2024-01-24T17:41:29.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.218+0000 7f5142200640 1 -- 172.21.15.39:0/2239652987 learned_addr learned my addr 172.21.15.39:0/2239652987 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f5141800640 1 --2- 172.21.15.39:0/2239652987 >> v2:172.21.15.100:3300/0 conn(0x7f513c15fff0 0x7f513c0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800806203 0 0) 0x7f513c0cda90 con 0x7f513c0d14c0 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511c003520 con 0x7f513c0d14c0 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726703715 0 0) 0x7f511c003520 con 0x7f513c0d14c0 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 >> v2:172.21.15.100:3300/0 conn(0x7f513c15fff0 msgr2=0x7f513c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 --2- 172.21.15.39:0/2239652987 >> v2:172.21.15.100:3300/0 conn(0x7f513c15fff0 0x7f513c0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 >> v1:172.21.15.100:6789/0 conn(0x7f513c0b2a10 legacy=0x7f513c0d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c0b30a0 con 0x7f513c0d14c0 2024-01-24T17:41:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f51340019f0 con 0x7f513c0d14c0 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 >> v1:172.21.15.39:6789/0 conn(0x7f513c0d14c0 legacy=0x7f513c0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c003970 0x7f511c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f511c0047c0 0x7f511c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f511c005180 0x7f511c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0b2e90 con 0x7f511c0047c0 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c00c020 con 0x7f511c003970 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.219+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511c00c1e0 con 0x7f511c005180 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.220+0000 7f5140e00640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f511c0047c0 0x7f511c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.220+0000 7f5141800640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c003970 0x7f511c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.220+0000 7f5140e00640 1 -- 172.21.15.39:0/2239652987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f511c005180 msgr2=0x7f511c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.220+0000 7f5140e00640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f511c005180 0x7f511c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.220+0000 7f5140e00640 1 -- 172.21.15.39:0/2239652987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c003970 msgr2=0x7f511c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.220+0000 7f5140e00640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c003970 0x7f511c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.220+0000 7f5140e00640 1 -- 172.21.15.39:0/2239652987 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c003520 con 0x7f511c0047c0 2024-01-24T17:41:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.220+0000 7f5142200640 1 -- 172.21.15.39:0/2239652987 reap_dead start 2024-01-24T17:41:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.221+0000 7f5140e00640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f511c0047c0 0x7f511c004c40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5124001720 tx=0x7f511c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc276bab4f815463 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.221+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51240052d0 con 0x7f511c0047c0 2024-01-24T17:41:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.221+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512402e050 con 0x7f511c0047c0 2024-01-24T17:41:29.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.221+0000 7f513be00640 1 -- 172.21.15.39:0/2239652987 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124003830 con 0x7f511c0047c0 2024-01-24T17:41:29.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.222+0000 7f5144200640 1 -- 172.21.15.39:0/2239652987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f511c0047c0 msgr2=0x7f511c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.222+0000 7f5144200640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f511c0047c0 0x7f511c004c40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5124001720 tx=0x7f511c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.222+0000 7f5144200640 1 -- 172.21.15.39:0/2239652987 shutdown_connections 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.222+0000 7f5144200640 1 --2- 172.21.15.39:0/2239652987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f511c0047c0 0x7f511c004c40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.222+0000 7f5144200640 1 -- 172.21.15.39:0/2239652987 >> 172.21.15.39:0/2239652987 conn(0x7f513c09bcc0 msgr2=0x7f513c0d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.222+0000 7f5144200640 1 -- 172.21.15.39:0/2239652987 shutdown_connections 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.222+0000 7f5144200640 1 -- 172.21.15.39:0/2239652987 wait complete. 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.224+0000 7f5144200640 1 Processor -- start 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.224+0000 7f5144200640 1 -- start start 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5144200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c0047c0 0x7f513c145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5144200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f513c158c80 0x7f513c0cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5144200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f513c15c180 0x7f513c0cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5144200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0c8e40 con 0x7f513c158c80 2024-01-24T17:41:29.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5144200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c0caa10 con 0x7f511c0047c0 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5144200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c146010 con 0x7f513c15c180 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5140e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f513c158c80 0x7f513c0cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5140e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f513c158c80 0x7f513c0cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34896/0 (socket says 172.21.15.39:34896) 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.225+0000 7f5140e00640 1 -- 172.21.15.39:0/673389951 learned_addr learned my addr 172.21.15.39:0/673389951 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5141800640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c0047c0 0x7f513c145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5142200640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f513c15c180 0x7f513c0cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5140e00640 1 -- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f513c15c180 msgr2=0x7f513c0cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5140e00640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f513c15c180 0x7f513c0cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5140e00640 1 -- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c0047c0 msgr2=0x7f513c145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5140e00640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c0047c0 0x7f513c145ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5140e00640 1 -- 172.21.15.39:0/673389951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c159410 con 0x7f513c158c80 2024-01-24T17:41:29.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5141800640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c0047c0 0x7f513c145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:29.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5140e00640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f513c158c80 0x7f513c0cb740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5124005040 tx=0x7f5124001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:29.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5139600640 1 -- 172.21.15.39:0/673389951 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124007a30 con 0x7f513c158c80 2024-01-24T17:41:29.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5139600640 1 -- 172.21.15.39:0/673389951 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51240069e0 con 0x7f513c158c80 2024-01-24T17:41:29.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5139600640 1 -- 172.21.15.39:0/673389951 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124006d70 con 0x7f513c158c80 2024-01-24T17:41:29.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.226+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c0cac60 con 0x7f513c158c80 2024-01-24T17:41:29.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.227+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c0cb220 con 0x7f513c158c80 2024-01-24T17:41:29.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.228+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c0cb3b0 con 0x7f513c158c80 2024-01-24T17:41:29.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.229+0000 7f5139600640 1 -- 172.21.15.39:0/673389951 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f51240035f0 con 0x7f513c158c80 2024-01-24T17:41:29.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.229+0000 7f5139600640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511406a460 0x7f511406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.230+0000 7f5139600640 1 -- 172.21.15.39:0/673389951 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f51240aa740 con 0x7f513c158c80 2024-01-24T17:41:29.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.230+0000 7f5141800640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511406a460 0x7f511406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.231+0000 7f5141800640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511406a460 0x7f511406c920 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f513c0690a0 tx=0x7f512c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:29.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.235+0000 7f5139600640 1 -- 172.21.15.39:0/673389951 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5124076070 con 0x7f513c158c80 2024-01-24T17:41:29.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.427+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f513c0c8e20 con 0x7f513c158c80 2024-01-24T17:41:29.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.428+0000 7f5139600640 1 -- 172.21.15.39:0/673389951 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f512404b0a0 con 0x7f513c158c80 2024-01-24T17:41:29.430 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:41:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511406a460 msgr2=0x7f511406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511406a460 0x7f511406c920 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f513c0690a0 tx=0x7f512c008040 comp rx=0 tx=0).stop 2024-01-24T17:41:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f513c158c80 msgr2=0x7f513c0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f513c158c80 0x7f513c0cb740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5124005040 tx=0x7f5124001120 comp rx=0 tx=0).stop 2024-01-24T17:41:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 shutdown_connections 2024-01-24T17:41:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f513c15c180 0x7f513c0cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511406a460 0x7f511406c920 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f513c158c80 0x7f513c0cb740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 --2- 172.21.15.39:0/673389951 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f511c0047c0 0x7f513c145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 >> 172.21.15.39:0/673389951 conn(0x7f513c09bcc0 msgr2=0x7f513c0af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:29.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 shutdown_connections 2024-01-24T17:41:29.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.432+0000 7f5144200640 1 -- 172.21.15.39:0/673389951 wait complete. 2024-01-24T17:41:29.453 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:41:29.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.669+0000 7f5ca5a00640 1 Processor -- start 2024-01-24T17:41:29.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.669+0000 7f5ca5a00640 1 -- start start 2024-01-24T17:41:29.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.671+0000 7f5ca5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015a720 0x7f5ca00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.671+0000 7f5ca5a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5ca015ab00 0x7f5ca00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.671+0000 7f5ca5a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca00ce400 con 0x7f5ca00ccd20 2024-01-24T17:41:29.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.671+0000 7f5ca5a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca00ce110 con 0x7f5ca015ab00 2024-01-24T17:41:29.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.671+0000 7f5ca5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca015b3b0 con 0x7f5ca015a720 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c9fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015a720 0x7f5ca00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c9fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015a720 0x7f5ca00cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:46130/0 (socket says 172.21.15.39:46130) 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c9fe00640 1 -- 172.21.15.39:0/2571526948 learned_addr learned my addr 172.21.15.39:0/2571526948 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5ca5000640 1 --1- 172.21.15.39:0/2571526948 >> v1:172.21.15.100:6789/0 conn(0x7f5ca00ccd20 0x7f5ca00ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:56886/0 (socket says 172.21.15.39:56886) 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c97200640 1 --2- 172.21.15.39:0/2571526948 >> v2:172.21.15.100:3300/0 conn(0x7f5ca015ab00 0x7f5ca00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c9fe00640 1 -- 172.21.15.39:0/2571526948 >> v2:172.21.15.100:3300/0 conn(0x7f5ca015ab00 msgr2=0x7f5ca00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c9fe00640 1 --2- 172.21.15.39:0/2571526948 >> v2:172.21.15.100:3300/0 conn(0x7f5ca015ab00 0x7f5ca00cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c9fe00640 1 -- 172.21.15.39:0/2571526948 >> v1:172.21.15.100:6789/0 conn(0x7f5ca00ccd20 legacy=0x7f5ca00ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c9fe00640 1 -- 172.21.15.39:0/2571526948 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca015b690 con 0x7f5ca015a720 2024-01-24T17:41:29.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.672+0000 7f5c9f400640 1 -- 172.21.15.39:0/2571526948 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688874662 0 0) 0x7f5ca00ce400 con 0x7f5ca00ccd20 2024-01-24T17:41:29.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.674+0000 7f5c9fe00640 1 --2- 172.21.15.39:0/2571526948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015a720 0x7f5ca00cdbd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009e50 tx=0x7f5c9002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af55fab4ae2796b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:29.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.674+0000 7f5c9f400640 1 -- 172.21.15.39:0/2571526948 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9002ca70 con 0x7f5ca015a720 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.674+0000 7f5c9f400640 1 -- 172.21.15.39:0/2571526948 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9002cbd0 con 0x7f5ca015a720 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.674+0000 7f5c9f400640 1 -- 172.21.15.39:0/2571526948 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c900343f0 con 0x7f5ca015a720 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.675+0000 7f5ca5a00640 1 -- 172.21.15.39:0/2571526948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015a720 msgr2=0x7f5ca00cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.675+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/2571526948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015a720 0x7f5ca00cdbd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5c90009e50 tx=0x7f5c9002bed0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.675+0000 7f5ca5a00640 1 -- 172.21.15.39:0/2571526948 shutdown_connections 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.675+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/2571526948 >> v2:172.21.15.100:3300/0 conn(0x7f5ca015ab00 0x7f5ca00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.675+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/2571526948 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015a720 0x7f5ca00cdbd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.675+0000 7f5ca5a00640 1 -- 172.21.15.39:0/2571526948 >> 172.21.15.39:0/2571526948 conn(0x7f5ca006a090 msgr2=0x7f5ca006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.675+0000 7f5ca5a00640 1 -- 172.21.15.39:0/2571526948 shutdown_connections 2024-01-24T17:41:29.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.675+0000 7f5ca5a00640 1 -- 172.21.15.39:0/2571526948 wait complete. 2024-01-24T17:41:29.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.677+0000 7f5ca5a00640 1 Processor -- start 2024-01-24T17:41:29.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.677+0000 7f5ca5a00640 1 -- start start 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ca015a720 0x7f5ca01efec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015ab00 0x7f5ca00aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca01f0400 0x7f5ca00b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca00b0ae0 con 0x7f5ca01f0400 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca00b0dc0 con 0x7f5ca015a720 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca00b10a0 con 0x7f5ca015ab00 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca01f0400 0x7f5ca00b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca01f0400 0x7f5ca00b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34898/0 (socket says 172.21.15.39:34898) 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5ca5000640 1 -- 172.21.15.39:0/1643022383 learned_addr learned my addr 172.21.15.39:0/1643022383 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.678+0000 7f5c97200640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015ab00 0x7f5ca00aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5c9fe00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ca015a720 0x7f5ca01efec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5ca5000640 1 -- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015ab00 msgr2=0x7f5ca00aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5ca5000640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015ab00 0x7f5ca00aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5ca5000640 1 -- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ca015a720 msgr2=0x7f5ca01efec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5ca5000640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ca015a720 0x7f5ca01efec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5ca5000640 1 -- 172.21.15.39:0/1643022383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90009b30 con 0x7f5ca01f0400 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5c9fe00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ca015a720 0x7f5ca01efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5ca5000640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca01f0400 0x7f5ca00b04b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800c410 tx=0x7f5c9800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5c9cc00640 1 -- 172.21.15.39:0/1643022383 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c980074e0 con 0x7f5ca01f0400 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5c9cc00640 1 -- 172.21.15.39:0/1643022383 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c98007640 con 0x7f5ca01f0400 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5c9cc00640 1 -- 172.21.15.39:0/1643022383 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c98004a20 con 0x7f5ca01f0400 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca00b1380 con 0x7f5ca01f0400 2024-01-24T17:41:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.679+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca0157410 con 0x7f5ca01f0400 2024-01-24T17:41:29.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.681+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca004dff0 con 0x7f5ca01f0400 2024-01-24T17:41:29.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.682+0000 7f5c9cc00640 1 -- 172.21.15.39:0/1643022383 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5c98004b80 con 0x7f5ca01f0400 2024-01-24T17:41:29.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.683+0000 7f5c9cc00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5c7806a4b0 0x7f5c7806c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:29.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.683+0000 7f5c9cc00640 1 -- 172.21.15.39:0/1643022383 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5c980881a0 con 0x7f5ca01f0400 2024-01-24T17:41:29.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.683+0000 7f5c9fe00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5c7806a4b0 0x7f5c7806c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:29.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.684+0000 7f5c9fe00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5c7806a4b0 0x7f5c7806c970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5c90035000 tx=0x7f5c90030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:29.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.687+0000 7f5c9cc00640 1 -- 172.21.15.39:0/1643022383 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5c98053d10 con 0x7f5ca01f0400 2024-01-24T17:41:29.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.923+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ca015b940 con 0x7f5ca01f0400 2024-01-24T17:41:29.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.924+0000 7f5c9cc00640 1 -- 172.21.15.39:0/1643022383 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3530 (secure 0 0 0) 0x7f5c98057330 con 0x7f5ca01f0400 2024-01-24T17:41:29.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:29.927 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-01-24T17:41:29.927 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 3 2024-01-24T17:41:29.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5c7806a4b0 msgr2=0x7f5c7806c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5c7806a4b0 0x7f5c7806c970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5c90035000 tx=0x7f5c90030000 comp rx=0 tx=0).stop 2024-01-24T17:41:29.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca01f0400 msgr2=0x7f5ca00b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:29.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca01f0400 0x7f5ca00b04b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800c410 tx=0x7f5c9800c8d0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 shutdown_connections 2024-01-24T17:41:29.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5c7806a4b0 0x7f5c7806c970 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ca01f0400 0x7f5ca00b04b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ca015ab00 0x7f5ca00aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 --2- 172.21.15.39:0/1643022383 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ca015a720 0x7f5ca01efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:29.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.927+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 >> 172.21.15.39:0/1643022383 conn(0x7f5ca006a090 msgr2=0x7f5ca015d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:29.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.928+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 shutdown_connections 2024-01-24T17:41:29.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:29.928+0000 7f5ca5a00640 1 -- 172.21.15.39:0/1643022383 wait complete. 2024-01-24T17:41:29.946 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:30.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.163+0000 7f185d200640 1 Processor -- start 2024-01-24T17:41:30.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.163+0000 7f185d200640 1 -- start start 2024-01-24T17:41:30.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.164+0000 7f185d200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f185815ca50 0x7f18580d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.164+0000 7f185d200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18580d1680 0x7f18580cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.164+0000 7f185d200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18580d0240 con 0x7f18580cf770 2024-01-24T17:41:30.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.164+0000 7f185d200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18580cff60 con 0x7f18580d1680 2024-01-24T17:41:30.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.164+0000 7f185d200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18580d0660 con 0x7f185815ca50 2024-01-24T17:41:30.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1856a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18580d1680 0x7f18580cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1857e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f18580cf770 0x7f18580cd400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39164/0 (socket says 172.21.15.39:39164) 2024-01-24T17:41:30.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1857e00640 1 -- 172.21.15.39:0/1834589242 learned_addr learned my addr 172.21.15.39:0/1834589242 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:30.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1857400640 1 --2- 172.21.15.39:0/1834589242 >> v2:172.21.15.100:3300/0 conn(0x7f185815ca50 0x7f18580d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1856a00640 1 -- 172.21.15.39:0/1834589242 >> v2:172.21.15.100:3300/0 conn(0x7f185815ca50 msgr2=0x7f18580d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:30.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1856a00640 1 --2- 172.21.15.39:0/1834589242 >> v2:172.21.15.100:3300/0 conn(0x7f185815ca50 0x7f18580d1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1856a00640 1 -- 172.21.15.39:0/1834589242 >> v1:172.21.15.39:6789/0 conn(0x7f18580cf770 legacy=0x7f18580cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1856a00640 1 -- 172.21.15.39:0/1834589242 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18580d0940 con 0x7f18580d1680 2024-01-24T17:41:30.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.165+0000 7f1856000640 1 -- 172.21.15.39:0/1834589242 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158253990 0 0) 0x7f18580d0240 con 0x7f18580cf770 2024-01-24T17:41:30.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.166+0000 7f1856a00640 1 --2- 172.21.15.39:0/1834589242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580d1680 0x7f18580cf230 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18580d3a60 tx=0x7f184002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d82907377aa9476 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:30.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f1856000640 1 -- 172.21.15.39:0/1834589242 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184002e070 con 0x7f18580d1680 2024-01-24T17:41:30.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f1856000640 1 -- 172.21.15.39:0/1834589242 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18400315a0 con 0x7f18580d1680 2024-01-24T17:41:30.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f1856000640 1 -- 172.21.15.39:0/1834589242 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840031930 con 0x7f18580d1680 2024-01-24T17:41:30.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f185d200640 1 -- 172.21.15.39:0/1834589242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580d1680 msgr2=0x7f18580cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f185d200640 1 --2- 172.21.15.39:0/1834589242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580d1680 0x7f18580cf230 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18580d3a60 tx=0x7f184002b840 comp rx=0 tx=0).stop 2024-01-24T17:41:30.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f185d200640 1 -- 172.21.15.39:0/1834589242 shutdown_connections 2024-01-24T17:41:30.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f185d200640 1 --2- 172.21.15.39:0/1834589242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580d1680 0x7f18580cf230 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f185d200640 1 --2- 172.21.15.39:0/1834589242 >> v2:172.21.15.100:3300/0 conn(0x7f185815ca50 0x7f18580d1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f185d200640 1 -- 172.21.15.39:0/1834589242 >> 172.21.15.39:0/1834589242 conn(0x7f1858071dd0 msgr2=0x7f185809bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:30.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f185d200640 1 -- 172.21.15.39:0/1834589242 shutdown_connections 2024-01-24T17:41:30.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.167+0000 7f185d200640 1 -- 172.21.15.39:0/1834589242 wait complete. 2024-01-24T17:41:30.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.169+0000 7f185d200640 1 Processor -- start 2024-01-24T17:41:30.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.169+0000 7f185d200640 1 -- start start 2024-01-24T17:41:30.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.170+0000 7f185d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580cf770 0x7f18580b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.170+0000 7f185d200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18580d1680 0x7f18580afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.170+0000 7f185d200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f185815ca50 0x7f18581f1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.170+0000 7f185d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185815e160 con 0x7f18580cf770 2024-01-24T17:41:30.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.170+0000 7f185d200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18580b0730 con 0x7f18580d1680 2024-01-24T17:41:30.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.170+0000 7f185d200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18580b0a10 con 0x7f185815ca50 2024-01-24T17:41:30.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580cf770 0x7f18580b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580cf770 0x7f18580b7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34914/0 (socket says 172.21.15.39:34914) 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 -- 172.21.15.39:0/388460687 learned_addr learned my addr 172.21.15.39:0/388460687 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1856a00640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18580d1680 0x7f18580afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 -- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f185815ca50 msgr2=0x7f18581f1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857e00640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f185815ca50 0x7f18581f1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f185815ca50 0x7f18581f1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 -- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18580d1680 msgr2=0x7f18580afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18580d1680 0x7f18580afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 -- 172.21.15.39:0/388460687 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1840009400 con 0x7f18580cf770 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1856a00640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18580d1680 0x7f18580afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857e00640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f185815ca50 0x7f18581f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.171+0000 7f1857400640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580cf770 0x7f18580b7c40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f184800d5f0 tx=0x7f184800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:30.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.172+0000 7f184f400640 1 -- 172.21.15.39:0/388460687 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18480125f0 con 0x7f18580cf770 2024-01-24T17:41:30.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.172+0000 7f184f400640 1 -- 172.21.15.39:0/388460687 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1848012750 con 0x7f18580cf770 2024-01-24T17:41:30.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.172+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18580b0cf0 con 0x7f18580cf770 2024-01-24T17:41:30.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.172+0000 7f184f400640 1 -- 172.21.15.39:0/388460687 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848012b00 con 0x7f18580cf770 2024-01-24T17:41:30.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.172+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18580b0420 con 0x7f18580cf770 2024-01-24T17:41:30.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.173+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18580677b0 con 0x7f18580cf770 2024-01-24T17:41:30.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.174+0000 7f184f400640 1 -- 172.21.15.39:0/388460687 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1848012c60 con 0x7f18580cf770 2024-01-24T17:41:30.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.175+0000 7f184f400640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f183006a430 0x7f183006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.175+0000 7f184f400640 1 -- 172.21.15.39:0/388460687 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1848088b40 con 0x7f18580cf770 2024-01-24T17:41:30.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.176+0000 7f1856a00640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f183006a430 0x7f183006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.177+0000 7f1856a00640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f183006a430 0x7f183006c8f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1840036000 tx=0x7f184003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:30.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.180+0000 7f184f400640 1 -- 172.21.15.39:0/388460687 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1848054400 con 0x7f18580cf770 2024-01-24T17:41:30.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.373+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1858067040 con 0x7f18580cf770 2024-01-24T17:41:30.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.375+0000 7f184f400640 1 -- 172.21.15.39:0/388460687 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12382 (secure 0 0 0) 0x7f1848057a20 con 0x7f18580cf770 2024-01-24T17:41:30.377 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:30.378 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":16,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:24.668408+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:30.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f183006a430 msgr2=0x7f183006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f183006a430 0x7f183006c8f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1840036000 tx=0x7f184003d040 comp rx=0 tx=0).stop 2024-01-24T17:41:30.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580cf770 msgr2=0x7f18580b7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580cf770 0x7f18580b7c40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f184800d5f0 tx=0x7f184800dab0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 shutdown_connections 2024-01-24T17:41:30.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f185815ca50 0x7f18581f1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18580d1680 0x7f18580afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f183006a430 0x7f183006c8f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 --2- 172.21.15.39:0/388460687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18580cf770 0x7f18580b7c40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.378+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 >> 172.21.15.39:0/388460687 conn(0x7f1858071dd0 msgr2=0x7f18580cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:30.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.379+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 shutdown_connections 2024-01-24T17:41:30.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.379+0000 7f185d200640 1 -- 172.21.15.39:0/388460687 wait complete. 2024-01-24T17:41:30.400 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:41:30.400 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:41:30.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.621+0000 7f4352400640 1 Processor -- start 2024-01-24T17:41:30.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.621+0000 7f4352400640 1 -- start start 2024-01-24T17:41:30.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.623+0000 7f4352400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f434c0b0350 0x7f434c06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.623+0000 7f4352400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c06c9f0 0x7f434c15da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.623+0000 7f4352400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434c06d4c0 con 0x7f434c0b0730 2024-01-24T17:41:30.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.623+0000 7f4352400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f434c06cf00 con 0x7f434c0b0350 2024-01-24T17:41:30.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.623+0000 7f4352400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c06d1e0 con 0x7f434c06c9f0 2024-01-24T17:41:30.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.624+0000 7f4343e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c06c9f0 0x7f434c15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.624+0000 7f4343e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c06c9f0 0x7f434c15da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:46148/0 (socket says 172.21.15.39:46148) 2024-01-24T17:41:30.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.624+0000 7f434b400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f434c0b0350 0x7f434c06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.624+0000 7f4343e00640 1 -- 172.21.15.39:0/2973676651 learned_addr learned my addr 172.21.15.39:0/2973676651 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:30.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.624+0000 7f4343e00640 1 -- 172.21.15.39:0/2973676651 >> v2:172.21.15.100:3300/0 conn(0x7f434c0b0350 msgr2=0x7f434c06c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:30.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.624+0000 7f4343e00640 1 --2- 172.21.15.39:0/2973676651 >> v2:172.21.15.100:3300/0 conn(0x7f434c0b0350 0x7f434c06c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.624+0000 7f4343e00640 1 -- 172.21.15.39:0/2973676651 >> v1:172.21.15.100:6789/0 conn(0x7f434c0b0730 legacy=0x7f434c070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.625+0000 7f4343e00640 1 -- 172.21.15.39:0/2973676651 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c0b15e0 con 0x7f434c06c9f0 2024-01-24T17:41:30.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.625+0000 7f434b400640 1 --2- 172.21.15.39:0/2973676651 >> v2:172.21.15.100:3300/0 conn(0x7f434c0b0350 0x7f434c06c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:30.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.626+0000 7f4343e00640 1 --2- 172.21.15.39:0/2973676651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c06c9f0 0x7f434c15da80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4334009870 tx=0x7f433402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eee62caed78de3f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:30.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.626+0000 7f434aa00640 1 -- 172.21.15.39:0/2973676651 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334031640 con 0x7f434c06c9f0 2024-01-24T17:41:30.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.626+0000 7f434aa00640 1 -- 172.21.15.39:0/2973676651 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43340317a0 con 0x7f434c06c9f0 2024-01-24T17:41:30.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.626+0000 7f434aa00640 1 -- 172.21.15.39:0/2973676651 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334031b30 con 0x7f434c06c9f0 2024-01-24T17:41:30.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.627+0000 7f4352400640 1 -- 172.21.15.39:0/2973676651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c06c9f0 msgr2=0x7f434c15da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.627+0000 7f4352400640 1 --2- 172.21.15.39:0/2973676651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c06c9f0 0x7f434c15da80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4334009870 tx=0x7f433402b8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.627+0000 7f4352400640 1 -- 172.21.15.39:0/2973676651 shutdown_connections 2024-01-24T17:41:30.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.627+0000 7f4352400640 1 --2- 172.21.15.39:0/2973676651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c06c9f0 0x7f434c15da80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.627+0000 7f4352400640 1 --2- 172.21.15.39:0/2973676651 >> v2:172.21.15.100:3300/0 conn(0x7f434c0b0350 0x7f434c06c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.627+0000 7f4352400640 1 -- 172.21.15.39:0/2973676651 >> 172.21.15.39:0/2973676651 conn(0x7f434c0cb0e0 msgr2=0x7f434c0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:30.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.627+0000 7f4352400640 1 -- 172.21.15.39:0/2973676651 shutdown_connections 2024-01-24T17:41:30.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.627+0000 7f4352400640 1 -- 172.21.15.39:0/2973676651 wait complete. 2024-01-24T17:41:30.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.629+0000 7f4352400640 1 Processor -- start 2024-01-24T17:41:30.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.629+0000 7f4352400640 1 -- start start 2024-01-24T17:41:30.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4352400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c06c9f0 0x7f434c0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4352400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434c0b0350 0x7f434c0c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4352400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b0730 0x7f434c0c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4352400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0c8590 con 0x7f434c0b0350 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4352400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0c8f60 con 0x7f434c06c9f0 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4352400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0c7980 con 0x7f434c0b0730 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4343e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434c0b0350 0x7f434c0c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4343e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434c0b0350 0x7f434c0c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34922/0 (socket says 172.21.15.39:34922) 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.630+0000 7f4343e00640 1 -- 172.21.15.39:0/3160004445 learned_addr learned my addr 172.21.15.39:0/3160004445 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f434be00640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b0730 0x7f434c0c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f434b400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c06c9f0 0x7f434c0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4343e00640 1 -- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b0730 msgr2=0x7f434c0c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4343e00640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b0730 0x7f434c0c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4343e00640 1 -- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c06c9f0 msgr2=0x7f434c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4343e00640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c06c9f0 0x7f434c0c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4343e00640 1 -- 172.21.15.39:0/3160004445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c009400 con 0x7f434c0b0350 2024-01-24T17:41:30.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4343e00640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434c0b0350 0x7f434c0c46f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4334000c00 tx=0x7f43340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:30.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4343400640 1 -- 172.21.15.39:0/3160004445 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433403a930 con 0x7f434c0b0350 2024-01-24T17:41:30.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4343400640 1 -- 172.21.15.39:0/3160004445 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433403aa90 con 0x7f434c0b0350 2024-01-24T17:41:30.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.631+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43340094d0 con 0x7f434c0b0350 2024-01-24T17:41:30.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.632+0000 7f4343400640 1 -- 172.21.15.39:0/3160004445 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433403ae40 con 0x7f434c0b0350 2024-01-24T17:41:30.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.632+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c0c7f40 con 0x7f434c0b0350 2024-01-24T17:41:30.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.633+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c0c8740 con 0x7f434c0b0350 2024-01-24T17:41:30.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.634+0000 7f4343400640 1 -- 172.21.15.39:0/3160004445 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f433402f050 con 0x7f434c0b0350 2024-01-24T17:41:30.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.635+0000 7f4343400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f432406a430 0x7f432406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:30.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.635+0000 7f4343400640 1 -- 172.21.15.39:0/3160004445 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f43340aa4a0 con 0x7f434c0b0350 2024-01-24T17:41:30.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.635+0000 7f434b400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f432406a430 0x7f432406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:30.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.636+0000 7f434b400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f432406a430 0x7f432406c8f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f433c009c60 tx=0x7f433c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:30.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.639+0000 7f4343400640 1 -- 172.21.15.39:0/3160004445 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f434c0c8740 con 0x7f434c0b0350 2024-01-24T17:41:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.832+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f434c073ef0 con 0x7f434c0b0350 2024-01-24T17:41:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.834+0000 7f4343400640 1 -- 172.21.15.39:0/3160004445 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4334075cc0 con 0x7f434c0b0350 2024-01-24T17:41:30.836 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:41:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f432406a430 msgr2=0x7f432406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f432406a430 0x7f432406c8f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f433c009c60 tx=0x7f433c008040 comp rx=0 tx=0).stop 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434c0b0350 msgr2=0x7f434c0c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434c0b0350 0x7f434c0c46f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4334000c00 tx=0x7f43340046c0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 shutdown_connections 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f434c0b0730 0x7f434c0c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f432406a430 0x7f432406c8f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434c0b0350 0x7f434c0c46f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 --2- 172.21.15.39:0/3160004445 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434c06c9f0 0x7f434c0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 >> 172.21.15.39:0/3160004445 conn(0x7f434c0cb0e0 msgr2=0x7f434c0cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 shutdown_connections 2024-01-24T17:41:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:30.837+0000 7f4352400640 1 -- 172.21.15.39:0/3160004445 wait complete. 2024-01-24T17:41:30.859 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T17:41:31.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.079+0000 7f9494a00640 1 Processor -- start 2024-01-24T17:41:31.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.079+0000 7f9494a00640 1 -- start start 2024-01-24T17:41:31.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.081+0000 7f9494a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900cb020 0x7f949015ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.081+0000 7f9494a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f94900cb400 0x7f94900ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.081+0000 7f9494a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949015b240 con 0x7f94900c8c10 2024-01-24T17:41:31.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.081+0000 7f9494a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f949015b520 con 0x7f94900cb400 2024-01-24T17:41:31.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.081+0000 7f9494a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949015b800 con 0x7f94900cb020 2024-01-24T17:41:31.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948f200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f94900c8c10 0x7f949015e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39166/0 (socket says 172.21.15.39:39166) 2024-01-24T17:41:31.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948f200640 1 -- 172.21.15.39:0/4201280151 learned_addr learned my addr 172.21.15.39:0/4201280151 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:31.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948e800640 1 --2- 172.21.15.39:0/4201280151 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900cb020 0x7f949015ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948de00640 1 --2- 172.21.15.39:0/4201280151 >> v2:172.21.15.100:3300/0 conn(0x7f94900cb400 0x7f94900ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948d400640 1 -- 172.21.15.39:0/4201280151 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423479312 0 0) 0x7f949015b240 con 0x7f94900c8c10 2024-01-24T17:41:31.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948d400640 1 -- 172.21.15.39:0/4201280151 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9470003520 con 0x7f94900c8c10 2024-01-24T17:41:31.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948de00640 1 -- 172.21.15.39:0/4201280151 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900cb020 msgr2=0x7f949015ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:31.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948d400640 1 -- 172.21.15.39:0/4201280151 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2881433373 0 0) 0x7f9470003520 con 0x7f94900c8c10 2024-01-24T17:41:31.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.082+0000 7f948de00640 1 --2- 172.21.15.39:0/4201280151 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900cb020 0x7f949015ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.083+0000 7f948de00640 1 -- 172.21.15.39:0/4201280151 >> v1:172.21.15.39:6789/0 conn(0x7f94900c8c10 legacy=0x7f949015e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.083+0000 7f948de00640 1 -- 172.21.15.39:0/4201280151 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949015bae0 con 0x7f94900cb400 2024-01-24T17:41:31.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.084+0000 7f948de00640 1 --2- 172.21.15.39:0/4201280151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb400 0x7f94900ca4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f94780097a0 tx=0x7f947802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8884682337931fb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:31.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.084+0000 7f948d400640 1 -- 172.21.15.39:0/4201280151 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488001670 con 0x7f94900cb400 2024-01-24T17:41:31.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.084+0000 7f948d400640 1 -- 172.21.15.39:0/4201280151 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947802e050 con 0x7f94900cb400 2024-01-24T17:41:31.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.084+0000 7f948d400640 1 -- 172.21.15.39:0/4201280151 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9478030710 con 0x7f94900cb400 2024-01-24T17:41:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.085+0000 7f9494a00640 1 -- 172.21.15.39:0/4201280151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb400 msgr2=0x7f94900ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.085+0000 7f9494a00640 1 --2- 172.21.15.39:0/4201280151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb400 0x7f94900ca4e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f94780097a0 tx=0x7f947802b790 comp rx=0 tx=0).stop 2024-01-24T17:41:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.085+0000 7f9494a00640 1 -- 172.21.15.39:0/4201280151 shutdown_connections 2024-01-24T17:41:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.085+0000 7f9494a00640 1 --2- 172.21.15.39:0/4201280151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb400 0x7f94900ca4e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.085+0000 7f9494a00640 1 --2- 172.21.15.39:0/4201280151 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900cb020 0x7f949015ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.085+0000 7f9494a00640 1 -- 172.21.15.39:0/4201280151 >> 172.21.15.39:0/4201280151 conn(0x7f949006a2a0 msgr2=0x7f94900afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.085+0000 7f9494a00640 1 -- 172.21.15.39:0/4201280151 shutdown_connections 2024-01-24T17:41:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.085+0000 7f9494a00640 1 -- 172.21.15.39:0/4201280151 wait complete. 2024-01-24T17:41:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.087+0000 7f9494a00640 1 Processor -- start 2024-01-24T17:41:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.087+0000 7f9494a00640 1 -- start start 2024-01-24T17:41:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f9494a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900c8c10 0x7f94900c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f9494a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb020 0x7f949015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f9494a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900cb400 0x7f9490157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f9494a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94901f1e00 con 0x7f94900cb400 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f9494a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949015c730 con 0x7f94900cb020 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f9494a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949015d2b0 con 0x7f94900c8c10 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f948f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900cb400 0x7f9490157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f948f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900cb400 0x7f9490157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34932/0 (socket says 172.21.15.39:34932) 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.088+0000 7f948f200640 1 -- 172.21.15.39:0/3852312976 learned_addr learned my addr 172.21.15.39:0/3852312976 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.089+0000 7f948e800640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900c8c10 0x7f94900c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.089+0000 7f948de00640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb020 0x7f949015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.089+0000 7f948f200640 1 -- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900c8c10 msgr2=0x7f94900c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.089+0000 7f948f200640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900c8c10 0x7f94900c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.089+0000 7f948f200640 1 -- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb020 msgr2=0x7f949015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.089+0000 7f948f200640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb020 0x7f949015d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.089+0000 7f948f200640 1 -- 172.21.15.39:0/3852312976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9478009400 con 0x7f94900cb400 2024-01-24T17:41:31.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.089+0000 7f948f200640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900cb400 0x7f9490157690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f948800c730 tx=0x7f948800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:31.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.090+0000 7f9486a00640 1 -- 172.21.15.39:0/3852312976 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488005ad0 con 0x7f94900cb400 2024-01-24T17:41:31.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.090+0000 7f9486a00640 1 -- 172.21.15.39:0/3852312976 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9488005c30 con 0x7f94900cb400 2024-01-24T17:41:31.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.090+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f949015df30 con 0x7f94900cb400 2024-01-24T17:41:31.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.090+0000 7f9486a00640 1 -- 172.21.15.39:0/3852312976 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488007070 con 0x7f94900cb400 2024-01-24T17:41:31.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.090+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949015cc60 con 0x7f94900cb400 2024-01-24T17:41:31.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.091+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949015d440 con 0x7f94900cb400 2024-01-24T17:41:31.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.092+0000 7f9486a00640 1 -- 172.21.15.39:0/3852312976 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f94880071d0 con 0x7f94900cb400 2024-01-24T17:41:31.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.093+0000 7f9486a00640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f946806a460 0x7f946806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.093+0000 7f9486a00640 1 -- 172.21.15.39:0/3852312976 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f94880883e0 con 0x7f94900cb400 2024-01-24T17:41:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.093+0000 7f948e800640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f946806a460 0x7f946806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.094+0000 7f948e800640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f946806a460 0x7f946806c920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f94800074a0 tx=0x7f948000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:31.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.097+0000 7f9486a00640 1 -- 172.21.15.39:0/3852312976 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9488053bf0 con 0x7f94900cb400 2024-01-24T17:41:31.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.296+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f949015d000 con 0x7f94900cb400 2024-01-24T17:41:31.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.297+0000 7f9486a00640 1 -- 172.21.15.39:0/3852312976 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9488057210 con 0x7f94900cb400 2024-01-24T17:41:31.298 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:31.299 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-01-24T17:41:31.299 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:41:31.299 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-01-24T17:41:31.299 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-01-24T17:41:31.299 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:41:31.299 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-01-24T17:41:31.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f946806a460 msgr2=0x7f946806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f946806a460 0x7f946806c920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f94800074a0 tx=0x7f948000b040 comp rx=0 tx=0).stop 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900cb400 msgr2=0x7f9490157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900cb400 0x7f9490157690 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f948800c730 tx=0x7f948800cbf0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 shutdown_connections 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f946806a460 0x7f946806c920 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94900cb400 0x7f9490157690 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94900cb020 0x7f949015d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 --2- 172.21.15.39:0/3852312976 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f94900c8c10 0x7f94900c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 >> 172.21.15.39:0/3852312976 conn(0x7f949006a2a0 msgr2=0x7f94901587f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 shutdown_connections 2024-01-24T17:41:31.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.300+0000 7f9494a00640 1 -- 172.21.15.39:0/3852312976 wait complete. 2024-01-24T17:41:31.322 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T17:41:31.322 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T17:41:31.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.548+0000 7f6f4d400640 1 Processor -- start 2024-01-24T17:41:31.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.548+0000 7f6f4d400640 1 -- start start 2024-01-24T17:41:31.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f4d400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f4815a7c0 0x7f6f4806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f4d400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6f4815ea00 0x7f6f480ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f4d400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f48070790 con 0x7f6f48070330 2024-01-24T17:41:31.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f4d400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4815f1b0 con 0x7f6f4815ea00 2024-01-24T17:41:31.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f4d400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480cbca0 con 0x7f6f4815a7c0 2024-01-24T17:41:31.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f4ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6f4815ea00 0x7f6f480ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f4ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6f4815ea00 0x7f6f480ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34938/0 (socket says 172.21.15.39:34938) 2024-01-24T17:41:31.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f4ca00640 1 -- 172.21.15.39:0/1685382192 learned_addr learned my addr 172.21.15.39:0/1685382192 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:31.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.550+0000 7f6f47e00640 1 --2- 172.21.15.39:0/1685382192 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f4815a7c0 0x7f6f4806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.551+0000 7f6f4ca00640 1 -- 172.21.15.39:0/1685382192 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f4815a7c0 msgr2=0x7f6f4806fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:31.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.551+0000 7f6f4ca00640 1 --2- 172.21.15.39:0/1685382192 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f4815a7c0 0x7f6f4806fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.551+0000 7f6f4ca00640 1 -- 172.21.15.39:0/1685382192 >> v1:172.21.15.100:6789/0 conn(0x7f6f48070330 legacy=0x7f6f4815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.551+0000 7f6f4ca00640 1 -- 172.21.15.39:0/1685382192 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4815eed0 con 0x7f6f4815ea00 2024-01-24T17:41:31.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.552+0000 7f6f4ca00640 1 --2- 172.21.15.39:0/1685382192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f480ccf40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009e50 tx=0x7f6f4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccb9c9ba4029c7a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:31.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.552+0000 7f6f46a00640 1 -- 172.21.15.39:0/1685382192 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4002ca80 con 0x7f6f4815ea00 2024-01-24T17:41:31.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.552+0000 7f6f46a00640 1 -- 172.21.15.39:0/1685382192 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4002cbe0 con 0x7f6f4815ea00 2024-01-24T17:41:31.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.552+0000 7f6f46a00640 1 -- 172.21.15.39:0/1685382192 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f400343f0 con 0x7f6f4815ea00 2024-01-24T17:41:31.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.553+0000 7f6f4d400640 1 -- 172.21.15.39:0/1685382192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 msgr2=0x7f6f480ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.553+0000 7f6f4d400640 1 --2- 172.21.15.39:0/1685382192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f480ccf40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009e50 tx=0x7f6f4002bee0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.553+0000 7f6f4d400640 1 -- 172.21.15.39:0/1685382192 shutdown_connections 2024-01-24T17:41:31.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.553+0000 7f6f4d400640 1 --2- 172.21.15.39:0/1685382192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f480ccf40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.553+0000 7f6f4d400640 1 --2- 172.21.15.39:0/1685382192 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f4815a7c0 0x7f6f4806fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.553+0000 7f6f4d400640 1 -- 172.21.15.39:0/1685382192 >> 172.21.15.39:0/1685382192 conn(0x7f6f480caeb0 msgr2=0x7f6f480cb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:31.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.553+0000 7f6f4d400640 1 -- 172.21.15.39:0/1685382192 shutdown_connections 2024-01-24T17:41:31.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.553+0000 7f6f4d400640 1 -- 172.21.15.39:0/1685382192 wait complete. 2024-01-24T17:41:31.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.555+0000 7f6f4d400640 1 Processor -- start 2024-01-24T17:41:31.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.555+0000 7f6f4d400640 1 -- start start 2024-01-24T17:41:31.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.556+0000 7f6f4d400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f48070330 0x7f6f481f4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.556+0000 7f6f4d400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f4815a7c0 0x7f6f480c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.556+0000 7f6f4d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f48154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.556+0000 7f6f4d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480c4960 con 0x7f6f4815ea00 2024-01-24T17:41:31.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.556+0000 7f6f4d400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f481f4ed0 con 0x7f6f4815a7c0 2024-01-24T17:41:31.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.556+0000 7f6f4d400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f480c9050 con 0x7f6f48070330 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.556+0000 7f6f4ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f48154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f4ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f48154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34948/0 (socket says 172.21.15.39:34948) 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f4ca00640 1 -- 172.21.15.39:0/4072214848 learned_addr learned my addr 172.21.15.39:0/4072214848 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f47400640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f4815a7c0 0x7f6f480c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f47e00640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f48070330 0x7f6f481f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f4ca00640 1 -- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f48070330 msgr2=0x7f6f481f4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f4ca00640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f48070330 0x7f6f481f4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f4ca00640 1 -- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f4815a7c0 msgr2=0x7f6f480c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f4ca00640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f4815a7c0 0x7f6f480c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f4ca00640 1 -- 172.21.15.39:0/4072214848 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f40009b30 con 0x7f6f4815ea00 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f47400640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f4815a7c0 0x7f6f480c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.557+0000 7f6f4ca00640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f48154a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009e50 tx=0x7f6f40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:31.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.558+0000 7f6f3fe00640 1 -- 172.21.15.39:0/4072214848 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40009510 con 0x7f6f4815ea00 2024-01-24T17:41:31.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.558+0000 7f6f3fe00640 1 -- 172.21.15.39:0/4072214848 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4003e450 con 0x7f6f4815ea00 2024-01-24T17:41:31.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.558+0000 7f6f3fe00640 1 -- 172.21.15.39:0/4072214848 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4003e800 con 0x7f6f4815ea00 2024-01-24T17:41:31.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.558+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f480c9330 con 0x7f6f4815ea00 2024-01-24T17:41:31.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.558+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f480c98f0 con 0x7f6f4815ea00 2024-01-24T17:41:31.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.559+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4804dff0 con 0x7f6f4815ea00 2024-01-24T17:41:31.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.560+0000 7f6f3fe00640 1 -- 172.21.15.39:0/4072214848 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6f4002cd50 con 0x7f6f4815ea00 2024-01-24T17:41:31.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.561+0000 7f6f3fe00640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f2006a430 0x7f6f2006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:31.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.561+0000 7f6f3fe00640 1 -- 172.21.15.39:0/4072214848 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6f400ab150 con 0x7f6f4815ea00 2024-01-24T17:41:31.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.562+0000 7f6f47e00640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f2006a430 0x7f6f2006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:31.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.563+0000 7f6f47e00640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f2006a430 0x7f6f2006c8f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6f38006fd0 tx=0x7f6f38008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:31.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.566+0000 7f6f3fe00640 1 -- 172.21.15.39:0/4072214848 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6f400769f0 con 0x7f6f4815ea00 2024-01-24T17:41:31.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.762+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6f480cb7c0 con 0x7f6f4815ea00 2024-01-24T17:41:31.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.849+0000 7f6f3fe00640 1 -- 172.21.15.39:0/4072214848 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6f4007a010 con 0x7f6f4815ea00 2024-01-24T17:41:31.851 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-01-24T17:41:31.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.852+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f2006a430 msgr2=0x7f6f2006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.852+0000 7f6f4d400640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f2006a430 0x7f6f2006c8f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6f38006fd0 tx=0x7f6f38008040 comp rx=0 tx=0).stop 2024-01-24T17:41:31.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.852+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 msgr2=0x7f6f48154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.852+0000 7f6f4d400640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f48154a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009e50 tx=0x7f6f40004a40 comp rx=0 tx=0).stop 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.853+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 shutdown_connections 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.853+0000 7f6f4d400640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f2006a430 0x7f6f2006c8f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.853+0000 7f6f4d400640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f4815ea00 0x7f6f48154a30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.853+0000 7f6f4d400640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f4815a7c0 0x7f6f480c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.853+0000 7f6f4d400640 1 --2- 172.21.15.39:0/4072214848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f48070330 0x7f6f481f4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.853+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 >> 172.21.15.39:0/4072214848 conn(0x7f6f480caeb0 msgr2=0x7f6f48157db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.853+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 shutdown_connections 2024-01-24T17:41:31.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:31.853+0000 7f6f4d400640 1 -- 172.21.15.39:0/4072214848 wait complete. 2024-01-24T17:41:31.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T17:41:32.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.090+0000 7fd8a0e00640 1 Processor -- start 2024-01-24T17:41:32.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.090+0000 7fd8a0e00640 1 -- start start 2024-01-24T17:41:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd8a0e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c071e70 0x7fd89c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd8a0e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd89c0b1210 0x7fd89c0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd8a0e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89c06fd80 con 0x7fd89c15a6e0 2024-01-24T17:41:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd8a0e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89c15aac0 con 0x7fd89c0b1210 2024-01-24T17:41:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd8a0e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c0708c0 con 0x7fd89c071e70 2024-01-24T17:41:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89b400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd89c15a6e0 0x7fd89c15eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39174/0 (socket says 172.21.15.39:39174) 2024-01-24T17:41:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89b400640 1 -- 172.21.15.39:0/3897839263 learned_addr learned my addr 172.21.15.39:0/3897839263 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89be00640 1 --2- 172.21.15.39:0/3897839263 >> v2:172.21.15.39:3300/0 conn(0x7fd89c0b1210 0x7fd89c0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89aa00640 1 --2- 172.21.15.39:0/3897839263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c071e70 0x7fd89c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89a000640 1 -- 172.21.15.39:0/3897839263 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26675513 0 0) 0x7fd89c06fd80 con 0x7fd89c15a6e0 2024-01-24T17:41:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89a000640 1 -- 172.21.15.39:0/3897839263 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd87c003520 con 0x7fd89c15a6e0 2024-01-24T17:41:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89be00640 1 -- 172.21.15.39:0/3897839263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c071e70 msgr2=0x7fd89c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89be00640 1 --2- 172.21.15.39:0/3897839263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c071e70 0x7fd89c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89be00640 1 -- 172.21.15.39:0/3897839263 >> v1:172.21.15.39:6789/0 conn(0x7fd89c15a6e0 legacy=0x7fd89c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.092+0000 7fd89be00640 1 -- 172.21.15.39:0/3897839263 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c070020 con 0x7fd89c0b1210 2024-01-24T17:41:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.093+0000 7fd89be00640 1 --2- 172.21.15.39:0/3897839263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0b1210 0x7fd89c0cb7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd894009d90 tx=0x7fd89402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33dfe0968ecbe3c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.093+0000 7fd89a000640 1 -- 172.21.15.39:0/3897839263 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89402cc50 con 0x7fd89c0b1210 2024-01-24T17:41:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.093+0000 7fd89a000640 1 -- 172.21.15.39:0/3897839263 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89402cdb0 con 0x7fd89c0b1210 2024-01-24T17:41:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.093+0000 7fd89a000640 1 -- 172.21.15.39:0/3897839263 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd894034640 con 0x7fd89c0b1210 2024-01-24T17:41:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.094+0000 7fd8a0e00640 1 -- 172.21.15.39:0/3897839263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0b1210 msgr2=0x7fd89c0cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.094+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/3897839263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0b1210 0x7fd89c0cb7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd894009d90 tx=0x7fd89402bea0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.094+0000 7fd8a0e00640 1 -- 172.21.15.39:0/3897839263 shutdown_connections 2024-01-24T17:41:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.094+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/3897839263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0b1210 0x7fd89c0cb7a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.094+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/3897839263 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c071e70 0x7fd89c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.094+0000 7fd8a0e00640 1 -- 172.21.15.39:0/3897839263 >> 172.21.15.39:0/3897839263 conn(0x7fd89c06a090 msgr2=0x7fd89c06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:32.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.094+0000 7fd8a0e00640 1 -- 172.21.15.39:0/3897839263 shutdown_connections 2024-01-24T17:41:32.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.094+0000 7fd8a0e00640 1 -- 172.21.15.39:0/3897839263 wait complete. 2024-01-24T17:41:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.096+0000 7fd8a0e00640 1 Processor -- start 2024-01-24T17:41:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.096+0000 7fd8a0e00640 1 -- start start 2024-01-24T17:41:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.097+0000 7fd8a0e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c0b1210 0x7fd89c156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.097+0000 7fd8a0e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd89c15a6e0 0x7fd89c0c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.097+0000 7fd8a0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0c93c0 0x7fd89c0ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.097+0000 7fd8a0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c0ae940 con 0x7fd89c0c93c0 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.097+0000 7fd8a0e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c1572e0 con 0x7fd89c15a6e0 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.097+0000 7fd8a0e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c0aeb90 con 0x7fd89c0b1210 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0c93c0 0x7fd89c0ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0c93c0 0x7fd89c0ab8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34954/0 (socket says 172.21.15.39:34954) 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 -- 172.21.15.39:0/201607281 learned_addr learned my addr 172.21.15.39:0/201607281 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89b400640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c0b1210 0x7fd89c156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89aa00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd89c15a6e0 0x7fd89c0c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 -- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c0b1210 msgr2=0x7fd89c156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c0b1210 0x7fd89c156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 -- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd89c15a6e0 msgr2=0x7fd89c0c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd89c15a6e0 0x7fd89c0c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 -- 172.21.15.39:0/201607281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd894009a70 con 0x7fd89c0c93c0 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.098+0000 7fd89be00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0c93c0 0x7fd89c0ab8c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd894009d90 tx=0x7fd894004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.099+0000 7fd893400640 1 -- 172.21.15.39:0/201607281 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd894004880 con 0x7fd89c0c93c0 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.099+0000 7fd893400640 1 -- 172.21.15.39:0/201607281 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd894009450 con 0x7fd89c0c93c0 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.099+0000 7fd893400640 1 -- 172.21.15.39:0/201607281 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd894034640 con 0x7fd89c0c93c0 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.099+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89c0aee70 con 0x7fd89c0c93c0 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.099+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89c0af430 con 0x7fd89c0c93c0 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.100+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89c04dff0 con 0x7fd89c0c93c0 2024-01-24T17:41:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.101+0000 7fd893400640 1 -- 172.21.15.39:0/201607281 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd89403b030 con 0x7fd89c0c93c0 2024-01-24T17:41:32.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.102+0000 7fd893400640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87406a430 0x7fd87406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:32.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.102+0000 7fd893400640 1 -- 172.21.15.39:0/201607281 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd8940aad50 con 0x7fd89c0c93c0 2024-01-24T17:41:32.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.103+0000 7fd89b400640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87406a430 0x7fd87406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:32.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.104+0000 7fd89b400640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87406a430 0x7fd87406c8f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd88c005880 tx=0x7fd88c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:32.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.106+0000 7fd893400640 1 -- 172.21.15.39:0/201607281 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd8940765c0 con 0x7fd89c0c93c0 2024-01-24T17:41:32.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.300+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd89c06c930 con 0x7fd89c0c93c0 2024-01-24T17:41:32.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.869+0000 7fd893400640 1 -- 172.21.15.39:0/201607281 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd894079be0 con 0x7fd89c0c93c0 2024-01-24T17:41:32.870 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-01-24T17:41:32.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.874+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87406a430 msgr2=0x7fd87406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:32.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.874+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87406a430 0x7fd87406c8f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd88c005880 tx=0x7fd88c009040 comp rx=0 tx=0).stop 2024-01-24T17:41:32.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.874+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0c93c0 msgr2=0x7fd89c0ab8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:32.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.874+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0c93c0 0x7fd89c0ab8c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd894009d90 tx=0x7fd894004a40 comp rx=0 tx=0).stop 2024-01-24T17:41:32.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.875+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 shutdown_connections 2024-01-24T17:41:32.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.875+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87406a430 0x7fd87406c8f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.875+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd89c0c93c0 0x7fd89c0ab8c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.875+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd89c15a6e0 0x7fd89c0c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.875+0000 7fd8a0e00640 1 --2- 172.21.15.39:0/201607281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd89c0b1210 0x7fd89c156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:32.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.875+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 >> 172.21.15.39:0/201607281 conn(0x7fd89c06a090 msgr2=0x7fd89c159d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:32.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.877+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 shutdown_connections 2024-01-24T17:41:32.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:32.877+0000 7fd8a0e00640 1 -- 172.21.15.39:0/201607281 wait complete. 2024-01-24T17:41:32.914 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T17:41:33.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.103+0000 7fadfcc00640 1 Processor -- start 2024-01-24T17:41:33.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.103+0000 7fadfcc00640 1 -- start start 2024-01-24T17:41:33.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.105+0000 7fadfcc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadf80b0350 0x7fadf806c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.105+0000 7fadfcc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf806d4c0 con 0x7fadf815e190 2024-01-24T17:41:33.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.105+0000 7fadfcc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf806cf00 con 0x7fadf806c9f0 2024-01-24T17:41:33.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.105+0000 7fadfcc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf806d1e0 con 0x7fadf80b0350 2024-01-24T17:41:33.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.105+0000 7fadf6800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fadf815e190 0x7fadf80b1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52304/0 (socket says 172.21.15.39:52304) 2024-01-24T17:41:33.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.105+0000 7fadf6800640 1 -- 172.21.15.39:0/1972696452 learned_addr learned my addr 172.21.15.39:0/1972696452 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5e00640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadf80b0350 0x7fadf806c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092923889 0 0) 0x7fadf806d4c0 con 0x7fadf815e190 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd8003520 con 0x7fadf815e190 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 812979668 0 0) 0x7fadd8003520 con 0x7fadf815e190 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadf80b0350 msgr2=0x7fadf806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadf80b0350 0x7fadf806c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 >> v1:172.21.15.100:6789/0 conn(0x7fadf806c9f0 legacy=0x7fadf815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80b15e0 con 0x7fadf815e190 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fadf00019f0 con 0x7fadf815e190 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 >> v1:172.21.15.39:6789/0 conn(0x7fadf815e190 legacy=0x7fadf80b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadf80b0350 0x7fadd80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd8003ee0 0x7fadd80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd8004d00 0x7fadd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf806d1e0 con 0x7fadd8003ee0 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd800ba90 con 0x7fadd8004d00 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.106+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd800bc50 con 0x7fadf80b0350 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.107+0000 7fadefe00640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd8003ee0 0x7fadd80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.107+0000 7fadf6800640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd8004d00 0x7fadd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.107+0000 7fadefe00640 1 -- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadf80b0350 msgr2=0x7fadd80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.107+0000 7fadefe00640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadf80b0350 0x7fadd80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.107+0000 7fadefe00640 1 -- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd8004d00 msgr2=0x7fadd8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.107+0000 7fadefe00640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd8004d00 0x7fadd8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.107+0000 7fadefe00640 1 -- 172.21.15.39:0/1972696452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd8003520 con 0x7fadd8003ee0 2024-01-24T17:41:33.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.108+0000 7fadefe00640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd8003ee0 0x7fadd80047c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fade0001830 tx=0x7fadd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329cf57372f9c51a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:33.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.108+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade002f050 con 0x7fadd8003ee0 2024-01-24T17:41:33.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade0033610 con 0x7fadd8003ee0 2024-01-24T17:41:33.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadf5400640 1 -- 172.21.15.39:0/1972696452 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade00339c0 con 0x7fadd8003ee0 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadfcc00640 1 -- 172.21.15.39:0/1972696452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd8003ee0 msgr2=0x7fadd80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadfcc00640 1 --2- 172.21.15.39:0/1972696452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd8003ee0 0x7fadd80047c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fade0001830 tx=0x7fadd80057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadf6800640 1 -- 172.21.15.39:0/1972696452 reap_dead start 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadfcc00640 1 -- 172.21.15.39:0/1972696452 shutdown_connections 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadfcc00640 1 -- 172.21.15.39:0/1972696452 >> 172.21.15.39:0/1972696452 conn(0x7fadf80cb0e0 msgr2=0x7fadf80cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadfcc00640 1 -- 172.21.15.39:0/1972696452 shutdown_connections 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.109+0000 7fadfcc00640 1 -- 172.21.15.39:0/1972696452 wait complete. 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.111+0000 7fadfcc00640 1 Processor -- start 2024-01-24T17:41:33.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.111+0000 7fadfcc00640 1 -- start start 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadfcc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd8003ee0 0x7fadf814fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadfcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadf81502e0 0x7fadf8155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadfcc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadf8155f90 0x7fadf80c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadfcc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80c75e0 con 0x7fadf81502e0 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadfcc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80c78c0 con 0x7fadf8155f90 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadfcc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf80c7ba0 con 0x7fadd8003ee0 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadefe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadf81502e0 0x7fadf8155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadefe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadf81502e0 0x7fadf8155a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58072/0 (socket says 172.21.15.39:58072) 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.112+0000 7fadefe00640 1 -- 172.21.15.39:0/2133137007 learned_addr learned my addr 172.21.15.39:0/2133137007 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:33.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadf6800640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadf8155f90 0x7fadf80c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadf5e00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd8003ee0 0x7fadf814fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadefe00640 1 -- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd8003ee0 msgr2=0x7fadf814fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadefe00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd8003ee0 0x7fadf814fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadefe00640 1 -- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadf8155f90 msgr2=0x7fadf80c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadefe00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadf8155f90 0x7fadf80c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadefe00640 1 -- 172.21.15.39:0/2133137007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf80c7e80 con 0x7fadf81502e0 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadf6800640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadf8155f90 0x7fadf80c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadefe00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadf81502e0 0x7fadf8155a50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fade002bd30 tx=0x7fade0001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadee000640 1 -- 172.21.15.39:0/2133137007 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade0007050 con 0x7fadf81502e0 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadee000640 1 -- 172.21.15.39:0/2133137007 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade00071b0 con 0x7fadf81502e0 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadee000640 1 -- 172.21.15.39:0/2133137007 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade0007560 con 0x7fadf81502e0 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.113+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf80c8160 con 0x7fadf81502e0 2024-01-24T17:41:33.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.114+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf8150e30 con 0x7fadf81502e0 2024-01-24T17:41:33.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.115+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf8073ef0 con 0x7fadf81502e0 2024-01-24T17:41:33.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.116+0000 7fadee000640 1 -- 172.21.15.39:0/2133137007 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fade00067d0 con 0x7fadf81502e0 2024-01-24T17:41:33.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.117+0000 7fadee000640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadd006a430 0x7fadd006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.117+0000 7fadee000640 1 -- 172.21.15.39:0/2133137007 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fade00aa6e0 con 0x7fadf81502e0 2024-01-24T17:41:33.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.117+0000 7fadf5e00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadd006a430 0x7fadd006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.118+0000 7fadf5e00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadd006a430 0x7fadd006c8f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fade8009970 tx=0x7fade8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:33.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.122+0000 7fadee000640 1 -- 172.21.15.39:0/2133137007 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fade0075c00 con 0x7fadf81502e0 2024-01-24T17:41:33.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.352+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fadf80c75e0 con 0x7fadf81502e0 2024-01-24T17:41:33.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.362+0000 7fadee000640 1 -- 172.21.15.39:0/2133137007 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fade00076c0 con 0x7fadf81502e0 2024-01-24T17:41:33.362 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 2 and data pool 3 2024-01-24T17:41:33.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.365+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadd006a430 msgr2=0x7fadd006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.365+0000 7fadfcc00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadd006a430 0x7fadd006c8f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fade8009970 tx=0x7fade8008040 comp rx=0 tx=0).stop 2024-01-24T17:41:33.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.365+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadf81502e0 msgr2=0x7fadf8155a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.365+0000 7fadfcc00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadf81502e0 0x7fadf8155a50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fade002bd30 tx=0x7fade0001230 comp rx=0 tx=0).stop 2024-01-24T17:41:33.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.367+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 shutdown_connections 2024-01-24T17:41:33.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.367+0000 7fadfcc00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadf8155f90 0x7fadf80c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.367+0000 7fadfcc00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadd006a430 0x7fadd006c8f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.367+0000 7fadfcc00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadf81502e0 0x7fadf8155a50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.368+0000 7fadfcc00640 1 --2- 172.21.15.39:0/2133137007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd8003ee0 0x7fadf814fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.368+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 >> 172.21.15.39:0/2133137007 conn(0x7fadf80cb0e0 msgr2=0x7fadf8152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:33.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.368+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 shutdown_connections 2024-01-24T17:41:33.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.369+0000 7fadfcc00640 1 -- 172.21.15.39:0/2133137007 wait complete. 2024-01-24T17:41:33.405 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:33.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.583+0000 7f214bc00640 1 Processor -- start 2024-01-24T17:41:33.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.583+0000 7f214bc00640 1 -- start start 2024-01-24T17:41:33.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.584+0000 7f214bc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f214415d020 0x7f214415d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.584+0000 7f214bc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f214415eb80 0x7f214415f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.584+0000 7f214bc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214415dac0 con 0x7f21440cf990 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.584+0000 7f214bc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214415e080 con 0x7f214415d020 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.584+0000 7f214bc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214415dda0 con 0x7f214415eb80 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f2149200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f21440cf990 0x7f21440cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52318/0 (socket says 172.21.15.39:52318) 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f2149200640 1 -- 172.21.15.39:0/4104133188 learned_addr learned my addr 172.21.15.39:0/4104133188 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f213be00640 1 --2- 172.21.15.39:0/4104133188 >> v2:172.21.15.39:3300/0 conn(0x7f214415d020 0x7f214415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f2149c00640 1 --2- 172.21.15.39:0/4104133188 >> v2:172.21.15.100:3300/0 conn(0x7f214415eb80 0x7f214415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f2143e00640 1 -- 172.21.15.39:0/4104133188 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173286554 0 0) 0x7f214415dac0 con 0x7f21440cf990 2024-01-24T17:41:33.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f2143e00640 1 -- 172.21.15.39:0/4104133188 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2124003520 con 0x7f21440cf990 2024-01-24T17:41:33.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f213be00640 1 -- 172.21.15.39:0/4104133188 >> v2:172.21.15.100:3300/0 conn(0x7f214415eb80 msgr2=0x7f214415f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:33.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f213be00640 1 --2- 172.21.15.39:0/4104133188 >> v2:172.21.15.100:3300/0 conn(0x7f214415eb80 0x7f214415f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f213be00640 1 -- 172.21.15.39:0/4104133188 >> v1:172.21.15.39:6789/0 conn(0x7f21440cf990 legacy=0x7f21440cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.585+0000 7f213be00640 1 -- 172.21.15.39:0/4104133188 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214415e380 con 0x7f214415d020 2024-01-24T17:41:33.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.587+0000 7f213be00640 1 --2- 172.21.15.39:0/4104133188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214415d020 0x7f214415d490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21440b4900 tx=0x7f212c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42921ee9040f753c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:33.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.587+0000 7f2143e00640 1 -- 172.21.15.39:0/4104133188 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c031830 con 0x7f214415d020 2024-01-24T17:41:33.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.587+0000 7f2143e00640 1 -- 172.21.15.39:0/4104133188 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c031990 con 0x7f214415d020 2024-01-24T17:41:33.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.587+0000 7f2143e00640 1 -- 172.21.15.39:0/4104133188 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c031d20 con 0x7f214415d020 2024-01-24T17:41:33.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.587+0000 7f214bc00640 1 -- 172.21.15.39:0/4104133188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214415d020 msgr2=0x7f214415d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.587+0000 7f214bc00640 1 --2- 172.21.15.39:0/4104133188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214415d020 0x7f214415d490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21440b4900 tx=0x7f212c02b8d0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.588+0000 7f214bc00640 1 -- 172.21.15.39:0/4104133188 shutdown_connections 2024-01-24T17:41:33.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.588+0000 7f214bc00640 1 --2- 172.21.15.39:0/4104133188 >> v2:172.21.15.100:3300/0 conn(0x7f214415eb80 0x7f214415f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.588+0000 7f214bc00640 1 --2- 172.21.15.39:0/4104133188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214415d020 0x7f214415d490 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.588+0000 7f214bc00640 1 -- 172.21.15.39:0/4104133188 >> 172.21.15.39:0/4104133188 conn(0x7f2144071790 msgr2=0x7f2144071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:33.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.588+0000 7f214bc00640 1 -- 172.21.15.39:0/4104133188 shutdown_connections 2024-01-24T17:41:33.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.588+0000 7f214bc00640 1 -- 172.21.15.39:0/4104133188 wait complete. 2024-01-24T17:41:33.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.589+0000 7f214bc00640 1 Processor -- start 2024-01-24T17:41:33.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.589+0000 7f214bc00640 1 -- start start 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.590+0000 7f214bc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21440cf990 0x7f2144156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.590+0000 7f214bc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214415d020 0x7f21440d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.590+0000 7f214bc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214415eb80 0x7f2144158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.590+0000 7f214bc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440d2dd0 con 0x7f21440cf990 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.590+0000 7f214bc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440d30b0 con 0x7f214415d020 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.590+0000 7f214bc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21440d3390 con 0x7f214415eb80 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21440cf990 0x7f2144156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21440cf990 0x7f2144156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58092/0 (socket says 172.21.15.39:58092) 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 -- 172.21.15.39:0/1777359670 learned_addr learned my addr 172.21.15.39:0/1777359670 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 -- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214415eb80 msgr2=0x7f2144158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214415eb80 0x7f2144158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 -- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214415d020 msgr2=0x7f21440d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214415d020 0x7f21440d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 -- 172.21.15.39:0/1777359670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c009400 con 0x7f21440cf990 2024-01-24T17:41:33.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.591+0000 7f2149200640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21440cf990 0x7f2144156fe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f213400c500 tx=0x7f213400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:33.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.592+0000 7f2141600640 1 -- 172.21.15.39:0/1777359670 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134005ce0 con 0x7f21440cf990 2024-01-24T17:41:33.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.592+0000 7f2141600640 1 -- 172.21.15.39:0/1777359670 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2134005e40 con 0x7f21440cf990 2024-01-24T17:41:33.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.592+0000 7f2141600640 1 -- 172.21.15.39:0/1777359670 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134007330 con 0x7f21440cf990 2024-01-24T17:41:33.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.592+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21440ca660 con 0x7f21440cf990 2024-01-24T17:41:33.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.592+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2144157700 con 0x7f21440cf990 2024-01-24T17:41:33.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.593+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21440ca850 con 0x7f21440cf990 2024-01-24T17:41:33.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.594+0000 7f2141600640 1 -- 172.21.15.39:0/1777359670 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2134007550 con 0x7f21440cf990 2024-01-24T17:41:33.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.595+0000 7f2141600640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211c06a430 0x7f211c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:33.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.595+0000 7f2141600640 1 -- 172.21.15.39:0/1777359670 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2134087ff0 con 0x7f21440cf990 2024-01-24T17:41:33.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.596+0000 7f213be00640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211c06a430 0x7f211c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:33.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.597+0000 7f213be00640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211c06a430 0x7f211c06c8f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f212c036000 tx=0x7f212c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:33.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.599+0000 7f2141600640 1 -- 172.21.15.39:0/1777359670 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f21340bacb0 con 0x7f21440cf990 2024-01-24T17:41:33.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.787+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2144073ef0 con 0x7f21440cf990 2024-01-24T17:41:33.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.788+0000 7f2141600640 1 -- 172.21.15.39:0/1777359670 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f2134017090 con 0x7f21440cf990 2024-01-24T17:41:33.789 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:33.790 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":19,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:33.353970+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211c06a430 msgr2=0x7f211c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211c06a430 0x7f211c06c8f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f212c036000 tx=0x7f212c030000 comp rx=0 tx=0).stop 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21440cf990 msgr2=0x7f2144156fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21440cf990 0x7f2144156fe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f213400c500 tx=0x7f213400c9c0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 shutdown_connections 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214415eb80 0x7f2144158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214415d020 0x7f21440d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211c06a430 0x7f211c06c8f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 --2- 172.21.15.39:0/1777359670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21440cf990 0x7f2144156fe0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 >> 172.21.15.39:0/1777359670 conn(0x7f2144071790 msgr2=0x7f214415eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:33.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 shutdown_connections 2024-01-24T17:41:33.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:33.791+0000 7f214bc00640 1 -- 172.21.15.39:0/1777359670 wait complete. 2024-01-24T17:41:33.809 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:34.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.022+0000 7f1ba9400640 1 Processor -- start 2024-01-24T17:41:34.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.022+0000 7f1ba9400640 1 -- start start 2024-01-24T17:41:34.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba9400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1ba415a720 0x7f1ba40cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba9400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1ba40ce8e0 0x7f1ba40ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba9400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba40cf260 con 0x7f1ba415ab00 2024-01-24T17:41:34.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba9400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba40ce400 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba9400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba40ccef0 con 0x7f1ba415a720 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba8a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1ba40ce8e0 0x7f1ba40ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba8a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1ba40ce8e0 0x7f1ba40ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58102/0 (socket says 172.21.15.39:58102) 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba8a00640 1 -- 172.21.15.39:0/558525674 learned_addr learned my addr 172.21.15.39:0/558525674 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.024+0000 7f1ba3e00640 1 --2- 172.21.15.39:0/558525674 >> v2:172.21.15.100:3300/0 conn(0x7f1ba415a720 0x7f1ba40cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.025+0000 7f1ba8a00640 1 -- 172.21.15.39:0/558525674 >> v2:172.21.15.100:3300/0 conn(0x7f1ba415a720 msgr2=0x7f1ba40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.025+0000 7f1ba8a00640 1 --2- 172.21.15.39:0/558525674 >> v2:172.21.15.100:3300/0 conn(0x7f1ba415a720 0x7f1ba40cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.025+0000 7f1ba8a00640 1 -- 172.21.15.39:0/558525674 >> v1:172.21.15.100:6789/0 conn(0x7f1ba415ab00 legacy=0x7f1ba40cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:34.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.025+0000 7f1ba8a00640 1 -- 172.21.15.39:0/558525674 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba40ce110 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.026+0000 7f1ba8a00640 1 --2- 172.21.15.39:0/558525674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba40ced20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ba406c3e0 tx=0x7f1b9c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b2c93f22aab74d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:34.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.026+0000 7f1ba3400640 1 -- 172.21.15.39:0/558525674 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c0317e0 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.026+0000 7f1ba3400640 1 -- 172.21.15.39:0/558525674 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9c031940 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.026+0000 7f1ba3400640 1 -- 172.21.15.39:0/558525674 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c031cd0 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.027+0000 7f1ba9400640 1 -- 172.21.15.39:0/558525674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 msgr2=0x7f1ba40ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.027+0000 7f1ba9400640 1 --2- 172.21.15.39:0/558525674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba40ced20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ba406c3e0 tx=0x7f1b9c02b880 comp rx=0 tx=0).stop 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.027+0000 7f1ba9400640 1 -- 172.21.15.39:0/558525674 shutdown_connections 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.027+0000 7f1ba9400640 1 --2- 172.21.15.39:0/558525674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba40ced20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.027+0000 7f1ba9400640 1 --2- 172.21.15.39:0/558525674 >> v2:172.21.15.100:3300/0 conn(0x7f1ba415a720 0x7f1ba40cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.027+0000 7f1ba9400640 1 -- 172.21.15.39:0/558525674 >> 172.21.15.39:0/558525674 conn(0x7f1ba406a090 msgr2=0x7f1ba406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.027+0000 7f1ba9400640 1 -- 172.21.15.39:0/558525674 shutdown_connections 2024-01-24T17:41:34.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.027+0000 7f1ba9400640 1 -- 172.21.15.39:0/558525674 wait complete. 2024-01-24T17:41:34.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.029+0000 7f1ba9400640 1 Processor -- start 2024-01-24T17:41:34.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.029+0000 7f1ba9400640 1 -- start start 2024-01-24T17:41:34.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba41f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba9400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ba415a720 0x7f1ba41f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba9400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ba415ab00 0x7f1ba415d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba9400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba40c8430 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba9400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba40c7ad0 con 0x7f1ba415a720 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba9400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba41f6400 con 0x7f1ba415ab00 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba41f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba41f5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58110/0 (socket says 172.21.15.39:58110) 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba3e00640 1 -- 172.21.15.39:0/2889992119 learned_addr learned my addr 172.21.15.39:0/2889992119 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba3e00640 1 -- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ba415ab00 msgr2=0x7f1ba415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba3e00640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ba415ab00 0x7f1ba415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba3e00640 1 -- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ba415a720 msgr2=0x7f1ba41f1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba3e00640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ba415a720 0x7f1ba41f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.030+0000 7f1ba3e00640 1 -- 172.21.15.39:0/2889992119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0094d0 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.031+0000 7f1ba3e00640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba41f5ec0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1b9400ce10 tx=0x7f1b9400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:34.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.031+0000 7f1ba0c00640 1 -- 172.21.15.39:0/2889992119 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94003b90 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.031+0000 7f1ba0c00640 1 -- 172.21.15.39:0/2889992119 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b94003cf0 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.031+0000 7f1ba0c00640 1 -- 172.21.15.39:0/2889992119 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94003ea0 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.031+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba40c7d70 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.031+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba40ca770 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.032+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba40c8570 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.033+0000 7f1ba0c00640 1 -- 172.21.15.39:0/2889992119 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1b94004240 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.034+0000 7f1ba0c00640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1b7c06a430 0x7f1b7c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.034+0000 7f1ba0c00640 1 -- 172.21.15.39:0/2889992119 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b94012070 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.035+0000 7f1b9b200640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1b7c06a430 0x7f1b7c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.036+0000 7f1b9b200640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1b7c06a430 0x7f1b7c06c8f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c006fd0 tx=0x7f1b8c006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:34.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.038+0000 7f1ba0c00640 1 -- 172.21.15.39:0/2889992119 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1ba40c8570 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.230+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ba40c8570 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.232+0000 7f1ba0c00640 1 -- 172.21.15.39:0/2889992119 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f1ba40c8570 con 0x7f1ba40ce8e0 2024-01-24T17:41:34.233 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:34.233 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":19,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:33.353970+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1b7c06a430 msgr2=0x7f1b7c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1b7c06a430 0x7f1b7c06c8f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c006fd0 tx=0x7f1b8c006bc0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 msgr2=0x7f1ba41f5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba41f5ec0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1b9400ce10 tx=0x7f1b9400a660 comp rx=0 tx=0).stop 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 shutdown_connections 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1ba415ab00 0x7f1ba415d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1ba415a720 0x7f1ba41f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1b7c06a430 0x7f1b7c06c8f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 --2- 172.21.15.39:0/2889992119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba40ce8e0 0x7f1ba41f5ec0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 >> 172.21.15.39:0/2889992119 conn(0x7f1ba406a090 msgr2=0x7f1ba40cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 shutdown_connections 2024-01-24T17:41:34.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.235+0000 7f1ba9400640 1 -- 172.21.15.39:0/2889992119 wait complete. 2024-01-24T17:41:34.252 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T17:41:34.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.469+0000 7f78a5000640 1 Processor -- start 2024-01-24T17:41:34.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.469+0000 7f78a5000640 1 -- start start 2024-01-24T17:41:34.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.470+0000 7f78a5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a00b0730 0x7f78a015da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.470+0000 7f78a5000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a015e450 con 0x7f78a00b0350 2024-01-24T17:41:34.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.470+0000 7f78a5000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a006d4c0 con 0x7f78a015dfc0 2024-01-24T17:41:34.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.470+0000 7f78a5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a006cf00 con 0x7f78a00b0730 2024-01-24T17:41:34.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f789de00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f78a00b0350 0x7f78a006c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52330/0 (socket says 172.21.15.39:52330) 2024-01-24T17:41:34.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f789de00640 1 -- 172.21.15.39:0/1063386850 learned_addr learned my addr 172.21.15.39:0/1063386850 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:34.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965906564 0 0) 0x7f7890001490 con 0x7f78a00b0350 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7880003520 con 0x7f78a00b0350 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f7895200640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a00b0730 0x7f78a015da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3015583715 0 0) 0x7f7880003520 con 0x7f78a00b0350 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a00b0730 msgr2=0x7f78a015da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f789d400640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a00b0730 0x7f78a015da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.471+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 >> v1:172.21.15.100:6789/0 conn(0x7f78a015dfc0 legacy=0x7f78a00b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a006d1e0 con 0x7f78a00b0350 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7890001b80 con 0x7f78a00b0350 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 >> v1:172.21.15.39:6789/0 conn(0x7f78a00b0350 legacy=0x7f78a006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78800039a0 0x7f7880003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a00b0730 0x7f78800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7880004d00 0x7f7880005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a006cf00 con 0x7f7880004d00 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788000bc30 con 0x7f78800039a0 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788000bdf0 con 0x7f78a00b0730 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789e800640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7880004d00 0x7f7880005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789de00640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78800039a0 0x7f7880003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789e800640 1 -- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a00b0730 msgr2=0x7f78800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f7895200640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a00b0730 0x7f78800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789e800640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a00b0730 0x7f78800047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789e800640 1 -- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78800039a0 msgr2=0x7f7880003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789e800640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78800039a0 0x7f7880003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789e800640 1 -- 172.21.15.39:0/1063386850 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7880003520 con 0x7f7880004d00 2024-01-24T17:41:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.472+0000 7f789de00640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78800039a0 0x7f7880003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.473+0000 7f789e800640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7880004d00 0x7f7880005180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7898001750 tx=0x7f78800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ad5ea8538ef096 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.474+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980052d0 con 0x7f7880004d00 2024-01-24T17:41:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.474+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789802e050 con 0x7f7880004d00 2024-01-24T17:41:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.474+0000 7f789d400640 1 -- 172.21.15.39:0/1063386850 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980035f0 con 0x7f7880004d00 2024-01-24T17:41:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.474+0000 7f78a5000640 1 -- 172.21.15.39:0/1063386850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7880004d00 msgr2=0x7f7880005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.474+0000 7f78a5000640 1 --2- 172.21.15.39:0/1063386850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7880004d00 0x7f7880005180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7898001750 tx=0x7f78800057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.474+0000 7f789e800640 1 -- 172.21.15.39:0/1063386850 reap_dead start 2024-01-24T17:41:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.474+0000 7f78a5000640 1 -- 172.21.15.39:0/1063386850 shutdown_connections 2024-01-24T17:41:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.474+0000 7f78a5000640 1 -- 172.21.15.39:0/1063386850 >> 172.21.15.39:0/1063386850 conn(0x7f78a00cb0e0 msgr2=0x7f78a00cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.475+0000 7f78a5000640 1 -- 172.21.15.39:0/1063386850 shutdown_connections 2024-01-24T17:41:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.475+0000 7f78a5000640 1 -- 172.21.15.39:0/1063386850 wait complete. 2024-01-24T17:41:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.476+0000 7f78a5000640 1 Processor -- start 2024-01-24T17:41:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.477+0000 7f78a5000640 1 -- start start 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.477+0000 7f78a5000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7880004d00 0x7f78a00c2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.477+0000 7f78a5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78a00c1ad0 0x7f78a01f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f78a5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a01f26b0 0x7f78a01f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f78a5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a01f30a0 con 0x7f78a00c1ad0 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f78a5000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00c2a30 con 0x7f7880004d00 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f78a5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00c2d10 con 0x7f78a01f26b0 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f7895200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78a00c1ad0 0x7f78a01f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f7895200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78a00c1ad0 0x7f78a01f2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58128/0 (socket says 172.21.15.39:58128) 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f7895200640 1 -- 172.21.15.39:0/4034218849 learned_addr learned my addr 172.21.15.39:0/4034218849 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f789de00640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7880004d00 0x7f78a00c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f789e800640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a01f26b0 0x7f78a01f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f7895200640 1 -- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a01f26b0 msgr2=0x7f78a01f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f7895200640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a01f26b0 0x7f78a01f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f7895200640 1 -- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7880004d00 msgr2=0x7f78a00c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f7895200640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7880004d00 0x7f78a00c2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.478+0000 7f7895200640 1 -- 172.21.15.39:0/4034218849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00c2ff0 con 0x7f78a00c1ad0 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.479+0000 7f7895200640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78a00c1ad0 0x7f78a01f2170 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f788800e7e0 tx=0x7f788800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.479+0000 7f7896a00640 1 -- 172.21.15.39:0/4034218849 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788800c6d0 con 0x7f78a00c1ad0 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.479+0000 7f7896a00640 1 -- 172.21.15.39:0/4034218849 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788800c830 con 0x7f78a00c1ad0 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.479+0000 7f7896a00640 1 -- 172.21.15.39:0/4034218849 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788800cba0 con 0x7f78a00c1ad0 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.479+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a00c32d0 con 0x7f78a00c1ad0 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.479+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a01592f0 con 0x7f78a00c1ad0 2024-01-24T17:41:34.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.480+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a0073ef0 con 0x7f78a00c1ad0 2024-01-24T17:41:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.481+0000 7f7896a00640 1 -- 172.21.15.39:0/4034218849 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f788800cd00 con 0x7f78a00c1ad0 2024-01-24T17:41:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.482+0000 7f7896a00640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787806a430 0x7f787806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.482+0000 7f7896a00640 1 -- 172.21.15.39:0/4034218849 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7888088730 con 0x7f78a00c1ad0 2024-01-24T17:41:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.482+0000 7f789de00640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787806a430 0x7f787806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.484+0000 7f789de00640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787806a430 0x7f787806c8f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f78900013b0 tx=0x7f7890009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.487+0000 7f7896a00640 1 -- 172.21.15.39:0/4034218849 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7888053d20 con 0x7f78a00c1ad0 2024-01-24T17:41:34.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:34.708+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f78a015d150 con 0x7f78a00c1ad0 2024-01-24T17:41:34.744 INFO:tasks.ceph.mds.b.smithi100.stderr:2024-01-24T17:41:34.743+0000 7ff3f2e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:41:35.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.373+0000 7f7896a00640 1 -- 172.21.15.39:0/4034218849 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7888057340 con 0x7f78a00c1ad0 2024-01-24T17:41:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787806a430 msgr2=0x7f787806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787806a430 0x7f787806c8f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f78900013b0 tx=0x7f7890009040 comp rx=0 tx=0).stop 2024-01-24T17:41:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78a00c1ad0 msgr2=0x7f78a01f2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78a00c1ad0 0x7f78a01f2170 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f788800e7e0 tx=0x7f788800eca0 comp rx=0 tx=0).stop 2024-01-24T17:41:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 shutdown_connections 2024-01-24T17:41:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f78a01f26b0 0x7f78a01f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:35.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787806a430 0x7f787806c8f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:35.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78a00c1ad0 0x7f78a01f2170 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:35.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 --2- 172.21.15.39:0/4034218849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7880004d00 0x7f78a00c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:35.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 >> 172.21.15.39:0/4034218849 conn(0x7f78a00cb0e0 msgr2=0x7f78a006ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:35.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 shutdown_connections 2024-01-24T17:41:35.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.376+0000 7f78a5000640 1 -- 172.21.15.39:0/4034218849 wait complete. 2024-01-24T17:41:35.397 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-24T17:41:35.397 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-24T17:41:35.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.621+0000 7f4521800640 1 Processor -- start 2024-01-24T17:41:35.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.621+0000 7f4521800640 1 -- start start 2024-01-24T17:41:35.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.622+0000 7f4521800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c0b00e0 0x7f451c0b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:35.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.622+0000 7f4521800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c159070 con 0x7f451c158c10 2024-01-24T17:41:35.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.623+0000 7f4521800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0cd5c0 con 0x7f451c06e100 2024-01-24T17:41:35.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.623+0000 7f4521800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451c0b0ab0 con 0x7f451c0b00e0 2024-01-24T17:41:35.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.623+0000 7f4511e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f451c158c10 0x7f451c06c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52334/0 (socket says 172.21.15.39:52334) 2024-01-24T17:41:35.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.623+0000 7f4511e00640 1 -- 172.21.15.39:0/3176439246 learned_addr learned my addr 172.21.15.39:0/3176439246 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:35.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.623+0000 7f451b400640 1 --2- 172.21.15.39:0/3176439246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c0b00e0 0x7f451c0b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:35.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.623+0000 7f451a000640 1 -- 172.21.15.39:0/3176439246 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948599258 0 0) 0x7f451c159070 con 0x7f451c158c10 2024-01-24T17:41:35.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.624+0000 7f451a000640 1 -- 172.21.15.39:0/3176439246 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc003520 con 0x7f451c158c10 2024-01-24T17:41:35.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.624+0000 7f451a000640 1 -- 172.21.15.39:0/3176439246 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085447388 0 0) 0x7f451c0cd5c0 con 0x7f451c06e100 2024-01-24T17:41:35.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.624+0000 7f451b400640 1 -- 172.21.15.39:0/3176439246 >> v1:172.21.15.100:6789/0 conn(0x7f451c06e100 legacy=0x7f451c0c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:35.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.624+0000 7f451b400640 1 -- 172.21.15.39:0/3176439246 >> v1:172.21.15.39:6789/0 conn(0x7f451c158c10 legacy=0x7f451c06c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:35.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.624+0000 7f451b400640 1 -- 172.21.15.39:0/3176439246 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0cd2e0 con 0x7f451c0b00e0 2024-01-24T17:41:35.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.625+0000 7f451b400640 1 --2- 172.21.15.39:0/3176439246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c0b00e0 0x7f451c0b0570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f45140097d0 tx=0x7f451402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6433a98166d8015 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:35.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.625+0000 7f451a000640 1 -- 172.21.15.39:0/3176439246 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514030800 con 0x7f451c0b00e0 2024-01-24T17:41:35.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.626+0000 7f451a000640 1 -- 172.21.15.39:0/3176439246 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4514030960 con 0x7f451c0b00e0 2024-01-24T17:41:35.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.626+0000 7f451a000640 1 -- 172.21.15.39:0/3176439246 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514030cd0 con 0x7f451c0b00e0 2024-01-24T17:41:35.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.626+0000 7f4521800640 1 -- 172.21.15.39:0/3176439246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c0b00e0 msgr2=0x7f451c0b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:35.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.626+0000 7f4521800640 1 --2- 172.21.15.39:0/3176439246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c0b00e0 0x7f451c0b0570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f45140097d0 tx=0x7f451402b940 comp rx=0 tx=0).stop 2024-01-24T17:41:35.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.627+0000 7f4521800640 1 -- 172.21.15.39:0/3176439246 shutdown_connections 2024-01-24T17:41:35.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.627+0000 7f4521800640 1 --2- 172.21.15.39:0/3176439246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c0b00e0 0x7f451c0b0570 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:35.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.627+0000 7f4521800640 1 -- 172.21.15.39:0/3176439246 >> 172.21.15.39:0/3176439246 conn(0x7f451c0cb0e0 msgr2=0x7f451c0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:35.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.627+0000 7f4521800640 1 -- 172.21.15.39:0/3176439246 shutdown_connections 2024-01-24T17:41:35.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.627+0000 7f4521800640 1 -- 172.21.15.39:0/3176439246 wait complete. 2024-01-24T17:41:35.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.629+0000 7f4521800640 1 Processor -- start 2024-01-24T17:41:35.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.629+0000 7f4521800640 1 -- start start 2024-01-24T17:41:35.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4521800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c06e100 0x7f451c0c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:35.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4521800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f451c0b00e0 0x7f451c0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:35.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4521800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f451c158c10 0x7f451c0c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:35.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4521800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c0c8620 con 0x7f451c0b00e0 2024-01-24T17:41:35.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4521800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c0c6a20 con 0x7f451c158c10 2024-01-24T17:41:35.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4521800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451c0c1530 con 0x7f451c06e100 2024-01-24T17:41:35.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4511e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f451c0b00e0 0x7f451c0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:35.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4511e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f451c0b00e0 0x7f451c0c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58136/0 (socket says 172.21.15.39:58136) 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4511e00640 1 -- 172.21.15.39:0/2948809604 learned_addr learned my addr 172.21.15.39:0/2948809604 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f451aa00640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c06e100 0x7f451c0c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f451b400640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f451c158c10 0x7f451c0c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4511e00640 1 -- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c06e100 msgr2=0x7f451c0c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.630+0000 7f4511e00640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c06e100 0x7f451c0c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4511e00640 1 -- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f451c158c10 msgr2=0x7f451c0c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4511e00640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f451c158c10 0x7f451c0c0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4511e00640 1 -- 172.21.15.39:0/2948809604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4514009400 con 0x7f451c0b00e0 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4511e00640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f451c0b00e0 0x7f451c0c80e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f450400c390 tx=0x7f450400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4513400640 1 -- 172.21.15.39:0/2948809604 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4504005ce0 con 0x7f451c0b00e0 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4513400640 1 -- 172.21.15.39:0/2948809604 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4504005e40 con 0x7f451c0b00e0 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451c0c5e10 con 0x7f451c0b00e0 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4513400640 1 -- 172.21.15.39:0/2948809604 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45040072f0 con 0x7f451c0b00e0 2024-01-24T17:41:35.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.631+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c0c63d0 con 0x7f451c0b00e0 2024-01-24T17:41:35.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.632+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451c0c6b90 con 0x7f451c0b00e0 2024-01-24T17:41:35.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.633+0000 7f4513400640 1 -- 172.21.15.39:0/2948809604 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4504007450 con 0x7f451c0b00e0 2024-01-24T17:41:35.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.634+0000 7f4513400640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44f406a430 0x7f44f406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:35.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.634+0000 7f4513400640 1 -- 172.21.15.39:0/2948809604 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4504087f30 con 0x7f451c0b00e0 2024-01-24T17:41:35.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.635+0000 7f451aa00640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44f406a430 0x7f44f406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:35.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.636+0000 7f451aa00640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44f406a430 0x7f44f406c8f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f450c007fd0 tx=0x7f450c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:35.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.639+0000 7f4513400640 1 -- 172.21.15.39:0/2948809604 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4504053520 con 0x7f451c0b00e0 2024-01-24T17:41:35.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:35.869+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f451c073ef0 con 0x7f451c0b00e0 2024-01-24T17:41:36.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.376+0000 7f4513400640 1 -- 172.21.15.39:0/2948809604 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f45040170c0 con 0x7f451c0b00e0 2024-01-24T17:41:36.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44f406a430 msgr2=0x7f44f406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:36.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44f406a430 0x7f44f406c8f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f450c007fd0 tx=0x7f450c009040 comp rx=0 tx=0).stop 2024-01-24T17:41:36.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f451c0b00e0 msgr2=0x7f451c0c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:36.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f451c0b00e0 0x7f451c0c80e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f450400c390 tx=0x7f450400c850 comp rx=0 tx=0).stop 2024-01-24T17:41:36.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 shutdown_connections 2024-01-24T17:41:36.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f451c158c10 0x7f451c0c0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44f406a430 0x7f44f406c8f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f451c0b00e0 0x7f451c0c80e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 --2- 172.21.15.39:0/2948809604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f451c06e100 0x7f451c0c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.380+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 >> 172.21.15.39:0/2948809604 conn(0x7f451c0cb0e0 msgr2=0x7f451c0adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:36.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.381+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 shutdown_connections 2024-01-24T17:41:36.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.381+0000 7f4521800640 1 -- 172.21.15.39:0/2948809604 wait complete. 2024-01-24T17:41:36.400 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-24T17:41:36.400 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:41:36.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.621+0000 7fd73b200640 1 Processor -- start 2024-01-24T17:41:36.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.622+0000 7fd73b200640 1 -- start start 2024-01-24T17:41:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.623+0000 7fd73b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd73c15cbf0 0x7fd73c0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.623+0000 7fd73b200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd73c15db60 0x7fd73c0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.623+0000 7fd73b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c0d0da0 0x7fd73c0d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:36.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.623+0000 7fd73b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c0d1770 con 0x7fd73c15cbf0 2024-01-24T17:41:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.623+0000 7fd73b200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73c15e010 con 0x7fd73c15db60 2024-01-24T17:41:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.623+0000 7fd73b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0d0a90 con 0x7fd73c0d0da0 2024-01-24T17:41:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd73c15cbf0 0x7fd73c0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd73c15cbf0 0x7fd73c0d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58142/0 (socket says 172.21.15.39:58142) 2024-01-24T17:41:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739e00640 1 -- 172.21.15.39:0/2264791295 learned_addr learned my addr 172.21.15.39:0/2264791295 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd73a800640 1 --2- 172.21.15.39:0/2264791295 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c0d0da0 0x7fd73c0d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739400640 1 --2- 172.21.15.39:0/2264791295 >> v2:172.21.15.100:3300/0 conn(0x7fd73c15db60 0x7fd73c0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:36.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739e00640 1 -- 172.21.15.39:0/2264791295 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c0d0da0 msgr2=0x7fd73c0d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739e00640 1 --2- 172.21.15.39:0/2264791295 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c0d0da0 0x7fd73c0d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739e00640 1 -- 172.21.15.39:0/2264791295 >> v2:172.21.15.100:3300/0 conn(0x7fd73c15db60 msgr2=0x7fd73c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739e00640 1 --2- 172.21.15.39:0/2264791295 >> v2:172.21.15.100:3300/0 conn(0x7fd73c15db60 0x7fd73c0ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.624+0000 7fd739e00640 1 -- 172.21.15.39:0/2264791295 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c0cd760 con 0x7fd73c15cbf0 2024-01-24T17:41:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.626+0000 7fd739e00640 1 --2- 172.21.15.39:0/2264791295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c15cbf0 0x7fd73c0d03b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd724009860 tx=0x7fd72402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86a37798919ef90 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:36.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.626+0000 7fd738a00640 1 -- 172.21.15.39:0/2264791295 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724031640 con 0x7fd73c15cbf0 2024-01-24T17:41:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.626+0000 7fd738a00640 1 -- 172.21.15.39:0/2264791295 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7240317a0 con 0x7fd73c15cbf0 2024-01-24T17:41:36.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.626+0000 7fd738a00640 1 -- 172.21.15.39:0/2264791295 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724031b30 con 0x7fd73c15cbf0 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 -- 172.21.15.39:0/2264791295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c15cbf0 msgr2=0x7fd73c0d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 --2- 172.21.15.39:0/2264791295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c15cbf0 0x7fd73c0d03b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd724009860 tx=0x7fd72402b8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 -- 172.21.15.39:0/2264791295 shutdown_connections 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 --2- 172.21.15.39:0/2264791295 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c0d0da0 0x7fd73c0d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 --2- 172.21.15.39:0/2264791295 >> v2:172.21.15.100:3300/0 conn(0x7fd73c15db60 0x7fd73c0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 --2- 172.21.15.39:0/2264791295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c15cbf0 0x7fd73c0d03b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 -- 172.21.15.39:0/2264791295 >> 172.21.15.39:0/2264791295 conn(0x7fd73c09bb90 msgr2=0x7fd73c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 -- 172.21.15.39:0/2264791295 shutdown_connections 2024-01-24T17:41:36.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.627+0000 7fd73b200640 1 -- 172.21.15.39:0/2264791295 wait complete. 2024-01-24T17:41:36.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.629+0000 7fd73b200640 1 Processor -- start 2024-01-24T17:41:36.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.629+0000 7fd73b200640 1 -- start start 2024-01-24T17:41:36.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd73b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c0d0da0 0x7fd73c1f6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:36.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd73b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd73c15cbf0 0x7fd73c0cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:36.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd73b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c15db60 0x7fd73c1f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:36.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd73b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c1f5d30 con 0x7fd73c0d0da0 2024-01-24T17:41:36.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd73b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c1f56e0 con 0x7fd73c15cbf0 2024-01-24T17:41:36.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd73b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c0c9c50 con 0x7fd73c15db60 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd739e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c0d0da0 0x7fd73c1f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd739e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c0d0da0 0x7fd73c1f6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58152/0 (socket says 172.21.15.39:58152) 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.630+0000 7fd739e00640 1 -- 172.21.15.39:0/4071984588 learned_addr learned my addr 172.21.15.39:0/4071984588 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd739400640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd73c15cbf0 0x7fd73c0cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd73a800640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c15db60 0x7fd73c1f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd739e00640 1 -- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c15db60 msgr2=0x7fd73c1f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd739e00640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c15db60 0x7fd73c1f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd739e00640 1 -- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd73c15cbf0 msgr2=0x7fd73c0cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd739e00640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd73c15cbf0 0x7fd73c0cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd739e00640 1 -- 172.21.15.39:0/4071984588 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd724009470 con 0x7fd73c0d0da0 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd73a800640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c15db60 0x7fd73c1f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd739400640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd73c15cbf0 0x7fd73c0cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd739e00640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c0d0da0 0x7fd73c1f6150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd724000c00 tx=0x7fd7240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:36.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd732000640 1 -- 172.21.15.39:0/4071984588 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72403ac60 con 0x7fd73c0d0da0 2024-01-24T17:41:36.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd732000640 1 -- 172.21.15.39:0/4071984588 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72403ade0 con 0x7fd73c0d0da0 2024-01-24T17:41:36.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd732000640 1 -- 172.21.15.39:0/4071984588 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724044630 con 0x7fd73c0d0da0 2024-01-24T17:41:36.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.631+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c1f5950 con 0x7fd73c0d0da0 2024-01-24T17:41:36.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.632+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c1f1ed0 con 0x7fd73c0d0da0 2024-01-24T17:41:36.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.633+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73c04dff0 con 0x7fd73c0d0da0 2024-01-24T17:41:36.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.633+0000 7fd732000640 1 -- 172.21.15.39:0/4071984588 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd72402f050 con 0x7fd73c0d0da0 2024-01-24T17:41:36.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.635+0000 7fd732000640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd71406a430 0x7fd71406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:36.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.635+0000 7fd732000640 1 -- 172.21.15.39:0/4071984588 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7240aa880 con 0x7fd73c0d0da0 2024-01-24T17:41:36.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.635+0000 7fd739400640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd71406a430 0x7fd71406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:36.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.636+0000 7fd739400640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd71406a430 0x7fd71406c8f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd73c1594c0 tx=0x7fd72c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:36.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.639+0000 7fd732000640 1 -- 172.21.15.39:0/4071984588 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd724075dd0 con 0x7fd73c0d0da0 2024-01-24T17:41:36.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.873+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd73c0b4f60 con 0x7fd73c0d0da0 2024-01-24T17:41:36.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.874+0000 7fd732000640 1 -- 172.21.15.39:0/4071984588 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7fd7240793f0 con 0x7fd73c0d0da0 2024-01-24T17:41:36.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:36.875 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:41:33.354808+0000","modified":"2024-01-24T17:41:36.370204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-24T17:41:36.875 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-01-24T17:41:36.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd71406a430 msgr2=0x7fd71406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:36.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd71406a430 0x7fd71406c8f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd73c1594c0 tx=0x7fd72c00b040 comp rx=0 tx=0).stop 2024-01-24T17:41:36.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c0d0da0 msgr2=0x7fd73c1f6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:36.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c0d0da0 0x7fd73c1f6150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd724000c00 tx=0x7fd7240046c0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 shutdown_connections 2024-01-24T17:41:36.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd73c15db60 0x7fd73c1f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd73c15cbf0 0x7fd73c0cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd71406a430 0x7fd71406c8f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 --2- 172.21.15.39:0/4071984588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd73c0d0da0 0x7fd73c1f6150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:36.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.877+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 >> 172.21.15.39:0/4071984588 conn(0x7fd73c09bb90 msgr2=0x7fd73c15e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:36.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.878+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 shutdown_connections 2024-01-24T17:41:36.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:36.878+0000 7fd73b200640 1 -- 172.21.15.39:0/4071984588 wait complete. 2024-01-24T17:41:36.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:37.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.113+0000 7ff3e8400640 1 Processor -- start 2024-01-24T17:41:37.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.114+0000 7ff3e8400640 1 -- start start 2024-01-24T17:41:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.115+0000 7ff3e8400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff3e015a950 0x7ff3e00ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.115+0000 7ff3e8400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 0x7ff3e00b1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.115+0000 7ff3e8400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e00c9180 con 0x7ff3e00caaa0 2024-01-24T17:41:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.115+0000 7ff3e8400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e00b43d0 con 0x7ff3e015a950 2024-01-24T17:41:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.115+0000 7ff3e8400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e00c8f70 con 0x7ff3e00cb660 2024-01-24T17:41:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.115+0000 7ff3e5000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff3e00caaa0 0x7ff3e00caf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52344/0 (socket says 172.21.15.39:52344) 2024-01-24T17:41:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3e5000640 1 -- 172.21.15.39:0/1286589616 learned_addr learned my addr 172.21.15.39:0/1286589616 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3e5a00640 1 --2- 172.21.15.39:0/1286589616 >> v2:172.21.15.100:3300/0 conn(0x7ff3e015a950 0x7ff3e00ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3e6400640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 0x7ff3e00b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717465835 0 0) 0x7ff3e00c9180 con 0x7ff3e00caaa0 2024-01-24T17:41:37.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c0003520 con 0x7ff3e00caaa0 2024-01-24T17:41:37.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1935848371 0 0) 0x7ff3c0003520 con 0x7ff3e00caaa0 2024-01-24T17:41:37.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 msgr2=0x7ff3e00b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:37.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 0x7ff3e00b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 >> v2:172.21.15.100:3300/0 conn(0x7ff3e015a950 msgr2=0x7ff3e00ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 --2- 172.21.15.39:0/1286589616 >> v2:172.21.15.100:3300/0 conn(0x7ff3e015a950 0x7ff3e00ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e00c9460 con 0x7ff3e00caaa0 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.116+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff3c8001690 con 0x7ff3e00caaa0 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 >> v1:172.21.15.39:6789/0 conn(0x7ff3e00caaa0 legacy=0x7ff3e00caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3dfe00640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 0x7ff3c0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3dfe00640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3c0003ef0 0x7ff3c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3dfe00640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e00b43d0 con 0x7ff3c0004d00 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e00c8f70 con 0x7ff3c0003ef0 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c000b980 con 0x7ff3e00cb660 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e6400640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e5a00640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 0x7ff3c0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e5000640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3c0003ef0 0x7ff3c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e6400640 1 -- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 msgr2=0x7ff3c0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e6400640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 0x7ff3c0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e6400640 1 -- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3c0003ef0 msgr2=0x7ff3c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e6400640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3c0003ef0 0x7ff3c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e6400640 1 -- 172.21.15.39:0/1286589616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0003520 con 0x7ff3c0004d00 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.117+0000 7ff3e5a00640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e00cb660 0x7ff3c0003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.118+0000 7ff3e6400640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3c0005180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009fc0 tx=0x7ff3c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a88b8e6e239f1a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.119+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d802cc00 con 0x7ff3c0004d00 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.119+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d800f040 con 0x7ff3c0004d00 2024-01-24T17:41:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.119+0000 7ff3dfe00640 1 -- 172.21.15.39:0/1286589616 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8034440 con 0x7ff3c0004d00 2024-01-24T17:41:37.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.119+0000 7ff3e8400640 1 -- 172.21.15.39:0/1286589616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 msgr2=0x7ff3c0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.119+0000 7ff3e8400640 1 --2- 172.21.15.39:0/1286589616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3c0005180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009fc0 tx=0x7ff3c00057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.119+0000 7ff3e6400640 1 -- 172.21.15.39:0/1286589616 reap_dead start 2024-01-24T17:41:37.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.119+0000 7ff3e8400640 1 -- 172.21.15.39:0/1286589616 shutdown_connections 2024-01-24T17:41:37.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.119+0000 7ff3e8400640 1 -- 172.21.15.39:0/1286589616 >> 172.21.15.39:0/1286589616 conn(0x7ff3e006a3e0 msgr2=0x7ff3e006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:37.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.120+0000 7ff3e8400640 1 -- 172.21.15.39:0/1286589616 shutdown_connections 2024-01-24T17:41:37.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.120+0000 7ff3e8400640 1 -- 172.21.15.39:0/1286589616 wait complete. 2024-01-24T17:41:37.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.121+0000 7ff3e8400640 1 Processor -- start 2024-01-24T17:41:37.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.122+0000 7ff3e8400640 1 -- start start 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.122+0000 7ff3e8400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3e014d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.122+0000 7ff3e8400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3e00c8570 0x7ff3e00c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.122+0000 7ff3e8400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e015daa0 0x7ff3e0157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.122+0000 7ff3e8400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e015cd70 con 0x7ff3c0004d00 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.122+0000 7ff3e8400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e014dde0 con 0x7ff3e00c8570 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.122+0000 7ff3e8400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e00c6d20 con 0x7ff3e015daa0 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3e014d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3e014d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58172/0 (socket says 172.21.15.39:58172) 2024-01-24T17:41:37.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5a00640 1 -- 172.21.15.39:0/1186350435 learned_addr learned my addr 172.21.15.39:0/1186350435 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e6400640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e015daa0 0x7ff3e0157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5000640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3e00c8570 0x7ff3e00c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5a00640 1 -- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e015daa0 msgr2=0x7ff3e0157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5a00640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e015daa0 0x7ff3e0157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5a00640 1 -- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3e00c8570 msgr2=0x7ff3e00c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5a00640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3e00c8570 0x7ff3e00c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.123+0000 7ff3e5a00640 1 -- 172.21.15.39:0/1186350435 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e015cfc0 con 0x7ff3c0004d00 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.124+0000 7ff3e5a00640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3e014d8a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00109d0 tx=0x7ff3d0010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.124+0000 7ff3dd600640 1 -- 172.21.15.39:0/1186350435 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d000ec20 con 0x7ff3c0004d00 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.124+0000 7ff3dd600640 1 -- 172.21.15.39:0/1186350435 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d000ed80 con 0x7ff3c0004d00 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.124+0000 7ff3dd600640 1 -- 172.21.15.39:0/1186350435 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d001c620 con 0x7ff3c0004d00 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.124+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e015d2a0 con 0x7ff3c0004d00 2024-01-24T17:41:37.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.124+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e00c68e0 con 0x7ff3c0004d00 2024-01-24T17:41:37.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.125+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e0073ef0 con 0x7ff3c0004d00 2024-01-24T17:41:37.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.126+0000 7ff3dd600640 1 -- 172.21.15.39:0/1186350435 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff3d001c780 con 0x7ff3c0004d00 2024-01-24T17:41:37.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.127+0000 7ff3dd600640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3b806a480 0x7ff3b806c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.127+0000 7ff3dd600640 1 -- 172.21.15.39:0/1186350435 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3d0088f60 con 0x7ff3c0004d00 2024-01-24T17:41:37.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.127+0000 7ff3e5000640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3b806a480 0x7ff3b806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.128+0000 7ff3e5000640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3b806a480 0x7ff3b806c940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8001eb0 tx=0x7ff3c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:37.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.131+0000 7ff3dd600640 1 -- 172.21.15.39:0/1186350435 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff3d00bbcb0 con 0x7ff3c0004d00 2024-01-24T17:41:37.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.328+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3e0159060 con 0x7ff3c0004d00 2024-01-24T17:41:37.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.329+0000 7ff3dd600640 1 -- 172.21.15.39:0/1186350435 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7ff3d0054650 con 0x7ff3c0004d00 2024-01-24T17:41:37.330 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:37.331 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.332+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3b806a480 msgr2=0x7ff3b806c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.332+0000 7ff3e8400640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3b806a480 0x7ff3b806c940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8001eb0 tx=0x7ff3c8009040 comp rx=0 tx=0).stop 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.332+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 msgr2=0x7ff3e014d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.332+0000 7ff3e8400640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3e014d8a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00109d0 tx=0x7ff3d0010e90 comp rx=0 tx=0).stop 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.333+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 shutdown_connections 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.333+0000 7ff3e8400640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3e015daa0 0x7ff3e0157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.333+0000 7ff3e8400640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3e00c8570 0x7ff3e00c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.333+0000 7ff3e8400640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3b806a480 0x7ff3b806c940 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.333+0000 7ff3e8400640 1 --2- 172.21.15.39:0/1186350435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3c0004d00 0x7ff3e014d8a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.333+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 >> 172.21.15.39:0/1186350435 conn(0x7ff3e006a3e0 msgr2=0x7ff3e006fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.333+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 shutdown_connections 2024-01-24T17:41:37.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.333+0000 7ff3e8400640 1 -- 172.21.15.39:0/1186350435 wait complete. 2024-01-24T17:41:37.353 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-24T17:41:37.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:41:37.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.573+0000 7f5cdbe00640 1 Processor -- start 2024-01-24T17:41:37.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.573+0000 7f5cdbe00640 1 -- start start 2024-01-24T17:41:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.575+0000 7f5cdbe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5cdc0cdd70 0x7f5cdc06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.575+0000 7f5cdbe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0cb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.575+0000 7f5cdbe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cdc0cd050 con 0x7f5cdc0ca670 2024-01-24T17:41:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.575+0000 7f5cdbe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc0cbc40 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.575+0000 7f5cdbe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc159fe0 con 0x7f5cdc0cb2b0 2024-01-24T17:41:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cdb400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cd1e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5cdc0cdd70 0x7f5cdc06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cdaa00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f5cdc0ca670 0x7f5cdc15a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48622/0 (socket says 172.21.15.39:48622) 2024-01-24T17:41:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cd1e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5cdc0cdd70 0x7f5cdc06bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49118/0 (socket says 172.21.15.39:49118) 2024-01-24T17:41:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cdaa00640 1 -- 172.21.15.39:0/3321468196 learned_addr learned my addr 172.21.15.39:0/3321468196 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cdb400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:47910/0 (socket says 172.21.15.39:47910) 2024-01-24T17:41:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cd1e00640 1 -- 172.21.15.39:0/3321468196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 msgr2=0x7f5cdc0cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cd1e00640 1 --2- 172.21.15.39:0/3321468196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0cb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cd1e00640 1 -- 172.21.15.39:0/3321468196 >> v1:172.21.15.100:6789/0 conn(0x7f5cdc0ca670 legacy=0x7f5cdc15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cd1e00640 1 -- 172.21.15.39:0/3321468196 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc0caa50 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.576+0000 7f5cdb400640 1 --2- 172.21.15.39:0/3321468196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.578+0000 7f5cd1e00640 1 --2- 172.21.15.39:0/3321468196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc06bde0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40097a0 tx=0x7f5cc402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95ed5b1acc3d8c23 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.578+0000 7f5cda000640 1 -- 172.21.15.39:0/3321468196 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4031780 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.578+0000 7f5cda000640 1 -- 172.21.15.39:0/3321468196 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc40318e0 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.578+0000 7f5cda000640 1 -- 172.21.15.39:0/3321468196 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4031c70 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.579+0000 7f5cdbe00640 1 -- 172.21.15.39:0/3321468196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0cdd70 msgr2=0x7f5cdc06bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.579+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/3321468196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc06bde0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40097a0 tx=0x7f5cc402b820 comp rx=0 tx=0).stop 2024-01-24T17:41:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.579+0000 7f5cdbe00640 1 -- 172.21.15.39:0/3321468196 shutdown_connections 2024-01-24T17:41:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.579+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/3321468196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc06bde0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.579+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/3321468196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.579+0000 7f5cdbe00640 1 -- 172.21.15.39:0/3321468196 >> 172.21.15.39:0/3321468196 conn(0x7f5cdc06a090 msgr2=0x7f5cdc06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.579+0000 7f5cdbe00640 1 -- 172.21.15.39:0/3321468196 shutdown_connections 2024-01-24T17:41:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.579+0000 7f5cdbe00640 1 -- 172.21.15.39:0/3321468196 wait complete. 2024-01-24T17:41:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.581+0000 7f5cdbe00640 1 Processor -- start 2024-01-24T17:41:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.581+0000 7f5cdbe00640 1 -- start start 2024-01-24T17:41:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdbe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0ca670 0x7f5cdc1f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdbe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc1f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc156f10 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdbe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc1f65f0 con 0x7f5cdc0ca670 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdbe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc157580 con 0x7f5cdc0cb2b0 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc1f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc1f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58184/0 (socket says 172.21.15.39:58184) 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.582+0000 7f5cdb400640 1 -- 172.21.15.39:0/1906804700 learned_addr learned my addr 172.21.15.39:0/1906804700 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cdaa00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0ca670 0x7f5cdc1f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cd1e00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cdb400640 1 -- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 msgr2=0x7f5cdc0ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cdb400640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cdb400640 1 -- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0ca670 msgr2=0x7f5cdc1f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cdb400640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0ca670 0x7f5cdc1f60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cdb400640 1 -- 172.21.15.39:0/1906804700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4009400 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cdb400640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc1f3270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400e880 tx=0x7f5cd400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cd3400640 1 -- 172.21.15.39:0/1906804700 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4009660 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cd3400640 1 -- 172.21.15.39:0/1906804700 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd4004020 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cd3400640 1 -- 172.21.15.39:0/1906804700 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd40043b0 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.583+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc4009400 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.584+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc157e20 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.585+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc04dff0 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.586+0000 7f5cd3400640 1 -- 172.21.15.39:0/1906804700 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5cd400c940 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.587+0000 7f5cd3400640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cb406a430 0x7f5cb406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:37.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.587+0000 7f5cd3400640 1 -- 172.21.15.39:0/1906804700 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cd40888d0 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.587+0000 7f5cdaa00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cb406a430 0x7f5cb406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.588+0000 7f5cdaa00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cb406a430 0x7f5cb406c8f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc005c00 tx=0x7f5ccc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:37.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.591+0000 7f5cd3400640 1 -- 172.21.15.39:0/1906804700 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5cd4053e20 con 0x7f5cdc0cdd70 2024-01-24T17:41:37.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.786+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5cdc15ab80 con 0x7f5cb406a430 2024-01-24T17:41:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.810+0000 7f5cd3400640 1 -- 172.21.15.39:0/1906804700 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+253729 (secure 0 0 0) 0x7f5ccc001950 con 0x7f5cb406a430 2024-01-24T17:41:37.813 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:37.816 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cb406a430 msgr2=0x7f5cb406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cb406a430 0x7f5cb406c8f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc005c00 tx=0x7f5ccc009040 comp rx=0 tx=0).stop 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cdc0cdd70 msgr2=0x7f5cdc1f3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc1f3270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400e880 tx=0x7f5cd400ed40 comp rx=0 tx=0).stop 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 shutdown_connections 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cb406a430 0x7f5cb406c8f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5cdc0cdd70 0x7f5cdc1f3270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5cdc0cb2b0 0x7f5cdc0ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 --2- 172.21.15.39:0/1906804700 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5cdc0ca670 0x7f5cdc1f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:37.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 >> 172.21.15.39:0/1906804700 conn(0x7f5cdc06a090 msgr2=0x7f5cdc0c98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:37.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 shutdown_connections 2024-01-24T17:41:37.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:37.817+0000 7f5cdbe00640 1 -- 172.21.15.39:0/1906804700 wait complete. 2024-01-24T17:41:37.836 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-01-24T17:41:36.208945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235032,"kb_used_data":2472,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754739688,"statfs":{"total":773094113280,"available":772853440512,"internally_reserved":0,"allocated":2531328,"data_stored":1593120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.358378"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.865660+0000","last_change":"2024-01-24T17:41:31.865660+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.865660+0000","last_undegraded":"2024-01-24T17:41:31.865660+0000","last_fullsized":"2024-01-24T17:41:31.865660+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.872070+0000","last_change":"2024-01-24T17:41:31.872070+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.872070+0000","last_undegraded":"2024-01-24T17:41:31.872070+0000","last_fullsized":"2024-01-24T17:41:31.872070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:56.825045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.855999+0000","last_change":"2024-01-24T17:41:31.855999+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.855999+0000","last_undegraded":"2024-01-24T17:41:31.855999+0000","last_fullsized":"2024-01-24T17:41:31.855999+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:01:37.962464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.867669+0000","last_change":"2024-01-24T17:41:31.867669+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.867669+0000","last_undegraded":"2024-01-24T17:41:31.867669+0000","last_fullsized":"2024-01-24T17:41:31.867669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.856759+0000","last_change":"2024-01-24T17:41:31.856759+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.856759+0000","last_undegraded":"2024-01-24T17:41:31.856759+0000","last_fullsized":"2024-01-24T17:41:31.856759+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:01:37.962464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.860548+0000","last_change":"2024-01-24T17:41:31.860548+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.860548+0000","last_undegraded":"2024-01-24T17:41:31.860548+0000","last_fullsized":"2024-01-24T17:41:31.860548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.860801+0000","last_change":"2024-01-24T17:41:31.860801+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.860801+0000","last_undegraded":"2024-01-24T17:41:31.860801+0000","last_fullsized":"2024-01-24T17:41:31.860801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.875662+0000","last_change":"2024-01-24T17:41:31.875662+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.875662+0000","last_undegraded":"2024-01-24T17:41:31.875662+0000","last_fullsized":"2024-01-24T17:41:31.875662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:06:04.595152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.860786+0000","last_change":"2024-01-24T17:41:31.860786+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.860786+0000","last_undegraded":"2024-01-24T17:41:31.860786+0000","last_fullsized":"2024-01-24T17:41:31.860786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.870923+0000","last_change":"2024-01-24T17:41:31.870923+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.870923+0000","last_undegraded":"2024-01-24T17:41:31.870923+0000","last_fullsized":"2024-01-24T17:41:31.870923+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:46.796323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.860311+0000","last_change":"2024-01-24T17:41:31.860311+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.860311+0000","last_undegraded":"2024-01-24T17:41:31.860311+0000","last_fullsized":"2024-01-24T17:41:31.860311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.868097+0000","last_change":"2024-01-24T17:41:31.868097+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.868097+0000","last_undegraded":"2024-01-24T17:41:31.868097+0000","last_fullsized":"2024-01-24T17:41:31.868097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:45.778394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.862779+0000","last_change":"2024-01-24T17:41:31.862779+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.862779+0000","last_undegraded":"2024-01-24T17:41:31.862779+0000","last_fullsized":"2024-01-24T17:41:31.862779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.869834+0000","last_change":"2024-01-24T17:41:31.869834+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.869834+0000","last_undegraded":"2024-01-24T17:41:31.869834+0000","last_fullsized":"2024-01-24T17:41:31.869834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:53.093743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.866458+0000","last_change":"2024-01-24T17:41:31.866458+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.866458+0000","last_undegraded":"2024-01-24T17:41:31.866458+0000","last_fullsized":"2024-01-24T17:41:31.866458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.856235+0000","last_change":"2024-01-24T17:41:31.856235+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.856235+0000","last_undegraded":"2024-01-24T17:41:31.856235+0000","last_fullsized":"2024-01-24T17:41:31.856235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:20.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":161,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-01-24T17:41:27.282054+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:41:27.282054+0000","last_peered":"2024-01-24T17:41:27.282054+0000","last_clean":"2024-01-24T17:41:27.282054+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:41:27.282054+0000","last_undegraded":"2024-01-24T17:41:27.282054+0000","last_fullsized":"2024-01-24T17:41:27.282054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.866554+0000","last_change":"2024-01-24T17:41:31.866554+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.866554+0000","last_undegraded":"2024-01-24T17:41:31.866554+0000","last_fullsized":"2024-01-24T17:41:31.866554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:16.636377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.866165+0000","last_change":"2024-01-24T17:41:31.866165+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.866165+0000","last_undegraded":"2024-01-24T17:41:31.866165+0000","last_fullsized":"2024-01-24T17:41:31.866165+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.860836+0000","last_change":"2024-01-24T17:41:31.860836+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.860836+0000","last_undegraded":"2024-01-24T17:41:31.860836+0000","last_fullsized":"2024-01-24T17:41:31.860836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.868366+0000","last_change":"2024-01-24T17:41:31.868366+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.868366+0000","last_undegraded":"2024-01-24T17:41:31.868366+0000","last_fullsized":"2024-01-24T17:41:31.868366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:51.152032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.869309+0000","last_change":"2024-01-24T17:41:31.869309+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.869309+0000","last_undegraded":"2024-01-24T17:41:31.869309+0000","last_fullsized":"2024-01-24T17:41:31.869309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.867609+0000","last_change":"2024-01-24T17:41:31.867609+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.867609+0000","last_undegraded":"2024-01-24T17:41:31.867609+0000","last_fullsized":"2024-01-24T17:41:31.867609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:45.778394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.866784+0000","last_change":"2024-01-24T17:41:31.866784+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.866784+0000","last_undegraded":"2024-01-24T17:41:31.866784+0000","last_fullsized":"2024-01-24T17:41:31.866784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.868287+0000","last_change":"2024-01-24T17:41:31.868287+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.868287+0000","last_undegraded":"2024-01-24T17:41:31.868287+0000","last_fullsized":"2024-01-24T17:41:31.868287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.860568+0000","last_change":"2024-01-24T17:41:31.860568+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.860568+0000","last_undegraded":"2024-01-24T17:41:31.860568+0000","last_fullsized":"2024-01-24T17:41:31.860568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:53.093743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.867192+0000","last_change":"2024-01-24T17:41:31.867192+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.867192+0000","last_undegraded":"2024-01-24T17:41:31.867192+0000","last_fullsized":"2024-01-24T17:41:31.867192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.856733+0000","last_change":"2024-01-24T17:41:31.856733+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.856733+0000","last_undegraded":"2024-01-24T17:41:31.856733+0000","last_fullsized":"2024-01-24T17:41:31.856733+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.868854+0000","last_change":"2024-01-24T17:41:31.868854+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.868854+0000","last_undegraded":"2024-01-24T17:41:31.868854+0000","last_fullsized":"2024-01-24T17:41:31.868854+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:26.459376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.861005+0000","last_change":"2024-01-24T17:41:31.861005+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.861005+0000","last_undegraded":"2024-01-24T17:41:31.861005+0000","last_fullsized":"2024-01-24T17:41:31.861005+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.861590+0000","last_change":"2024-01-24T17:41:31.861590+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.861590+0000","last_undegraded":"2024-01-24T17:41:31.861590+0000","last_fullsized":"2024-01-24T17:41:31.861590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.866381+0000","last_change":"2024-01-24T17:41:31.866381+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.866381+0000","last_undegraded":"2024-01-24T17:41:31.866381+0000","last_fullsized":"2024-01-24T17:41:31.866381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:41:32.847621+0000","last_change":"2024-01-24T17:41:32.847621+0000","last_active":"2024-01-24T17:41:32.847621+0000","last_peered":"2024-01-24T17:41:32.847621+0000","last_clean":"2024-01-24T17:41:32.847621+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:32.847621+0000","last_undegraded":"2024-01-24T17:41:32.847621+0000","last_fullsized":"2024-01-24T17:41:32.847621+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-01-24T17:41:31.868033+0000","last_change":"2024-01-24T17:41:31.868033+0000","last_active":"2024-01-24T17:41:31.832671+0000","last_peered":"2024-01-24T17:41:31.832671+0000","last_clean":"2024-01-24T17:41:31.832671+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:41:31.868033+0000","last_undegraded":"2024-01-24T17:41:31.868033+0000","last_fullsized":"2024-01-24T17:41:31.868033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":51548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.649}]}]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":51548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.621}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9770000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2170000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":641916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.097}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.056}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.621}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.081}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7889999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31280,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340560,"statfs":{"total":96636764160,"available":96604733440,"internally_reserved":0,"allocated":167936,"data_stored":51548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31280,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340560,"statfs":{"total":96636764160,"available":96604733440,"internally_reserved":0,"allocated":167936,"data_stored":51548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6659999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":761856,"data_stored":641916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.748}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7010000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31280,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340560,"statfs":{"total":96636764160,"available":96604733440,"internally_reserved":0,"allocated":167936,"data_stored":51548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1560000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.274}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:41:37.841 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:41:38.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.100+0000 7f1fd1000640 1 Processor -- start 2024-01-24T17:41:38.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.100+0000 7f1fd1000640 1 -- start start 2024-01-24T17:41:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fd1000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1fcc0cdd70 0x7f1fcc06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fd1000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc0cd050 con 0x7f1fcc0cb2b0 2024-01-24T17:41:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fd1000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc159fe0 con 0x7f1fcc0ca670 2024-01-24T17:41:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fd1000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0caa50 con 0x7f1fcc0cdd70 2024-01-24T17:41:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fcbe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1fcc0cb2b0 0x7f1fcc0cb6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52356/0 (socket says 172.21.15.39:52356) 2024-01-24T17:41:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fcbe00640 1 -- 172.21.15.39:0/3367508270 learned_addr learned my addr 172.21.15.39:0/3367508270 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238891835 0 0) 0x7f1fcc0cd050 con 0x7f1fcc0cb2b0 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fcaa00640 1 --2- 172.21.15.39:0/3367508270 >> v2:172.21.15.100:3300/0 conn(0x7f1fcc0cdd70 0x7f1fcc06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.102+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fac003520 con 0x7f1fcc0cb2b0 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941364367 0 0) 0x7f1fac003520 con 0x7f1fcc0cb2b0 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 >> v2:172.21.15.100:3300/0 conn(0x7f1fcc0cdd70 msgr2=0x7f1fcc06bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 --2- 172.21.15.39:0/3367508270 >> v2:172.21.15.100:3300/0 conn(0x7f1fcc0cdd70 0x7f1fcc06bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 >> v1:172.21.15.100:6789/0 conn(0x7f1fcc0ca670 legacy=0x7f1fcc15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc159c90 con 0x7f1fcc0cb2b0 2024-01-24T17:41:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1fc40019f0 con 0x7f1fcc0cb2b0 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 >> v1:172.21.15.39:6789/0 conn(0x7f1fcc0cb2b0 legacy=0x7f1fcc0cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac003970 0x7f1fac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac0047c0 0x7f1fac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fac0052a0 0x7f1fac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0caa50 con 0x7f1fac0052a0 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac00c170 con 0x7f1fac0047c0 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac00c330 con 0x7f1fac003970 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fcbe00640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fac0052a0 0x7f1fac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.103+0000 7f1fcaa00640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac0047c0 0x7f1fac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.104+0000 7f1fcb400640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac003970 0x7f1fac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.104+0000 7f1fcbe00640 1 -- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac003970 msgr2=0x7f1fac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.104+0000 7f1fcbe00640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac003970 0x7f1fac003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.104+0000 7f1fcbe00640 1 -- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac0047c0 msgr2=0x7f1fac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.104+0000 7f1fcbe00640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fac0047c0 0x7f1fac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.104+0000 7f1fcbe00640 1 -- 172.21.15.39:0/3367508270 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac003520 con 0x7f1fac0052a0 2024-01-24T17:41:38.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.104+0000 7f1fcb400640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac003970 0x7f1fac003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:38.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.104+0000 7f1fcbe00640 1 -- 172.21.15.39:0/3367508270 reap_dead start 2024-01-24T17:41:38.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.105+0000 7f1fcbe00640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fac0052a0 0x7f1fac00b980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400a410 tx=0x7f1fac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e983795dc8efd7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.105+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40072d0 con 0x7f1fac0052a0 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.105+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc4007430 con 0x7f1fac0052a0 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.105+0000 7f1fca000640 1 -- 172.21.15.39:0/3367508270 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40077e0 con 0x7f1fac0052a0 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.106+0000 7f1fd1000640 1 -- 172.21.15.39:0/3367508270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fac0052a0 msgr2=0x7f1fac00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.106+0000 7f1fd1000640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fac0052a0 0x7f1fac00b980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400a410 tx=0x7f1fac0057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.106+0000 7f1fd1000640 1 -- 172.21.15.39:0/3367508270 shutdown_connections 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.106+0000 7f1fd1000640 1 --2- 172.21.15.39:0/3367508270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fac0052a0 0x7f1fac00b980 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.106+0000 7f1fd1000640 1 -- 172.21.15.39:0/3367508270 >> 172.21.15.39:0/3367508270 conn(0x7f1fcc06a090 msgr2=0x7f1fcc06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.106+0000 7f1fd1000640 1 -- 172.21.15.39:0/3367508270 shutdown_connections 2024-01-24T17:41:38.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.106+0000 7f1fd1000640 1 -- 172.21.15.39:0/3367508270 wait complete. 2024-01-24T17:41:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.108+0000 7f1fd1000640 1 Processor -- start 2024-01-24T17:41:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.108+0000 7f1fd1000640 1 -- start start 2024-01-24T17:41:38.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.109+0000 7f1fd1000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac0052a0 0x7f1fcc0c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fd1000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fcc0c4070 0x7f1fcc1ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fd1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fcc15b610 0x7f1fcc1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fd1000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0c67f0 con 0x7f1fcc15b610 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fd1000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0c5b70 con 0x7f1fcc0c4070 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fd1000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0c5e50 con 0x7f1fac0052a0 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fcc15b610 0x7f1fcc1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fcc15b610 0x7f1fcc1584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58204/0 (socket says 172.21.15.39:58204) 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcbe00640 1 -- 172.21.15.39:0/173888158 learned_addr learned my addr 172.21.15.39:0/173888158 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcaa00640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fcc0c4070 0x7f1fcc1ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcb400640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac0052a0 0x7f1fcc0c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcbe00640 1 -- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac0052a0 msgr2=0x7f1fcc0c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcbe00640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac0052a0 0x7f1fcc0c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcbe00640 1 -- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fcc0c4070 msgr2=0x7f1fcc1ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcbe00640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fcc0c4070 0x7f1fcc1ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcbe00640 1 -- 172.21.15.39:0/173888158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc0c6130 con 0x7f1fcc15b610 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.110+0000 7f1fcb400640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac0052a0 0x7f1fcc0c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.111+0000 7f1fcbe00640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fcc15b610 0x7f1fcc1584b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1fc402ee30 tx=0x7f1fc4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.111+0000 7f1fc3400640 1 -- 172.21.15.39:0/173888158 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40075a0 con 0x7f1fcc15b610 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.111+0000 7f1fc3400640 1 -- 172.21.15.39:0/173888158 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc4007700 con 0x7f1fcc15b610 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.111+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc0c6410 con 0x7f1fcc15b610 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.111+0000 7f1fc3400640 1 -- 172.21.15.39:0/173888158 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4007a90 con 0x7f1fcc15b610 2024-01-24T17:41:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.111+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc1ee910 con 0x7f1fcc15b610 2024-01-24T17:41:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.112+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc04dff0 con 0x7f1fcc15b610 2024-01-24T17:41:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.113+0000 7f1fc3400640 1 -- 172.21.15.39:0/173888158 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1fc404b460 con 0x7f1fcc15b610 2024-01-24T17:41:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.114+0000 7f1fc3400640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1fa406a480 0x7f1fa406c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.114+0000 7f1fc3400640 1 -- 172.21.15.39:0/173888158 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fc40ad130 con 0x7f1fcc15b610 2024-01-24T17:41:38.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.115+0000 7f1fcb400640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1fa406a480 0x7f1fa406c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.116+0000 7f1fcb400640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1fa406a480 0x7f1fa406c940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc001690 tx=0x7f1fbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:38.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.119+0000 7f1fc3400640 1 -- 172.21.15.39:0/173888158 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1fc40788b0 con 0x7f1fcc15b610 2024-01-24T17:41:38.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.310+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1fcc06c750 con 0x7f1fa406a480 2024-01-24T17:41:38.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.333+0000 7f1fc3400640 1 -- 172.21.15.39:0/173888158 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264457 (secure 0 0 0) 0x7f1fbc00d040 con 0x7f1fa406a480 2024-01-24T17:41:38.337 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:38.339 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:41:38.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1fa406a480 msgr2=0x7f1fa406c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1fa406a480 0x7f1fa406c940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc001690 tx=0x7f1fbc009040 comp rx=0 tx=0).stop 2024-01-24T17:41:38.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fcc15b610 msgr2=0x7f1fcc1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fcc15b610 0x7f1fcc1584b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1fc402ee30 tx=0x7f1fc4001d80 comp rx=0 tx=0).stop 2024-01-24T17:41:38.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 shutdown_connections 2024-01-24T17:41:38.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1fa406a480 0x7f1fa406c940 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fcc15b610 0x7f1fcc1584b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fcc0c4070 0x7f1fcc1ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 --2- 172.21.15.39:0/173888158 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fac0052a0 0x7f1fcc0c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 >> 172.21.15.39:0/173888158 conn(0x7f1fcc06a090 msgr2=0x7f1fcc0b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:38.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 shutdown_connections 2024-01-24T17:41:38.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.339+0000 7f1fd1000640 1 -- 172.21.15.39:0/173888158 wait complete. 2024-01-24T17:41:38.359 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-24T17:41:38.209659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235372,"kb_used_data":2824,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754739348,"statfs":{"total":773094113280,"available":772853092352,"internally_reserved":0,"allocated":2891776,"data_stored":1752644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002004"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382873+0000","last_change":"2024-01-24T17:41:33.393186+0000","last_active":"2024-01-24T17:41:34.382873+0000","last_peered":"2024-01-24T17:41:34.382873+0000","last_clean":"2024-01-24T17:41:34.382873+0000","last_became_active":"2024-01-24T17:41:33.391034+0000","last_became_peered":"2024-01-24T17:41:33.391034+0000","last_unstale":"2024-01-24T17:41:34.382873+0000","last_undegraded":"2024-01-24T17:41:34.382873+0000","last_fullsized":"2024-01-24T17:41:34.382873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376217+0000","last_change":"2024-01-24T17:41:33.384027+0000","last_active":"2024-01-24T17:41:34.376217+0000","last_peered":"2024-01-24T17:41:34.376217+0000","last_clean":"2024-01-24T17:41:34.376217+0000","last_became_active":"2024-01-24T17:41:33.383720+0000","last_became_peered":"2024-01-24T17:41:33.383720+0000","last_unstale":"2024-01-24T17:41:34.376217+0000","last_undegraded":"2024-01-24T17:41:34.376217+0000","last_fullsized":"2024-01-24T17:41:34.376217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376005+0000","last_change":"2024-01-24T17:41:33.392448+0000","last_active":"2024-01-24T17:41:34.376005+0000","last_peered":"2024-01-24T17:41:34.376005+0000","last_clean":"2024-01-24T17:41:34.376005+0000","last_became_active":"2024-01-24T17:41:33.392125+0000","last_became_peered":"2024-01-24T17:41:33.392125+0000","last_unstale":"2024-01-24T17:41:34.376005+0000","last_undegraded":"2024-01-24T17:41:34.376005+0000","last_fullsized":"2024-01-24T17:41:34.376005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:20:35.949143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.385632+0000","last_change":"2024-01-24T17:41:33.395630+0000","last_active":"2024-01-24T17:41:34.385632+0000","last_peered":"2024-01-24T17:41:34.385632+0000","last_clean":"2024-01-24T17:41:34.385632+0000","last_became_active":"2024-01-24T17:41:33.395378+0000","last_became_peered":"2024-01-24T17:41:33.395378+0000","last_unstale":"2024-01-24T17:41:34.385632+0000","last_undegraded":"2024-01-24T17:41:34.385632+0000","last_fullsized":"2024-01-24T17:41:34.385632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382553+0000","last_change":"2024-01-24T17:41:33.382453+0000","last_active":"2024-01-24T17:41:34.382553+0000","last_peered":"2024-01-24T17:41:34.382553+0000","last_clean":"2024-01-24T17:41:34.382553+0000","last_became_active":"2024-01-24T17:41:33.382151+0000","last_became_peered":"2024-01-24T17:41:33.382151+0000","last_unstale":"2024-01-24T17:41:34.382553+0000","last_undegraded":"2024-01-24T17:41:34.382553+0000","last_fullsized":"2024-01-24T17:41:34.382553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.384184+0000","last_change":"2024-01-24T17:41:33.396138+0000","last_active":"2024-01-24T17:41:34.384184+0000","last_peered":"2024-01-24T17:41:34.384184+0000","last_clean":"2024-01-24T17:41:34.384184+0000","last_became_active":"2024-01-24T17:41:33.395890+0000","last_became_peered":"2024-01-24T17:41:33.395890+0000","last_unstale":"2024-01-24T17:41:34.384184+0000","last_undegraded":"2024-01-24T17:41:34.384184+0000","last_fullsized":"2024-01-24T17:41:34.384184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:17.651326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381377+0000","last_change":"2024-01-24T17:41:33.393003+0000","last_active":"2024-01-24T17:41:34.381377+0000","last_peered":"2024-01-24T17:41:34.381377+0000","last_clean":"2024-01-24T17:41:34.381377+0000","last_became_active":"2024-01-24T17:41:33.391259+0000","last_became_peered":"2024-01-24T17:41:33.391259+0000","last_unstale":"2024-01-24T17:41:34.381377+0000","last_undegraded":"2024-01-24T17:41:34.381377+0000","last_fullsized":"2024-01-24T17:41:34.381377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:50:21.701964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374081+0000","last_change":"2024-01-24T17:41:33.402016+0000","last_active":"2024-01-24T17:41:34.374081+0000","last_peered":"2024-01-24T17:41:34.374081+0000","last_clean":"2024-01-24T17:41:34.374081+0000","last_became_active":"2024-01-24T17:41:33.401767+0000","last_became_peered":"2024-01-24T17:41:33.401767+0000","last_unstale":"2024-01-24T17:41:34.374081+0000","last_undegraded":"2024-01-24T17:41:34.374081+0000","last_fullsized":"2024-01-24T17:41:34.374081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:35:09.121979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.387416+0000","last_change":"2024-01-24T17:41:33.391456+0000","last_active":"2024-01-24T17:41:34.387416+0000","last_peered":"2024-01-24T17:41:34.387416+0000","last_clean":"2024-01-24T17:41:34.387416+0000","last_became_active":"2024-01-24T17:41:33.391192+0000","last_became_peered":"2024-01-24T17:41:33.391192+0000","last_unstale":"2024-01-24T17:41:34.387416+0000","last_undegraded":"2024-01-24T17:41:34.387416+0000","last_fullsized":"2024-01-24T17:41:34.387416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382754+0000","last_change":"2024-01-24T17:41:33.380252+0000","last_active":"2024-01-24T17:41:34.382754+0000","last_peered":"2024-01-24T17:41:34.382754+0000","last_clean":"2024-01-24T17:41:34.382754+0000","last_became_active":"2024-01-24T17:41:33.378238+0000","last_became_peered":"2024-01-24T17:41:33.378238+0000","last_unstale":"2024-01-24T17:41:34.382754+0000","last_undegraded":"2024-01-24T17:41:34.382754+0000","last_fullsized":"2024-01-24T17:41:34.382754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:01:48.740461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382398+0000","last_change":"2024-01-24T17:41:33.382673+0000","last_active":"2024-01-24T17:41:34.382398+0000","last_peered":"2024-01-24T17:41:34.382398+0000","last_clean":"2024-01-24T17:41:34.382398+0000","last_became_active":"2024-01-24T17:41:33.382404+0000","last_became_peered":"2024-01-24T17:41:33.382404+0000","last_unstale":"2024-01-24T17:41:34.382398+0000","last_undegraded":"2024-01-24T17:41:34.382398+0000","last_fullsized":"2024-01-24T17:41:34.382398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:44:38.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383532+0000","last_change":"2024-01-24T17:41:33.395359+0000","last_active":"2024-01-24T17:41:34.383532+0000","last_peered":"2024-01-24T17:41:34.383532+0000","last_clean":"2024-01-24T17:41:34.383532+0000","last_became_active":"2024-01-24T17:41:33.395102+0000","last_became_peered":"2024-01-24T17:41:33.395102+0000","last_unstale":"2024-01-24T17:41:34.383532+0000","last_undegraded":"2024-01-24T17:41:34.383532+0000","last_fullsized":"2024-01-24T17:41:34.383532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372275+0000","last_change":"2024-01-24T17:41:33.391912+0000","last_active":"2024-01-24T17:41:34.372275+0000","last_peered":"2024-01-24T17:41:34.372275+0000","last_clean":"2024-01-24T17:41:34.372275+0000","last_became_active":"2024-01-24T17:41:33.391421+0000","last_became_peered":"2024-01-24T17:41:33.391421+0000","last_unstale":"2024-01-24T17:41:34.372275+0000","last_undegraded":"2024-01-24T17:41:34.372275+0000","last_fullsized":"2024-01-24T17:41:34.372275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:36:51.994706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.371942+0000","last_change":"2024-01-24T17:41:33.381744+0000","last_active":"2024-01-24T17:41:34.371942+0000","last_peered":"2024-01-24T17:41:34.371942+0000","last_clean":"2024-01-24T17:41:34.371942+0000","last_became_active":"2024-01-24T17:41:33.381499+0000","last_became_peered":"2024-01-24T17:41:33.381499+0000","last_unstale":"2024-01-24T17:41:34.371942+0000","last_undegraded":"2024-01-24T17:41:34.371942+0000","last_fullsized":"2024-01-24T17:41:34.371942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:27.474325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373889+0000","last_change":"2024-01-24T17:41:33.382763+0000","last_active":"2024-01-24T17:41:34.373889+0000","last_peered":"2024-01-24T17:41:34.373889+0000","last_clean":"2024-01-24T17:41:34.373889+0000","last_became_active":"2024-01-24T17:41:33.382506+0000","last_became_peered":"2024-01-24T17:41:33.382506+0000","last_unstale":"2024-01-24T17:41:34.373889+0000","last_undegraded":"2024-01-24T17:41:34.373889+0000","last_fullsized":"2024-01-24T17:41:34.373889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:29.008097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383246+0000","last_change":"2024-01-24T17:41:33.398656+0000","last_active":"2024-01-24T17:41:34.383246+0000","last_peered":"2024-01-24T17:41:34.383246+0000","last_clean":"2024-01-24T17:41:34.383246+0000","last_became_active":"2024-01-24T17:41:33.398404+0000","last_became_peered":"2024-01-24T17:41:33.398404+0000","last_unstale":"2024-01-24T17:41:34.383246+0000","last_undegraded":"2024-01-24T17:41:34.383246+0000","last_fullsized":"2024-01-24T17:41:34.383246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:36:23.946397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383245+0000","last_change":"2024-01-24T17:41:33.401897+0000","last_active":"2024-01-24T17:41:34.383245+0000","last_peered":"2024-01-24T17:41:34.383245+0000","last_clean":"2024-01-24T17:41:34.383245+0000","last_became_active":"2024-01-24T17:41:33.401641+0000","last_became_peered":"2024-01-24T17:41:33.401641+0000","last_unstale":"2024-01-24T17:41:34.383245+0000","last_undegraded":"2024-01-24T17:41:34.383245+0000","last_fullsized":"2024-01-24T17:41:34.383245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:52:01.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382136+0000","last_change":"2024-01-24T17:41:33.397837+0000","last_active":"2024-01-24T17:41:34.382136+0000","last_peered":"2024-01-24T17:41:34.382136+0000","last_clean":"2024-01-24T17:41:34.382136+0000","last_became_active":"2024-01-24T17:41:33.397597+0000","last_became_peered":"2024-01-24T17:41:33.397597+0000","last_unstale":"2024-01-24T17:41:34.382136+0000","last_undegraded":"2024-01-24T17:41:34.382136+0000","last_fullsized":"2024-01-24T17:41:34.382136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381810+0000","last_change":"2024-01-24T17:41:33.389940+0000","last_active":"2024-01-24T17:41:34.381810+0000","last_peered":"2024-01-24T17:41:34.381810+0000","last_clean":"2024-01-24T17:41:34.381810+0000","last_became_active":"2024-01-24T17:41:33.389673+0000","last_became_peered":"2024-01-24T17:41:33.389673+0000","last_unstale":"2024-01-24T17:41:34.381810+0000","last_undegraded":"2024-01-24T17:41:34.381810+0000","last_fullsized":"2024-01-24T17:41:34.381810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:48:57.485742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383781+0000","last_change":"2024-01-24T17:41:33.394799+0000","last_active":"2024-01-24T17:41:34.383781+0000","last_peered":"2024-01-24T17:41:34.383781+0000","last_clean":"2024-01-24T17:41:34.383781+0000","last_became_active":"2024-01-24T17:41:33.394396+0000","last_became_peered":"2024-01-24T17:41:33.394396+0000","last_unstale":"2024-01-24T17:41:34.383781+0000","last_undegraded":"2024-01-24T17:41:34.383781+0000","last_fullsized":"2024-01-24T17:41:34.383781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:35:09.121979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372973+0000","last_change":"2024-01-24T17:41:33.402158+0000","last_active":"2024-01-24T17:41:34.372973+0000","last_peered":"2024-01-24T17:41:34.372973+0000","last_clean":"2024-01-24T17:41:34.372973+0000","last_became_active":"2024-01-24T17:41:33.400801+0000","last_became_peered":"2024-01-24T17:41:33.400801+0000","last_unstale":"2024-01-24T17:41:34.372973+0000","last_undegraded":"2024-01-24T17:41:34.372973+0000","last_fullsized":"2024-01-24T17:41:34.372973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:29.008097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386543+0000","last_change":"2024-01-24T17:41:33.399257+0000","last_active":"2024-01-24T17:41:34.386543+0000","last_peered":"2024-01-24T17:41:34.386543+0000","last_clean":"2024-01-24T17:41:34.386543+0000","last_became_active":"2024-01-24T17:41:33.398569+0000","last_became_peered":"2024-01-24T17:41:33.398569+0000","last_unstale":"2024-01-24T17:41:34.386543+0000","last_undegraded":"2024-01-24T17:41:34.386543+0000","last_fullsized":"2024-01-24T17:41:34.386543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:54.108693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374158+0000","last_change":"2024-01-24T17:41:33.372320+0000","last_active":"2024-01-24T17:41:34.374158+0000","last_peered":"2024-01-24T17:41:34.374158+0000","last_clean":"2024-01-24T17:41:34.374158+0000","last_became_active":"2024-01-24T17:41:33.372080+0000","last_became_peered":"2024-01-24T17:41:33.372080+0000","last_unstale":"2024-01-24T17:41:34.374158+0000","last_undegraded":"2024-01-24T17:41:34.374158+0000","last_fullsized":"2024-01-24T17:41:34.374158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381086+0000","last_change":"2024-01-24T17:41:33.388373+0000","last_active":"2024-01-24T17:41:34.381086+0000","last_peered":"2024-01-24T17:41:34.381086+0000","last_clean":"2024-01-24T17:41:34.381086+0000","last_became_active":"2024-01-24T17:41:33.387185+0000","last_became_peered":"2024-01-24T17:41:33.387185+0000","last_unstale":"2024-01-24T17:41:34.381086+0000","last_undegraded":"2024-01-24T17:41:34.381086+0000","last_fullsized":"2024-01-24T17:41:34.381086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376756+0000","last_change":"2024-01-24T17:41:33.396814+0000","last_active":"2024-01-24T17:41:34.376756+0000","last_peered":"2024-01-24T17:41:34.376756+0000","last_clean":"2024-01-24T17:41:34.376756+0000","last_became_active":"2024-01-24T17:41:33.396446+0000","last_became_peered":"2024-01-24T17:41:33.396446+0000","last_unstale":"2024-01-24T17:41:34.376756+0000","last_undegraded":"2024-01-24T17:41:34.376756+0000","last_fullsized":"2024-01-24T17:41:34.376756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386135+0000","last_change":"2024-01-24T17:41:33.386328+0000","last_active":"2024-01-24T17:41:34.386135+0000","last_peered":"2024-01-24T17:41:34.386135+0000","last_clean":"2024-01-24T17:41:34.386135+0000","last_became_active":"2024-01-24T17:41:33.384795+0000","last_became_peered":"2024-01-24T17:41:33.384795+0000","last_unstale":"2024-01-24T17:41:34.386135+0000","last_undegraded":"2024-01-24T17:41:34.386135+0000","last_fullsized":"2024-01-24T17:41:34.386135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372759+0000","last_change":"2024-01-24T17:41:33.381873+0000","last_active":"2024-01-24T17:41:34.372759+0000","last_peered":"2024-01-24T17:41:34.372759+0000","last_clean":"2024-01-24T17:41:34.372759+0000","last_became_active":"2024-01-24T17:41:33.381356+0000","last_became_peered":"2024-01-24T17:41:33.381356+0000","last_unstale":"2024-01-24T17:41:34.372759+0000","last_undegraded":"2024-01-24T17:41:34.372759+0000","last_fullsized":"2024-01-24T17:41:34.372759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372500+0000","last_change":"2024-01-24T17:41:33.401722+0000","last_active":"2024-01-24T17:41:34.372500+0000","last_peered":"2024-01-24T17:41:34.372500+0000","last_clean":"2024-01-24T17:41:34.372500+0000","last_became_active":"2024-01-24T17:41:33.401437+0000","last_became_peered":"2024-01-24T17:41:33.401437+0000","last_unstale":"2024-01-24T17:41:34.372500+0000","last_undegraded":"2024-01-24T17:41:34.372500+0000","last_fullsized":"2024-01-24T17:41:34.372500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:53:07.257519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.375137+0000","last_change":"2024-01-24T17:41:33.386028+0000","last_active":"2024-01-24T17:41:34.375137+0000","last_peered":"2024-01-24T17:41:34.375137+0000","last_clean":"2024-01-24T17:41:34.375137+0000","last_became_active":"2024-01-24T17:41:33.385728+0000","last_became_peered":"2024-01-24T17:41:33.385728+0000","last_unstale":"2024-01-24T17:41:34.375137+0000","last_undegraded":"2024-01-24T17:41:34.375137+0000","last_fullsized":"2024-01-24T17:41:34.375137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:14.826051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381062+0000","last_change":"2024-01-24T17:41:33.392322+0000","last_active":"2024-01-24T17:41:34.381062+0000","last_peered":"2024-01-24T17:41:34.381062+0000","last_clean":"2024-01-24T17:41:34.381062+0000","last_became_active":"2024-01-24T17:41:33.391949+0000","last_became_peered":"2024-01-24T17:41:33.391949+0000","last_unstale":"2024-01-24T17:41:34.381062+0000","last_undegraded":"2024-01-24T17:41:34.381062+0000","last_fullsized":"2024-01-24T17:41:34.381062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381378+0000","last_change":"2024-01-24T17:41:33.392982+0000","last_active":"2024-01-24T17:41:34.381378+0000","last_peered":"2024-01-24T17:41:34.381378+0000","last_clean":"2024-01-24T17:41:34.381378+0000","last_became_active":"2024-01-24T17:41:33.392653+0000","last_became_peered":"2024-01-24T17:41:33.392653+0000","last_unstale":"2024-01-24T17:41:34.381378+0000","last_undegraded":"2024-01-24T17:41:34.381378+0000","last_fullsized":"2024-01-24T17:41:34.381378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382379+0000","last_change":"2024-01-24T17:41:33.388829+0000","last_active":"2024-01-24T17:41:34.382379+0000","last_peered":"2024-01-24T17:41:34.382379+0000","last_clean":"2024-01-24T17:41:34.382379+0000","last_became_active":"2024-01-24T17:41:33.387836+0000","last_became_peered":"2024-01-24T17:41:33.387836+0000","last_unstale":"2024-01-24T17:41:34.382379+0000","last_undegraded":"2024-01-24T17:41:34.382379+0000","last_fullsized":"2024-01-24T17:41:34.382379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386559+0000","last_change":"2024-01-24T17:41:33.386169+0000","last_active":"2024-01-24T17:41:34.386559+0000","last_peered":"2024-01-24T17:41:34.386559+0000","last_clean":"2024-01-24T17:41:34.386559+0000","last_became_active":"2024-01-24T17:41:33.385898+0000","last_became_peered":"2024-01-24T17:41:33.385898+0000","last_unstale":"2024-01-24T17:41:34.386559+0000","last_undegraded":"2024-01-24T17:41:34.386559+0000","last_fullsized":"2024-01-24T17:41:34.386559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:46.793343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402349+0000","last_change":"2024-01-24T17:41:32.895862+0000","last_active":"2024-01-24T17:41:33.402349+0000","last_peered":"2024-01-24T17:41:33.402349+0000","last_clean":"2024-01-24T17:41:33.402349+0000","last_became_active":"2024-01-24T17:41:32.895370+0000","last_became_peered":"2024-01-24T17:41:32.895370+0000","last_unstale":"2024-01-24T17:41:33.402349+0000","last_undegraded":"2024-01-24T17:41:33.402349+0000","last_fullsized":"2024-01-24T17:41:33.402349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373080+0000","last_change":"2024-01-24T17:41:33.405202+0000","last_active":"2024-01-24T17:41:34.373080+0000","last_peered":"2024-01-24T17:41:34.373080+0000","last_clean":"2024-01-24T17:41:34.373080+0000","last_became_active":"2024-01-24T17:41:33.404917+0000","last_became_peered":"2024-01-24T17:41:33.404917+0000","last_unstale":"2024-01-24T17:41:34.373080+0000","last_undegraded":"2024-01-24T17:41:34.373080+0000","last_fullsized":"2024-01-24T17:41:34.373080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404334+0000","last_change":"2024-01-24T17:41:32.910250+0000","last_active":"2024-01-24T17:41:33.404334+0000","last_peered":"2024-01-24T17:41:33.404334+0000","last_clean":"2024-01-24T17:41:33.404334+0000","last_became_active":"2024-01-24T17:41:32.909619+0000","last_became_peered":"2024-01-24T17:41:32.909619+0000","last_unstale":"2024-01-24T17:41:33.404334+0000","last_undegraded":"2024-01-24T17:41:33.404334+0000","last_fullsized":"2024-01-24T17:41:33.404334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:56.825045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.373938+0000","last_change":"2024-01-24T17:41:32.895589+0000","last_active":"2024-01-24T17:41:33.373938+0000","last_peered":"2024-01-24T17:41:33.373938+0000","last_clean":"2024-01-24T17:41:33.373938+0000","last_became_active":"2024-01-24T17:41:32.895305+0000","last_became_peered":"2024-01-24T17:41:32.895305+0000","last_unstale":"2024-01-24T17:41:33.373938+0000","last_undegraded":"2024-01-24T17:41:33.373938+0000","last_fullsized":"2024-01-24T17:41:33.373938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:01:37.962464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374431+0000","last_change":"2024-01-24T17:41:33.385480+0000","last_active":"2024-01-24T17:41:34.374431+0000","last_peered":"2024-01-24T17:41:34.374431+0000","last_clean":"2024-01-24T17:41:34.374431+0000","last_became_active":"2024-01-24T17:41:33.384848+0000","last_became_peered":"2024-01-24T17:41:33.384848+0000","last_unstale":"2024-01-24T17:41:34.374431+0000","last_undegraded":"2024-01-24T17:41:34.374431+0000","last_fullsized":"2024-01-24T17:41:34.374431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:50:21.701964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377136+0000","last_change":"2024-01-24T17:41:32.897056+0000","last_active":"2024-01-24T17:41:33.377136+0000","last_peered":"2024-01-24T17:41:33.377136+0000","last_clean":"2024-01-24T17:41:33.377136+0000","last_became_active":"2024-01-24T17:41:32.888851+0000","last_became_peered":"2024-01-24T17:41:32.888851+0000","last_unstale":"2024-01-24T17:41:33.377136+0000","last_undegraded":"2024-01-24T17:41:33.377136+0000","last_fullsized":"2024-01-24T17:41:33.377136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374368+0000","last_change":"2024-01-24T17:41:33.395388+0000","last_active":"2024-01-24T17:41:34.374368+0000","last_peered":"2024-01-24T17:41:34.374368+0000","last_clean":"2024-01-24T17:41:34.374368+0000","last_became_active":"2024-01-24T17:41:33.395100+0000","last_became_peered":"2024-01-24T17:41:33.395100+0000","last_unstale":"2024-01-24T17:41:34.374368+0000","last_undegraded":"2024-01-24T17:41:34.374368+0000","last_fullsized":"2024-01-24T17:41:34.374368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:17:59.263270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.383197+0000","last_change":"2024-01-24T17:41:32.892978+0000","last_active":"2024-01-24T17:41:33.383197+0000","last_peered":"2024-01-24T17:41:33.383197+0000","last_clean":"2024-01-24T17:41:33.383197+0000","last_became_active":"2024-01-24T17:41:32.892668+0000","last_became_peered":"2024-01-24T17:41:32.892668+0000","last_unstale":"2024-01-24T17:41:33.383197+0000","last_undegraded":"2024-01-24T17:41:33.383197+0000","last_fullsized":"2024-01-24T17:41:33.383197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:01:37.962464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386042+0000","last_change":"2024-01-24T17:41:33.384625+0000","last_active":"2024-01-24T17:41:34.386042+0000","last_peered":"2024-01-24T17:41:34.386042+0000","last_clean":"2024-01-24T17:41:34.386042+0000","last_became_active":"2024-01-24T17:41:33.384293+0000","last_became_peered":"2024-01-24T17:41:33.384293+0000","last_unstale":"2024-01-24T17:41:34.386042+0000","last_undegraded":"2024-01-24T17:41:34.386042+0000","last_fullsized":"2024-01-24T17:41:34.386042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:27.474325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.369468+0000","last_change":"2024-01-24T17:41:32.896336+0000","last_active":"2024-01-24T17:41:33.369468+0000","last_peered":"2024-01-24T17:41:33.369468+0000","last_clean":"2024-01-24T17:41:33.369468+0000","last_became_active":"2024-01-24T17:41:32.896023+0000","last_became_peered":"2024-01-24T17:41:32.896023+0000","last_unstale":"2024-01-24T17:41:33.369468+0000","last_undegraded":"2024-01-24T17:41:33.369468+0000","last_fullsized":"2024-01-24T17:41:33.369468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.377112+0000","last_change":"2024-01-24T17:41:33.384509+0000","last_active":"2024-01-24T17:41:34.377112+0000","last_peered":"2024-01-24T17:41:34.377112+0000","last_clean":"2024-01-24T17:41:34.377112+0000","last_became_active":"2024-01-24T17:41:33.384188+0000","last_became_peered":"2024-01-24T17:41:33.384188+0000","last_unstale":"2024-01-24T17:41:34.377112+0000","last_undegraded":"2024-01-24T17:41:34.377112+0000","last_fullsized":"2024-01-24T17:41:34.377112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:02.263592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.394541+0000","last_change":"2024-01-24T17:41:32.898747+0000","last_active":"2024-01-24T17:41:33.394541+0000","last_peered":"2024-01-24T17:41:33.394541+0000","last_clean":"2024-01-24T17:41:33.394541+0000","last_became_active":"2024-01-24T17:41:32.896625+0000","last_became_peered":"2024-01-24T17:41:32.896625+0000","last_unstale":"2024-01-24T17:41:33.394541+0000","last_undegraded":"2024-01-24T17:41:33.394541+0000","last_fullsized":"2024-01-24T17:41:33.394541+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381067+0000","last_change":"2024-01-24T17:41:33.385876+0000","last_active":"2024-01-24T17:41:34.381067+0000","last_peered":"2024-01-24T17:41:34.381067+0000","last_clean":"2024-01-24T17:41:34.381067+0000","last_became_active":"2024-01-24T17:41:33.385566+0000","last_became_peered":"2024-01-24T17:41:33.385566+0000","last_unstale":"2024-01-24T17:41:34.381067+0000","last_undegraded":"2024-01-24T17:41:34.381067+0000","last_fullsized":"2024-01-24T17:41:34.381067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:44:38.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.411432+0000","last_change":"2024-01-24T17:41:32.901159+0000","last_active":"2024-01-24T17:41:33.411432+0000","last_peered":"2024-01-24T17:41:33.411432+0000","last_clean":"2024-01-24T17:41:33.411432+0000","last_became_active":"2024-01-24T17:41:32.900832+0000","last_became_peered":"2024-01-24T17:41:32.900832+0000","last_unstale":"2024-01-24T17:41:33.411432+0000","last_undegraded":"2024-01-24T17:41:33.411432+0000","last_fullsized":"2024-01-24T17:41:33.411432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:06:04.595152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383859+0000","last_change":"2024-01-24T17:41:33.382102+0000","last_active":"2024-01-24T17:41:34.383859+0000","last_peered":"2024-01-24T17:41:34.383859+0000","last_clean":"2024-01-24T17:41:34.383859+0000","last_became_active":"2024-01-24T17:41:33.381813+0000","last_became_peered":"2024-01-24T17:41:33.381813+0000","last_unstale":"2024-01-24T17:41:34.383859+0000","last_undegraded":"2024-01-24T17:41:34.383859+0000","last_fullsized":"2024-01-24T17:41:34.383859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.405772+0000","last_change":"2024-01-24T17:41:32.896282+0000","last_active":"2024-01-24T17:41:33.405772+0000","last_peered":"2024-01-24T17:41:33.405772+0000","last_clean":"2024-01-24T17:41:33.405772+0000","last_became_active":"2024-01-24T17:41:32.896029+0000","last_became_peered":"2024-01-24T17:41:32.896029+0000","last_unstale":"2024-01-24T17:41:33.405772+0000","last_undegraded":"2024-01-24T17:41:33.405772+0000","last_fullsized":"2024-01-24T17:41:33.405772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372253+0000","last_change":"2024-01-24T17:41:33.380019+0000","last_active":"2024-01-24T17:41:34.372253+0000","last_peered":"2024-01-24T17:41:34.372253+0000","last_clean":"2024-01-24T17:41:34.372253+0000","last_became_active":"2024-01-24T17:41:33.379743+0000","last_became_peered":"2024-01-24T17:41:33.379743+0000","last_unstale":"2024-01-24T17:41:34.372253+0000","last_undegraded":"2024-01-24T17:41:34.372253+0000","last_fullsized":"2024-01-24T17:41:34.372253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:01:48.740461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377512+0000","last_change":"2024-01-24T17:41:32.906190+0000","last_active":"2024-01-24T17:41:33.377512+0000","last_peered":"2024-01-24T17:41:33.377512+0000","last_clean":"2024-01-24T17:41:33.377512+0000","last_became_active":"2024-01-24T17:41:32.903525+0000","last_became_peered":"2024-01-24T17:41:32.903525+0000","last_unstale":"2024-01-24T17:41:33.377512+0000","last_undegraded":"2024-01-24T17:41:33.377512+0000","last_fullsized":"2024-01-24T17:41:33.377512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:46.796323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382677+0000","last_change":"2024-01-24T17:41:33.400221+0000","last_active":"2024-01-24T17:41:34.382677+0000","last_peered":"2024-01-24T17:41:34.382677+0000","last_clean":"2024-01-24T17:41:34.382677+0000","last_became_active":"2024-01-24T17:41:33.399980+0000","last_became_peered":"2024-01-24T17:41:33.399980+0000","last_unstale":"2024-01-24T17:41:34.382677+0000","last_undegraded":"2024-01-24T17:41:34.382677+0000","last_fullsized":"2024-01-24T17:41:34.382677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:54:11.431445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.367726+0000","last_change":"2024-01-24T17:41:32.907626+0000","last_active":"2024-01-24T17:41:33.367726+0000","last_peered":"2024-01-24T17:41:33.367726+0000","last_clean":"2024-01-24T17:41:33.367726+0000","last_became_active":"2024-01-24T17:41:32.907361+0000","last_became_peered":"2024-01-24T17:41:32.907361+0000","last_unstale":"2024-01-24T17:41:33.367726+0000","last_undegraded":"2024-01-24T17:41:33.367726+0000","last_fullsized":"2024-01-24T17:41:33.367726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386410+0000","last_change":"2024-01-24T17:41:33.392634+0000","last_active":"2024-01-24T17:41:34.386410+0000","last_peered":"2024-01-24T17:41:34.386410+0000","last_clean":"2024-01-24T17:41:34.386410+0000","last_became_active":"2024-01-24T17:41:33.392428+0000","last_became_peered":"2024-01-24T17:41:33.392428+0000","last_unstale":"2024-01-24T17:41:34.386410+0000","last_undegraded":"2024-01-24T17:41:34.386410+0000","last_fullsized":"2024-01-24T17:41:34.386410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.409695+0000","last_change":"2024-01-24T17:41:32.902055+0000","last_active":"2024-01-24T17:41:33.409695+0000","last_peered":"2024-01-24T17:41:33.409695+0000","last_clean":"2024-01-24T17:41:33.409695+0000","last_became_active":"2024-01-24T17:41:32.901738+0000","last_became_peered":"2024-01-24T17:41:32.901738+0000","last_unstale":"2024-01-24T17:41:33.409695+0000","last_undegraded":"2024-01-24T17:41:33.409695+0000","last_fullsized":"2024-01-24T17:41:33.409695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:45.778394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381845+0000","last_change":"2024-01-24T17:41:33.388186+0000","last_active":"2024-01-24T17:41:34.381845+0000","last_peered":"2024-01-24T17:41:34.381845+0000","last_clean":"2024-01-24T17:41:34.381845+0000","last_became_active":"2024-01-24T17:41:33.387930+0000","last_became_peered":"2024-01-24T17:41:33.387930+0000","last_unstale":"2024-01-24T17:41:34.381845+0000","last_undegraded":"2024-01-24T17:41:34.381845+0000","last_fullsized":"2024-01-24T17:41:34.381845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:19:54.318762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402116+0000","last_change":"2024-01-24T17:41:32.903388+0000","last_active":"2024-01-24T17:41:33.402116+0000","last_peered":"2024-01-24T17:41:33.402116+0000","last_clean":"2024-01-24T17:41:33.402116+0000","last_became_active":"2024-01-24T17:41:32.900342+0000","last_became_peered":"2024-01-24T17:41:32.900342+0000","last_unstale":"2024-01-24T17:41:33.402116+0000","last_undegraded":"2024-01-24T17:41:33.402116+0000","last_fullsized":"2024-01-24T17:41:33.402116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386143+0000","last_change":"2024-01-24T17:41:33.379754+0000","last_active":"2024-01-24T17:41:34.386143+0000","last_peered":"2024-01-24T17:41:34.386143+0000","last_clean":"2024-01-24T17:41:34.386143+0000","last_became_active":"2024-01-24T17:41:33.379454+0000","last_became_peered":"2024-01-24T17:41:33.379454+0000","last_unstale":"2024-01-24T17:41:34.386143+0000","last_undegraded":"2024-01-24T17:41:34.386143+0000","last_fullsized":"2024-01-24T17:41:34.386143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:46.793343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.408082+0000","last_change":"2024-01-24T17:41:32.904632+0000","last_active":"2024-01-24T17:41:33.408082+0000","last_peered":"2024-01-24T17:41:33.408082+0000","last_clean":"2024-01-24T17:41:33.408082+0000","last_became_active":"2024-01-24T17:41:32.904220+0000","last_became_peered":"2024-01-24T17:41:32.904220+0000","last_unstale":"2024-01-24T17:41:33.408082+0000","last_undegraded":"2024-01-24T17:41:33.408082+0000","last_fullsized":"2024-01-24T17:41:33.408082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:53.093743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382977+0000","last_change":"2024-01-24T17:41:33.395831+0000","last_active":"2024-01-24T17:41:34.382977+0000","last_peered":"2024-01-24T17:41:34.382977+0000","last_clean":"2024-01-24T17:41:34.382977+0000","last_became_active":"2024-01-24T17:41:33.392642+0000","last_became_peered":"2024-01-24T17:41:33.392642+0000","last_unstale":"2024-01-24T17:41:34.382977+0000","last_undegraded":"2024-01-24T17:41:34.382977+0000","last_fullsized":"2024-01-24T17:41:34.382977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:20:35.949143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.406390+0000","last_change":"2024-01-24T17:41:32.898336+0000","last_active":"2024-01-24T17:41:33.406390+0000","last_peered":"2024-01-24T17:41:33.406390+0000","last_clean":"2024-01-24T17:41:33.406390+0000","last_became_active":"2024-01-24T17:41:32.897951+0000","last_became_peered":"2024-01-24T17:41:32.897951+0000","last_unstale":"2024-01-24T17:41:33.406390+0000","last_undegraded":"2024-01-24T17:41:33.406390+0000","last_fullsized":"2024-01-24T17:41:33.406390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383662+0000","last_change":"2024-01-24T17:41:33.388321+0000","last_active":"2024-01-24T17:41:34.383662+0000","last_peered":"2024-01-24T17:41:34.383662+0000","last_clean":"2024-01-24T17:41:34.383662+0000","last_became_active":"2024-01-24T17:41:33.386908+0000","last_became_peered":"2024-01-24T17:41:33.386908+0000","last_unstale":"2024-01-24T17:41:34.383662+0000","last_undegraded":"2024-01-24T17:41:34.383662+0000","last_fullsized":"2024-01-24T17:41:34.383662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:14.826051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.376101+0000","last_change":"2024-01-24T17:41:32.898929+0000","last_active":"2024-01-24T17:41:33.376101+0000","last_peered":"2024-01-24T17:41:33.376101+0000","last_clean":"2024-01-24T17:41:33.376101+0000","last_became_active":"2024-01-24T17:41:32.898648+0000","last_became_peered":"2024-01-24T17:41:32.898648+0000","last_unstale":"2024-01-24T17:41:33.376101+0000","last_undegraded":"2024-01-24T17:41:33.376101+0000","last_fullsized":"2024-01-24T17:41:33.376101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:20.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381588+0000","last_change":"2024-01-24T17:41:33.394760+0000","last_active":"2024-01-24T17:41:34.381588+0000","last_peered":"2024-01-24T17:41:34.381588+0000","last_clean":"2024-01-24T17:41:34.381588+0000","last_became_active":"2024-01-24T17:41:33.394503+0000","last_became_peered":"2024-01-24T17:41:33.394503+0000","last_unstale":"2024-01-24T17:41:34.381588+0000","last_undegraded":"2024-01-24T17:41:34.381588+0000","last_fullsized":"2024-01-24T17:41:34.381588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:36:20.611114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.382896+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:41:33.382896+0000","last_peered":"2024-01-24T17:41:33.382896+0000","last_clean":"2024-01-24T17:41:33.382896+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:41:33.382896+0000","last_undegraded":"2024-01-24T17:41:33.382896+0000","last_fullsized":"2024-01-24T17:41:33.382896+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.378136+0000","last_change":"2024-01-24T17:41:32.906502+0000","last_active":"2024-01-24T17:41:33.378136+0000","last_peered":"2024-01-24T17:41:33.378136+0000","last_clean":"2024-01-24T17:41:33.378136+0000","last_became_active":"2024-01-24T17:41:32.905833+0000","last_became_peered":"2024-01-24T17:41:32.905833+0000","last_unstale":"2024-01-24T17:41:33.378136+0000","last_undegraded":"2024-01-24T17:41:33.378136+0000","last_fullsized":"2024-01-24T17:41:33.378136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:16.636377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374358+0000","last_change":"2024-01-24T17:41:33.400209+0000","last_active":"2024-01-24T17:41:34.374358+0000","last_peered":"2024-01-24T17:41:34.374358+0000","last_clean":"2024-01-24T17:41:34.374358+0000","last_became_active":"2024-01-24T17:41:33.399935+0000","last_became_peered":"2024-01-24T17:41:33.399935+0000","last_unstale":"2024-01-24T17:41:34.374358+0000","last_undegraded":"2024-01-24T17:41:34.374358+0000","last_fullsized":"2024-01-24T17:41:34.374358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:02.263592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377275+0000","last_change":"2024-01-24T17:41:32.903275+0000","last_active":"2024-01-24T17:41:33.377275+0000","last_peered":"2024-01-24T17:41:33.377275+0000","last_clean":"2024-01-24T17:41:33.377275+0000","last_became_active":"2024-01-24T17:41:32.902650+0000","last_became_peered":"2024-01-24T17:41:32.902650+0000","last_unstale":"2024-01-24T17:41:33.377275+0000","last_undegraded":"2024-01-24T17:41:33.377275+0000","last_fullsized":"2024-01-24T17:41:33.377275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.370807+0000","last_change":"2024-01-24T17:41:33.385359+0000","last_active":"2024-01-24T17:41:34.370807+0000","last_peered":"2024-01-24T17:41:34.370807+0000","last_clean":"2024-01-24T17:41:34.370807+0000","last_became_active":"2024-01-24T17:41:33.385048+0000","last_became_peered":"2024-01-24T17:41:33.385048+0000","last_unstale":"2024-01-24T17:41:34.370807+0000","last_undegraded":"2024-01-24T17:41:34.370807+0000","last_fullsized":"2024-01-24T17:41:34.370807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:17.651326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.381252+0000","last_change":"2024-01-24T17:41:32.899789+0000","last_active":"2024-01-24T17:41:33.381252+0000","last_peered":"2024-01-24T17:41:33.381252+0000","last_clean":"2024-01-24T17:41:33.381252+0000","last_became_active":"2024-01-24T17:41:32.899458+0000","last_became_peered":"2024-01-24T17:41:32.899458+0000","last_unstale":"2024-01-24T17:41:33.381252+0000","last_undegraded":"2024-01-24T17:41:33.381252+0000","last_fullsized":"2024-01-24T17:41:33.381252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374273+0000","last_change":"2024-01-24T17:41:33.385653+0000","last_active":"2024-01-24T17:41:34.374273+0000","last_peered":"2024-01-24T17:41:34.374273+0000","last_clean":"2024-01-24T17:41:34.374273+0000","last_became_active":"2024-01-24T17:41:33.385180+0000","last_became_peered":"2024-01-24T17:41:33.385180+0000","last_unstale":"2024-01-24T17:41:34.374273+0000","last_undegraded":"2024-01-24T17:41:34.374273+0000","last_fullsized":"2024-01-24T17:41:34.374273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:01:35.279502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377607+0000","last_change":"2024-01-24T17:41:32.903904+0000","last_active":"2024-01-24T17:41:33.377607+0000","last_peered":"2024-01-24T17:41:33.377607+0000","last_clean":"2024-01-24T17:41:33.377607+0000","last_became_active":"2024-01-24T17:41:32.902915+0000","last_became_peered":"2024-01-24T17:41:32.902915+0000","last_unstale":"2024-01-24T17:41:33.377607+0000","last_undegraded":"2024-01-24T17:41:33.377607+0000","last_fullsized":"2024-01-24T17:41:33.377607+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:51.152032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382323+0000","last_change":"2024-01-24T17:41:33.379050+0000","last_active":"2024-01-24T17:41:34.382323+0000","last_peered":"2024-01-24T17:41:34.382323+0000","last_clean":"2024-01-24T17:41:34.382323+0000","last_became_active":"2024-01-24T17:41:33.378782+0000","last_became_peered":"2024-01-24T17:41:33.378782+0000","last_unstale":"2024-01-24T17:41:34.382323+0000","last_undegraded":"2024-01-24T17:41:34.382323+0000","last_fullsized":"2024-01-24T17:41:34.382323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377892+0000","last_change":"2024-01-24T17:41:32.895887+0000","last_active":"2024-01-24T17:41:33.377892+0000","last_peered":"2024-01-24T17:41:33.377892+0000","last_clean":"2024-01-24T17:41:33.377892+0000","last_became_active":"2024-01-24T17:41:32.895460+0000","last_became_peered":"2024-01-24T17:41:32.895460+0000","last_unstale":"2024-01-24T17:41:33.377892+0000","last_undegraded":"2024-01-24T17:41:33.377892+0000","last_fullsized":"2024-01-24T17:41:33.377892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.385945+0000","last_change":"2024-01-24T17:41:33.381026+0000","last_active":"2024-01-24T17:41:34.385945+0000","last_peered":"2024-01-24T17:41:34.385945+0000","last_clean":"2024-01-24T17:41:34.385945+0000","last_became_active":"2024-01-24T17:41:33.380649+0000","last_became_peered":"2024-01-24T17:41:33.380649+0000","last_unstale":"2024-01-24T17:41:34.385945+0000","last_undegraded":"2024-01-24T17:41:34.385945+0000","last_fullsized":"2024-01-24T17:41:34.385945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.378103+0000","last_change":"2024-01-24T17:41:32.907390+0000","last_active":"2024-01-24T17:41:33.378103+0000","last_peered":"2024-01-24T17:41:33.378103+0000","last_clean":"2024-01-24T17:41:33.378103+0000","last_became_active":"2024-01-24T17:41:32.906724+0000","last_became_peered":"2024-01-24T17:41:32.906724+0000","last_unstale":"2024-01-24T17:41:33.378103+0000","last_undegraded":"2024-01-24T17:41:33.378103+0000","last_fullsized":"2024-01-24T17:41:33.378103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:45.778394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373347+0000","last_change":"2024-01-24T17:41:33.390087+0000","last_active":"2024-01-24T17:41:34.373347+0000","last_peered":"2024-01-24T17:41:34.373347+0000","last_clean":"2024-01-24T17:41:34.373347+0000","last_became_active":"2024-01-24T17:41:33.389820+0000","last_became_peered":"2024-01-24T17:41:33.389820+0000","last_unstale":"2024-01-24T17:41:34.373347+0000","last_undegraded":"2024-01-24T17:41:34.373347+0000","last_fullsized":"2024-01-24T17:41:34.373347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.384053+0000","last_change":"2024-01-24T17:41:32.902595+0000","last_active":"2024-01-24T17:41:33.384053+0000","last_peered":"2024-01-24T17:41:33.384053+0000","last_clean":"2024-01-24T17:41:33.384053+0000","last_became_active":"2024-01-24T17:41:32.902291+0000","last_became_peered":"2024-01-24T17:41:32.902291+0000","last_unstale":"2024-01-24T17:41:33.384053+0000","last_undegraded":"2024-01-24T17:41:33.384053+0000","last_fullsized":"2024-01-24T17:41:33.384053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374204+0000","last_change":"2024-01-24T17:41:33.401908+0000","last_active":"2024-01-24T17:41:34.374204+0000","last_peered":"2024-01-24T17:41:34.374204+0000","last_clean":"2024-01-24T17:41:34.374204+0000","last_became_active":"2024-01-24T17:41:33.401634+0000","last_became_peered":"2024-01-24T17:41:33.401634+0000","last_unstale":"2024-01-24T17:41:34.374204+0000","last_undegraded":"2024-01-24T17:41:34.374204+0000","last_fullsized":"2024-01-24T17:41:34.374204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.405956+0000","last_change":"2024-01-24T17:41:32.909941+0000","last_active":"2024-01-24T17:41:33.405956+0000","last_peered":"2024-01-24T17:41:33.405956+0000","last_clean":"2024-01-24T17:41:33.405956+0000","last_became_active":"2024-01-24T17:41:32.909592+0000","last_became_peered":"2024-01-24T17:41:32.909592+0000","last_unstale":"2024-01-24T17:41:33.405956+0000","last_undegraded":"2024-01-24T17:41:33.405956+0000","last_fullsized":"2024-01-24T17:41:33.405956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373574+0000","last_change":"2024-01-24T17:41:33.394464+0000","last_active":"2024-01-24T17:41:34.373574+0000","last_peered":"2024-01-24T17:41:34.373574+0000","last_clean":"2024-01-24T17:41:34.373574+0000","last_became_active":"2024-01-24T17:41:33.393898+0000","last_became_peered":"2024-01-24T17:41:33.393898+0000","last_unstale":"2024-01-24T17:41:34.373574+0000","last_undegraded":"2024-01-24T17:41:34.373574+0000","last_fullsized":"2024-01-24T17:41:34.373574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:01:35.279502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.366360+0000","last_change":"2024-01-24T17:41:32.896815+0000","last_active":"2024-01-24T17:41:33.366360+0000","last_peered":"2024-01-24T17:41:33.366360+0000","last_clean":"2024-01-24T17:41:33.366360+0000","last_became_active":"2024-01-24T17:41:32.896263+0000","last_became_peered":"2024-01-24T17:41:32.896263+0000","last_unstale":"2024-01-24T17:41:33.366360+0000","last_undegraded":"2024-01-24T17:41:33.366360+0000","last_fullsized":"2024-01-24T17:41:33.366360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:53.093743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382082+0000","last_change":"2024-01-24T17:41:33.393488+0000","last_active":"2024-01-24T17:41:34.382082+0000","last_peered":"2024-01-24T17:41:34.382082+0000","last_clean":"2024-01-24T17:41:34.382082+0000","last_became_active":"2024-01-24T17:41:33.391266+0000","last_became_peered":"2024-01-24T17:41:33.391266+0000","last_unstale":"2024-01-24T17:41:34.382082+0000","last_undegraded":"2024-01-24T17:41:34.382082+0000","last_fullsized":"2024-01-24T17:41:34.382082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:08:31.383389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.387230+0000","last_change":"2024-01-24T17:41:32.902635+0000","last_active":"2024-01-24T17:41:33.387230+0000","last_peered":"2024-01-24T17:41:33.387230+0000","last_clean":"2024-01-24T17:41:33.387230+0000","last_became_active":"2024-01-24T17:41:32.902369+0000","last_became_peered":"2024-01-24T17:41:32.902369+0000","last_unstale":"2024-01-24T17:41:33.387230+0000","last_undegraded":"2024-01-24T17:41:33.387230+0000","last_fullsized":"2024-01-24T17:41:33.387230+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372780+0000","last_change":"2024-01-24T17:41:33.390610+0000","last_active":"2024-01-24T17:41:34.372780+0000","last_peered":"2024-01-24T17:41:34.372780+0000","last_clean":"2024-01-24T17:41:34.372780+0000","last_became_active":"2024-01-24T17:41:33.390337+0000","last_became_peered":"2024-01-24T17:41:33.390337+0000","last_unstale":"2024-01-24T17:41:34.372780+0000","last_undegraded":"2024-01-24T17:41:34.372780+0000","last_fullsized":"2024-01-24T17:41:34.372780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.374492+0000","last_change":"2024-01-24T17:41:32.891585+0000","last_active":"2024-01-24T17:41:33.374492+0000","last_peered":"2024-01-24T17:41:33.374492+0000","last_clean":"2024-01-24T17:41:33.374492+0000","last_became_active":"2024-01-24T17:41:32.891376+0000","last_became_peered":"2024-01-24T17:41:32.891376+0000","last_unstale":"2024-01-24T17:41:33.374492+0000","last_undegraded":"2024-01-24T17:41:33.374492+0000","last_fullsized":"2024-01-24T17:41:33.374492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381998+0000","last_change":"2024-01-24T17:41:33.397462+0000","last_active":"2024-01-24T17:41:34.381998+0000","last_peered":"2024-01-24T17:41:34.381998+0000","last_clean":"2024-01-24T17:41:34.381998+0000","last_became_active":"2024-01-24T17:41:33.397113+0000","last_became_peered":"2024-01-24T17:41:33.397113+0000","last_unstale":"2024-01-24T17:41:34.381998+0000","last_undegraded":"2024-01-24T17:41:34.381998+0000","last_fullsized":"2024-01-24T17:41:34.381998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.397708+0000","last_change":"2024-01-24T17:41:32.904150+0000","last_active":"2024-01-24T17:41:33.397708+0000","last_peered":"2024-01-24T17:41:33.397708+0000","last_clean":"2024-01-24T17:41:33.397708+0000","last_became_active":"2024-01-24T17:41:32.902550+0000","last_became_peered":"2024-01-24T17:41:32.902550+0000","last_unstale":"2024-01-24T17:41:33.397708+0000","last_undegraded":"2024-01-24T17:41:33.397708+0000","last_fullsized":"2024-01-24T17:41:33.397708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:26.459376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381261+0000","last_change":"2024-01-24T17:41:33.399418+0000","last_active":"2024-01-24T17:41:34.381261+0000","last_peered":"2024-01-24T17:41:34.381261+0000","last_clean":"2024-01-24T17:41:34.381261+0000","last_became_active":"2024-01-24T17:41:33.395852+0000","last_became_peered":"2024-01-24T17:41:33.395852+0000","last_unstale":"2024-01-24T17:41:34.381261+0000","last_undegraded":"2024-01-24T17:41:34.381261+0000","last_fullsized":"2024-01-24T17:41:34.381261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381377+0000","last_change":"2024-01-24T17:41:33.401099+0000","last_active":"2024-01-24T17:41:34.381377+0000","last_peered":"2024-01-24T17:41:34.381377+0000","last_clean":"2024-01-24T17:41:34.381377+0000","last_became_active":"2024-01-24T17:41:33.399957+0000","last_became_peered":"2024-01-24T17:41:33.399957+0000","last_unstale":"2024-01-24T17:41:34.381377+0000","last_undegraded":"2024-01-24T17:41:34.381377+0000","last_fullsized":"2024-01-24T17:41:34.381377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:12:52.644481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404497+0000","last_change":"2024-01-24T17:41:32.900545+0000","last_active":"2024-01-24T17:41:33.404497+0000","last_peered":"2024-01-24T17:41:33.404497+0000","last_clean":"2024-01-24T17:41:33.404497+0000","last_became_active":"2024-01-24T17:41:32.900228+0000","last_became_peered":"2024-01-24T17:41:32.900228+0000","last_unstale":"2024-01-24T17:41:33.404497+0000","last_undegraded":"2024-01-24T17:41:33.404497+0000","last_fullsized":"2024-01-24T17:41:33.404497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386811+0000","last_change":"2024-01-24T17:41:33.401985+0000","last_active":"2024-01-24T17:41:34.386811+0000","last_peered":"2024-01-24T17:41:34.386811+0000","last_clean":"2024-01-24T17:41:34.386811+0000","last_became_active":"2024-01-24T17:41:33.401623+0000","last_became_peered":"2024-01-24T17:41:33.401623+0000","last_unstale":"2024-01-24T17:41:34.386811+0000","last_undegraded":"2024-01-24T17:41:34.386811+0000","last_fullsized":"2024-01-24T17:41:34.386811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402556+0000","last_change":"2024-01-24T17:41:32.899336+0000","last_active":"2024-01-24T17:41:33.402556+0000","last_peered":"2024-01-24T17:41:33.402556+0000","last_clean":"2024-01-24T17:41:33.402556+0000","last_became_active":"2024-01-24T17:41:32.898773+0000","last_became_peered":"2024-01-24T17:41:32.898773+0000","last_unstale":"2024-01-24T17:41:33.402556+0000","last_undegraded":"2024-01-24T17:41:33.402556+0000","last_fullsized":"2024-01-24T17:41:33.402556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376757+0000","last_change":"2024-01-24T17:41:33.396094+0000","last_active":"2024-01-24T17:41:34.376757+0000","last_peered":"2024-01-24T17:41:34.376757+0000","last_clean":"2024-01-24T17:41:34.376757+0000","last_became_active":"2024-01-24T17:41:33.395819+0000","last_became_peered":"2024-01-24T17:41:33.395819+0000","last_unstale":"2024-01-24T17:41:34.376757+0000","last_undegraded":"2024-01-24T17:41:34.376757+0000","last_fullsized":"2024-01-24T17:41:34.376757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404773+0000","last_change":"2024-01-24T17:41:32.890539+0000","last_active":"2024-01-24T17:41:33.404773+0000","last_peered":"2024-01-24T17:41:33.404773+0000","last_clean":"2024-01-24T17:41:33.404773+0000","last_became_active":"2024-01-24T17:41:32.890175+0000","last_became_peered":"2024-01-24T17:41:32.890175+0000","last_unstale":"2024-01-24T17:41:33.404773+0000","last_undegraded":"2024-01-24T17:41:33.404773+0000","last_fullsized":"2024-01-24T17:41:33.404773+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376495+0000","last_change":"2024-01-24T17:41:33.374778+0000","last_active":"2024-01-24T17:41:34.376495+0000","last_peered":"2024-01-24T17:41:34.376495+0000","last_clean":"2024-01-24T17:41:34.376495+0000","last_became_active":"2024-01-24T17:41:33.374507+0000","last_became_peered":"2024-01-24T17:41:33.374507+0000","last_unstale":"2024-01-24T17:41:34.376495+0000","last_undegraded":"2024-01-24T17:41:34.376495+0000","last_fullsized":"2024-01-24T17:41:34.376495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.410169+0000","last_change":"2024-01-24T17:41:32.909855+0000","last_active":"2024-01-24T17:41:33.410169+0000","last_peered":"2024-01-24T17:41:33.410169+0000","last_clean":"2024-01-24T17:41:33.410169+0000","last_became_active":"2024-01-24T17:41:32.909553+0000","last_became_peered":"2024-01-24T17:41:32.909553+0000","last_unstale":"2024-01-24T17:41:33.410169+0000","last_undegraded":"2024-01-24T17:41:33.410169+0000","last_fullsized":"2024-01-24T17:41:33.410169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.631}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2930000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6739999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5209999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8460000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6579999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2050000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":661358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.569}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":73134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":208896,"data_stored":71480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.718}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0209999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":802816,"data_stored":661336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":212992,"data_stored":70954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:41:38.363 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-24T17:41:38.363 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-24T17:41:38.363 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-24T17:41:38.363 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-24T17:41:38.364 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-24T17:41:38.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.626+0000 7f9702e00640 1 Processor -- start 2024-01-24T17:41:38.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.627+0000 7f9702e00640 1 -- start start 2024-01-24T17:41:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.628+0000 7f9702e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96fc15cbf0 0x7f96fc0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.628+0000 7f9702e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f96fc0d0da0 0x7f96fc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.628+0000 7f9702e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96fc15e010 con 0x7f96fc15db60 2024-01-24T17:41:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.628+0000 7f9702e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc0d1750 con 0x7f96fc15cbf0 2024-01-24T17:41:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.628+0000 7f9702e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96fc0d0a90 con 0x7f96fc0d0da0 2024-01-24T17:41:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.628+0000 7f9701a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96fc15cbf0 0x7f96fc0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.628+0000 7f9701000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f96fc15db60 0x7f96fc0ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52366/0 (socket says 172.21.15.39:52366) 2024-01-24T17:41:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.628+0000 7f9701000640 1 -- 172.21.15.39:0/1191703235 learned_addr learned my addr 172.21.15.39:0/1191703235 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.629+0000 7f9702400640 1 --2- 172.21.15.39:0/1191703235 >> v2:172.21.15.100:3300/0 conn(0x7f96fc0d0da0 0x7f96fc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.629+0000 7f9701a00640 1 -- 172.21.15.39:0/1191703235 >> v2:172.21.15.100:3300/0 conn(0x7f96fc0d0da0 msgr2=0x7f96fc0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.629+0000 7f9701a00640 1 --2- 172.21.15.39:0/1191703235 >> v2:172.21.15.100:3300/0 conn(0x7f96fc0d0da0 0x7f96fc0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.629+0000 7f9701a00640 1 -- 172.21.15.39:0/1191703235 >> v1:172.21.15.39:6789/0 conn(0x7f96fc15db60 legacy=0x7f96fc0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.629+0000 7f9701a00640 1 -- 172.21.15.39:0/1191703235 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc0cd760 con 0x7f96fc15cbf0 2024-01-24T17:41:38.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.630+0000 7f9701a00640 1 --2- 172.21.15.39:0/1191703235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15cbf0 0x7f96fc0d03b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009790 tx=0x7f96ec02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3e66512e686761 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:38.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.630+0000 7f96fbe00640 1 -- 172.21.15.39:0/1191703235 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec02d040 con 0x7f96fc15cbf0 2024-01-24T17:41:38.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.630+0000 7f96fbe00640 1 -- 172.21.15.39:0/1191703235 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ec031810 con 0x7f96fc15cbf0 2024-01-24T17:41:38.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.630+0000 7f96fbe00640 1 -- 172.21.15.39:0/1191703235 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec031ba0 con 0x7f96fc15cbf0 2024-01-24T17:41:38.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.631+0000 7f9702e00640 1 -- 172.21.15.39:0/1191703235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15cbf0 msgr2=0x7f96fc0d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.631+0000 7f9702e00640 1 --2- 172.21.15.39:0/1191703235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15cbf0 0x7f96fc0d03b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009790 tx=0x7f96ec02b8b0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.631+0000 7f9702e00640 1 -- 172.21.15.39:0/1191703235 shutdown_connections 2024-01-24T17:41:38.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.631+0000 7f9702e00640 1 --2- 172.21.15.39:0/1191703235 >> v2:172.21.15.100:3300/0 conn(0x7f96fc0d0da0 0x7f96fc0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.631+0000 7f9702e00640 1 --2- 172.21.15.39:0/1191703235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15cbf0 0x7f96fc0d03b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.631+0000 7f9702e00640 1 -- 172.21.15.39:0/1191703235 >> 172.21.15.39:0/1191703235 conn(0x7f96fc09bb90 msgr2=0x7f96fc0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:38.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.631+0000 7f9702e00640 1 -- 172.21.15.39:0/1191703235 shutdown_connections 2024-01-24T17:41:38.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.631+0000 7f9702e00640 1 -- 172.21.15.39:0/1191703235 wait complete. 2024-01-24T17:41:38.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.633+0000 7f9702e00640 1 Processor -- start 2024-01-24T17:41:38.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.633+0000 7f9702e00640 1 -- start start 2024-01-24T17:41:38.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96fc0d0da0 0x7f96fc0cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96fc15cbf0 0x7f96fc0c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15db60 0x7f96fc0c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc0ca530 con 0x7f96fc15db60 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc0ca810 con 0x7f96fc15cbf0 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc0caaf0 con 0x7f96fc0d0da0 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15db60 0x7f96fc0c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15db60 0x7f96fc0c8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58216/0 (socket says 172.21.15.39:58216) 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9702400640 1 -- 172.21.15.39:0/1862628029 learned_addr learned my addr 172.21.15.39:0/1862628029 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9701a00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96fc0d0da0 0x7f96fc0cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.634+0000 7f9701000640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96fc15cbf0 0x7f96fc0c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.635+0000 7f9702400640 1 -- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96fc0d0da0 msgr2=0x7f96fc0cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.635+0000 7f9702400640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96fc0d0da0 0x7f96fc0cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.635+0000 7f9702400640 1 -- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96fc15cbf0 msgr2=0x7f96fc0c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.635+0000 7f9702400640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96fc15cbf0 0x7f96fc0c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.635+0000 7f9702400640 1 -- 172.21.15.39:0/1862628029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec0093a0 con 0x7f96fc15db60 2024-01-24T17:41:38.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.635+0000 7f9702400640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15db60 0x7f96fc0c8ad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96f4007e60 tx=0x7f96f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:38.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.635+0000 7f96f9600640 1 -- 172.21.15.39:0/1862628029 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4014420 con 0x7f96fc15db60 2024-01-24T17:41:38.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.636+0000 7f96f9600640 1 -- 172.21.15.39:0/1862628029 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f40145a0 con 0x7f96fc15db60 2024-01-24T17:41:38.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.636+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc0cadd0 con 0x7f96fc15db60 2024-01-24T17:41:38.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.636+0000 7f96f9600640 1 -- 172.21.15.39:0/1862628029 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4014950 con 0x7f96fc15db60 2024-01-24T17:41:38.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.636+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc1591c0 con 0x7f96fc15db60 2024-01-24T17:41:38.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.637+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc04dff0 con 0x7f96fc15db60 2024-01-24T17:41:38.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.638+0000 7f96f9600640 1 -- 172.21.15.39:0/1862628029 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f96f4014ab0 con 0x7f96fc15db60 2024-01-24T17:41:38.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.638+0000 7f96f9600640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f96d406a430 0x7f96d406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:38.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.639+0000 7f96f9600640 1 -- 172.21.15.39:0/1862628029 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96f40890f0 con 0x7f96fc15db60 2024-01-24T17:41:38.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.639+0000 7f9701a00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f96d406a430 0x7f96d406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:38.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.640+0000 7f9701a00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f96d406a430 0x7f96d406c8f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f96ec036000 tx=0x7f96ec03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:38.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.643+0000 7f96f9600640 1 -- 172.21.15.39:0/1862628029 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f96f4054780 con 0x7f96fc15db60 2024-01-24T17:41:38.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.885+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f96fc0a0490 con 0x7f96fc15db60 2024-01-24T17:41:38.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.894+0000 7f96f9600640 1 -- 172.21.15.39:0/1862628029 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+169969 (secure 0 0 0) 0x7f96f4057da0 con 0x7f96fc15db60 2024-01-24T17:41:38.897 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f96d406a430 msgr2=0x7f96d406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f96d406a430 0x7f96d406c8f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f96ec036000 tx=0x7f96ec03c040 comp rx=0 tx=0).stop 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15db60 msgr2=0x7f96fc0c8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15db60 0x7f96fc0c8ad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96f4007e60 tx=0x7f96f400e4b0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 shutdown_connections 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f96d406a430 0x7f96d406c8f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc15db60 0x7f96fc0c8ad0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96fc15cbf0 0x7f96fc0c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 --2- 172.21.15.39:0/1862628029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96fc0d0da0 0x7f96fc0cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:38.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 >> 172.21.15.39:0/1862628029 conn(0x7f96fc09bb90 msgr2=0x7f96fc06ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:38.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 shutdown_connections 2024-01-24T17:41:38.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:38.898+0000 7f9702e00640 1 -- 172.21.15.39:0/1862628029 wait complete. 2024-01-24T17:41:38.918 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":5,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6832","nonce":36263},{"type":"v1","addr":"172.21.15.100:6833","nonce":36263}]},"active_addr":"172.21.15.100:6833/36263","active_change":"2024-01-24T17:41:14.183787+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":2917038322}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":2854236919}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":2538848925}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":464311454}]}]} 2024-01-24T17:41:38.922 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-24T17:41:38.922 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-24T17:41:38.923 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.140+0000 7f6069200640 1 Processor -- start 2024-01-24T17:41:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.140+0000 7f6069200640 1 -- start start 2024-01-24T17:41:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.142+0000 7f6069200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606415a720 0x7f60640cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.142+0000 7f6069200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f60640ce8e0 0x7f60640ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.142+0000 7f6069200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60640cf290 con 0x7f606415ab00 2024-01-24T17:41:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.142+0000 7f6069200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60640ce400 con 0x7f60640ce8e0 2024-01-24T17:41:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.142+0000 7f6069200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60640ccef0 con 0x7f606415a720 2024-01-24T17:41:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.143+0000 7f6063400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606415a720 0x7f60640cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.143+0000 7f6063400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606415a720 0x7f60640cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:47960/0 (socket says 172.21.15.39:47960) 2024-01-24T17:41:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.143+0000 7f6063400640 1 -- 172.21.15.39:0/1090120216 learned_addr learned my addr 172.21.15.39:0/1090120216 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.143+0000 7f6063e00640 1 --2- 172.21.15.39:0/1090120216 >> v2:172.21.15.100:3300/0 conn(0x7f60640ce8e0 0x7f60640ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.143+0000 7f6063400640 1 -- 172.21.15.39:0/1090120216 >> v2:172.21.15.100:3300/0 conn(0x7f60640ce8e0 msgr2=0x7f60640ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.143+0000 7f6063400640 1 --2- 172.21.15.39:0/1090120216 >> v2:172.21.15.100:3300/0 conn(0x7f60640ce8e0 0x7f60640ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.143+0000 7f6063400640 1 -- 172.21.15.39:0/1090120216 >> v1:172.21.15.100:6789/0 conn(0x7f606415ab00 legacy=0x7f60640cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.143+0000 7f6063400640 1 -- 172.21.15.39:0/1090120216 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60640ce110 con 0x7f606415a720 2024-01-24T17:41:39.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.145+0000 7f6063400640 1 --2- 172.21.15.39:0/1090120216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606415a720 0x7f60640cdbd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6054009870 tx=0x7f605402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7683c2262e719471 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:39.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.145+0000 7f6062a00640 1 -- 172.21.15.39:0/1090120216 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054031640 con 0x7f606415a720 2024-01-24T17:41:39.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.145+0000 7f6062a00640 1 -- 172.21.15.39:0/1090120216 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60540317a0 con 0x7f606415a720 2024-01-24T17:41:39.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.145+0000 7f6062a00640 1 -- 172.21.15.39:0/1090120216 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6054031b30 con 0x7f606415a720 2024-01-24T17:41:39.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.146+0000 7f6069200640 1 -- 172.21.15.39:0/1090120216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606415a720 msgr2=0x7f60640cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.146+0000 7f6069200640 1 --2- 172.21.15.39:0/1090120216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606415a720 0x7f60640cdbd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6054009870 tx=0x7f605402b8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.146+0000 7f6069200640 1 -- 172.21.15.39:0/1090120216 shutdown_connections 2024-01-24T17:41:39.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.146+0000 7f6069200640 1 --2- 172.21.15.39:0/1090120216 >> v2:172.21.15.100:3300/0 conn(0x7f60640ce8e0 0x7f60640ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.146+0000 7f6069200640 1 --2- 172.21.15.39:0/1090120216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606415a720 0x7f60640cdbd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.146+0000 7f6069200640 1 -- 172.21.15.39:0/1090120216 >> 172.21.15.39:0/1090120216 conn(0x7f606406a090 msgr2=0x7f606406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:39.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.146+0000 7f6069200640 1 -- 172.21.15.39:0/1090120216 shutdown_connections 2024-01-24T17:41:39.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.146+0000 7f6069200640 1 -- 172.21.15.39:0/1090120216 wait complete. 2024-01-24T17:41:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.148+0000 7f6069200640 1 Processor -- start 2024-01-24T17:41:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.149+0000 7f6069200640 1 -- start start 2024-01-24T17:41:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f6069200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60640ce8e0 0x7f606414e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f6069200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606415a720 0x7f6064150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f6069200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606415ab00 0x7f60641580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f6069200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60641513e0 con 0x7f606415a720 2024-01-24T17:41:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f6069200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60641516c0 con 0x7f606415ab00 2024-01-24T17:41:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f6069200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60641519a0 con 0x7f60640ce8e0 2024-01-24T17:41:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f605be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606415a720 0x7f6064150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f605be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606415a720 0x7f6064150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58222/0 (socket says 172.21.15.39:58222) 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f605be00640 1 -- 172.21.15.39:0/1139859787 learned_addr learned my addr 172.21.15.39:0/1139859787 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f6063400640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60640ce8e0 0x7f606414e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.150+0000 7f6063e00640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606415ab00 0x7f60641580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605be00640 1 -- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60640ce8e0 msgr2=0x7f606414e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605be00640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60640ce8e0 0x7f606414e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605be00640 1 -- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606415ab00 msgr2=0x7f60641580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605be00640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606415ab00 0x7f60641580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605be00640 1 -- 172.21.15.39:0/1139859787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60540094d0 con 0x7f606415a720 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605be00640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606415a720 0x7f6064150a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f604c00c450 tx=0x7f604c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605b400640 1 -- 172.21.15.39:0/1139859787 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c0052d0 con 0x7f606415a720 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605b400640 1 -- 172.21.15.39:0/1139859787 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c005430 con 0x7f606415a720 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f605b400640 1 -- 172.21.15.39:0/1139859787 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c012870 con 0x7f606415a720 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.151+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6064151c80 con 0x7f606415a720 2024-01-24T17:41:39.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.152+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606414ed60 con 0x7f606415a720 2024-01-24T17:41:39.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.153+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60641521a0 con 0x7f606415a720 2024-01-24T17:41:39.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.154+0000 7f605b400640 1 -- 172.21.15.39:0/1139859787 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f604c0129d0 con 0x7f606415a720 2024-01-24T17:41:39.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.154+0000 7f605b400640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f603c06a430 0x7f603c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.155+0000 7f605b400640 1 -- 172.21.15.39:0/1139859787 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f604c088480 con 0x7f606415a720 2024-01-24T17:41:39.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.155+0000 7f6063400640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f603c06a430 0x7f603c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.156+0000 7f6063400640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f603c06a430 0x7f603c06c8f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6054036000 tx=0x7f605403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:39.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.159+0000 7f605b400640 1 -- 172.21.15.39:0/1139859787 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f60641521a0 con 0x7f606415a720 2024-01-24T17:41:39.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.354+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60641521a0 con 0x7f606415a720 2024-01-24T17:41:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.356+0000 7f605b400640 1 -- 172.21.15.39:0/1139859787 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f60641521a0 con 0x7f606415a720 2024-01-24T17:41:39.357 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:39.357 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:39.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f603c06a430 msgr2=0x7f603c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f603c06a430 0x7f603c06c8f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6054036000 tx=0x7f605403c040 comp rx=0 tx=0).stop 2024-01-24T17:41:39.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606415a720 msgr2=0x7f6064150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606415a720 0x7f6064150a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f604c00c450 tx=0x7f604c00c910 comp rx=0 tx=0).stop 2024-01-24T17:41:39.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 shutdown_connections 2024-01-24T17:41:39.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606415ab00 0x7f60641580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f603c06a430 0x7f603c06c8f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606415a720 0x7f6064150a00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 --2- 172.21.15.39:0/1139859787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60640ce8e0 0x7f606414e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 >> 172.21.15.39:0/1139859787 conn(0x7f606406a090 msgr2=0x7f60640ad740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:39.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 shutdown_connections 2024-01-24T17:41:39.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.359+0000 7f6069200640 1 -- 172.21.15.39:0/1139859787 wait complete. 2024-01-24T17:41:39.380 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-24T17:41:39.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:39.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.600+0000 7fc638000640 1 Processor -- start 2024-01-24T17:41:39.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.601+0000 7fc638000640 1 -- start start 2024-01-24T17:41:39.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.602+0000 7fc638000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6300cb3e0 0x7fc6300b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.602+0000 7fc638000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6300cb850 con 0x7fc6300caf50 2024-01-24T17:41:39.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.602+0000 7fc638000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6300c9180 con 0x7fc63015a950 2024-01-24T17:41:39.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.602+0000 7fc638000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300b43d0 con 0x7fc6300cb3e0 2024-01-24T17:41:39.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.602+0000 7fc634c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc6300caf50 0x7fc6300b1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52368/0 (socket says 172.21.15.39:52368) 2024-01-24T17:41:39.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.602+0000 7fc634c00640 1 -- 172.21.15.39:0/1903453110 learned_addr learned my addr 172.21.15.39:0/1903453110 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:39.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc636000640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6300cb3e0 0x7fc6300b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361146669 0 0) 0x7fc6300cb850 con 0x7fc6300caf50 2024-01-24T17:41:39.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc610003520 con 0x7fc6300caf50 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095140416 0 0) 0x7fc610003520 con 0x7fc6300caf50 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6300cb3e0 msgr2=0x7fc6300b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6300cb3e0 0x7fc6300b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 >> v1:172.21.15.100:6789/0 conn(0x7fc63015a950 legacy=0x7fc6300ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6300c9460 con 0x7fc6300caf50 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc618001670 con 0x7fc6300caf50 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 >> v1:172.21.15.39:6789/0 conn(0x7fc6300caf50 legacy=0x7fc6300b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6100039a0 0x7fc610003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6300cb3e0 0x7fc6100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc610004d00 0x7fc610005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300b43d0 con 0x7fc610004d00 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61000ba30 con 0x7fc6100039a0 2024-01-24T17:41:39.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.603+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61000bba0 con 0x7fc6300cb3e0 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc636000640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc610004d00 0x7fc610005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc635600640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6100039a0 0x7fc610003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc634c00640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6300cb3e0 0x7fc6100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc636000640 1 -- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6300cb3e0 msgr2=0x7fc6100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc636000640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6300cb3e0 0x7fc6100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc636000640 1 -- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6100039a0 msgr2=0x7fc610003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc636000640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6100039a0 0x7fc610003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc636000640 1 -- 172.21.15.39:0/1903453110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc610003520 con 0x7fc610004d00 2024-01-24T17:41:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.604+0000 7fc635600640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6100039a0 0x7fc610003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:39.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.605+0000 7fc636000640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc610004d00 0x7fc610005180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc628009fc0 tx=0x7fc6100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac9206f49bee975 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62802cc20 con 0x7fc610004d00 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62800f040 con 0x7fc610004d00 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc62fe00640 1 -- 172.21.15.39:0/1903453110 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc628034440 con 0x7fc610004d00 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc638000640 1 -- 172.21.15.39:0/1903453110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc610004d00 msgr2=0x7fc610005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc638000640 1 --2- 172.21.15.39:0/1903453110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc610004d00 0x7fc610005180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc628009fc0 tx=0x7fc6100057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc636000640 1 -- 172.21.15.39:0/1903453110 reap_dead start 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc638000640 1 -- 172.21.15.39:0/1903453110 shutdown_connections 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc638000640 1 -- 172.21.15.39:0/1903453110 >> 172.21.15.39:0/1903453110 conn(0x7fc63006a3e0 msgr2=0x7fc63006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc638000640 1 -- 172.21.15.39:0/1903453110 shutdown_connections 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.606+0000 7fc638000640 1 -- 172.21.15.39:0/1903453110 wait complete. 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.608+0000 7fc638000640 1 Processor -- start 2024-01-24T17:41:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.608+0000 7fc638000640 1 -- start start 2024-01-24T17:41:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.609+0000 7fc638000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc610004d00 0x7fc630151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.609+0000 7fc638000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6300ccb00 0x7fc6300b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.609+0000 7fc638000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300ccf80 0x7fc6300b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.609+0000 7fc638000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300b0980 con 0x7fc6300ccb00 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.609+0000 7fc638000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300b0e10 con 0x7fc6300ccf80 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.609+0000 7fc638000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300afd70 con 0x7fc610004d00 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6300ccb00 0x7fc6300b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6300ccb00 0x7fc6300b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58228/0 (socket says 172.21.15.39:58228) 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 -- 172.21.15.39:0/2476111966 learned_addr learned my addr 172.21.15.39:0/2476111966 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc636000640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300ccf80 0x7fc6300b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc635600640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc610004d00 0x7fc630151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 -- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc610004d00 msgr2=0x7fc630151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc610004d00 0x7fc630151be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 -- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300ccf80 msgr2=0x7fc6300b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300ccf80 0x7fc6300b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 -- 172.21.15.39:0/2476111966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6300b0050 con 0x7fc6300ccb00 2024-01-24T17:41:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.610+0000 7fc634c00640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6300ccb00 0x7fc6300b22e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc618001050 tx=0x7fc61800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.611+0000 7fc62d600640 1 -- 172.21.15.39:0/2476111966 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800eed0 con 0x7fc6300ccb00 2024-01-24T17:41:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.611+0000 7fc62d600640 1 -- 172.21.15.39:0/2476111966 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61800c870 con 0x7fc6300ccb00 2024-01-24T17:41:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.611+0000 7fc62d600640 1 -- 172.21.15.39:0/2476111966 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800cbe0 con 0x7fc6300ccb00 2024-01-24T17:41:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.611+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6300b0330 con 0x7fc6300ccb00 2024-01-24T17:41:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.611+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc630152120 con 0x7fc6300ccb00 2024-01-24T17:41:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.612+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6300b0af0 con 0x7fc6300ccb00 2024-01-24T17:41:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.613+0000 7fc62d600640 1 -- 172.21.15.39:0/2476111966 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc61800cd40 con 0x7fc6300ccb00 2024-01-24T17:41:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.614+0000 7fc62d600640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc60806a430 0x7fc60806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:39.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.614+0000 7fc62d600640 1 -- 172.21.15.39:0/2476111966 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc618088920 con 0x7fc6300ccb00 2024-01-24T17:41:39.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.615+0000 7fc635600640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc60806a430 0x7fc60806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.616+0000 7fc635600640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc60806a430 0x7fc60806c8f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc6300b1020 tx=0x7fc620009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:39.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.618+0000 7fc62d600640 1 -- 172.21.15.39:0/2476111966 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc618053f00 con 0x7fc6300ccb00 2024-01-24T17:41:39.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.813+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc630073ef0 con 0x7fc6300ccb00 2024-01-24T17:41:39.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.816+0000 7fc62d600640 1 -- 172.21.15.39:0/2476111966 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7fc618057520 con 0x7fc6300ccb00 2024-01-24T17:41:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:39.817 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:39.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.818+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc60806a430 msgr2=0x7fc60806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.818+0000 7fc638000640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc60806a430 0x7fc60806c8f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc6300b1020 tx=0x7fc620009040 comp rx=0 tx=0).stop 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.818+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6300ccb00 msgr2=0x7fc6300b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6300ccb00 0x7fc6300b22e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc618001050 tx=0x7fc61800ec80 comp rx=0 tx=0).stop 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 shutdown_connections 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6300ccf80 0x7fc6300b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc60806a430 0x7fc60806c8f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6300ccb00 0x7fc6300b22e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 --2- 172.21.15.39:0/2476111966 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc610004d00 0x7fc630151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 >> 172.21.15.39:0/2476111966 conn(0x7fc63006a3e0 msgr2=0x7fc630070740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 shutdown_connections 2024-01-24T17:41:39.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:39.819+0000 7fc638000640 1 -- 172.21.15.39:0/2476111966 wait complete. 2024-01-24T17:41:39.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-24T17:41:39.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-24T17:41:39.841 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-24T17:41:39.841 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-24T17:41:39.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-24T17:41:39.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-24T17:41:39.843 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-24T17:41:39.844 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-24T17:41:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.152+0000 7f1f29000640 1 Processor -- start 2024-01-24T17:41:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.152+0000 7f69caa00640 1 Processor -- start 2024-01-24T17:41:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.152+0000 7f1f29000640 1 -- start start 2024-01-24T17:41:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.152+0000 7f69caa00640 1 -- start start 2024-01-24T17:41:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.154+0000 7f69caa00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f69c415c9b0 0x7f69c40cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.154+0000 7f69caa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f69c415cd90 0x7f69c40d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.154+0000 7f69caa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c40d1350 0x7f69c40cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.154+0000 7f69caa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c40d17a0 con 0x7f69c415cd90 2024-01-24T17:41:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.154+0000 7f69caa00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c406b6f0 con 0x7f69c415c9b0 2024-01-24T17:41:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.154+0000 7f69caa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40af3a0 con 0x7f69c40d1350 2024-01-24T17:41:40.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.155+0000 7f1f29000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1f24156550 0x7f1f24159710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.155+0000 7f69c9600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f69c415c9b0 0x7f69c40cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.155+0000 7f69c8c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f69c415cd90 0x7f69c40d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.155+0000 7f69c8c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f69c415cd90 0x7f69c40d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58236/0 (socket says 172.21.15.39:58236) 2024-01-24T17:41:40.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.155+0000 7f69c8c00640 1 -- 172.21.15.39:0/2174803456 learned_addr learned my addr 172.21.15.39:0/2174803456 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.156+0000 7f69ca000640 1 --2- 172.21.15.39:0/2174803456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c40d1350 0x7f69c40cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.156+0000 7f69c8c00640 1 -- 172.21.15.39:0/2174803456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c40d1350 msgr2=0x7f69c40cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.156+0000 7f1f29000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1f240ce950 0x7f1f240cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.156+0000 7f69c8c00640 1 --2- 172.21.15.39:0/2174803456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c40d1350 0x7f69c40cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.156+0000 7f69c8c00640 1 -- 172.21.15.39:0/2174803456 >> v2:172.21.15.100:3300/0 conn(0x7f69c415c9b0 msgr2=0x7f69c40cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.156+0000 7f1f29000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f240c9780 con 0x7f1f24159c50 2024-01-24T17:41:40.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.157+0000 7f69c8c00640 1 --2- 172.21.15.39:0/2174803456 >> v2:172.21.15.100:3300/0 conn(0x7f69c415c9b0 0x7f69c40cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.157+0000 7f1f29000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f240c8b70 con 0x7f1f240ce950 2024-01-24T17:41:40.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.157+0000 7f1f29000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f240c8e50 con 0x7f1f24156550 2024-01-24T17:41:40.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.157+0000 7f1f22a00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1f24159c50 0x7f1f240ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48668/0 (socket says 172.21.15.39:48668) 2024-01-24T17:41:40.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.157+0000 7f1f22a00640 1 -- 172.21.15.39:0/1618421792 learned_addr learned my addr 172.21.15.39:0/1618421792 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.157+0000 7f69c8c00640 1 -- 172.21.15.39:0/2174803456 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c40d0a60 con 0x7f69c415cd90 2024-01-24T17:41:40.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.157+0000 7f1f23e00640 1 --2- 172.21.15.39:0/1618421792 >> v2:172.21.15.39:3300/0 conn(0x7f1f240ce950 0x7f1f240cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.157+0000 7f1f23400640 1 --2- 172.21.15.39:0/1618421792 >> v2:172.21.15.100:3300/0 conn(0x7f1f24156550 0x7f1f24159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.158+0000 7f69c9600640 1 --2- 172.21.15.39:0/2174803456 >> v2:172.21.15.100:3300/0 conn(0x7f69c415c9b0 0x7f69c40cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:40.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.159+0000 7f69c8c00640 1 --2- 172.21.15.39:0/2174803456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c415cd90 0x7f69c40d0e10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69c408b610 tx=0x7f69bc02be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30317d4b4195553 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.158+0000 7f1f23400640 1 -- 172.21.15.39:0/1618421792 >> v2:172.21.15.39:3300/0 conn(0x7f1f240ce950 msgr2=0x7f1f240cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.158+0000 7f1f23400640 1 --2- 172.21.15.39:0/1618421792 >> v2:172.21.15.39:3300/0 conn(0x7f1f240ce950 0x7f1f240cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.158+0000 7f1f23400640 1 -- 172.21.15.39:0/1618421792 >> v1:172.21.15.100:6789/0 conn(0x7f1f24159c50 legacy=0x7f1f240ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.158+0000 7f1f23400640 1 -- 172.21.15.39:0/1618421792 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240c9130 con 0x7f1f24156550 2024-01-24T17:41:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.160+0000 7f69c3e00640 1 -- 172.21.15.39:0/2174803456 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc02ca10 con 0x7f69c415cd90 2024-01-24T17:41:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.160+0000 7f69c3e00640 1 -- 172.21.15.39:0/2174803456 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc02cb70 con 0x7f69c415cd90 2024-01-24T17:41:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.160+0000 7f69c3e00640 1 -- 172.21.15.39:0/2174803456 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0343f0 con 0x7f69c415cd90 2024-01-24T17:41:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.161+0000 7f69caa00640 1 -- 172.21.15.39:0/2174803456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c415cd90 msgr2=0x7f69c40d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.161+0000 7f69caa00640 1 --2- 172.21.15.39:0/2174803456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c415cd90 0x7f69c40d0e10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69c408b610 tx=0x7f69bc02be70 comp rx=0 tx=0).stop 2024-01-24T17:41:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.161+0000 7f1f23400640 1 --2- 172.21.15.39:0/1618421792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f24156550 0x7f1f24159710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1f240cf280 tx=0x7f1f1c02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1f127fa372e3629 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.161+0000 7f69caa00640 1 -- 172.21.15.39:0/2174803456 shutdown_connections 2024-01-24T17:41:40.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.162+0000 7f69caa00640 1 --2- 172.21.15.39:0/2174803456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c40d1350 0x7f69c40cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.162+0000 7f69caa00640 1 --2- 172.21.15.39:0/2174803456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c415cd90 0x7f69c40d0e10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.162+0000 7f69caa00640 1 --2- 172.21.15.39:0/2174803456 >> v2:172.21.15.100:3300/0 conn(0x7f69c415c9b0 0x7f69c40cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.162+0000 7f69caa00640 1 -- 172.21.15.39:0/2174803456 >> 172.21.15.39:0/2174803456 conn(0x7f69c40d2830 msgr2=0x7f69c40d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.162+0000 7f69caa00640 1 -- 172.21.15.39:0/2174803456 shutdown_connections 2024-01-24T17:41:40.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.163+0000 7f69caa00640 1 -- 172.21.15.39:0/2174803456 wait complete. 2024-01-24T17:41:40.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.162+0000 7f1f22000640 1 -- 172.21.15.39:0/1618421792 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c02ebe0 con 0x7f1f24156550 2024-01-24T17:41:40.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.163+0000 7f1f22000640 1 -- 172.21.15.39:0/1618421792 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c02ed40 con 0x7f1f24156550 2024-01-24T17:41:40.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.163+0000 7f1f22000640 1 -- 172.21.15.39:0/1618421792 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c008660 con 0x7f1f24156550 2024-01-24T17:41:40.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.164+0000 7f1f29000640 1 -- 172.21.15.39:0/1618421792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f24156550 msgr2=0x7f1f24159710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.164+0000 7f1f29000640 1 --2- 172.21.15.39:0/1618421792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f24156550 0x7f1f24159710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1f240cf280 tx=0x7f1f1c02e040 comp rx=0 tx=0).stop 2024-01-24T17:41:40.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.163+0000 7fbf12400640 1 Processor -- start 2024-01-24T17:41:40.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.163+0000 7fbf12400640 1 -- start start 2024-01-24T17:41:40.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7f1f29000640 1 -- 172.21.15.39:0/1618421792 shutdown_connections 2024-01-24T17:41:40.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7f1f29000640 1 --2- 172.21.15.39:0/1618421792 >> v2:172.21.15.39:3300/0 conn(0x7f1f240ce950 0x7f1f240cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7f1f29000640 1 --2- 172.21.15.39:0/1618421792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f24156550 0x7f1f24159710 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7f1f29000640 1 -- 172.21.15.39:0/1618421792 >> 172.21.15.39:0/1618421792 conn(0x7f1f2406c220 msgr2=0x7f1f2406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7f69caa00640 1 Processor -- start 2024-01-24T17:41:40.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7f1f29000640 1 -- 172.21.15.39:0/1618421792 shutdown_connections 2024-01-24T17:41:40.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7f1f29000640 1 -- 172.21.15.39:0/1618421792 wait complete. 2024-01-24T17:41:40.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7fbf12400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbf0c160b80 0x7fbf0c164a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7fbf12400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbf0c0cbad0 0x7fbf0c06cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7fbf12400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0c0cde40 con 0x7fbf0c06ca70 2024-01-24T17:41:40.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7fbf12400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c06c650 con 0x7fbf0c160b80 2024-01-24T17:41:40.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.165+0000 7fbf12400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c06c2b0 con 0x7fbf0c0cbad0 2024-01-24T17:41:40.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.166+0000 7f69caa00640 1 -- start start 2024-01-24T17:41:40.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.166+0000 7fa04a600640 1 Processor -- start 2024-01-24T17:41:40.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.166+0000 7fa04a600640 1 -- start start 2024-01-24T17:41:40.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.166+0000 7fbf0aa00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fbf0c06ca70 0x7fbf0c0cb3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52376/0 (socket says 172.21.15.39:52376) 2024-01-24T17:41:40.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.167+0000 7fbf0aa00640 1 -- 172.21.15.39:0/4139484862 learned_addr learned my addr 172.21.15.39:0/4139484862 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.167+0000 7f69caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c40d1350 0x7f69c40cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.167+0000 7f69caa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c415c9b0 0x7f69c40b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.167+0000 7f69caa00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c415cd90 0x7f69c41594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.167+0000 7f69caa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c4159f60 con 0x7f69c40d1350 2024-01-24T17:41:40.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.167+0000 7f69caa00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40cb110 con 0x7f69c415cd90 2024-01-24T17:41:40.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.167+0000 7f69caa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40cb3f0 con 0x7f69c415c9b0 2024-01-24T17:41:40.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7fbf0b400640 1 --2- 172.21.15.39:0/4139484862 >> v2:172.21.15.39:3300/0 conn(0x7fbf0c160b80 0x7fbf0c164a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7fbf0be00640 1 --2- 172.21.15.39:0/4139484862 >> v2:172.21.15.100:3300/0 conn(0x7fbf0c0cbad0 0x7fbf0c06cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445376355 0 0) 0x7fbf0c0cde40 con 0x7fbf0c06ca70 2024-01-24T17:41:40.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec003520 con 0x7fbf0c06ca70 2024-01-24T17:41:40.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7fa04a600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa03c0544b0 0x7fa03c038920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7fa04a600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c0566a0 con 0x7fa03c0e46a0 2024-01-24T17:41:40.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7fa04a600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c056980 con 0x7fa03c038e60 2024-01-24T17:41:40.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7fa04a600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa03c039310 con 0x7fa03c0544b0 2024-01-24T17:41:40.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3110579604 0 0) 0x7fbeec003520 con 0x7fbf0c06ca70 2024-01-24T17:41:40.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 >> v2:172.21.15.100:3300/0 conn(0x7fbf0c0cbad0 msgr2=0x7fbf0c06cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 --2- 172.21.15.39:0/4139484862 >> v2:172.21.15.100:3300/0 conn(0x7fbf0c0cbad0 0x7fbf0c06cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 >> v2:172.21.15.39:3300/0 conn(0x7fbf0c160b80 msgr2=0x7fbf0c164a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 --2- 172.21.15.39:0/4139484862 >> v2:172.21.15.39:3300/0 conn(0x7fbf0c160b80 0x7fbf0c164a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fa049200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa03c0e46a0 0x7fa03c051d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52388/0 (socket says 172.21.15.39:52388) 2024-01-24T17:41:40.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fa049c00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fa03c038e60 0x7fa03c03a690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48676/0 (socket says 172.21.15.39:48676) 2024-01-24T17:41:40.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fa049c00640 1 -- 172.21.15.39:0/3155144785 learned_addr learned my addr 172.21.15.39:0/3155144785 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c164f90 con 0x7fbf0c06ca70 2024-01-24T17:41:40.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fbefc001670 con 0x7fbf0c06ca70 2024-01-24T17:41:40.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.168+0000 7f1f29000640 1 Processor -- start 2024-01-24T17:41:40.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7f1f29000640 1 -- start start 2024-01-24T17:41:40.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fa043e00640 1 --2- 172.21.15.39:0/3155144785 >> v2:172.21.15.100:3300/0 conn(0x7fa03c0544b0 0x7fa03c038920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7f69c8c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c415c9b0 0x7f69c40b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 >> v1:172.21.15.39:6789/0 conn(0x7fbf0c06ca70 legacy=0x7fbf0c0cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeec003970 0x7fbeec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec0047c0 0x7fbeec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeec005180 0x7fbeec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c06c650 con 0x7fbeec003970 2024-01-24T17:41:40.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c06c2b0 con 0x7fbeec005180 2024-01-24T17:41:40.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec00bf10 con 0x7fbeec0047c0 2024-01-24T17:41:40.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7fbf0be00640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeec005180 0x7fbeec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7fbf0aa00640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec0047c0 0x7fbeec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f69ca000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c415cd90 0x7f69c41594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f69ca000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c415cd90 0x7f69c41594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49252/0 (socket says 172.21.15.39:49252) 2024-01-24T17:41:40.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f69ca000640 1 -- 172.21.15.39:0/3897544962 learned_addr learned my addr 172.21.15.39:0/3897544962 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.169+0000 7f69c9600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c40d1350 0x7f69c40cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7fa043e00640 1 -- 172.21.15.39:0/3155144785 >> v1:172.21.15.100:6789/0 conn(0x7fa03c038e60 legacy=0x7fa03c03a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7fa043e00640 1 -- 172.21.15.39:0/3155144785 >> v1:172.21.15.39:6789/0 conn(0x7fa03c0e46a0 legacy=0x7fa03c051d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7fa043e00640 1 -- 172.21.15.39:0/3155144785 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c056c60 con 0x7fa03c0544b0 2024-01-24T17:41:40.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7fbf0be00640 1 -- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec0047c0 msgr2=0x7fbeec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7fbf0be00640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec0047c0 0x7fbeec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7fbf0be00640 1 -- 172.21.15.39:0/4139484862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeec003970 msgr2=0x7fbeec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7fbf0be00640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeec003970 0x7fbeec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7fbf0be00640 1 -- 172.21.15.39:0/4139484862 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c164f90 con 0x7fbeec005180 2024-01-24T17:41:40.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f1f29000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f240ce950 0x7f1f24154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f1f29000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f24156550 0x7f1f240c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f1f29000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f24159c50 0x7f1f2415b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f1f29000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f241548f0 con 0x7f1f240ce950 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f1f29000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240c5d20 con 0x7f1f24156550 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.170+0000 7f1f29000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f240c6000 con 0x7f1f24159c50 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7f1f23e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f24159c50 0x7f1f2415b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7f1f23e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f24159c50 0x7f1f2415b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48034/0 (socket says 172.21.15.39:48034) 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7f1f23e00640 1 -- 172.21.15.39:0/2223869276 learned_addr learned my addr 172.21.15.39:0/2223869276 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.171+0000 7f1f23400640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f240ce950 0x7f1f24154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f1f22a00640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f24156550 0x7f1f240c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f8162c00640 1 Processor -- start 2024-01-24T17:41:40.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f8162c00640 1 -- start start 2024-01-24T17:41:40.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f69c9600640 1 -- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c415c9b0 msgr2=0x7f69c40b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f69c9600640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f69c415c9b0 0x7f69c40b1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f69c9600640 1 -- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c415cd90 msgr2=0x7f69c41594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f69c9600640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f69c415cd90 0x7f69c41594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f69c9600640 1 -- 172.21.15.39:0/3897544962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc009b30 con 0x7f69c40d1350 2024-01-24T17:41:40.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7fbf0be00640 1 -- 172.21.15.39:0/4139484862 reap_dead start 2024-01-24T17:41:40.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f1f23400640 1 -- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f24159c50 msgr2=0x7f1f2415b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7f69c9600640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c40d1350 0x7f69c40cbbf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69b400c5e0 tx=0x7f69b400caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f1f23400640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f24159c50 0x7f1f2415b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f1f23400640 1 -- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f24156550 msgr2=0x7f1f240c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f1f23400640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f24156550 0x7f1f240c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.172+0000 7f1f23400640 1 -- 172.21.15.39:0/2223869276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c00ab80 con 0x7f1f240ce950 2024-01-24T17:41:40.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7fa043e00640 1 --2- 172.21.15.39:0/3155144785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa03c0544b0 0x7fa03c038920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa04402c840 tx=0x7fa0440a5970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42630f54e17bf69e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7fbf0be00640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeec005180 0x7fbeec00b980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009fc0 tx=0x7fbeec0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6caefe5df6c158e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4011800 con 0x7f69c40d1350 2024-01-24T17:41:40.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b4011960 con 0x7f69c40d1350 2024-01-24T17:41:40.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4011d30 con 0x7f69c40d1350 2024-01-24T17:41:40.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7fa043400640 1 -- 172.21.15.39:0/3155144785 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0440ab720 con 0x7fa03c0544b0 2024-01-24T17:41:40.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c40cb6d0 con 0x7f69c40d1350 2024-01-24T17:41:40.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0402cc40 con 0x7fbeec005180 2024-01-24T17:41:40.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.173+0000 7fa043400640 1 -- 172.21.15.39:0/3155144785 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0440ab880 con 0x7fa03c0544b0 2024-01-24T17:41:40.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7feb9da00640 1 Processor -- start 2024-01-24T17:41:40.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7f8162c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f815c0aff40 0x7f815c158740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7f8162c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0ce3c0 0x7f815c0ce870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7f8162c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815c0ccac0 con 0x7f815c0cbaf0 2024-01-24T17:41:40.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7f8162c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f815c0cedb0 con 0x7f815c0aff40 2024-01-24T17:41:40.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7f8162c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c0c9190 con 0x7f815c0ce3c0 2024-01-24T17:41:40.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0400f040 con 0x7fbeec005180 2024-01-24T17:41:40.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7fbf0a000640 1 -- 172.21.15.39:0/4139484862 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04003990 con 0x7fbeec005180 2024-01-24T17:41:40.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7fa043400640 1 -- 172.21.15.39:0/3155144785 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0440abc70 con 0x7fa03c0544b0 2024-01-24T17:41:40.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.175+0000 7f8161800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f815c0aff40 0x7f815c158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.175+0000 7f8161800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f815c0aff40 0x7f815c158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49286/0 (socket says 172.21.15.39:49286) 2024-01-24T17:41:40.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.175+0000 7f8161800640 1 -- 172.21.15.39:0/1761912775 learned_addr learned my addr 172.21.15.39:0/1761912775 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.175+0000 7f1f23400640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f240ce950 0x7f1f24154b10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c034000 tx=0x7f1f1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.174+0000 7feb9da00640 1 -- start start 2024-01-24T17:41:40.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.175+0000 7f8162200640 1 --2- 172.21.15.39:0/1761912775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0ce3c0 0x7f815c0ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7f8161800640 1 -- 172.21.15.39:0/1761912775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0ce3c0 msgr2=0x7f815c0ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.175+0000 7fa04a600640 1 -- 172.21.15.39:0/3155144785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa03c0544b0 msgr2=0x7fa03c038920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.175+0000 7fa04a600640 1 --2- 172.21.15.39:0/3155144785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa03c0544b0 0x7fa03c038920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa04402c840 tx=0x7fa0440a5970 comp rx=0 tx=0).stop 2024-01-24T17:41:40.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7fa04a600640 1 -- 172.21.15.39:0/3155144785 shutdown_connections 2024-01-24T17:41:40.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7fa04a600640 1 --2- 172.21.15.39:0/3155144785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa03c0544b0 0x7fa03c038920 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7fa04a600640 1 -- 172.21.15.39:0/3155144785 >> 172.21.15.39:0/3155144785 conn(0x7fa03c053920 msgr2=0x7fa03c053d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.175+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c415a240 con 0x7f69c40d1350 2024-01-24T17:41:40.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c04f400 con 0x7f1f240ce950 2024-01-24T17:41:40.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c04f560 con 0x7f1f240ce950 2024-01-24T17:41:40.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c04f910 con 0x7f1f240ce950 2024-01-24T17:41:40.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7fa04a600640 1 -- 172.21.15.39:0/3155144785 shutdown_connections 2024-01-24T17:41:40.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7f8161800640 1 --2- 172.21.15.39:0/1761912775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0ce3c0 0x7f815c0ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7f8161800640 1 -- 172.21.15.39:0/1761912775 >> v1:172.21.15.39:6789/0 conn(0x7f815c0cbaf0 legacy=0x7f815c0c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb9da00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7feb980b2c60 0x7feb980b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb9da00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb980d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f69c406b0a0 con 0x7f69c40d1350 2024-01-24T17:41:40.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f240c62e0 con 0x7f1f240ce950 2024-01-24T17:41:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.177+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f240c68a0 con 0x7f1f240ce950 2024-01-24T17:41:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7fbf12400640 1 -- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeec005180 msgr2=0x7fbeec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7fbf12400640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeec005180 0x7fbeec00b980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009fc0 tx=0x7fbeec0057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7fa04a600640 1 -- 172.21.15.39:0/3155144785 wait complete. 2024-01-24T17:41:40.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7f8161800640 1 -- 172.21.15.39:0/1761912775 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c0700c0 con 0x7f815c0aff40 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb9da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb980b3040 con 0x7feb980c59b0 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb9da00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb980d15e0 con 0x7feb980b2c60 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb9da00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb980cd100 con 0x7feb980c62b0 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb97e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7feb980b2c60 0x7feb980b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb97e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7feb980b2c60 0x7feb980b33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58300/0 (socket says 172.21.15.39:58300) 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb97e00640 1 -- 172.21.15.39:0/1905146039 learned_addr learned my addr 172.21.15.39:0/1905146039 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.176+0000 7feb9d000640 1 --1- 172.21.15.39:0/1905146039 >> v1:172.21.15.39:6789/0 conn(0x7feb980c59b0 0x7feb980cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52416/0 (socket says 172.21.15.39:52416) 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.177+0000 7feb8f200640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb980d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.177+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086887997 0 0) 0x7feb980b3040 con 0x7feb980c59b0 2024-01-24T17:41:40.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.177+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb78003520 con 0x7feb980c59b0 2024-01-24T17:41:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 359850043 0 0) 0x7feb78003520 con 0x7feb980c59b0 2024-01-24T17:41:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 msgr2=0x7feb980d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb980d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 >> v2:172.21.15.39:3300/0 conn(0x7feb980b2c60 msgr2=0x7feb980b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 --2- 172.21.15.39:0/1905146039 >> v2:172.21.15.39:3300/0 conn(0x7feb980b2c60 0x7feb980b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb980d0d40 con 0x7feb980c59b0 2024-01-24T17:41:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.177+0000 7f46df200640 1 Processor -- start 2024-01-24T17:41:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7fbf12400640 1 -- 172.21.15.39:0/4139484862 shutdown_connections 2024-01-24T17:41:40.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7fbf12400640 1 --2- 172.21.15.39:0/4139484862 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbeec005180 0x7fbeec00b980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7fbf12400640 1 -- 172.21.15.39:0/4139484862 >> 172.21.15.39:0/4139484862 conn(0x7fbf0c06a090 msgr2=0x7fbf0c06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.177+0000 7f46df200640 1 -- start start 2024-01-24T17:41:40.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7feb90001a40 con 0x7feb980c59b0 2024-01-24T17:41:40.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 >> v1:172.21.15.39:6789/0 conn(0x7feb980c59b0 legacy=0x7feb980cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7f8161800640 1 --2- 172.21.15.39:0/1761912775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f815c0aff40 0x7f815c158740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f81440098a0 tx=0x7f814402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24a75a513b98374f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb78003970 0x7feb78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78004d00 0x7feb78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb980d15e0 con 0x7feb78004d00 2024-01-24T17:41:40.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7f46df200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f46d80e77b0 0x7f46d8034410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7fa04a600640 1 Processor -- start 2024-01-24T17:41:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7f815be00640 1 -- 172.21.15.39:0/1761912775 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8144030860 con 0x7f815c0aff40 2024-01-24T17:41:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7f815be00640 1 -- 172.21.15.39:0/1761912775 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81440309c0 con 0x7f815c0aff40 2024-01-24T17:41:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7f815be00640 1 -- 172.21.15.39:0/1761912775 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8144030d90 con 0x7f815c0aff40 2024-01-24T17:41:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb980cd100 con 0x7feb78003970 2024-01-24T17:41:40.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7f46df200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 0x7f46d8049010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.178+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7800b980 con 0x7feb980c62b0 2024-01-24T17:41:40.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7fa04a600640 1 -- start start 2024-01-24T17:41:40.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7feb8f200640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7feb97e00640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb78003970 0x7feb78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7f46df200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d8053010 con 0x7f46d80540b0 2024-01-24T17:41:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7f46df200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d80e7b90 con 0x7f46d80e77b0 2024-01-24T17:41:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7f46df200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80532f0 con 0x7f46d8048bb0 2024-01-24T17:41:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7feb8f200640 1 -- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb78003970 msgr2=0x7feb78003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7f46dde00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f46d80e77b0 0x7f46d8034410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1f24156930 con 0x7f1f240ce950 2024-01-24T17:41:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7feb8f200640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb78003970 0x7feb78003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7fbf12400640 1 -- 172.21.15.39:0/4139484862 shutdown_connections 2024-01-24T17:41:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7fbf12400640 1 -- 172.21.15.39:0/4139484862 wait complete. 2024-01-24T17:41:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7feb9d000640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78004d00 0x7feb78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7feb8f200640 1 -- 172.21.15.39:0/1905146039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78004d00 msgr2=0x7feb78005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7feb8f200640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78004d00 0x7feb78005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7feb8f200640 1 -- 172.21.15.39:0/1905146039 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78003520 con 0x7feb980c62b0 2024-01-24T17:41:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f46dde00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f46d80e77b0 0x7f46d8034410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49302/0 (socket says 172.21.15.39:49302) 2024-01-24T17:41:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.179+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f69b401a450 con 0x7f69c40d1350 2024-01-24T17:41:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.180+0000 7f69c1600640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f699c06a480 0x7f699c06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7feb9d000640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78004d00 0x7feb78005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f46dde00640 1 -- 172.21.15.39:0/690580592 learned_addr learned my addr 172.21.15.39:0/690580592 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f46dd400640 1 --2- 172.21.15.39:0/690580592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 0x7f46d8049010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7feac6e00640 1 Processor -- start 2024-01-24T17:41:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7fa04a600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa03c038e60 0x7fa03c185ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7feac6e00640 1 -- start start 2024-01-24T17:41:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7fa04a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa03c0544b0 0x7fa03c04e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7fa04a600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa03c0e46a0 0x7fa03c009050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f69c8c00640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f699c06a480 0x7f699c06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7fa04a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c178db0 con 0x7fa03c0544b0 2024-01-24T17:41:40.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7fa04a600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c0509d0 con 0x7fa03c0e46a0 2024-01-24T17:41:40.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7fa04a600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c04ed30 con 0x7fa03c038e60 2024-01-24T17:41:40.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7fbf12400640 1 Processor -- start 2024-01-24T17:41:40.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7fa049200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa03c038e60 0x7fa03c185ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7fa043e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa03c0544b0 0x7fa03c04e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7fa049200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa03c038e60 0x7fa03c185ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48102/0 (socket says 172.21.15.39:48102) 2024-01-24T17:41:40.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7fa049200640 1 -- 172.21.15.39:0/2809492536 learned_addr learned my addr 172.21.15.39:0/2809492536 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f8162c00640 1 -- 172.21.15.39:0/1761912775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f815c0aff40 msgr2=0x7f815c158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f8162c00640 1 --2- 172.21.15.39:0/1761912775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f815c0aff40 0x7f815c158740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f81440098a0 tx=0x7f814402b9a0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f8162c00640 1 -- 172.21.15.39:0/1761912775 shutdown_connections 2024-01-24T17:41:40.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f8162c00640 1 --2- 172.21.15.39:0/1761912775 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0ce3c0 0x7f815c0ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f8162c00640 1 --2- 172.21.15.39:0/1761912775 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f815c0aff40 0x7f815c158740 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.181+0000 7f8162c00640 1 -- 172.21.15.39:0/1761912775 >> 172.21.15.39:0/1761912775 conn(0x7f815c06c020 msgr2=0x7f815c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7fbf12400640 1 -- start start 2024-01-24T17:41:40.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7f8162c00640 1 -- 172.21.15.39:0/1761912775 shutdown_connections 2024-01-24T17:41:40.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7f8162c00640 1 -- 172.21.15.39:0/1761912775 wait complete. 2024-01-24T17:41:40.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fa049c00640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa03c0e46a0 0x7fa03c009050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7feb8f200640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb780047c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7feb80009a90 tx=0x7feb780057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6797928d81d077e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69b4057060 con 0x7f69c40d1350 2024-01-24T17:41:40.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fa043e00640 1 -- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa03c038e60 msgr2=0x7fa03c185ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fa043e00640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa03c038e60 0x7fa03c185ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fa043e00640 1 -- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa03c0e46a0 msgr2=0x7fa03c009050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fa043e00640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa03c0e46a0 0x7fa03c009050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fa043e00640 1 -- 172.21.15.39:0/2809492536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04404eee0 con 0x7fa03c0544b0 2024-01-24T17:41:40.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7f69c1600640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f699c0726a0 0x7f699c074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 --> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f699c075170 con 0x7f699c0726a0 2024-01-24T17:41:40.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7feac6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac00cbad0 0x7feac006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7feac6e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac00cde40 con 0x7feac0160b80 2024-01-24T17:41:40.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7feac6e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac006c2b0 con 0x7feac006ca70 2024-01-24T17:41:40.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7feac6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac006c650 con 0x7feac00cbad0 2024-01-24T17:41:40.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fa043e00640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa03c0544b0 0x7fa03c04e7f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa04404f260 tx=0x7fa044069a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69b40533a0 con 0x7f69c40d1350 2024-01-24T17:41:40.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fbf12400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec005180 0x7fbf0c1f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.183+0000 7fbf12400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf0c0b7c80 0x7fbf0c0b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7fbf12400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf0c0b16e0 0x7fbf0c0b1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7fbf12400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0b20a0 con 0x7fbf0c0b7c80 2024-01-24T17:41:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7fbf12400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c162e40 con 0x7fbf0c0b16e0 2024-01-24T17:41:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7fbf12400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c1630b0 con 0x7fbeec005180 2024-01-24T17:41:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8002c6f0 con 0x7feb980c62b0 2024-01-24T17:41:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8002c850 con 0x7feb980c62b0 2024-01-24T17:41:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7feb97400640 1 -- 172.21.15.39:0/1905146039 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8002cbe0 con 0x7feb980c62b0 2024-01-24T17:41:40.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7f46dde00640 1 -- 172.21.15.39:0/690580592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 msgr2=0x7f46d8049010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7f46dde00640 1 --2- 172.21.15.39:0/690580592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 0x7f46d8049010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7f46dde00640 1 -- 172.21.15.39:0/690580592 >> v1:172.21.15.39:6789/0 conn(0x7f46d80540b0 legacy=0x7f46d804f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.182+0000 7f46dde00640 1 -- 172.21.15.39:0/690580592 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d80535d0 con 0x7f46d80e77b0 2024-01-24T17:41:40.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7f69ca000640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f699c0726a0 0x7f699c074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7fbf0b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec005180 0x7fbf0c1f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7fbf0b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec005180 0x7fbf0c1f8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48108/0 (socket says 172.21.15.39:48108) 2024-01-24T17:41:40.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7fbf0b400640 1 -- 172.21.15.39:0/3449195227 learned_addr learned my addr 172.21.15.39:0/3449195227 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fbf0be00640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf0c0b16e0 0x7fbf0c0b1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044060aa0 con 0x7fa03c0544b0 2024-01-24T17:41:40.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa044060c00 con 0x7fa03c0544b0 2024-01-24T17:41:40.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0440b4420 con 0x7fa03c0544b0 2024-01-24T17:41:40.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa03c1869e0 con 0x7fa03c0544b0 2024-01-24T17:41:40.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7f69c8c00640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f699c06a480 0x7f699c06c940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f69bc035000 tx=0x7f69bc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1f1c04fa70 con 0x7f1f240ce950 2024-01-24T17:41:40.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fbf0b400640 1 -- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf0c0b16e0 msgr2=0x7fbf0c0b1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fbf0b400640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf0c0b16e0 0x7fbf0c0b1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fbf0b400640 1 -- 172.21.15.39:0/3449195227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf0c0b7c80 msgr2=0x7fbf0c0b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fbf0b400640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf0c0b7c80 0x7fbf0c0b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fbf0b400640 1 -- 172.21.15.39:0/3449195227 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c163390 con 0x7fbeec005180 2024-01-24T17:41:40.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7feb9da00640 1 -- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 msgr2=0x7feb780047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7feb9da00640 1 --2- 172.21.15.39:0/1905146039 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb780047c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7feb80009a90 tx=0x7feb780057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7feac4e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac00cbad0 0x7feac006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7feac4e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac00cbad0 0x7feac006cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48106/0 (socket says 172.21.15.39:48106) 2024-01-24T17:41:40.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.184+0000 7feac4e00640 1 -- 172.21.15.39:0/138396154 learned_addr learned my addr 172.21.15.39:0/138396154 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7feac4e00640 1 -- 172.21.15.39:0/138396154 >> v1:172.21.15.100:6789/0 conn(0x7feac006ca70 legacy=0x7feac00cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7feac4e00640 1 -- 172.21.15.39:0/138396154 >> v1:172.21.15.39:6789/0 conn(0x7feac0160b80 legacy=0x7feac0164a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7feac4e00640 1 -- 172.21.15.39:0/138396154 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac0165230 con 0x7feac00cbad0 2024-01-24T17:41:40.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7fbf0b400640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec005180 0x7fbf0c1f8d60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbef400e800 tx=0x7fbef400ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03c186420 con 0x7fa03c0544b0 2024-01-24T17:41:40.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7f69ca000640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f699c0726a0 0x7f699c074ac0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7f46dde00640 1 --2- 172.21.15.39:0/690580592 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46d80e77b0 0x7f46d8034410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009870 tx=0x7f46d002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e12959293ea8a87 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7feb9d000640 1 -- 172.21.15.39:0/1905146039 reap_dead start 2024-01-24T17:41:40.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.185+0000 7f8162c00640 1 Processor -- start 2024-01-24T17:41:40.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7f8162c00640 1 -- start start 2024-01-24T17:41:40.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4005990 con 0x7fbeec005180 2024-01-24T17:41:40.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef4005b10 con 0x7fbeec005180 2024-01-24T17:41:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0c163670 con 0x7fbeec005180 2024-01-24T17:41:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.186+0000 7feb9da00640 1 -- 172.21.15.39:0/1905146039 shutdown_connections 2024-01-24T17:41:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7feb9da00640 1 -- 172.21.15.39:0/1905146039 >> 172.21.15.39:0/1905146039 conn(0x7feb98071ea0 msgr2=0x7feb9809bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7feb9da00640 1 -- 172.21.15.39:0/1905146039 shutdown_connections 2024-01-24T17:41:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0c1f92a0 con 0x7fbeec005180 2024-01-24T17:41:40.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa03c0e4a80 con 0x7fa03c0544b0 2024-01-24T17:41:40.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef400c940 con 0x7fbeec005180 2024-01-24T17:41:40.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7f1f1b400640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1efc06a480 0x7f1efc06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f1c0bbf40 con 0x7f1f240ce950 2024-01-24T17:41:40.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7feb9da00640 1 -- 172.21.15.39:0/1905146039 wait complete. 2024-01-24T17:41:40.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7f8162c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f815c0aff40 0x7f815c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f1f1b400640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f1efc0726a0 0x7f1efc074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7f8162c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0cbaf0 0x7f815c1f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7f8162c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f815c0ce3c0 0x7f815c156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7f8162c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c1f6330 con 0x7f815c0ce3c0 2024-01-24T17:41:40.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7f8162c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c158df0 con 0x7f815c0aff40 2024-01-24T17:41:40.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.187+0000 7f8162c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c1590d0 con 0x7f815c0cbaf0 2024-01-24T17:41:40.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f8160e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0cbaf0 0x7f815c1f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f8160e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0cbaf0 0x7f815c1f5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48122/0 (socket says 172.21.15.39:48122) 2024-01-24T17:41:40.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f8160e00640 1 -- 172.21.15.39:0/548453310 learned_addr learned my addr 172.21.15.39:0/548453310 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7feac4e00640 1 --2- 172.21.15.39:0/138396154 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac00cbad0 0x7feac006cef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feaa80098a0 tx=0x7feaa802b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3b705597aa70d7e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 <== osd.0 v2:172.21.15.39:6800/36720 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f699c075170 con 0x7f699c0726a0 2024-01-24T17:41:40.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f8161800640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f815c0aff40 0x7f815c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f8162200640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f815c0ce3c0 0x7f815c156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 --> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1efc075170 con 0x7f1efc0726a0 2024-01-24T17:41:40.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f1c0be520 con 0x7f1f240ce950 2024-01-24T17:41:40.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7f1f23e00640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f1efc0726a0 0x7f1efc074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.188+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbedc000fc0 con 0x7fbeec005180 2024-01-24T17:41:40.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7feabea00640 1 -- 172.21.15.39:0/138396154 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8030860 con 0x7feac00cbad0 2024-01-24T17:41:40.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7feabea00640 1 -- 172.21.15.39:0/138396154 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa80309c0 con 0x7feac00cbad0 2024-01-24T17:41:40.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7feabea00640 1 -- 172.21.15.39:0/138396154 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8030d90 con 0x7feac00cbad0 2024-01-24T17:41:40.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7f8160e00640 1 -- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f815c0aff40 msgr2=0x7f815c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7f8160e00640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f815c0aff40 0x7f815c0b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7f8160e00640 1 -- 172.21.15.39:0/548453310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f815c0ce3c0 msgr2=0x7f815c156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7f8160e00640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f815c0ce3c0 0x7f815c156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7f8160e00640 1 -- 172.21.15.39:0/548453310 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81440094d0 con 0x7f815c0cbaf0 2024-01-24T17:41:40.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7f1f22a00640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1efc06a480 0x7f1efc06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7f8162200640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f815c0ce3c0 0x7f815c156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:40.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.189+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa0440b4580 con 0x7fa03c0544b0 2024-01-24T17:41:40.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7feac6e00640 1 -- 172.21.15.39:0/138396154 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac00cbad0 msgr2=0x7feac006cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7feac6e00640 1 --2- 172.21.15.39:0/138396154 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac00cbad0 0x7feac006cef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feaa80098a0 tx=0x7feaa802b9a0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7f1f23e00640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f1efc0726a0 0x7f1efc074ac0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7f8160e00640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0cbaf0 0x7f815c1f5df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f814c00c480 tx=0x7f814c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7f46dca00640 1 -- 172.21.15.39:0/690580592 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0031640 con 0x7f46d80e77b0 2024-01-24T17:41:40.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7feac6e00640 1 -- 172.21.15.39:0/138396154 shutdown_connections 2024-01-24T17:41:40.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7feac6e00640 1 --2- 172.21.15.39:0/138396154 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac00cbad0 0x7feac006cef0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7feac6e00640 1 -- 172.21.15.39:0/138396154 >> 172.21.15.39:0/138396154 conn(0x7feac006a090 msgr2=0x7feac006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7feac6e00640 1 -- 172.21.15.39:0/138396154 shutdown_connections 2024-01-24T17:41:40.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.190+0000 7feac6e00640 1 -- 172.21.15.39:0/138396154 wait complete. 2024-01-24T17:41:40.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7f46dca00640 1 -- 172.21.15.39:0/690580592 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d00317a0 con 0x7f46d80e77b0 2024-01-24T17:41:40.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7f46dca00640 1 -- 172.21.15.39:0/690580592 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0031b30 con 0x7f46d80e77b0 2024-01-24T17:41:40.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7fa040c00640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa01c06a460 0x7fa01c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c0052d0 con 0x7f815c0cbaf0 2024-01-24T17:41:40.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814c005430 con 0x7f815c0cbaf0 2024-01-24T17:41:40.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c012870 con 0x7f815c0cbaf0 2024-01-24T17:41:40.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0440a7040 con 0x7fa03c0544b0 2024-01-24T17:41:40.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 <== osd.1 v2:172.21.15.39:6820/36723 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1efc075170 con 0x7f1efc0726a0 2024-01-24T17:41:40.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.191+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbef400caa0 con 0x7fbeec005180 2024-01-24T17:41:40.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7f1f22a00640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1efc06a480 0x7f1efc06c940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1f240c99f0 tx=0x7f1f1400b210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7feb9da00640 1 Processor -- start 2024-01-24T17:41:40.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7fa049200640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa01c06a460 0x7fa01c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7fbf03400640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee406a460 0x7fbee406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7feb9da00640 1 -- start start 2024-01-24T17:41:40.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7fa040c00640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fa01c072600 0x7fa01c074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 --> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa01c0750d0 con 0x7fa01c072600 2024-01-24T17:41:40.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0441292c0 con 0x7fa03c0544b0 2024-01-24T17:41:40.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7f46df200640 1 -- 172.21.15.39:0/690580592 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46d80e77b0 msgr2=0x7f46d8034410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7f46df200640 1 --2- 172.21.15.39:0/690580592 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46d80e77b0 0x7f46d8034410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009870 tx=0x7f46d002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbef4088f30 con 0x7fbeec005180 2024-01-24T17:41:40.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7fbf03400640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fbee4072600 0x7fbee4074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.192+0000 7fa049c00640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fa01c072600 0x7fa01c074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7feac6e00640 1 Processor -- start 2024-01-24T17:41:40.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7fbf0aa00640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee406a460 0x7fbee406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7f46df200640 1 -- 172.21.15.39:0/690580592 shutdown_connections 2024-01-24T17:41:40.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7f46df200640 1 --2- 172.21.15.39:0/690580592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 0x7f46d8049010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7f46df200640 1 --2- 172.21.15.39:0/690580592 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46d80e77b0 0x7f46d8034410 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009870 tx=0x7f46d002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7f46df200640 1 -- 172.21.15.39:0/690580592 >> 172.21.15.39:0/690580592 conn(0x7f46d800a550 msgr2=0x7f46d800a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7fa049c00640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fa01c072600 0x7fa01c074a20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7fa049200640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa01c06a460 0x7fa01c06c920 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa034000f60 tx=0x7fa034007f00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7f46df200640 1 -- 172.21.15.39:0/690580592 shutdown_connections 2024-01-24T17:41:40.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.193+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815c1f6560 con 0x7f815c0cbaf0 2024-01-24T17:41:40.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815c1f6b20 con 0x7f815c0cbaf0 2024-01-24T17:41:40.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7fbf0be00640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fbee4072600 0x7fbee4074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 --> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbee40750d0 con 0x7fbee4072600 2024-01-24T17:41:40.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbef4089380 con 0x7fbeec005180 2024-01-24T17:41:40.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7feb9da00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb981f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7feb9da00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb980c9830 0x7feb98161a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7feb9da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb98161fa0 0x7feb980ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7feb9da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb980ca840 con 0x7feb98161fa0 2024-01-24T17:41:40.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7feb9da00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb981f6280 con 0x7feb980c9830 2024-01-24T17:41:40.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7feb9da00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb980c9cc0 con 0x7feb980c62b0 2024-01-24T17:41:40.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.195+0000 7fbf0aa00640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee406a460 0x7fbee406c920 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0017e0 tx=0x7fbefc001490 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.195+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 <== osd.3 v2:172.21.15.39:6816/36725 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa01c0750d0 con 0x7fa01c072600 2024-01-24T17:41:40.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.195+0000 7feac6e00640 1 -- start start 2024-01-24T17:41:40.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.195+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f815c0c9690 con 0x7f815c0cbaf0 2024-01-24T17:41:40.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.196+0000 7feb8f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb980c9830 0x7feb98161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.196+0000 7feb9d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb98161fa0 0x7feb980ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.194+0000 7f46df200640 1 -- 172.21.15.39:0/690580592 wait complete. 2024-01-24T17:41:40.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.196+0000 7feb8f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb980c9830 0x7feb98161a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49332/0 (socket says 172.21.15.39:49332) 2024-01-24T17:41:40.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.196+0000 7feb8f200640 1 -- 172.21.15.39:0/2252091503 learned_addr learned my addr 172.21.15.39:0/2252091503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.196+0000 7fbf0be00640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fbee4072600 0x7fbee4074a20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7f46df200640 1 Processor -- start 2024-01-24T17:41:40.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feac6e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feac006ca70 0x7feac01554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feac6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00cbad0 0x7feac015cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feac6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac0160b80 0x7feac015d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feac6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01559f0 con 0x7feac00cbad0 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feac6e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac015d550 con 0x7feac006ca70 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feac6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac015d800 con 0x7feac0160b80 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feb97e00640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb981f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feb9d000640 1 -- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 msgr2=0x7feb981f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feb9d000640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb980c62b0 0x7feb981f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feb9d000640 1 -- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb980c9830 msgr2=0x7feb98161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feb9d000640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb980c9830 0x7feb98161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7feb9d000640 1 -- 172.21.15.39:0/2252091503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb980c9fa0 con 0x7feb98161fa0 2024-01-24T17:41:40.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.198+0000 7feb9d000640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb98161fa0 0x7feb980ce960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feb980c5d90 tx=0x7feb9000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.199+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f814c0129d0 con 0x7f815c0cbaf0 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.197+0000 7f46df200640 1 -- start start 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.198+0000 7f46df200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 0x7f46d80e3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.198+0000 7f46df200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46d80540b0 0x7f46d8185350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.198+0000 7f46df200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46d8004c00 0x7f46d8005080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.199+0000 7f46df200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8189fe0 con 0x7f46d8004c00 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.199+0000 7f46df200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d80055c0 con 0x7f46d80540b0 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.199+0000 7f46df200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8189110 con 0x7f46d8048bb0 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.199+0000 7f46de800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46d8004c00 0x7f46d8005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.199+0000 7f46de800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46d8004c00 0x7f46d8005080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58364/0 (socket says 172.21.15.39:58364) 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.199+0000 7f46de800640 1 -- 172.21.15.39:0/828320853 learned_addr learned my addr 172.21.15.39:0/828320853 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.199+0000 7f46dd400640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46d80540b0 0x7f46d8185350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7feabf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00cbad0 0x7feac015cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7feabf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00cbad0 0x7feac015cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58372/0 (socket says 172.21.15.39:58372) 2024-01-24T17:41:40.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7feabf400640 1 -- 172.21.15.39:0/2085663921 learned_addr learned my addr 172.21.15.39:0/2085663921 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7f46dde00640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 0x7f46d80e3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7f46de800640 1 -- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 msgr2=0x7f46d80e3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7f46de800640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46d8048bb0 0x7f46d80e3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7f46de800640 1 -- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46d80540b0 msgr2=0x7f46d8185350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7f46de800640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46d80540b0 0x7f46d8185350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7f46de800640 1 -- 172.21.15.39:0/828320853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d00094d0 con 0x7f46d8004c00 2024-01-24T17:41:40.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7feac4e00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac0160b80 0x7feac015d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.200+0000 7feabfe00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feac006ca70 0x7feac01554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 <== osd.2 v2:172.21.15.39:6808/36726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fbee40750d0 con 0x7fbee4072600 2024-01-24T17:41:40.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feabf400640 1 -- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac0160b80 msgr2=0x7feac015d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feabf400640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feac0160b80 0x7feac015d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feabf400640 1 -- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feac006ca70 msgr2=0x7feac01554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feabf400640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feac006ca70 0x7feac01554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feabf400640 1 -- 172.21.15.39:0/2085663921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa80094d0 con 0x7feac00cbad0 2024-01-24T17:41:40.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90004db0 con 0x7feb98161fa0 2024-01-24T17:41:40.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9000bb80 con 0x7feb98161fa0 2024-01-24T17:41:40.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90007110 con 0x7feb98161fa0 2024-01-24T17:41:40.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f8159600640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f813406a500 0x7f813406c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f814c088de0 con 0x7f815c0cbaf0 2024-01-24T17:41:40.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb980ca280 con 0x7feb98161fa0 2024-01-24T17:41:40.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb980ceea0 con 0x7feb98161fa0 2024-01-24T17:41:40.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f46de800640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46d8004c00 0x7f46d8005080 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46e0068990 tx=0x7f46e0066fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f8159600640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f81340726a0 0x7f8134074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 --> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8134075170 con 0x7f81340726a0 2024-01-24T17:41:40.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f8161800640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f813406a500 0x7f813406c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0073570 con 0x7f46d8004c00 2024-01-24T17:41:40.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e00736f0 con 0x7f46d8004c00 2024-01-24T17:41:40.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0092470 con 0x7f46d8004c00 2024-01-24T17:41:40.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.201+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d8189360 con 0x7f46d8004c00 2024-01-24T17:41:40.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.202+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d8189920 con 0x7f46d8004c00 2024-01-24T17:41:40.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.202+0000 7feabf400640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00cbad0 0x7feac015cad0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feab000c450 tx=0x7feab000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.203+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb980d1300 con 0x7feb98161fa0 2024-01-24T17:41:40.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.203+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f46d8053b90 con 0x7f46d8004c00 2024-01-24T17:41:40.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.203+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f814c054440 con 0x7f815c0cbaf0 2024-01-24T17:41:40.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.203+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab00052d0 con 0x7feac00cbad0 2024-01-24T17:41:40.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.204+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0005430 con 0x7feac00cbad0 2024-01-24T17:41:40.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.204+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab00052d0 con 0x7feac00cbad0 2024-01-24T17:41:40.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.206+0000 7f8162200640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f81340726a0 0x7f8134074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7f8162200640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f81340726a0 0x7f8134074ac0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.205+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7feb90007270 con 0x7feb98161fa0 2024-01-24T17:41:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.205+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac015dae0 con 0x7feac00cbad0 2024-01-24T17:41:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.205+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac015e0a0 con 0x7feac00cbad0 2024-01-24T17:41:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feac01657f0 con 0x7feac00cbad0 2024-01-24T17:41:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7feb94c00640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb7006a500 0x7feb7006c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb90088a80 con 0x7feb98161fa0 2024-01-24T17:41:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7feb94c00640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7feb70072720 0x7feb70074b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 --> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb700751f0 con 0x7feb70072720 2024-01-24T17:41:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb9008d2a0 con 0x7feb98161fa0 2024-01-24T17:41:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.206+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f46e008f070 con 0x7f46d8004c00 2024-01-24T17:41:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.208+0000 7feb97e00640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb7006a500 0x7feb7006c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7f46d6000640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46b806a430 0x7f46b806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46e0102260 con 0x7f46d8004c00 2024-01-24T17:41:40.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7f46d6000640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f46b80725d0 0x7f46b80749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.208+0000 7feb8f200640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7feb70072720 0x7feb70074b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46b80750a0 con 0x7f46b80725d0 2024-01-24T17:41:40.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.207+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46e01026b0 con 0x7f46d8004c00 2024-01-24T17:41:40.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.208+0000 7f46dd400640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f46b80725d0 0x7f46b80749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.209+0000 7f46dde00640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46b806a430 0x7f46b806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.208+0000 7f8161800640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f813406a500 0x7f813406c9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8144036000 tx=0x7f8144008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.210+0000 7f46dd400640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f46b80725d0 0x7f46b80749f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.210+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 <== osd.4 v2:172.21.15.100:6800/36832 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8134075170 con 0x7f81340726a0 2024-01-24T17:41:40.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.211+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7feab0012630 con 0x7feac00cbad0 2024-01-24T17:41:40.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.212+0000 7feab7e00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea9806a490 0x7fea9806c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.212+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 <== osd.6 v2:172.21.15.100:6808/36833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f46b80750a0 con 0x7f46b80725d0 2024-01-24T17:41:40.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.212+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feab0088be0 con 0x7feac00cbad0 2024-01-24T17:41:40.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.211+0000 7feb8f200640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7feb70072720 0x7feb70074b40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.211+0000 7feb97e00640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb7006a500 0x7feb7006c9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feb88009c10 tx=0x7feb88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.212+0000 7feab7e00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7fea980726b0 0x7fea98074ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.212+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 --> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea98075180 con 0x7fea980726b0 2024-01-24T17:41:40.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.213+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7feab0088f90 con 0x7feac00cbad0 2024-01-24T17:41:40.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.213+0000 7feac4e00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7fea980726b0 0x7fea98074ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.214+0000 7feac4e00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7fea980726b0 0x7fea98074ad0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.215+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 <== osd.5 v2:172.21.15.100:6816/36844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7feb700751f0 con 0x7feb70072720 2024-01-24T17:41:40.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.215+0000 7feabfe00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea9806a490 0x7fea9806c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.216+0000 7f46dde00640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46b806a430 0x7f46b806c8f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f46d0036000 tx=0x7f46d003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.216+0000 7feabfe00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea9806a490 0x7fea9806c950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feab8001d00 tx=0x7feab8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.218+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 <== osd.7 v2:172.21.15.100:6820/36857 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fea98075180 con 0x7fea980726b0 2024-01-24T17:41:40.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.219+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 --> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f69c406adc0 con 0x7f699c0726a0 2024-01-24T17:41:40.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.222+0000 7f69c1600640 1 -- 172.21.15.39:0/3897544962 <== osd.0 v2:172.21.15.39:6800/36720 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f69ac009bd0 con 0x7f699c0726a0 2024-01-24T17:41:40.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.222+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f699c0726a0 msgr2=0x7f699c074ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.224+0000 7f69caa00640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f699c0726a0 0x7f699c074ac0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.224+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f699c06a480 msgr2=0x7f699c06c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.224+0000 7f69caa00640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f699c06a480 0x7f699c06c940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f69bc035000 tx=0x7f69bc030000 comp rx=0 tx=0).stop 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.224+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c40d1350 msgr2=0x7f69c40cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.224+0000 7f69caa00640 1 --2- 172.21.15.39:0/3897544962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f69c40d1350 0x7f69c40cbbf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f69b400c5e0 tx=0x7f69b400caa0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.224+0000 7f69ca000640 1 -- 172.21.15.39:0/3897544962 reap_dead start 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.225+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 shutdown_connections 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.225+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 >> 172.21.15.39:0/3897544962 conn(0x7f69c40d2830 msgr2=0x7f69c40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.225+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 shutdown_connections 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.225+0000 7f69caa00640 1 -- 172.21.15.39:0/3897544962 wait complete. 2024-01-24T17:41:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.227+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 --> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1f240c9410 con 0x7f1efc0726a0 2024-01-24T17:41:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.227+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 --> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbedc002cb0 con 0x7fbee4072600 2024-01-24T17:41:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.228+0000 7f1f1b400640 1 -- 172.21.15.39:0/2223869276 <== osd.1 v2:172.21.15.39:6820/36723 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1f0c00c040 con 0x7f1efc0726a0 2024-01-24T17:41:40.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.228+0000 7fbf03400640 1 -- 172.21.15.39:0/3449195227 <== osd.2 v2:172.21.15.39:6808/36726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbedc002cb0 con 0x7fbee4072600 2024-01-24T17:41:40.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fbee4072600 msgr2=0x7fbee4074a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fbee4072600 0x7fbee4074a20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee406a460 msgr2=0x7fbee406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee406a460 0x7fbee406c920 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0017e0 tx=0x7fbefc001490 comp rx=0 tx=0).stop 2024-01-24T17:41:40.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec005180 msgr2=0x7fbf0c1f8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 --2- 172.21.15.39:0/3449195227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbeec005180 0x7fbf0c1f8d60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbef400e800 tx=0x7fbef400ecc0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf0be00640 1 -- 172.21.15.39:0/3449195227 reap_dead start 2024-01-24T17:41:40.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f1efc0726a0 msgr2=0x7f1efc074ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 shutdown_connections 2024-01-24T17:41:40.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 >> 172.21.15.39:0/3449195227 conn(0x7fbf0c06a090 msgr2=0x7fbf0c06fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 --> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa03c03a170 con 0x7fa01c072600 2024-01-24T17:41:40.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7f1f29000640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f1efc0726a0 0x7f1efc074ac0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.229+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 shutdown_connections 2024-01-24T17:41:40.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1efc06a480 msgr2=0x7f1efc06c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7f1f29000640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1efc06a480 0x7f1efc06c940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1f240c99f0 tx=0x7f1f1400b210 comp rx=0 tx=0).stop 2024-01-24T17:41:40.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7fa040c00640 1 -- 172.21.15.39:0/2809492536 <== osd.3 v2:172.21.15.39:6816/36725 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa02c001260 con 0x7fa01c072600 2024-01-24T17:41:40.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f240ce950 msgr2=0x7f1f24154b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7f1f29000640 1 --2- 172.21.15.39:0/2223869276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f240ce950 0x7f1f24154b10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c034000 tx=0x7f1f1c004a40 comp rx=0 tx=0).stop 2024-01-24T17:41:40.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7f1f23e00640 1 -- 172.21.15.39:0/2223869276 reap_dead start 2024-01-24T17:41:40.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 shutdown_connections 2024-01-24T17:41:40.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 >> 172.21.15.39:0/2223869276 conn(0x7f1f2406c220 msgr2=0x7f1f24155460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.230+0000 7fbf12400640 1 -- 172.21.15.39:0/3449195227 wait complete. 2024-01-24T17:41:40.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fa01c072600 msgr2=0x7fa01c074a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa04a600640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fa01c072600 0x7fa01c074a20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 shutdown_connections 2024-01-24T17:41:40.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa01c06a460 msgr2=0x7fa01c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa04a600640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa01c06a460 0x7fa01c06c920 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa034000f60 tx=0x7fa034007f00 comp rx=0 tx=0).stop 2024-01-24T17:41:40.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa03c0544b0 msgr2=0x7fa03c04e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa04a600640 1 --2- 172.21.15.39:0/2809492536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa03c0544b0 0x7fa03c04e7f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa04404f260 tx=0x7fa044069a80 comp rx=0 tx=0).stop 2024-01-24T17:41:40.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa049c00640 1 -- 172.21.15.39:0/2809492536 reap_dead start 2024-01-24T17:41:40.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 shutdown_connections 2024-01-24T17:41:40.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.231+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 >> 172.21.15.39:0/2809492536 conn(0x7fa03c053920 msgr2=0x7fa03c036a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.233+0000 7f1f29000640 1 -- 172.21.15.39:0/2223869276 wait complete. 2024-01-24T17:41:40.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.233+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 shutdown_connections 2024-01-24T17:41:40.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.233+0000 7fa04a600640 1 -- 172.21.15.39:0/2809492536 wait complete. 2024-01-24T17:41:40.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.243+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 --> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f815c0c93f0 con 0x7f81340726a0 2024-01-24T17:41:40.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.243+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 --> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feac0165510 con 0x7fea980726b0 2024-01-24T17:41:40.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.244+0000 7feab7e00640 1 -- 172.21.15.39:0/2085663921 <== osd.7 v2:172.21.15.100:6820/36857 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feaa8030b30 con 0x7fea980726b0 2024-01-24T17:41:40.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.244+0000 7f8159600640 1 -- 172.21.15.39:0/548453310 <== osd.4 v2:172.21.15.100:6800/36832 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8154007b30 con 0x7f81340726a0 2024-01-24T17:41:40.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.245+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7fea980726b0 msgr2=0x7fea98074ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.245+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 --> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb980d1020 con 0x7feb70072720 2024-01-24T17:41:40.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.245+0000 7feac6e00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7fea980726b0 0x7fea98074ad0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.245+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea9806a490 msgr2=0x7fea9806c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.245+0000 7feac6e00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea9806a490 0x7fea9806c950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feab8001d00 tx=0x7feab8009040 comp rx=0 tx=0).stop 2024-01-24T17:41:40.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.245+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00cbad0 msgr2=0x7feac015cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.245+0000 7feac6e00640 1 --2- 172.21.15.39:0/2085663921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feac00cbad0 0x7feac015cad0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feab000c450 tx=0x7feab000c910 comp rx=0 tx=0).stop 2024-01-24T17:41:40.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.245+0000 7feac4e00640 1 -- 172.21.15.39:0/2085663921 reap_dead start 2024-01-24T17:41:40.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f81340726a0 msgr2=0x7f8134074ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f81340726a0 0x7f8134074ac0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 shutdown_connections 2024-01-24T17:41:40.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 >> 172.21.15.39:0/2085663921 conn(0x7feac006a090 msgr2=0x7feac0160f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f813406a500 msgr2=0x7f813406c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f813406a500 0x7f813406c9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8144036000 tx=0x7f8144008030 comp rx=0 tx=0).stop 2024-01-24T17:41:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 shutdown_connections 2024-01-24T17:41:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7feac6e00640 1 -- 172.21.15.39:0/2085663921 wait complete. 2024-01-24T17:41:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7feb94c00640 1 -- 172.21.15.39:0/2252091503 <== osd.5 v2:172.21.15.100:6816/36844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feb80004040 con 0x7feb70072720 2024-01-24T17:41:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0cbaf0 msgr2=0x7f815c1f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 --2- 172.21.15.39:0/548453310 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f815c0cbaf0 0x7f815c1f5df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f814c00c480 tx=0x7f814c00c940 comp rx=0 tx=0).stop 2024-01-24T17:41:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162200640 1 -- 172.21.15.39:0/548453310 reap_dead start 2024-01-24T17:41:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 shutdown_connections 2024-01-24T17:41:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 >> 172.21.15.39:0/548453310 conn(0x7f815c06c020 msgr2=0x7f815c0cdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46d80538b0 con 0x7f46b80725d0 2024-01-24T17:41:40.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 shutdown_connections 2024-01-24T17:41:40.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.246+0000 7f8162c00640 1 -- 172.21.15.39:0/548453310 wait complete. 2024-01-24T17:41:40.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7feb70072720 msgr2=0x7feb70074b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7feb70072720 0x7feb70074b40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb7006a500 msgr2=0x7feb7006c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb7006a500 0x7feb7006c9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feb88009c10 tx=0x7feb88008040 comp rx=0 tx=0).stop 2024-01-24T17:41:40.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb98161fa0 msgr2=0x7feb980ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 --2- 172.21.15.39:0/2252091503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb98161fa0 0x7feb980ce960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feb980c5d90 tx=0x7feb9000b4a0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9d000640 1 -- 172.21.15.39:0/2252091503 reap_dead start 2024-01-24T17:41:40.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 shutdown_connections 2024-01-24T17:41:40.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 >> 172.21.15.39:0/2252091503 conn(0x7feb98071ea0 msgr2=0x7feb980c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7f46d6000640 1 -- 172.21.15.39:0/828320853 <== osd.6 v2:172.21.15.100:6808/36833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f46c8009680 con 0x7f46b80725d0 2024-01-24T17:41:40.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.247+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 shutdown_connections 2024-01-24T17:41:40.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.248+0000 7feb9da00640 1 -- 172.21.15.39:0/2252091503 wait complete. 2024-01-24T17:41:40.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.248+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f46b80725d0 msgr2=0x7f46b80749f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.248+0000 7f46df200640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f46b80725d0 0x7f46b80749f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.249+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46b806a430 msgr2=0x7f46b806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.249+0000 7f46df200640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46b806a430 0x7f46b806c8f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f46d0036000 tx=0x7f46d003c040 comp rx=0 tx=0).stop 2024-01-24T17:41:40.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.249+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46d8004c00 msgr2=0x7f46d8005080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.249+0000 7f46df200640 1 --2- 172.21.15.39:0/828320853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46d8004c00 0x7f46d8005080 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46e0068990 tx=0x7f46e0066fc0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.250+0000 7f46de800640 1 -- 172.21.15.39:0/828320853 reap_dead start 2024-01-24T17:41:40.281 INFO:teuthology.orchestra.run.smithi039.stdout:55834574854 2024-01-24T17:41:40.282 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-24T17:41:40.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.253+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 shutdown_connections 2024-01-24T17:41:40.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.253+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 >> 172.21.15.39:0/828320853 conn(0x7f46d800a550 msgr2=0x7f46d804b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.253+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 shutdown_connections 2024-01-24T17:41:40.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.253+0000 7f46df200640 1 -- 172.21.15.39:0/828320853 wait complete. 2024-01-24T17:41:40.283 INFO:teuthology.orchestra.run.smithi039.stdout:55834574854 2024-01-24T17:41:40.284 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-24T17:41:40.284 INFO:teuthology.orchestra.run.smithi039.stdout:55834574855 2024-01-24T17:41:40.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-24T17:41:40.285 INFO:teuthology.orchestra.run.smithi039.stdout:55834574856 2024-01-24T17:41:40.286 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-24T17:41:40.286 INFO:teuthology.orchestra.run.smithi039.stdout:55834574856 2024-01-24T17:41:40.287 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-24T17:41:40.287 INFO:teuthology.orchestra.run.smithi039.stdout:55834574854 2024-01-24T17:41:40.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-24T17:41:40.288 INFO:teuthology.orchestra.run.smithi039.stdout:55834574853 2024-01-24T17:41:40.289 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-24T17:41:40.290 INFO:teuthology.orchestra.run.smithi039.stdout:55834574855 2024-01-24T17:41:40.290 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-24T17:41:40.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.617+0000 7f9648000640 1 Processor -- start 2024-01-24T17:41:40.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.617+0000 7f9648000640 1 -- start start 2024-01-24T17:41:40.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.619+0000 7f9648000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96380df530 0x7f96380075b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.619+0000 7f9648000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9638006010 0x7f9638054290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.619+0000 7f9648000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9638006510 con 0x7f963804fca0 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.619+0000 7f9648000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96380df910 con 0x7f96380df530 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.619+0000 7f9648000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96380067f0 con 0x7f9638006010 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.620+0000 7f9645600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96380df530 0x7f96380075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.620+0000 7f9645600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96380df530 0x7f96380075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58386/0 (socket says 172.21.15.39:58386) 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.620+0000 7f9645600640 1 -- 172.21.15.39:0/2480098820 learned_addr learned my addr 172.21.15.39:0/2480098820 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.621+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381087858 0 0) 0x7f9638006510 con 0x7f963804fca0 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.622+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620003520 con 0x7f963804fca0 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.623+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790988522 0 0) 0x7f9620003520 con 0x7f963804fca0 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.623+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 >> v2:172.21.15.100:3300/0 conn(0x7f9638006010 msgr2=0x7f9638054290 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.623+0000 7f963fe00640 1 --2- 172.21.15.39:0/2480098820 >> v2:172.21.15.100:3300/0 conn(0x7f9638006010 0x7f9638054290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.623+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 >> v2:172.21.15.39:3300/0 conn(0x7f96380df530 msgr2=0x7f96380075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.623+0000 7f963fe00640 1 --2- 172.21.15.39:0/2480098820 >> v2:172.21.15.39:3300/0 conn(0x7f96380df530 0x7f96380075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.623+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638006ad0 con 0x7f963804fca0 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f964004e840 con 0x7f963804fca0 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 >> v1:172.21.15.39:6789/0 conn(0x7f963804fca0 legacy=0x7f96380500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f963fe00640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620003970 0x7f9620003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f963fe00640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9620004310 0x7f96200047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f963fe00640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9620004fe0 0x7f962000b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380df910 con 0x7f9620004fe0 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380067f0 con 0x7f9620004310 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962000bf90 con 0x7f9620003970 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f9645600640 1 --2- 172.21.15.39:0/2480098820 >> v2:172.21.15.39:3300/0 conn(0x7f96380df530 0x7f96380075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.624+0000 7f9644c00640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9620004310 0x7f96200047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.625+0000 7f9645600640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620003970 0x7f9620003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.625+0000 7f9644c00640 1 -- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620003970 msgr2=0x7f9620003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.625+0000 7f9644c00640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620003970 0x7f9620003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.625+0000 7f9644c00640 1 -- 172.21.15.39:0/2480098820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9620004fe0 msgr2=0x7f962000b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.625+0000 7f9644c00640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9620004fe0 0x7f962000b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.625+0000 7f9644c00640 1 -- 172.21.15.39:0/2480098820 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620003520 con 0x7f9620004310 2024-01-24T17:41:40.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.626+0000 7f9646000640 1 -- 172.21.15.39:0/2480098820 reap_dead start 2024-01-24T17:41:40.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.627+0000 7fc6a5600640 1 Processor -- start 2024-01-24T17:41:40.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.627+0000 7fc6a5600640 1 -- start start 2024-01-24T17:41:40.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.629+0000 7f9644c00640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9620004310 0x7f96200047e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f96280097d0 tx=0x7f96200057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6599e7dd2024cd58 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.629+0000 7f21b5800640 1 Processor -- start 2024-01-24T17:41:40.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.629+0000 7f21b5800640 1 -- start start 2024-01-24T17:41:40.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.629+0000 7fc6a5600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6980e3060 0x7fc69803ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.629+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628032600 con 0x7f9620004310 2024-01-24T17:41:40.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.630+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962802e050 con 0x7f9620004310 2024-01-24T17:41:40.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.630+0000 7f9648000640 1 -- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9620004310 msgr2=0x7f96200047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7f963fe00640 1 -- 172.21.15.39:0/2480098820 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628032600 con 0x7f9620004310 2024-01-24T17:41:40.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7f9648000640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9620004310 0x7f96200047e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f96280097d0 tx=0x7f96200057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.629+0000 7fc6a5600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69803c060 con 0x7fc698006b60 2024-01-24T17:41:40.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.629+0000 7fc6a5600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6980070e0 con 0x7fc698054350 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.629+0000 7fc6a5600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc69803ef70 con 0x7fc6980e3060 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.630+0000 7fc69ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6980e3060 0x7fc69803ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.630+0000 7fc69ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6980e3060 0x7fc69803ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58396/0 (socket says 172.21.15.39:58396) 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.630+0000 7fc69e000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc698006b60 0x7fc698053c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52444/0 (socket says 172.21.15.39:52444) 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.630+0000 7fc69ea00640 1 -- 172.21.15.39:0/1474585442 learned_addr learned my addr 172.21.15.39:0/1474585442 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7f9648000640 1 -- 172.21.15.39:0/2480098820 shutdown_connections 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7f9648000640 1 --2- 172.21.15.39:0/2480098820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9620004310 0x7f96200047e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7f9648000640 1 -- 172.21.15.39:0/2480098820 >> 172.21.15.39:0/2480098820 conn(0x7f9638054c40 msgr2=0x7f9638055050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7f9648000640 1 -- 172.21.15.39:0/2480098820 shutdown_connections 2024-01-24T17:41:40.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7fc69ea00640 1 -- 172.21.15.39:0/1474585442 >> v1:172.21.15.100:6789/0 conn(0x7fc698054350 legacy=0x7fc698057710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7fc69ea00640 1 -- 172.21.15.39:0/1474585442 >> v1:172.21.15.39:6789/0 conn(0x7fc698006b60 legacy=0x7fc698053c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.632+0000 7f21b5800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f21a8053150 0x7f21a8006190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.632+0000 7f21b5800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f21a80066d0 0x7f21a8006b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7fc69ea00640 1 -- 172.21.15.39:0/1474585442 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980073a0 con 0x7fc6980e3060 2024-01-24T17:41:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.631+0000 7f9648000640 1 -- 172.21.15.39:0/2480098820 wait complete. 2024-01-24T17:41:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.632+0000 7f21b5800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a80543f0 con 0x7f21a8033fc0 2024-01-24T17:41:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.632+0000 7f21b5800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80535d0 con 0x7f21a8053150 2024-01-24T17:41:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.632+0000 7f21b5800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80343a0 con 0x7f21a80066d0 2024-01-24T17:41:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.633+0000 7f21afe00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f21a8033fc0 0x7f21a8050230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48704/0 (socket says 172.21.15.39:48704) 2024-01-24T17:41:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.633+0000 7f21afe00640 1 -- 172.21.15.39:0/2152301481 learned_addr learned my addr 172.21.15.39:0/2152301481 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.633+0000 7f21b4e00640 1 --2- 172.21.15.39:0/2152301481 >> v2:172.21.15.100:3300/0 conn(0x7f21a80066d0 0x7f21a8006b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.633+0000 7f21af400640 1 --2- 172.21.15.39:0/2152301481 >> v2:172.21.15.39:3300/0 conn(0x7f21a8053150 0x7f21a8006190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.634+0000 7fc69ea00640 1 --2- 172.21.15.39:0/1474585442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6980e3060 0x7fc69803ea30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0068230 tx=0x7fc6a00a5850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1ab83ea0024237 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.633+0000 7f21aea00640 1 -- 172.21.15.39:0/2152301481 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437676611 0 0) 0x7f21a80543f0 con 0x7f21a8033fc0 2024-01-24T17:41:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.633+0000 7f21aea00640 1 -- 172.21.15.39:0/2152301481 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2190007930 con 0x7f21a8033fc0 2024-01-24T17:41:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.634+0000 7f21af400640 1 -- 172.21.15.39:0/2152301481 >> v2:172.21.15.100:3300/0 conn(0x7f21a80066d0 msgr2=0x7f21a8006b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.634+0000 7f21af400640 1 --2- 172.21.15.39:0/2152301481 >> v2:172.21.15.100:3300/0 conn(0x7f21a80066d0 0x7f21a8006b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.634+0000 7f21af400640 1 -- 172.21.15.39:0/2152301481 >> v1:172.21.15.100:6789/0 conn(0x7f21a8033fc0 legacy=0x7f21a8050230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.634+0000 7f21af400640 1 -- 172.21.15.39:0/2152301481 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80537e0 con 0x7f21a8053150 2024-01-24T17:41:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.634+0000 7fc69d600640 1 -- 172.21.15.39:0/1474585442 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00ab810 con 0x7fc6980e3060 2024-01-24T17:41:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.635+0000 7fc69d600640 1 -- 172.21.15.39:0/1474585442 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a00ab970 con 0x7fc6980e3060 2024-01-24T17:41:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.635+0000 7fc69d600640 1 -- 172.21.15.39:0/1474585442 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00abd60 con 0x7fc6980e3060 2024-01-24T17:41:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.636+0000 7f21af400640 1 --2- 172.21.15.39:0/2152301481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8053150 0x7f21a8006190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f21a8054bc0 tx=0x7f21a002c080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a73e08bf19553a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.636+0000 7f21aea00640 1 -- 172.21.15.39:0/2152301481 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a002ce30 con 0x7f21a8053150 2024-01-24T17:41:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.636+0000 7f21aea00640 1 -- 172.21.15.39:0/2152301481 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a0030070 con 0x7f21a8053150 2024-01-24T17:41:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.636+0000 7f21aea00640 1 -- 172.21.15.39:0/2152301481 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0036600 con 0x7f21a8053150 2024-01-24T17:41:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.636+0000 7f9648000640 1 Processor -- start 2024-01-24T17:41:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.636+0000 7fc6a5600640 1 -- 172.21.15.39:0/1474585442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6980e3060 msgr2=0x7fc69803ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.637+0000 7f9648000640 1 -- start start 2024-01-24T17:41:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.636+0000 7fc6a5600640 1 --2- 172.21.15.39:0/1474585442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6980e3060 0x7fc69803ea30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0068230 tx=0x7fc6a00a5850 comp rx=0 tx=0).stop 2024-01-24T17:41:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.637+0000 7fc6a5600640 1 -- 172.21.15.39:0/1474585442 shutdown_connections 2024-01-24T17:41:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.637+0000 7fc6a5600640 1 --2- 172.21.15.39:0/1474585442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6980e3060 0x7fc69803ea30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.637+0000 7fc6a5600640 1 -- 172.21.15.39:0/1474585442 >> 172.21.15.39:0/1474585442 conn(0x7fc698055a80 msgr2=0x7fc698055e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.637+0000 7f2ff4000640 1 Processor -- start 2024-01-24T17:41:40.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.637+0000 7f2ff4000640 1 -- start start 2024-01-24T17:41:40.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.638+0000 7fc6a5600640 1 -- 172.21.15.39:0/1474585442 shutdown_connections 2024-01-24T17:41:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.638+0000 7f21b5800640 1 -- 172.21.15.39:0/2152301481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8053150 msgr2=0x7f21a8006190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.638+0000 7f21b5800640 1 --2- 172.21.15.39:0/2152301481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8053150 0x7f21a8006190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f21a8054bc0 tx=0x7f21a002c080 comp rx=0 tx=0).stop 2024-01-24T17:41:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.638+0000 7fc6a5600640 1 -- 172.21.15.39:0/1474585442 wait complete. 2024-01-24T17:41:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.638+0000 7f9648000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620004310 0x7f963817a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.639+0000 7f9648000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96380d3a50 0x7f963817a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.639+0000 7f762d800640 1 Processor -- start 2024-01-24T17:41:40.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.639+0000 7f762d800640 1 -- start start 2024-01-24T17:41:40.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.639+0000 7f9645600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620004310 0x7f963817a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.639+0000 7f9645600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620004310 0x7f963817a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48150/0 (socket says 172.21.15.39:48150) 2024-01-24T17:41:40.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.639+0000 7f9645600640 1 -- 172.21.15.39:0/238821829 learned_addr learned my addr 172.21.15.39:0/238821829 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f9644c00640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96380d3a50 0x7f963817a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f21b5800640 1 -- 172.21.15.39:0/2152301481 shutdown_connections 2024-01-24T17:41:40.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f21b5800640 1 --2- 172.21.15.39:0/2152301481 >> v2:172.21.15.100:3300/0 conn(0x7f21a80066d0 0x7f21a8006b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f21b5800640 1 --2- 172.21.15.39:0/2152301481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8053150 0x7f21a8006190 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f21b5800640 1 -- 172.21.15.39:0/2152301481 >> 172.21.15.39:0/2152301481 conn(0x7f21a8054c10 msgr2=0x7f21a8054ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f9648000640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96380d3f10 0x7f963817aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f2ff4000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec0cf990 0x7f2fec0cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f2ff4000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2fec15fb90 0x7f2fec15d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f2ff4000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec0d02b0 con 0x7f2fec15d020 2024-01-24T17:41:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f2ff4000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec15e080 con 0x7f2fec15fb90 2024-01-24T17:41:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f2ff4000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec15dac0 con 0x7f2fec0cf990 2024-01-24T17:41:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f21b5800640 1 -- 172.21.15.39:0/2152301481 shutdown_connections 2024-01-24T17:41:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380d4800 con 0x7f96380d3f10 2024-01-24T17:41:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963816aa20 con 0x7f96380d3a50 2024-01-24T17:41:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963804a250 con 0x7f9620004310 2024-01-24T17:41:40.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f2ff0c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2fec15d020 0x7f2fec160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52446/0 (socket says 172.21.15.39:52446) 2024-01-24T17:41:40.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f2ff0c00640 1 -- 172.21.15.39:0/1843847448 learned_addr learned my addr 172.21.15.39:0/1843847448 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f9646000640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96380d3f10 0x7f963817aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f2ff1600640 1 --2- 172.21.15.39:0/1843847448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec0cf990 0x7f2fec0cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f2ff2000640 1 --2- 172.21.15.39:0/1843847448 >> v2:172.21.15.100:3300/0 conn(0x7f2fec15fb90 0x7f2fec15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.640+0000 7f21b5800640 1 -- 172.21.15.39:0/2152301481 wait complete. 2024-01-24T17:41:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f85bcc00640 1 Processor -- start 2024-01-24T17:41:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f2febe00640 1 -- 172.21.15.39:0/1843847448 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230192070 0 0) 0x7f2fec0d02b0 con 0x7f2fec15d020 2024-01-24T17:41:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f2febe00640 1 -- 172.21.15.39:0/1843847448 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc003520 con 0x7f2fec15d020 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f762d800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f76280cfe60 0x7f76280d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f762d800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76280ce160 con 0x7f76280cf9e0 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f762d800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76280d27a0 con 0x7f762815cc90 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f762d800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762815d070 con 0x7f76280cfe60 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f9645600640 1 -- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96380d3a50 msgr2=0x7f963817a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f9645600640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96380d3a50 0x7f963817a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f9645600640 1 -- 172.21.15.39:0/238821829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96380d3f10 msgr2=0x7f963817aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f9645600640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96380d3f10 0x7f963817aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f9645600640 1 -- 172.21.15.39:0/238821829 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963804a4d0 con 0x7f9620004310 2024-01-24T17:41:40.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f9644c00640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96380d3a50 0x7f963817a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:40.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f2ff1600640 1 -- 172.21.15.39:0/1843847448 >> v2:172.21.15.100:3300/0 conn(0x7f2fec15fb90 msgr2=0x7f2fec15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f2ff1600640 1 --2- 172.21.15.39:0/1843847448 >> v2:172.21.15.100:3300/0 conn(0x7f2fec15fb90 0x7f2fec15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f2ff1600640 1 -- 172.21.15.39:0/1843847448 >> v1:172.21.15.39:6789/0 conn(0x7f2fec15d020 legacy=0x7f2fec160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f2ff1600640 1 -- 172.21.15.39:0/1843847448 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec15dda0 con 0x7f2fec0cf990 2024-01-24T17:41:40.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.641+0000 7f85bcc00640 1 -- start start 2024-01-24T17:41:40.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f7626a00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f762815cc90 0x7f76280d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48718/0 (socket says 172.21.15.39:48718) 2024-01-24T17:41:40.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f7626a00640 1 -- 172.21.15.39:0/47911136 learned_addr learned my addr 172.21.15.39:0/47911136 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f7627400640 1 --2- 172.21.15.39:0/47911136 >> v2:172.21.15.39:3300/0 conn(0x7f76280cfe60 0x7f76280d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.642+0000 7f9645600640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620004310 0x7f963817a030 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f96300108b0 tx=0x7f9630010d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.643+0000 7f7625600640 1 -- 172.21.15.39:0/47911136 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889625140 0 0) 0x7f76280d27a0 con 0x7f762815cc90 2024-01-24T17:41:40.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.643+0000 7f7625600640 1 -- 172.21.15.39:0/47911136 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608003520 con 0x7f762815cc90 2024-01-24T17:41:40.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.643+0000 7fc6a5600640 1 Processor -- start 2024-01-24T17:41:40.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.643+0000 7f85bcc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f85b80b1b20 0x7f85b80d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.643+0000 7f963d600640 1 -- 172.21.15.39:0/238821829 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963000eb00 con 0x7f9620004310 2024-01-24T17:41:40.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.643+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963804a7b0 con 0x7f9620004310 2024-01-24T17:41:40.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f963d600640 1 -- 172.21.15.39:0/238821829 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963000c050 con 0x7f9620004310 2024-01-24T17:41:40.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f963d600640 1 -- 172.21.15.39:0/238821829 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963000ee60 con 0x7f9620004310 2024-01-24T17:41:40.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.643+0000 7f2ff1600640 1 --2- 172.21.15.39:0/1843847448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec0cf990 0x7f2fec0cfd70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009d90 tx=0x7f2fdc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5853b029db57cb57 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f7625600640 1 -- 172.21.15.39:0/47911136 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586210291 0 0) 0x7f7618001110 con 0x7f76280cf9e0 2024-01-24T17:41:40.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f7627400640 1 -- 172.21.15.39:0/47911136 >> v1:172.21.15.100:6789/0 conn(0x7f762815cc90 legacy=0x7f76280d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.643+0000 7f85bcc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f85b806ac00 0x7f85b806b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f85bcc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b80c5ed0 con 0x7f85b80ce1f0 2024-01-24T17:41:40.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f85bcc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b80d16c0 con 0x7f85b80b1b20 2024-01-24T17:41:40.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f85bcc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b80cd710 con 0x7f85b806ac00 2024-01-24T17:41:40.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f85b7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f85b80ce1f0 0x7f85b80cf080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52458/0 (socket says 172.21.15.39:52458) 2024-01-24T17:41:40.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f85b7e00640 1 -- 172.21.15.39:0/1468140536 learned_addr learned my addr 172.21.15.39:0/1468140536 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f21b5800640 1 Processor -- start 2024-01-24T17:41:40.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7fc6a5600640 1 -- start start 2024-01-24T17:41:40.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f85b7400640 1 --2- 172.21.15.39:0/1468140536 >> v2:172.21.15.39:3300/0 conn(0x7f85b80b1b20 0x7f85b80d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f85b6a00640 1 --2- 172.21.15.39:0/1468140536 >> v2:172.21.15.100:3300/0 conn(0x7f85b806ac00 0x7f85b806b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f7627400640 1 -- 172.21.15.39:0/47911136 >> v1:172.21.15.39:6789/0 conn(0x7f76280cf9e0 legacy=0x7f76280d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f7627400640 1 -- 172.21.15.39:0/47911136 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76280d2a80 con 0x7f76280cfe60 2024-01-24T17:41:40.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.645+0000 7f85b6a00640 1 -- 172.21.15.39:0/1468140536 >> v2:172.21.15.39:3300/0 conn(0x7f85b80b1b20 msgr2=0x7f85b80d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.645+0000 7f85b6000640 1 -- 172.21.15.39:0/1468140536 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307683619 0 0) 0x7f85b80c5ed0 con 0x7f85b80ce1f0 2024-01-24T17:41:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.645+0000 7f5f0d400640 1 Processor -- start 2024-01-24T17:41:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.645+0000 7f5f0d400640 1 -- start start 2024-01-24T17:41:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963816a6b0 con 0x7f9620004310 2024-01-24T17:41:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f85b6a00640 1 --2- 172.21.15.39:0/1468140536 >> v2:172.21.15.39:3300/0 conn(0x7f85b80b1b20 0x7f85b80d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f85b6a00640 1 -- 172.21.15.39:0/1468140536 >> v1:172.21.15.39:6789/0 conn(0x7f85b80ce1f0 legacy=0x7f85b80cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f85b6a00640 1 -- 172.21.15.39:0/1468140536 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80cd9f0 con 0x7f85b806ac00 2024-01-24T17:41:40.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9638050da0 con 0x7f9620004310 2024-01-24T17:41:40.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f2febe00640 1 -- 172.21.15.39:0/1843847448 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc02cc50 con 0x7f2fec0cf990 2024-01-24T17:41:40.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f2febe00640 1 -- 172.21.15.39:0/1843847448 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc02cdb0 con 0x7f2fec0cf990 2024-01-24T17:41:40.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f2febe00640 1 -- 172.21.15.39:0/1843847448 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc034660 con 0x7f2fec0cf990 2024-01-24T17:41:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.644+0000 7f21b5800640 1 -- start start 2024-01-24T17:41:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21b5800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21a80066d0 0x7f21a80d8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21b5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8033fc0 0x7f21a817c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21b5800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21a8053150 0x7f21a80dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21b5800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80d8eb0 con 0x7f21a8033fc0 2024-01-24T17:41:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21b5800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a817cb40 con 0x7f21a8053150 2024-01-24T17:41:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21b5800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80d9130 con 0x7f21a80066d0 2024-01-24T17:41:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21afe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21a80066d0 0x7f21a80d8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f85b7400640 1 --2- 172.21.15.39:0/1468140536 >> v2:172.21.15.39:3300/0 conn(0x7f85b80b1b20 0x7f85b80d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:40.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21afe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21a80066d0 0x7f21a80d8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48162/0 (socket says 172.21.15.39:48162) 2024-01-24T17:41:40.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.646+0000 7f21afe00640 1 -- 172.21.15.39:0/2244025809 learned_addr learned my addr 172.21.15.39:0/2244025809 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f21b4e00640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21a8053150 0x7f21a80dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f9586a00640 1 Processor -- start 2024-01-24T17:41:40.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f2ff4000640 1 -- 172.21.15.39:0/1843847448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec0cf990 msgr2=0x7f2fec0cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f2ff4000640 1 --2- 172.21.15.39:0/1843847448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec0cf990 0x7f2fec0cfd70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009d90 tx=0x7f2fdc02bea0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f5f0d400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5f08158af0 0x7f5f0815d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f21afe00640 1 -- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21a8053150 msgr2=0x7f21a80dfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f21afe00640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21a8053150 0x7f21a80dfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f5f0d400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f080cb470 0x7f5f08071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f21afe00640 1 -- 172.21.15.39:0/2244025809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8033fc0 msgr2=0x7f21a817c600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T17:41:40.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f21afe00640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8033fc0 0x7f21a817c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f21afe00640 1 -- 172.21.15.39:0/2244025809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a000c4c0 con 0x7f21a80066d0 2024-01-24T17:41:40.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f5f0d400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f080704b0 con 0x7f5f080c95a0 2024-01-24T17:41:40.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f5f0d400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f0806ffa0 con 0x7f5f080cb470 2024-01-24T17:41:40.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f5f0d400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f080706e0 con 0x7f5f08158af0 2024-01-24T17:41:40.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f2ff4000640 1 -- 172.21.15.39:0/1843847448 shutdown_connections 2024-01-24T17:41:40.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f2ff4000640 1 --2- 172.21.15.39:0/1843847448 >> v2:172.21.15.100:3300/0 conn(0x7f2fec15fb90 0x7f2fec15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f2ff4000640 1 --2- 172.21.15.39:0/1843847448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec0cf990 0x7f2fec0cfd70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f2ff4000640 1 -- 172.21.15.39:0/1843847448 >> 172.21.15.39:0/1843847448 conn(0x7f2fec071790 msgr2=0x7f2fec071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.647+0000 7f9586a00640 1 -- start start 2024-01-24T17:41:40.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f2ff4000640 1 -- 172.21.15.39:0/1843847448 shutdown_connections 2024-01-24T17:41:40.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7fc6a5600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc698006b60 0x7fc6980da540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7fc6a5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc698054350 0x7fc6980dff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7fc6a5600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6980e3060 0x7fc698050c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f21afe00640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21a80066d0 0x7f21a80d8970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f21b004eda0 tx=0x7f21b0066fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7fc6a5600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69804fe50 con 0x7fc698054350 2024-01-24T17:41:40.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7fc6a5600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6980daa80 con 0x7fc6980e3060 2024-01-24T17:41:40.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7fc6a5600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69804fb70 con 0x7fc698006b60 2024-01-24T17:41:40.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f7627400640 1 --2- 172.21.15.39:0/47911136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 0x7f76280d0d60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7610009720 tx=0x7f761002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c39fbba74cd970 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f2ff4000640 1 -- 172.21.15.39:0/1843847448 wait complete. 2024-01-24T17:41:40.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f5effe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f5f080c95a0 0x7f5f080cad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52468/0 (socket says 172.21.15.39:52468) 2024-01-24T17:41:40.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f5effe00640 1 -- 172.21.15.39:0/568556785 learned_addr learned my addr 172.21.15.39:0/568556785 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f5f06800640 1 --2- 172.21.15.39:0/568556785 >> v2:172.21.15.100:3300/0 conn(0x7f5f08158af0 0x7f5f0815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7fc69ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc698006b60 0x7fc6980da540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f85b6a00640 1 --2- 172.21.15.39:0/1468140536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b806ac00 0x7f85b806b060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f85b80d2510 tx=0x7f85a802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73e66d78eaa24f1b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f7625600640 1 -- 172.21.15.39:0/47911136 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76200019f0 con 0x7f76280cfe60 2024-01-24T17:41:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7fc69f400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6980e3060 0x7fc698050c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7fc69ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc698006b60 0x7fc6980da540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48166/0 (socket says 172.21.15.39:48166) 2024-01-24T17:41:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7fc69ea00640 1 -- 172.21.15.39:0/919210988 learned_addr learned my addr 172.21.15.39:0/919210988 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7fc69f400640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6980e3060 0x7fc698050c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49440/0 (socket says 172.21.15.39:49440) 2024-01-24T17:41:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f5f07200640 1 --2- 172.21.15.39:0/568556785 >> v2:172.21.15.39:3300/0 conn(0x7f5f080cb470 0x7f5f08071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f9586a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95800cbab0 0x7f9580158a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f9586a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f95800affc0 0x7f95800b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f9586a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95800708c0 con 0x7f958015c6c0 2024-01-24T17:41:40.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f9586a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9580158f70 con 0x7f95800affc0 2024-01-24T17:41:40.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f9586a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95800b0980 con 0x7f95800cbab0 2024-01-24T17:41:40.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f5f06800640 1 -- 172.21.15.39:0/568556785 >> v2:172.21.15.39:3300/0 conn(0x7f5f080cb470 msgr2=0x7f5f08071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f5f06800640 1 --2- 172.21.15.39:0/568556785 >> v2:172.21.15.39:3300/0 conn(0x7f5f080cb470 0x7f5f08071180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f5f06800640 1 -- 172.21.15.39:0/568556785 >> v1:172.21.15.39:6789/0 conn(0x7f5f080c95a0 legacy=0x7f5f080cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f5f06800640 1 -- 172.21.15.39:0/568556785 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f080cc300 con 0x7f5f08158af0 2024-01-24T17:41:40.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f5f07200640 1 --2- 172.21.15.39:0/568556785 >> v2:172.21.15.39:3300/0 conn(0x7f5f080cb470 0x7f5f08071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:40.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7fc69e000640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc698054350 0x7fc6980dff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7fc69ea00640 1 -- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6980e3060 msgr2=0x7fc698050c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7fc69ea00640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6980e3060 0x7fc698050c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7fc69ea00640 1 -- 172.21.15.39:0/919210988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc698054350 msgr2=0x7fc6980dff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7fc69ea00640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc698054350 0x7fc6980dff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7fc69ea00640 1 -- 172.21.15.39:0/919210988 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69804d100 con 0x7fc698006b60 2024-01-24T17:41:40.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7fc69e000640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc698054350 0x7fc6980dff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:40.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f7625600640 1 -- 172.21.15.39:0/47911136 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761002e050 con 0x7f76280cfe60 2024-01-24T17:41:40.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f7625600640 1 -- 172.21.15.39:0/47911136 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610030730 con 0x7f76280cfe60 2024-01-24T17:41:40.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f957f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f95800affc0 0x7f95800b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.649+0000 7f963d600640 1 -- 172.21.15.39:0/238821829 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9630015470 con 0x7f9620004310 2024-01-24T17:41:40.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f963d600640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f961806a460 0x7f961806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f9584a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f958015c6c0 0x7f958015cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52474/0 (socket says 172.21.15.39:52474) 2024-01-24T17:41:40.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f2ff4000640 1 Processor -- start 2024-01-24T17:41:40.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f21a7e00640 1 -- 172.21.15.39:0/2244025809 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b0073370 con 0x7f21a80066d0 2024-01-24T17:41:40.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.648+0000 7f21a7e00640 1 -- 172.21.15.39:0/2244025809 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b008d070 con 0x7f21a80066d0 2024-01-24T17:41:40.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f9584a00640 1 -- 172.21.15.39:0/1150538454 learned_addr learned my addr 172.21.15.39:0/1150538454 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:40.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f2ff4000640 1 -- start start 2024-01-24T17:41:40.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f963d600640 1 -- 172.21.15.39:0/238821829 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9630089460 con 0x7f9620004310 2024-01-24T17:41:40.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f21a7e00640 1 -- 172.21.15.39:0/2244025809 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b0073750 con 0x7f21a80066d0 2024-01-24T17:41:40.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a80d9410 con 0x7f21a80066d0 2024-01-24T17:41:40.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a80d99d0 con 0x7f21a80066d0 2024-01-24T17:41:40.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f9644c00640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f961806a460 0x7f961806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f85b6000640 1 -- 172.21.15.39:0/1468140536 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a802e070 con 0x7f85b806ac00 2024-01-24T17:41:40.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f85b6000640 1 -- 172.21.15.39:0/1468140536 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a80315a0 con 0x7f85b806ac00 2024-01-24T17:41:40.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.650+0000 7f85b6000640 1 -- 172.21.15.39:0/1468140536 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8031930 con 0x7f85b806ac00 2024-01-24T17:41:40.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f957f400640 1 -- 172.21.15.39:0/1150538454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95800cbab0 msgr2=0x7f9580158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:40.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f957f400640 1 --2- 172.21.15.39:0/1150538454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95800cbab0 0x7f9580158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f957f400640 1 -- 172.21.15.39:0/1150538454 >> v1:172.21.15.39:6789/0 conn(0x7f958015c6c0 legacy=0x7f958015cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f957ea00640 1 -- 172.21.15.39:0/1150538454 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780015802 0 0) 0x7f95800708c0 con 0x7f958015c6c0 2024-01-24T17:41:40.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f85bcc00640 1 -- 172.21.15.39:0/1468140536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b806ac00 msgr2=0x7f85b806b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f85bcc00640 1 --2- 172.21.15.39:0/1468140536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b806ac00 0x7f85b806b060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f85b80d2510 tx=0x7f85a802b840 comp rx=0 tx=0).stop 2024-01-24T17:41:40.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f2ff4000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fec0cf990 0x7f2fec152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.651+0000 7f2ff4000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec15d020 0x7f2fec0d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff4000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15fb90 0x7f2fec0d3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff4000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0c8f40 con 0x7f2fec15fb90 2024-01-24T17:41:40.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff4000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1533c0 con 0x7f2fec0cf990 2024-01-24T17:41:40.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff4000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1f6100 con 0x7f2fec15d020 2024-01-24T17:41:40.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f762d800640 1 -- 172.21.15.39:0/47911136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 msgr2=0x7f76280d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f762d800640 1 --2- 172.21.15.39:0/47911136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 0x7f76280d0d60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7610009720 tx=0x7f761002b710 comp rx=0 tx=0).stop 2024-01-24T17:41:40.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f957f400640 1 -- 172.21.15.39:0/1150538454 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580070ba0 con 0x7f95800affc0 2024-01-24T17:41:40.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f9644c00640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f961806a460 0x7f961806c920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9638006440 tx=0x7f9628008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15fb90 0x7f2fec0d3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a8005490 con 0x7f21a80066d0 2024-01-24T17:41:40.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7fc69ea00640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc698006b60 0x7fc6980da540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0074a60 tx=0x7fc6a0061a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff1600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fec0cf990 0x7f2fec152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec15d020 0x7f2fec0d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff1600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fec0cf990 0x7f2fec152a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49456/0 (socket says 172.21.15.39:49456) 2024-01-24T17:41:40.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f2ff1600640 1 -- 172.21.15.39:0/3249828091 learned_addr learned my addr 172.21.15.39:0/3249828091 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f762d800640 1 -- 172.21.15.39:0/47911136 shutdown_connections 2024-01-24T17:41:40.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f762d800640 1 --2- 172.21.15.39:0/47911136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 0x7f76280d0d60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f762d800640 1 -- 172.21.15.39:0/47911136 >> 172.21.15.39:0/47911136 conn(0x7f7628071790 msgr2=0x7f7628071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f85bcc00640 1 -- 172.21.15.39:0/1468140536 shutdown_connections 2024-01-24T17:41:40.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f85bcc00640 1 --2- 172.21.15.39:0/1468140536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b806ac00 0x7f85b806b060 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f85bcc00640 1 --2- 172.21.15.39:0/1468140536 >> v2:172.21.15.39:3300/0 conn(0x7f85b80b1b20 0x7f85b80d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f85bcc00640 1 -- 172.21.15.39:0/1468140536 >> 172.21.15.39:0/1468140536 conn(0x7f85b80d27d0 msgr2=0x7f85b80d2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f762d800640 1 -- 172.21.15.39:0/47911136 shutdown_connections 2024-01-24T17:41:40.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f762d800640 1 -- 172.21.15.39:0/47911136 wait complete. 2024-01-24T17:41:40.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.652+0000 7f85bcc00640 1 -- 172.21.15.39:0/1468140536 shutdown_connections 2024-01-24T17:41:40.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.653+0000 7f5f06800640 1 --2- 172.21.15.39:0/568556785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f08158af0 0x7f5f0815d130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b10d0 tx=0x7f5ef802bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbe29ed246ca6e43 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.653+0000 7f2ff2000640 1 -- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec15d020 msgr2=0x7f2fec0d2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.653+0000 7f85bcc00640 1 -- 172.21.15.39:0/1468140536 wait complete. 2024-01-24T17:41:40.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.653+0000 7f2ff2000640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec15d020 0x7f2fec0d2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.653+0000 7f2ff2000640 1 -- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fec0cf990 msgr2=0x7f2fec152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.653+0000 7f2ff2000640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fec0cf990 0x7f2fec152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.653+0000 7f2ff2000640 1 -- 172.21.15.39:0/3249828091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc009a70 con 0x7f2fec15fb90 2024-01-24T17:41:40.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.653+0000 7f2ff2000640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15fb90 0x7f2fec0d3130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400e7f0 tx=0x7f2fe400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f957f400640 1 --2- 172.21.15.39:0/1150538454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 0x7f95800b0440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f95700097a0 tx=0x7f957002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ea5c8d2d1ea6c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f2fe9600640 1 -- 172.21.15.39:0/3249828091 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe400c940 con 0x7f2fec15fb90 2024-01-24T17:41:40.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f2fe9600640 1 -- 172.21.15.39:0/3249828091 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe400cac0 con 0x7f2fec15fb90 2024-01-24T17:41:40.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f2fe9600640 1 -- 172.21.15.39:0/3249828091 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe400ce70 con 0x7f2fec15fb90 2024-01-24T17:41:40.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f2ff4000640 1 -- 172.21.15.39:0/3249828091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec0c82c0 con 0x7f2fec15fb90 2024-01-24T17:41:40.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f2ff4000640 1 -- 172.21.15.39:0/3249828091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec0c8880 con 0x7f2fec15fb90 2024-01-24T17:41:40.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f957ea00640 1 -- 172.21.15.39:0/1150538454 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570030610 con 0x7f95800affc0 2024-01-24T17:41:40.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f957ea00640 1 -- 172.21.15.39:0/1150538454 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9570030770 con 0x7f95800affc0 2024-01-24T17:41:40.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.654+0000 7f957ea00640 1 -- 172.21.15.39:0/1150538454 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570030b00 con 0x7f95800affc0 2024-01-24T17:41:40.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f5f05e00640 1 -- 172.21.15.39:0/568556785 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef802cc70 con 0x7f5f08158af0 2024-01-24T17:41:40.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f5f05e00640 1 -- 172.21.15.39:0/568556785 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef802cdd0 con 0x7f5f08158af0 2024-01-24T17:41:40.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f5f05e00640 1 -- 172.21.15.39:0/568556785 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8034660 con 0x7f5f08158af0 2024-01-24T17:41:40.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f762d800640 1 Processor -- start 2024-01-24T17:41:40.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f9586a00640 1 -- 172.21.15.39:0/1150538454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 msgr2=0x7f95800b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f9586a00640 1 --2- 172.21.15.39:0/1150538454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 0x7f95800b0440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f95700097a0 tx=0x7f957002b960 comp rx=0 tx=0).stop 2024-01-24T17:41:40.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f762d800640 1 -- start start 2024-01-24T17:41:40.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f2fe2a00640 1 -- 172.21.15.39:0/3249828091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb4005250 con 0x7f2fec15fb90 2024-01-24T17:41:40.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f5f0d400640 1 -- 172.21.15.39:0/568556785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f08158af0 msgr2=0x7f5f0815d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f5f0d400640 1 --2- 172.21.15.39:0/568556785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f08158af0 0x7f5f0815d130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b10d0 tx=0x7f5ef802bec0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f85bcc00640 1 Processor -- start 2024-01-24T17:41:40.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f85bcc00640 1 -- start start 2024-01-24T17:41:40.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f9586a00640 1 -- 172.21.15.39:0/1150538454 shutdown_connections 2024-01-24T17:41:40.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.655+0000 7f9586a00640 1 --2- 172.21.15.39:0/1150538454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 0x7f95800b0440 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f9586a00640 1 --2- 172.21.15.39:0/1150538454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f95800cbab0 0x7f9580158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f9586a00640 1 -- 172.21.15.39:0/1150538454 >> 172.21.15.39:0/1150538454 conn(0x7f95800cafc0 msgr2=0x7f95800cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f9586a00640 1 -- 172.21.15.39:0/1150538454 shutdown_connections 2024-01-24T17:41:40.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f9586a00640 1 -- 172.21.15.39:0/1150538454 wait complete. 2024-01-24T17:41:40.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f5f0d400640 1 -- 172.21.15.39:0/568556785 shutdown_connections 2024-01-24T17:41:40.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f5f0d400640 1 --2- 172.21.15.39:0/568556785 >> v2:172.21.15.39:3300/0 conn(0x7f5f080cb470 0x7f5f08071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f5f0d400640 1 --2- 172.21.15.39:0/568556785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f08158af0 0x7f5f0815d130 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b10d0 tx=0x7f5ef802bec0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.656+0000 7f5f0d400640 1 -- 172.21.15.39:0/568556785 >> 172.21.15.39:0/568556785 conn(0x7f5f0806a3e0 msgr2=0x7f5f0806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:40.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.657+0000 7f5f0d400640 1 -- 172.21.15.39:0/568556785 shutdown_connections 2024-01-24T17:41:40.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.657+0000 7f2fe9600640 1 -- 172.21.15.39:0/3249828091 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2fe4013470 con 0x7f2fec15fb90 2024-01-24T17:41:40.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.657+0000 7f5f0d400640 1 -- 172.21.15.39:0/568556785 wait complete. 2024-01-24T17:41:40.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f762d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76280cf9e0 0x7f76281f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85bcc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b806ac00 0x7f85b80cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85bcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85b80b1b20 0x7f85b8161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85bcc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80ce1f0 0x7f85b81f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85bcc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8160b70 con 0x7f85b80b1b20 2024-01-24T17:41:40.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85bcc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8160560 con 0x7f85b80ce1f0 2024-01-24T17:41:40.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85bcc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8162440 con 0x7f85b806ac00 2024-01-24T17:41:40.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f762d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 0x7f762815f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f2fe9600640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2fc406a430 0x7f2fc406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f762d800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f762815cc90 0x7f762815ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f2fe9600640 1 -- 172.21.15.39:0/3249828091 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fe40885c0 con 0x7f2fec15fb90 2024-01-24T17:41:40.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f762d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280c70d0 con 0x7f76280cfe60 2024-01-24T17:41:40.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f762d800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280c6730 con 0x7f762815cc90 2024-01-24T17:41:40.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f762d800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76280c6a10 con 0x7f76280cf9e0 2024-01-24T17:41:40.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85b7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80ce1f0 0x7f85b81f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85b7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80ce1f0 0x7f85b81f6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49468/0 (socket says 172.21.15.39:49468) 2024-01-24T17:41:40.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.658+0000 7f85b7e00640 1 -- 172.21.15.39:0/377561047 learned_addr learned my addr 172.21.15.39:0/377561047 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f9586a00640 1 Processor -- start 2024-01-24T17:41:40.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f9586a00640 1 -- start start 2024-01-24T17:41:40.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f2ff1600640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2fc406a430 0x7f2fc406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f7626a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76280cf9e0 0x7f76281f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f7627400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f762815cc90 0x7f762815ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f7627400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f762815cc90 0x7f762815ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49480/0 (socket says 172.21.15.39:49480) 2024-01-24T17:41:40.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f7627400640 1 -- 172.21.15.39:0/1188765688 learned_addr learned my addr 172.21.15.39:0/1188765688 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f85b7400640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b806ac00 0x7f85b80cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f85b7e00640 1 -- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b806ac00 msgr2=0x7f85b80cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f85b7e00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b806ac00 0x7f85b80cc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f85b7e00640 1 -- 172.21.15.39:0/377561047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85b80b1b20 msgr2=0x7f85b8161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f85b7e00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85b80b1b20 0x7f85b8161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.659+0000 7f85b7e00640 1 -- 172.21.15.39:0/377561047 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a8009400 con 0x7f85b80ce1f0 2024-01-24T17:41:40.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f7626000640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 0x7f762815f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f7627400640 1 -- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76280cf9e0 msgr2=0x7f76281f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f7627400640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76280cf9e0 0x7f76281f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f7627400640 1 -- 172.21.15.39:0/1188765688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 msgr2=0x7f762815f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f7627400640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 0x7f762815f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f7627400640 1 -- 172.21.15.39:0/1188765688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610009400 con 0x7f762815cc90 2024-01-24T17:41:40.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f85b7e00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80ce1f0 0x7f85b81f6810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c450 tx=0x7f85a000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f85af400640 1 -- 172.21.15.39:0/377561047 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00074e0 con 0x7f85b80ce1f0 2024-01-24T17:41:40.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.660+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b81f73b0 con 0x7f85b80ce1f0 2024-01-24T17:41:40.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.661+0000 7f85af400640 1 -- 172.21.15.39:0/377561047 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a0007660 con 0x7f85b80ce1f0 2024-01-24T17:41:40.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.661+0000 7f85af400640 1 -- 172.21.15.39:0/377561047 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0004a20 con 0x7f85b80ce1f0 2024-01-24T17:41:40.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.661+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b8160800 con 0x7f85b80ce1f0 2024-01-24T17:41:40.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.661+0000 7f2ff1600640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2fc406a430 0x7f2fc406c8f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc000f80 tx=0x7f2fdc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.661+0000 7f9586a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 0x7f95800b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.661+0000 7f9586a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95800cbab0 0x7f95800b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.661+0000 7f9586a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f958015c6c0 0x7f95800c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f7627400640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f762815cc90 0x7f762815ac10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f761002bd20 tx=0x7f76100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f5f0d400640 1 Processor -- start 2024-01-24T17:41:40.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f9586a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95800c5fb0 con 0x7f95800affc0 2024-01-24T17:41:40.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f5f0d400640 1 -- start start 2024-01-24T17:41:40.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f9586a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95800b3850 con 0x7f95800cbab0 2024-01-24T17:41:40.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b80b1940 con 0x7f85b80ce1f0 2024-01-24T17:41:40.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f957fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 0x7f95800b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f761ea00640 1 -- 172.21.15.39:0/1188765688 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761003d950 con 0x7f762815cc90 2024-01-24T17:41:40.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.662+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76280c6cf0 con 0x7f762815cc90 2024-01-24T17:41:40.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762815f730 con 0x7f762815cc90 2024-01-24T17:41:40.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f9584a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f958015c6c0 0x7f95800c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f9584a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f958015c6c0 0x7f95800c31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48208/0 (socket says 172.21.15.39:48208) 2024-01-24T17:41:40.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f9584a00640 1 -- 172.21.15.39:0/743836298 learned_addr learned my addr 172.21.15.39:0/743836298 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95800b3b30 con 0x7f958015c6c0 2024-01-24T17:41:40.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f957f400640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95800cbab0 0x7f95800b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f5f0d400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f080c95a0 0x7f5f08152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f5f0d400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f080cb470 0x7f5f080adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f5f0d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f080ae3e0 0x7f5f08154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f5f0d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080c4020 con 0x7f5f080ae3e0 2024-01-24T17:41:40.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f5f0d400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080c3410 con 0x7f5f080cb470 2024-01-24T17:41:40.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f5f0d400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080c36f0 con 0x7f5f080c95a0 2024-01-24T17:41:40.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f761ea00640 1 -- 172.21.15.39:0/1188765688 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761003dab0 con 0x7f762815cc90 2024-01-24T17:41:40.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.663+0000 7f761ea00640 1 -- 172.21.15.39:0/1188765688 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761003de40 con 0x7f762815cc90 2024-01-24T17:41:40.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.664+0000 7f957fe00640 1 -- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f958015c6c0 msgr2=0x7f95800c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.664+0000 7f5f06800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f080c95a0 0x7f5f08152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.664+0000 7f5f06800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f080c95a0 0x7f5f08152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48210/0 (socket says 172.21.15.39:48210) 2024-01-24T17:41:40.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.664+0000 7f5f06800640 1 -- 172.21.15.39:0/4210124337 learned_addr learned my addr 172.21.15.39:0/4210124337 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:40.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.664+0000 7f5f07200640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f080ae3e0 0x7f5f08154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.664+0000 7f957fe00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f958015c6c0 0x7f95800c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.664+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f8005250 con 0x7f762815cc90 2024-01-24T17:41:40.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f85af400640 1 -- 172.21.15.39:0/377561047 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f85a001c3b0 con 0x7f85b80ce1f0 2024-01-24T17:41:40.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f957fe00640 1 -- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95800cbab0 msgr2=0x7f95800b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f957fe00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95800cbab0 0x7f95800b2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f957fe00640 1 -- 172.21.15.39:0/743836298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9570009400 con 0x7f95800affc0 2024-01-24T17:41:40.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f2fe9600640 1 -- 172.21.15.39:0/3249828091 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2fe4053cf0 con 0x7f2fec15fb90 2024-01-24T17:41:40.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f957fe00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 0x7f95800b2a80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f957800bd70 tx=0x7f957800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.666+0000 7f9577e00640 1 -- 172.21.15.39:0/743836298 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578014440 con 0x7f95800affc0 2024-01-24T17:41:40.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.666+0000 7f9577e00640 1 -- 172.21.15.39:0/743836298 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95780145c0 con 0x7f95800affc0 2024-01-24T17:41:40.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.666+0000 7f9577e00640 1 -- 172.21.15.39:0/743836298 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578014970 con 0x7f95800affc0 2024-01-24T17:41:40.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.666+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95800b3e10 con 0x7f95800affc0 2024-01-24T17:41:40.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.666+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95800c5990 con 0x7f95800affc0 2024-01-24T17:41:40.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f5f06800640 1 -- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f080cb470 msgr2=0x7f5f080adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f5f06800640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f080cb470 0x7f5f080adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f5f06800640 1 -- 172.21.15.39:0/4210124337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f080ae3e0 msgr2=0x7f5f08154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:40.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f5f06800640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f080ae3e0 0x7f5f08154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:40.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.665+0000 7f5f06800640 1 -- 172.21.15.39:0/4210124337 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef8009a70 con 0x7f5f080c95a0 2024-01-24T17:41:40.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.666+0000 7f5f06800640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f080c95a0 0x7f5f08152bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5ef802c450 tx=0x7f5ef8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.666+0000 7f85af400640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f859006a460 0x7f859006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.666+0000 7f85af400640 1 -- 172.21.15.39:0/377561047 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85a00880a0 con 0x7f85b80ce1f0 2024-01-24T17:41:40.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.667+0000 7f85b7400640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f859006a460 0x7f859006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.667+0000 7f761ea00640 1 -- 172.21.15.39:0/1188765688 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7610039040 con 0x7f762815cc90 2024-01-24T17:41:40.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.668+0000 7f761ea00640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f760006a430 0x7f760006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.668+0000 7f761ea00640 1 -- 172.21.15.39:0/1188765688 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76100aab60 con 0x7f762815cc90 2024-01-24T17:41:40.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.667+0000 7f957f400640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95800cbab0 0x7f95800b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:40.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.668+0000 7f9577e00640 1 -- 172.21.15.39:0/743836298 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f957801d3b0 con 0x7f95800affc0 2024-01-24T17:41:40.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.668+0000 7f7626a00640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f760006a430 0x7f760006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.669+0000 7f85b7400640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f859006a460 0x7f859006c920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f85b0003b80 tx=0x7f85b0009890 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.669+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9580073ef0 con 0x7f95800affc0 2024-01-24T17:41:40.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.669+0000 7f9577e00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f955806a500 0x7f955806c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.669+0000 7f9577e00640 1 -- 172.21.15.39:0/743836298 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9578088b00 con 0x7f95800affc0 2024-01-24T17:41:40.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.669+0000 7f957f400640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f955806a500 0x7f955806c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.670+0000 7f7626a00640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f760006a430 0x7f760006c8f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7620001a80 tx=0x7f7620005bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.670+0000 7f957f400640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f955806a500 0x7f955806c9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9570036000 tx=0x7f957003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.674+0000 7f9577e00640 1 -- 172.21.15.39:0/743836298 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9578054180 con 0x7f95800affc0 2024-01-24T17:41:40.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.676+0000 7f963d600640 1 -- 172.21.15.39:0/238821829 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f96300549b0 con 0x7f9620004310 2024-01-24T17:41:40.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7fc696a00640 1 -- 172.21.15.39:0/919210988 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00add90 con 0x7fc698006b60 2024-01-24T17:41:40.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7f5efea00640 1 -- 172.21.15.39:0/4210124337 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef803e610 con 0x7f5f080c95a0 2024-01-24T17:41:40.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69804d3e0 con 0x7fc698006b60 2024-01-24T17:41:40.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f080c39d0 con 0x7f5f080c95a0 2024-01-24T17:41:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7fc696a00640 1 -- 172.21.15.39:0/919210988 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a00b6400 con 0x7fc698006b60 2024-01-24T17:41:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6980e0440 con 0x7fc698006b60 2024-01-24T17:41:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f08155230 con 0x7f5f080c95a0 2024-01-24T17:41:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7f5efea00640 1 -- 172.21.15.39:0/4210124337 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef803e770 con 0x7f5f080c95a0 2024-01-24T17:41:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.677+0000 7f5efea00640 1 -- 172.21.15.39:0/4210124337 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef803eae0 con 0x7f5f080c95a0 2024-01-24T17:41:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.678+0000 7fc696a00640 1 -- 172.21.15.39:0/919210988 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00b6770 con 0x7fc698006b60 2024-01-24T17:41:40.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.678+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed8005250 con 0x7f5f080c95a0 2024-01-24T17:41:40.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.678+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69804cf60 con 0x7fc698006b60 2024-01-24T17:41:40.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.683+0000 7f21a7e00640 1 -- 172.21.15.39:0/2244025809 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f21b00738b0 con 0x7f21a80066d0 2024-01-24T17:41:40.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.684+0000 7f21a7e00640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f218806a500 0x7f218806c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.684+0000 7f21a7e00640 1 -- 172.21.15.39:0/2244025809 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21b0102260 con 0x7f21a80066d0 2024-01-24T17:41:40.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.685+0000 7f21af400640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f218806a500 0x7f218806c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.686+0000 7f21af400640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f218806a500 0x7f218806c9c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f21a0037000 tx=0x7f21a0032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.692+0000 7f85af400640 1 -- 172.21.15.39:0/377561047 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f85a00571b0 con 0x7f85b80ce1f0 2024-01-24T17:41:40.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.693+0000 7f21a7e00640 1 -- 172.21.15.39:0/2244025809 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f21b00d1400 con 0x7f21a80066d0 2024-01-24T17:41:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.696+0000 7f5efea00640 1 -- 172.21.15.39:0/4210124337 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5ef803b050 con 0x7f5f080c95a0 2024-01-24T17:41:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.698+0000 7fc696a00640 1 -- 172.21.15.39:0/919210988 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc6a00b4040 con 0x7fc698006b60 2024-01-24T17:41:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.698+0000 7f5efea00640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ee006a460 0x7f5ee006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.698+0000 7f5efea00640 1 -- 172.21.15.39:0/4210124337 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ef80ab690 con 0x7f5f080c95a0 2024-01-24T17:41:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.699+0000 7f5effe00640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ee006a460 0x7f5ee006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.699+0000 7fc696a00640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc67806a490 0x7fc67806c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.699+0000 7fc696a00640 1 -- 172.21.15.39:0/919210988 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6a01237a0 con 0x7fc698006b60 2024-01-24T17:41:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.700+0000 7fc69e000640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc67806a490 0x7fc67806c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:40.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.700+0000 7f5effe00640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ee006a460 0x7f5ee006c920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5f080cd1a0 tx=0x7f5ef0007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.701+0000 7fc69e000640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc67806a490 0x7fc67806c950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc69803c330 tx=0x7fc690009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:40.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.704+0000 7f761ea00640 1 -- 172.21.15.39:0/1188765688 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7610076230 con 0x7f762815cc90 2024-01-24T17:41:40.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.707+0000 7f5efea00640 1 -- 172.21.15.39:0/4210124337 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5ef8076d30 con 0x7f5f080c95a0 2024-01-24T17:41:40.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:40.718+0000 7fc696a00640 1 -- 172.21.15.39:0/919210988 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc6a00eed90 con 0x7fc698006b60 2024-01-24T17:41:41.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.052+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9638008b50 con 0x7f9620004310 2024-01-24T17:41:41.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.054+0000 7f963d600640 1 -- 172.21.15.39:0/238821829 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9630057fd0 con 0x7f9620004310 2024-01-24T17:41:41.055 INFO:teuthology.orchestra.run.smithi039.stdout:55834574853 2024-01-24T17:41:41.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.057+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f961806a460 msgr2=0x7f961806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.057+0000 7f9648000640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f961806a460 0x7f961806c920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9638006440 tx=0x7f9628008030 comp rx=0 tx=0).stop 2024-01-24T17:41:41.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.058+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620004310 msgr2=0x7f963817a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.058+0000 7f9648000640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620004310 0x7f963817a030 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f96300108b0 tx=0x7f9630010d70 comp rx=0 tx=0).stop 2024-01-24T17:41:41.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.058+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 shutdown_connections 2024-01-24T17:41:41.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.058+0000 7f9648000640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f961806a460 0x7f961806c920 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.058+0000 7f9648000640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96380d3f10 0x7f963817aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.058+0000 7f9648000640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96380d3a50 0x7f963817a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.058+0000 7f9648000640 1 --2- 172.21.15.39:0/238821829 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9620004310 0x7f963817a030 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.058+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 >> 172.21.15.39:0/238821829 conn(0x7f9638054c40 msgr2=0x7f96380da580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:41.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.059+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 shutdown_connections 2024-01-24T17:41:41.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.059+0000 7f9648000640 1 -- 172.21.15.39:0/238821829 wait complete. 2024-01-24T17:41:41.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.067+0000 7f2fe2a00640 1 -- 172.21.15.39:0/3249828091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2fb40057d0 con 0x7f2fec15fb90 2024-01-24T17:41:41.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.068+0000 7f2fe9600640 1 -- 172.21.15.39:0/3249828091 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2fe4057310 con 0x7f2fec15fb90 2024-01-24T17:41:41.069 INFO:teuthology.orchestra.run.smithi039.stdout:55834574854 2024-01-24T17:41:41.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 -- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2fc406a430 msgr2=0x7f2fc406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2fc406a430 0x7f2fc406c8f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc000f80 tx=0x7f2fdc030000 comp rx=0 tx=0).stop 2024-01-24T17:41:41.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 -- 172.21.15.39:0/3249828091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15fb90 msgr2=0x7f2fec0d3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15fb90 0x7f2fec0d3130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400e7f0 tx=0x7f2fe400ecb0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 -- 172.21.15.39:0/3249828091 shutdown_connections 2024-01-24T17:41:41.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2fc406a430 0x7f2fc406c8f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fec15fb90 0x7f2fec0d3130 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fec15d020 0x7f2fec0d2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 --2- 172.21.15.39:0/3249828091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fec0cf990 0x7f2fec152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.072+0000 7f2fe2a00640 1 -- 172.21.15.39:0/3249828091 >> 172.21.15.39:0/3249828091 conn(0x7f2fec071790 msgr2=0x7f2fec160050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:41.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.073+0000 7f2fe2a00640 1 -- 172.21.15.39:0/3249828091 shutdown_connections 2024-01-24T17:41:41.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.073+0000 7f2fe2a00640 1 -- 172.21.15.39:0/3249828091 wait complete. 2024-01-24T17:41:41.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.082+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f95800b44c0 con 0x7f95800affc0 2024-01-24T17:41:41.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.083+0000 7f9577e00640 1 -- 172.21.15.39:0/743836298 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9578019020 con 0x7f95800affc0 2024-01-24T17:41:41.086 INFO:teuthology.orchestra.run.smithi039.stdout:55834574854 2024-01-24T17:41:41.087 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-01-24T17:41:41.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f955806a500 msgr2=0x7f955806c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f955806a500 0x7f955806c9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9570036000 tx=0x7f957003b040 comp rx=0 tx=0).stop 2024-01-24T17:41:41.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 msgr2=0x7f95800b2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 0x7f95800b2a80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f957800bd70 tx=0x7f957800be60 comp rx=0 tx=0).stop 2024-01-24T17:41:41.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 shutdown_connections 2024-01-24T17:41:41.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f958015c6c0 0x7f95800c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f95800cbab0 0x7f95800b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f955806a500 0x7f955806c9c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 --2- 172.21.15.39:0/743836298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95800affc0 0x7f95800b2a80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.087+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 >> 172.21.15.39:0/743836298 conn(0x7f95800cafc0 msgr2=0x7f958006d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:41.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.088+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 shutdown_connections 2024-01-24T17:41:41.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.088+0000 7f9586a00640 1 -- 172.21.15.39:0/743836298 wait complete. 2024-01-24T17:41:41.099 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-01-24T17:41:41.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.105+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f21a8052d00 con 0x7f21a80066d0 2024-01-24T17:41:41.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.106+0000 7f21a7e00640 1 -- 172.21.15.39:0/2244025809 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f21b0098090 con 0x7f21a80066d0 2024-01-24T17:41:41.107 INFO:teuthology.orchestra.run.smithi039.stdout:55834574855 2024-01-24T17:41:41.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.108+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f218806a500 msgr2=0x7f218806c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.108+0000 7f21b5800640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f218806a500 0x7f218806c9c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f21a0037000 tx=0x7f21a0032000 comp rx=0 tx=0).stop 2024-01-24T17:41:41.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.108+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21a80066d0 msgr2=0x7f21a80d8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.108+0000 7f21b5800640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21a80066d0 0x7f21a80d8970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f21b004eda0 tx=0x7f21b0066fc0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.109+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 shutdown_connections 2024-01-24T17:41:41.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.109+0000 7f21b5800640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21a8053150 0x7f21a80dfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.109+0000 7f21b5800640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f218806a500 0x7f218806c9c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.109+0000 7f21b5800640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21a8033fc0 0x7f21a817c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.109+0000 7f21b5800640 1 --2- 172.21.15.39:0/2244025809 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21a80066d0 0x7f21a80d8970 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.109+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 >> 172.21.15.39:0/2244025809 conn(0x7f21a8054c10 msgr2=0x7f21a8035b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:41.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.109+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 shutdown_connections 2024-01-24T17:41:41.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.109+0000 7f21b5800640 1 -- 172.21.15.39:0/2244025809 wait complete. 2024-01-24T17:41:41.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.110+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f75f80057d0 con 0x7f762815cc90 2024-01-24T17:41:41.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.111+0000 7f761ea00640 1 -- 172.21.15.39:0/1188765688 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f761004d090 con 0x7f762815cc90 2024-01-24T17:41:41.112 INFO:teuthology.orchestra.run.smithi039.stdout:55834574855 2024-01-24T17:41:41.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f760006a430 msgr2=0x7f760006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f760006a430 0x7f760006c8f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7620001a80 tx=0x7f7620005bd0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f762815cc90 msgr2=0x7f762815ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f762815cc90 0x7f762815ac10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f761002bd20 tx=0x7f76100046c0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 shutdown_connections 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f762815cc90 0x7f762815ac10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f760006a430 0x7f760006c8f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76280cfe60 0x7f762815f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 --2- 172.21.15.39:0/1188765688 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76280cf9e0 0x7f76281f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 >> 172.21.15.39:0/1188765688 conn(0x7f7628071790 msgr2=0x7f76280d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 shutdown_connections 2024-01-24T17:41:41.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.114+0000 7f762d800640 1 -- 172.21.15.39:0/1188765688 wait complete. 2024-01-24T17:41:41.116 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2024-01-24T17:41:41.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.122+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f85b8160ce0 con 0x7f85b80ce1f0 2024-01-24T17:41:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.123+0000 7f85af400640 1 -- 172.21.15.39:0/377561047 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f85a0018090 con 0x7f85b80ce1f0 2024-01-24T17:41:41.123 INFO:teuthology.orchestra.run.smithi039.stdout:55834574852 2024-01-24T17:41:41.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc69804d900 con 0x7fc698006b60 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f859006a460 msgr2=0x7f859006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f859006a460 0x7f859006c920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f85b0003b80 tx=0x7f85b0009890 comp rx=0 tx=0).stop 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80ce1f0 msgr2=0x7f85b81f6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80ce1f0 0x7f85b81f6810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c450 tx=0x7f85a000c910 comp rx=0 tx=0).stop 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 shutdown_connections 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85b80ce1f0 0x7f85b81f6810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f859006a460 0x7f859006c920 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85b80b1b20 0x7f85b8161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 --2- 172.21.15.39:0/377561047 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85b806ac00 0x7f85b80cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.126+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 >> 172.21.15.39:0/377561047 conn(0x7f85b80d27d0 msgr2=0x7f85b806b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:41.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.127+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 shutdown_connections 2024-01-24T17:41:41.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.127+0000 7fc696a00640 1 -- 172.21.15.39:0/919210988 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc69804d900 con 0x7fc698006b60 2024-01-24T17:41:41.129 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2024-01-24T17:41:41.129 INFO:teuthology.orchestra.run.smithi039.stdout:55834574853 2024-01-24T17:41:41.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.127+0000 7f85bcc00640 1 -- 172.21.15.39:0/377561047 wait complete. 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc67806a490 msgr2=0x7fc67806c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc67806a490 0x7fc67806c950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc69803c330 tx=0x7fc690009040 comp rx=0 tx=0).stop 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc698006b60 msgr2=0x7fc6980da540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc698006b60 0x7fc6980da540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0074a60 tx=0x7fc6a0061a20 comp rx=0 tx=0).stop 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 shutdown_connections 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6980e3060 0x7fc698050c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc67806a490 0x7fc67806c950 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc698054350 0x7fc6980dff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 --2- 172.21.15.39:0/919210988 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc698006b60 0x7fc6980da540 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 >> 172.21.15.39:0/919210988 conn(0x7fc698055a80 msgr2=0x7fc6980dd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 shutdown_connections 2024-01-24T17:41:41.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.130+0000 7fc6a5600640 1 -- 172.21.15.39:0/919210988 wait complete. 2024-01-24T17:41:41.132 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2024-01-24T17:41:41.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.132+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5ed80057d0 con 0x7f5f080c95a0 2024-01-24T17:41:41.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.132+0000 7f5efea00640 1 -- 172.21.15.39:0/4210124337 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ef804e020 con 0x7f5f080c95a0 2024-01-24T17:41:41.133 INFO:teuthology.orchestra.run.smithi039.stdout:55834574853 2024-01-24T17:41:41.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ee006a460 msgr2=0x7f5ee006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ee006a460 0x7f5ee006c920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5f080cd1a0 tx=0x7f5ef0007a40 comp rx=0 tx=0).stop 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f080c95a0 msgr2=0x7f5f08152bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f080c95a0 0x7f5f08152bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5ef802c450 tx=0x7f5ef8004a40 comp rx=0 tx=0).stop 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 shutdown_connections 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ee006a460 0x7f5ee006c920 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f080ae3e0 0x7f5f08154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f080cb470 0x7f5f080adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 --2- 172.21.15.39:0/4210124337 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f080c95a0 0x7f5f08152bb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 >> 172.21.15.39:0/4210124337 conn(0x7f5f0806a3e0 msgr2=0x7f5f080cda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 shutdown_connections 2024-01-24T17:41:41.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:41.135+0000 7f5f0d400640 1 -- 172.21.15.39:0/4210124337 wait complete. 2024-01-24T17:41:41.148 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.6 2024-01-24T17:41:41.151 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-01-24T17:41:41.154 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-01-24T17:41:42.088 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-24T17:41:42.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-24T17:41:42.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-24T17:41:42.130 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-24T17:41:42.133 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-24T17:41:42.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-24T17:41:42.151 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-24T17:41:42.155 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-24T17:41:42.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.378+0000 7f6fe9800640 1 Processor -- start 2024-01-24T17:41:42.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.378+0000 7f6fe9800640 1 -- start start 2024-01-24T17:41:42.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.380+0000 7f6fe9800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6fdc0e78b0 0x7f6fdc0545a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.380+0000 7f6fe9800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc006ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.381+0000 7f6fe9800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc053b00 con 0x7f6fdc04f260 2024-01-24T17:41:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.381+0000 7f6fe9800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fdc048570 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.381+0000 7f6fe9800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc04ffe0 con 0x7f6fdc038450 2024-01-24T17:41:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.381+0000 7f6fe3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6fdc0e78b0 0x7f6fdc0545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.381+0000 7f6fe3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6fdc0e78b0 0x7f6fdc0545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58492/0 (socket says 172.21.15.39:58492) 2024-01-24T17:41:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.381+0000 7f6fe3e00640 1 -- 172.21.15.39:0/2106747503 learned_addr learned my addr 172.21.15.39:0/2106747503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.381+0000 7f6fe8e00640 1 --2- 172.21.15.39:0/2106747503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc006ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.382+0000 7f6fe3e00640 1 -- 172.21.15.39:0/2106747503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 msgr2=0x7f6fdc006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.383+0000 7f6fe3e00640 1 --2- 172.21.15.39:0/2106747503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc006ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.383+0000 7f6fe3e00640 1 -- 172.21.15.39:0/2106747503 >> v1:172.21.15.100:6789/0 conn(0x7f6fdc04f260 legacy=0x7f6fdc037d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.383+0000 7f6fe3e00640 1 -- 172.21.15.39:0/2106747503 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc053da0 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.384+0000 7f6fe3e00640 1 --2- 172.21.15.39:0/2106747503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0545a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4068a50 tx=0x7f6fe40a5800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a18fd23b4c4aff5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.385+0000 7f6fe2a00640 1 -- 172.21.15.39:0/2106747503 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40a9050 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.385+0000 7f6fe2a00640 1 -- 172.21.15.39:0/2106747503 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe40ad5f0 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.386+0000 7f6fe2a00640 1 -- 172.21.15.39:0/2106747503 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40ada20 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.387+0000 7f6fe9800640 1 -- 172.21.15.39:0/2106747503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 msgr2=0x7f6fdc0545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.387+0000 7f6fe9800640 1 --2- 172.21.15.39:0/2106747503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0545a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4068a50 tx=0x7f6fe40a5800 comp rx=0 tx=0).stop 2024-01-24T17:41:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.388+0000 7f6fe9800640 1 -- 172.21.15.39:0/2106747503 shutdown_connections 2024-01-24T17:41:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.388+0000 7f6fe9800640 1 --2- 172.21.15.39:0/2106747503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc006ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.388+0000 7f6fe9800640 1 --2- 172.21.15.39:0/2106747503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0545a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.388+0000 7f6fe9800640 1 -- 172.21.15.39:0/2106747503 >> 172.21.15.39:0/2106747503 conn(0x7f6fdc054c10 msgr2=0x7f6fdc055020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.388+0000 7f6fe9800640 1 -- 172.21.15.39:0/2106747503 shutdown_connections 2024-01-24T17:41:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.388+0000 7f6fe9800640 1 -- 172.21.15.39:0/2106747503 wait complete. 2024-01-24T17:41:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.391+0000 7f6fe9800640 1 Processor -- start 2024-01-24T17:41:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.392+0000 7f6fe9800640 1 -- start start 2024-01-24T17:41:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.394+0000 7f6fe9800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc0ea310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.395+0000 7f6fe9800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fdc04f260 0x7f6fdc17d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.395+0000 7f6fe9800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0e8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.395+0000 7f6fe8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0e8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.396+0000 7f6fe8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0e8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58494/0 (socket says 172.21.15.39:58494) 2024-01-24T17:41:42.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.396+0000 7f6fe8e00640 1 -- 172.21.15.39:0/3218346727 learned_addr learned my addr 172.21.15.39:0/3218346727 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.396+0000 7f6fe3e00640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc0ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.396+0000 7f6fe9800640 1 -- 172.21.15.39:0/3218346727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc0487e0 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.396+0000 7f6fe9800640 1 -- 172.21.15.39:0/3218346727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc048ac0 con 0x7f6fdc04f260 2024-01-24T17:41:42.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.397+0000 7f6fe3400640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fdc04f260 0x7f6fdc17d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.397+0000 7f6fe9800640 1 -- 172.21.15.39:0/3218346727 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc048da0 con 0x7f6fdc038450 2024-01-24T17:41:42.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.398+0000 7f6fe8e00640 1 -- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 msgr2=0x7f6fdc0ea310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.398+0000 7f6fe8e00640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc0ea310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.398+0000 7f6fe8e00640 1 -- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fdc04f260 msgr2=0x7f6fdc17d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.398+0000 7f6fe8e00640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fdc04f260 0x7f6fdc17d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.398+0000 7f6fe8e00640 1 -- 172.21.15.39:0/3218346727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe404eee0 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.398+0000 7f6fe8e00640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0e8730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00c7f0 tx=0x7f6fcc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.399+0000 7f6fe3e00640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc0ea310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:42.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.400+0000 7f6fe3400640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fdc04f260 0x7f6fdc17d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:42.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.400+0000 7f6fdbe00640 1 -- 172.21.15.39:0/3218346727 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc004220 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.400+0000 7f6fdbe00640 1 -- 172.21.15.39:0/3218346727 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc004380 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.400+0000 7f6fdbe00640 1 -- 172.21.15.39:0/3218346727 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc007820 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.400+0000 7f6fe9800640 1 -- 172.21.15.39:0/3218346727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc049080 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.402+0000 7f6fe9800640 1 -- 172.21.15.39:0/3218346727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc0e8c70 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.406+0000 7f6fdbe00640 1 -- 172.21.15.39:0/3218346727 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6fcc007980 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.407+0000 7f6fd9600640 1 -- 172.21.15.39:0/3218346727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac005250 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.409+0000 7f6fdbe00640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a4b0 0x7f6fbc06c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.410+0000 7f6fe3e00640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a4b0 0x7f6fbc06c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.410+0000 7f6fdbe00640 1 -- 172.21.15.39:0/3218346727 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fcc057430 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.413+0000 7f6fe3e00640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a4b0 0x7f6fbc06c970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00da90 tx=0x7f6fe406e030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.418+0000 7f6fdbe00640 1 -- 172.21.15.39:0/3218346727 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6fcc08d060 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.427+0000 7fa535800640 1 Processor -- start 2024-01-24T17:41:42.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.428+0000 7fa535800640 1 -- start start 2024-01-24T17:41:42.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.429+0000 7fa535800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015d020 0x7fa53015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.429+0000 7fa535800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa53015eb80 0x7fa53015f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.430+0000 7fa535800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53015dac0 con 0x7fa5300cf990 2024-01-24T17:41:42.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.430+0000 7fa535800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53015e080 con 0x7fa53015eb80 2024-01-24T17:41:42.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.430+0000 7fa535800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53015dda0 con 0x7fa53015d020 2024-01-24T17:41:42.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.430+0000 7f827ea00640 1 Processor -- start 2024-01-24T17:41:42.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.431+0000 7fa52e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015d020 0x7fa53015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.431+0000 7fa52f400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa53015eb80 0x7fa53015f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.431+0000 7fa52f400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa53015eb80 0x7fa53015f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49506/0 (socket says 172.21.15.39:49506) 2024-01-24T17:41:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.431+0000 7fa52f400640 1 -- 172.21.15.39:0/226609722 learned_addr learned my addr 172.21.15.39:0/226609722 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.431+0000 7f827ea00640 1 -- start start 2024-01-24T17:41:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.432+0000 7fa52e000640 1 -- 172.21.15.39:0/226609722 >> v2:172.21.15.100:3300/0 conn(0x7fa53015eb80 msgr2=0x7fa53015f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:42.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.432+0000 7fa52d600640 1 -- 172.21.15.39:0/226609722 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845527145 0 0) 0x7fa53015dac0 con 0x7fa5300cf990 2024-01-24T17:41:42.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.432+0000 7f7fdaa00640 1 Processor -- start 2024-01-24T17:41:42.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.433+0000 7f827ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278158a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.433+0000 7f827ea00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82780708c0 con 0x7f82780b06d0 2024-01-24T17:41:42.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.433+0000 7f827ea00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8278070ba0 con 0x7f82780cbab0 2024-01-24T17:41:42.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.433+0000 7f827ea00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8278158f70 con 0x7f827815cdd0 2024-01-24T17:41:42.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.432+0000 7fa52e000640 1 --2- 172.21.15.39:0/226609722 >> v2:172.21.15.100:3300/0 conn(0x7fa53015eb80 0x7fa53015f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.432+0000 7fa52e000640 1 -- 172.21.15.39:0/226609722 >> v1:172.21.15.100:6789/0 conn(0x7fa5300cf990 legacy=0x7fa5300cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.432+0000 7fa52e000640 1 -- 172.21.15.39:0/226609722 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53015e380 con 0x7fa53015d020 2024-01-24T17:41:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7fa52f400640 1 --2- 172.21.15.39:0/226609722 >> v2:172.21.15.100:3300/0 conn(0x7fa53015eb80 0x7fa53015f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.432+0000 7f7fdaa00640 1 -- start start 2024-01-24T17:41:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7f7fdaa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd415cc90 0x7f7fd40d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7f7fdaa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7fd40cf9e0 0x7f7fd40cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7f7fdaa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40ce160 con 0x7f7fd40b5b10 2024-01-24T17:41:42.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7f7fdaa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd415d070 con 0x7f7fd40cf9e0 2024-01-24T17:41:42.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7f7fdaa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40d03a0 con 0x7f7fd415cc90 2024-01-24T17:41:42.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7f827ca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7f8277e00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f82780cbab0 0x7f82780affc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48748/0 (socket says 172.21.15.39:48748) 2024-01-24T17:41:42.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7f8277e00640 1 -- 172.21.15.39:0/1137195911 learned_addr learned my addr 172.21.15.39:0/1137195911 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:42.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.434+0000 7fa52e000640 1 --2- 172.21.15.39:0/226609722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015d020 0x7fa53015d460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa5200097a0 tx=0x7fa52002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59a256e1fb6425de server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7f7fd3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd415cc90 0x7f7fd40d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7f7fd8a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7fd40b5b10 0x7f7fd40b5fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52482/0 (socket says 172.21.15.39:52482) 2024-01-24T17:41:42.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7f7fd8a00640 1 -- 172.21.15.39:0/670990330 learned_addr learned my addr 172.21.15.39:0/670990330 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:42.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7f7fd3400640 1 --2- 172.21.15.39:0/670990330 >> v2:172.21.15.39:3300/0 conn(0x7f7fd40cf9e0 0x7f7fd40cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7fa52d600640 1 -- 172.21.15.39:0/226609722 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520031640 con 0x7fa53015d020 2024-01-24T17:41:42.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7fa52d600640 1 -- 172.21.15.39:0/226609722 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5200317a0 con 0x7fa53015d020 2024-01-24T17:41:42.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7fa52d600640 1 -- 172.21.15.39:0/226609722 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520031b30 con 0x7fa53015d020 2024-01-24T17:41:42.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7f8276a00640 1 -- 172.21.15.39:0/1137195911 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068579940 0 0) 0x7f8278070ba0 con 0x7f82780cbab0 2024-01-24T17:41:42.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.436+0000 7f7fd3e00640 1 -- 172.21.15.39:0/670990330 >> v2:172.21.15.39:3300/0 conn(0x7f7fd40cf9e0 msgr2=0x7f7fd40cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:42.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.435+0000 7f8276a00640 1 -- 172.21.15.39:0/1137195911 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8258003520 con 0x7f82780cbab0 2024-01-24T17:41:42.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.437+0000 7fa535800640 1 -- 172.21.15.39:0/226609722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015d020 msgr2=0x7fa53015d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.436+0000 7f827ca00640 1 -- 172.21.15.39:0/1137195911 >> v1:172.21.15.100:6789/0 conn(0x7f82780cbab0 legacy=0x7f82780affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.436+0000 7f827ca00640 1 -- 172.21.15.39:0/1137195911 >> v1:172.21.15.39:6789/0 conn(0x7f82780b06d0 legacy=0x7f827815c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:42.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.436+0000 7f7fd3e00640 1 --2- 172.21.15.39:0/670990330 >> v2:172.21.15.39:3300/0 conn(0x7f7fd40cf9e0 0x7f7fd40cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.436+0000 7f7fd3e00640 1 -- 172.21.15.39:0/670990330 >> v1:172.21.15.39:6789/0 conn(0x7f7fd40b5b10 legacy=0x7f7fd40b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.436+0000 7f7fd3e00640 1 -- 172.21.15.39:0/670990330 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd40d2500 con 0x7f7fd415cc90 2024-01-24T17:41:42.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.436+0000 7f7fd2a00640 1 -- 172.21.15.39:0/670990330 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812680994 0 0) 0x7f7fbc001110 con 0x7f7fd40b5b10 2024-01-24T17:41:42.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.437+0000 7fa535800640 1 --2- 172.21.15.39:0/226609722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015d020 0x7fa53015d460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa5200097a0 tx=0x7fa52002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.437+0000 7f827ca00640 1 -- 172.21.15.39:0/1137195911 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8278070e80 con 0x7f827815cdd0 2024-01-24T17:41:42.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.437+0000 7fa535800640 1 -- 172.21.15.39:0/226609722 shutdown_connections 2024-01-24T17:41:42.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.437+0000 7fa535800640 1 --2- 172.21.15.39:0/226609722 >> v2:172.21.15.100:3300/0 conn(0x7fa53015eb80 0x7fa53015f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.437+0000 7fa535800640 1 --2- 172.21.15.39:0/226609722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015d020 0x7fa53015d460 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa5200097a0 tx=0x7fa52002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.438+0000 7fa535800640 1 -- 172.21.15.39:0/226609722 >> 172.21.15.39:0/226609722 conn(0x7fa530071790 msgr2=0x7fa530071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.438+0000 7fa535800640 1 -- 172.21.15.39:0/226609722 shutdown_connections 2024-01-24T17:41:42.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.438+0000 7fa535800640 1 -- 172.21.15.39:0/226609722 wait complete. 2024-01-24T17:41:42.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.439+0000 7f7fd3400640 1 --2- 172.21.15.39:0/670990330 >> v2:172.21.15.39:3300/0 conn(0x7f7fd40cf9e0 0x7f7fd40cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:42.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.439+0000 7f827ca00640 1 --2- 172.21.15.39:0/1137195911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278158a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82680097d0 tx=0x7f826802b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19334a9687feee81 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.440+0000 7f7fd3e00640 1 --2- 172.21.15.39:0/670990330 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd415cc90 0x7f7fd40d1fc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009d90 tx=0x7f7fcc02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a83df9715891a76d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.440+0000 7f8276a00640 1 -- 172.21.15.39:0/1137195911 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826802f810 con 0x7f827815cdd0 2024-01-24T17:41:42.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.440+0000 7f8276a00640 1 -- 172.21.15.39:0/1137195911 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826802f970 con 0x7f827815cdd0 2024-01-24T17:41:42.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.440+0000 7f8276a00640 1 -- 172.21.15.39:0/1137195911 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826802fd40 con 0x7f827815cdd0 2024-01-24T17:41:42.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.441+0000 7f7fd2a00640 1 -- 172.21.15.39:0/670990330 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc02cc60 con 0x7f7fd415cc90 2024-01-24T17:41:42.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.441+0000 7fa535800640 1 Processor -- start 2024-01-24T17:41:42.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.441+0000 7fa535800640 1 -- start start 2024-01-24T17:41:42.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.441+0000 7f7fd2a00640 1 -- 172.21.15.39:0/670990330 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc02cdc0 con 0x7f7fd415cc90 2024-01-24T17:41:42.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.441+0000 7f7fd2a00640 1 -- 172.21.15.39:0/670990330 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc034660 con 0x7f7fd415cc90 2024-01-24T17:41:42.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f7fdaa00640 1 -- 172.21.15.39:0/670990330 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd415cc90 msgr2=0x7f7fd40d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/670990330 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd415cc90 0x7f7fd40d1fc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009d90 tx=0x7f7fcc02beb0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f7fdaa00640 1 -- 172.21.15.39:0/670990330 shutdown_connections 2024-01-24T17:41:42.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/670990330 >> v2:172.21.15.39:3300/0 conn(0x7f7fd40cf9e0 0x7f7fd40cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/670990330 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd415cc90 0x7f7fd40d1fc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f7fdaa00640 1 -- 172.21.15.39:0/670990330 >> 172.21.15.39:0/670990330 conn(0x7f7fd4071790 msgr2=0x7f7fd4071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f827ea00640 1 -- 172.21.15.39:0/1137195911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 msgr2=0x7f8278158a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f827ea00640 1 --2- 172.21.15.39:0/1137195911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278158a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82680097d0 tx=0x7f826802b9f0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.442+0000 7f7fdaa00640 1 -- 172.21.15.39:0/670990330 shutdown_connections 2024-01-24T17:41:42.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7f827ea00640 1 -- 172.21.15.39:0/1137195911 shutdown_connections 2024-01-24T17:41:42.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7f827ea00640 1 --2- 172.21.15.39:0/1137195911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278158a30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7fa535800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5300cf990 0x7fa5300cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7fa535800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015eb80 0x7fa5300cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7fa535800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530157c30 0x7fa5300b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7fa535800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300cb810 con 0x7fa530157c30 2024-01-24T17:41:42.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7fa535800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5301f5510 con 0x7fa5300cf990 2024-01-24T17:41:42.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7fa535800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5301f6290 con 0x7fa53015eb80 2024-01-24T17:41:42.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7f7fdaa00640 1 -- 172.21.15.39:0/670990330 wait complete. 2024-01-24T17:41:42.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7f827ea00640 1 -- 172.21.15.39:0/1137195911 >> 172.21.15.39:0/1137195911 conn(0x7f82780cafc0 msgr2=0x7f82780cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7f827ea00640 1 -- 172.21.15.39:0/1137195911 shutdown_connections 2024-01-24T17:41:42.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.443+0000 7f827ea00640 1 -- 172.21.15.39:0/1137195911 wait complete. 2024-01-24T17:41:42.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.444+0000 7fa52ea00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5300cf990 0x7fa5300cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.444+0000 7fa52ea00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5300cf990 0x7fa5300cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49510/0 (socket says 172.21.15.39:49510) 2024-01-24T17:41:42.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.444+0000 7fa52ea00640 1 -- 172.21.15.39:0/1058980600 learned_addr learned my addr 172.21.15.39:0/1058980600 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.444+0000 7fa52f400640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530157c30 0x7fa5300b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.445+0000 7fa52e000640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015eb80 0x7fa5300cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.446+0000 7f7fdaa00640 1 Processor -- start 2024-01-24T17:41:42.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.446+0000 7f7fdaa00640 1 -- start start 2024-01-24T17:41:42.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.446+0000 7fa52f400640 1 -- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015eb80 msgr2=0x7fa5300cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.446+0000 7f827ea00640 1 Processor -- start 2024-01-24T17:41:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.446+0000 7fa52f400640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015eb80 0x7fa5300cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.446+0000 7fa52f400640 1 -- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5300cf990 msgr2=0x7fa5300cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.446+0000 7fa52f400640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5300cf990 0x7fa5300cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.446+0000 7fa52f400640 1 -- 172.21.15.39:0/1058980600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5180094d0 con 0x7fa530157c30 2024-01-24T17:41:42.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.447+0000 7f827ea00640 1 -- start start 2024-01-24T17:41:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.447+0000 7fa52f400640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530157c30 0x7fa5300b6fd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa51800e870 tx=0x7fa51800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.448+0000 7fa526a00640 1 -- 172.21.15.39:0/1058980600 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518004020 con 0x7fa530157c30 2024-01-24T17:41:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.448+0000 7f827ea00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82780b06d0 0x7f82781fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.448+0000 7f7fdaa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd40b5b10 0x7f7fd415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.448+0000 7f827ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82780cbab0 0x7f82780c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.448+0000 7f7fdaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd40cf9e0 0x7f7fd415f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7fa526a00640 1 -- 172.21.15.39:0/1058980600 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5180041a0 con 0x7fa530157c30 2024-01-24T17:41:42.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7fa526a00640 1 -- 172.21.15.39:0/1058980600 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51800c940 con 0x7fa530157c30 2024-01-24T17:41:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f827ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f8277400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82780cbab0 0x7f82780c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f7fd3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd40b5b10 0x7f7fd415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f7fd3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd40b5b10 0x7f7fd415ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48268/0 (socket says 172.21.15.39:48268) 2024-01-24T17:41:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f827ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82781fedf0 con 0x7f82780cbab0 2024-01-24T17:41:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f827ea00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82780c5790 con 0x7f82780b06d0 2024-01-24T17:41:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f8277400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82780cbab0 0x7f82780c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58526/0 (socket says 172.21.15.39:58526) 2024-01-24T17:41:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f8277400640 1 -- 172.21.15.39:0/2569243853 learned_addr learned my addr 172.21.15.39:0/2569243853 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82780c5a70 con 0x7f827815cdd0 2024-01-24T17:41:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f7fdaa00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fd415cc90 0x7f7fd40c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f7fdaa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40c7b70 con 0x7f7fd40cf9e0 2024-01-24T17:41:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f7fdaa00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40c7e50 con 0x7f7fd415cc90 2024-01-24T17:41:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f7fdaa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40c8130 con 0x7f7fd40b5b10 2024-01-24T17:41:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.449+0000 7f7fd3e00640 1 -- 172.21.15.39:0/4103684832 learned_addr learned my addr 172.21.15.39:0/4103684832 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.450+0000 7f7fd3400640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd40cf9e0 0x7f7fd415f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.450+0000 7f7fd8a00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fd415cc90 0x7f7fd40c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.450+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa520009400 con 0x7fa530157c30 2024-01-24T17:41:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.450+0000 7f7fd3e00640 1 -- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fd415cc90 msgr2=0x7f7fd40c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.450+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5301f5a90 con 0x7fa530157c30 2024-01-24T17:41:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f7fd3e00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fd415cc90 0x7f7fd40c6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.450+0000 7f827ca00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f7fd3e00640 1 -- 172.21.15.39:0/4103684832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd40cf9e0 msgr2=0x7f7fd415f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f7fd3e00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd40cf9e0 0x7f7fd415f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f7fd3e00640 1 -- 172.21.15.39:0/4103684832 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc009a70 con 0x7f7fd40b5b10 2024-01-24T17:41:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.450+0000 7f8277e00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82780b06d0 0x7f82781fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f827ca00640 1 -- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82780b06d0 msgr2=0x7f82781fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f7fd3400640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd40cf9e0 0x7f7fd415f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f827ca00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82780b06d0 0x7f82781fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f827ca00640 1 -- 172.21.15.39:0/2569243853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82780cbab0 msgr2=0x7f82780c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f827ca00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82780cbab0 0x7f82780c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.451+0000 7f827ca00640 1 -- 172.21.15.39:0/2569243853 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8268009400 con 0x7f827815cdd0 2024-01-24T17:41:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.452+0000 7f8277e00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82780b06d0 0x7f82781fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.452+0000 7f8277400640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82780cbab0 0x7f82780c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.453+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa530073ef0 con 0x7fa530157c30 2024-01-24T17:41:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.452+0000 7f7fd3e00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd40b5b10 0x7f7fd415ab30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009d90 tx=0x7f7fcc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.453+0000 7f7fcbe00640 1 -- 172.21.15.39:0/4103684832 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc040650 con 0x7f7fd40b5b10 2024-01-24T17:41:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.453+0000 7f7fcbe00640 1 -- 172.21.15.39:0/4103684832 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc0407b0 con 0x7f7fd40b5b10 2024-01-24T17:41:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.453+0000 7f7fcbe00640 1 -- 172.21.15.39:0/4103684832 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc040b60 con 0x7f7fd40b5b10 2024-01-24T17:41:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.453+0000 7f827ca00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278157b00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8268033000 tx=0x7f82680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.453+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd40c8410 con 0x7f7fd40b5b10 2024-01-24T17:41:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.454+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd40c89d0 con 0x7f7fd40b5b10 2024-01-24T17:41:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7f826fe00640 1 -- 172.21.15.39:0/2569243853 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826803b040 con 0x7f827815cdd0 2024-01-24T17:41:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7f826fe00640 1 -- 172.21.15.39:0/2569243853 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8268037830 con 0x7f827815cdd0 2024-01-24T17:41:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7f826fe00640 1 -- 172.21.15.39:0/2569243853 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8268037be0 con 0x7f827815cdd0 2024-01-24T17:41:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82780c5d50 con 0x7f827815cdd0 2024-01-24T17:41:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7f003b200640 1 Processor -- start 2024-01-24T17:41:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd40b4320 con 0x7f7fd40b5b10 2024-01-24T17:41:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82780c6310 con 0x7f827815cdd0 2024-01-24T17:41:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7f003b200640 1 -- start start 2024-01-24T17:41:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.457+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8248005250 con 0x7f827815cdd0 2024-01-24T17:41:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.455+0000 7fa526a00640 1 -- 172.21.15.39:0/1058980600 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa51800caa0 con 0x7fa530157c30 2024-01-24T17:41:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.456+0000 7fa526a00640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa50806a460 0x7fa50806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.457+0000 7fa526a00640 1 -- 172.21.15.39:0/1058980600 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa518057ad0 con 0x7fa530157c30 2024-01-24T17:41:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.458+0000 7fa52ea00640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa50806a460 0x7fa50806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.459+0000 7fa52ea00640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa50806a460 0x7fa50806c920 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5280012e0 tx=0x7fa528009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.459+0000 7f826fe00640 1 -- 172.21.15.39:0/2569243853 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8268037d40 con 0x7f827815cdd0 2024-01-24T17:41:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.461+0000 7f826fe00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825006a500 0x7f825006c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.462+0000 7f826fe00640 1 -- 172.21.15.39:0/2569243853 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82680abfa0 con 0x7f827815cdd0 2024-01-24T17:41:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.462+0000 7f8277e00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825006a500 0x7f825006c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.457+0000 7f003b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0034035130 0x7f00340e45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.457+0000 7f003b200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0034006b40 0x7f0034006f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.457+0000 7f003b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0034050960 con 0x7f00340e3b00 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.457+0000 7f003b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0034007cf0 con 0x7f0034035130 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.457+0000 7f003b200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340e03f0 con 0x7f0034006b40 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.458+0000 7f0039e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f00340e3b00 0x7f00340e3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52500/0 (socket says 172.21.15.39:52500) 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.459+0000 7f0039e00640 1 -- 172.21.15.39:0/722079747 learned_addr learned my addr 172.21.15.39:0/722079747 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.459+0000 7f0039400640 1 --2- 172.21.15.39:0/722079747 >> v2:172.21.15.39:3300/0 conn(0x7f0034035130 0x7f00340e45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.459+0000 7f003a800640 1 --2- 172.21.15.39:0/722079747 >> v2:172.21.15.100:3300/0 conn(0x7f0034006b40 0x7f0034006f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.460+0000 7f0039400640 1 -- 172.21.15.39:0/722079747 >> v2:172.21.15.100:3300/0 conn(0x7f0034006b40 msgr2=0x7f0034006f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.460+0000 7f0039400640 1 --2- 172.21.15.39:0/722079747 >> v2:172.21.15.100:3300/0 conn(0x7f0034006b40 0x7f0034006f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.460+0000 7f0039400640 1 -- 172.21.15.39:0/722079747 >> v1:172.21.15.39:6789/0 conn(0x7f00340e3b00 legacy=0x7f00340e3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.460+0000 7f0039400640 1 -- 172.21.15.39:0/722079747 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0034050eb0 con 0x7f0034035130 2024-01-24T17:41:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.462+0000 7f0039400640 1 --2- 172.21.15.39:0/722079747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034035130 0x7f00340e45f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f003400c6f0 tx=0x7f002c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448dc3ac5c1ce0a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.460+0000 7f7fcbe00640 1 -- 172.21.15.39:0/4103684832 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7fcc040cc0 con 0x7f7fd40b5b10 2024-01-24T17:41:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.461+0000 7f7fcbe00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fac06a430 0x7f7fac06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.462+0000 7f7fcbe00640 1 -- 172.21.15.39:0/4103684832 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fcc0ab8e0 con 0x7f7fd40b5b10 2024-01-24T17:41:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.462+0000 7f7fd3400640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fac06a430 0x7f7fac06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.463+0000 7f7fd3400640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fac06a430 0x7f7fac06c8f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40d1360 tx=0x7f7fc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.464+0000 7f8277e00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825006a500 0x7f825006c9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8260007fd0 tx=0x7f8260009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.466+0000 7f0038a00640 1 -- 172.21.15.39:0/722079747 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c0317e0 con 0x7f0034035130 2024-01-24T17:41:42.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.467+0000 7fa526a00640 1 -- 172.21.15.39:0/1058980600 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa5180545a0 con 0x7fa530157c30 2024-01-24T17:41:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.467+0000 7f0038a00640 1 -- 172.21.15.39:0/722079747 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c031940 con 0x7f0034035130 2024-01-24T17:41:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.468+0000 7f0038a00640 1 -- 172.21.15.39:0/722079747 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c0317e0 con 0x7f0034035130 2024-01-24T17:41:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.468+0000 7f003b200640 1 -- 172.21.15.39:0/722079747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034035130 msgr2=0x7f00340e45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.468+0000 7f003b200640 1 --2- 172.21.15.39:0/722079747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034035130 0x7f00340e45f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f003400c6f0 tx=0x7f002c02b880 comp rx=0 tx=0).stop 2024-01-24T17:41:42.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.468+0000 7f003b200640 1 -- 172.21.15.39:0/722079747 shutdown_connections 2024-01-24T17:41:42.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.468+0000 7f003b200640 1 --2- 172.21.15.39:0/722079747 >> v2:172.21.15.100:3300/0 conn(0x7f0034006b40 0x7f0034006f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.468+0000 7f003b200640 1 --2- 172.21.15.39:0/722079747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034035130 0x7f00340e45f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.468+0000 7f003b200640 1 -- 172.21.15.39:0/722079747 >> 172.21.15.39:0/722079747 conn(0x7f0034055d20 msgr2=0x7f0034056130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.468+0000 7f003b200640 1 -- 172.21.15.39:0/722079747 shutdown_connections 2024-01-24T17:41:42.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.469+0000 7f003b200640 1 -- 172.21.15.39:0/722079747 wait complete. 2024-01-24T17:41:42.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.471+0000 7f003b200640 1 Processor -- start 2024-01-24T17:41:42.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.471+0000 7f003b200640 1 -- start start 2024-01-24T17:41:42.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.473+0000 7f003b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034006b40 0x7f00340daf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.473+0000 7f826fe00640 1 -- 172.21.15.39:0/2569243853 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8268077690 con 0x7f827815cdd0 2024-01-24T17:41:42.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.474+0000 7f003b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0034035130 0x7f00340d7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.474+0000 7f0039e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034006b40 0x7f00340daf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.474+0000 7f0039e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034006b40 0x7f00340daf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58552/0 (socket says 172.21.15.39:58552) 2024-01-24T17:41:42.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.474+0000 7f0039e00640 1 -- 172.21.15.39:0/3310833023 learned_addr learned my addr 172.21.15.39:0/3310833023 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.474+0000 7f003b200640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00340e3b00 0x7f00340d7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.474+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340dc460 con 0x7f0034006b40 2024-01-24T17:41:42.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.474+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340dc880 con 0x7f0034035130 2024-01-24T17:41:42.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.475+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00340db7e0 con 0x7f00340e3b00 2024-01-24T17:41:42.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.475+0000 7f003a800640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00340e3b00 0x7f00340d7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.475+0000 7f0039e00640 1 -- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00340e3b00 msgr2=0x7f00340d7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.475+0000 7f0039e00640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00340e3b00 0x7f00340d7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.475+0000 7f0039e00640 1 -- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0034035130 msgr2=0x7f00340d7a10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-24T17:41:42.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.475+0000 7f0039e00640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0034035130 0x7f00340d7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.475+0000 7f0039e00640 1 -- 172.21.15.39:0/3310833023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c009400 con 0x7f0034006b40 2024-01-24T17:41:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.476+0000 7f0039e00640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034006b40 0x7f00340daf90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f002400c500 tx=0x7f002400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.477+0000 7f0032000640 1 -- 172.21.15.39:0/3310833023 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024007560 con 0x7f0034006b40 2024-01-24T17:41:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.477+0000 7f0032000640 1 -- 172.21.15.39:0/3310833023 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00240047e0 con 0x7f0034006b40 2024-01-24T17:41:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.477+0000 7f0032000640 1 -- 172.21.15.39:0/3310833023 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024004b50 con 0x7f0034006b40 2024-01-24T17:41:42.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.477+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00340dbac0 con 0x7f0034006b40 2024-01-24T17:41:42.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.477+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00340dc080 con 0x7f0034006b40 2024-01-24T17:41:42.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.478+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00340dc5a0 con 0x7f0034006b40 2024-01-24T17:41:42.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.481+0000 7f0032000640 1 -- 172.21.15.39:0/3310833023 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0024004cb0 con 0x7f0034006b40 2024-01-24T17:41:42.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.483+0000 7f0032000640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f001406a480 0x7f001406c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.483+0000 7f0032000640 1 -- 172.21.15.39:0/3310833023 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00240883f0 con 0x7f0034006b40 2024-01-24T17:41:42.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.484+0000 7f0039400640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f001406a480 0x7f001406c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.485+0000 7f0039400640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f001406a480 0x7f001406c940 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f002c036000 tx=0x7f002c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.487+0000 7f7fcbe00640 1 -- 172.21.15.39:0/4103684832 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7fcc076e30 con 0x7f7fd40b5b10 2024-01-24T17:41:42.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.490+0000 7f0032000640 1 -- 172.21.15.39:0/3310833023 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0024053a50 con 0x7f0034006b40 2024-01-24T17:41:42.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.499+0000 7efd40c00640 1 Processor -- start 2024-01-24T17:41:42.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.499+0000 7efd40c00640 1 -- start start 2024-01-24T17:41:42.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.501+0000 7efd40c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd340581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.502+0000 7efd40c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd34057700 con 0x7efd3403e3b0 2024-01-24T17:41:42.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.502+0000 7efd40c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd34054460 con 0x7efd3403e9a0 2024-01-24T17:41:42.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.502+0000 7efd40c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd34053a90 con 0x7efd340079d0 2024-01-24T17:41:42.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.502+0000 7efd3aa00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7efd3403e9a0 0x7efd340072c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48762/0 (socket says 172.21.15.39:48762) 2024-01-24T17:41:42.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.502+0000 7efd3aa00640 1 -- 172.21.15.39:0/3014476600 learned_addr learned my addr 172.21.15.39:0/3014476600 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:42.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.502+0000 7efd3be00640 1 --2- 172.21.15.39:0/3014476600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd340581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.503+0000 7efd3a000640 1 -- 172.21.15.39:0/3014476600 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619945825 0 0) 0x7efd34054460 con 0x7efd3403e9a0 2024-01-24T17:41:42.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.503+0000 7efd3a000640 1 -- 172.21.15.39:0/3014476600 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1c003520 con 0x7efd3403e9a0 2024-01-24T17:41:42.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.503+0000 7efd3be00640 1 -- 172.21.15.39:0/3014476600 >> v1:172.21.15.100:6789/0 conn(0x7efd3403e9a0 legacy=0x7efd340072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.503+0000 7efd3be00640 1 -- 172.21.15.39:0/3014476600 >> v1:172.21.15.39:6789/0 conn(0x7efd3403e3b0 legacy=0x7efd3403c2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-24T17:41:42.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.504+0000 7efd3be00640 1 -- 172.21.15.39:0/3014476600 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd34054740 con 0x7efd340079d0 2024-01-24T17:41:42.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.506+0000 7efd3be00640 1 --2- 172.21.15.39:0/3014476600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd340581a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efd240097d0 tx=0x7efd2402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4369b47113e54769 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.507+0000 7efd3a000640 1 -- 172.21.15.39:0/3014476600 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd24031770 con 0x7efd340079d0 2024-01-24T17:41:42.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.507+0000 7efd3a000640 1 -- 172.21.15.39:0/3014476600 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd240318d0 con 0x7efd340079d0 2024-01-24T17:41:42.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.507+0000 7efd3a000640 1 -- 172.21.15.39:0/3014476600 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd24031c60 con 0x7efd340079d0 2024-01-24T17:41:42.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.508+0000 7efd40c00640 1 -- 172.21.15.39:0/3014476600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 msgr2=0x7efd340581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.508+0000 7efd40c00640 1 --2- 172.21.15.39:0/3014476600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd340581a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efd240097d0 tx=0x7efd2402b810 comp rx=0 tx=0).stop 2024-01-24T17:41:42.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.508+0000 7f1d9ba00640 1 Processor -- start 2024-01-24T17:41:42.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.508+0000 7efd40c00640 1 -- 172.21.15.39:0/3014476600 shutdown_connections 2024-01-24T17:41:42.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.508+0000 7efd40c00640 1 --2- 172.21.15.39:0/3014476600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd340581a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.508+0000 7f1d9ba00640 1 -- start start 2024-01-24T17:41:42.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.508+0000 7efd40c00640 1 -- 172.21.15.39:0/3014476600 >> 172.21.15.39:0/3014476600 conn(0x7efd34059120 msgr2=0x7efd34059530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.509+0000 7efd40c00640 1 -- 172.21.15.39:0/3014476600 shutdown_connections 2024-01-24T17:41:42.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.510+0000 7efd40c00640 1 -- 172.21.15.39:0/3014476600 wait complete. 2024-01-24T17:41:42.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.510+0000 7f1d9ba00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1d940d0a30 0x7f1d9406ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.510+0000 7f1d9ba00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d940d0e10 0x7f1d940b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.510+0000 7f1d9ba00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9406b640 con 0x7f1d9415ddd0 2024-01-24T17:41:42.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.510+0000 7f1d9ba00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d940cf210 con 0x7f1d940d0a30 2024-01-24T17:41:42.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.510+0000 7f1d9ba00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9406b920 con 0x7f1d940d0e10 2024-01-24T17:41:42.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.511+0000 7f1d93e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1d9415ddd0 0x7f1d940ceb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52524/0 (socket says 172.21.15.39:52524) 2024-01-24T17:41:42.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.511+0000 7f1d93e00640 1 -- 172.21.15.39:0/2892802160 learned_addr learned my addr 172.21.15.39:0/2892802160 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:42.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.513+0000 7f86ff200640 1 Processor -- start 2024-01-24T17:41:42.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.513+0000 7f86ff200640 1 -- start start 2024-01-24T17:41:42.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222407210 0 0) 0x7f1d9406b640 con 0x7f1d9415ddd0 2024-01-24T17:41:42.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d74003520 con 0x7f1d9415ddd0 2024-01-24T17:41:42.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572402867 0 0) 0x7f1d74003520 con 0x7f1d9415ddd0 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d940d0e10 msgr2=0x7f1d940b0dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d940d0e10 0x7f1d940b0dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 >> v2:172.21.15.39:3300/0 conn(0x7f1d940d0a30 msgr2=0x7f1d9406ad80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 --2- 172.21.15.39:0/2892802160 >> v2:172.21.15.39:3300/0 conn(0x7f1d940d0a30 0x7f1d9406ad80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9406bc00 con 0x7f1d9415ddd0 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1d84001670 con 0x7f1d9415ddd0 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 >> v1:172.21.15.39:6789/0 conn(0x7f1d9415ddd0 legacy=0x7f1d940ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d74003970 0x7f1d74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740047c0 0x7f1d74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740052a0 0x7f1d74014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940cf210 con 0x7f1d740052a0 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9406b920 con 0x7f1d74003970 2024-01-24T17:41:42.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74005a00 con 0x7f1d740047c0 2024-01-24T17:41:42.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.513+0000 7f1d93e00640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740047c0 0x7f1d74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.514+0000 7f1d93e00640 1 -- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d74003970 msgr2=0x7f1d74003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-24T17:41:42.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.514+0000 7f1d93e00640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d74003970 0x7f1d74003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.514+0000 7f1d93e00640 1 -- 172.21.15.39:0/2892802160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740052a0 msgr2=0x7f1d74014190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-24T17:41:42.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.514+0000 7f1d93e00640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740052a0 0x7f1d74014190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.514+0000 7f1d93e00640 1 -- 172.21.15.39:0/2892802160 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9406bc00 con 0x7f1d740047c0 2024-01-24T17:41:42.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.515+0000 7f86ff200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f86f00e75d0 0x7f86f0038790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.515+0000 7f86ff200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f86f00e79b0 0x7f86f0038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.515+0000 7f86ff200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f0007b40 con 0x7f86f004a320 2024-01-24T17:41:42.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.515+0000 7f86ff200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f0049dd0 con 0x7f86f00e75d0 2024-01-24T17:41:42.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.515+0000 7f86ff200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f00542b0 con 0x7f86f00e79b0 2024-01-24T17:41:42.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.516+0000 7f86f7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f86f00e75d0 0x7f86f0038790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.516+0000 7f86f7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f86f00e75d0 0x7f86f0038790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58562/0 (socket says 172.21.15.39:58562) 2024-01-24T17:41:42.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.516+0000 7f86f7e00640 1 -- 172.21.15.39:0/3347066217 learned_addr learned my addr 172.21.15.39:0/3347066217 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.516+0000 7f86f7400640 1 --2- 172.21.15.39:0/3347066217 >> v2:172.21.15.100:3300/0 conn(0x7f86f00e79b0 0x7f86f0038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.516+0000 7f86f7400640 1 -- 172.21.15.39:0/3347066217 >> v2:172.21.15.39:3300/0 conn(0x7f86f00e75d0 msgr2=0x7f86f0038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:42.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.516+0000 7f86f7400640 1 --2- 172.21.15.39:0/3347066217 >> v2:172.21.15.39:3300/0 conn(0x7f86f00e75d0 0x7f86f0038790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.516+0000 7f86f7400640 1 -- 172.21.15.39:0/3347066217 >> v1:172.21.15.100:6789/0 conn(0x7f86f004a320 legacy=0x7f86f00eb330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-24T17:41:42.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.516+0000 7f86f7400640 1 -- 172.21.15.39:0/3347066217 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0049fe0 con 0x7f86f00e79b0 2024-01-24T17:41:42.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.512+0000 7efd40c00640 1 Processor -- start 2024-01-24T17:41:42.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.517+0000 7efd40c00640 1 -- start start 2024-01-24T17:41:42.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.518+0000 7f86f7400640 1 --2- 172.21.15.39:0/3347066217 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f00e79b0 0x7f86f0038cd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f86f0006ae0 tx=0x7f86e002b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0ffada3ffc6e6bf server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.519+0000 7f86f6a00640 1 -- 172.21.15.39:0/3347066217 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e0031520 con 0x7f86f00e79b0 2024-01-24T17:41:42.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.519+0000 7f86f6a00640 1 -- 172.21.15.39:0/3347066217 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e0031680 con 0x7f86f00e79b0 2024-01-24T17:41:42.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.519+0000 7f86f6a00640 1 -- 172.21.15.39:0/3347066217 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e0031a10 con 0x7f86f00e79b0 2024-01-24T17:41:42.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.517+0000 7f1d93e00640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740047c0 0x7f1d74004c40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1d8400c240 tx=0x7f1d740057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a94ec7d2458e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.518+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8402cb70 con 0x7f1d740047c0 2024-01-24T17:41:42.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.518+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8402ccd0 con 0x7f1d740047c0 2024-01-24T17:41:42.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.518+0000 7f1d93400640 1 -- 172.21.15.39:0/2892802160 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84005af0 con 0x7f1d740047c0 2024-01-24T17:41:42.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.519+0000 7f1d9ba00640 1 -- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740047c0 msgr2=0x7f1d74004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.519+0000 7f1d9ba00640 1 --2- 172.21.15.39:0/2892802160 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740047c0 0x7f1d74004c40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1d8400c240 tx=0x7f1d740057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.519+0000 7f1d99a00640 1 -- 172.21.15.39:0/2892802160 reap_dead start 2024-01-24T17:41:42.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.519+0000 7f1d9ba00640 1 -- 172.21.15.39:0/2892802160 shutdown_connections 2024-01-24T17:41:42.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.519+0000 7f1d9ba00640 1 -- 172.21.15.39:0/2892802160 >> 172.21.15.39:0/2892802160 conn(0x7f1d94089fd0 msgr2=0x7f1d94099b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7efd40c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd3400d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7f1d9ba00640 1 -- 172.21.15.39:0/2892802160 shutdown_connections 2024-01-24T17:41:42.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7f1d9ba00640 1 -- 172.21.15.39:0/2892802160 wait complete. 2024-01-24T17:41:42.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7f86ff200640 1 -- 172.21.15.39:0/3347066217 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f00e79b0 msgr2=0x7f86f0038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7f86ff200640 1 --2- 172.21.15.39:0/3347066217 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f00e79b0 0x7f86f0038cd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f86f0006ae0 tx=0x7f86e002b7c0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7efd40c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd3403e3b0 0x7efd3404f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7efd40c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd3403e9a0 0x7efd34185080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7f86ff200640 1 -- 172.21.15.39:0/3347066217 shutdown_connections 2024-01-24T17:41:42.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7f86ff200640 1 --2- 172.21.15.39:0/3347066217 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f00e79b0 0x7f86f0038cd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.520+0000 7f86ff200640 1 --2- 172.21.15.39:0/3347066217 >> v2:172.21.15.39:3300/0 conn(0x7f86f00e75d0 0x7f86f0038790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7f86ff200640 1 -- 172.21.15.39:0/3347066217 >> 172.21.15.39:0/3347066217 conn(0x7f86f0053790 msgr2=0x7f86f0053ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7f86ff200640 1 -- 172.21.15.39:0/3347066217 shutdown_connections 2024-01-24T17:41:42.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7f86ff200640 1 -- 172.21.15.39:0/3347066217 wait complete. 2024-01-24T17:41:42.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7efd40c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3404f220 con 0x7efd3403e3b0 2024-01-24T17:41:42.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7efd3b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd3400d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7efd3aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd3403e3b0 0x7efd3404f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7efd3b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd3400d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:48294/0 (socket says 172.21.15.39:48294) 2024-01-24T17:41:42.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7efd3b400640 1 -- 172.21.15.39:0/29879640 learned_addr learned my addr 172.21.15.39:0/29879640 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.521+0000 7efd3be00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd3403e9a0 0x7efd34185080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.522+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3404e3b0 con 0x7efd3403e9a0 2024-01-24T17:41:42.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.522+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3400dc20 con 0x7efd340079d0 2024-01-24T17:41:42.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.522+0000 7f1d9ba00640 1 Processor -- start 2024-01-24T17:41:42.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.523+0000 7efd3aa00640 1 -- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 msgr2=0x7efd3400d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.523+0000 7efd3aa00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd3400d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.523+0000 7efd3aa00640 1 -- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd3403e9a0 msgr2=0x7efd34185080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.523+0000 7efd3aa00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd3403e9a0 0x7efd34185080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.523+0000 7efd3aa00640 1 -- 172.21.15.39:0/29879640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd24009400 con 0x7efd3403e3b0 2024-01-24T17:41:42.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.523+0000 7f86ff200640 1 Processor -- start 2024-01-24T17:41:42.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.523+0000 7efd3aa00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd3403e3b0 0x7efd3404f640 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00c450 tx=0x7efd2c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.524+0000 7f1d9ba00640 1 -- start start 2024-01-24T17:41:42.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.524+0000 7efd33400640 1 -- 172.21.15.39:0/29879640 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0074e0 con 0x7efd3403e3b0 2024-01-24T17:41:42.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.524+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3404e5a0 con 0x7efd3403e3b0 2024-01-24T17:41:42.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.524+0000 7f86ff200640 1 -- start start 2024-01-24T17:41:42.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.525+0000 7f86ff200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f004a320 0x7f86f00db2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.525+0000 7f86ff200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f00e75d0 0x7f86f004c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.525+0000 7f86ff200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f00e79b0 0x7f86f004ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.525+0000 7f86ff200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00db830 con 0x7f86f00e79b0 2024-01-24T17:41:42.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.525+0000 7f86ff200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f004baf0 con 0x7f86f004a320 2024-01-24T17:41:42.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.525+0000 7f86ff200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f004b220 con 0x7f86f00e75d0 2024-01-24T17:41:42.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.524+0000 7efd33400640 1 -- 172.21.15.39:0/29879640 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c007640 con 0x7efd3403e3b0 2024-01-24T17:41:42.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.524+0000 7efd33400640 1 -- 172.21.15.39:0/29879640 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0049e0 con 0x7efd3403e3b0 2024-01-24T17:41:42.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.524+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3404eb60 con 0x7efd3403e3b0 2024-01-24T17:41:42.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.526+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3404f360 con 0x7efd3403e3b0 2024-01-24T17:41:42.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.526+0000 7f86f7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f004a320 0x7f86f00db2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.526+0000 7f86fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f00e79b0 0x7f86f004ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.526+0000 7f86fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f00e79b0 0x7f86f004ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58572/0 (socket says 172.21.15.39:58572) 2024-01-24T17:41:42.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.526+0000 7f86fd200640 1 -- 172.21.15.39:0/32553162 learned_addr learned my addr 172.21.15.39:0/32553162 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f1d9ba00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d94154e10 0x7f1d941590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f86fd200640 1 -- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f00e75d0 msgr2=0x7f86f004c920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-24T17:41:42.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f86fd200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f00e75d0 0x7f86f004c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f86fd200640 1 -- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f004a320 msgr2=0x7f86f00db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f86fd200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f004a320 0x7f86f00db2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f1d9ba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d9415b000 0x7f1d940b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f86fd200640 1 -- 172.21.15.39:0/32553162 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e0009400 con 0x7f86f00e79b0 2024-01-24T17:41:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f1d9ba00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d940b2630 0x7f1d94156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f86f7e00640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f004a320 0x7f86f00db2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:42.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f1d9ba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d940b2ae0 con 0x7f1d9415b000 2024-01-24T17:41:42.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f1d9ba00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9415a530 con 0x7f1d94154e10 2024-01-24T17:41:42.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.527+0000 7f1d9ba00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d94159920 con 0x7f1d940b2630 2024-01-24T17:41:42.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.528+0000 7f86fd200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f00e79b0 0x7f86f004ce60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f86f8068110 tx=0x7f86f8066fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.528+0000 7f1d99000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d94154e10 0x7f1d941590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.528+0000 7f1d99a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d940b2630 0x7f1d94156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.528+0000 7f1d99000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d94154e10 0x7f1d941590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49576/0 (socket says 172.21.15.39:49576) 2024-01-24T17:41:42.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.528+0000 7f1d99000640 1 -- 172.21.15.39:0/1330701402 learned_addr learned my addr 172.21.15.39:0/1330701402 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:42.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.528+0000 7f86efe00640 1 -- 172.21.15.39:0/32553162 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8073700 con 0x7f86f00e79b0 2024-01-24T17:41:42.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f86efe00640 1 -- 172.21.15.39:0/32553162 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f80738a0 con 0x7f86f00e79b0 2024-01-24T17:41:42.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f86efe00640 1 -- 172.21.15.39:0/32553162 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f80926b0 con 0x7f86f00e79b0 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f0050ab0 con 0x7f86f00e79b0 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f004b780 con 0x7f86f00e79b0 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.532+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f00395f0 con 0x7f86f00e79b0 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f1d99000640 1 -- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d940b2630 msgr2=0x7f1d94156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f1d99000640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d940b2630 0x7f1d94156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f1d99000640 1 -- 172.21.15.39:0/1330701402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d9415b000 msgr2=0x7f1d940b20f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f1d99000640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d9415b000 0x7f1d940b20f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f1d99000640 1 -- 172.21.15.39:0/1330701402 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d94159c00 con 0x7f1d94154e10 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.529+0000 7f1d99000640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d94154e10 0x7f1d941590d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00b420 tx=0x7f1d7c00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.530+0000 7f1d90c00640 1 -- 172.21.15.39:0/1330701402 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c011830 con 0x7f1d94154e10 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.530+0000 7f1d90c00640 1 -- 172.21.15.39:0/1330701402 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c011990 con 0x7f1d94154e10 2024-01-24T17:41:42.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.530+0000 7f1d90c00640 1 -- 172.21.15.39:0/1330701402 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c011d40 con 0x7f1d94154e10 2024-01-24T17:41:42.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.532+0000 7efd33400640 1 -- 172.21.15.39:0/29879640 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efd2c004b40 con 0x7efd3403e3b0 2024-01-24T17:41:42.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.534+0000 7efd33400640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd1406a430 0x7efd1406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.533+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d94159ee0 con 0x7f1d94154e10 2024-01-24T17:41:42.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.533+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d94159610 con 0x7f1d94154e10 2024-01-24T17:41:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.535+0000 7f86efe00640 1 -- 172.21.15.39:0/32553162 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f86f808f070 con 0x7f86f00e79b0 2024-01-24T17:41:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.537+0000 7efd33400640 1 -- 172.21.15.39:0/29879640 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd2c056f00 con 0x7efd3403e3b0 2024-01-24T17:41:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.537+0000 7f86efe00640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86d006a530 0x7f86d006c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.537+0000 7f86efe00640 1 -- 172.21.15.39:0/32553162 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86f8101e40 con 0x7f86f00e79b0 2024-01-24T17:41:42.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.538+0000 7f86f7e00640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86d006a530 0x7f86d006c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.538+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d940b0290 con 0x7f1d94154e10 2024-01-24T17:41:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.538+0000 7efd3b400640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd1406a430 0x7efd1406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.539+0000 7f86f7e00640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86d006a530 0x7f86d006c9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009c20 tx=0x7f86e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.540+0000 7efd3b400640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd1406a430 0x7efd1406c8f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efd3c074280 tx=0x7efd3c072cb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.540+0000 7f1d90c00640 1 -- 172.21.15.39:0/1330701402 <== mon.1 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1d7c01a410 con 0x7f1d94154e10 2024-01-24T17:41:42.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.542+0000 7f1d90c00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d6c06a460 0x7f1d6c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:42.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.542+0000 7f1d90c00640 1 -- 172.21.15.39:0/1330701402 <== mon.1 v2:172.21.15.100:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d7c088100 con 0x7f1d94154e10 2024-01-24T17:41:42.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.542+0000 7f1d93e00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d6c06a460 0x7f1d6c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:42.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.544+0000 7efd33400640 1 -- 172.21.15.39:0/29879640 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efd3404f360 con 0x7efd3403e3b0 2024-01-24T17:41:42.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.544+0000 7f1d93e00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d6c06a460 0x7f1d6c06c920 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1d84034000 tx=0x7f1d84009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:42.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.553+0000 7f86efe00640 1 -- 172.21.15.39:0/32553162 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f86f80d0e10 con 0x7f86f00e79b0 2024-01-24T17:41:42.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.555+0000 7f1d90c00640 1 -- 172.21.15.39:0/1330701402 <== mon.1 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1d7c0570e0 con 0x7f1d94154e10 2024-01-24T17:41:42.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.817+0000 7f6fd9600640 1 -- 172.21.15.39:0/3218346727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6fac0057d0 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.818+0000 7f6fdbe00640 1 -- 172.21.15.39:0/3218346727 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6fcc08d090 con 0x7f6fdc0e78b0 2024-01-24T17:41:42.819 INFO:teuthology.orchestra.run.smithi039.stdout:55834574854 2024-01-24T17:41:42.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.822+0000 7f6fd9600640 1 -- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a4b0 msgr2=0x7f6fbc06c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.822+0000 7f6fd9600640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a4b0 0x7f6fbc06c970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00da90 tx=0x7f6fe406e030 comp rx=0 tx=0).stop 2024-01-24T17:41:42.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.822+0000 7f6fd9600640 1 -- 172.21.15.39:0/3218346727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 msgr2=0x7f6fdc0e8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.822+0000 7f6fd9600640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0e8730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc00c7f0 tx=0x7f6fcc00ccb0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.823+0000 7f6fd9600640 1 -- 172.21.15.39:0/3218346727 shutdown_connections 2024-01-24T17:41:42.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.823+0000 7f6fd9600640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a4b0 0x7f6fbc06c970 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.823+0000 7f6fd9600640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc0e78b0 0x7f6fdc0e8730 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.823+0000 7f6fd9600640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fdc04f260 0x7f6fdc17d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.823+0000 7f6fd9600640 1 --2- 172.21.15.39:0/3218346727 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fdc038450 0x7f6fdc0ea310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.823+0000 7f6fd9600640 1 -- 172.21.15.39:0/3218346727 >> 172.21.15.39:0/3218346727 conn(0x7f6fdc054c10 msgr2=0x7f6fdc0e6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.823+0000 7f6fd9600640 1 -- 172.21.15.39:0/3218346727 shutdown_connections 2024-01-24T17:41:42.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.823+0000 7f6fd9600640 1 -- 172.21.15.39:0/3218346727 wait complete. 2024-01-24T17:41:42.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-01-24T17:41:42.850 DEBUG:teuthology.parallel:result is None 2024-01-24T17:41:42.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.860+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa5300b2140 con 0x7fa530157c30 2024-01-24T17:41:42.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.861+0000 7fa526a00640 1 -- 172.21.15.39:0/1058980600 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa518020090 con 0x7fa530157c30 2024-01-24T17:41:42.861 INFO:teuthology.orchestra.run.smithi039.stdout:55834574855 2024-01-24T17:41:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa50806a460 msgr2=0x7fa50806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa50806a460 0x7fa50806c920 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5280012e0 tx=0x7fa528009040 comp rx=0 tx=0).stop 2024-01-24T17:41:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530157c30 msgr2=0x7fa5300b6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530157c30 0x7fa5300b6fd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa51800e870 tx=0x7fa51800ed30 comp rx=0 tx=0).stop 2024-01-24T17:41:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 shutdown_connections 2024-01-24T17:41:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa50806a460 0x7fa50806c920 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530157c30 0x7fa5300b6fd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa53015eb80 0x7fa5300cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 --2- 172.21.15.39:0/1058980600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa5300cf990 0x7fa5300cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.864+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 >> 172.21.15.39:0/1058980600 conn(0x7fa530071790 msgr2=0x7fa53015ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.865+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 shutdown_connections 2024-01-24T17:41:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.865+0000 7fa535800640 1 -- 172.21.15.39:0/1058980600 wait complete. 2024-01-24T17:41:42.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.877+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f82480057d0 con 0x7f827815cdd0 2024-01-24T17:41:42.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.878+0000 7f826fe00640 1 -- 172.21.15.39:0/2569243853 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f826807acb0 con 0x7f827815cdd0 2024-01-24T17:41:42.879 INFO:teuthology.orchestra.run.smithi039.stdout:55834574855 2024-01-24T17:41:42.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.881+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825006a500 msgr2=0x7f825006c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.881+0000 7f827ea00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825006a500 0x7f825006c9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8260007fd0 tx=0x7f8260009040 comp rx=0 tx=0).stop 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.881+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 msgr2=0x7f8278157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.881+0000 7f827ea00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278157b00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8268033000 tx=0x7f82680046c0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.882+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 shutdown_connections 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.882+0000 7f827ea00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f827815cdd0 0x7f8278157b00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.882+0000 7f827ea00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825006a500 0x7f825006c9c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.882+0000 7f827ea00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82780cbab0 0x7f82780c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.882+0000 7f827ea00640 1 --2- 172.21.15.39:0/2569243853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82780b06d0 0x7f82781fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.882+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 >> 172.21.15.39:0/2569243853 conn(0x7f82780cafc0 msgr2=0x7f82780affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.882+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 shutdown_connections 2024-01-24T17:41:42.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.883+0000 7f827ea00640 1 -- 172.21.15.39:0/2569243853 wait complete. 2024-01-24T17:41:42.891 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-01-24T17:41:42.891 DEBUG:teuthology.parallel:result is None 2024-01-24T17:41:42.908 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-01-24T17:41:42.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.907+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0034009c30 con 0x7f0034006b40 2024-01-24T17:41:42.908 DEBUG:teuthology.parallel:result is None 2024-01-24T17:41:42.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.908+0000 7f0032000640 1 -- 172.21.15.39:0/3310833023 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0024057070 con 0x7f0034006b40 2024-01-24T17:41:42.909 INFO:teuthology.orchestra.run.smithi039.stdout:55834574856 2024-01-24T17:41:42.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.910+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1d940b6780 con 0x7f1d94154e10 2024-01-24T17:41:42.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.910+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f001406a480 msgr2=0x7f001406c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.910+0000 7f003b200640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f001406a480 0x7f001406c940 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f002c036000 tx=0x7f002c008030 comp rx=0 tx=0).stop 2024-01-24T17:41:42.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034006b40 msgr2=0x7f00340daf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034006b40 0x7f00340daf90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f002400c500 tx=0x7f002400c9c0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 shutdown_connections 2024-01-24T17:41:42.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f00340e3b00 0x7f00340d7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0034035130 0x7f00340d7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f001406a480 0x7f001406c940 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 --2- 172.21.15.39:0/3310833023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0034006b40 0x7f00340daf90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 >> 172.21.15.39:0/3310833023 conn(0x7f0034055d20 msgr2=0x7f0034009060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.912 INFO:teuthology.orchestra.run.smithi039.stdout:55834574854 2024-01-24T17:41:42.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f1d90c00640 1 -- 172.21.15.39:0/1330701402 <== mon.1 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1d7c056c70 con 0x7f1d94154e10 2024-01-24T17:41:42.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 shutdown_connections 2024-01-24T17:41:42.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.911+0000 7f003b200640 1 -- 172.21.15.39:0/3310833023 wait complete. 2024-01-24T17:41:42.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d6c06a460 msgr2=0x7f1d6c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d6c06a460 0x7f1d6c06c920 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1d84034000 tx=0x7f1d84009030 comp rx=0 tx=0).stop 2024-01-24T17:41:42.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d94154e10 msgr2=0x7f1d941590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d94154e10 0x7f1d941590d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00b420 tx=0x7f1d7c00b8e0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 shutdown_connections 2024-01-24T17:41:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d940b2630 0x7f1d94156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d6c06a460 0x7f1d6c06c920 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d9415b000 0x7f1d940b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 --2- 172.21.15.39:0/1330701402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d94154e10 0x7f1d941590d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 >> 172.21.15.39:0/1330701402 conn(0x7f1d94089fd0 msgr2=0x7f1d940ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 shutdown_connections 2024-01-24T17:41:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.913+0000 7f1d9ba00640 1 -- 172.21.15.39:0/1330701402 wait complete. 2024-01-24T17:41:42.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.916+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7fd4067040 con 0x7f7fd40b5b10 2024-01-24T17:41:42.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.917+0000 7f7fcbe00640 1 -- 172.21.15.39:0/4103684832 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7fcc044020 con 0x7f7fd40b5b10 2024-01-24T17:41:42.917 INFO:teuthology.orchestra.run.smithi039.stdout:55834574856 2024-01-24T17:41:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fac06a430 msgr2=0x7f7fac06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fac06a430 0x7f7fac06c8f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40d1360 tx=0x7f7fc4008040 comp rx=0 tx=0).stop 2024-01-24T17:41:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd40b5b10 msgr2=0x7f7fd415ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd40b5b10 0x7f7fd415ab30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009d90 tx=0x7f7fcc004a40 comp rx=0 tx=0).stop 2024-01-24T17:41:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 shutdown_connections 2024-01-24T17:41:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fd415cc90 0x7f7fd40c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fac06a430 0x7f7fac06c8f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd40cf9e0 0x7f7fd415f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 --2- 172.21.15.39:0/4103684832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fd40b5b10 0x7f7fd415ab30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 >> 172.21.15.39:0/4103684832 conn(0x7f7fd4071790 msgr2=0x7f7fd40b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.920+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 shutdown_connections 2024-01-24T17:41:42.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.921+0000 7f7fdaa00640 1 -- 172.21.15.39:0/4103684832 wait complete. 2024-01-24T17:41:42.930 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-01-24T17:41:42.930 DEBUG:teuthology.parallel:result is None 2024-01-24T17:41:42.932 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-01-24T17:41:42.932 DEBUG:teuthology.parallel:result is None 2024-01-24T17:41:42.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.932+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7efd3404f360 con 0x7efd3403e3b0 2024-01-24T17:41:42.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.932+0000 7efd33400640 1 -- 172.21.15.39:0/29879640 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd3404f360 con 0x7efd3403e3b0 2024-01-24T17:41:42.933 INFO:teuthology.orchestra.run.smithi039.stdout:55834574853 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd1406a430 msgr2=0x7efd1406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd1406a430 0x7efd1406c8f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efd3c074280 tx=0x7efd3c072cb0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd3403e3b0 msgr2=0x7efd3404f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd3403e3b0 0x7efd3404f640 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00c450 tx=0x7efd2c00c910 comp rx=0 tx=0).stop 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 shutdown_connections 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd3403e9a0 0x7efd34185080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd1406a430 0x7efd1406c8f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd3403e3b0 0x7efd3404f640 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 --2- 172.21.15.39:0/29879640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd340079d0 0x7efd3400d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 >> 172.21.15.39:0/29879640 conn(0x7efd34059120 msgr2=0x7efd340f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 shutdown_connections 2024-01-24T17:41:42.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.935+0000 7efd40c00640 1 -- 172.21.15.39:0/29879640 wait complete. 2024-01-24T17:41:42.945 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-01-24T17:41:42.945 DEBUG:teuthology.parallel:result is None 2024-01-24T17:41:42.951 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2024-01-24T17:41:42.951 DEBUG:teuthology.parallel:result is None 2024-01-24T17:41:42.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.955+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f86f004bc60 con 0x7f86f00e79b0 2024-01-24T17:41:42.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.956+0000 7f86efe00640 1 -- 172.21.15.39:0/32553162 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f86f80d09a0 con 0x7f86f00e79b0 2024-01-24T17:41:42.956 INFO:teuthology.orchestra.run.smithi039.stdout:55834574854 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86d006a530 msgr2=0x7f86d006c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86d006a530 0x7f86d006c9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009c20 tx=0x7f86e8008040 comp rx=0 tx=0).stop 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f00e79b0 msgr2=0x7f86f004ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f00e79b0 0x7f86f004ce60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f86f8068110 tx=0x7f86f8066fc0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 shutdown_connections 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86d006a530 0x7f86d006c9f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f00e79b0 0x7f86f004ce60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f00e75d0 0x7f86f004c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 --2- 172.21.15.39:0/32553162 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f004a320 0x7f86f00db2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 >> 172.21.15.39:0/32553162 conn(0x7f86f0053790 msgr2=0x7f86f0038780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 shutdown_connections 2024-01-24T17:41:42.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:42.958+0000 7f86ff200640 1 -- 172.21.15.39:0/32553162 wait complete. 2024-01-24T17:41:42.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-01-24T17:41:42.977 DEBUG:teuthology.parallel:result is None 2024-01-24T17:41:42.977 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-24T17:41:42.977 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:41:43.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.225+0000 7f6307e00640 1 Processor -- start 2024-01-24T17:41:43.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.226+0000 7f6307e00640 1 -- start start 2024-01-24T17:41:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.227+0000 7f6307e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630815ab00 0x7f63080cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.227+0000 7f6307e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63080ccd20 0x7f63080ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.227+0000 7f6307e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080ce400 con 0x7f630815a720 2024-01-24T17:41:43.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.227+0000 7f6307e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630815b3b0 con 0x7f63080ccd20 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.227+0000 7f6307e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630815b690 con 0x7f630815ab00 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.227+0000 7f6307400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63080ccd20 0x7f63080ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.227+0000 7f6307400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63080ccd20 0x7f63080ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58062/0 (socket says 172.21.15.39:58062) 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.227+0000 7f6307400640 1 -- 172.21.15.39:0/829490072 learned_addr learned my addr 172.21.15.39:0/829490072 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.228+0000 7f62fde00640 1 --2- 172.21.15.39:0/829490072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630815ab00 0x7f63080cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.228+0000 7f6307400640 1 -- 172.21.15.39:0/829490072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630815ab00 msgr2=0x7f63080cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.228+0000 7f6307400640 1 --2- 172.21.15.39:0/829490072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630815ab00 0x7f63080cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.228+0000 7f6307400640 1 -- 172.21.15.39:0/829490072 >> v1:172.21.15.100:6789/0 conn(0x7f630815a720 legacy=0x7f63080cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:43.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.228+0000 7f6307400640 1 -- 172.21.15.39:0/829490072 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630815b970 con 0x7f63080ccd20 2024-01-24T17:41:43.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.229+0000 7f6307400640 1 --2- 172.21.15.39:0/829490072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63080ccd20 0x7f63080ce8e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6300009e50 tx=0x7f630002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57a6a0f19ab1cad server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:43.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.229+0000 7f6306000640 1 -- 172.21.15.39:0/829490072 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630002c9e0 con 0x7f63080ccd20 2024-01-24T17:41:43.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.229+0000 7f6306000640 1 -- 172.21.15.39:0/829490072 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630002cb40 con 0x7f63080ccd20 2024-01-24T17:41:43.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.229+0000 7f6306000640 1 -- 172.21.15.39:0/829490072 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63000343f0 con 0x7f63080ccd20 2024-01-24T17:41:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.230+0000 7f6307e00640 1 -- 172.21.15.39:0/829490072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63080ccd20 msgr2=0x7f63080ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.230+0000 7f6307e00640 1 --2- 172.21.15.39:0/829490072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63080ccd20 0x7f63080ce8e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6300009e50 tx=0x7f630002be40 comp rx=0 tx=0).stop 2024-01-24T17:41:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.230+0000 7f6307e00640 1 -- 172.21.15.39:0/829490072 shutdown_connections 2024-01-24T17:41:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.230+0000 7f6307e00640 1 --2- 172.21.15.39:0/829490072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63080ccd20 0x7f63080ce8e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.230+0000 7f6307e00640 1 --2- 172.21.15.39:0/829490072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f630815ab00 0x7f63080cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.230+0000 7f6307e00640 1 -- 172.21.15.39:0/829490072 >> 172.21.15.39:0/829490072 conn(0x7f630806a090 msgr2=0x7f630806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.230+0000 7f6307e00640 1 -- 172.21.15.39:0/829490072 shutdown_connections 2024-01-24T17:41:43.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.230+0000 7f6307e00640 1 -- 172.21.15.39:0/829490072 wait complete. 2024-01-24T17:41:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.232+0000 7f6307e00640 1 Processor -- start 2024-01-24T17:41:43.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.232+0000 7f6307e00640 1 -- start start 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f6307e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f630815a720 0x7f63080c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f6307e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f630815ab00 0x7f63080acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f6307e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63080ad000 0x7f6308147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f6307e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080ad490 con 0x7f630815ab00 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f6307e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308149890 con 0x7f630815a720 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f6307e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6308149b30 con 0x7f63080ad000 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f62fde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f630815ab00 0x7f63080acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f62fde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f630815ab00 0x7f63080acac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58074/0 (socket says 172.21.15.39:58074) 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f62fde00640 1 -- 172.21.15.39:0/3912509166 learned_addr learned my addr 172.21.15.39:0/3912509166 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f62fde00640 1 -- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63080ad000 msgr2=0x7f6308147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f62fde00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63080ad000 0x7f6308147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f62fde00640 1 -- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f630815a720 msgr2=0x7f63080c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f62fde00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f630815a720 0x7f63080c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.233+0000 7f62fde00640 1 -- 172.21.15.39:0/3912509166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6300009b30 con 0x7f630815ab00 2024-01-24T17:41:43.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.234+0000 7f62fde00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f630815ab00 0x7f63080acac0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f62f000d800 tx=0x7f62f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:43.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.234+0000 7f62ff400640 1 -- 172.21.15.39:0/3912509166 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f000b6c0 con 0x7f630815ab00 2024-01-24T17:41:43.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.234+0000 7f62ff400640 1 -- 172.21.15.39:0/3912509166 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f000b820 con 0x7f630815ab00 2024-01-24T17:41:43.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.234+0000 7f62ff400640 1 -- 172.21.15.39:0/3912509166 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f000bbb0 con 0x7f630815ab00 2024-01-24T17:41:43.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.234+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6308149e10 con 0x7f630815ab00 2024-01-24T17:41:43.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.234+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630814a3d0 con 0x7f630815ab00 2024-01-24T17:41:43.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.235+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630804dff0 con 0x7f630815ab00 2024-01-24T17:41:43.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.236+0000 7f62ff400640 1 -- 172.21.15.39:0/3912509166 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f62f000bd10 con 0x7f630815ab00 2024-01-24T17:41:43.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.237+0000 7f62ff400640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62e006a430 0x7f62e006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.237+0000 7f62ff400640 1 -- 172.21.15.39:0/3912509166 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62f0088d90 con 0x7f630815ab00 2024-01-24T17:41:43.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.238+0000 7f6306a00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62e006a430 0x7f62e006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:43.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.238+0000 7f6306a00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62e006a430 0x7f62e006c8f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f62f8003cc0 tx=0x7f62f80079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:43.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.242+0000 7f62ff400640 1 -- 172.21.15.39:0/3912509166 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f62f0054410 con 0x7f630815ab00 2024-01-24T17:41:43.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.427+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f630815bc50 con 0x7f62e006a430 2024-01-24T17:41:43.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.447+0000 7f62ff400640 1 -- 172.21.15.39:0/3912509166 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264436 (secure 0 0 0) 0x7f62f8014400 con 0x7f62e006a430 2024-01-24T17:41:43.452 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:43.453 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62e006a430 msgr2=0x7f62e006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62e006a430 0x7f62e006c8f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f62f8003cc0 tx=0x7f62f80079b0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f630815ab00 msgr2=0x7f63080acac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f630815ab00 0x7f63080acac0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f62f000d800 tx=0x7f62f000dcc0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 shutdown_connections 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63080ad000 0x7f6308147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62e006a430 0x7f62e006c8f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f630815ab00 0x7f63080acac0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.453+0000 7f6307e00640 1 --2- 172.21.15.39:0/3912509166 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f630815a720 0x7f63080c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.454+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 >> 172.21.15.39:0/3912509166 conn(0x7f630806a090 msgr2=0x7f630815bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.454+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 shutdown_connections 2024-01-24T17:41:43.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.454+0000 7f6307e00640 1 -- 172.21.15.39:0/3912509166 wait complete. 2024-01-24T17:41:43.473 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-24T17:41:42.211403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235372,"kb_used_data":2824,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754739348,"statfs":{"total":773094113280,"available":772853092352,"internally_reserved":0,"allocated":2891776,"data_stored":1752644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.847782"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382873+0000","last_change":"2024-01-24T17:41:33.393186+0000","last_active":"2024-01-24T17:41:34.382873+0000","last_peered":"2024-01-24T17:41:34.382873+0000","last_clean":"2024-01-24T17:41:34.382873+0000","last_became_active":"2024-01-24T17:41:33.391034+0000","last_became_peered":"2024-01-24T17:41:33.391034+0000","last_unstale":"2024-01-24T17:41:34.382873+0000","last_undegraded":"2024-01-24T17:41:34.382873+0000","last_fullsized":"2024-01-24T17:41:34.382873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376217+0000","last_change":"2024-01-24T17:41:33.384027+0000","last_active":"2024-01-24T17:41:34.376217+0000","last_peered":"2024-01-24T17:41:34.376217+0000","last_clean":"2024-01-24T17:41:34.376217+0000","last_became_active":"2024-01-24T17:41:33.383720+0000","last_became_peered":"2024-01-24T17:41:33.383720+0000","last_unstale":"2024-01-24T17:41:34.376217+0000","last_undegraded":"2024-01-24T17:41:34.376217+0000","last_fullsized":"2024-01-24T17:41:34.376217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376005+0000","last_change":"2024-01-24T17:41:33.392448+0000","last_active":"2024-01-24T17:41:34.376005+0000","last_peered":"2024-01-24T17:41:34.376005+0000","last_clean":"2024-01-24T17:41:34.376005+0000","last_became_active":"2024-01-24T17:41:33.392125+0000","last_became_peered":"2024-01-24T17:41:33.392125+0000","last_unstale":"2024-01-24T17:41:34.376005+0000","last_undegraded":"2024-01-24T17:41:34.376005+0000","last_fullsized":"2024-01-24T17:41:34.376005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:20:35.949143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.385632+0000","last_change":"2024-01-24T17:41:33.395630+0000","last_active":"2024-01-24T17:41:34.385632+0000","last_peered":"2024-01-24T17:41:34.385632+0000","last_clean":"2024-01-24T17:41:34.385632+0000","last_became_active":"2024-01-24T17:41:33.395378+0000","last_became_peered":"2024-01-24T17:41:33.395378+0000","last_unstale":"2024-01-24T17:41:34.385632+0000","last_undegraded":"2024-01-24T17:41:34.385632+0000","last_fullsized":"2024-01-24T17:41:34.385632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382553+0000","last_change":"2024-01-24T17:41:33.382453+0000","last_active":"2024-01-24T17:41:34.382553+0000","last_peered":"2024-01-24T17:41:34.382553+0000","last_clean":"2024-01-24T17:41:34.382553+0000","last_became_active":"2024-01-24T17:41:33.382151+0000","last_became_peered":"2024-01-24T17:41:33.382151+0000","last_unstale":"2024-01-24T17:41:34.382553+0000","last_undegraded":"2024-01-24T17:41:34.382553+0000","last_fullsized":"2024-01-24T17:41:34.382553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.384184+0000","last_change":"2024-01-24T17:41:33.396138+0000","last_active":"2024-01-24T17:41:34.384184+0000","last_peered":"2024-01-24T17:41:34.384184+0000","last_clean":"2024-01-24T17:41:34.384184+0000","last_became_active":"2024-01-24T17:41:33.395890+0000","last_became_peered":"2024-01-24T17:41:33.395890+0000","last_unstale":"2024-01-24T17:41:34.384184+0000","last_undegraded":"2024-01-24T17:41:34.384184+0000","last_fullsized":"2024-01-24T17:41:34.384184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:17.651326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381377+0000","last_change":"2024-01-24T17:41:33.393003+0000","last_active":"2024-01-24T17:41:34.381377+0000","last_peered":"2024-01-24T17:41:34.381377+0000","last_clean":"2024-01-24T17:41:34.381377+0000","last_became_active":"2024-01-24T17:41:33.391259+0000","last_became_peered":"2024-01-24T17:41:33.391259+0000","last_unstale":"2024-01-24T17:41:34.381377+0000","last_undegraded":"2024-01-24T17:41:34.381377+0000","last_fullsized":"2024-01-24T17:41:34.381377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:50:21.701964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374081+0000","last_change":"2024-01-24T17:41:33.402016+0000","last_active":"2024-01-24T17:41:34.374081+0000","last_peered":"2024-01-24T17:41:34.374081+0000","last_clean":"2024-01-24T17:41:34.374081+0000","last_became_active":"2024-01-24T17:41:33.401767+0000","last_became_peered":"2024-01-24T17:41:33.401767+0000","last_unstale":"2024-01-24T17:41:34.374081+0000","last_undegraded":"2024-01-24T17:41:34.374081+0000","last_fullsized":"2024-01-24T17:41:34.374081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:35:09.121979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.387416+0000","last_change":"2024-01-24T17:41:33.391456+0000","last_active":"2024-01-24T17:41:34.387416+0000","last_peered":"2024-01-24T17:41:34.387416+0000","last_clean":"2024-01-24T17:41:34.387416+0000","last_became_active":"2024-01-24T17:41:33.391192+0000","last_became_peered":"2024-01-24T17:41:33.391192+0000","last_unstale":"2024-01-24T17:41:34.387416+0000","last_undegraded":"2024-01-24T17:41:34.387416+0000","last_fullsized":"2024-01-24T17:41:34.387416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382754+0000","last_change":"2024-01-24T17:41:33.380252+0000","last_active":"2024-01-24T17:41:34.382754+0000","last_peered":"2024-01-24T17:41:34.382754+0000","last_clean":"2024-01-24T17:41:34.382754+0000","last_became_active":"2024-01-24T17:41:33.378238+0000","last_became_peered":"2024-01-24T17:41:33.378238+0000","last_unstale":"2024-01-24T17:41:34.382754+0000","last_undegraded":"2024-01-24T17:41:34.382754+0000","last_fullsized":"2024-01-24T17:41:34.382754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:01:48.740461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382398+0000","last_change":"2024-01-24T17:41:33.382673+0000","last_active":"2024-01-24T17:41:34.382398+0000","last_peered":"2024-01-24T17:41:34.382398+0000","last_clean":"2024-01-24T17:41:34.382398+0000","last_became_active":"2024-01-24T17:41:33.382404+0000","last_became_peered":"2024-01-24T17:41:33.382404+0000","last_unstale":"2024-01-24T17:41:34.382398+0000","last_undegraded":"2024-01-24T17:41:34.382398+0000","last_fullsized":"2024-01-24T17:41:34.382398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:44:38.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383532+0000","last_change":"2024-01-24T17:41:33.395359+0000","last_active":"2024-01-24T17:41:34.383532+0000","last_peered":"2024-01-24T17:41:34.383532+0000","last_clean":"2024-01-24T17:41:34.383532+0000","last_became_active":"2024-01-24T17:41:33.395102+0000","last_became_peered":"2024-01-24T17:41:33.395102+0000","last_unstale":"2024-01-24T17:41:34.383532+0000","last_undegraded":"2024-01-24T17:41:34.383532+0000","last_fullsized":"2024-01-24T17:41:34.383532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372275+0000","last_change":"2024-01-24T17:41:33.391912+0000","last_active":"2024-01-24T17:41:34.372275+0000","last_peered":"2024-01-24T17:41:34.372275+0000","last_clean":"2024-01-24T17:41:34.372275+0000","last_became_active":"2024-01-24T17:41:33.391421+0000","last_became_peered":"2024-01-24T17:41:33.391421+0000","last_unstale":"2024-01-24T17:41:34.372275+0000","last_undegraded":"2024-01-24T17:41:34.372275+0000","last_fullsized":"2024-01-24T17:41:34.372275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:36:51.994706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.371942+0000","last_change":"2024-01-24T17:41:33.381744+0000","last_active":"2024-01-24T17:41:34.371942+0000","last_peered":"2024-01-24T17:41:34.371942+0000","last_clean":"2024-01-24T17:41:34.371942+0000","last_became_active":"2024-01-24T17:41:33.381499+0000","last_became_peered":"2024-01-24T17:41:33.381499+0000","last_unstale":"2024-01-24T17:41:34.371942+0000","last_undegraded":"2024-01-24T17:41:34.371942+0000","last_fullsized":"2024-01-24T17:41:34.371942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:27.474325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373889+0000","last_change":"2024-01-24T17:41:33.382763+0000","last_active":"2024-01-24T17:41:34.373889+0000","last_peered":"2024-01-24T17:41:34.373889+0000","last_clean":"2024-01-24T17:41:34.373889+0000","last_became_active":"2024-01-24T17:41:33.382506+0000","last_became_peered":"2024-01-24T17:41:33.382506+0000","last_unstale":"2024-01-24T17:41:34.373889+0000","last_undegraded":"2024-01-24T17:41:34.373889+0000","last_fullsized":"2024-01-24T17:41:34.373889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:29.008097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383246+0000","last_change":"2024-01-24T17:41:33.398656+0000","last_active":"2024-01-24T17:41:34.383246+0000","last_peered":"2024-01-24T17:41:34.383246+0000","last_clean":"2024-01-24T17:41:34.383246+0000","last_became_active":"2024-01-24T17:41:33.398404+0000","last_became_peered":"2024-01-24T17:41:33.398404+0000","last_unstale":"2024-01-24T17:41:34.383246+0000","last_undegraded":"2024-01-24T17:41:34.383246+0000","last_fullsized":"2024-01-24T17:41:34.383246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:36:23.946397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383245+0000","last_change":"2024-01-24T17:41:33.401897+0000","last_active":"2024-01-24T17:41:34.383245+0000","last_peered":"2024-01-24T17:41:34.383245+0000","last_clean":"2024-01-24T17:41:34.383245+0000","last_became_active":"2024-01-24T17:41:33.401641+0000","last_became_peered":"2024-01-24T17:41:33.401641+0000","last_unstale":"2024-01-24T17:41:34.383245+0000","last_undegraded":"2024-01-24T17:41:34.383245+0000","last_fullsized":"2024-01-24T17:41:34.383245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:52:01.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382136+0000","last_change":"2024-01-24T17:41:33.397837+0000","last_active":"2024-01-24T17:41:34.382136+0000","last_peered":"2024-01-24T17:41:34.382136+0000","last_clean":"2024-01-24T17:41:34.382136+0000","last_became_active":"2024-01-24T17:41:33.397597+0000","last_became_peered":"2024-01-24T17:41:33.397597+0000","last_unstale":"2024-01-24T17:41:34.382136+0000","last_undegraded":"2024-01-24T17:41:34.382136+0000","last_fullsized":"2024-01-24T17:41:34.382136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381810+0000","last_change":"2024-01-24T17:41:33.389940+0000","last_active":"2024-01-24T17:41:34.381810+0000","last_peered":"2024-01-24T17:41:34.381810+0000","last_clean":"2024-01-24T17:41:34.381810+0000","last_became_active":"2024-01-24T17:41:33.389673+0000","last_became_peered":"2024-01-24T17:41:33.389673+0000","last_unstale":"2024-01-24T17:41:34.381810+0000","last_undegraded":"2024-01-24T17:41:34.381810+0000","last_fullsized":"2024-01-24T17:41:34.381810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:48:57.485742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383781+0000","last_change":"2024-01-24T17:41:33.394799+0000","last_active":"2024-01-24T17:41:34.383781+0000","last_peered":"2024-01-24T17:41:34.383781+0000","last_clean":"2024-01-24T17:41:34.383781+0000","last_became_active":"2024-01-24T17:41:33.394396+0000","last_became_peered":"2024-01-24T17:41:33.394396+0000","last_unstale":"2024-01-24T17:41:34.383781+0000","last_undegraded":"2024-01-24T17:41:34.383781+0000","last_fullsized":"2024-01-24T17:41:34.383781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:35:09.121979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372973+0000","last_change":"2024-01-24T17:41:33.402158+0000","last_active":"2024-01-24T17:41:34.372973+0000","last_peered":"2024-01-24T17:41:34.372973+0000","last_clean":"2024-01-24T17:41:34.372973+0000","last_became_active":"2024-01-24T17:41:33.400801+0000","last_became_peered":"2024-01-24T17:41:33.400801+0000","last_unstale":"2024-01-24T17:41:34.372973+0000","last_undegraded":"2024-01-24T17:41:34.372973+0000","last_fullsized":"2024-01-24T17:41:34.372973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:29.008097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386543+0000","last_change":"2024-01-24T17:41:33.399257+0000","last_active":"2024-01-24T17:41:34.386543+0000","last_peered":"2024-01-24T17:41:34.386543+0000","last_clean":"2024-01-24T17:41:34.386543+0000","last_became_active":"2024-01-24T17:41:33.398569+0000","last_became_peered":"2024-01-24T17:41:33.398569+0000","last_unstale":"2024-01-24T17:41:34.386543+0000","last_undegraded":"2024-01-24T17:41:34.386543+0000","last_fullsized":"2024-01-24T17:41:34.386543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:54.108693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374158+0000","last_change":"2024-01-24T17:41:33.372320+0000","last_active":"2024-01-24T17:41:34.374158+0000","last_peered":"2024-01-24T17:41:34.374158+0000","last_clean":"2024-01-24T17:41:34.374158+0000","last_became_active":"2024-01-24T17:41:33.372080+0000","last_became_peered":"2024-01-24T17:41:33.372080+0000","last_unstale":"2024-01-24T17:41:34.374158+0000","last_undegraded":"2024-01-24T17:41:34.374158+0000","last_fullsized":"2024-01-24T17:41:34.374158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381086+0000","last_change":"2024-01-24T17:41:33.388373+0000","last_active":"2024-01-24T17:41:34.381086+0000","last_peered":"2024-01-24T17:41:34.381086+0000","last_clean":"2024-01-24T17:41:34.381086+0000","last_became_active":"2024-01-24T17:41:33.387185+0000","last_became_peered":"2024-01-24T17:41:33.387185+0000","last_unstale":"2024-01-24T17:41:34.381086+0000","last_undegraded":"2024-01-24T17:41:34.381086+0000","last_fullsized":"2024-01-24T17:41:34.381086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376756+0000","last_change":"2024-01-24T17:41:33.396814+0000","last_active":"2024-01-24T17:41:34.376756+0000","last_peered":"2024-01-24T17:41:34.376756+0000","last_clean":"2024-01-24T17:41:34.376756+0000","last_became_active":"2024-01-24T17:41:33.396446+0000","last_became_peered":"2024-01-24T17:41:33.396446+0000","last_unstale":"2024-01-24T17:41:34.376756+0000","last_undegraded":"2024-01-24T17:41:34.376756+0000","last_fullsized":"2024-01-24T17:41:34.376756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386135+0000","last_change":"2024-01-24T17:41:33.386328+0000","last_active":"2024-01-24T17:41:34.386135+0000","last_peered":"2024-01-24T17:41:34.386135+0000","last_clean":"2024-01-24T17:41:34.386135+0000","last_became_active":"2024-01-24T17:41:33.384795+0000","last_became_peered":"2024-01-24T17:41:33.384795+0000","last_unstale":"2024-01-24T17:41:34.386135+0000","last_undegraded":"2024-01-24T17:41:34.386135+0000","last_fullsized":"2024-01-24T17:41:34.386135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372759+0000","last_change":"2024-01-24T17:41:33.381873+0000","last_active":"2024-01-24T17:41:34.372759+0000","last_peered":"2024-01-24T17:41:34.372759+0000","last_clean":"2024-01-24T17:41:34.372759+0000","last_became_active":"2024-01-24T17:41:33.381356+0000","last_became_peered":"2024-01-24T17:41:33.381356+0000","last_unstale":"2024-01-24T17:41:34.372759+0000","last_undegraded":"2024-01-24T17:41:34.372759+0000","last_fullsized":"2024-01-24T17:41:34.372759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372500+0000","last_change":"2024-01-24T17:41:33.401722+0000","last_active":"2024-01-24T17:41:34.372500+0000","last_peered":"2024-01-24T17:41:34.372500+0000","last_clean":"2024-01-24T17:41:34.372500+0000","last_became_active":"2024-01-24T17:41:33.401437+0000","last_became_peered":"2024-01-24T17:41:33.401437+0000","last_unstale":"2024-01-24T17:41:34.372500+0000","last_undegraded":"2024-01-24T17:41:34.372500+0000","last_fullsized":"2024-01-24T17:41:34.372500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:53:07.257519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.375137+0000","last_change":"2024-01-24T17:41:33.386028+0000","last_active":"2024-01-24T17:41:34.375137+0000","last_peered":"2024-01-24T17:41:34.375137+0000","last_clean":"2024-01-24T17:41:34.375137+0000","last_became_active":"2024-01-24T17:41:33.385728+0000","last_became_peered":"2024-01-24T17:41:33.385728+0000","last_unstale":"2024-01-24T17:41:34.375137+0000","last_undegraded":"2024-01-24T17:41:34.375137+0000","last_fullsized":"2024-01-24T17:41:34.375137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:14.826051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381062+0000","last_change":"2024-01-24T17:41:33.392322+0000","last_active":"2024-01-24T17:41:34.381062+0000","last_peered":"2024-01-24T17:41:34.381062+0000","last_clean":"2024-01-24T17:41:34.381062+0000","last_became_active":"2024-01-24T17:41:33.391949+0000","last_became_peered":"2024-01-24T17:41:33.391949+0000","last_unstale":"2024-01-24T17:41:34.381062+0000","last_undegraded":"2024-01-24T17:41:34.381062+0000","last_fullsized":"2024-01-24T17:41:34.381062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381378+0000","last_change":"2024-01-24T17:41:33.392982+0000","last_active":"2024-01-24T17:41:34.381378+0000","last_peered":"2024-01-24T17:41:34.381378+0000","last_clean":"2024-01-24T17:41:34.381378+0000","last_became_active":"2024-01-24T17:41:33.392653+0000","last_became_peered":"2024-01-24T17:41:33.392653+0000","last_unstale":"2024-01-24T17:41:34.381378+0000","last_undegraded":"2024-01-24T17:41:34.381378+0000","last_fullsized":"2024-01-24T17:41:34.381378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382379+0000","last_change":"2024-01-24T17:41:33.388829+0000","last_active":"2024-01-24T17:41:34.382379+0000","last_peered":"2024-01-24T17:41:34.382379+0000","last_clean":"2024-01-24T17:41:34.382379+0000","last_became_active":"2024-01-24T17:41:33.387836+0000","last_became_peered":"2024-01-24T17:41:33.387836+0000","last_unstale":"2024-01-24T17:41:34.382379+0000","last_undegraded":"2024-01-24T17:41:34.382379+0000","last_fullsized":"2024-01-24T17:41:34.382379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386559+0000","last_change":"2024-01-24T17:41:33.386169+0000","last_active":"2024-01-24T17:41:34.386559+0000","last_peered":"2024-01-24T17:41:34.386559+0000","last_clean":"2024-01-24T17:41:34.386559+0000","last_became_active":"2024-01-24T17:41:33.385898+0000","last_became_peered":"2024-01-24T17:41:33.385898+0000","last_unstale":"2024-01-24T17:41:34.386559+0000","last_undegraded":"2024-01-24T17:41:34.386559+0000","last_fullsized":"2024-01-24T17:41:34.386559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:46.793343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402349+0000","last_change":"2024-01-24T17:41:32.895862+0000","last_active":"2024-01-24T17:41:33.402349+0000","last_peered":"2024-01-24T17:41:33.402349+0000","last_clean":"2024-01-24T17:41:33.402349+0000","last_became_active":"2024-01-24T17:41:32.895370+0000","last_became_peered":"2024-01-24T17:41:32.895370+0000","last_unstale":"2024-01-24T17:41:33.402349+0000","last_undegraded":"2024-01-24T17:41:33.402349+0000","last_fullsized":"2024-01-24T17:41:33.402349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373080+0000","last_change":"2024-01-24T17:41:33.405202+0000","last_active":"2024-01-24T17:41:34.373080+0000","last_peered":"2024-01-24T17:41:34.373080+0000","last_clean":"2024-01-24T17:41:34.373080+0000","last_became_active":"2024-01-24T17:41:33.404917+0000","last_became_peered":"2024-01-24T17:41:33.404917+0000","last_unstale":"2024-01-24T17:41:34.373080+0000","last_undegraded":"2024-01-24T17:41:34.373080+0000","last_fullsized":"2024-01-24T17:41:34.373080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404334+0000","last_change":"2024-01-24T17:41:32.910250+0000","last_active":"2024-01-24T17:41:33.404334+0000","last_peered":"2024-01-24T17:41:33.404334+0000","last_clean":"2024-01-24T17:41:33.404334+0000","last_became_active":"2024-01-24T17:41:32.909619+0000","last_became_peered":"2024-01-24T17:41:32.909619+0000","last_unstale":"2024-01-24T17:41:33.404334+0000","last_undegraded":"2024-01-24T17:41:33.404334+0000","last_fullsized":"2024-01-24T17:41:33.404334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:56.825045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.373938+0000","last_change":"2024-01-24T17:41:32.895589+0000","last_active":"2024-01-24T17:41:33.373938+0000","last_peered":"2024-01-24T17:41:33.373938+0000","last_clean":"2024-01-24T17:41:33.373938+0000","last_became_active":"2024-01-24T17:41:32.895305+0000","last_became_peered":"2024-01-24T17:41:32.895305+0000","last_unstale":"2024-01-24T17:41:33.373938+0000","last_undegraded":"2024-01-24T17:41:33.373938+0000","last_fullsized":"2024-01-24T17:41:33.373938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:01:37.962464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374431+0000","last_change":"2024-01-24T17:41:33.385480+0000","last_active":"2024-01-24T17:41:34.374431+0000","last_peered":"2024-01-24T17:41:34.374431+0000","last_clean":"2024-01-24T17:41:34.374431+0000","last_became_active":"2024-01-24T17:41:33.384848+0000","last_became_peered":"2024-01-24T17:41:33.384848+0000","last_unstale":"2024-01-24T17:41:34.374431+0000","last_undegraded":"2024-01-24T17:41:34.374431+0000","last_fullsized":"2024-01-24T17:41:34.374431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:50:21.701964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377136+0000","last_change":"2024-01-24T17:41:32.897056+0000","last_active":"2024-01-24T17:41:33.377136+0000","last_peered":"2024-01-24T17:41:33.377136+0000","last_clean":"2024-01-24T17:41:33.377136+0000","last_became_active":"2024-01-24T17:41:32.888851+0000","last_became_peered":"2024-01-24T17:41:32.888851+0000","last_unstale":"2024-01-24T17:41:33.377136+0000","last_undegraded":"2024-01-24T17:41:33.377136+0000","last_fullsized":"2024-01-24T17:41:33.377136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374368+0000","last_change":"2024-01-24T17:41:33.395388+0000","last_active":"2024-01-24T17:41:34.374368+0000","last_peered":"2024-01-24T17:41:34.374368+0000","last_clean":"2024-01-24T17:41:34.374368+0000","last_became_active":"2024-01-24T17:41:33.395100+0000","last_became_peered":"2024-01-24T17:41:33.395100+0000","last_unstale":"2024-01-24T17:41:34.374368+0000","last_undegraded":"2024-01-24T17:41:34.374368+0000","last_fullsized":"2024-01-24T17:41:34.374368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:17:59.263270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.383197+0000","last_change":"2024-01-24T17:41:32.892978+0000","last_active":"2024-01-24T17:41:33.383197+0000","last_peered":"2024-01-24T17:41:33.383197+0000","last_clean":"2024-01-24T17:41:33.383197+0000","last_became_active":"2024-01-24T17:41:32.892668+0000","last_became_peered":"2024-01-24T17:41:32.892668+0000","last_unstale":"2024-01-24T17:41:33.383197+0000","last_undegraded":"2024-01-24T17:41:33.383197+0000","last_fullsized":"2024-01-24T17:41:33.383197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:01:37.962464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386042+0000","last_change":"2024-01-24T17:41:33.384625+0000","last_active":"2024-01-24T17:41:34.386042+0000","last_peered":"2024-01-24T17:41:34.386042+0000","last_clean":"2024-01-24T17:41:34.386042+0000","last_became_active":"2024-01-24T17:41:33.384293+0000","last_became_peered":"2024-01-24T17:41:33.384293+0000","last_unstale":"2024-01-24T17:41:34.386042+0000","last_undegraded":"2024-01-24T17:41:34.386042+0000","last_fullsized":"2024-01-24T17:41:34.386042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:27.474325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.369468+0000","last_change":"2024-01-24T17:41:32.896336+0000","last_active":"2024-01-24T17:41:33.369468+0000","last_peered":"2024-01-24T17:41:33.369468+0000","last_clean":"2024-01-24T17:41:33.369468+0000","last_became_active":"2024-01-24T17:41:32.896023+0000","last_became_peered":"2024-01-24T17:41:32.896023+0000","last_unstale":"2024-01-24T17:41:33.369468+0000","last_undegraded":"2024-01-24T17:41:33.369468+0000","last_fullsized":"2024-01-24T17:41:33.369468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.377112+0000","last_change":"2024-01-24T17:41:33.384509+0000","last_active":"2024-01-24T17:41:34.377112+0000","last_peered":"2024-01-24T17:41:34.377112+0000","last_clean":"2024-01-24T17:41:34.377112+0000","last_became_active":"2024-01-24T17:41:33.384188+0000","last_became_peered":"2024-01-24T17:41:33.384188+0000","last_unstale":"2024-01-24T17:41:34.377112+0000","last_undegraded":"2024-01-24T17:41:34.377112+0000","last_fullsized":"2024-01-24T17:41:34.377112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:02.263592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.394541+0000","last_change":"2024-01-24T17:41:32.898747+0000","last_active":"2024-01-24T17:41:33.394541+0000","last_peered":"2024-01-24T17:41:33.394541+0000","last_clean":"2024-01-24T17:41:33.394541+0000","last_became_active":"2024-01-24T17:41:32.896625+0000","last_became_peered":"2024-01-24T17:41:32.896625+0000","last_unstale":"2024-01-24T17:41:33.394541+0000","last_undegraded":"2024-01-24T17:41:33.394541+0000","last_fullsized":"2024-01-24T17:41:33.394541+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381067+0000","last_change":"2024-01-24T17:41:33.385876+0000","last_active":"2024-01-24T17:41:34.381067+0000","last_peered":"2024-01-24T17:41:34.381067+0000","last_clean":"2024-01-24T17:41:34.381067+0000","last_became_active":"2024-01-24T17:41:33.385566+0000","last_became_peered":"2024-01-24T17:41:33.385566+0000","last_unstale":"2024-01-24T17:41:34.381067+0000","last_undegraded":"2024-01-24T17:41:34.381067+0000","last_fullsized":"2024-01-24T17:41:34.381067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:44:38.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.411432+0000","last_change":"2024-01-24T17:41:32.901159+0000","last_active":"2024-01-24T17:41:33.411432+0000","last_peered":"2024-01-24T17:41:33.411432+0000","last_clean":"2024-01-24T17:41:33.411432+0000","last_became_active":"2024-01-24T17:41:32.900832+0000","last_became_peered":"2024-01-24T17:41:32.900832+0000","last_unstale":"2024-01-24T17:41:33.411432+0000","last_undegraded":"2024-01-24T17:41:33.411432+0000","last_fullsized":"2024-01-24T17:41:33.411432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:06:04.595152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383859+0000","last_change":"2024-01-24T17:41:33.382102+0000","last_active":"2024-01-24T17:41:34.383859+0000","last_peered":"2024-01-24T17:41:34.383859+0000","last_clean":"2024-01-24T17:41:34.383859+0000","last_became_active":"2024-01-24T17:41:33.381813+0000","last_became_peered":"2024-01-24T17:41:33.381813+0000","last_unstale":"2024-01-24T17:41:34.383859+0000","last_undegraded":"2024-01-24T17:41:34.383859+0000","last_fullsized":"2024-01-24T17:41:34.383859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.405772+0000","last_change":"2024-01-24T17:41:32.896282+0000","last_active":"2024-01-24T17:41:33.405772+0000","last_peered":"2024-01-24T17:41:33.405772+0000","last_clean":"2024-01-24T17:41:33.405772+0000","last_became_active":"2024-01-24T17:41:32.896029+0000","last_became_peered":"2024-01-24T17:41:32.896029+0000","last_unstale":"2024-01-24T17:41:33.405772+0000","last_undegraded":"2024-01-24T17:41:33.405772+0000","last_fullsized":"2024-01-24T17:41:33.405772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372253+0000","last_change":"2024-01-24T17:41:33.380019+0000","last_active":"2024-01-24T17:41:34.372253+0000","last_peered":"2024-01-24T17:41:34.372253+0000","last_clean":"2024-01-24T17:41:34.372253+0000","last_became_active":"2024-01-24T17:41:33.379743+0000","last_became_peered":"2024-01-24T17:41:33.379743+0000","last_unstale":"2024-01-24T17:41:34.372253+0000","last_undegraded":"2024-01-24T17:41:34.372253+0000","last_fullsized":"2024-01-24T17:41:34.372253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:01:48.740461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377512+0000","last_change":"2024-01-24T17:41:32.906190+0000","last_active":"2024-01-24T17:41:33.377512+0000","last_peered":"2024-01-24T17:41:33.377512+0000","last_clean":"2024-01-24T17:41:33.377512+0000","last_became_active":"2024-01-24T17:41:32.903525+0000","last_became_peered":"2024-01-24T17:41:32.903525+0000","last_unstale":"2024-01-24T17:41:33.377512+0000","last_undegraded":"2024-01-24T17:41:33.377512+0000","last_fullsized":"2024-01-24T17:41:33.377512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:46.796323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382677+0000","last_change":"2024-01-24T17:41:33.400221+0000","last_active":"2024-01-24T17:41:34.382677+0000","last_peered":"2024-01-24T17:41:34.382677+0000","last_clean":"2024-01-24T17:41:34.382677+0000","last_became_active":"2024-01-24T17:41:33.399980+0000","last_became_peered":"2024-01-24T17:41:33.399980+0000","last_unstale":"2024-01-24T17:41:34.382677+0000","last_undegraded":"2024-01-24T17:41:34.382677+0000","last_fullsized":"2024-01-24T17:41:34.382677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:54:11.431445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.367726+0000","last_change":"2024-01-24T17:41:32.907626+0000","last_active":"2024-01-24T17:41:33.367726+0000","last_peered":"2024-01-24T17:41:33.367726+0000","last_clean":"2024-01-24T17:41:33.367726+0000","last_became_active":"2024-01-24T17:41:32.907361+0000","last_became_peered":"2024-01-24T17:41:32.907361+0000","last_unstale":"2024-01-24T17:41:33.367726+0000","last_undegraded":"2024-01-24T17:41:33.367726+0000","last_fullsized":"2024-01-24T17:41:33.367726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386410+0000","last_change":"2024-01-24T17:41:33.392634+0000","last_active":"2024-01-24T17:41:34.386410+0000","last_peered":"2024-01-24T17:41:34.386410+0000","last_clean":"2024-01-24T17:41:34.386410+0000","last_became_active":"2024-01-24T17:41:33.392428+0000","last_became_peered":"2024-01-24T17:41:33.392428+0000","last_unstale":"2024-01-24T17:41:34.386410+0000","last_undegraded":"2024-01-24T17:41:34.386410+0000","last_fullsized":"2024-01-24T17:41:34.386410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.409695+0000","last_change":"2024-01-24T17:41:32.902055+0000","last_active":"2024-01-24T17:41:33.409695+0000","last_peered":"2024-01-24T17:41:33.409695+0000","last_clean":"2024-01-24T17:41:33.409695+0000","last_became_active":"2024-01-24T17:41:32.901738+0000","last_became_peered":"2024-01-24T17:41:32.901738+0000","last_unstale":"2024-01-24T17:41:33.409695+0000","last_undegraded":"2024-01-24T17:41:33.409695+0000","last_fullsized":"2024-01-24T17:41:33.409695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:45.778394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381845+0000","last_change":"2024-01-24T17:41:33.388186+0000","last_active":"2024-01-24T17:41:34.381845+0000","last_peered":"2024-01-24T17:41:34.381845+0000","last_clean":"2024-01-24T17:41:34.381845+0000","last_became_active":"2024-01-24T17:41:33.387930+0000","last_became_peered":"2024-01-24T17:41:33.387930+0000","last_unstale":"2024-01-24T17:41:34.381845+0000","last_undegraded":"2024-01-24T17:41:34.381845+0000","last_fullsized":"2024-01-24T17:41:34.381845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:19:54.318762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402116+0000","last_change":"2024-01-24T17:41:32.903388+0000","last_active":"2024-01-24T17:41:33.402116+0000","last_peered":"2024-01-24T17:41:33.402116+0000","last_clean":"2024-01-24T17:41:33.402116+0000","last_became_active":"2024-01-24T17:41:32.900342+0000","last_became_peered":"2024-01-24T17:41:32.900342+0000","last_unstale":"2024-01-24T17:41:33.402116+0000","last_undegraded":"2024-01-24T17:41:33.402116+0000","last_fullsized":"2024-01-24T17:41:33.402116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386143+0000","last_change":"2024-01-24T17:41:33.379754+0000","last_active":"2024-01-24T17:41:34.386143+0000","last_peered":"2024-01-24T17:41:34.386143+0000","last_clean":"2024-01-24T17:41:34.386143+0000","last_became_active":"2024-01-24T17:41:33.379454+0000","last_became_peered":"2024-01-24T17:41:33.379454+0000","last_unstale":"2024-01-24T17:41:34.386143+0000","last_undegraded":"2024-01-24T17:41:34.386143+0000","last_fullsized":"2024-01-24T17:41:34.386143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:46.793343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.408082+0000","last_change":"2024-01-24T17:41:32.904632+0000","last_active":"2024-01-24T17:41:33.408082+0000","last_peered":"2024-01-24T17:41:33.408082+0000","last_clean":"2024-01-24T17:41:33.408082+0000","last_became_active":"2024-01-24T17:41:32.904220+0000","last_became_peered":"2024-01-24T17:41:32.904220+0000","last_unstale":"2024-01-24T17:41:33.408082+0000","last_undegraded":"2024-01-24T17:41:33.408082+0000","last_fullsized":"2024-01-24T17:41:33.408082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:53.093743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382977+0000","last_change":"2024-01-24T17:41:33.395831+0000","last_active":"2024-01-24T17:41:34.382977+0000","last_peered":"2024-01-24T17:41:34.382977+0000","last_clean":"2024-01-24T17:41:34.382977+0000","last_became_active":"2024-01-24T17:41:33.392642+0000","last_became_peered":"2024-01-24T17:41:33.392642+0000","last_unstale":"2024-01-24T17:41:34.382977+0000","last_undegraded":"2024-01-24T17:41:34.382977+0000","last_fullsized":"2024-01-24T17:41:34.382977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:20:35.949143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.406390+0000","last_change":"2024-01-24T17:41:32.898336+0000","last_active":"2024-01-24T17:41:33.406390+0000","last_peered":"2024-01-24T17:41:33.406390+0000","last_clean":"2024-01-24T17:41:33.406390+0000","last_became_active":"2024-01-24T17:41:32.897951+0000","last_became_peered":"2024-01-24T17:41:32.897951+0000","last_unstale":"2024-01-24T17:41:33.406390+0000","last_undegraded":"2024-01-24T17:41:33.406390+0000","last_fullsized":"2024-01-24T17:41:33.406390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383662+0000","last_change":"2024-01-24T17:41:33.388321+0000","last_active":"2024-01-24T17:41:34.383662+0000","last_peered":"2024-01-24T17:41:34.383662+0000","last_clean":"2024-01-24T17:41:34.383662+0000","last_became_active":"2024-01-24T17:41:33.386908+0000","last_became_peered":"2024-01-24T17:41:33.386908+0000","last_unstale":"2024-01-24T17:41:34.383662+0000","last_undegraded":"2024-01-24T17:41:34.383662+0000","last_fullsized":"2024-01-24T17:41:34.383662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:14.826051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.376101+0000","last_change":"2024-01-24T17:41:32.898929+0000","last_active":"2024-01-24T17:41:33.376101+0000","last_peered":"2024-01-24T17:41:33.376101+0000","last_clean":"2024-01-24T17:41:33.376101+0000","last_became_active":"2024-01-24T17:41:32.898648+0000","last_became_peered":"2024-01-24T17:41:32.898648+0000","last_unstale":"2024-01-24T17:41:33.376101+0000","last_undegraded":"2024-01-24T17:41:33.376101+0000","last_fullsized":"2024-01-24T17:41:33.376101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:20.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381588+0000","last_change":"2024-01-24T17:41:33.394760+0000","last_active":"2024-01-24T17:41:34.381588+0000","last_peered":"2024-01-24T17:41:34.381588+0000","last_clean":"2024-01-24T17:41:34.381588+0000","last_became_active":"2024-01-24T17:41:33.394503+0000","last_became_peered":"2024-01-24T17:41:33.394503+0000","last_unstale":"2024-01-24T17:41:34.381588+0000","last_undegraded":"2024-01-24T17:41:34.381588+0000","last_fullsized":"2024-01-24T17:41:34.381588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:36:20.611114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.382896+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:41:33.382896+0000","last_peered":"2024-01-24T17:41:33.382896+0000","last_clean":"2024-01-24T17:41:33.382896+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:41:33.382896+0000","last_undegraded":"2024-01-24T17:41:33.382896+0000","last_fullsized":"2024-01-24T17:41:33.382896+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.378136+0000","last_change":"2024-01-24T17:41:32.906502+0000","last_active":"2024-01-24T17:41:33.378136+0000","last_peered":"2024-01-24T17:41:33.378136+0000","last_clean":"2024-01-24T17:41:33.378136+0000","last_became_active":"2024-01-24T17:41:32.905833+0000","last_became_peered":"2024-01-24T17:41:32.905833+0000","last_unstale":"2024-01-24T17:41:33.378136+0000","last_undegraded":"2024-01-24T17:41:33.378136+0000","last_fullsized":"2024-01-24T17:41:33.378136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:16.636377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374358+0000","last_change":"2024-01-24T17:41:33.400209+0000","last_active":"2024-01-24T17:41:34.374358+0000","last_peered":"2024-01-24T17:41:34.374358+0000","last_clean":"2024-01-24T17:41:34.374358+0000","last_became_active":"2024-01-24T17:41:33.399935+0000","last_became_peered":"2024-01-24T17:41:33.399935+0000","last_unstale":"2024-01-24T17:41:34.374358+0000","last_undegraded":"2024-01-24T17:41:34.374358+0000","last_fullsized":"2024-01-24T17:41:34.374358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:02.263592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377275+0000","last_change":"2024-01-24T17:41:32.903275+0000","last_active":"2024-01-24T17:41:33.377275+0000","last_peered":"2024-01-24T17:41:33.377275+0000","last_clean":"2024-01-24T17:41:33.377275+0000","last_became_active":"2024-01-24T17:41:32.902650+0000","last_became_peered":"2024-01-24T17:41:32.902650+0000","last_unstale":"2024-01-24T17:41:33.377275+0000","last_undegraded":"2024-01-24T17:41:33.377275+0000","last_fullsized":"2024-01-24T17:41:33.377275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.370807+0000","last_change":"2024-01-24T17:41:33.385359+0000","last_active":"2024-01-24T17:41:34.370807+0000","last_peered":"2024-01-24T17:41:34.370807+0000","last_clean":"2024-01-24T17:41:34.370807+0000","last_became_active":"2024-01-24T17:41:33.385048+0000","last_became_peered":"2024-01-24T17:41:33.385048+0000","last_unstale":"2024-01-24T17:41:34.370807+0000","last_undegraded":"2024-01-24T17:41:34.370807+0000","last_fullsized":"2024-01-24T17:41:34.370807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:17.651326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.381252+0000","last_change":"2024-01-24T17:41:32.899789+0000","last_active":"2024-01-24T17:41:33.381252+0000","last_peered":"2024-01-24T17:41:33.381252+0000","last_clean":"2024-01-24T17:41:33.381252+0000","last_became_active":"2024-01-24T17:41:32.899458+0000","last_became_peered":"2024-01-24T17:41:32.899458+0000","last_unstale":"2024-01-24T17:41:33.381252+0000","last_undegraded":"2024-01-24T17:41:33.381252+0000","last_fullsized":"2024-01-24T17:41:33.381252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374273+0000","last_change":"2024-01-24T17:41:33.385653+0000","last_active":"2024-01-24T17:41:34.374273+0000","last_peered":"2024-01-24T17:41:34.374273+0000","last_clean":"2024-01-24T17:41:34.374273+0000","last_became_active":"2024-01-24T17:41:33.385180+0000","last_became_peered":"2024-01-24T17:41:33.385180+0000","last_unstale":"2024-01-24T17:41:34.374273+0000","last_undegraded":"2024-01-24T17:41:34.374273+0000","last_fullsized":"2024-01-24T17:41:34.374273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:01:35.279502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377607+0000","last_change":"2024-01-24T17:41:32.903904+0000","last_active":"2024-01-24T17:41:33.377607+0000","last_peered":"2024-01-24T17:41:33.377607+0000","last_clean":"2024-01-24T17:41:33.377607+0000","last_became_active":"2024-01-24T17:41:32.902915+0000","last_became_peered":"2024-01-24T17:41:32.902915+0000","last_unstale":"2024-01-24T17:41:33.377607+0000","last_undegraded":"2024-01-24T17:41:33.377607+0000","last_fullsized":"2024-01-24T17:41:33.377607+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:51.152032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382323+0000","last_change":"2024-01-24T17:41:33.379050+0000","last_active":"2024-01-24T17:41:34.382323+0000","last_peered":"2024-01-24T17:41:34.382323+0000","last_clean":"2024-01-24T17:41:34.382323+0000","last_became_active":"2024-01-24T17:41:33.378782+0000","last_became_peered":"2024-01-24T17:41:33.378782+0000","last_unstale":"2024-01-24T17:41:34.382323+0000","last_undegraded":"2024-01-24T17:41:34.382323+0000","last_fullsized":"2024-01-24T17:41:34.382323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377892+0000","last_change":"2024-01-24T17:41:32.895887+0000","last_active":"2024-01-24T17:41:33.377892+0000","last_peered":"2024-01-24T17:41:33.377892+0000","last_clean":"2024-01-24T17:41:33.377892+0000","last_became_active":"2024-01-24T17:41:32.895460+0000","last_became_peered":"2024-01-24T17:41:32.895460+0000","last_unstale":"2024-01-24T17:41:33.377892+0000","last_undegraded":"2024-01-24T17:41:33.377892+0000","last_fullsized":"2024-01-24T17:41:33.377892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.385945+0000","last_change":"2024-01-24T17:41:33.381026+0000","last_active":"2024-01-24T17:41:34.385945+0000","last_peered":"2024-01-24T17:41:34.385945+0000","last_clean":"2024-01-24T17:41:34.385945+0000","last_became_active":"2024-01-24T17:41:33.380649+0000","last_became_peered":"2024-01-24T17:41:33.380649+0000","last_unstale":"2024-01-24T17:41:34.385945+0000","last_undegraded":"2024-01-24T17:41:34.385945+0000","last_fullsized":"2024-01-24T17:41:34.385945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.378103+0000","last_change":"2024-01-24T17:41:32.907390+0000","last_active":"2024-01-24T17:41:33.378103+0000","last_peered":"2024-01-24T17:41:33.378103+0000","last_clean":"2024-01-24T17:41:33.378103+0000","last_became_active":"2024-01-24T17:41:32.906724+0000","last_became_peered":"2024-01-24T17:41:32.906724+0000","last_unstale":"2024-01-24T17:41:33.378103+0000","last_undegraded":"2024-01-24T17:41:33.378103+0000","last_fullsized":"2024-01-24T17:41:33.378103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:45.778394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373347+0000","last_change":"2024-01-24T17:41:33.390087+0000","last_active":"2024-01-24T17:41:34.373347+0000","last_peered":"2024-01-24T17:41:34.373347+0000","last_clean":"2024-01-24T17:41:34.373347+0000","last_became_active":"2024-01-24T17:41:33.389820+0000","last_became_peered":"2024-01-24T17:41:33.389820+0000","last_unstale":"2024-01-24T17:41:34.373347+0000","last_undegraded":"2024-01-24T17:41:34.373347+0000","last_fullsized":"2024-01-24T17:41:34.373347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373963+0000","last_change":"2024-01-24T17:41:32.902595+0000","last_active":"2024-01-24T17:41:34.373963+0000","last_peered":"2024-01-24T17:41:34.373963+0000","last_clean":"2024-01-24T17:41:34.373963+0000","last_became_active":"2024-01-24T17:41:32.902291+0000","last_became_peered":"2024-01-24T17:41:32.902291+0000","last_unstale":"2024-01-24T17:41:34.373963+0000","last_undegraded":"2024-01-24T17:41:34.373963+0000","last_fullsized":"2024-01-24T17:41:34.373963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374204+0000","last_change":"2024-01-24T17:41:33.401908+0000","last_active":"2024-01-24T17:41:34.374204+0000","last_peered":"2024-01-24T17:41:34.374204+0000","last_clean":"2024-01-24T17:41:34.374204+0000","last_became_active":"2024-01-24T17:41:33.401634+0000","last_became_peered":"2024-01-24T17:41:33.401634+0000","last_unstale":"2024-01-24T17:41:34.374204+0000","last_undegraded":"2024-01-24T17:41:34.374204+0000","last_fullsized":"2024-01-24T17:41:34.374204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.405956+0000","last_change":"2024-01-24T17:41:32.909941+0000","last_active":"2024-01-24T17:41:33.405956+0000","last_peered":"2024-01-24T17:41:33.405956+0000","last_clean":"2024-01-24T17:41:33.405956+0000","last_became_active":"2024-01-24T17:41:32.909592+0000","last_became_peered":"2024-01-24T17:41:32.909592+0000","last_unstale":"2024-01-24T17:41:33.405956+0000","last_undegraded":"2024-01-24T17:41:33.405956+0000","last_fullsized":"2024-01-24T17:41:33.405956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373574+0000","last_change":"2024-01-24T17:41:33.394464+0000","last_active":"2024-01-24T17:41:34.373574+0000","last_peered":"2024-01-24T17:41:34.373574+0000","last_clean":"2024-01-24T17:41:34.373574+0000","last_became_active":"2024-01-24T17:41:33.393898+0000","last_became_peered":"2024-01-24T17:41:33.393898+0000","last_unstale":"2024-01-24T17:41:34.373574+0000","last_undegraded":"2024-01-24T17:41:34.373574+0000","last_fullsized":"2024-01-24T17:41:34.373574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:01:35.279502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.366360+0000","last_change":"2024-01-24T17:41:32.896815+0000","last_active":"2024-01-24T17:41:33.366360+0000","last_peered":"2024-01-24T17:41:33.366360+0000","last_clean":"2024-01-24T17:41:33.366360+0000","last_became_active":"2024-01-24T17:41:32.896263+0000","last_became_peered":"2024-01-24T17:41:32.896263+0000","last_unstale":"2024-01-24T17:41:33.366360+0000","last_undegraded":"2024-01-24T17:41:33.366360+0000","last_fullsized":"2024-01-24T17:41:33.366360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:53.093743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382082+0000","last_change":"2024-01-24T17:41:33.393488+0000","last_active":"2024-01-24T17:41:34.382082+0000","last_peered":"2024-01-24T17:41:34.382082+0000","last_clean":"2024-01-24T17:41:34.382082+0000","last_became_active":"2024-01-24T17:41:33.391266+0000","last_became_peered":"2024-01-24T17:41:33.391266+0000","last_unstale":"2024-01-24T17:41:34.382082+0000","last_undegraded":"2024-01-24T17:41:34.382082+0000","last_fullsized":"2024-01-24T17:41:34.382082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:08:31.383389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.387230+0000","last_change":"2024-01-24T17:41:32.902635+0000","last_active":"2024-01-24T17:41:33.387230+0000","last_peered":"2024-01-24T17:41:33.387230+0000","last_clean":"2024-01-24T17:41:33.387230+0000","last_became_active":"2024-01-24T17:41:32.902369+0000","last_became_peered":"2024-01-24T17:41:32.902369+0000","last_unstale":"2024-01-24T17:41:33.387230+0000","last_undegraded":"2024-01-24T17:41:33.387230+0000","last_fullsized":"2024-01-24T17:41:33.387230+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372780+0000","last_change":"2024-01-24T17:41:33.390610+0000","last_active":"2024-01-24T17:41:34.372780+0000","last_peered":"2024-01-24T17:41:34.372780+0000","last_clean":"2024-01-24T17:41:34.372780+0000","last_became_active":"2024-01-24T17:41:33.390337+0000","last_became_peered":"2024-01-24T17:41:33.390337+0000","last_unstale":"2024-01-24T17:41:34.372780+0000","last_undegraded":"2024-01-24T17:41:34.372780+0000","last_fullsized":"2024-01-24T17:41:34.372780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.374492+0000","last_change":"2024-01-24T17:41:32.891585+0000","last_active":"2024-01-24T17:41:33.374492+0000","last_peered":"2024-01-24T17:41:33.374492+0000","last_clean":"2024-01-24T17:41:33.374492+0000","last_became_active":"2024-01-24T17:41:32.891376+0000","last_became_peered":"2024-01-24T17:41:32.891376+0000","last_unstale":"2024-01-24T17:41:33.374492+0000","last_undegraded":"2024-01-24T17:41:33.374492+0000","last_fullsized":"2024-01-24T17:41:33.374492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381998+0000","last_change":"2024-01-24T17:41:33.397462+0000","last_active":"2024-01-24T17:41:34.381998+0000","last_peered":"2024-01-24T17:41:34.381998+0000","last_clean":"2024-01-24T17:41:34.381998+0000","last_became_active":"2024-01-24T17:41:33.397113+0000","last_became_peered":"2024-01-24T17:41:33.397113+0000","last_unstale":"2024-01-24T17:41:34.381998+0000","last_undegraded":"2024-01-24T17:41:34.381998+0000","last_fullsized":"2024-01-24T17:41:34.381998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.397708+0000","last_change":"2024-01-24T17:41:32.904150+0000","last_active":"2024-01-24T17:41:33.397708+0000","last_peered":"2024-01-24T17:41:33.397708+0000","last_clean":"2024-01-24T17:41:33.397708+0000","last_became_active":"2024-01-24T17:41:32.902550+0000","last_became_peered":"2024-01-24T17:41:32.902550+0000","last_unstale":"2024-01-24T17:41:33.397708+0000","last_undegraded":"2024-01-24T17:41:33.397708+0000","last_fullsized":"2024-01-24T17:41:33.397708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:26.459376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381261+0000","last_change":"2024-01-24T17:41:33.399418+0000","last_active":"2024-01-24T17:41:34.381261+0000","last_peered":"2024-01-24T17:41:34.381261+0000","last_clean":"2024-01-24T17:41:34.381261+0000","last_became_active":"2024-01-24T17:41:33.395852+0000","last_became_peered":"2024-01-24T17:41:33.395852+0000","last_unstale":"2024-01-24T17:41:34.381261+0000","last_undegraded":"2024-01-24T17:41:34.381261+0000","last_fullsized":"2024-01-24T17:41:34.381261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381377+0000","last_change":"2024-01-24T17:41:33.401099+0000","last_active":"2024-01-24T17:41:34.381377+0000","last_peered":"2024-01-24T17:41:34.381377+0000","last_clean":"2024-01-24T17:41:34.381377+0000","last_became_active":"2024-01-24T17:41:33.399957+0000","last_became_peered":"2024-01-24T17:41:33.399957+0000","last_unstale":"2024-01-24T17:41:34.381377+0000","last_undegraded":"2024-01-24T17:41:34.381377+0000","last_fullsized":"2024-01-24T17:41:34.381377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:12:52.644481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404497+0000","last_change":"2024-01-24T17:41:32.900545+0000","last_active":"2024-01-24T17:41:33.404497+0000","last_peered":"2024-01-24T17:41:33.404497+0000","last_clean":"2024-01-24T17:41:33.404497+0000","last_became_active":"2024-01-24T17:41:32.900228+0000","last_became_peered":"2024-01-24T17:41:32.900228+0000","last_unstale":"2024-01-24T17:41:33.404497+0000","last_undegraded":"2024-01-24T17:41:33.404497+0000","last_fullsized":"2024-01-24T17:41:33.404497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386811+0000","last_change":"2024-01-24T17:41:33.401985+0000","last_active":"2024-01-24T17:41:34.386811+0000","last_peered":"2024-01-24T17:41:34.386811+0000","last_clean":"2024-01-24T17:41:34.386811+0000","last_became_active":"2024-01-24T17:41:33.401623+0000","last_became_peered":"2024-01-24T17:41:33.401623+0000","last_unstale":"2024-01-24T17:41:34.386811+0000","last_undegraded":"2024-01-24T17:41:34.386811+0000","last_fullsized":"2024-01-24T17:41:34.386811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402556+0000","last_change":"2024-01-24T17:41:32.899336+0000","last_active":"2024-01-24T17:41:33.402556+0000","last_peered":"2024-01-24T17:41:33.402556+0000","last_clean":"2024-01-24T17:41:33.402556+0000","last_became_active":"2024-01-24T17:41:32.898773+0000","last_became_peered":"2024-01-24T17:41:32.898773+0000","last_unstale":"2024-01-24T17:41:33.402556+0000","last_undegraded":"2024-01-24T17:41:33.402556+0000","last_fullsized":"2024-01-24T17:41:33.402556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376757+0000","last_change":"2024-01-24T17:41:33.396094+0000","last_active":"2024-01-24T17:41:34.376757+0000","last_peered":"2024-01-24T17:41:34.376757+0000","last_clean":"2024-01-24T17:41:34.376757+0000","last_became_active":"2024-01-24T17:41:33.395819+0000","last_became_peered":"2024-01-24T17:41:33.395819+0000","last_unstale":"2024-01-24T17:41:34.376757+0000","last_undegraded":"2024-01-24T17:41:34.376757+0000","last_fullsized":"2024-01-24T17:41:34.376757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404773+0000","last_change":"2024-01-24T17:41:32.890539+0000","last_active":"2024-01-24T17:41:33.404773+0000","last_peered":"2024-01-24T17:41:33.404773+0000","last_clean":"2024-01-24T17:41:33.404773+0000","last_became_active":"2024-01-24T17:41:32.890175+0000","last_became_peered":"2024-01-24T17:41:32.890175+0000","last_unstale":"2024-01-24T17:41:33.404773+0000","last_undegraded":"2024-01-24T17:41:33.404773+0000","last_fullsized":"2024-01-24T17:41:33.404773+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376495+0000","last_change":"2024-01-24T17:41:33.374778+0000","last_active":"2024-01-24T17:41:34.376495+0000","last_peered":"2024-01-24T17:41:34.376495+0000","last_clean":"2024-01-24T17:41:34.376495+0000","last_became_active":"2024-01-24T17:41:33.374507+0000","last_became_peered":"2024-01-24T17:41:33.374507+0000","last_unstale":"2024-01-24T17:41:34.376495+0000","last_undegraded":"2024-01-24T17:41:34.376495+0000","last_fullsized":"2024-01-24T17:41:34.376495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.410169+0000","last_change":"2024-01-24T17:41:32.909855+0000","last_active":"2024-01-24T17:41:33.410169+0000","last_peered":"2024-01-24T17:41:33.410169+0000","last_clean":"2024-01-24T17:41:33.410169+0000","last_became_active":"2024-01-24T17:41:32.909553+0000","last_became_peered":"2024-01-24T17:41:32.909553+0000","last_unstale":"2024-01-24T17:41:33.410169+0000","last_undegraded":"2024-01-24T17:41:33.410169+0000","last_fullsized":"2024-01-24T17:41:33.410169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":661358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.694}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":73134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":208896,"data_stored":71480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.792}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1589999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0110000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8540000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":802816,"data_stored":661336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":212992,"data_stored":70954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:41:43.478 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:41:43.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.734+0000 7f9c1c400640 1 Processor -- start 2024-01-24T17:41:43.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.734+0000 7f9c1c400640 1 -- start start 2024-01-24T17:41:43.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.735+0000 7f9c1c400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9c1415a950 0x7f9c140ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.735+0000 7f9c1c400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140c9180 con 0x7f9c140caaa0 2024-01-24T17:41:43.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.735+0000 7f9c1c400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140c9460 con 0x7f9c140cb660 2024-01-24T17:41:43.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.735+0000 7f9c1c400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c140b43d0 con 0x7f9c1415a950 2024-01-24T17:41:43.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.735+0000 7f9c19000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9c140caaa0 0x7f9c140caf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54370/0 (socket says 172.21.15.39:54370) 2024-01-24T17:41:43.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.735+0000 7f9c19000640 1 -- 172.21.15.39:0/2888778030 learned_addr learned my addr 172.21.15.39:0/2888778030 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:43.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869972661 0 0) 0x7f9c140c9180 con 0x7f9c140caaa0 2024-01-24T17:41:43.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4003520 con 0x7f9c140caaa0 2024-01-24T17:41:43.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c19a00640 1 --2- 172.21.15.39:0/2888778030 >> v2:172.21.15.100:3300/0 conn(0x7f9c1415a950 0x7f9c140ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:43.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 687990956 0 0) 0x7f9bf4003520 con 0x7f9c140caaa0 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 >> v2:172.21.15.100:3300/0 conn(0x7f9c1415a950 msgr2=0x7f9c140ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 --2- 172.21.15.39:0/2888778030 >> v2:172.21.15.100:3300/0 conn(0x7f9c1415a950 0x7f9c140ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 >> v1:172.21.15.100:6789/0 conn(0x7f9c140cb660 legacy=0x7f9c140b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140c9740 con 0x7f9c140caaa0 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f9bfc001670 con 0x7f9c140caaa0 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 >> v1:172.21.15.39:6789/0 conn(0x7f9c140caaa0 legacy=0x7f9c140caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bf4003970 0x7f9bf4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf40047c0 0x7f9bf4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bf4005180 0x7f9bf400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140b43d0 con 0x7f9bf40047c0 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf400c020 con 0x7f9bf4003970 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.736+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf400c1e0 con 0x7f9bf4005180 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.737+0000 7f9c19000640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf40047c0 0x7f9bf4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:43.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.737+0000 7f9c19000640 1 -- 172.21.15.39:0/2888778030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bf4005180 msgr2=0x7f9bf400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:43.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.737+0000 7f9c19000640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bf4005180 0x7f9bf400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.737+0000 7f9c19000640 1 -- 172.21.15.39:0/2888778030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bf4003970 msgr2=0x7f9bf4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:43.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.737+0000 7f9c19000640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bf4003970 0x7f9bf4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.737+0000 7f9c19000640 1 -- 172.21.15.39:0/2888778030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4003520 con 0x7f9bf40047c0 2024-01-24T17:41:43.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.737+0000 7f9c1a400640 1 -- 172.21.15.39:0/2888778030 reap_dead start 2024-01-24T17:41:43.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.738+0000 7f9c19000640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf40047c0 0x7f9bf4004c40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00c240 tx=0x7f9bf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe43255bdb8ab78a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:43.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.738+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc005ad0 con 0x7f9bf40047c0 2024-01-24T17:41:43.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.738+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc005c30 con 0x7f9bf40047c0 2024-01-24T17:41:43.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.738+0000 7f9c13e00640 1 -- 172.21.15.39:0/2888778030 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc02caf0 con 0x7f9bf40047c0 2024-01-24T17:41:43.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.739+0000 7f9c1c400640 1 -- 172.21.15.39:0/2888778030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf40047c0 msgr2=0x7f9bf4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.739+0000 7f9c1c400640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf40047c0 0x7f9bf4004c40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00c240 tx=0x7f9bf40057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.739+0000 7f9c1c400640 1 -- 172.21.15.39:0/2888778030 shutdown_connections 2024-01-24T17:41:43.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.739+0000 7f9c1c400640 1 --2- 172.21.15.39:0/2888778030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bf40047c0 0x7f9bf4004c40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.739+0000 7f9c1c400640 1 -- 172.21.15.39:0/2888778030 >> 172.21.15.39:0/2888778030 conn(0x7f9c1406a3e0 msgr2=0x7f9c1406bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:43.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.739+0000 7f9c1c400640 1 -- 172.21.15.39:0/2888778030 shutdown_connections 2024-01-24T17:41:43.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.739+0000 7f9c1c400640 1 -- 172.21.15.39:0/2888778030 wait complete. 2024-01-24T17:41:43.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.741+0000 7f9c1c400640 1 Processor -- start 2024-01-24T17:41:43.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.741+0000 7f9c1c400640 1 -- start start 2024-01-24T17:41:43.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c1c400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bf40047c0 0x7f9c1415c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c1c400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c14154940 0x7f9c1415d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c1c400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c14154da0 0x7f9c1415d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c1c400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14152820 con 0x7f9c14154940 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c1c400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14152b00 con 0x7f9bf40047c0 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c1c400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14152de0 con 0x7f9c14154da0 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c19000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c14154940 0x7f9c1415d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c19000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c14154940 0x7f9c1415d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58094/0 (socket says 172.21.15.39:58094) 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c19000640 1 -- 172.21.15.39:0/1233625059 learned_addr learned my addr 172.21.15.39:0/1233625059 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c19000640 1 -- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c14154da0 msgr2=0x7f9c1415d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c19000640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c14154da0 0x7f9c1415d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c19000640 1 -- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bf40047c0 msgr2=0x7f9c1415c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c19000640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bf40047c0 0x7f9c1415c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.742+0000 7f9c19000640 1 -- 172.21.15.39:0/1233625059 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c141530c0 con 0x7f9c14154940 2024-01-24T17:41:43.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.743+0000 7f9c19000640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c14154940 0x7f9c1415d420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc001930 tx=0x7f9bfc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:43.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.743+0000 7f9c11600640 1 -- 172.21.15.39:0/1233625059 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc03c040 con 0x7f9c14154940 2024-01-24T17:41:43.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.743+0000 7f9c11600640 1 -- 172.21.15.39:0/1233625059 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc02cd80 con 0x7f9c14154940 2024-01-24T17:41:43.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.743+0000 7f9c11600640 1 -- 172.21.15.39:0/1233625059 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc0072b0 con 0x7f9c14154940 2024-01-24T17:41:43.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.743+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c141533a0 con 0x7f9c14154940 2024-01-24T17:41:43.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.743+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c14154360 con 0x7f9c14154940 2024-01-24T17:41:43.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.744+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c14153530 con 0x7f9c14154940 2024-01-24T17:41:43.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.745+0000 7f9c11600640 1 -- 172.21.15.39:0/1233625059 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9bfc032070 con 0x7f9c14154940 2024-01-24T17:41:43.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.746+0000 7f9c11600640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bec06a460 0x7f9bec06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:43.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.746+0000 7f9c11600640 1 -- 172.21.15.39:0/1233625059 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bfc0a9f50 con 0x7f9c14154940 2024-01-24T17:41:43.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.746+0000 7f9c19a00640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bec06a460 0x7f9bec06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:43.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.747+0000 7f9c19a00640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bec06a460 0x7f9bec06c920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c04006a90 tx=0x7f9c04006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:43.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.750+0000 7f9c11600640 1 -- 172.21.15.39:0/1233625059 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9bfc0754a0 con 0x7f9c14154940 2024-01-24T17:41:43.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.937+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c14152c70 con 0x7f9bec06a460 2024-01-24T17:41:43.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.957+0000 7f9c11600640 1 -- 172.21.15.39:0/1233625059 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264436 (secure 0 0 0) 0x7f9c14152c70 con 0x7f9bec06a460 2024-01-24T17:41:43.961 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:43.963 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:41:43.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bec06a460 msgr2=0x7f9bec06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bec06a460 0x7f9bec06c920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9c04006a90 tx=0x7f9c04006a20 comp rx=0 tx=0).stop 2024-01-24T17:41:43.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c14154940 msgr2=0x7f9c1415d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c14154940 0x7f9c1415d420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc001930 tx=0x7f9bfc02e070 comp rx=0 tx=0).stop 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 shutdown_connections 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9c14154da0 0x7f9c1415d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bec06a460 0x7f9bec06c920 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c14154940 0x7f9c1415d420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 --2- 172.21.15.39:0/1233625059 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bf40047c0 0x7f9c1415c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 >> 172.21.15.39:0/1233625059 conn(0x7f9c1406a3e0 msgr2=0x7f9c140caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 shutdown_connections 2024-01-24T17:41:43.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:43.964+0000 7f9c1c400640 1 -- 172.21.15.39:0/1233625059 wait complete. 2024-01-24T17:41:43.982 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-24T17:41:42.211403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235372,"kb_used_data":2824,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754739348,"statfs":{"total":773094113280,"available":772853092352,"internally_reserved":0,"allocated":2891776,"data_stored":1752644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.847782"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382873+0000","last_change":"2024-01-24T17:41:33.393186+0000","last_active":"2024-01-24T17:41:34.382873+0000","last_peered":"2024-01-24T17:41:34.382873+0000","last_clean":"2024-01-24T17:41:34.382873+0000","last_became_active":"2024-01-24T17:41:33.391034+0000","last_became_peered":"2024-01-24T17:41:33.391034+0000","last_unstale":"2024-01-24T17:41:34.382873+0000","last_undegraded":"2024-01-24T17:41:34.382873+0000","last_fullsized":"2024-01-24T17:41:34.382873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376217+0000","last_change":"2024-01-24T17:41:33.384027+0000","last_active":"2024-01-24T17:41:34.376217+0000","last_peered":"2024-01-24T17:41:34.376217+0000","last_clean":"2024-01-24T17:41:34.376217+0000","last_became_active":"2024-01-24T17:41:33.383720+0000","last_became_peered":"2024-01-24T17:41:33.383720+0000","last_unstale":"2024-01-24T17:41:34.376217+0000","last_undegraded":"2024-01-24T17:41:34.376217+0000","last_fullsized":"2024-01-24T17:41:34.376217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376005+0000","last_change":"2024-01-24T17:41:33.392448+0000","last_active":"2024-01-24T17:41:34.376005+0000","last_peered":"2024-01-24T17:41:34.376005+0000","last_clean":"2024-01-24T17:41:34.376005+0000","last_became_active":"2024-01-24T17:41:33.392125+0000","last_became_peered":"2024-01-24T17:41:33.392125+0000","last_unstale":"2024-01-24T17:41:34.376005+0000","last_undegraded":"2024-01-24T17:41:34.376005+0000","last_fullsized":"2024-01-24T17:41:34.376005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:20:35.949143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.385632+0000","last_change":"2024-01-24T17:41:33.395630+0000","last_active":"2024-01-24T17:41:34.385632+0000","last_peered":"2024-01-24T17:41:34.385632+0000","last_clean":"2024-01-24T17:41:34.385632+0000","last_became_active":"2024-01-24T17:41:33.395378+0000","last_became_peered":"2024-01-24T17:41:33.395378+0000","last_unstale":"2024-01-24T17:41:34.385632+0000","last_undegraded":"2024-01-24T17:41:34.385632+0000","last_fullsized":"2024-01-24T17:41:34.385632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382553+0000","last_change":"2024-01-24T17:41:33.382453+0000","last_active":"2024-01-24T17:41:34.382553+0000","last_peered":"2024-01-24T17:41:34.382553+0000","last_clean":"2024-01-24T17:41:34.382553+0000","last_became_active":"2024-01-24T17:41:33.382151+0000","last_became_peered":"2024-01-24T17:41:33.382151+0000","last_unstale":"2024-01-24T17:41:34.382553+0000","last_undegraded":"2024-01-24T17:41:34.382553+0000","last_fullsized":"2024-01-24T17:41:34.382553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.384184+0000","last_change":"2024-01-24T17:41:33.396138+0000","last_active":"2024-01-24T17:41:34.384184+0000","last_peered":"2024-01-24T17:41:34.384184+0000","last_clean":"2024-01-24T17:41:34.384184+0000","last_became_active":"2024-01-24T17:41:33.395890+0000","last_became_peered":"2024-01-24T17:41:33.395890+0000","last_unstale":"2024-01-24T17:41:34.384184+0000","last_undegraded":"2024-01-24T17:41:34.384184+0000","last_fullsized":"2024-01-24T17:41:34.384184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:17.651326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381377+0000","last_change":"2024-01-24T17:41:33.393003+0000","last_active":"2024-01-24T17:41:34.381377+0000","last_peered":"2024-01-24T17:41:34.381377+0000","last_clean":"2024-01-24T17:41:34.381377+0000","last_became_active":"2024-01-24T17:41:33.391259+0000","last_became_peered":"2024-01-24T17:41:33.391259+0000","last_unstale":"2024-01-24T17:41:34.381377+0000","last_undegraded":"2024-01-24T17:41:34.381377+0000","last_fullsized":"2024-01-24T17:41:34.381377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:50:21.701964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374081+0000","last_change":"2024-01-24T17:41:33.402016+0000","last_active":"2024-01-24T17:41:34.374081+0000","last_peered":"2024-01-24T17:41:34.374081+0000","last_clean":"2024-01-24T17:41:34.374081+0000","last_became_active":"2024-01-24T17:41:33.401767+0000","last_became_peered":"2024-01-24T17:41:33.401767+0000","last_unstale":"2024-01-24T17:41:34.374081+0000","last_undegraded":"2024-01-24T17:41:34.374081+0000","last_fullsized":"2024-01-24T17:41:34.374081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:35:09.121979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.387416+0000","last_change":"2024-01-24T17:41:33.391456+0000","last_active":"2024-01-24T17:41:34.387416+0000","last_peered":"2024-01-24T17:41:34.387416+0000","last_clean":"2024-01-24T17:41:34.387416+0000","last_became_active":"2024-01-24T17:41:33.391192+0000","last_became_peered":"2024-01-24T17:41:33.391192+0000","last_unstale":"2024-01-24T17:41:34.387416+0000","last_undegraded":"2024-01-24T17:41:34.387416+0000","last_fullsized":"2024-01-24T17:41:34.387416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382754+0000","last_change":"2024-01-24T17:41:33.380252+0000","last_active":"2024-01-24T17:41:34.382754+0000","last_peered":"2024-01-24T17:41:34.382754+0000","last_clean":"2024-01-24T17:41:34.382754+0000","last_became_active":"2024-01-24T17:41:33.378238+0000","last_became_peered":"2024-01-24T17:41:33.378238+0000","last_unstale":"2024-01-24T17:41:34.382754+0000","last_undegraded":"2024-01-24T17:41:34.382754+0000","last_fullsized":"2024-01-24T17:41:34.382754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:01:48.740461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382398+0000","last_change":"2024-01-24T17:41:33.382673+0000","last_active":"2024-01-24T17:41:34.382398+0000","last_peered":"2024-01-24T17:41:34.382398+0000","last_clean":"2024-01-24T17:41:34.382398+0000","last_became_active":"2024-01-24T17:41:33.382404+0000","last_became_peered":"2024-01-24T17:41:33.382404+0000","last_unstale":"2024-01-24T17:41:34.382398+0000","last_undegraded":"2024-01-24T17:41:34.382398+0000","last_fullsized":"2024-01-24T17:41:34.382398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:44:38.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383532+0000","last_change":"2024-01-24T17:41:33.395359+0000","last_active":"2024-01-24T17:41:34.383532+0000","last_peered":"2024-01-24T17:41:34.383532+0000","last_clean":"2024-01-24T17:41:34.383532+0000","last_became_active":"2024-01-24T17:41:33.395102+0000","last_became_peered":"2024-01-24T17:41:33.395102+0000","last_unstale":"2024-01-24T17:41:34.383532+0000","last_undegraded":"2024-01-24T17:41:34.383532+0000","last_fullsized":"2024-01-24T17:41:34.383532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372275+0000","last_change":"2024-01-24T17:41:33.391912+0000","last_active":"2024-01-24T17:41:34.372275+0000","last_peered":"2024-01-24T17:41:34.372275+0000","last_clean":"2024-01-24T17:41:34.372275+0000","last_became_active":"2024-01-24T17:41:33.391421+0000","last_became_peered":"2024-01-24T17:41:33.391421+0000","last_unstale":"2024-01-24T17:41:34.372275+0000","last_undegraded":"2024-01-24T17:41:34.372275+0000","last_fullsized":"2024-01-24T17:41:34.372275+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:36:51.994706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.371942+0000","last_change":"2024-01-24T17:41:33.381744+0000","last_active":"2024-01-24T17:41:34.371942+0000","last_peered":"2024-01-24T17:41:34.371942+0000","last_clean":"2024-01-24T17:41:34.371942+0000","last_became_active":"2024-01-24T17:41:33.381499+0000","last_became_peered":"2024-01-24T17:41:33.381499+0000","last_unstale":"2024-01-24T17:41:34.371942+0000","last_undegraded":"2024-01-24T17:41:34.371942+0000","last_fullsized":"2024-01-24T17:41:34.371942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:27.474325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373889+0000","last_change":"2024-01-24T17:41:33.382763+0000","last_active":"2024-01-24T17:41:34.373889+0000","last_peered":"2024-01-24T17:41:34.373889+0000","last_clean":"2024-01-24T17:41:34.373889+0000","last_became_active":"2024-01-24T17:41:33.382506+0000","last_became_peered":"2024-01-24T17:41:33.382506+0000","last_unstale":"2024-01-24T17:41:34.373889+0000","last_undegraded":"2024-01-24T17:41:34.373889+0000","last_fullsized":"2024-01-24T17:41:34.373889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:29.008097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383246+0000","last_change":"2024-01-24T17:41:33.398656+0000","last_active":"2024-01-24T17:41:34.383246+0000","last_peered":"2024-01-24T17:41:34.383246+0000","last_clean":"2024-01-24T17:41:34.383246+0000","last_became_active":"2024-01-24T17:41:33.398404+0000","last_became_peered":"2024-01-24T17:41:33.398404+0000","last_unstale":"2024-01-24T17:41:34.383246+0000","last_undegraded":"2024-01-24T17:41:34.383246+0000","last_fullsized":"2024-01-24T17:41:34.383246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:36:23.946397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383245+0000","last_change":"2024-01-24T17:41:33.401897+0000","last_active":"2024-01-24T17:41:34.383245+0000","last_peered":"2024-01-24T17:41:34.383245+0000","last_clean":"2024-01-24T17:41:34.383245+0000","last_became_active":"2024-01-24T17:41:33.401641+0000","last_became_peered":"2024-01-24T17:41:33.401641+0000","last_unstale":"2024-01-24T17:41:34.383245+0000","last_undegraded":"2024-01-24T17:41:34.383245+0000","last_fullsized":"2024-01-24T17:41:34.383245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:52:01.420886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382136+0000","last_change":"2024-01-24T17:41:33.397837+0000","last_active":"2024-01-24T17:41:34.382136+0000","last_peered":"2024-01-24T17:41:34.382136+0000","last_clean":"2024-01-24T17:41:34.382136+0000","last_became_active":"2024-01-24T17:41:33.397597+0000","last_became_peered":"2024-01-24T17:41:33.397597+0000","last_unstale":"2024-01-24T17:41:34.382136+0000","last_undegraded":"2024-01-24T17:41:34.382136+0000","last_fullsized":"2024-01-24T17:41:34.382136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381810+0000","last_change":"2024-01-24T17:41:33.389940+0000","last_active":"2024-01-24T17:41:34.381810+0000","last_peered":"2024-01-24T17:41:34.381810+0000","last_clean":"2024-01-24T17:41:34.381810+0000","last_became_active":"2024-01-24T17:41:33.389673+0000","last_became_peered":"2024-01-24T17:41:33.389673+0000","last_unstale":"2024-01-24T17:41:34.381810+0000","last_undegraded":"2024-01-24T17:41:34.381810+0000","last_fullsized":"2024-01-24T17:41:34.381810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:48:57.485742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383781+0000","last_change":"2024-01-24T17:41:33.394799+0000","last_active":"2024-01-24T17:41:34.383781+0000","last_peered":"2024-01-24T17:41:34.383781+0000","last_clean":"2024-01-24T17:41:34.383781+0000","last_became_active":"2024-01-24T17:41:33.394396+0000","last_became_peered":"2024-01-24T17:41:33.394396+0000","last_unstale":"2024-01-24T17:41:34.383781+0000","last_undegraded":"2024-01-24T17:41:34.383781+0000","last_fullsized":"2024-01-24T17:41:34.383781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:35:09.121979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372973+0000","last_change":"2024-01-24T17:41:33.402158+0000","last_active":"2024-01-24T17:41:34.372973+0000","last_peered":"2024-01-24T17:41:34.372973+0000","last_clean":"2024-01-24T17:41:34.372973+0000","last_became_active":"2024-01-24T17:41:33.400801+0000","last_became_peered":"2024-01-24T17:41:33.400801+0000","last_unstale":"2024-01-24T17:41:34.372973+0000","last_undegraded":"2024-01-24T17:41:34.372973+0000","last_fullsized":"2024-01-24T17:41:34.372973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:29.008097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386543+0000","last_change":"2024-01-24T17:41:33.399257+0000","last_active":"2024-01-24T17:41:34.386543+0000","last_peered":"2024-01-24T17:41:34.386543+0000","last_clean":"2024-01-24T17:41:34.386543+0000","last_became_active":"2024-01-24T17:41:33.398569+0000","last_became_peered":"2024-01-24T17:41:33.398569+0000","last_unstale":"2024-01-24T17:41:34.386543+0000","last_undegraded":"2024-01-24T17:41:34.386543+0000","last_fullsized":"2024-01-24T17:41:34.386543+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:54.108693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374158+0000","last_change":"2024-01-24T17:41:33.372320+0000","last_active":"2024-01-24T17:41:34.374158+0000","last_peered":"2024-01-24T17:41:34.374158+0000","last_clean":"2024-01-24T17:41:34.374158+0000","last_became_active":"2024-01-24T17:41:33.372080+0000","last_became_peered":"2024-01-24T17:41:33.372080+0000","last_unstale":"2024-01-24T17:41:34.374158+0000","last_undegraded":"2024-01-24T17:41:34.374158+0000","last_fullsized":"2024-01-24T17:41:34.374158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381086+0000","last_change":"2024-01-24T17:41:33.388373+0000","last_active":"2024-01-24T17:41:34.381086+0000","last_peered":"2024-01-24T17:41:34.381086+0000","last_clean":"2024-01-24T17:41:34.381086+0000","last_became_active":"2024-01-24T17:41:33.387185+0000","last_became_peered":"2024-01-24T17:41:33.387185+0000","last_unstale":"2024-01-24T17:41:34.381086+0000","last_undegraded":"2024-01-24T17:41:34.381086+0000","last_fullsized":"2024-01-24T17:41:34.381086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:24:32.206794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376756+0000","last_change":"2024-01-24T17:41:33.396814+0000","last_active":"2024-01-24T17:41:34.376756+0000","last_peered":"2024-01-24T17:41:34.376756+0000","last_clean":"2024-01-24T17:41:34.376756+0000","last_became_active":"2024-01-24T17:41:33.396446+0000","last_became_peered":"2024-01-24T17:41:33.396446+0000","last_unstale":"2024-01-24T17:41:34.376756+0000","last_undegraded":"2024-01-24T17:41:34.376756+0000","last_fullsized":"2024-01-24T17:41:34.376756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386135+0000","last_change":"2024-01-24T17:41:33.386328+0000","last_active":"2024-01-24T17:41:34.386135+0000","last_peered":"2024-01-24T17:41:34.386135+0000","last_clean":"2024-01-24T17:41:34.386135+0000","last_became_active":"2024-01-24T17:41:33.384795+0000","last_became_peered":"2024-01-24T17:41:33.384795+0000","last_unstale":"2024-01-24T17:41:34.386135+0000","last_undegraded":"2024-01-24T17:41:34.386135+0000","last_fullsized":"2024-01-24T17:41:34.386135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372759+0000","last_change":"2024-01-24T17:41:33.381873+0000","last_active":"2024-01-24T17:41:34.372759+0000","last_peered":"2024-01-24T17:41:34.372759+0000","last_clean":"2024-01-24T17:41:34.372759+0000","last_became_active":"2024-01-24T17:41:33.381356+0000","last_became_peered":"2024-01-24T17:41:33.381356+0000","last_unstale":"2024-01-24T17:41:34.372759+0000","last_undegraded":"2024-01-24T17:41:34.372759+0000","last_fullsized":"2024-01-24T17:41:34.372759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372500+0000","last_change":"2024-01-24T17:41:33.401722+0000","last_active":"2024-01-24T17:41:34.372500+0000","last_peered":"2024-01-24T17:41:34.372500+0000","last_clean":"2024-01-24T17:41:34.372500+0000","last_became_active":"2024-01-24T17:41:33.401437+0000","last_became_peered":"2024-01-24T17:41:33.401437+0000","last_unstale":"2024-01-24T17:41:34.372500+0000","last_undegraded":"2024-01-24T17:41:34.372500+0000","last_fullsized":"2024-01-24T17:41:34.372500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:53:07.257519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.375137+0000","last_change":"2024-01-24T17:41:33.386028+0000","last_active":"2024-01-24T17:41:34.375137+0000","last_peered":"2024-01-24T17:41:34.375137+0000","last_clean":"2024-01-24T17:41:34.375137+0000","last_became_active":"2024-01-24T17:41:33.385728+0000","last_became_peered":"2024-01-24T17:41:33.385728+0000","last_unstale":"2024-01-24T17:41:34.375137+0000","last_undegraded":"2024-01-24T17:41:34.375137+0000","last_fullsized":"2024-01-24T17:41:34.375137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:14.826051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381062+0000","last_change":"2024-01-24T17:41:33.392322+0000","last_active":"2024-01-24T17:41:34.381062+0000","last_peered":"2024-01-24T17:41:34.381062+0000","last_clean":"2024-01-24T17:41:34.381062+0000","last_became_active":"2024-01-24T17:41:33.391949+0000","last_became_peered":"2024-01-24T17:41:33.391949+0000","last_unstale":"2024-01-24T17:41:34.381062+0000","last_undegraded":"2024-01-24T17:41:34.381062+0000","last_fullsized":"2024-01-24T17:41:34.381062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381378+0000","last_change":"2024-01-24T17:41:33.392982+0000","last_active":"2024-01-24T17:41:34.381378+0000","last_peered":"2024-01-24T17:41:34.381378+0000","last_clean":"2024-01-24T17:41:34.381378+0000","last_became_active":"2024-01-24T17:41:33.392653+0000","last_became_peered":"2024-01-24T17:41:33.392653+0000","last_unstale":"2024-01-24T17:41:34.381378+0000","last_undegraded":"2024-01-24T17:41:34.381378+0000","last_fullsized":"2024-01-24T17:41:34.381378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382379+0000","last_change":"2024-01-24T17:41:33.388829+0000","last_active":"2024-01-24T17:41:34.382379+0000","last_peered":"2024-01-24T17:41:34.382379+0000","last_clean":"2024-01-24T17:41:34.382379+0000","last_became_active":"2024-01-24T17:41:33.387836+0000","last_became_peered":"2024-01-24T17:41:33.387836+0000","last_unstale":"2024-01-24T17:41:34.382379+0000","last_undegraded":"2024-01-24T17:41:34.382379+0000","last_fullsized":"2024-01-24T17:41:34.382379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386559+0000","last_change":"2024-01-24T17:41:33.386169+0000","last_active":"2024-01-24T17:41:34.386559+0000","last_peered":"2024-01-24T17:41:34.386559+0000","last_clean":"2024-01-24T17:41:34.386559+0000","last_became_active":"2024-01-24T17:41:33.385898+0000","last_became_peered":"2024-01-24T17:41:33.385898+0000","last_unstale":"2024-01-24T17:41:34.386559+0000","last_undegraded":"2024-01-24T17:41:34.386559+0000","last_fullsized":"2024-01-24T17:41:34.386559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:46.793343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402349+0000","last_change":"2024-01-24T17:41:32.895862+0000","last_active":"2024-01-24T17:41:33.402349+0000","last_peered":"2024-01-24T17:41:33.402349+0000","last_clean":"2024-01-24T17:41:33.402349+0000","last_became_active":"2024-01-24T17:41:32.895370+0000","last_became_peered":"2024-01-24T17:41:32.895370+0000","last_unstale":"2024-01-24T17:41:33.402349+0000","last_undegraded":"2024-01-24T17:41:33.402349+0000","last_fullsized":"2024-01-24T17:41:33.402349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373080+0000","last_change":"2024-01-24T17:41:33.405202+0000","last_active":"2024-01-24T17:41:34.373080+0000","last_peered":"2024-01-24T17:41:34.373080+0000","last_clean":"2024-01-24T17:41:34.373080+0000","last_became_active":"2024-01-24T17:41:33.404917+0000","last_became_peered":"2024-01-24T17:41:33.404917+0000","last_unstale":"2024-01-24T17:41:34.373080+0000","last_undegraded":"2024-01-24T17:41:34.373080+0000","last_fullsized":"2024-01-24T17:41:34.373080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:42.134883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404334+0000","last_change":"2024-01-24T17:41:32.910250+0000","last_active":"2024-01-24T17:41:33.404334+0000","last_peered":"2024-01-24T17:41:33.404334+0000","last_clean":"2024-01-24T17:41:33.404334+0000","last_became_active":"2024-01-24T17:41:32.909619+0000","last_became_peered":"2024-01-24T17:41:32.909619+0000","last_unstale":"2024-01-24T17:41:33.404334+0000","last_undegraded":"2024-01-24T17:41:33.404334+0000","last_fullsized":"2024-01-24T17:41:33.404334+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:56.825045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.373938+0000","last_change":"2024-01-24T17:41:32.895589+0000","last_active":"2024-01-24T17:41:33.373938+0000","last_peered":"2024-01-24T17:41:33.373938+0000","last_clean":"2024-01-24T17:41:33.373938+0000","last_became_active":"2024-01-24T17:41:32.895305+0000","last_became_peered":"2024-01-24T17:41:32.895305+0000","last_unstale":"2024-01-24T17:41:33.373938+0000","last_undegraded":"2024-01-24T17:41:33.373938+0000","last_fullsized":"2024-01-24T17:41:33.373938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:01:37.962464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374431+0000","last_change":"2024-01-24T17:41:33.385480+0000","last_active":"2024-01-24T17:41:34.374431+0000","last_peered":"2024-01-24T17:41:34.374431+0000","last_clean":"2024-01-24T17:41:34.374431+0000","last_became_active":"2024-01-24T17:41:33.384848+0000","last_became_peered":"2024-01-24T17:41:33.384848+0000","last_unstale":"2024-01-24T17:41:34.374431+0000","last_undegraded":"2024-01-24T17:41:34.374431+0000","last_fullsized":"2024-01-24T17:41:34.374431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:50:21.701964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377136+0000","last_change":"2024-01-24T17:41:32.897056+0000","last_active":"2024-01-24T17:41:33.377136+0000","last_peered":"2024-01-24T17:41:33.377136+0000","last_clean":"2024-01-24T17:41:33.377136+0000","last_became_active":"2024-01-24T17:41:32.888851+0000","last_became_peered":"2024-01-24T17:41:32.888851+0000","last_unstale":"2024-01-24T17:41:33.377136+0000","last_undegraded":"2024-01-24T17:41:33.377136+0000","last_fullsized":"2024-01-24T17:41:33.377136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374368+0000","last_change":"2024-01-24T17:41:33.395388+0000","last_active":"2024-01-24T17:41:34.374368+0000","last_peered":"2024-01-24T17:41:34.374368+0000","last_clean":"2024-01-24T17:41:34.374368+0000","last_became_active":"2024-01-24T17:41:33.395100+0000","last_became_peered":"2024-01-24T17:41:33.395100+0000","last_unstale":"2024-01-24T17:41:34.374368+0000","last_undegraded":"2024-01-24T17:41:34.374368+0000","last_fullsized":"2024-01-24T17:41:34.374368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:17:59.263270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.383197+0000","last_change":"2024-01-24T17:41:32.892978+0000","last_active":"2024-01-24T17:41:33.383197+0000","last_peered":"2024-01-24T17:41:33.383197+0000","last_clean":"2024-01-24T17:41:33.383197+0000","last_became_active":"2024-01-24T17:41:32.892668+0000","last_became_peered":"2024-01-24T17:41:32.892668+0000","last_unstale":"2024-01-24T17:41:33.383197+0000","last_undegraded":"2024-01-24T17:41:33.383197+0000","last_fullsized":"2024-01-24T17:41:33.383197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:01:37.962464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386042+0000","last_change":"2024-01-24T17:41:33.384625+0000","last_active":"2024-01-24T17:41:34.386042+0000","last_peered":"2024-01-24T17:41:34.386042+0000","last_clean":"2024-01-24T17:41:34.386042+0000","last_became_active":"2024-01-24T17:41:33.384293+0000","last_became_peered":"2024-01-24T17:41:33.384293+0000","last_unstale":"2024-01-24T17:41:34.386042+0000","last_undegraded":"2024-01-24T17:41:34.386042+0000","last_fullsized":"2024-01-24T17:41:34.386042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:27.474325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.369468+0000","last_change":"2024-01-24T17:41:32.896336+0000","last_active":"2024-01-24T17:41:33.369468+0000","last_peered":"2024-01-24T17:41:33.369468+0000","last_clean":"2024-01-24T17:41:33.369468+0000","last_became_active":"2024-01-24T17:41:32.896023+0000","last_became_peered":"2024-01-24T17:41:32.896023+0000","last_unstale":"2024-01-24T17:41:33.369468+0000","last_undegraded":"2024-01-24T17:41:33.369468+0000","last_fullsized":"2024-01-24T17:41:33.369468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.377112+0000","last_change":"2024-01-24T17:41:33.384509+0000","last_active":"2024-01-24T17:41:34.377112+0000","last_peered":"2024-01-24T17:41:34.377112+0000","last_clean":"2024-01-24T17:41:34.377112+0000","last_became_active":"2024-01-24T17:41:33.384188+0000","last_became_peered":"2024-01-24T17:41:33.384188+0000","last_unstale":"2024-01-24T17:41:34.377112+0000","last_undegraded":"2024-01-24T17:41:34.377112+0000","last_fullsized":"2024-01-24T17:41:34.377112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:02.263592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.394541+0000","last_change":"2024-01-24T17:41:32.898747+0000","last_active":"2024-01-24T17:41:33.394541+0000","last_peered":"2024-01-24T17:41:33.394541+0000","last_clean":"2024-01-24T17:41:33.394541+0000","last_became_active":"2024-01-24T17:41:32.896625+0000","last_became_peered":"2024-01-24T17:41:32.896625+0000","last_unstale":"2024-01-24T17:41:33.394541+0000","last_undegraded":"2024-01-24T17:41:33.394541+0000","last_fullsized":"2024-01-24T17:41:33.394541+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381067+0000","last_change":"2024-01-24T17:41:33.385876+0000","last_active":"2024-01-24T17:41:34.381067+0000","last_peered":"2024-01-24T17:41:34.381067+0000","last_clean":"2024-01-24T17:41:34.381067+0000","last_became_active":"2024-01-24T17:41:33.385566+0000","last_became_peered":"2024-01-24T17:41:33.385566+0000","last_unstale":"2024-01-24T17:41:34.381067+0000","last_undegraded":"2024-01-24T17:41:34.381067+0000","last_fullsized":"2024-01-24T17:41:34.381067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:44:38.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.411432+0000","last_change":"2024-01-24T17:41:32.901159+0000","last_active":"2024-01-24T17:41:33.411432+0000","last_peered":"2024-01-24T17:41:33.411432+0000","last_clean":"2024-01-24T17:41:33.411432+0000","last_became_active":"2024-01-24T17:41:32.900832+0000","last_became_peered":"2024-01-24T17:41:32.900832+0000","last_unstale":"2024-01-24T17:41:33.411432+0000","last_undegraded":"2024-01-24T17:41:33.411432+0000","last_fullsized":"2024-01-24T17:41:33.411432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:06:04.595152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383859+0000","last_change":"2024-01-24T17:41:33.382102+0000","last_active":"2024-01-24T17:41:34.383859+0000","last_peered":"2024-01-24T17:41:34.383859+0000","last_clean":"2024-01-24T17:41:34.383859+0000","last_became_active":"2024-01-24T17:41:33.381813+0000","last_became_peered":"2024-01-24T17:41:33.381813+0000","last_unstale":"2024-01-24T17:41:34.383859+0000","last_undegraded":"2024-01-24T17:41:34.383859+0000","last_fullsized":"2024-01-24T17:41:34.383859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:13:31.518238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.405772+0000","last_change":"2024-01-24T17:41:32.896282+0000","last_active":"2024-01-24T17:41:33.405772+0000","last_peered":"2024-01-24T17:41:33.405772+0000","last_clean":"2024-01-24T17:41:33.405772+0000","last_became_active":"2024-01-24T17:41:32.896029+0000","last_became_peered":"2024-01-24T17:41:32.896029+0000","last_unstale":"2024-01-24T17:41:33.405772+0000","last_undegraded":"2024-01-24T17:41:33.405772+0000","last_fullsized":"2024-01-24T17:41:33.405772+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372253+0000","last_change":"2024-01-24T17:41:33.380019+0000","last_active":"2024-01-24T17:41:34.372253+0000","last_peered":"2024-01-24T17:41:34.372253+0000","last_clean":"2024-01-24T17:41:34.372253+0000","last_became_active":"2024-01-24T17:41:33.379743+0000","last_became_peered":"2024-01-24T17:41:33.379743+0000","last_unstale":"2024-01-24T17:41:34.372253+0000","last_undegraded":"2024-01-24T17:41:34.372253+0000","last_fullsized":"2024-01-24T17:41:34.372253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:01:48.740461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377512+0000","last_change":"2024-01-24T17:41:32.906190+0000","last_active":"2024-01-24T17:41:33.377512+0000","last_peered":"2024-01-24T17:41:33.377512+0000","last_clean":"2024-01-24T17:41:33.377512+0000","last_became_active":"2024-01-24T17:41:32.903525+0000","last_became_peered":"2024-01-24T17:41:32.903525+0000","last_unstale":"2024-01-24T17:41:33.377512+0000","last_undegraded":"2024-01-24T17:41:33.377512+0000","last_fullsized":"2024-01-24T17:41:33.377512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:46.796323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382677+0000","last_change":"2024-01-24T17:41:33.400221+0000","last_active":"2024-01-24T17:41:34.382677+0000","last_peered":"2024-01-24T17:41:34.382677+0000","last_clean":"2024-01-24T17:41:34.382677+0000","last_became_active":"2024-01-24T17:41:33.399980+0000","last_became_peered":"2024-01-24T17:41:33.399980+0000","last_unstale":"2024-01-24T17:41:34.382677+0000","last_undegraded":"2024-01-24T17:41:34.382677+0000","last_fullsized":"2024-01-24T17:41:34.382677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:54:11.431445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.367726+0000","last_change":"2024-01-24T17:41:32.907626+0000","last_active":"2024-01-24T17:41:33.367726+0000","last_peered":"2024-01-24T17:41:33.367726+0000","last_clean":"2024-01-24T17:41:33.367726+0000","last_became_active":"2024-01-24T17:41:32.907361+0000","last_became_peered":"2024-01-24T17:41:32.907361+0000","last_unstale":"2024-01-24T17:41:33.367726+0000","last_undegraded":"2024-01-24T17:41:33.367726+0000","last_fullsized":"2024-01-24T17:41:33.367726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386410+0000","last_change":"2024-01-24T17:41:33.392634+0000","last_active":"2024-01-24T17:41:34.386410+0000","last_peered":"2024-01-24T17:41:34.386410+0000","last_clean":"2024-01-24T17:41:34.386410+0000","last_became_active":"2024-01-24T17:41:33.392428+0000","last_became_peered":"2024-01-24T17:41:33.392428+0000","last_unstale":"2024-01-24T17:41:34.386410+0000","last_undegraded":"2024-01-24T17:41:34.386410+0000","last_fullsized":"2024-01-24T17:41:34.386410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.409695+0000","last_change":"2024-01-24T17:41:32.902055+0000","last_active":"2024-01-24T17:41:33.409695+0000","last_peered":"2024-01-24T17:41:33.409695+0000","last_clean":"2024-01-24T17:41:33.409695+0000","last_became_active":"2024-01-24T17:41:32.901738+0000","last_became_peered":"2024-01-24T17:41:32.901738+0000","last_unstale":"2024-01-24T17:41:33.409695+0000","last_undegraded":"2024-01-24T17:41:33.409695+0000","last_fullsized":"2024-01-24T17:41:33.409695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:45.778394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381845+0000","last_change":"2024-01-24T17:41:33.388186+0000","last_active":"2024-01-24T17:41:34.381845+0000","last_peered":"2024-01-24T17:41:34.381845+0000","last_clean":"2024-01-24T17:41:34.381845+0000","last_became_active":"2024-01-24T17:41:33.387930+0000","last_became_peered":"2024-01-24T17:41:33.387930+0000","last_unstale":"2024-01-24T17:41:34.381845+0000","last_undegraded":"2024-01-24T17:41:34.381845+0000","last_fullsized":"2024-01-24T17:41:34.381845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:19:54.318762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402116+0000","last_change":"2024-01-24T17:41:32.903388+0000","last_active":"2024-01-24T17:41:33.402116+0000","last_peered":"2024-01-24T17:41:33.402116+0000","last_clean":"2024-01-24T17:41:33.402116+0000","last_became_active":"2024-01-24T17:41:32.900342+0000","last_became_peered":"2024-01-24T17:41:32.900342+0000","last_unstale":"2024-01-24T17:41:33.402116+0000","last_undegraded":"2024-01-24T17:41:33.402116+0000","last_fullsized":"2024-01-24T17:41:33.402116+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386143+0000","last_change":"2024-01-24T17:41:33.379754+0000","last_active":"2024-01-24T17:41:34.386143+0000","last_peered":"2024-01-24T17:41:34.386143+0000","last_clean":"2024-01-24T17:41:34.386143+0000","last_became_active":"2024-01-24T17:41:33.379454+0000","last_became_peered":"2024-01-24T17:41:33.379454+0000","last_unstale":"2024-01-24T17:41:34.386143+0000","last_undegraded":"2024-01-24T17:41:34.386143+0000","last_fullsized":"2024-01-24T17:41:34.386143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:46.793343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.408082+0000","last_change":"2024-01-24T17:41:32.904632+0000","last_active":"2024-01-24T17:41:33.408082+0000","last_peered":"2024-01-24T17:41:33.408082+0000","last_clean":"2024-01-24T17:41:33.408082+0000","last_became_active":"2024-01-24T17:41:32.904220+0000","last_became_peered":"2024-01-24T17:41:32.904220+0000","last_unstale":"2024-01-24T17:41:33.408082+0000","last_undegraded":"2024-01-24T17:41:33.408082+0000","last_fullsized":"2024-01-24T17:41:33.408082+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:53.093743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382977+0000","last_change":"2024-01-24T17:41:33.395831+0000","last_active":"2024-01-24T17:41:34.382977+0000","last_peered":"2024-01-24T17:41:34.382977+0000","last_clean":"2024-01-24T17:41:34.382977+0000","last_became_active":"2024-01-24T17:41:33.392642+0000","last_became_peered":"2024-01-24T17:41:33.392642+0000","last_unstale":"2024-01-24T17:41:34.382977+0000","last_undegraded":"2024-01-24T17:41:34.382977+0000","last_fullsized":"2024-01-24T17:41:34.382977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:20:35.949143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.406390+0000","last_change":"2024-01-24T17:41:32.898336+0000","last_active":"2024-01-24T17:41:33.406390+0000","last_peered":"2024-01-24T17:41:33.406390+0000","last_clean":"2024-01-24T17:41:33.406390+0000","last_became_active":"2024-01-24T17:41:32.897951+0000","last_became_peered":"2024-01-24T17:41:32.897951+0000","last_unstale":"2024-01-24T17:41:33.406390+0000","last_undegraded":"2024-01-24T17:41:33.406390+0000","last_fullsized":"2024-01-24T17:41:33.406390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.383662+0000","last_change":"2024-01-24T17:41:33.388321+0000","last_active":"2024-01-24T17:41:34.383662+0000","last_peered":"2024-01-24T17:41:34.383662+0000","last_clean":"2024-01-24T17:41:34.383662+0000","last_became_active":"2024-01-24T17:41:33.386908+0000","last_became_peered":"2024-01-24T17:41:33.386908+0000","last_unstale":"2024-01-24T17:41:34.383662+0000","last_undegraded":"2024-01-24T17:41:34.383662+0000","last_fullsized":"2024-01-24T17:41:34.383662+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:14.826051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.376101+0000","last_change":"2024-01-24T17:41:32.898929+0000","last_active":"2024-01-24T17:41:33.376101+0000","last_peered":"2024-01-24T17:41:33.376101+0000","last_clean":"2024-01-24T17:41:33.376101+0000","last_became_active":"2024-01-24T17:41:32.898648+0000","last_became_peered":"2024-01-24T17:41:32.898648+0000","last_unstale":"2024-01-24T17:41:33.376101+0000","last_undegraded":"2024-01-24T17:41:33.376101+0000","last_fullsized":"2024-01-24T17:41:33.376101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:04:20.925548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381588+0000","last_change":"2024-01-24T17:41:33.394760+0000","last_active":"2024-01-24T17:41:34.381588+0000","last_peered":"2024-01-24T17:41:34.381588+0000","last_clean":"2024-01-24T17:41:34.381588+0000","last_became_active":"2024-01-24T17:41:33.394503+0000","last_became_peered":"2024-01-24T17:41:33.394503+0000","last_unstale":"2024-01-24T17:41:34.381588+0000","last_undegraded":"2024-01-24T17:41:34.381588+0000","last_fullsized":"2024-01-24T17:41:34.381588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:36:20.611114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.382896+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:41:33.382896+0000","last_peered":"2024-01-24T17:41:33.382896+0000","last_clean":"2024-01-24T17:41:33.382896+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:41:33.382896+0000","last_undegraded":"2024-01-24T17:41:33.382896+0000","last_fullsized":"2024-01-24T17:41:33.382896+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.378136+0000","last_change":"2024-01-24T17:41:32.906502+0000","last_active":"2024-01-24T17:41:33.378136+0000","last_peered":"2024-01-24T17:41:33.378136+0000","last_clean":"2024-01-24T17:41:33.378136+0000","last_became_active":"2024-01-24T17:41:32.905833+0000","last_became_peered":"2024-01-24T17:41:32.905833+0000","last_unstale":"2024-01-24T17:41:33.378136+0000","last_undegraded":"2024-01-24T17:41:33.378136+0000","last_fullsized":"2024-01-24T17:41:33.378136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:16.636377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374358+0000","last_change":"2024-01-24T17:41:33.400209+0000","last_active":"2024-01-24T17:41:34.374358+0000","last_peered":"2024-01-24T17:41:34.374358+0000","last_clean":"2024-01-24T17:41:34.374358+0000","last_became_active":"2024-01-24T17:41:33.399935+0000","last_became_peered":"2024-01-24T17:41:33.399935+0000","last_unstale":"2024-01-24T17:41:34.374358+0000","last_undegraded":"2024-01-24T17:41:34.374358+0000","last_fullsized":"2024-01-24T17:41:34.374358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:02.263592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377275+0000","last_change":"2024-01-24T17:41:32.903275+0000","last_active":"2024-01-24T17:41:33.377275+0000","last_peered":"2024-01-24T17:41:33.377275+0000","last_clean":"2024-01-24T17:41:33.377275+0000","last_became_active":"2024-01-24T17:41:32.902650+0000","last_became_peered":"2024-01-24T17:41:32.902650+0000","last_unstale":"2024-01-24T17:41:33.377275+0000","last_undegraded":"2024-01-24T17:41:33.377275+0000","last_fullsized":"2024-01-24T17:41:33.377275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.370807+0000","last_change":"2024-01-24T17:41:33.385359+0000","last_active":"2024-01-24T17:41:34.370807+0000","last_peered":"2024-01-24T17:41:34.370807+0000","last_clean":"2024-01-24T17:41:34.370807+0000","last_became_active":"2024-01-24T17:41:33.385048+0000","last_became_peered":"2024-01-24T17:41:33.385048+0000","last_unstale":"2024-01-24T17:41:34.370807+0000","last_undegraded":"2024-01-24T17:41:34.370807+0000","last_fullsized":"2024-01-24T17:41:34.370807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:17.651326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.381252+0000","last_change":"2024-01-24T17:41:32.899789+0000","last_active":"2024-01-24T17:41:33.381252+0000","last_peered":"2024-01-24T17:41:33.381252+0000","last_clean":"2024-01-24T17:41:33.381252+0000","last_became_active":"2024-01-24T17:41:32.899458+0000","last_became_peered":"2024-01-24T17:41:32.899458+0000","last_unstale":"2024-01-24T17:41:33.381252+0000","last_undegraded":"2024-01-24T17:41:33.381252+0000","last_fullsized":"2024-01-24T17:41:33.381252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374273+0000","last_change":"2024-01-24T17:41:33.385653+0000","last_active":"2024-01-24T17:41:34.374273+0000","last_peered":"2024-01-24T17:41:34.374273+0000","last_clean":"2024-01-24T17:41:34.374273+0000","last_became_active":"2024-01-24T17:41:33.385180+0000","last_became_peered":"2024-01-24T17:41:33.385180+0000","last_unstale":"2024-01-24T17:41:34.374273+0000","last_undegraded":"2024-01-24T17:41:34.374273+0000","last_fullsized":"2024-01-24T17:41:34.374273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:01:35.279502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377607+0000","last_change":"2024-01-24T17:41:32.903904+0000","last_active":"2024-01-24T17:41:33.377607+0000","last_peered":"2024-01-24T17:41:33.377607+0000","last_clean":"2024-01-24T17:41:33.377607+0000","last_became_active":"2024-01-24T17:41:32.902915+0000","last_became_peered":"2024-01-24T17:41:32.902915+0000","last_unstale":"2024-01-24T17:41:33.377607+0000","last_undegraded":"2024-01-24T17:41:33.377607+0000","last_fullsized":"2024-01-24T17:41:33.377607+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:51.152032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382323+0000","last_change":"2024-01-24T17:41:33.379050+0000","last_active":"2024-01-24T17:41:34.382323+0000","last_peered":"2024-01-24T17:41:34.382323+0000","last_clean":"2024-01-24T17:41:34.382323+0000","last_became_active":"2024-01-24T17:41:33.378782+0000","last_became_peered":"2024-01-24T17:41:33.378782+0000","last_unstale":"2024-01-24T17:41:34.382323+0000","last_undegraded":"2024-01-24T17:41:34.382323+0000","last_fullsized":"2024-01-24T17:41:34.382323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.377892+0000","last_change":"2024-01-24T17:41:32.895887+0000","last_active":"2024-01-24T17:41:33.377892+0000","last_peered":"2024-01-24T17:41:33.377892+0000","last_clean":"2024-01-24T17:41:33.377892+0000","last_became_active":"2024-01-24T17:41:32.895460+0000","last_became_peered":"2024-01-24T17:41:32.895460+0000","last_unstale":"2024-01-24T17:41:33.377892+0000","last_undegraded":"2024-01-24T17:41:33.377892+0000","last_fullsized":"2024-01-24T17:41:33.377892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.385945+0000","last_change":"2024-01-24T17:41:33.381026+0000","last_active":"2024-01-24T17:41:34.385945+0000","last_peered":"2024-01-24T17:41:34.385945+0000","last_clean":"2024-01-24T17:41:34.385945+0000","last_became_active":"2024-01-24T17:41:33.380649+0000","last_became_peered":"2024-01-24T17:41:33.380649+0000","last_unstale":"2024-01-24T17:41:34.385945+0000","last_undegraded":"2024-01-24T17:41:34.385945+0000","last_fullsized":"2024-01-24T17:41:34.385945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:57.839994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.378103+0000","last_change":"2024-01-24T17:41:32.907390+0000","last_active":"2024-01-24T17:41:33.378103+0000","last_peered":"2024-01-24T17:41:33.378103+0000","last_clean":"2024-01-24T17:41:33.378103+0000","last_became_active":"2024-01-24T17:41:32.906724+0000","last_became_peered":"2024-01-24T17:41:32.906724+0000","last_unstale":"2024-01-24T17:41:33.378103+0000","last_undegraded":"2024-01-24T17:41:33.378103+0000","last_fullsized":"2024-01-24T17:41:33.378103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:45.778394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373347+0000","last_change":"2024-01-24T17:41:33.390087+0000","last_active":"2024-01-24T17:41:34.373347+0000","last_peered":"2024-01-24T17:41:34.373347+0000","last_clean":"2024-01-24T17:41:34.373347+0000","last_became_active":"2024-01-24T17:41:33.389820+0000","last_became_peered":"2024-01-24T17:41:33.389820+0000","last_unstale":"2024-01-24T17:41:34.373347+0000","last_undegraded":"2024-01-24T17:41:34.373347+0000","last_fullsized":"2024-01-24T17:41:34.373347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373963+0000","last_change":"2024-01-24T17:41:32.902595+0000","last_active":"2024-01-24T17:41:34.373963+0000","last_peered":"2024-01-24T17:41:34.373963+0000","last_clean":"2024-01-24T17:41:34.373963+0000","last_became_active":"2024-01-24T17:41:32.902291+0000","last_became_peered":"2024-01-24T17:41:32.902291+0000","last_unstale":"2024-01-24T17:41:34.373963+0000","last_undegraded":"2024-01-24T17:41:34.373963+0000","last_fullsized":"2024-01-24T17:41:34.373963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.374204+0000","last_change":"2024-01-24T17:41:33.401908+0000","last_active":"2024-01-24T17:41:34.374204+0000","last_peered":"2024-01-24T17:41:34.374204+0000","last_clean":"2024-01-24T17:41:34.374204+0000","last_became_active":"2024-01-24T17:41:33.401634+0000","last_became_peered":"2024-01-24T17:41:33.401634+0000","last_unstale":"2024-01-24T17:41:34.374204+0000","last_undegraded":"2024-01-24T17:41:34.374204+0000","last_fullsized":"2024-01-24T17:41:34.374204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.405956+0000","last_change":"2024-01-24T17:41:32.909941+0000","last_active":"2024-01-24T17:41:33.405956+0000","last_peered":"2024-01-24T17:41:33.405956+0000","last_clean":"2024-01-24T17:41:33.405956+0000","last_became_active":"2024-01-24T17:41:32.909592+0000","last_became_peered":"2024-01-24T17:41:32.909592+0000","last_unstale":"2024-01-24T17:41:33.405956+0000","last_undegraded":"2024-01-24T17:41:33.405956+0000","last_fullsized":"2024-01-24T17:41:33.405956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.373574+0000","last_change":"2024-01-24T17:41:33.394464+0000","last_active":"2024-01-24T17:41:34.373574+0000","last_peered":"2024-01-24T17:41:34.373574+0000","last_clean":"2024-01-24T17:41:34.373574+0000","last_became_active":"2024-01-24T17:41:33.393898+0000","last_became_peered":"2024-01-24T17:41:33.393898+0000","last_unstale":"2024-01-24T17:41:34.373574+0000","last_undegraded":"2024-01-24T17:41:34.373574+0000","last_fullsized":"2024-01-24T17:41:34.373574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:01:35.279502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.366360+0000","last_change":"2024-01-24T17:41:32.896815+0000","last_active":"2024-01-24T17:41:33.366360+0000","last_peered":"2024-01-24T17:41:33.366360+0000","last_clean":"2024-01-24T17:41:33.366360+0000","last_became_active":"2024-01-24T17:41:32.896263+0000","last_became_peered":"2024-01-24T17:41:32.896263+0000","last_unstale":"2024-01-24T17:41:33.366360+0000","last_undegraded":"2024-01-24T17:41:33.366360+0000","last_fullsized":"2024-01-24T17:41:33.366360+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:45:53.093743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.382082+0000","last_change":"2024-01-24T17:41:33.393488+0000","last_active":"2024-01-24T17:41:34.382082+0000","last_peered":"2024-01-24T17:41:34.382082+0000","last_clean":"2024-01-24T17:41:34.382082+0000","last_became_active":"2024-01-24T17:41:33.391266+0000","last_became_peered":"2024-01-24T17:41:33.391266+0000","last_unstale":"2024-01-24T17:41:34.382082+0000","last_undegraded":"2024-01-24T17:41:34.382082+0000","last_fullsized":"2024-01-24T17:41:34.382082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:08:31.383389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.387230+0000","last_change":"2024-01-24T17:41:32.902635+0000","last_active":"2024-01-24T17:41:33.387230+0000","last_peered":"2024-01-24T17:41:33.387230+0000","last_clean":"2024-01-24T17:41:33.387230+0000","last_became_active":"2024-01-24T17:41:32.902369+0000","last_became_peered":"2024-01-24T17:41:32.902369+0000","last_unstale":"2024-01-24T17:41:33.387230+0000","last_undegraded":"2024-01-24T17:41:33.387230+0000","last_fullsized":"2024-01-24T17:41:33.387230+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:05.371996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.372780+0000","last_change":"2024-01-24T17:41:33.390610+0000","last_active":"2024-01-24T17:41:34.372780+0000","last_peered":"2024-01-24T17:41:34.372780+0000","last_clean":"2024-01-24T17:41:34.372780+0000","last_became_active":"2024-01-24T17:41:33.390337+0000","last_became_peered":"2024-01-24T17:41:33.390337+0000","last_unstale":"2024-01-24T17:41:34.372780+0000","last_undegraded":"2024-01-24T17:41:34.372780+0000","last_fullsized":"2024-01-24T17:41:34.372780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:20:54.123755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.374492+0000","last_change":"2024-01-24T17:41:32.891585+0000","last_active":"2024-01-24T17:41:33.374492+0000","last_peered":"2024-01-24T17:41:33.374492+0000","last_clean":"2024-01-24T17:41:33.374492+0000","last_became_active":"2024-01-24T17:41:32.891376+0000","last_became_peered":"2024-01-24T17:41:32.891376+0000","last_unstale":"2024-01-24T17:41:33.374492+0000","last_undegraded":"2024-01-24T17:41:33.374492+0000","last_fullsized":"2024-01-24T17:41:33.374492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381998+0000","last_change":"2024-01-24T17:41:33.397462+0000","last_active":"2024-01-24T17:41:34.381998+0000","last_peered":"2024-01-24T17:41:34.381998+0000","last_clean":"2024-01-24T17:41:34.381998+0000","last_became_active":"2024-01-24T17:41:33.397113+0000","last_became_peered":"2024-01-24T17:41:33.397113+0000","last_unstale":"2024-01-24T17:41:34.381998+0000","last_undegraded":"2024-01-24T17:41:34.381998+0000","last_fullsized":"2024-01-24T17:41:34.381998+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:00:52.166982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.397708+0000","last_change":"2024-01-24T17:41:32.904150+0000","last_active":"2024-01-24T17:41:33.397708+0000","last_peered":"2024-01-24T17:41:33.397708+0000","last_clean":"2024-01-24T17:41:33.397708+0000","last_became_active":"2024-01-24T17:41:32.902550+0000","last_became_peered":"2024-01-24T17:41:32.902550+0000","last_unstale":"2024-01-24T17:41:33.397708+0000","last_undegraded":"2024-01-24T17:41:33.397708+0000","last_fullsized":"2024-01-24T17:41:33.397708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:29:26.459376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381261+0000","last_change":"2024-01-24T17:41:33.399418+0000","last_active":"2024-01-24T17:41:34.381261+0000","last_peered":"2024-01-24T17:41:34.381261+0000","last_clean":"2024-01-24T17:41:34.381261+0000","last_became_active":"2024-01-24T17:41:33.395852+0000","last_became_peered":"2024-01-24T17:41:33.395852+0000","last_unstale":"2024-01-24T17:41:34.381261+0000","last_undegraded":"2024-01-24T17:41:34.381261+0000","last_fullsized":"2024-01-24T17:41:34.381261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.381377+0000","last_change":"2024-01-24T17:41:33.401099+0000","last_active":"2024-01-24T17:41:34.381377+0000","last_peered":"2024-01-24T17:41:34.381377+0000","last_clean":"2024-01-24T17:41:34.381377+0000","last_became_active":"2024-01-24T17:41:33.399957+0000","last_became_peered":"2024-01-24T17:41:33.399957+0000","last_unstale":"2024-01-24T17:41:34.381377+0000","last_undegraded":"2024-01-24T17:41:34.381377+0000","last_fullsized":"2024-01-24T17:41:34.381377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:12:52.644481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404497+0000","last_change":"2024-01-24T17:41:32.900545+0000","last_active":"2024-01-24T17:41:33.404497+0000","last_peered":"2024-01-24T17:41:33.404497+0000","last_clean":"2024-01-24T17:41:33.404497+0000","last_became_active":"2024-01-24T17:41:32.900228+0000","last_became_peered":"2024-01-24T17:41:32.900228+0000","last_unstale":"2024-01-24T17:41:33.404497+0000","last_undegraded":"2024-01-24T17:41:33.404497+0000","last_fullsized":"2024-01-24T17:41:33.404497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:13:42.604966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.386811+0000","last_change":"2024-01-24T17:41:33.401985+0000","last_active":"2024-01-24T17:41:34.386811+0000","last_peered":"2024-01-24T17:41:34.386811+0000","last_clean":"2024-01-24T17:41:34.386811+0000","last_became_active":"2024-01-24T17:41:33.401623+0000","last_became_peered":"2024-01-24T17:41:33.401623+0000","last_unstale":"2024-01-24T17:41:34.386811+0000","last_undegraded":"2024-01-24T17:41:34.386811+0000","last_fullsized":"2024-01-24T17:41:34.386811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:09:47.811272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.402556+0000","last_change":"2024-01-24T17:41:32.899336+0000","last_active":"2024-01-24T17:41:33.402556+0000","last_peered":"2024-01-24T17:41:33.402556+0000","last_clean":"2024-01-24T17:41:33.402556+0000","last_became_active":"2024-01-24T17:41:32.898773+0000","last_became_peered":"2024-01-24T17:41:32.898773+0000","last_unstale":"2024-01-24T17:41:33.402556+0000","last_undegraded":"2024-01-24T17:41:33.402556+0000","last_fullsized":"2024-01-24T17:41:33.402556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:47.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376757+0000","last_change":"2024-01-24T17:41:33.396094+0000","last_active":"2024-01-24T17:41:34.376757+0000","last_peered":"2024-01-24T17:41:34.376757+0000","last_clean":"2024-01-24T17:41:34.376757+0000","last_became_active":"2024-01-24T17:41:33.395819+0000","last_became_peered":"2024-01-24T17:41:33.395819+0000","last_unstale":"2024-01-24T17:41:34.376757+0000","last_undegraded":"2024-01-24T17:41:34.376757+0000","last_fullsized":"2024-01-24T17:41:34.376757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:22.550977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.404773+0000","last_change":"2024-01-24T17:41:32.890539+0000","last_active":"2024-01-24T17:41:33.404773+0000","last_peered":"2024-01-24T17:41:33.404773+0000","last_clean":"2024-01-24T17:41:33.404773+0000","last_became_active":"2024-01-24T17:41:32.890175+0000","last_became_peered":"2024-01-24T17:41:32.890175+0000","last_unstale":"2024-01-24T17:41:33.404773+0000","last_undegraded":"2024-01-24T17:41:33.404773+0000","last_fullsized":"2024-01-24T17:41:33.404773+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:18:13.894628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-24T17:41:34.376495+0000","last_change":"2024-01-24T17:41:33.374778+0000","last_active":"2024-01-24T17:41:34.376495+0000","last_peered":"2024-01-24T17:41:34.376495+0000","last_clean":"2024-01-24T17:41:34.376495+0000","last_became_active":"2024-01-24T17:41:33.374507+0000","last_became_peered":"2024-01-24T17:41:33.374507+0000","last_unstale":"2024-01-24T17:41:34.376495+0000","last_undegraded":"2024-01-24T17:41:34.376495+0000","last_fullsized":"2024-01-24T17:41:34.376495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:32.847621+0000","last_clean_scrub_stamp":"2024-01-24T17:41:32.847621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:41.884460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-24T17:41:33.410169+0000","last_change":"2024-01-24T17:41:32.909855+0000","last_active":"2024-01-24T17:41:33.410169+0000","last_peered":"2024-01-24T17:41:33.410169+0000","last_clean":"2024-01-24T17:41:33.410169+0000","last_became_active":"2024-01-24T17:41:32.909553+0000","last_became_peered":"2024-01-24T17:41:32.909553+0000","last_unstale":"2024-01-24T17:41:33.410169+0000","last_undegraded":"2024-01-24T17:41:33.410169+0000","last_fullsized":"2024-01-24T17:41:33.410169+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:31.832671+0000","last_clean_scrub_stamp":"2024-01-24T17:41:31.832671+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:13:38.934796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2310000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":661358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.694}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8009999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.647}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":73134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":208896,"data_stored":71480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.792}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1589999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0110000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8540000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31904,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339936,"statfs":{"total":96636764160,"available":96604094464,"internally_reserved":0,"allocated":802816,"data_stored":661336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.29}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":212992,"data_stored":70954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:41:43.986 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-24T17:41:43.987 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-24T17:41:43.987 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-24T17:41:43.987 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-24T17:41:44.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.247+0000 7f01ae000640 1 Processor -- start 2024-01-24T17:41:44.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.247+0000 7f01ae000640 1 -- start start 2024-01-24T17:41:44.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.249+0000 7f01ae000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f01a815cdf0 0x7f01a80cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.249+0000 7f01ae000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f01a80cef60 0x7f01a80d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.249+0000 7f01ae000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80cf5a0 con 0x7f01a80d0d30 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.249+0000 7f01ae000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a80cf3c0 con 0x7f01a80cef60 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.249+0000 7f01ae000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a80d11e0 con 0x7f01a815cdf0 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.249+0000 7f01a7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f01a80d0d30 0x7f01a80cd110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54374/0 (socket says 172.21.15.39:54374) 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.249+0000 7f01a7e00640 1 -- 172.21.15.39:0/2173301624 learned_addr learned my addr 172.21.15.39:0/2173301624 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.249+0000 7f019e800640 1 --2- 172.21.15.39:0/2173301624 >> v2:172.21.15.39:3300/0 conn(0x7f01a80cef60 0x7f01a80d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.250+0000 7f01a7400640 1 --2- 172.21.15.39:0/2173301624 >> v2:172.21.15.100:3300/0 conn(0x7f01a815cdf0 0x7f01a80cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.250+0000 7f01a6a00640 1 -- 172.21.15.39:0/2173301624 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349925423 0 0) 0x7f01a80cf5a0 con 0x7f01a80d0d30 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.250+0000 7f01a6a00640 1 -- 172.21.15.39:0/2173301624 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0188003520 con 0x7f01a80d0d30 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.250+0000 7f019e800640 1 -- 172.21.15.39:0/2173301624 >> v2:172.21.15.100:3300/0 conn(0x7f01a815cdf0 msgr2=0x7f01a80cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.250+0000 7f019e800640 1 --2- 172.21.15.39:0/2173301624 >> v2:172.21.15.100:3300/0 conn(0x7f01a815cdf0 0x7f01a80cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.250+0000 7f019e800640 1 -- 172.21.15.39:0/2173301624 >> v1:172.21.15.39:6789/0 conn(0x7f01a80d0d30 legacy=0x7f01a80cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.250+0000 7f019e800640 1 -- 172.21.15.39:0/2173301624 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80cf880 con 0x7f01a80cef60 2024-01-24T17:41:44.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.250+0000 7f01a7400640 1 --2- 172.21.15.39:0/2173301624 >> v2:172.21.15.100:3300/0 conn(0x7f01a815cdf0 0x7f01a80cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.251+0000 7f019e800640 1 --2- 172.21.15.39:0/2173301624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a80cef60 0x7f01a80d07f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f01a80d15b0 tx=0x7f019002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45806fa4d074c27 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.251+0000 7f01a6a00640 1 -- 172.21.15.39:0/2173301624 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190031830 con 0x7f01a80cef60 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.251+0000 7f01a6a00640 1 -- 172.21.15.39:0/2173301624 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0190031990 con 0x7f01a80cef60 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.251+0000 7f01a6a00640 1 -- 172.21.15.39:0/2173301624 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190031d20 con 0x7f01a80cef60 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.252+0000 7f01ae000640 1 -- 172.21.15.39:0/2173301624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a80cef60 msgr2=0x7f01a80d07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.252+0000 7f01ae000640 1 --2- 172.21.15.39:0/2173301624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a80cef60 0x7f01a80d07f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f01a80d15b0 tx=0x7f019002b8d0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.252+0000 7f01ae000640 1 -- 172.21.15.39:0/2173301624 shutdown_connections 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.252+0000 7f01ae000640 1 --2- 172.21.15.39:0/2173301624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a80cef60 0x7f01a80d07f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.252+0000 7f01ae000640 1 --2- 172.21.15.39:0/2173301624 >> v2:172.21.15.100:3300/0 conn(0x7f01a815cdf0 0x7f01a80cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.252+0000 7f01ae000640 1 -- 172.21.15.39:0/2173301624 >> 172.21.15.39:0/2173301624 conn(0x7f01a8099b60 msgr2=0x7f01a8089fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:44.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.252+0000 7f01ae000640 1 -- 172.21.15.39:0/2173301624 shutdown_connections 2024-01-24T17:41:44.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.252+0000 7f01ae000640 1 -- 172.21.15.39:0/2173301624 wait complete. 2024-01-24T17:41:44.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.254+0000 7f01ae000640 1 Processor -- start 2024-01-24T17:41:44.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.254+0000 7f01ae000640 1 -- start start 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01ae000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01a80cef60 0x7f01a815af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01ae000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01a80d0d30 0x7f01a81546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01ae000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a815cdf0 0x7f01a8157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01ae000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a809e5d0 con 0x7f01a815cdf0 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01ae000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a809a640 con 0x7f01a80d0d30 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01ae000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8154c30 con 0x7f01a80cef60 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01a7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a815cdf0 0x7f01a8157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01a7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a815cdf0 0x7f01a8157d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58104/0 (socket says 172.21.15.39:58104) 2024-01-24T17:41:44.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.255+0000 7f01a7e00640 1 -- 172.21.15.39:0/635391074 learned_addr learned my addr 172.21.15.39:0/635391074 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f019e800640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01a80d0d30 0x7f01a81546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01a7400640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01a80cef60 0x7f01a815af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01a7e00640 1 -- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01a80cef60 msgr2=0x7f01a815af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01a7e00640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01a80cef60 0x7f01a815af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01a7e00640 1 -- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01a80d0d30 msgr2=0x7f01a81546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01a7e00640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01a80d0d30 0x7f01a81546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01a7e00640 1 -- 172.21.15.39:0/635391074 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190009400 con 0x7f01a815cdf0 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01a7e00640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a815cdf0 0x7f01a8157d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f01a000c460 tx=0x7f01a000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f019fe00640 1 -- 172.21.15.39:0/635391074 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0005d30 con 0x7f01a815cdf0 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f019fe00640 1 -- 172.21.15.39:0/635391074 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a0005e90 con 0x7f01a815cdf0 2024-01-24T17:41:44.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f019fe00640 1 -- 172.21.15.39:0/635391074 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0007290 con 0x7f01a815cdf0 2024-01-24T17:41:44.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a8159ed0 con 0x7f01a815cdf0 2024-01-24T17:41:44.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.256+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a815a710 con 0x7f01a815cdf0 2024-01-24T17:41:44.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.257+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a8073ef0 con 0x7f01a815cdf0 2024-01-24T17:41:44.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.258+0000 7f019fe00640 1 -- 172.21.15.39:0/635391074 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f01a00073f0 con 0x7f01a815cdf0 2024-01-24T17:41:44.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.259+0000 7f019fe00640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018006a430 0x7f018006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.259+0000 7f019fe00640 1 -- 172.21.15.39:0/635391074 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01a0087c70 con 0x7f01a815cdf0 2024-01-24T17:41:44.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.260+0000 7f01a7400640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018006a430 0x7f018006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.261+0000 7f01a7400640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018006a430 0x7f018006c8f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0198009ed0 tx=0x7f019800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:44.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.264+0000 7f019fe00640 1 -- 172.21.15.39:0/635391074 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f01a0053130 con 0x7f01a815cdf0 2024-01-24T17:41:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.497+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f01a80b1c90 con 0x7f01a815cdf0 2024-01-24T17:41:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.498+0000 7f019fe00640 1 -- 172.21.15.39:0/635391074 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f01a0017070 con 0x7f01a815cdf0 2024-01-24T17:41:44.498 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:44.498 INFO:teuthology.orchestra.run.smithi039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-24T17:41:44.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018006a430 msgr2=0x7f018006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018006a430 0x7f018006c8f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0198009ed0 tx=0x7f019800b040 comp rx=0 tx=0).stop 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a815cdf0 msgr2=0x7f01a8157d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a815cdf0 0x7f01a8157d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f01a000c460 tx=0x7f01a000c920 comp rx=0 tx=0).stop 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 shutdown_connections 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018006a430 0x7f018006c8f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01a815cdf0 0x7f01a8157d80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01a80d0d30 0x7f01a81546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 --2- 172.21.15.39:0/635391074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01a80cef60 0x7f01a815af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 >> 172.21.15.39:0/635391074 conn(0x7f01a8099b60 msgr2=0x7f01a806eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 shutdown_connections 2024-01-24T17:41:44.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.501+0000 7f01ae000640 1 -- 172.21.15.39:0/635391074 wait complete. 2024-01-24T17:41:44.519 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-24T17:41:44.520 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:44.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.725+0000 7f4348c00640 1 Processor -- start 2024-01-24T17:41:44.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.725+0000 7f4348c00640 1 -- start start 2024-01-24T17:41:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.727+0000 7f4348c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f434415a950 0x7f43440ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.727+0000 7f4348c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f43440caaa0 0x7f43440caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.727+0000 7f4348c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43440cb870 con 0x7f43440cb490 2024-01-24T17:41:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.727+0000 7f4348c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43440b43d0 con 0x7f434415a950 2024-01-24T17:41:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.727+0000 7f4348c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43440c8f70 con 0x7f43440caaa0 2024-01-24T17:41:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.727+0000 7f4341e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f434415a950 0x7f43440ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.727+0000 7f4341e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f434415a950 0x7f43440ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58106/0 (socket says 172.21.15.39:58106) 2024-01-24T17:41:44.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.727+0000 7f4341e00640 1 -- 172.21.15.39:0/3322956952 learned_addr learned my addr 172.21.15.39:0/3322956952 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.728+0000 7f4341e00640 1 -- 172.21.15.39:0/3322956952 >> v2:172.21.15.100:3300/0 conn(0x7f43440caaa0 msgr2=0x7f43440caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.728+0000 7f4341e00640 1 --2- 172.21.15.39:0/3322956952 >> v2:172.21.15.100:3300/0 conn(0x7f43440caaa0 0x7f43440caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.728+0000 7f4341e00640 1 -- 172.21.15.39:0/3322956952 >> v1:172.21.15.100:6789/0 conn(0x7f43440cb490 legacy=0x7f43440affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.728+0000 7f4341e00640 1 -- 172.21.15.39:0/3322956952 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43440c9180 con 0x7f434415a950 2024-01-24T17:41:44.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.729+0000 7f4341e00640 1 --2- 172.21.15.39:0/3322956952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434415a950 0x7f43440ca560 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f434415e2e0 tx=0x7f433402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252c4573665695ab server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:44.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.729+0000 7f4341400640 1 -- 172.21.15.39:0/3322956952 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334031870 con 0x7f434415a950 2024-01-24T17:41:44.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.729+0000 7f4341400640 1 -- 172.21.15.39:0/3322956952 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43340319d0 con 0x7f434415a950 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.729+0000 7f4341400640 1 -- 172.21.15.39:0/3322956952 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334031d60 con 0x7f434415a950 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.730+0000 7f4348c00640 1 -- 172.21.15.39:0/3322956952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434415a950 msgr2=0x7f43440ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.730+0000 7f4348c00640 1 --2- 172.21.15.39:0/3322956952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434415a950 0x7f43440ca560 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f434415e2e0 tx=0x7f433402b910 comp rx=0 tx=0).stop 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.730+0000 7f4348c00640 1 -- 172.21.15.39:0/3322956952 shutdown_connections 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.730+0000 7f4348c00640 1 --2- 172.21.15.39:0/3322956952 >> v2:172.21.15.100:3300/0 conn(0x7f43440caaa0 0x7f43440caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.730+0000 7f4348c00640 1 --2- 172.21.15.39:0/3322956952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f434415a950 0x7f43440ca560 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.730+0000 7f4348c00640 1 -- 172.21.15.39:0/3322956952 >> 172.21.15.39:0/3322956952 conn(0x7f434406a3e0 msgr2=0x7f434406bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.730+0000 7f4348c00640 1 -- 172.21.15.39:0/3322956952 shutdown_connections 2024-01-24T17:41:44.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.730+0000 7f4348c00640 1 -- 172.21.15.39:0/3322956952 wait complete. 2024-01-24T17:41:44.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.732+0000 7f4348c00640 1 Processor -- start 2024-01-24T17:41:44.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.732+0000 7f4348c00640 1 -- start start 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f4348c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43440caaa0 0x7f43440c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f4348c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43440cb490 0x7f43440cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f4348c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434415a950 0x7f43441519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f4348c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43440c4350 con 0x7f43440cb490 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f4348c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43440c4630 con 0x7f434415a950 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f4348c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43440c4910 con 0x7f43440caaa0 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f433be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43440cb490 0x7f43440cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f433be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43440cb490 0x7f43440cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58112/0 (socket says 172.21.15.39:58112) 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f433be00640 1 -- 172.21.15.39:0/620454626 learned_addr learned my addr 172.21.15.39:0/620454626 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f433be00640 1 -- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43440caaa0 msgr2=0x7f43440c3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f433be00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43440caaa0 0x7f43440c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f433be00640 1 -- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434415a950 msgr2=0x7f43441519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f4342800640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434415a950 0x7f43441519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f433be00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434415a950 0x7f43441519a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.733+0000 7f433be00640 1 -- 172.21.15.39:0/620454626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340094d0 con 0x7f43440cb490 2024-01-24T17:41:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.734+0000 7f433be00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43440cb490 0x7f43440cd190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f432c00b4a0 tx=0x7f432c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.734+0000 7f433a000640 1 -- 172.21.15.39:0/620454626 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c010610 con 0x7f43440cb490 2024-01-24T17:41:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.734+0000 7f433a000640 1 -- 172.21.15.39:0/620454626 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c010770 con 0x7f43440cb490 2024-01-24T17:41:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.734+0000 7f433a000640 1 -- 172.21.15.39:0/620454626 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c010b20 con 0x7f43440cb490 2024-01-24T17:41:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.734+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43440c4bf0 con 0x7f43440cb490 2024-01-24T17:41:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.734+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43440cd6d0 con 0x7f43440cb490 2024-01-24T17:41:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.735+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4344073ef0 con 0x7f43440cb490 2024-01-24T17:41:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.736+0000 7f433a000640 1 -- 172.21.15.39:0/620454626 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f432c010c80 con 0x7f43440cb490 2024-01-24T17:41:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.737+0000 7f433a000640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431c06a430 0x7f431c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.737+0000 7f433a000640 1 -- 172.21.15.39:0/620454626 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f432c0884a0 con 0x7f43440cb490 2024-01-24T17:41:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.738+0000 7f4341e00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431c06a430 0x7f431c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:44.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.739+0000 7f4341e00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431c06a430 0x7f431c06c8f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4334036000 tx=0x7f4334030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.742+0000 7f433a000640 1 -- 172.21.15.39:0/620454626 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f432c053b00 con 0x7f43440cb490 2024-01-24T17:41:44.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.939+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43440ae950 con 0x7f43440cb490 2024-01-24T17:41:44.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.941+0000 7f433a000640 1 -- 172.21.15.39:0/620454626 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f432c057120 con 0x7f43440cb490 2024-01-24T17:41:44.942 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:44.942 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:44.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431c06a430 msgr2=0x7f431c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431c06a430 0x7f431c06c8f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4334036000 tx=0x7f4334030000 comp rx=0 tx=0).stop 2024-01-24T17:41:44.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43440cb490 msgr2=0x7f43440cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:44.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43440cb490 0x7f43440cd190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f432c00b4a0 tx=0x7f432c00b960 comp rx=0 tx=0).stop 2024-01-24T17:41:44.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 shutdown_connections 2024-01-24T17:41:44.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f434415a950 0x7f43441519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431c06a430 0x7f431c06c8f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43440cb490 0x7f43440cd190 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 --2- 172.21.15.39:0/620454626 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43440caaa0 0x7f43440c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:44.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.943+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 >> 172.21.15.39:0/620454626 conn(0x7f434406a3e0 msgr2=0x7f434415cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:44.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.944+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 shutdown_connections 2024-01-24T17:41:44.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:44.944+0000 7f4348c00640 1 -- 172.21.15.39:0/620454626 wait complete. 2024-01-24T17:41:44.964 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:41:44.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:41:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.187+0000 7f9f09c00640 1 Processor -- start 2024-01-24T17:41:45.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.188+0000 7f9f09c00640 1 -- start start 2024-01-24T17:41:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.189+0000 7f9f09c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9f040b0350 0x7f9f0406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.189+0000 7f9f09c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9f0415e190 0x7f9f040b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.189+0000 7f9f09c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0406d4c0 con 0x7f9f0406c9f0 2024-01-24T17:41:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.189+0000 7f9f09c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0406cf00 con 0x7f9f0415e190 2024-01-24T17:41:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.189+0000 7f9f09c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0406d1e0 con 0x7f9f040b0350 2024-01-24T17:41:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9f03400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9f0415e190 0x7f9f040b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9efbe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9f0406c9f0 0x7f9f0415da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54386/0 (socket says 172.21.15.39:54386) 2024-01-24T17:41:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9efbe00640 1 -- 172.21.15.39:0/3543589164 learned_addr learned my addr 172.21.15.39:0/3543589164 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9f02a00640 1 --2- 172.21.15.39:0/3543589164 >> v2:172.21.15.100:3300/0 conn(0x7f9f040b0350 0x7f9f0406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9f02000640 1 -- 172.21.15.39:0/3543589164 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711478014 0 0) 0x7f9f0406d4c0 con 0x7f9f0406c9f0 2024-01-24T17:41:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9f02000640 1 -- 172.21.15.39:0/3543589164 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee4003520 con 0x7f9f0406c9f0 2024-01-24T17:41:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9f03400640 1 -- 172.21.15.39:0/3543589164 >> v2:172.21.15.100:3300/0 conn(0x7f9f040b0350 msgr2=0x7f9f0406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9f03400640 1 --2- 172.21.15.39:0/3543589164 >> v2:172.21.15.100:3300/0 conn(0x7f9f040b0350 0x7f9f0406c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9f03400640 1 -- 172.21.15.39:0/3543589164 >> v1:172.21.15.39:6789/0 conn(0x7f9f0406c9f0 legacy=0x7f9f0415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.190+0000 7f9f03400640 1 -- 172.21.15.39:0/3543589164 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f040b15e0 con 0x7f9f0415e190 2024-01-24T17:41:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.192+0000 7f9f03400640 1 --2- 172.21.15.39:0/3543589164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0415e190 0x7f9f040b1de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9f0406e170 tx=0x7f9efc02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd0da5c541f32b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.192+0000 7f9f02000640 1 -- 172.21.15.39:0/3543589164 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc02cbe0 con 0x7f9f0415e190 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.192+0000 7f9f02000640 1 -- 172.21.15.39:0/3543589164 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc02cd40 con 0x7f9f0415e190 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.192+0000 7f9f02000640 1 -- 172.21.15.39:0/3543589164 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc034640 con 0x7f9f0415e190 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.193+0000 7f9f09c00640 1 -- 172.21.15.39:0/3543589164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0415e190 msgr2=0x7f9f040b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.193+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3543589164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0415e190 0x7f9f040b1de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9f0406e170 tx=0x7f9efc02be30 comp rx=0 tx=0).stop 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.193+0000 7f9f09c00640 1 -- 172.21.15.39:0/3543589164 shutdown_connections 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.193+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3543589164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0415e190 0x7f9f040b1de0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.193+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3543589164 >> v2:172.21.15.100:3300/0 conn(0x7f9f040b0350 0x7f9f0406c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.193+0000 7f9f09c00640 1 -- 172.21.15.39:0/3543589164 >> 172.21.15.39:0/3543589164 conn(0x7f9f040cb0e0 msgr2=0x7f9f040cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.193+0000 7f9f09c00640 1 -- 172.21.15.39:0/3543589164 shutdown_connections 2024-01-24T17:41:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.193+0000 7f9f09c00640 1 -- 172.21.15.39:0/3543589164 wait complete. 2024-01-24T17:41:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.195+0000 7f9f09c00640 1 Processor -- start 2024-01-24T17:41:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.195+0000 7f9f09c00640 1 -- start start 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f09c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0406c9f0 0x7f9f04151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f09c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f040b0350 0x7f9f040af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f09c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f0415e190 0x7f9f040af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f09c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040c5c80 con 0x7f9f0406c9f0 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f09c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040cd240 con 0x7f9f0415e190 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f09c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040cd520 con 0x7f9f040b0350 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f02a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0406c9f0 0x7f9f04151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f02a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0406c9f0 0x7f9f04151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58132/0 (socket says 172.21.15.39:58132) 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f02a00640 1 -- 172.21.15.39:0/3654611693 learned_addr learned my addr 172.21.15.39:0/3654611693 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f02a00640 1 -- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f040b0350 msgr2=0x7f9f040af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f02a00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f040b0350 0x7f9f040af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f02a00640 1 -- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f0415e190 msgr2=0x7f9f040af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f02a00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f0415e190 0x7f9f040af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.196+0000 7f9f02a00640 1 -- 172.21.15.39:0/3654611693 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc009a70 con 0x7f9f0406c9f0 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.197+0000 7f9f02a00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0406c9f0 0x7f9f04151c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400e9b0 tx=0x7f9ef400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.197+0000 7f9efaa00640 1 -- 172.21.15.39:0/3654611693 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4009670 con 0x7f9f0406c9f0 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.197+0000 7f9efaa00640 1 -- 172.21.15.39:0/3654611693 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef400c9a0 con 0x7f9f0406c9f0 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.197+0000 7f9efaa00640 1 -- 172.21.15.39:0/3654611693 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef400cd50 con 0x7f9f0406c9f0 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.197+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f040cd800 con 0x7f9f0406c9f0 2024-01-24T17:41:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.197+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f04152170 con 0x7f9f0406c9f0 2024-01-24T17:41:45.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.198+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f040cdd20 con 0x7f9f0406c9f0 2024-01-24T17:41:45.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.199+0000 7f9efaa00640 1 -- 172.21.15.39:0/3654611693 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9ef4013470 con 0x7f9f0406c9f0 2024-01-24T17:41:45.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.200+0000 7f9efaa00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9edc06a430 0x7f9edc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.201+0000 7f9efaa00640 1 -- 172.21.15.39:0/3654611693 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ef4088be0 con 0x7f9f0406c9f0 2024-01-24T17:41:45.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.201+0000 7f9efbe00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9edc06a430 0x7f9edc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.202+0000 7f9efbe00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9edc06a430 0x7f9edc06c8f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9eec000f60 tx=0x7f9eec0078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:45.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.205+0000 7f9efaa00640 1 -- 172.21.15.39:0/3654611693 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9ef40541d0 con 0x7f9f0406c9f0 2024-01-24T17:41:45.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.399+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f040c5dc0 con 0x7f9f0406c9f0 2024-01-24T17:41:45.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.400+0000 7f9efaa00640 1 -- 172.21.15.39:0/3654611693 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ef40170a0 con 0x7f9f0406c9f0 2024-01-24T17:41:45.401 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:41:45.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.402+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9edc06a430 msgr2=0x7f9edc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.402+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9edc06a430 0x7f9edc06c8f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9eec000f60 tx=0x7f9eec0078f0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.402+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0406c9f0 msgr2=0x7f9f04151c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.402+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0406c9f0 0x7f9f04151c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400e9b0 tx=0x7f9ef400ee70 comp rx=0 tx=0).stop 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.403+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 shutdown_connections 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.403+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f0415e190 0x7f9f040af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.403+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f040b0350 0x7f9f040af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.403+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9edc06a430 0x7f9edc06c8f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.403+0000 7f9f09c00640 1 --2- 172.21.15.39:0/3654611693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f0406c9f0 0x7f9f04151c30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.403+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 >> 172.21.15.39:0/3654611693 conn(0x7f9f040cb0e0 msgr2=0x7f9f040c2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.403+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 shutdown_connections 2024-01-24T17:41:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.403+0000 7f9f09c00640 1 -- 172.21.15.39:0/3654611693 wait complete. 2024-01-24T17:41:45.422 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:41:45.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.643+0000 7f8454400640 1 Processor -- start 2024-01-24T17:41:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.643+0000 7f8454400640 1 -- start start 2024-01-24T17:41:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.644+0000 7f8454400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844c0d14c0 0x7f844c0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.644+0000 7f8454400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0cda90 con 0x7f844c15fff0 2024-01-24T17:41:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.644+0000 7f8454400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0d32c0 con 0x7f844c0b2a10 2024-01-24T17:41:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.644+0000 7f8454400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0b2ec0 con 0x7f844c0d14c0 2024-01-24T17:41:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.645+0000 7f8451a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f844c15fff0 0x7f844c0d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54390/0 (socket says 172.21.15.39:54390) 2024-01-24T17:41:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.645+0000 7f8451a00640 1 -- 172.21.15.39:0/3607289937 learned_addr learned my addr 172.21.15.39:0/3607289937 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.645+0000 7f8451000640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844c0d14c0 0x7f844c0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.645+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984886319 0 0) 0x7f844c0cda90 con 0x7f844c15fff0 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.645+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842c003520 con 0x7f844c15fff0 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4149235764 0 0) 0x7f842c003520 con 0x7f844c15fff0 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844c0d14c0 msgr2=0x7f844c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844c0d14c0 0x7f844c0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 >> v1:172.21.15.100:6789/0 conn(0x7f844c0b2a10 legacy=0x7f844c0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0b30a0 con 0x7f844c15fff0 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f843c001670 con 0x7f844c15fff0 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 >> v1:172.21.15.39:6789/0 conn(0x7f844c15fff0 legacy=0x7f844c0d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f842c0039a0 0x7f842c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844c0d14c0 0x7f842c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f842c004d00 0x7f842c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0b2ec0 con 0x7f842c0039a0 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c00ba60 con 0x7f842c004d00 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842c00bc20 con 0x7f844c0d14c0 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f8451a00640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f842c0039a0 0x7f842c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f8451000640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844c0d14c0 0x7f842c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.646+0000 7f8452400640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f842c004d00 0x7f842c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.647+0000 7f8451a00640 1 -- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844c0d14c0 msgr2=0x7f842c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.647+0000 7f8451a00640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f844c0d14c0 0x7f842c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.647+0000 7f8451a00640 1 -- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f842c004d00 msgr2=0x7f842c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.647+0000 7f8451a00640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f842c004d00 0x7f842c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.647+0000 7f8451a00640 1 -- 172.21.15.39:0/3607289937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c003520 con 0x7f842c0039a0 2024-01-24T17:41:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.648+0000 7f8451a00640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f842c0039a0 0x7f842c003dc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f843c007fb0 tx=0x7f842c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1eaee6aa5f31a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.648+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c005ad0 con 0x7f842c0039a0 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.648+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c005c30 con 0x7f842c0039a0 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.648+0000 7f844be00640 1 -- 172.21.15.39:0/3607289937 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c02c7c0 con 0x7f842c0039a0 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.648+0000 7f8454400640 1 -- 172.21.15.39:0/3607289937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f842c0039a0 msgr2=0x7f842c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.648+0000 7f8454400640 1 --2- 172.21.15.39:0/3607289937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f842c0039a0 0x7f842c003dc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f843c007fb0 tx=0x7f842c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.649+0000 7f8452400640 1 -- 172.21.15.39:0/3607289937 reap_dead start 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.649+0000 7f8454400640 1 -- 172.21.15.39:0/3607289937 shutdown_connections 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.649+0000 7f8454400640 1 -- 172.21.15.39:0/3607289937 >> 172.21.15.39:0/3607289937 conn(0x7f844c09bcc0 msgr2=0x7f844c0d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.649+0000 7f8454400640 1 -- 172.21.15.39:0/3607289937 shutdown_connections 2024-01-24T17:41:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.649+0000 7f8454400640 1 -- 172.21.15.39:0/3607289937 wait complete. 2024-01-24T17:41:45.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.650+0000 7f8454400640 1 Processor -- start 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.651+0000 7f8454400640 1 -- start start 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8454400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f842c0039a0 0x7f844c1f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8454400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f844c1f56c0 0x7f844c1f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8454400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f844c15b7f0 0x7f844c15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8454400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c15b3d0 con 0x7f844c1f56c0 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8454400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c1f6080 con 0x7f844c15b7f0 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8454400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c15a750 con 0x7f842c0039a0 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f844c1f56c0 0x7f844c1f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f844c1f56c0 0x7f844c1f5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58144/0 (socket says 172.21.15.39:58144) 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451000640 1 -- 172.21.15.39:0/1115213186 learned_addr learned my addr 172.21.15.39:0/1115213186 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451a00640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f842c0039a0 0x7f844c1f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8452400640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f844c15b7f0 0x7f844c15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451000640 1 -- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f842c0039a0 msgr2=0x7f844c1f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451000640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f842c0039a0 0x7f844c1f3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451000640 1 -- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f844c15b7f0 msgr2=0x7f844c15bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451000640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f844c15b7f0 0x7f844c15bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.652+0000 7f8451000640 1 -- 172.21.15.39:0/1115213186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c15aa30 con 0x7f844c1f56c0 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.653+0000 7f8451000640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f844c1f56c0 0x7f844c1f5b40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8434007f30 tx=0x7f8434011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.653+0000 7f8449600640 1 -- 172.21.15.39:0/1115213186 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843400e030 con 0x7f844c1f56c0 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.653+0000 7f8449600640 1 -- 172.21.15.39:0/1115213186 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8434016470 con 0x7f844c1f56c0 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.653+0000 7f8449600640 1 -- 172.21.15.39:0/1115213186 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434016820 con 0x7f844c1f56c0 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.653+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c15ad10 con 0x7f844c1f56c0 2024-01-24T17:41:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.653+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c15c1e0 con 0x7f844c1f56c0 2024-01-24T17:41:45.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.655+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c073ef0 con 0x7f844c1f56c0 2024-01-24T17:41:45.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.655+0000 7f8449600640 1 -- 172.21.15.39:0/1115213186 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8434016980 con 0x7f844c1f56c0 2024-01-24T17:41:45.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.656+0000 7f8449600640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f842406a430 0x7f842406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:45.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.656+0000 7f8449600640 1 -- 172.21.15.39:0/1115213186 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f843400f070 con 0x7f844c1f56c0 2024-01-24T17:41:45.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.657+0000 7f8451a00640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f842406a430 0x7f842406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:45.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.658+0000 7f8451a00640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f842406a430 0x7f842406c8f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f843c034000 tx=0x7f843c037040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:45.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.660+0000 7f8449600640 1 -- 172.21.15.39:0/1115213186 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8434054de0 con 0x7f844c1f56c0 2024-01-24T17:41:45.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.856+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f844c15c400 con 0x7f844c1f56c0 2024-01-24T17:41:45.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.857+0000 7f8449600640 1 -- 172.21.15.39:0/1115213186 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8434058400 con 0x7f844c1f56c0 2024-01-24T17:41:45.857 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:41:45.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.859+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f842406a430 msgr2=0x7f842406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.859+0000 7f8454400640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f842406a430 0x7f842406c8f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f843c034000 tx=0x7f843c037040 comp rx=0 tx=0).stop 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.859+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f844c1f56c0 msgr2=0x7f844c1f5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.859+0000 7f8454400640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f844c1f56c0 0x7f844c1f5b40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8434007f30 tx=0x7f8434011660 comp rx=0 tx=0).stop 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.860+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 shutdown_connections 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.860+0000 7f8454400640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f844c15b7f0 0x7f844c15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.860+0000 7f8454400640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f842406a430 0x7f842406c8f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.860+0000 7f8454400640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f844c1f56c0 0x7f844c1f5b40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.860+0000 7f8454400640 1 --2- 172.21.15.39:0/1115213186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f842c0039a0 0x7f844c1f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.860+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 >> 172.21.15.39:0/1115213186 conn(0x7f844c09bcc0 msgr2=0x7f844c15f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.860+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 shutdown_connections 2024-01-24T17:41:45.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:45.860+0000 7f8454400640 1 -- 172.21.15.39:0/1115213186 wait complete. 2024-01-24T17:41:45.879 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:41:46.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.080+0000 7f2f4a400640 1 Processor -- start 2024-01-24T17:41:46.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.080+0000 7f2f4a400640 1 -- start start 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.081+0000 7f2f4a400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2f440cb540 0x7f2f4406d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.081+0000 7f2f4a400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f4406c3a0 0x7f2f440ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.081+0000 7f2f4a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f440ad0b0 con 0x7f2f440accd0 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.081+0000 7f2f4a400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4406d840 con 0x7f2f440cb540 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.081+0000 7f2f4a400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f440b2370 con 0x7f2f4406c3a0 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f49000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2f440accd0 0x7f2f440cae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54398/0 (socket says 172.21.15.39:54398) 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f49000640 1 -- 172.21.15.39:0/3083373302 learned_addr learned my addr 172.21.15.39:0/3083373302 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f43e00640 1 --2- 172.21.15.39:0/3083373302 >> v2:172.21.15.39:3300/0 conn(0x7f2f440cb540 0x7f2f4406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f49a00640 1 --2- 172.21.15.39:0/3083373302 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f4406c3a0 0x7f2f440ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f43400640 1 -- 172.21.15.39:0/3083373302 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212168574 0 0) 0x7f2f440ad0b0 con 0x7f2f440accd0 2024-01-24T17:41:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f43400640 1 -- 172.21.15.39:0/3083373302 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f24003520 con 0x7f2f440accd0 2024-01-24T17:41:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f43e00640 1 -- 172.21.15.39:0/3083373302 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f4406c3a0 msgr2=0x7f2f440ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f43e00640 1 --2- 172.21.15.39:0/3083373302 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f4406c3a0 0x7f2f440ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f43e00640 1 -- 172.21.15.39:0/3083373302 >> v1:172.21.15.39:6789/0 conn(0x7f2f440accd0 legacy=0x7f2f440cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.082+0000 7f2f43e00640 1 -- 172.21.15.39:0/3083373302 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f440b2650 con 0x7f2f440cb540 2024-01-24T17:41:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.083+0000 7f2f43e00640 1 --2- 172.21.15.39:0/3083373302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4406d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0097a0 tx=0x7f2f2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0241e26061b732 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.084+0000 7f2f43400640 1 -- 172.21.15.39:0/3083373302 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c031830 con 0x7f2f440cb540 2024-01-24T17:41:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.084+0000 7f2f43400640 1 -- 172.21.15.39:0/3083373302 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c031990 con 0x7f2f440cb540 2024-01-24T17:41:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.084+0000 7f2f43400640 1 -- 172.21.15.39:0/3083373302 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c031d20 con 0x7f2f440cb540 2024-01-24T17:41:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.084+0000 7f2f4a400640 1 -- 172.21.15.39:0/3083373302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 msgr2=0x7f2f4406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.084+0000 7f2f4a400640 1 --2- 172.21.15.39:0/3083373302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4406d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0097a0 tx=0x7f2f2c02b8d0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.085+0000 7f2f4a400640 1 -- 172.21.15.39:0/3083373302 shutdown_connections 2024-01-24T17:41:46.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.085+0000 7f2f4a400640 1 --2- 172.21.15.39:0/3083373302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4406d300 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.085+0000 7f2f4a400640 1 --2- 172.21.15.39:0/3083373302 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f4406c3a0 0x7f2f440ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.085+0000 7f2f4a400640 1 -- 172.21.15.39:0/3083373302 >> 172.21.15.39:0/3083373302 conn(0x7f2f44069e50 msgr2=0x7f2f4406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:46.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.085+0000 7f2f4a400640 1 -- 172.21.15.39:0/3083373302 shutdown_connections 2024-01-24T17:41:46.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.085+0000 7f2f4a400640 1 -- 172.21.15.39:0/3083373302 wait complete. 2024-01-24T17:41:46.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.086+0000 7f2f4a400640 1 Processor -- start 2024-01-24T17:41:46.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.087+0000 7f2f4a400640 1 -- start start 2024-01-24T17:41:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f4a400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f4406c3a0 0x7f2f4415bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f4a400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f440accd0 0x7f2f44092410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f4a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4415cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f4a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4415c6b0 con 0x7f2f440cb540 2024-01-24T17:41:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f4a400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f440ae630 con 0x7f2f4406c3a0 2024-01-24T17:41:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f4a400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f44152510 con 0x7f2f440accd0 2024-01-24T17:41:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4415cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4415cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58162/0 (socket says 172.21.15.39:58162) 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49a00640 1 -- 172.21.15.39:0/82586904 learned_addr learned my addr 172.21.15.39:0/82586904 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49000640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f4406c3a0 0x7f2f4415bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f43e00640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f440accd0 0x7f2f44092410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49a00640 1 -- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f440accd0 msgr2=0x7f2f44092410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49a00640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f440accd0 0x7f2f44092410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49a00640 1 -- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f4406c3a0 msgr2=0x7f2f4415bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49a00640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f4406c3a0 0x7f2f4415bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.088+0000 7f2f49a00640 1 -- 172.21.15.39:0/82586904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c009400 con 0x7f2f440cb540 2024-01-24T17:41:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.089+0000 7f2f49a00640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4415cb90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c007e40 tx=0x7f2f3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.089+0000 7f2f40c00640 1 -- 172.21.15.39:0/82586904 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c014620 con 0x7f2f440cb540 2024-01-24T17:41:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.089+0000 7f2f40c00640 1 -- 172.21.15.39:0/82586904 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c00ee90 con 0x7f2f440cb540 2024-01-24T17:41:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.089+0000 7f2f40c00640 1 -- 172.21.15.39:0/82586904 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c0149e0 con 0x7f2f440cb540 2024-01-24T17:41:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.089+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f440c1f70 con 0x7f2f440cb540 2024-01-24T17:41:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.089+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4415c320 con 0x7f2f440cb540 2024-01-24T17:41:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.091+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4415c4b0 con 0x7f2f440cb540 2024-01-24T17:41:46.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.091+0000 7f2f40c00640 1 -- 172.21.15.39:0/82586904 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2f3c014b40 con 0x7f2f440cb540 2024-01-24T17:41:46.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.092+0000 7f2f40c00640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f1c06a430 0x7f2f1c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.092+0000 7f2f40c00640 1 -- 172.21.15.39:0/82586904 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f3c089230 con 0x7f2f440cb540 2024-01-24T17:41:46.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.093+0000 7f2f49000640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f1c06a430 0x7f2f1c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.094+0000 7f2f49000640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f1c06a430 0x7f2f1c06c8f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2f340010e0 tx=0x7f2f34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:46.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.097+0000 7f2f40c00640 1 -- 172.21.15.39:0/82586904 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2f3c0548b0 con 0x7f2f440cb540 2024-01-24T17:41:46.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.294+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2f4404dff0 con 0x7f2f440cb540 2024-01-24T17:41:46.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.295+0000 7f2f40c00640 1 -- 172.21.15.39:0/82586904 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2f3c057ed0 con 0x7f2f440cb540 2024-01-24T17:41:46.296 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f1c06a430 msgr2=0x7f2f1c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f1c06a430 0x7f2f1c06c8f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2f340010e0 tx=0x7f2f34009040 comp rx=0 tx=0).stop 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 msgr2=0x7f2f4415cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4415cb90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c007e40 tx=0x7f2f3c00e4b0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 shutdown_connections 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f1c06a430 0x7f2f1c06c8f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f440cb540 0x7f2f4415cb90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f440accd0 0x7f2f44092410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 --2- 172.21.15.39:0/82586904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f4406c3a0 0x7f2f4415bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 >> 172.21.15.39:0/82586904 conn(0x7f2f44069e50 msgr2=0x7f2f4406a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:46.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 shutdown_connections 2024-01-24T17:41:46.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.298+0000 7f2f4a400640 1 -- 172.21.15.39:0/82586904 wait complete. 2024-01-24T17:41:46.317 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:41:46.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.533+0000 7fd816c00640 1 Processor -- start 2024-01-24T17:41:46.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.533+0000 7fd816c00640 1 -- start start 2024-01-24T17:41:46.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.535+0000 7fd816c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd8100cb9c0 0x7fd81006bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.535+0000 7fd816c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8100cd050 con 0x7fd8100ca670 2024-01-24T17:41:46.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.535+0000 7fd816c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd810159fe0 con 0x7fd8100cdd70 2024-01-24T17:41:46.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.535+0000 7fd816c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8100caa50 con 0x7fd8100cb9c0 2024-01-24T17:41:46.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.535+0000 7fd815800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd8100ca670 0x7fd81015a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54410/0 (socket says 172.21.15.39:54410) 2024-01-24T17:41:46.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.535+0000 7fd815800640 1 -- 172.21.15.39:0/1163107246 learned_addr learned my addr 172.21.15.39:0/1163107246 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:46.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd816200640 1 --2- 172.21.15.39:0/1163107246 >> v2:172.21.15.100:3300/0 conn(0x7fd8100cb9c0 0x7fd81006bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782455071 0 0) 0x7fd8100cd050 con 0x7fd8100ca670 2024-01-24T17:41:46.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f0005730 con 0x7fd8100ca670 2024-01-24T17:41:46.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540011812 0 0) 0x7fd7f0005730 con 0x7fd8100ca670 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 >> v2:172.21.15.100:3300/0 conn(0x7fd8100cb9c0 msgr2=0x7fd81006bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 --2- 172.21.15.39:0/1163107246 >> v2:172.21.15.100:3300/0 conn(0x7fd8100cb9c0 0x7fd81006bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 >> v1:172.21.15.100:6789/0 conn(0x7fd8100cdd70 legacy=0x7fd8100cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd810159c90 con 0x7fd8100ca670 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fd800001670 con 0x7fd8100ca670 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 >> v1:172.21.15.39:6789/0 conn(0x7fd8100ca670 legacy=0x7fd81015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7f0005b80 0x7fd7f0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd7f0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7f0007390 0x7fd7f000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100caa50 con 0x7fd7f00069d0 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f000e330 con 0x7fd7f0007390 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f000e4f0 con 0x7fd7f0005b80 2024-01-24T17:41:46.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.536+0000 7fd814e00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd7f0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd816200640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7f0007390 0x7fd7f000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd815800640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7f0005b80 0x7fd7f0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd814e00640 1 -- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7f0005b80 msgr2=0x7fd7f0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd814e00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd7f0005b80 0x7fd7f0005fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd814e00640 1 -- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7f0007390 msgr2=0x7fd7f000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd814e00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7f0007390 0x7fd7f000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd814e00640 1 -- 172.21.15.39:0/1163107246 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f0005730 con 0x7fd7f00069d0 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd816200640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd7f0007390 0x7fd7f000db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.537+0000 7fd816200640 1 -- 172.21.15.39:0/1163107246 reap_dead start 2024-01-24T17:41:46.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.538+0000 7fd814e00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd7f0006e50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8001750 tx=0x7fd7f0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903fff74723ad4af server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:46.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.538+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80052d0 con 0x7fd7f00069d0 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.538+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f802e050 con 0x7fd7f00069d0 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.538+0000 7fd80fe00640 1 -- 172.21.15.39:0/1163107246 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8003830 con 0x7fd7f00069d0 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.539+0000 7fd816c00640 1 -- 172.21.15.39:0/1163107246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 msgr2=0x7fd7f0006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.539+0000 7fd816c00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd7f0006e50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8001750 tx=0x7fd7f0007a00 comp rx=0 tx=0).stop 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.539+0000 7fd816c00640 1 -- 172.21.15.39:0/1163107246 shutdown_connections 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.539+0000 7fd816c00640 1 --2- 172.21.15.39:0/1163107246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd7f0006e50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.539+0000 7fd816c00640 1 -- 172.21.15.39:0/1163107246 >> 172.21.15.39:0/1163107246 conn(0x7fd81006a090 msgr2=0x7fd81006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.539+0000 7fd816c00640 1 -- 172.21.15.39:0/1163107246 shutdown_connections 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.539+0000 7fd816c00640 1 -- 172.21.15.39:0/1163107246 wait complete. 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.541+0000 7fd816c00640 1 Processor -- start 2024-01-24T17:41:46.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.541+0000 7fd816c00640 1 -- start start 2024-01-24T17:41:46.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd816c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd810152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd816c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8101ee4a0 0x7fd810154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd816c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8100af880 0x7fd8100afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd816c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100b0240 con 0x7fd7f00069d0 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd816c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100c8690 con 0x7fd8100af880 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd816c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100c7a80 con 0x7fd8101ee4a0 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd815800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd810152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd815800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd810152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58174/0 (socket says 172.21.15.39:58174) 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.542+0000 7fd815800640 1 -- 172.21.15.39:0/3972838589 learned_addr learned my addr 172.21.15.39:0/3972838589 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd816200640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8100af880 0x7fd8100afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd814e00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8101ee4a0 0x7fd810154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd815800640 1 -- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8101ee4a0 msgr2=0x7fd810154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd815800640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8101ee4a0 0x7fd810154c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd815800640 1 -- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8100af880 msgr2=0x7fd8100afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd815800640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8100af880 0x7fd8100afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd815800640 1 -- 172.21.15.39:0/3972838589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8100c7d60 con 0x7fd7f00069d0 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd816200640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8100af880 0x7fd8100afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd815800640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd810152c80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd80000d7a0 tx=0x7fd80000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd80d600640 1 -- 172.21.15.39:0/3972838589 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8000074e0 con 0x7fd7f00069d0 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd80d600640 1 -- 172.21.15.39:0/3972838589 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd800007640 con 0x7fd7f00069d0 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd80d600640 1 -- 172.21.15.39:0/3972838589 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800004a20 con 0x7fd7f00069d0 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.543+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8100c8040 con 0x7fd7f00069d0 2024-01-24T17:41:46.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.544+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8101ee990 con 0x7fd7f00069d0 2024-01-24T17:41:46.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.545+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81004dff0 con 0x7fd7f00069d0 2024-01-24T17:41:46.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.545+0000 7fd80d600640 1 -- 172.21.15.39:0/3972838589 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd800004b80 con 0x7fd7f00069d0 2024-01-24T17:41:46.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.546+0000 7fd80d600640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd7e806a430 0x7fd7e806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:46.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.546+0000 7fd80d600640 1 -- 172.21.15.39:0/3972838589 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd800088200 con 0x7fd7f00069d0 2024-01-24T17:41:46.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.547+0000 7fd814e00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd7e806a430 0x7fd7e806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:46.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.548+0000 7fd814e00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd7e806a430 0x7fd7e806c8f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8033000 tx=0x7fd7f8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:46.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.550+0000 7fd80d600640 1 -- 172.21.15.39:0/3972838589 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd8000537e0 con 0x7fd7f00069d0 2024-01-24T17:41:46.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.781+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8100ae010 con 0x7fd7f00069d0 2024-01-24T17:41:46.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.782+0000 7fd80d600640 1 -- 172.21.15.39:0/3972838589 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7fd800056e00 con 0x7fd7f00069d0 2024-01-24T17:41:46.783 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:46.783 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:41:33.354808+0000","modified":"2024-01-24T17:41:36.370204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-24T17:41:46.783 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd7e806a430 msgr2=0x7fd7e806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd7e806a430 0x7fd7e806c8f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8033000 tx=0x7fd7f8009030 comp rx=0 tx=0).stop 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 msgr2=0x7fd810152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd810152c80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd80000d7a0 tx=0x7fd80000dc60 comp rx=0 tx=0).stop 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 shutdown_connections 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd8100af880 0x7fd8100afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8101ee4a0 0x7fd810154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd7e806a430 0x7fd7e806c8f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 --2- 172.21.15.39:0/3972838589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7f00069d0 0x7fd810152c80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 >> 172.21.15.39:0/3972838589 conn(0x7fd81006a090 msgr2=0x7fd81006a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 shutdown_connections 2024-01-24T17:41:46.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:46.785+0000 7fd816c00640 1 -- 172.21.15.39:0/3972838589 wait complete. 2024-01-24T17:41:46.804 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:47.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.023+0000 7efe33200640 1 Processor -- start 2024-01-24T17:41:47.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.024+0000 7efe33200640 1 -- start start 2024-01-24T17:41:47.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.025+0000 7efe33200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efe340ce8e0 0x7efe340ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.025+0000 7efe33200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe340cf260 con 0x7efe3415ab00 2024-01-24T17:41:47.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.025+0000 7efe33200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe340ce400 con 0x7efe3415a720 2024-01-24T17:41:47.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.025+0000 7efe33200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe340ccef0 con 0x7efe340ce8e0 2024-01-24T17:41:47.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.025+0000 7efe2be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efe3415ab00 0x7efe340cc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54418/0 (socket says 172.21.15.39:54418) 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe2be00640 1 -- 172.21.15.39:0/252772032 learned_addr learned my addr 172.21.15.39:0/252772032 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe32800640 1 --2- 172.21.15.39:0/252772032 >> v2:172.21.15.100:3300/0 conn(0x7efe340ce8e0 0x7efe340ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941688369 0 0) 0x7efe340cf260 con 0x7efe3415ab00 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe14003520 con 0x7efe3415ab00 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748093020 0 0) 0x7efe14003520 con 0x7efe3415ab00 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 >> v2:172.21.15.100:3300/0 conn(0x7efe340ce8e0 msgr2=0x7efe340ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 --2- 172.21.15.39:0/252772032 >> v2:172.21.15.100:3300/0 conn(0x7efe340ce8e0 0x7efe340ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 >> v1:172.21.15.100:6789/0 conn(0x7efe3415a720 legacy=0x7efe340cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3415b3b0 con 0x7efe3415ab00 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7efe1c001690 con 0x7efe3415ab00 2024-01-24T17:41:47.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.026+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 >> v1:172.21.15.39:6789/0 conn(0x7efe3415ab00 legacy=0x7efe340cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe31400640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe14003970 0x7efe14003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe31400640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140047c0 0x7efe14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe31400640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe140052a0 0x7efe1400fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe340ccef0 con 0x7efe140052a0 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe14010580 con 0x7efe14003970 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe14010740 con 0x7efe140047c0 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe32800640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe140052a0 0x7efe1400fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe31e00640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe14003970 0x7efe14003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe2be00640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140047c0 0x7efe14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe32800640 1 -- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140047c0 msgr2=0x7efe14004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe32800640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140047c0 0x7efe14004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe32800640 1 -- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe14003970 msgr2=0x7efe14003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe32800640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe14003970 0x7efe14003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe32800640 1 -- 172.21.15.39:0/252772032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe14003520 con 0x7efe140052a0 2024-01-24T17:41:47.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.027+0000 7efe31e00640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe14003970 0x7efe14003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:47.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.028+0000 7efe32800640 1 -- 172.21.15.39:0/252772032 reap_dead start 2024-01-24T17:41:47.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.028+0000 7efe32800640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe140052a0 0x7efe1400fd90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efe2c007330 tx=0x7efe140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f0329fa387f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.029+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2c02ccd0 con 0x7efe140052a0 2024-01-24T17:41:47.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.029+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2c02ce30 con 0x7efe140052a0 2024-01-24T17:41:47.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.029+0000 7efe31400640 1 -- 172.21.15.39:0/252772032 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2c003bd0 con 0x7efe140052a0 2024-01-24T17:41:47.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.029+0000 7efe33200640 1 -- 172.21.15.39:0/252772032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe140052a0 msgr2=0x7efe1400fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.029+0000 7efe33200640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe140052a0 0x7efe1400fd90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efe2c007330 tx=0x7efe140057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.029+0000 7efe33200640 1 -- 172.21.15.39:0/252772032 shutdown_connections 2024-01-24T17:41:47.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.029+0000 7efe33200640 1 --2- 172.21.15.39:0/252772032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe140052a0 0x7efe1400fd90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.029+0000 7efe33200640 1 -- 172.21.15.39:0/252772032 >> 172.21.15.39:0/252772032 conn(0x7efe3406a090 msgr2=0x7efe3406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:47.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.030+0000 7efe33200640 1 -- 172.21.15.39:0/252772032 shutdown_connections 2024-01-24T17:41:47.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.030+0000 7efe33200640 1 -- 172.21.15.39:0/252772032 wait complete. 2024-01-24T17:41:47.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.031+0000 7efe33200640 1 Processor -- start 2024-01-24T17:41:47.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.031+0000 7efe33200640 1 -- start start 2024-01-24T17:41:47.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.032+0000 7efe33200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140052a0 0x7efe34153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.032+0000 7efe33200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe340c7f90 0x7efe340c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.032+0000 7efe33200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe340c60f0 0x7efe341595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.032+0000 7efe33200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe34153fd0 con 0x7efe340c7f90 2024-01-24T17:41:47.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.032+0000 7efe33200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe340c9e70 con 0x7efe340c60f0 2024-01-24T17:41:47.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.032+0000 7efe33200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe340ca3f0 con 0x7efe140052a0 2024-01-24T17:41:47.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe340c7f90 0x7efe340c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe340c7f90 0x7efe340c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58194/0 (socket says 172.21.15.39:58194) 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 -- 172.21.15.39:0/365163723 learned_addr learned my addr 172.21.15.39:0/365163723 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe31e00640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140052a0 0x7efe34153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe32800640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe340c60f0 0x7efe341595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 -- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140052a0 msgr2=0x7efe34153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140052a0 0x7efe34153a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 -- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe340c60f0 msgr2=0x7efe341595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe340c60f0 0x7efe341595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 -- 172.21.15.39:0/365163723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe340ca110 con 0x7efe340c7f90 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.033+0000 7efe2be00640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe340c7f90 0x7efe340c9930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00d7e0 tx=0x7efe1c00dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.034+0000 7efe2a000640 1 -- 172.21.15.39:0/365163723 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c0074e0 con 0x7efe340c7f90 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.034+0000 7efe2a000640 1 -- 172.21.15.39:0/365163723 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c007640 con 0x7efe340c7f90 2024-01-24T17:41:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.034+0000 7efe2a000640 1 -- 172.21.15.39:0/365163723 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c0049e0 con 0x7efe340c7f90 2024-01-24T17:41:47.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.034+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe340c5610 con 0x7efe340c7f90 2024-01-24T17:41:47.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.034+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe340c5bd0 con 0x7efe340c7f90 2024-01-24T17:41:47.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.035+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe340c5df0 con 0x7efe340c7f90 2024-01-24T17:41:47.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.036+0000 7efe2a000640 1 -- 172.21.15.39:0/365163723 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efe1c004b40 con 0x7efe340c7f90 2024-01-24T17:41:47.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.037+0000 7efe2a000640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe0c06a430 0x7efe0c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.037+0000 7efe2a000640 1 -- 172.21.15.39:0/365163723 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe1c088300 con 0x7efe340c7f90 2024-01-24T17:41:47.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.037+0000 7efe31e00640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe0c06a430 0x7efe0c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.038+0000 7efe31e00640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe0c06a430 0x7efe0c06c8f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe24000f60 tx=0x7efe24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.041+0000 7efe2a000640 1 -- 172.21.15.39:0/365163723 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efe1c0538e0 con 0x7efe340c7f90 2024-01-24T17:41:47.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.237+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe3404dff0 con 0x7efe340c7f90 2024-01-24T17:41:47.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.239+0000 7efe2a000640 1 -- 172.21.15.39:0/365163723 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7efe1c056f00 con 0x7efe340c7f90 2024-01-24T17:41:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:47.240 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:47.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe0c06a430 msgr2=0x7efe0c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe0c06a430 0x7efe0c06c8f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efe24000f60 tx=0x7efe24009040 comp rx=0 tx=0).stop 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe340c7f90 msgr2=0x7efe340c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe340c7f90 0x7efe340c9930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00d7e0 tx=0x7efe1c00dca0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 shutdown_connections 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe340c60f0 0x7efe341595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe0c06a430 0x7efe0c06c8f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe340c7f90 0x7efe340c9930 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 --2- 172.21.15.39:0/365163723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140052a0 0x7efe34153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 >> 172.21.15.39:0/365163723 conn(0x7efe3406a090 msgr2=0x7efe340cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 shutdown_connections 2024-01-24T17:41:47.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.242+0000 7efe33200640 1 -- 172.21.15.39:0/365163723 wait complete. 2024-01-24T17:41:47.262 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:41:47.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:41:47.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.478+0000 7f7f21800640 1 Processor -- start 2024-01-24T17:41:47.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.478+0000 7f7f21800640 1 -- start start 2024-01-24T17:41:47.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.479+0000 7f7f21800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c15c8d0 0x7f7f1c0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.479+0000 7f7f21800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1c0b38a0 con 0x7f7f1c0b2710 2024-01-24T17:41:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.480+0000 7f7f21800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1c0d13e0 con 0x7f7f1c0d2e80 2024-01-24T17:41:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.480+0000 7f7f21800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c0d0ab0 con 0x7f7f1c15c8d0 2024-01-24T17:41:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.480+0000 7f7f1b400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7f1c0b2710 0x7f7f1c0d2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54424/0 (socket says 172.21.15.39:54424) 2024-01-24T17:41:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.480+0000 7f7f1b400640 1 -- 172.21.15.39:0/621740818 learned_addr learned my addr 172.21.15.39:0/621740818 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.480+0000 7f7f1be00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c15c8d0 0x7f7f1c0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.480+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509202961 0 0) 0x7f7f1c0b38a0 con 0x7f7f1c0b2710 2024-01-24T17:41:47.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.480+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7efc003520 con 0x7f7f1c0b2710 2024-01-24T17:41:47.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 244640200 0 0) 0x7f7efc003520 con 0x7f7f1c0b2710 2024-01-24T17:41:47.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c15c8d0 msgr2=0x7f7f1c0d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c15c8d0 0x7f7f1c0d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 >> v1:172.21.15.100:6789/0 conn(0x7f7f1c0d2e80 legacy=0x7f7f1c0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c0d0d90 con 0x7f7f1c0b2710 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7f040019f0 con 0x7f7f1c0b2710 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 >> v1:172.21.15.39:6789/0 conn(0x7f7f1c0b2710 legacy=0x7f7f1c0d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7efc0039a0 0x7f7efc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0047c0 0x7f7efc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c15c8d0 0x7f7efc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c0d0ab0 con 0x7f7efc0039a0 2024-01-24T17:41:47.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc00ba30 con 0x7f7efc0047c0 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc00bbf0 con 0x7f7f1c15c8d0 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1be00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7efc0039a0 0x7f7efc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1b400640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0047c0 0x7f7efc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1be00640 1 -- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c15c8d0 msgr2=0x7f7efc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.481+0000 7f7f1be00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c15c8d0 0x7f7efc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.482+0000 7f7f1be00640 1 -- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0047c0 msgr2=0x7f7efc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.482+0000 7f7f1be00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0047c0 0x7f7efc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.482+0000 7f7f1be00640 1 -- 172.21.15.39:0/621740818 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc003520 con 0x7f7efc0039a0 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.482+0000 7f7f1b400640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0047c0 0x7f7efc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:47.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.483+0000 7f7f1be00640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7efc0039a0 0x7f7efc003dc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009a90 tx=0x7f7efc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24241655e9dcdd95 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.483+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c02c6b0 con 0x7f7efc0039a0 2024-01-24T17:41:47.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.483+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c02c810 con 0x7f7efc0039a0 2024-01-24T17:41:47.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.483+0000 7f7f1aa00640 1 -- 172.21.15.39:0/621740818 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c02cba0 con 0x7f7efc0039a0 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.484+0000 7f7f21800640 1 -- 172.21.15.39:0/621740818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7efc0039a0 msgr2=0x7f7efc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.484+0000 7f7f21800640 1 --2- 172.21.15.39:0/621740818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7efc0039a0 0x7f7efc003dc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009a90 tx=0x7f7efc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.484+0000 7f7f20e00640 1 -- 172.21.15.39:0/621740818 reap_dead start 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.484+0000 7f7f21800640 1 -- 172.21.15.39:0/621740818 shutdown_connections 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.484+0000 7f7f21800640 1 -- 172.21.15.39:0/621740818 >> 172.21.15.39:0/621740818 conn(0x7f7f1c071dd0 msgr2=0x7f7f1c09bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.484+0000 7f7f21800640 1 -- 172.21.15.39:0/621740818 shutdown_connections 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.484+0000 7f7f21800640 1 -- 172.21.15.39:0/621740818 wait complete. 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.486+0000 7f7f21800640 1 Processor -- start 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.486+0000 7f7f21800640 1 -- start start 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f21800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0039a0 0x7f7f1c151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f21800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1c15f1a0 0x7f7f1c0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f21800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c0ca950 0x7f7f1c0b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f21800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c152440 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f21800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c0b5510 con 0x7f7efc0039a0 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f21800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c0b57f0 con 0x7f7f1c0ca950 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f1b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1c15f1a0 0x7f7f1c0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f1b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1c15f1a0 0x7f7f1c0ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58210/0 (socket says 172.21.15.39:58210) 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f1b400640 1 -- 172.21.15.39:0/1278789728 learned_addr learned my addr 172.21.15.39:0/1278789728 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f1b400640 1 -- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c0ca950 msgr2=0x7f7f1c0b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f1b400640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c0ca950 0x7f7f1c0b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f1b400640 1 -- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0039a0 msgr2=0x7f7f1c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f1b400640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0039a0 0x7f7f1c151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.487+0000 7f7f1b400640 1 -- 172.21.15.39:0/1278789728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c0b5ad0 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.488+0000 7f7f1b400640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1c15f1a0 0x7f7f1c0ca410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400dfc0 tx=0x7f7f0400b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.488+0000 7f7f13e00640 1 -- 172.21.15.39:0/1278789728 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04004d60 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.488+0000 7f7f13e00640 1 -- 172.21.15.39:0/1278789728 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f040070d0 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.488+0000 7f7f13e00640 1 -- 172.21.15.39:0/1278789728 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f04007480 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.488+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c0b5db0 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.488+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1c15eac0 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.489+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1c04dff0 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.490+0000 7f7f13e00640 1 -- 172.21.15.39:0/1278789728 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7f040075e0 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.491+0000 7f7f13e00640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ef406a460 0x7f7ef406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.491+0000 7f7f13e00640 1 -- 172.21.15.39:0/1278789728 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f04088a40 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.492+0000 7f7f1be00640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ef406a460 0x7f7ef406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.493+0000 7f7f1be00640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ef406a460 0x7f7ef406c920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c036000 tx=0x7f7f0c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.496+0000 7f7f13e00640 1 -- 172.21.15.39:0/1278789728 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7f0408ae60 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.688+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f1c0b0ab0 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.689+0000 7f7f13e00640 1 -- 172.21.15.39:0/1278789728 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f04019020 con 0x7f7f1c15f1a0 2024-01-24T17:41:47.689 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:41:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ef406a460 msgr2=0x7f7ef406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ef406a460 0x7f7ef406c920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c036000 tx=0x7f7f0c02e000 comp rx=0 tx=0).stop 2024-01-24T17:41:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1c15f1a0 msgr2=0x7f7f1c0ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1c15f1a0 0x7f7f1c0ca410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400dfc0 tx=0x7f7f0400b560 comp rx=0 tx=0).stop 2024-01-24T17:41:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 shutdown_connections 2024-01-24T17:41:47.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1c0ca950 0x7f7f1c0b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ef406a460 0x7f7ef406c920 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1c15f1a0 0x7f7f1c0ca410 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 --2- 172.21.15.39:0/1278789728 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7efc0039a0 0x7f7f1c151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 >> 172.21.15.39:0/1278789728 conn(0x7f7f1c071dd0 msgr2=0x7f7f1c0b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 shutdown_connections 2024-01-24T17:41:47.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.691+0000 7f7f21800640 1 -- 172.21.15.39:0/1278789728 wait complete. 2024-01-24T17:41:47.711 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:41:47.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.931+0000 7efd52000640 1 Processor -- start 2024-01-24T17:41:47.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.931+0000 7efd52000640 1 -- start start 2024-01-24T17:41:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd52000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 0x7efd4c0b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd52000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4c0cda90 con 0x7efd4c0d14c0 2024-01-24T17:41:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd52000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4c0d32c0 con 0x7efd4c15fff0 2024-01-24T17:41:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd52000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c0b30a0 con 0x7efd4c0b3660 2024-01-24T17:41:47.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd4aa00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efd4c0d14c0 0x7efd4c0cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54434/0 (socket says 172.21.15.39:54434) 2024-01-24T17:41:47.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd4aa00640 1 -- 172.21.15.39:0/1034093605 learned_addr learned my addr 172.21.15.39:0/1034093605 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:47.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395165004 0 0) 0x7efd4c0cda90 con 0x7efd4c0d14c0 2024-01-24T17:41:47.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2c003520 con 0x7efd4c0d14c0 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.933+0000 7efd4be00640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 0x7efd4c0b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 670996162 0 0) 0x7efd2c003520 con 0x7efd4c0d14c0 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 msgr2=0x7efd4c0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 0x7efd4c0b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 >> v1:172.21.15.100:6789/0 conn(0x7efd4c15fff0 legacy=0x7efd4c0d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4c0b3380 con 0x7efd4c0d14c0 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7efd34001670 con 0x7efd4c0d14c0 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 >> v1:172.21.15.39:6789/0 conn(0x7efd4c0d14c0 legacy=0x7efd4c0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c0039a0 0x7efd2c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 0x7efd2c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd2c004d00 0x7efd2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c0b30a0 con 0x7efd2c004d00 2024-01-24T17:41:47.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c00bb60 con 0x7efd2c0039a0 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c00bd20 con 0x7efd4c0b3660 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4be00640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd2c004d00 0x7efd2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4aa00640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 0x7efd2c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4b400640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c0039a0 0x7efd2c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.934+0000 7efd4be00640 1 -- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 msgr2=0x7efd2c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.935+0000 7efd4be00640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 0x7efd2c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.935+0000 7efd4be00640 1 -- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c0039a0 msgr2=0x7efd2c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.935+0000 7efd4be00640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c0039a0 0x7efd2c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.935+0000 7efd4be00640 1 -- 172.21.15.39:0/1034093605 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c003520 con 0x7efd2c004d00 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.935+0000 7efd4b400640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c0039a0 0x7efd2c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.935+0000 7efd4aa00640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0b3660 0x7efd2c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.936+0000 7efd4be00640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd2c004d00 0x7efd2c005180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efd44007360 tx=0x7efd2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fd821580dafe0f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.936+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4402ccb0 con 0x7efd2c004d00 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.936+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4402ce10 con 0x7efd2c004d00 2024-01-24T17:41:47.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.936+0000 7efd4a000640 1 -- 172.21.15.39:0/1034093605 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44034660 con 0x7efd2c004d00 2024-01-24T17:41:47.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.937+0000 7efd52000640 1 -- 172.21.15.39:0/1034093605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd2c004d00 msgr2=0x7efd2c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.937+0000 7efd52000640 1 --2- 172.21.15.39:0/1034093605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd2c004d00 0x7efd2c005180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efd44007360 tx=0x7efd2c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.937+0000 7efd4be00640 1 -- 172.21.15.39:0/1034093605 reap_dead start 2024-01-24T17:41:47.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.937+0000 7efd52000640 1 -- 172.21.15.39:0/1034093605 shutdown_connections 2024-01-24T17:41:47.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.937+0000 7efd52000640 1 -- 172.21.15.39:0/1034093605 >> 172.21.15.39:0/1034093605 conn(0x7efd4c09bcc0 msgr2=0x7efd4c0d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:47.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.937+0000 7efd52000640 1 -- 172.21.15.39:0/1034093605 shutdown_connections 2024-01-24T17:41:47.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.937+0000 7efd52000640 1 -- 172.21.15.39:0/1034093605 wait complete. 2024-01-24T17:41:47.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.938+0000 7efd52000640 1 Processor -- start 2024-01-24T17:41:47.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.939+0000 7efd52000640 1 -- start start 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.939+0000 7efd52000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c004d00 0x7efd4c1f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd52000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4c1f2ce0 0x7efd4c0c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd52000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0c9f00 0x7efd4c0c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd52000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c0ca3b0 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd52000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c0c6380 con 0x7efd2c004d00 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd52000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c0c65f0 con 0x7efd4c0c9f00 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4c1f2ce0 0x7efd4c0c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4c1f2ce0 0x7efd4c0c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58224/0 (socket says 172.21.15.39:58224) 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4aa00640 1 -- 172.21.15.39:0/1782933485 learned_addr learned my addr 172.21.15.39:0/1782933485 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4b400640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c004d00 0x7efd4c1f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4be00640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0c9f00 0x7efd4c0c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4aa00640 1 -- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0c9f00 msgr2=0x7efd4c0c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4aa00640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0c9f00 0x7efd4c0c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4aa00640 1 -- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c004d00 msgr2=0x7efd4c1f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4aa00640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c004d00 0x7efd4c1f34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.940+0000 7efd4aa00640 1 -- 172.21.15.39:0/1782933485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4c0c68d0 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.941+0000 7efd4aa00640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4c1f2ce0 0x7efd4c0c99c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efd3400d890 tx=0x7efd3400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.941+0000 7efd43400640 1 -- 172.21.15.39:0/1782933485 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340074e0 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.941+0000 7efd43400640 1 -- 172.21.15.39:0/1782933485 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd34007640 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.941+0000 7efd43400640 1 -- 172.21.15.39:0/1782933485 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340049e0 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.941+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4c0c6bb0 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.941+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4c1f3160 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.942+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd4c0ccf20 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.943+0000 7efd43400640 1 -- 172.21.15.39:0/1782933485 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efd34004b40 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.944+0000 7efd43400640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2406a460 0x7efd2406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:47.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.944+0000 7efd43400640 1 -- 172.21.15.39:0/1782933485 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd340883b0 con 0x7efd4c1f2ce0 2024-01-24T17:41:47.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.944+0000 7efd4b400640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2406a460 0x7efd2406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:47.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.945+0000 7efd4b400640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2406a460 0x7efd2406c920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0b2fd0 tx=0x7efd3c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:47.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:47.948+0000 7efd43400640 1 -- 172.21.15.39:0/1782933485 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efd34053900 con 0x7efd4c1f2ce0 2024-01-24T17:41:48.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.142+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efd4c073ef0 con 0x7efd4c1f2ce0 2024-01-24T17:41:48.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.143+0000 7efd43400640 1 -- 172.21.15.39:0/1782933485 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7efd34056f20 con 0x7efd4c1f2ce0 2024-01-24T17:41:48.144 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:41:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2406a460 msgr2=0x7efd2406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2406a460 0x7efd2406c920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0b2fd0 tx=0x7efd3c009040 comp rx=0 tx=0).stop 2024-01-24T17:41:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4c1f2ce0 msgr2=0x7efd4c0c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4c1f2ce0 0x7efd4c0c99c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efd3400d890 tx=0x7efd3400dd50 comp rx=0 tx=0).stop 2024-01-24T17:41:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 shutdown_connections 2024-01-24T17:41:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd4c0c9f00 0x7efd4c0c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2406a460 0x7efd2406c920 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4c1f2ce0 0x7efd4c0c99c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 --2- 172.21.15.39:0/1782933485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd2c004d00 0x7efd4c1f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 >> 172.21.15.39:0/1782933485 conn(0x7efd4c09bcc0 msgr2=0x7efd4c0cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:48.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 shutdown_connections 2024-01-24T17:41:48.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.146+0000 7efd52000640 1 -- 172.21.15.39:0/1782933485 wait complete. 2024-01-24T17:41:48.168 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:41:48.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.392+0000 7efe9b200640 1 Processor -- start 2024-01-24T17:41:48.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.392+0000 7efe9b200640 1 -- start start 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.393+0000 7efe9b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe940b0350 0x7efe9406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.393+0000 7efe9b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efe9415e190 0x7efe940b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.393+0000 7efe9b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9406d4c0 con 0x7efe9406c9f0 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.393+0000 7efe9b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe9406cf00 con 0x7efe9415e190 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.393+0000 7efe9b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9406d1e0 con 0x7efe940b0350 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.394+0000 7efe99200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efe9415e190 0x7efe940b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.394+0000 7efe99200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efe9415e190 0x7efe940b1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58230/0 (socket says 172.21.15.39:58230) 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.394+0000 7efe99200640 1 -- 172.21.15.39:0/2986246838 learned_addr learned my addr 172.21.15.39:0/2986246838 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.394+0000 7efe93e00640 1 --2- 172.21.15.39:0/2986246838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe940b0350 0x7efe9406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.394+0000 7efe99200640 1 -- 172.21.15.39:0/2986246838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe940b0350 msgr2=0x7efe9406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.394+0000 7efe99200640 1 --2- 172.21.15.39:0/2986246838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe940b0350 0x7efe9406c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.394+0000 7efe99200640 1 -- 172.21.15.39:0/2986246838 >> v1:172.21.15.39:6789/0 conn(0x7efe9406c9f0 legacy=0x7efe9415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.394+0000 7efe99200640 1 -- 172.21.15.39:0/2986246838 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe940b15e0 con 0x7efe9415e190 2024-01-24T17:41:48.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.395+0000 7efe99200640 1 --2- 172.21.15.39:0/2986246838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe9415e190 0x7efe940b1de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009870 tx=0x7efe8c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3814051076c334 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:48.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe92a00640 1 -- 172.21.15.39:0/2986246838 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c031640 con 0x7efe9415e190 2024-01-24T17:41:48.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe92a00640 1 -- 172.21.15.39:0/2986246838 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c0317a0 con 0x7efe9415e190 2024-01-24T17:41:48.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe92a00640 1 -- 172.21.15.39:0/2986246838 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c031b30 con 0x7efe9415e190 2024-01-24T17:41:48.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe9b200640 1 -- 172.21.15.39:0/2986246838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe9415e190 msgr2=0x7efe940b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe9b200640 1 --2- 172.21.15.39:0/2986246838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe9415e190 0x7efe940b1de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009870 tx=0x7efe8c02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe9b200640 1 -- 172.21.15.39:0/2986246838 shutdown_connections 2024-01-24T17:41:48.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe9b200640 1 --2- 172.21.15.39:0/2986246838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe9415e190 0x7efe940b1de0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe9b200640 1 --2- 172.21.15.39:0/2986246838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe940b0350 0x7efe9406c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.396+0000 7efe9b200640 1 -- 172.21.15.39:0/2986246838 >> 172.21.15.39:0/2986246838 conn(0x7efe940cb0e0 msgr2=0x7efe940cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:48.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.397+0000 7efe9b200640 1 -- 172.21.15.39:0/2986246838 shutdown_connections 2024-01-24T17:41:48.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.397+0000 7efe9b200640 1 -- 172.21.15.39:0/2986246838 wait complete. 2024-01-24T17:41:48.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.398+0000 7efe9b200640 1 Processor -- start 2024-01-24T17:41:48.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.398+0000 7efe9b200640 1 -- start start 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.399+0000 7efe9b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe9406c9f0 0x7efe94153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.399+0000 7efe9b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe940b0350 0x7efe940cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.399+0000 7efe9b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe9415e190 0x7efe940ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe9b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe941535d0 con 0x7efe940b0350 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe9b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe940cc680 con 0x7efe9415e190 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe9b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe940cc960 con 0x7efe9406c9f0 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe940b0350 0x7efe940cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe940b0350 0x7efe940cbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58234/0 (socket says 172.21.15.39:58234) 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93400640 1 -- 172.21.15.39:0/4014142259 learned_addr learned my addr 172.21.15.39:0/4014142259 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe99200640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe9415e190 0x7efe940ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93e00640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe9406c9f0 0x7efe94153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93400640 1 -- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe9406c9f0 msgr2=0x7efe94153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93400640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe9406c9f0 0x7efe94153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93400640 1 -- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe9415e190 msgr2=0x7efe940ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93400640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe9415e190 0x7efe940ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93400640 1 -- 172.21.15.39:0/4014142259 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c0094d0 con 0x7efe940b0350 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.400+0000 7efe93e00640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe9406c9f0 0x7efe94153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.401+0000 7efe93400640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe940b0350 0x7efe940cbe30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00c400 tx=0x7efe7c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.401+0000 7efe8be00640 1 -- 172.21.15.39:0/4014142259 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c0052d0 con 0x7efe940b0350 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.401+0000 7efe8be00640 1 -- 172.21.15.39:0/4014142259 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c005430 con 0x7efe940b0350 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.401+0000 7efe8be00640 1 -- 172.21.15.39:0/4014142259 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c003830 con 0x7efe940b0350 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.401+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe940ccc40 con 0x7efe940b0350 2024-01-24T17:41:48.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.401+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe940c7350 con 0x7efe940b0350 2024-01-24T17:41:48.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.402+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe94073ef0 con 0x7efe940b0350 2024-01-24T17:41:48.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.403+0000 7efe8be00640 1 -- 172.21.15.39:0/4014142259 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efe7c003990 con 0x7efe940b0350 2024-01-24T17:41:48.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.404+0000 7efe8be00640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe6c06a430 0x7efe6c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.404+0000 7efe8be00640 1 -- 172.21.15.39:0/4014142259 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe7c087f20 con 0x7efe940b0350 2024-01-24T17:41:48.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.404+0000 7efe93e00640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe6c06a430 0x7efe6c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.405+0000 7efe93e00640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe6c06a430 0x7efe6c06c8f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe84007330 tx=0x7efe8400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:48.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.408+0000 7efe8be00640 1 -- 172.21.15.39:0/4014142259 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efe7c053500 con 0x7efe940b0350 2024-01-24T17:41:48.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.603+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efe9406de70 con 0x7efe940b0350 2024-01-24T17:41:48.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.604+0000 7efe8be00640 1 -- 172.21.15.39:0/4014142259 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7efe7c017020 con 0x7efe940b0350 2024-01-24T17:41:48.604 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:41:48.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe6c06a430 msgr2=0x7efe6c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe6c06a430 0x7efe6c06c8f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe84007330 tx=0x7efe8400b040 comp rx=0 tx=0).stop 2024-01-24T17:41:48.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe940b0350 msgr2=0x7efe940cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe940b0350 0x7efe940cbe30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00c400 tx=0x7efe7c00c8c0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 shutdown_connections 2024-01-24T17:41:48.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe9415e190 0x7efe940ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe6c06a430 0x7efe6c06c8f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe940b0350 0x7efe940cbe30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 --2- 172.21.15.39:0/4014142259 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe9406c9f0 0x7efe94153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 >> 172.21.15.39:0/4014142259 conn(0x7efe940cb0e0 msgr2=0x7efe940c2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 shutdown_connections 2024-01-24T17:41:48.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.606+0000 7efe9b200640 1 -- 172.21.15.39:0/4014142259 wait complete. 2024-01-24T17:41:48.626 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:41:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.848+0000 7f04de400640 1 Processor -- start 2024-01-24T17:41:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.848+0000 7f04de400640 1 -- start start 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.850+0000 7f04de400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f04d815cbf0 0x7f04d80d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.850+0000 7f04de400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d80d0da0 0x7f04d80d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.850+0000 7f04de400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d815e010 con 0x7f04d815db60 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.850+0000 7f04de400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d80d1770 con 0x7f04d815cbf0 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.850+0000 7f04de400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80d0a90 con 0x7f04d80d0da0 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.850+0000 7f04dd000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f04d815cbf0 0x7f04d80d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.850+0000 7f04dd000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f04d815cbf0 0x7f04d80d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58240/0 (socket says 172.21.15.39:58240) 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.850+0000 7f04dd000640 1 -- 172.21.15.39:0/780452726 learned_addr learned my addr 172.21.15.39:0/780452726 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:48.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.851+0000 7f04dda00640 1 --2- 172.21.15.39:0/780452726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d80d0da0 0x7f04d80d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.851+0000 7f04dd000640 1 -- 172.21.15.39:0/780452726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d80d0da0 msgr2=0x7f04d80d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.851+0000 7f04dd000640 1 --2- 172.21.15.39:0/780452726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d80d0da0 0x7f04d80d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.851+0000 7f04dd000640 1 -- 172.21.15.39:0/780452726 >> v1:172.21.15.100:6789/0 conn(0x7f04d815db60 legacy=0x7f04d80ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.851+0000 7f04dd000640 1 -- 172.21.15.39:0/780452726 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d80cd760 con 0x7f04d815cbf0 2024-01-24T17:41:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.852+0000 7f04dd000640 1 --2- 172.21.15.39:0/780452726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815cbf0 0x7f04d80d03b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009880 tx=0x7f04c802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29296bd8182961f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.852+0000 7f04d7400640 1 -- 172.21.15.39:0/780452726 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8004230 con 0x7f04d815cbf0 2024-01-24T17:41:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.852+0000 7f04d7400640 1 -- 172.21.15.39:0/780452726 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8004390 con 0x7f04d815cbf0 2024-01-24T17:41:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.852+0000 7f04d7400640 1 -- 172.21.15.39:0/780452726 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8003810 con 0x7f04d815cbf0 2024-01-24T17:41:48.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.853+0000 7f04de400640 1 -- 172.21.15.39:0/780452726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815cbf0 msgr2=0x7f04d80d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.853+0000 7f04de400640 1 --2- 172.21.15.39:0/780452726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815cbf0 0x7f04d80d03b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009880 tx=0x7f04c802b960 comp rx=0 tx=0).stop 2024-01-24T17:41:48.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.853+0000 7f04de400640 1 -- 172.21.15.39:0/780452726 shutdown_connections 2024-01-24T17:41:48.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.853+0000 7f04de400640 1 --2- 172.21.15.39:0/780452726 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d80d0da0 0x7f04d80d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.853+0000 7f04de400640 1 --2- 172.21.15.39:0/780452726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815cbf0 0x7f04d80d03b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.853+0000 7f04de400640 1 -- 172.21.15.39:0/780452726 >> 172.21.15.39:0/780452726 conn(0x7f04d809bb90 msgr2=0x7f04d80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:48.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.853+0000 7f04de400640 1 -- 172.21.15.39:0/780452726 shutdown_connections 2024-01-24T17:41:48.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.853+0000 7f04de400640 1 -- 172.21.15.39:0/780452726 wait complete. 2024-01-24T17:41:48.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.855+0000 7f04de400640 1 Processor -- start 2024-01-24T17:41:48.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.855+0000 7f04de400640 1 -- start start 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04de400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04d80d0da0 0x7f04d81e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04de400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d815cbf0 0x7f04d8158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04de400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815db60 0x7f04d80c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04de400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d81e7e30 con 0x7f04d815db60 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04de400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d8159420 con 0x7f04d80d0da0 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04de400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d80c91f0 con 0x7f04d815cbf0 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04dda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815db60 0x7f04d80c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04dda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815db60 0x7f04d80c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58254/0 (socket says 172.21.15.39:58254) 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04dda00640 1 -- 172.21.15.39:0/4147153355 learned_addr learned my addr 172.21.15.39:0/4147153355 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:48.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04dd000640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04d80d0da0 0x7f04d81e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.856+0000 7f04d7e00640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d815cbf0 0x7f04d8158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04dda00640 1 -- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d815cbf0 msgr2=0x7f04d8158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04dda00640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d815cbf0 0x7f04d8158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04dda00640 1 -- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04d80d0da0 msgr2=0x7f04d81e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04dda00640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04d80d0da0 0x7f04d81e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04dda00640 1 -- 172.21.15.39:0/4147153355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c8009460 con 0x7f04d815db60 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04dd000640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04d80d0da0 0x7f04d81e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04d7e00640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d815cbf0 0x7f04d8158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04dda00640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815db60 0x7f04d80c7040 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04d000d5d0 tx=0x7f04d000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04d4c00640 1 -- 172.21.15.39:0/4147153355 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d00125f0 con 0x7f04d815db60 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04d4c00640 1 -- 172.21.15.39:0/4147153355 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d0012770 con 0x7f04d815db60 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04d4c00640 1 -- 172.21.15.39:0/4147153355 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0012b20 con 0x7f04d815db60 2024-01-24T17:41:48.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d80c6310 con 0x7f04d815db60 2024-01-24T17:41:48.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.857+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d80c6840 con 0x7f04d815db60 2024-01-24T17:41:48.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.858+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d804dff0 con 0x7f04d815db60 2024-01-24T17:41:48.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.859+0000 7f04d4c00640 1 -- 172.21.15.39:0/4147153355 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f04d0012c80 con 0x7f04d815db60 2024-01-24T17:41:48.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.861+0000 7f04d4c00640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04b006a430 0x7f04b006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:48.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.861+0000 7f04d4c00640 1 -- 172.21.15.39:0/4147153355 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04d0088d10 con 0x7f04d815db60 2024-01-24T17:41:48.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.861+0000 7f04dd000640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04b006a430 0x7f04b006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:48.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.862+0000 7f04dd000640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04b006a430 0x7f04b006c8f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f04c8035000 tx=0x7f04c8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:48.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:48.865+0000 7f04d4c00640 1 -- 172.21.15.39:0/4147153355 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f04d0054310 con 0x7f04d815db60 2024-01-24T17:41:49.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.098+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04d80b4f60 con 0x7f04d815db60 2024-01-24T17:41:49.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.100+0000 7f04d4c00640 1 -- 172.21.15.39:0/4147153355 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7f04d0057930 con 0x7f04d815db60 2024-01-24T17:41:49.100 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:49.101 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:41:33.354808+0000","modified":"2024-01-24T17:41:36.370204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-24T17:41:49.101 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-01-24T17:41:49.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.102+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04b006a430 msgr2=0x7f04b006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.102+0000 7f04de400640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04b006a430 0x7f04b006c8f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f04c8035000 tx=0x7f04c8008030 comp rx=0 tx=0).stop 2024-01-24T17:41:49.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.102+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815db60 msgr2=0x7f04d80c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.102+0000 7f04de400640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815db60 0x7f04d80c7040 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04d000d5d0 tx=0x7f04d000da90 comp rx=0 tx=0).stop 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.103+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 shutdown_connections 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.103+0000 7f04de400640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04b006a430 0x7f04b006c8f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.103+0000 7f04de400640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04d815db60 0x7f04d80c7040 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.103+0000 7f04de400640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04d815cbf0 0x7f04d8158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.103+0000 7f04de400640 1 --2- 172.21.15.39:0/4147153355 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04d80d0da0 0x7f04d81e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.103+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 >> 172.21.15.39:0/4147153355 conn(0x7f04d809bb90 msgr2=0x7f04d80b7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.103+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 shutdown_connections 2024-01-24T17:41:49.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.103+0000 7f04de400640 1 -- 172.21.15.39:0/4147153355 wait complete. 2024-01-24T17:41:49.120 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:41:49.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.341+0000 7f9a51400640 1 Processor -- start 2024-01-24T17:41:49.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.342+0000 7f9a51400640 1 -- start start 2024-01-24T17:41:49.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.343+0000 7f9a51400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9a4c15d020 0x7f9a4c15d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.343+0000 7f9a51400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c15dac0 con 0x7f9a4c15eb80 2024-01-24T17:41:49.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a51400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4c15dda0 con 0x7f9a4c0cf990 2024-01-24T17:41:49.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a51400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4c15e080 con 0x7f9a4c15d020 2024-01-24T17:41:49.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a4b200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9a4c15eb80 0x7f9a4c15f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54458/0 (socket says 172.21.15.39:54458) 2024-01-24T17:41:49.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a4b200640 1 -- 172.21.15.39:0/3796482648 learned_addr learned my addr 172.21.15.39:0/3796482648 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:49.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a49e00640 1 --2- 172.21.15.39:0/3796482648 >> v2:172.21.15.39:3300/0 conn(0x7f9a4c15d020 0x7f9a4c15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a49e00640 1 -- 172.21.15.39:0/3796482648 >> v1:172.21.15.100:6789/0 conn(0x7f9a4c0cf990 legacy=0x7f9a4c0cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a49e00640 1 -- 172.21.15.39:0/3796482648 >> v1:172.21.15.39:6789/0 conn(0x7f9a4c15eb80 legacy=0x7f9a4c15f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a49400640 1 -- 172.21.15.39:0/3796482648 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311216394 0 0) 0x7f9a4c15dac0 con 0x7f9a4c15eb80 2024-01-24T17:41:49.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.344+0000 7f9a49e00640 1 -- 172.21.15.39:0/3796482648 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c15e380 con 0x7f9a4c15d020 2024-01-24T17:41:49.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.345+0000 7f9a49e00640 1 --2- 172.21.15.39:0/3796482648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15d020 0x7f9a4c15d490 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009d90 tx=0x7f9a3c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4b2599ebf91967 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:49.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.346+0000 7f9a49400640 1 -- 172.21.15.39:0/3796482648 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c02cd00 con 0x7f9a4c15d020 2024-01-24T17:41:49.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.346+0000 7f9a49400640 1 -- 172.21.15.39:0/3796482648 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c02ce60 con 0x7f9a4c15d020 2024-01-24T17:41:49.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.346+0000 7f9a49400640 1 -- 172.21.15.39:0/3796482648 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c034660 con 0x7f9a4c15d020 2024-01-24T17:41:49.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.346+0000 7f9a51400640 1 -- 172.21.15.39:0/3796482648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15d020 msgr2=0x7f9a4c15d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.346+0000 7f9a51400640 1 --2- 172.21.15.39:0/3796482648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15d020 0x7f9a4c15d490 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009d90 tx=0x7f9a3c02bf50 comp rx=0 tx=0).stop 2024-01-24T17:41:49.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.346+0000 7f9a51400640 1 -- 172.21.15.39:0/3796482648 shutdown_connections 2024-01-24T17:41:49.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.346+0000 7f9a51400640 1 --2- 172.21.15.39:0/3796482648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15d020 0x7f9a4c15d490 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.346+0000 7f9a51400640 1 -- 172.21.15.39:0/3796482648 >> 172.21.15.39:0/3796482648 conn(0x7f9a4c071790 msgr2=0x7f9a4c071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:49.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.347+0000 7f9a51400640 1 -- 172.21.15.39:0/3796482648 shutdown_connections 2024-01-24T17:41:49.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.347+0000 7f9a51400640 1 -- 172.21.15.39:0/3796482648 wait complete. 2024-01-24T17:41:49.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.348+0000 7f9a51400640 1 Processor -- start 2024-01-24T17:41:49.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.348+0000 7f9a51400640 1 -- start start 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.349+0000 7f9a51400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a4c0cf990 0x7f9a4c0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.349+0000 7f9a51400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a4c15d020 0x7f9a4c1f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.349+0000 7f9a51400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15eb80 0x7f9a4c156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.349+0000 7f9a51400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0c77e0 con 0x7f9a4c15eb80 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.349+0000 7f9a51400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0d11f0 con 0x7f9a4c15d020 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.349+0000 7f9a51400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4c0d14d0 con 0x7f9a4c0cf990 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15eb80 0x7f9a4c156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15eb80 0x7f9a4c156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58260/0 (socket says 172.21.15.39:58260) 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 -- 172.21.15.39:0/2396190358 learned_addr learned my addr 172.21.15.39:0/2396190358 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4a800640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a4c0cf990 0x7f9a4c0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a49e00640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a4c15d020 0x7f9a4c1f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 -- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a4c0cf990 msgr2=0x7f9a4c0d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a4c0cf990 0x7f9a4c0d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 -- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a4c15d020 msgr2=0x7f9a4c1f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a4c15d020 0x7f9a4c1f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 -- 172.21.15.39:0/2396190358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c009a70 con 0x7f9a4c15eb80 2024-01-24T17:41:49.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4a800640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a4c0cf990 0x7f9a4c0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:41:49.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.350+0000 7f9a4b200640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15eb80 0x7f9a4c156e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9a44001eb0 tx=0x7f9a4400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:49.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.351+0000 7f9a42a00640 1 -- 172.21.15.39:0/2396190358 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a440074e0 con 0x7f9a4c15eb80 2024-01-24T17:41:49.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.351+0000 7f9a42a00640 1 -- 172.21.15.39:0/2396190358 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a44007640 con 0x7f9a4c15eb80 2024-01-24T17:41:49.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.351+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4c0d17b0 con 0x7f9a4c15eb80 2024-01-24T17:41:49.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.351+0000 7f9a42a00640 1 -- 172.21.15.39:0/2396190358 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44004a00 con 0x7f9a4c15eb80 2024-01-24T17:41:49.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.351+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4c157390 con 0x7f9a4c15eb80 2024-01-24T17:41:49.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.352+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4c0c7950 con 0x7f9a4c15eb80 2024-01-24T17:41:49.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.353+0000 7f9a42a00640 1 -- 172.21.15.39:0/2396190358 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9a44004b60 con 0x7f9a4c15eb80 2024-01-24T17:41:49.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.354+0000 7f9a42a00640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a2406a430 0x7f9a2406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.354+0000 7f9a42a00640 1 -- 172.21.15.39:0/2396190358 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a44088760 con 0x7f9a4c15eb80 2024-01-24T17:41:49.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.354+0000 7f9a4a800640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a2406a430 0x7f9a2406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.355+0000 7f9a4a800640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a2406a430 0x7f9a2406c8f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9a34007fd0 tx=0x7f9a34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:49.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.358+0000 7f9a42a00640 1 -- 172.21.15.39:0/2396190358 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9a44057740 con 0x7f9a4c15eb80 2024-01-24T17:41:49.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.558+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a4c073ef0 con 0x7f9a4c15eb80 2024-01-24T17:41:49.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.561+0000 7f9a42a00640 1 -- 172.21.15.39:0/2396190358 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f9a440572d0 con 0x7f9a4c15eb80 2024-01-24T17:41:49.561 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:49.561 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.563+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a2406a430 msgr2=0x7f9a2406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.563+0000 7f9a51400640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a2406a430 0x7f9a2406c8f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9a34007fd0 tx=0x7f9a34009040 comp rx=0 tx=0).stop 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.563+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15eb80 msgr2=0x7f9a4c156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.563+0000 7f9a51400640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15eb80 0x7f9a4c156e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9a44001eb0 tx=0x7f9a4400c870 comp rx=0 tx=0).stop 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.564+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 shutdown_connections 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.564+0000 7f9a51400640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a2406a430 0x7f9a2406c8f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.564+0000 7f9a51400640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a4c15eb80 0x7f9a4c156e50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.564+0000 7f9a51400640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a4c15d020 0x7f9a4c1f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.564+0000 7f9a51400640 1 --2- 172.21.15.39:0/2396190358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a4c0cf990 0x7f9a4c0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.564+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 >> 172.21.15.39:0/2396190358 conn(0x7f9a4c071790 msgr2=0x7f9a4c15c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.564+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 shutdown_connections 2024-01-24T17:41:49.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.564+0000 7f9a51400640 1 -- 172.21.15.39:0/2396190358 wait complete. 2024-01-24T17:41:49.582 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:41:49.670 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:41:49.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:41:49.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.774+0000 7f31ae600640 1 Processor -- start 2024-01-24T17:41:49.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.774+0000 7f31ae600640 1 -- start start 2024-01-24T17:41:49.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.775+0000 7f31ae600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f31a80b06d0 0x7f31a80ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.775+0000 7f31ae600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80710e0 0x7f31a80cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.775+0000 7f31ae600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a80cad00 con 0x7f31a8159620 2024-01-24T17:41:49.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.775+0000 7f31ae600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a8071650 con 0x7f31a80b06d0 2024-01-24T17:41:49.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.775+0000 7f31ae600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8071930 con 0x7f31a80710e0 2024-01-24T17:41:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.776+0000 7f31a7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80710e0 0x7f31a80cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.776+0000 7f31a7400640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f31a8159620 0x7f31a80affc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:59622/0 (socket says 172.21.15.39:59622) 2024-01-24T17:41:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.776+0000 7f31a7400640 1 -- 172.21.15.39:0/673897225 learned_addr learned my addr 172.21.15.39:0/673897225 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.776+0000 7f31a6a00640 1 --2- 172.21.15.39:0/673897225 >> v2:172.21.15.100:3300/0 conn(0x7f31a80b06d0 0x7f31a80ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.777+0000 7f31a6a00640 1 -- 172.21.15.39:0/673897225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80710e0 msgr2=0x7f31a80cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.777+0000 7f31a6a00640 1 --2- 172.21.15.39:0/673897225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80710e0 0x7f31a80cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.777+0000 7f31a6a00640 1 -- 172.21.15.39:0/673897225 >> v1:172.21.15.100:6789/0 conn(0x7f31a8159620 legacy=0x7f31a80affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.777+0000 7f31a6a00640 1 -- 172.21.15.39:0/673897225 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8071c10 con 0x7f31a80b06d0 2024-01-24T17:41:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.777+0000 7f31a7e00640 1 --2- 172.21.15.39:0/673897225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80710e0 0x7f31a80cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:41:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.778+0000 7f31a6a00640 1 --2- 172.21.15.39:0/673897225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b06d0 0x7f31a80ca7c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f31900097a0 tx=0x7f319002b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bed4e4061a3c0e51 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:49.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.778+0000 7f31a6000640 1 -- 172.21.15.39:0/673897225 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190030800 con 0x7f31a80b06d0 2024-01-24T17:41:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.778+0000 7f31a6000640 1 -- 172.21.15.39:0/673897225 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3190030960 con 0x7f31a80b06d0 2024-01-24T17:41:49.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.778+0000 7f31a6000640 1 -- 172.21.15.39:0/673897225 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190030cf0 con 0x7f31a80b06d0 2024-01-24T17:41:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.779+0000 7f31ae600640 1 -- 172.21.15.39:0/673897225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b06d0 msgr2=0x7f31a80ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.779+0000 7f31ae600640 1 --2- 172.21.15.39:0/673897225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b06d0 0x7f31a80ca7c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f31900097a0 tx=0x7f319002b940 comp rx=0 tx=0).stop 2024-01-24T17:41:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.779+0000 7f31ae600640 1 -- 172.21.15.39:0/673897225 shutdown_connections 2024-01-24T17:41:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.779+0000 7f31ae600640 1 --2- 172.21.15.39:0/673897225 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a80b06d0 0x7f31a80ca7c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.779+0000 7f31ae600640 1 --2- 172.21.15.39:0/673897225 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80710e0 0x7f31a80cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.779+0000 7f31ae600640 1 -- 172.21.15.39:0/673897225 >> 172.21.15.39:0/673897225 conn(0x7f31a806a3e0 msgr2=0x7f31a806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.779+0000 7f31ae600640 1 -- 172.21.15.39:0/673897225 shutdown_connections 2024-01-24T17:41:49.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.779+0000 7f31ae600640 1 -- 172.21.15.39:0/673897225 wait complete. 2024-01-24T17:41:49.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.781+0000 7f31ae600640 1 Processor -- start 2024-01-24T17:41:49.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.781+0000 7f31ae600640 1 -- start start 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31ae600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31a80710e0 0x7f31a80c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31ae600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80b06d0 0x7f31a815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31ae600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a8159620 0x7f31a8157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31ae600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a815c830 con 0x7f31a80710e0 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31ae600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80c4ea0 con 0x7f31a8159620 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31ae600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a815bf00 con 0x7f31a80b06d0 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31a80710e0 0x7f31a80c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31a80710e0 0x7f31a80c4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58276/0 (socket says 172.21.15.39:58276) 2024-01-24T17:41:49.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7400640 1 -- 172.21.15.39:0/3244429596 learned_addr learned my addr 172.21.15.39:0/3244429596 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a6a00640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80b06d0 0x7f31a815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7e00640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a8159620 0x7f31a8157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7400640 1 -- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80b06d0 msgr2=0x7f31a815b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7400640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80b06d0 0x7f31a815b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7400640 1 -- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a8159620 msgr2=0x7f31a8157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7400640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a8159620 0x7f31a8157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7400640 1 -- 172.21.15.39:0/3244429596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a0009a70 con 0x7f31a80710e0 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.782+0000 7f31a7e00640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a8159620 0x7f31a8157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:49.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.783+0000 7f31a7400640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31a80710e0 0x7f31a80c4810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f319800c480 tx=0x7f319800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:49.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.783+0000 7f319f400640 1 -- 172.21.15.39:0/3244429596 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3198005ce0 con 0x7f31a80710e0 2024-01-24T17:41:49.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.783+0000 7f319f400640 1 -- 172.21.15.39:0/3244429596 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3198005e40 con 0x7f31a80710e0 2024-01-24T17:41:49.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.783+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3190009400 con 0x7f31a80710e0 2024-01-24T17:41:49.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.783+0000 7f319f400640 1 -- 172.21.15.39:0/3244429596 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31980072f0 con 0x7f31a80710e0 2024-01-24T17:41:49.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.783+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a815c4c0 con 0x7f31a80710e0 2024-01-24T17:41:49.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.784+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a806c840 con 0x7f31a80710e0 2024-01-24T17:41:49.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.786+0000 7f319f400640 1 -- 172.21.15.39:0/3244429596 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3198007510 con 0x7f31a80710e0 2024-01-24T17:41:49.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.786+0000 7f319f400640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f318006a430 0x7f318006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:49.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.787+0000 7f319f400640 1 -- 172.21.15.39:0/3244429596 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3198087da0 con 0x7f31a80710e0 2024-01-24T17:41:49.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.787+0000 7f31a6a00640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f318006a430 0x7f318006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:49.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.788+0000 7f31a6a00640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f318006a430 0x7f318006c8f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3190036000 tx=0x7f3190008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:49.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:49.791+0000 7f319f400640 1 -- 172.21.15.39:0/3244429596 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f31980534d0 con 0x7f31a80710e0 2024-01-24T17:41:50.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.021+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31a8073ef0 con 0x7f31a80710e0 2024-01-24T17:41:50.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.023+0000 7f319f400640 1 -- 172.21.15.39:0/3244429596 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7f3198017090 con 0x7f31a80710e0 2024-01-24T17:41:50.024 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:41:50.024 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:41:33.354808+0000","modified":"2024-01-24T17:41:36.370204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-24T17:41:50.024 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-01-24T17:41:50.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.025+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f318006a430 msgr2=0x7f318006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.025+0000 7f31ae600640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f318006a430 0x7f318006c8f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3190036000 tx=0x7f3190008030 comp rx=0 tx=0).stop 2024-01-24T17:41:50.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31a80710e0 msgr2=0x7f31a80c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31a80710e0 0x7f31a80c4810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f319800c480 tx=0x7f319800c940 comp rx=0 tx=0).stop 2024-01-24T17:41:50.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 shutdown_connections 2024-01-24T17:41:50.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31a8159620 0x7f31a8157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31a80b06d0 0x7f31a815b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f318006a430 0x7f318006c8f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 --2- 172.21.15.39:0/3244429596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31a80710e0 0x7f31a80c4810 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 >> 172.21.15.39:0/3244429596 conn(0x7f31a806a3e0 msgr2=0x7f31a80ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:50.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 shutdown_connections 2024-01-24T17:41:50.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.026+0000 7f31ae600640 1 -- 172.21.15.39:0/3244429596 wait complete. 2024-01-24T17:41:50.447 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:41:33.354808+0000', 'modified': '2024-01-24T17:41:36.370204+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.100:6835/3015590938', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6834', 'nonce': 3015590938}, {'type': 'v1', 'addr': '172.21.15.100:6835', 'nonce': 3015590938}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:41:50.448 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T17:41:50.448 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-24T17:41:50.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.674+0000 7fc2e6000640 1 Processor -- start 2024-01-24T17:41:50.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.675+0000 7fc2e6000640 1 -- start start 2024-01-24T17:41:50.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.676+0000 7fc2e6000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc2e015ddd0 0x7fc2e00ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.676+0000 7fc2e6000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00d0e10 0x7fc2e00b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.676+0000 7fc2e6000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e006b640 con 0x7fc2e00d0a30 2024-01-24T17:41:50.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.676+0000 7fc2e6000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e006b920 con 0x7fc2e015ddd0 2024-01-24T17:41:50.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.676+0000 7fc2e6000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e006bc00 con 0x7fc2e00d0e10 2024-01-24T17:41:50.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2df400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc2e00d0a30 0x7fc2e006ad80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54460/0 (socket says 172.21.15.39:54460) 2024-01-24T17:41:50.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2df400640 1 -- 172.21.15.39:0/1356809448 learned_addr learned my addr 172.21.15.39:0/1356809448 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2d6800640 1 --2- 172.21.15.39:0/1356809448 >> v2:172.21.15.100:3300/0 conn(0x7fc2e015ddd0 0x7fc2e00ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00d0e10 0x7fc2e00b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107926762 0 0) 0x7fc2e006b640 con 0x7fc2e00d0a30 2024-01-24T17:41:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c0003520 con 0x7fc2e00d0a30 2024-01-24T17:41:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428827205 0 0) 0x7fc2c0003520 con 0x7fc2e00d0a30 2024-01-24T17:41:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00d0e10 msgr2=0x7fc2e00b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.677+0000 7fc2dea00640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00d0e10 0x7fc2e00b0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 >> v2:172.21.15.100:3300/0 conn(0x7fc2e015ddd0 msgr2=0x7fc2e00ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 --2- 172.21.15.39:0/1356809448 >> v2:172.21.15.100:3300/0 conn(0x7fc2e015ddd0 0x7fc2e00ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e00b08b0 con 0x7fc2e00d0a30 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc2d0001670 con 0x7fc2e00d0a30 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 >> v1:172.21.15.39:6789/0 conn(0x7fc2e00d0a30 legacy=0x7fc2e006ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c0003970 0x7fc2c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c00047c0 0x7fc2c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00d0e10 0x7fc2c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e006b920 con 0x7fc2c0003970 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e006bc00 con 0x7fc2c00047c0 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c000b9d0 con 0x7fc2e00d0e10 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2df400640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c0003970 0x7fc2c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00d0e10 0x7fc2c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2d6800640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c00047c0 0x7fc2c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2df400640 1 -- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00d0e10 msgr2=0x7fc2c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2df400640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00d0e10 0x7fc2c0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2df400640 1 -- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c00047c0 msgr2=0x7fc2c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.678+0000 7fc2df400640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c00047c0 0x7fc2c0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.679+0000 7fc2df400640 1 -- 172.21.15.39:0/1356809448 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c0003520 con 0x7fc2c0003970 2024-01-24T17:41:50.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.680+0000 7fc2df400640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c0003970 0x7fc2c0003d90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000c110 tx=0x7fc2c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fac8df854c2c1e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:50.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.680+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0005ad0 con 0x7fc2c0003970 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.680+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d0005c30 con 0x7fc2c0003970 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.680+0000 7fc2dea00640 1 -- 172.21.15.39:0/1356809448 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d002c7e0 con 0x7fc2c0003970 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.681+0000 7fc2e6000640 1 -- 172.21.15.39:0/1356809448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c0003970 msgr2=0x7fc2c0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.681+0000 7fc2e6000640 1 --2- 172.21.15.39:0/1356809448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2c0003970 0x7fc2c0003d90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000c110 tx=0x7fc2c00057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.681+0000 7fc2dfe00640 1 -- 172.21.15.39:0/1356809448 reap_dead start 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.681+0000 7fc2e6000640 1 -- 172.21.15.39:0/1356809448 shutdown_connections 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.681+0000 7fc2e6000640 1 -- 172.21.15.39:0/1356809448 >> 172.21.15.39:0/1356809448 conn(0x7fc2e0089fd0 msgr2=0x7fc2e0099b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.681+0000 7fc2e6000640 1 -- 172.21.15.39:0/1356809448 shutdown_connections 2024-01-24T17:41:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.681+0000 7fc2e6000640 1 -- 172.21.15.39:0/1356809448 wait complete. 2024-01-24T17:41:50.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.683+0000 7fc2e6000640 1 Processor -- start 2024-01-24T17:41:50.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.683+0000 7fc2e6000640 1 -- start start 2024-01-24T17:41:50.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2e6000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c0003970 0x7fc2e014d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2e6000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00b5e80 0x7fc2e015ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2e6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2e00b3420 0x7fc2e00b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2e6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e015c0a0 con 0x7fc2e00b3420 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2e6000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e015b490 con 0x7fc2c0003970 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2e6000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e015b770 con 0x7fc2e00b5e80 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2e00b3420 0x7fc2e00b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2e00b3420 0x7fc2e00b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58304/0 (socket says 172.21.15.39:58304) 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.684+0000 7fc2dfe00640 1 -- 172.21.15.39:0/2915638253 learned_addr learned my addr 172.21.15.39:0/2915638253 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2dfe00640 1 -- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00b5e80 msgr2=0x7fc2e015ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2d6800640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00b5e80 0x7fc2e015ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2df400640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c0003970 0x7fc2e014d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00b5e80 0x7fc2e015ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2dfe00640 1 -- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c0003970 msgr2=0x7fc2e014d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c0003970 0x7fc2e014d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2dfe00640 1 -- 172.21.15.39:0/2915638253 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e015ba50 con 0x7fc2e00b3420 2024-01-24T17:41:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2df400640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c0003970 0x7fc2e014d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2e00b3420 0x7fc2e00b38d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc2e015e1b0 tx=0x7fc2d800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2d7e00640 1 -- 172.21.15.39:0/2915638253 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8017400 con 0x7fc2e00b3420 2024-01-24T17:41:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2d7e00640 1 -- 172.21.15.39:0/2915638253 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8011040 con 0x7fc2e00b3420 2024-01-24T17:41:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2e6000640 1 -- 172.21.15.39:0/2915638253 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e015bd30 con 0x7fc2e00b3420 2024-01-24T17:41:50.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.685+0000 7fc2d7e00640 1 -- 172.21.15.39:0/2915638253 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80177c0 con 0x7fc2e00b3420 2024-01-24T17:41:50.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.686+0000 7fc2e6000640 1 -- 172.21.15.39:0/2915638253 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e015b0c0 con 0x7fc2e00b3420 2024-01-24T17:41:50.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.688+0000 7fc2d7e00640 1 -- 172.21.15.39:0/2915638253 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc2d8017920 con 0x7fc2e00b3420 2024-01-24T17:41:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.689+0000 7fc2d7e00640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc2b806a460 0x7fc2b806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.689+0000 7fc2d7e00640 1 -- 172.21.15.39:0/2915638253 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2d8088ba0 con 0x7fc2e00b3420 2024-01-24T17:41:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.689+0000 7fc2df400640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc2b806a460 0x7fc2b806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.689+0000 7fc2e8bf6740 1 Processor -- start 2024-01-24T17:41:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.690+0000 7fc2e8bf6740 1 -- start start 2024-01-24T17:41:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.690+0000 7fc2df400640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc2b806a460 0x7fc2b806c920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0034000 tx=0x7fc2d0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:50.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a868a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a8659f90 0x5619a8687280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a865db90 0x5619a86877c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a86356c0 con 0x5619a868a010 2024-01-24T17:41:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a85ca070 con 0x5619a8659f90 2024-01-24T17:41:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619a85c7800 con 0x5619a865db90 2024-01-24T17:41:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a868a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a868a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58314/0 (socket says 172.21.15.39:58314) 2024-01-24T17:41:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2df400640 1 -- 172.21.15.39:0/1716385986 learned_addr learned my addr 172.21.15.39:0/1716385986 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2df400640 1 -- 172.21.15.39:0/1716385986 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a865db90 msgr2=0x5619a86877c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2df400640 1 --2- 172.21.15.39:0/1716385986 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a865db90 0x5619a86877c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2df400640 1 -- 172.21.15.39:0/1716385986 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a8659f90 msgr2=0x5619a8687280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2df400640 1 --2- 172.21.15.39:0/1716385986 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a8659f90 0x5619a8687280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.691+0000 7fc2df400640 1 -- 172.21.15.39:0/1716385986 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619a858a780 con 0x5619a868a010 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.692+0000 7fc2df400640 1 --2- 172.21.15.39:0/1716385986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a868a3f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0007e90 tx=0x7fc2d0007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802e99b86623e636 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.692+0000 7fc2cfe00640 1 -- 172.21.15.39:0/1716385986 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d002cd30 con 0x5619a868a010 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.692+0000 7fc2cfe00640 1 -- 172.21.15.39:0/1716385986 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d002ce90 con 0x5619a868a010 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.692+0000 7fc2cfe00640 1 -- 172.21.15.39:0/1716385986 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0007270 con 0x5619a868a010 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.692+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1716385986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 msgr2=0x5619a868a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.692+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1716385986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a868a3f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0007e90 tx=0x7fc2d0007000 comp rx=0 tx=0).stop 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.693+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1716385986 shutdown_connections 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.693+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1716385986 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a865db90 0x5619a86877c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.693+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1716385986 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a8659f90 0x5619a8687280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.693+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1716385986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a868a3f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.693+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1716385986 >> 172.21.15.39:0/1716385986 conn(0x5619a865c2a0 msgr2=0x5619a86a4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.693+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1716385986 shutdown_connections 2024-01-24T17:41:50.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.693+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1716385986 wait complete. 2024-01-24T17:41:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.693+0000 7fc2e8bf6740 1 -- start start 2024-01-24T17:41:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.694+0000 7fc2e8bf6740 1 Processor -- start 2024-01-24T17:41:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a865db90 0x5619a868cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a868a010 0x5619a868fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a86866d0 0x5619a8690070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a85c30f0 con 0x5619a86866d0 2024-01-24T17:41:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a85648c0 con 0x5619a865db90 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619a8564560 con 0x5619a868a010 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2e8bf6740 10 client.0 _ll_register_callbacks cb 0x5619a85bca30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a86866d0 0x5619a8690070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a86866d0 0x5619a8690070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58320/0 (socket says 172.21.15.39:58320) 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2dfe00640 1 -- 172.21.15.39:0/220489876 learned_addr learned my addr 172.21.15.39:0/220489876 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2dfe00640 1 -- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a868a010 msgr2=0x5619a868fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a868a010 0x5619a868fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2dfe00640 1 -- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a865db90 msgr2=0x5619a868cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a865db90 0x5619a868cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.695+0000 7fc2dfe00640 1 -- 172.21.15.39:0/220489876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c0003520 con 0x5619a86866d0 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a86866d0 0x5619a8690070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc2d805f860 tx=0x7fc2d8057f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e7225a7d64f52c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2ccc00640 1 -- 172.21.15.39:0/220489876 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d806f3b0 con 0x5619a86866d0 2024-01-24T17:41:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2ccc00640 1 -- 172.21.15.39:0/220489876 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2d806ce90 con 0x5619a86866d0 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/220489876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619a85ed640 con 0x5619a86866d0 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2ccc00640 1 -- 172.21.15.39:0/220489876 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2d8017bd0 con 0x5619a86866d0 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2e8bf6740 10 client.4629 fetch_fsmap learned FSMap version 7 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/220489876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5619a85c1770 con 0x5619a86866d0 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2ccc00640 1 -- 172.21.15.39:0/220489876 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5619a85ed640 con 0x5619a86866d0 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2ccc00640 1 -- 172.21.15.39:0/220489876 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 7) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc2d8014070 con 0x5619a86866d0 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2e8bf6740 10 client.4629 fetch_fsmap finished waiting for FSMap version 7 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2e8bf6740 10 client.4629 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.696+0000 7fc2e8bf6740 20 client.4629 populate_metadata read hostname 'smithi039' 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.697+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] conn(0x5619a86bfe10 0x5619a86c2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.697+0000 7fc2e8bf6740 4 client.4629 mds_command: new command op to 4282 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:41:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.697+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/220489876 --> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619a85b5940 con 0x5619a86bfe10 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.697+0000 7fc2df400640 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] conn(0x5619a86bfe10 0x5619a86c2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.698+0000 7fc2df400640 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] conn(0x5619a86bfe10 0x5619a86c2200 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cadacf409b52930 server_cookie=962f94aa42d69e62 in_seq=0 out_seq=0 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.698+0000 7fc2ccc00640 10 client.4629 ms_handle_connect on v2:172.21.15.100:6834/3015590938 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2ccc00640 1 -- 172.21.15.39:0/220489876 <== mds.0 v2:172.21.15.100:6834/3015590938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5619a85b5940 con 0x5619a86bfe10 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2ccc00640 10 client.4629 handle_command_reply: tid=0 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 client.4629 shutdown 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/220489876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a86866d0 msgr2=0x5619a8690070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a86866d0 0x5619a8690070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc2d805f860 tx=0x7fc2d8057f20 comp rx=0 tx=0).stop 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/220489876 shutdown_connections 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] conn(0x5619a86bfe10 0x5619a86c2200 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a86866d0 0x5619a8690070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a868a010 0x5619a868fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/220489876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a865db90 0x5619a868cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2ccc00640 0 client.4629 ms_handle_reset on v2:172.21.15.100:6834/3015590938 2024-01-24T17:41:50.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/220489876 >> 172.21.15.39:0/220489876 conn(0x5619a8659f90 msgr2=0x5619a865e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:50.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/220489876 shutdown_connections 2024-01-24T17:41:50.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.700+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/220489876 wait complete. 2024-01-24T17:41:50.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.701+0000 7fc2e8bf6740 20 client.4629 trim_cache size 0 max 16384 2024-01-24T17:41:50.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.717+0000 7fc2e8bf6740 1 Processor -- start 2024-01-24T17:41:50.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.717+0000 7fc2e8bf6740 1 -- start start 2024-01-24T17:41:50.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a865db90 0x5619a865a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a86866d0 0x5619a87053a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a8686c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a832af20 con 0x5619a868a010 2024-01-24T17:41:50.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a86a90b0 con 0x5619a86866d0 2024-01-24T17:41:50.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619a8660c60 con 0x5619a865db90 2024-01-24T17:41:50.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a8686c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a8686c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58326/0 (socket says 172.21.15.39:58326) 2024-01-24T17:41:50.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.718+0000 7fc2dfe00640 1 -- 172.21.15.39:0/514686776 learned_addr learned my addr 172.21.15.39:0/514686776 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:50.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2dfe00640 1 -- 172.21.15.39:0/514686776 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a865db90 msgr2=0x5619a865a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/514686776 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a865db90 0x5619a865a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2dfe00640 1 -- 172.21.15.39:0/514686776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a86866d0 msgr2=0x5619a87053a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/514686776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a86866d0 0x5619a87053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2dfe00640 1 -- 172.21.15.39:0/514686776 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619a85f72f0 con 0x5619a868a010 2024-01-24T17:41:50.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2dfe00640 1 --2- 172.21.15.39:0/514686776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a8686c30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8057b60 tx=0x7fc2d8022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731f0bd3aa0863f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:50.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2d4a00640 1 -- 172.21.15.39:0/514686776 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d806f1b0 con 0x5619a868a010 2024-01-24T17:41:50.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2d4a00640 1 -- 172.21.15.39:0/514686776 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d806f310 con 0x5619a868a010 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.719+0000 7fc2d4a00640 1 -- 172.21.15.39:0/514686776 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d805fb40 con 0x5619a868a010 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/514686776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 msgr2=0x5619a8686c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/514686776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a8686c30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8057b60 tx=0x7fc2d8022000 comp rx=0 tx=0).stop 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/514686776 shutdown_connections 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/514686776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a868a010 0x5619a8686c30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/514686776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a86866d0 0x5619a87053a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/514686776 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a865db90 0x5619a865a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/514686776 >> 172.21.15.39:0/514686776 conn(0x5619a8659f90 msgr2=0x5619a868daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/514686776 shutdown_connections 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.720+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/514686776 wait complete. 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.721+0000 7fc2e8bf6740 1 -- start start 2024-01-24T17:41:50.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.721+0000 7fc2e8bf6740 1 Processor -- start 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a865db90 0x5619a8685040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a86866d0 0x5619a8701250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2e8bf6740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a868a010 0x5619a8701790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a8701e00 con 0x5619a865db90 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619a86904d0 con 0x5619a868a010 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2e8bf6740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619a8690c00 con 0x5619a86866d0 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2e8bf6740 10 client.0 _ll_register_callbacks cb 0x7fc2d8007080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a865db90 0x5619a8685040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a865db90 0x5619a8685040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58342/0 (socket says 172.21.15.39:58342) 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2df400640 1 -- 172.21.15.39:0/1402198746 learned_addr learned my addr 172.21.15.39:0/1402198746 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2df400640 1 -- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a86866d0 msgr2=0x5619a8701250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2df400640 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a86866d0 0x5619a8701250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2df400640 1 -- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a868a010 msgr2=0x5619a8701790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2df400640 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a868a010 0x5619a8701790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.722+0000 7fc2df400640 1 -- 172.21.15.39:0/1402198746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619a85c1770 con 0x5619a865db90 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.723+0000 7fc2df400640 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a865db90 0x5619a8685040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0007e90 tx=0x7fc2d0001820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db78ffa25bcd7e16 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.723+0000 7fc2ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.723+0000 7fc2ccc00640 1 -- 172.21.15.39:0/1402198746 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0007450 con 0x5619a865db90 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.723+0000 7fc2ccc00640 1 -- 172.21.15.39:0/1402198746 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2d0005870 con 0x5619a865db90 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.723+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1402198746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2d002ce90 con 0x5619a865db90 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.723+0000 7fc2ccc00640 1 -- 172.21.15.39:0/1402198746 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2d0005ab0 con 0x5619a865db90 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.723+0000 7fc2e8bf6740 10 client.4635 fetch_fsmap learned FSMap version 7 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.723+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1402198746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5619a8563ad0 con 0x5619a865db90 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2ccc00640 1 -- 172.21.15.39:0/1402198746 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d002ce90 con 0x5619a865db90 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2ccc00640 1 -- 172.21.15.39:0/1402198746 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 7) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc2d0055040 con 0x5619a865db90 2024-01-24T17:41:50.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2e8bf6740 10 client.4635 fetch_fsmap finished waiting for FSMap version 7 2024-01-24T17:41:50.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2e8bf6740 10 client.4635 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-24T17:41:50.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2e8bf6740 20 client.4635 populate_metadata read hostname 'smithi039' 2024-01-24T17:41:50.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] conn(0x5619a8681e40 0x5619a8682220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:50.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2e8bf6740 4 client.4635 mds_command: new command op to 4282 tid=0[{"prefix": "status"}] 2024-01-24T17:41:50.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1402198746 --> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5619a85bc170 con 0x5619a8681e40 2024-01-24T17:41:50.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.724+0000 7fc2d6800640 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] conn(0x5619a8681e40 0x5619a8682220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:50.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.725+0000 7fc2d6800640 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] conn(0x5619a8681e40 0x5619a8682220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98a26fa73e1bf59 server_cookie=c01721d05d008ba6 in_seq=0 out_seq=0 2024-01-24T17:41:50.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.725+0000 7fc2ccc00640 10 client.4635 ms_handle_connect on v2:172.21.15.100:6834/3015590938 2024-01-24T17:41:50.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2ccc00640 1 -- 172.21.15.39:0/1402198746 <== mds.0 v2:172.21.15.100:6834/3015590938 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5619a85bc170 con 0x5619a8681e40 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2ccc00640 10 client.4635 handle_command_reply: tid=0 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 client.4635 shutdown 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1402198746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a865db90 msgr2=0x5619a8685040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a865db90 0x5619a8685040 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0007e90 tx=0x7fc2d0001820 comp rx=0 tx=0).stop 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1402198746 shutdown_connections 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:6834/3015590938,v1:172.21.15.100:6835/3015590938] conn(0x5619a8681e40 0x5619a8682220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5619a868a010 0x5619a8701790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5619a86866d0 0x5619a8701250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2ccc00640 0 client.4635 ms_handle_reset on v2:172.21.15.100:6834/3015590938 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 --2- 172.21.15.39:0/1402198746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619a865db90 0x5619a8685040 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.726+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1402198746 >> 172.21.15.39:0/1402198746 conn(0x5619a8659f90 msgr2=0x5619a8685b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1402198746 shutdown_connections 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e8bf6740 1 -- 172.21.15.39:0/1402198746 wait complete. 2024-01-24T17:41:50.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e8bf6740 20 client.4635 trim_cache size 0 max 16384 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 4282, 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 17.364312103, 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 7, 2024-01-24T17:41:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 20, 2024-01-24T17:41:50.731 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 20, 2024-01-24T17:41:50.731 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 21.974860898999999 2024-01-24T17:41:50.731 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:41:50.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 -- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc2b806a460 msgr2=0x7fc2b806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc2b806a460 0x7fc2b806c920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0034000 tx=0x7fc2d0009030 comp rx=0 tx=0).stop 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 -- 172.21.15.39:0/2915638253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2e00b3420 msgr2=0x7fc2e00b38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2e00b3420 0x7fc2e00b38d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc2e015e1b0 tx=0x7fc2d800f4a0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 -- 172.21.15.39:0/2915638253 shutdown_connections 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc2b806a460 0x7fc2b806c920 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2e00b3420 0x7fc2e00b38d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2e00b5e80 0x7fc2e015ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 --2- 172.21.15.39:0/2915638253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c0003970 0x7fc2e014d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.727+0000 7fc2e6000640 1 -- 172.21.15.39:0/2915638253 >> 172.21.15.39:0/2915638253 conn(0x7fc2e0089fd0 msgr2=0x7fc2e00b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.728+0000 7fc2e6000640 1 -- 172.21.15.39:0/2915638253 shutdown_connections 2024-01-24T17:41:50.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:50.728+0000 7fc2e6000640 1 -- 172.21.15.39:0/2915638253 wait complete. 2024-01-24T17:41:50.745 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-01-24T17:41:50.768 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-01-24T17:41:50.768 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-24T17:41:50.768 INFO:tasks.ceph_fuse:client.0 config is {} 2024-01-24T17:41:50.769 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-24T17:41:50.769 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:41:50.769 INFO:tasks.ceph_fuse:client.2 config is {} 2024-01-24T17:41:50.769 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-24T17:41:50.769 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-24T17:41:50.769 INFO:tasks.ceph_fuse:client.1 config is {} 2024-01-24T17:41:50.769 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-24T17:41:50.769 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:41:50.769 INFO:tasks.ceph_fuse:client.3 config is {} 2024-01-24T17:41:50.769 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-24T17:41:50.769 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-24T17:41:50.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:50.770 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:41:50.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:50.827 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-01-24T17:41:50.929 INFO:teuthology.orchestra.run.smithi039.stderr:Cannot find device "ceph-brx" 2024-01-24T17:41:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:41:50.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:50.933 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:41:50.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:50.990 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link delete ceph-brx 2024-01-24T17:41:51.094 INFO:teuthology.orchestra.run.smithi100.stderr:Cannot find device "ceph-brx" 2024-01-24T17:41:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:41:51.098 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-01-24T17:41:51.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:41:51.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.320+0000 7f8ec0e00640 1 Processor -- start 2024-01-24T17:41:51.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.320+0000 7f8ec0e00640 1 -- start start 2024-01-24T17:41:51.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.322+0000 7f8ec0e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8ebc0cb3e0 0x7f8ebc0b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:51.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.322+0000 7f8ec0e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ebc0c9180 con 0x7f8ebc15a950 2024-01-24T17:41:51.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.322+0000 7f8ec0e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ebc0c9460 con 0x7f8ebc0caf50 2024-01-24T17:41:51.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.322+0000 7f8ec0e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc0b43d0 con 0x7f8ebc0cb3e0 2024-01-24T17:41:51.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.322+0000 7f8eb9e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8ebc15a950 0x7f8ebc0ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54466/0 (socket says 172.21.15.39:54466) 2024-01-24T17:41:51.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.322+0000 7f8eb9e00640 1 -- 172.21.15.39:0/1195403027 learned_addr learned my addr 172.21.15.39:0/1195403027 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:51.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eba800640 1 --2- 172.21.15.39:0/1195403027 >> v2:172.21.15.100:3300/0 conn(0x7f8ebc0cb3e0 0x7f8ebc0b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271294510 0 0) 0x7f8ebc0c9180 con 0x7f8ebc15a950 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9c003520 con 0x7f8ebc15a950 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2810040450 0 0) 0x7f8e9c003520 con 0x7f8ebc15a950 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 >> v2:172.21.15.100:3300/0 conn(0x7f8ebc0cb3e0 msgr2=0x7f8ebc0b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 --2- 172.21.15.39:0/1195403027 >> v2:172.21.15.100:3300/0 conn(0x7f8ebc0cb3e0 0x7f8ebc0b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 >> v1:172.21.15.100:6789/0 conn(0x7f8ebc0caf50 legacy=0x7f8ebc0b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc0c9740 con 0x7f8ebc15a950 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8eac001670 con 0x7f8ebc15a950 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 >> v1:172.21.15.39:6789/0 conn(0x7f8ebc15a950 legacy=0x7f8ebc0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c003970 0x7f8e9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.323+0000 7f8eb8a00640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0047c0 0x7f8e9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:51.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eb8a00640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c005180 0x7f8e9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0b43d0 con 0x7f8e9c005180 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c00c020 con 0x7f8e9c0047c0 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c00c1e0 con 0x7f8e9c003970 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eba800640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c005180 0x7f8e9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eb9e00640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c003970 0x7f8e9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eb9400640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0047c0 0x7f8e9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eba800640 1 -- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c003970 msgr2=0x7f8e9c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eba800640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c003970 0x7f8e9c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eba800640 1 -- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0047c0 msgr2=0x7f8e9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:51.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eba800640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0047c0 0x7f8e9c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eba800640 1 -- 172.21.15.39:0/1195403027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c003520 con 0x7f8e9c005180 2024-01-24T17:41:51.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.324+0000 7f8eb9e00640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c003970 0x7f8e9c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:51.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.325+0000 7f8eba800640 1 -- 172.21.15.39:0/1195403027 reap_dead start 2024-01-24T17:41:51.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.325+0000 7f8eba800640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c005180 0x7f8e9c00b980 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009fc0 tx=0x7f8e9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f69a7fe5581f37ee server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb402cc60 con 0x7f8e9c005180 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb400f040 con 0x7f8e9c005180 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8eb8a00640 1 -- 172.21.15.39:0/1195403027 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4003990 con 0x7f8e9c005180 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1195403027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c005180 msgr2=0x7f8e9c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8ec0e00640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c005180 0x7f8e9c00b980 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009fc0 tx=0x7f8e9c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1195403027 shutdown_connections 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8ec0e00640 1 --2- 172.21.15.39:0/1195403027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c005180 0x7f8e9c00b980 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1195403027 >> 172.21.15.39:0/1195403027 conn(0x7f8ebc06a3e0 msgr2=0x7f8ebc06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:51.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1195403027 shutdown_connections 2024-01-24T17:41:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.326+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1195403027 wait complete. 2024-01-24T17:41:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.328+0000 7f8ec0e00640 1 Processor -- start 2024-01-24T17:41:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.328+0000 7f8ec0e00640 1 -- start start 2024-01-24T17:41:51.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.329+0000 7f8ec0e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c005180 0x7f8ebc0b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.329+0000 7f8ec0e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ebc15c790 0x7f8ebc15cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.329+0000 7f8ec0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ebc14e200 0x7f8ebc14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.329+0000 7f8ec0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc14f160 con 0x7f8ebc14e200 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.329+0000 7f8ec0e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0b0f70 con 0x7f8e9c005180 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.329+0000 7f8ec0e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc15d150 con 0x7f8ebc15c790 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ebc14e200 0x7f8ebc14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ebc14e200 0x7f8ebc14e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58362/0 (socket says 172.21.15.39:58362) 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 -- 172.21.15.39:0/1221499671 learned_addr learned my addr 172.21.15.39:0/1221499671 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eb9400640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ebc15c790 0x7f8ebc15cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eb9e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c005180 0x7f8ebc0b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 -- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ebc15c790 msgr2=0x7f8ebc15cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ebc15c790 0x7f8ebc15cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 -- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c005180 msgr2=0x7f8ebc0b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:51.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c005180 0x7f8ebc0b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 -- 172.21.15.39:0/1221499671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc14ebc0 con 0x7f8ebc14e200 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eb9e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c005180 0x7f8ebc0b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.330+0000 7f8eba800640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ebc14e200 0x7f8ebc14e680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8eb402c6d0 tx=0x7f8eb4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.331+0000 7f8eb2000640 1 -- 172.21.15.39:0/1221499671 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4035dc0 con 0x7f8ebc14e200 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.331+0000 7f8eb2000640 1 -- 172.21.15.39:0/1221499671 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb4003990 con 0x7f8ebc14e200 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.331+0000 7f8eb2000640 1 -- 172.21.15.39:0/1221499671 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4003d40 con 0x7f8ebc14e200 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.331+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc15d6a0 con 0x7f8ebc14e200 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.331+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc0c7110 con 0x7f8ebc14e200 2024-01-24T17:41:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.332+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc0c6f70 con 0x7f8ebc14e200 2024-01-24T17:41:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.333+0000 7f8eb2000640 1 -- 172.21.15.39:0/1221499671 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8eb4004030 con 0x7f8ebc14e200 2024-01-24T17:41:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.334+0000 7f8eb2000640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e9406a4b0 0x7f8e9406c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.334+0000 7f8eb2000640 1 -- 172.21.15.39:0/1221499671 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8eb40ab4c0 con 0x7f8ebc14e200 2024-01-24T17:41:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.334+0000 7f8eb9e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e9406a4b0 0x7f8e9406c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:51.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.335+0000 7f8eb9e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e9406a4b0 0x7f8e9406c970 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8eac001eb0 tx=0x7f8eac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.338+0000 7f8eb2000640 1 -- 172.21.15.39:0/1221499671 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8ebc0c6f70 con 0x7f8ebc14e200 2024-01-24T17:41:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.572+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8ebc0c6f70 con 0x7f8ebc14e200 2024-01-24T17:41:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.573+0000 7f8eb2000640 1 -- 172.21.15.39:0/1221499671 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8ebc0c6f70 con 0x7f8ebc14e200 2024-01-24T17:41:51.574 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e9406a4b0 msgr2=0x7f8e9406c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e9406a4b0 0x7f8e9406c970 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8eac001eb0 tx=0x7f8eac009040 comp rx=0 tx=0).stop 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ebc14e200 msgr2=0x7f8ebc14e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ebc14e200 0x7f8ebc14e680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8eb402c6d0 tx=0x7f8eb4009750 comp rx=0 tx=0).stop 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 shutdown_connections 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e9406a4b0 0x7f8e9406c970 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ebc14e200 0x7f8ebc14e680 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ebc15c790 0x7f8ebc15cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 --2- 172.21.15.39:0/1221499671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c005180 0x7f8ebc0b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 >> 172.21.15.39:0/1221499671 conn(0x7f8ebc06a3e0 msgr2=0x7f8ebc0cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 shutdown_connections 2024-01-24T17:41:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:51.576+0000 7f8ec0e00640 1 -- 172.21.15.39:0/1221499671 wait complete. 2024-01-24T17:41:51.594 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:41:51.594 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:41:51.595 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:41:51.595 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:41:51.595 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:41:51.595 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:41:51.595 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:41:51.595 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:41:51.595 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:41:51.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:51.595 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:41:51.644 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:41:51.644 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42620sec preferred_lft 42620sec 2024-01-24T17:41:51.645 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:41:51.646 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:51.646 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:41:51.646 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:51.646 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:41:51.647 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-24T17:41:51.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:51.647 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:51.647 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:51.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add name ceph-brx type bridge 2024-01-24T17:41:51.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr flush dev ceph-brx 2024-01-24T17:41:51.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set ceph-brx up 2024-01-24T17:41:51.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-24T17:41:51.647 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:52.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:52.105 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:41:52.196 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:41:52.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:52.200 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:41:52.258 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:41:52.258 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:41:52.258 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:41:52.258 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:41:52.259 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:41:52.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:52.259 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:52.259 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:52.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:41:52.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:41:52.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:41:52.259 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:53.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:53.013 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:41:53.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:53.060 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-01-24T17:41:53.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:53.153 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:53.153 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:53.153 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:41:53.153 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-24T17:41:53.153 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:53.284 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-24T17:41:53.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:53.285 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:53.285 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:53.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-24T17:41:53.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-24T17:41:53.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-24T17:41:53.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-24T17:41:53.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-24T17:41:53.285 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:53.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:53.563 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:53.564 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:53.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:41:53.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.0 master ceph-brx 2024-01-24T17:41:53.564 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:53.728 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:41:53.728 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:41:53.729 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:41:53.774 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:41:53.774 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:41:53.774 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:41:53.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:41:53.947 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:41:53.947 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:41:54.013 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:41:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:41:54.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:54.016 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:41:54.057 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:41:54.057 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:41:54.105 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:41:54.180 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:41:54.180 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:41:54.253 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:41:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:41:54.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:54.256 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:41:54.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:41:54.294+0000 7f9283177480 -1 init, newargv = 0x561da4d030c0 newargc=15 2024-01-24T17:41:54.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[45111]: starting ceph client 2024-01-24T17:41:54.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[45111]: starting fuse 2024-01-24T17:41:55.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:41:55.512 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:41:55.512 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:41:55.609 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:41:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:41:55.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:55.614 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:41:55.670 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:41:55.670 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:41:55.670 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:41:55.670 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:41:55.670 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:41:55.670 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:41:55.670 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:41:55.670 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:41:55.670 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:41:55.670 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:41:55.671 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:41:55.672 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:41:55.672 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:41:55.672 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:41:55.852 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.45111.asok 2024-01-24T17:41:56.031 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45111.asok 2024-01-24T17:41:56.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:56.031 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45111.asok status 2024-01-24T17:41:56.278 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "45111", 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 4647, 2024-01-24T17:41:56.279 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4647 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 221306170 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:41:56.280 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 221306170 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.4647 192.168.0.1:0/221306170", 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/221306170", 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 7, 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 20, 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:41:56.281 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:41:56.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:41:56.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.510+0000 7f687fe00640 1 Processor -- start 2024-01-24T17:41:56.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.510+0000 7f687fe00640 1 -- start start 2024-01-24T17:41:56.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f688015a720 0x7f68800cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:56.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f68800ce8e0 0x7f68800ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:56.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800cf260 con 0x7f688015ab00 2024-01-24T17:41:56.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687fe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68800ce400 con 0x7f68800ce8e0 2024-01-24T17:41:56.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687fe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68800ccef0 con 0x7f688015a720 2024-01-24T17:41:56.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f68800ce8e0 0x7f68800ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:56.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687e000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f688015ab00 0x7f68800cc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35414/0 (socket says 172.21.15.39:35414) 2024-01-24T17:41:56.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687e000640 1 -- 172.21.15.39:0/1889188170 learned_addr learned my addr 172.21.15.39:0/1889188170 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:41:56.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.512+0000 7f687ea00640 1 --2- 172.21.15.39:0/1889188170 >> v2:172.21.15.100:3300/0 conn(0x7f688015a720 0x7f68800cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:56.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.513+0000 7f687f400640 1 -- 172.21.15.39:0/1889188170 >> v2:172.21.15.100:3300/0 conn(0x7f688015a720 msgr2=0x7f68800cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:41:56.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.513+0000 7f687f400640 1 --2- 172.21.15.39:0/1889188170 >> v2:172.21.15.100:3300/0 conn(0x7f688015a720 0x7f68800cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.513+0000 7f687f400640 1 -- 172.21.15.39:0/1889188170 >> v1:172.21.15.39:6789/0 conn(0x7f688015ab00 legacy=0x7f68800cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:56.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.513+0000 7f687f400640 1 -- 172.21.15.39:0/1889188170 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68800ce110 con 0x7f68800ce8e0 2024-01-24T17:41:56.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.514+0000 7f687f400640 1 --2- 172.21.15.39:0/1889188170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68800ce8e0 0x7f68800ced20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f688006c3e0 tx=0x7f687802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab60b0f004d6dd server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:56.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.514+0000 7f687d600640 1 -- 172.21.15.39:0/1889188170 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687802cbe0 con 0x7f68800ce8e0 2024-01-24T17:41:56.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.514+0000 7f687d600640 1 -- 172.21.15.39:0/1889188170 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687802cd40 con 0x7f68800ce8e0 2024-01-24T17:41:56.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.514+0000 7f687d600640 1 -- 172.21.15.39:0/1889188170 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878034660 con 0x7f68800ce8e0 2024-01-24T17:41:56.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.514+0000 7f687fe00640 1 -- 172.21.15.39:0/1889188170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68800ce8e0 msgr2=0x7f68800ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:56.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.514+0000 7f687fe00640 1 --2- 172.21.15.39:0/1889188170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68800ce8e0 0x7f68800ced20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f688006c3e0 tx=0x7f687802be30 comp rx=0 tx=0).stop 2024-01-24T17:41:56.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.515+0000 7f687fe00640 1 -- 172.21.15.39:0/1889188170 shutdown_connections 2024-01-24T17:41:56.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.515+0000 7f687fe00640 1 --2- 172.21.15.39:0/1889188170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68800ce8e0 0x7f68800ced20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.515+0000 7f687fe00640 1 --2- 172.21.15.39:0/1889188170 >> v2:172.21.15.100:3300/0 conn(0x7f688015a720 0x7f68800cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.515+0000 7f687fe00640 1 -- 172.21.15.39:0/1889188170 >> 172.21.15.39:0/1889188170 conn(0x7f688006a090 msgr2=0x7f688006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:56.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.515+0000 7f687fe00640 1 -- 172.21.15.39:0/1889188170 shutdown_connections 2024-01-24T17:41:56.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.515+0000 7f687fe00640 1 -- 172.21.15.39:0/1889188170 wait complete. 2024-01-24T17:41:56.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.516+0000 7f687fe00640 1 Processor -- start 2024-01-24T17:41:56.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.517+0000 7f687fe00640 1 -- start start 2024-01-24T17:41:56.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.517+0000 7f687fe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68800ce8e0 0x7f68801f1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:56.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f688015a720 0x7f68801e41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688015ab00 0x7f68800c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68801e4c70 con 0x7f688015a720 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687fe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68801e4f50 con 0x7f68800ce8e0 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68801e5230 con 0x7f688015ab00 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f688015a720 0x7f68801e41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f688015a720 0x7f68801e41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56406/0 (socket says 172.21.15.39:56406) 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687e000640 1 -- 172.21.15.39:0/2269946097 learned_addr learned my addr 172.21.15.39:0/2269946097 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687ea00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68800ce8e0 0x7f68801f1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687f400640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688015ab00 0x7f68800c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687e000640 1 -- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688015ab00 msgr2=0x7f68800c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687e000640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688015ab00 0x7f68800c5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687e000640 1 -- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68800ce8e0 msgr2=0x7f68801f1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:56.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687e000640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68800ce8e0 0x7f68801f1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.518+0000 7f687e000640 1 -- 172.21.15.39:0/2269946097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6878009a70 con 0x7f688015a720 2024-01-24T17:41:56.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.519+0000 7f687e000640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f688015a720 0x7f68801e41b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f686800c480 tx=0x7f686800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:56.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.519+0000 7f6876a00640 1 -- 172.21.15.39:0/2269946097 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868005ce0 con 0x7f688015a720 2024-01-24T17:41:56.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.519+0000 7f6876a00640 1 -- 172.21.15.39:0/2269946097 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6868005e40 con 0x7f688015a720 2024-01-24T17:41:56.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.519+0000 7f6876a00640 1 -- 172.21.15.39:0/2269946097 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68680072f0 con 0x7f688015a720 2024-01-24T17:41:56.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.519+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68801e5510 con 0x7f688015a720 2024-01-24T17:41:56.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.519+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68801f1e90 con 0x7f688015a720 2024-01-24T17:41:56.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.520+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68801e5a30 con 0x7f688015a720 2024-01-24T17:41:56.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.521+0000 7f6876a00640 1 -- 172.21.15.39:0/2269946097 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6868007510 con 0x7f688015a720 2024-01-24T17:41:56.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.522+0000 7f6876a00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f685806a430 0x7f685806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:41:56.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.522+0000 7f6876a00640 1 -- 172.21.15.39:0/2269946097 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6868087e40 con 0x7f688015a720 2024-01-24T17:41:56.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.523+0000 7f687ea00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f685806a430 0x7f685806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:41:56.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.524+0000 7f687ea00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f685806a430 0x7f685806c8f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6870006fd0 tx=0x7f6870008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:41:56.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.527+0000 7f6876a00640 1 -- 172.21.15.39:0/2269946097 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f68801e5a30 con 0x7f688015a720 2024-01-24T17:41:56.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.760+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f68801e5a30 con 0x7f688015a720 2024-01-24T17:41:56.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.760+0000 7f6876a00640 1 -- 172.21.15.39:0/2269946097 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f68801e5a30 con 0x7f688015a720 2024-01-24T17:41:56.761 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:41:56.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f685806a430 msgr2=0x7f685806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:56.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f685806a430 0x7f685806c8f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6870006fd0 tx=0x7f6870008040 comp rx=0 tx=0).stop 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f688015a720 msgr2=0x7f68801e41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f688015a720 0x7f68801e41b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f686800c480 tx=0x7f686800c940 comp rx=0 tx=0).stop 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 shutdown_connections 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f688015ab00 0x7f68800c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f685806a430 0x7f685806c8f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f688015a720 0x7f68801e41b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.763+0000 7f687fe00640 1 --2- 172.21.15.39:0/2269946097 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68800ce8e0 0x7f68801f1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.764+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 >> 172.21.15.39:0/2269946097 conn(0x7f688006a090 msgr2=0x7f68801598e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.764+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 shutdown_connections 2024-01-24T17:41:56.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:41:56.764+0000 7f687fe00640 1 -- 172.21.15.39:0/2269946097 wait complete. 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:41:56.784 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-24T17:41:56.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:56.784 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42615sec preferred_lft 42615sec 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:41:56.840 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:41:56.841 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:56.841 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:41:56.841 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:56.841 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:41:56.841 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:41:56.841 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:41:56.841 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:56.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:56.842 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:41:56.937 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:41:56.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:56.941 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:41:56.997 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:41:56.997 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:41:56.997 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:41:56.997 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:41:56.997 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:41:56.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:56.998 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:56.998 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:56.998 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:41:56.998 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:41:56.998 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:41:56.999 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:57.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:57.194 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:41:57.246 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:41:57.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:57.247 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-01-24T17:41:57.299 INFO:teuthology.orchestra.run.smithi039.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-24T17:41:57.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:57.300 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:57.300 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:57.300 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:41:57.301 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-24T17:41:57.301 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:57.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:57.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether ba:58:2c:a4:c6:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::b858:2cff:fea4:c601/64 scope link 2024-01-24T17:41:57.537 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:57.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:57.542 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether ba:58:2c:a4:c6:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::b858:2cff:fea4:c601/64 scope link 2024-01-24T17:41:57.646 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:41:57.650 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-24T17:41:57.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:57.650 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:57.650 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:57.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-24T17:41:57.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-24T17:41:57.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-24T17:41:57.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-24T17:41:57.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-24T17:41:57.650 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:57.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:57.903 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:41:57.903 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:41:57.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.1 up 2024-01-24T17:41:57.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.1 master ceph-brx 2024-01-24T17:41:57.904 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:41:58.072 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-01-24T17:41:58.072 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:41:58.072 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-24T17:41:58.121 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-24T17:41:58.121 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:41:58.121 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-24T17:41:58.165 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:41:58.310 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:41:58.310 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:41:58.406 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:41:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:41:58.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:58.411 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:41:58.471 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:41:58.471 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [39] 2024-01-24T17:41:58.472 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-01-24T17:41:58.479 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:41:58.559 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:41:58.559 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:41:58.629 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:41:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:41:58.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:58.633 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:41:58.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi039.stderr:2024-01-24T17:41:58.674+0000 7f89c4787480 -1 init, newargv = 0x555d3c5c3d10 newargc=15 2024-01-24T17:41:58.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi039.stderr:ceph-fuse[45674]: starting ceph client 2024-01-24T17:41:58.676 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:41:58.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi039.stderr:ceph-fuse[45674]: starting fuse 2024-01-24T17:41:59.778 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:41:59.884 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:41:59.884 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:41:59.980 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:41:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:41:59.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:41:59.985 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:42:00.042 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:42:00.042 INFO:teuthology.orchestra.run.smithi039.stdout:42 2024-01-24T17:42:00.043 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39, 42] 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-01-24T17:42:00.043 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:42:00.044 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.2")) 2024-01-24T17:42:00.045 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:42:00.231 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.2.45674.asok 2024-01-24T17:42:00.412 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.45674.asok 2024-01-24T17:42:00.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:00.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.45674.asok status 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "2", 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "45674", 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:42:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 4662, 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 4662 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:42:00.658 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.2:0", 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3408073372 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.2:0", 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3408073372 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.4662 192.168.0.2:0/3408073372", 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.2:0/3408073372", 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 7, 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 20, 2024-01-24T17:42:00.659 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:42:00.660 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:42:00.660 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:42:00.660 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:42:00.673 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:42:00.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.896+0000 7f89dfe00640 1 Processor -- start 2024-01-24T17:42:00.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.896+0000 7f89dfe00640 1 -- start start 2024-01-24T17:42:00.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.897+0000 7f89dfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 0x7f89e00cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:00.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.897+0000 7f89dfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89e00ccd20 0x7f89e00ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:00.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.897+0000 7f89dfe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00ce400 con 0x7f89e015a720 2024-01-24T17:42:00.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.897+0000 7f89dfe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e015b3b0 con 0x7f89e00ccd20 2024-01-24T17:42:00.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.897+0000 7f89dfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e015b690 con 0x7f89e015ab00 2024-01-24T17:42:00.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.898+0000 7f89df400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89e00ccd20 0x7f89e00ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:00.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.898+0000 7f89df400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89e00ccd20 0x7f89e00ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56416/0 (socket says 172.21.15.39:56416) 2024-01-24T17:42:00.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.898+0000 7f89df400640 1 -- 172.21.15.39:0/4122237334 learned_addr learned my addr 172.21.15.39:0/4122237334 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:00.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.898+0000 7f89d7e00640 1 --2- 172.21.15.39:0/4122237334 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 0x7f89e00cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:00.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.898+0000 7f89df400640 1 -- 172.21.15.39:0/4122237334 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 msgr2=0x7f89e00cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:00.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.898+0000 7f89df400640 1 --2- 172.21.15.39:0/4122237334 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 0x7f89e00cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:00.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.898+0000 7f89df400640 1 -- 172.21.15.39:0/4122237334 >> v1:172.21.15.100:6789/0 conn(0x7f89e015a720 legacy=0x7f89e00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:00.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.898+0000 7f89df400640 1 -- 172.21.15.39:0/4122237334 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e015b970 con 0x7f89e00ccd20 2024-01-24T17:42:00.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.899+0000 7f89df400640 1 --2- 172.21.15.39:0/4122237334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00ccd20 0x7f89e00ce8e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009e50 tx=0x7f89d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec84e224de261271 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:00.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89de000640 1 -- 172.21.15.39:0/4122237334 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d802c9e0 con 0x7f89e00ccd20 2024-01-24T17:42:00.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89de000640 1 -- 172.21.15.39:0/4122237334 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d802cb40 con 0x7f89e00ccd20 2024-01-24T17:42:00.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89de000640 1 -- 172.21.15.39:0/4122237334 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d80343f0 con 0x7f89e00ccd20 2024-01-24T17:42:00.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89dfe00640 1 -- 172.21.15.39:0/4122237334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00ccd20 msgr2=0x7f89e00ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:00.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89dfe00640 1 --2- 172.21.15.39:0/4122237334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00ccd20 0x7f89e00ce8e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009e50 tx=0x7f89d802be40 comp rx=0 tx=0).stop 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89dfe00640 1 -- 172.21.15.39:0/4122237334 shutdown_connections 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89dfe00640 1 --2- 172.21.15.39:0/4122237334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00ccd20 0x7f89e00ce8e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89dfe00640 1 --2- 172.21.15.39:0/4122237334 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 0x7f89e00cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.900+0000 7f89dfe00640 1 -- 172.21.15.39:0/4122237334 >> 172.21.15.39:0/4122237334 conn(0x7f89e006a090 msgr2=0x7f89e006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.901+0000 7f89dfe00640 1 -- 172.21.15.39:0/4122237334 shutdown_connections 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.901+0000 7f89dfe00640 1 -- 172.21.15.39:0/4122237334 wait complete. 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.902+0000 7f89dfe00640 1 Processor -- start 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.902+0000 7f89dfe00640 1 -- start start 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.903+0000 7f89dfe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89e015a720 0x7f89e01f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.903+0000 7f89dfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 0x7f89e0154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.903+0000 7f89dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00b2b10 0x7f89e00b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.903+0000 7f89dfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e01f40d0 con 0x7f89e00b2b10 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.903+0000 7f89dfe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e0154dc0 con 0x7f89e015a720 2024-01-24T17:42:00.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.903+0000 7f89dfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e00b3060 con 0x7f89e015ab00 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00b2b10 0x7f89e00b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00b2b10 0x7f89e00b0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56418/0 (socket says 172.21.15.39:56418) 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 -- 172.21.15.39:0/2048289224 learned_addr learned my addr 172.21.15.39:0/2048289224 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89d7e00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 0x7f89e0154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89dea00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89e015a720 0x7f89e01f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 -- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 msgr2=0x7f89e0154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 0x7f89e0154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 -- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89e015a720 msgr2=0x7f89e01f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89e015a720 0x7f89e01f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 -- 172.21.15.39:0/2048289224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d8009b30 con 0x7f89e00b2b10 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.904+0000 7f89df400640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00b2b10 0x7f89e00b0b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f89d8036040 tx=0x7f89d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.905+0000 7f89d6a00640 1 -- 172.21.15.39:0/2048289224 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8009510 con 0x7f89e00b2b10 2024-01-24T17:42:00.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.905+0000 7f89d6a00640 1 -- 172.21.15.39:0/2048289224 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d803e450 con 0x7f89e00b2b10 2024-01-24T17:42:00.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.905+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e00b3340 con 0x7f89e00b2b10 2024-01-24T17:42:00.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.905+0000 7f89d6a00640 1 -- 172.21.15.39:0/2048289224 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d803e7e0 con 0x7f89e00b2b10 2024-01-24T17:42:00.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.905+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e00b3900 con 0x7f89e00b2b10 2024-01-24T17:42:00.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.906+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e00c70b0 con 0x7f89e00b2b10 2024-01-24T17:42:00.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.907+0000 7f89d6a00640 1 -- 172.21.15.39:0/2048289224 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f89d803e940 con 0x7f89e00b2b10 2024-01-24T17:42:00.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.908+0000 7f89d6a00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89b806a430 0x7f89b806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:00.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.908+0000 7f89d6a00640 1 -- 172.21.15.39:0/2048289224 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89d80ab710 con 0x7f89e00b2b10 2024-01-24T17:42:00.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.908+0000 7f89dea00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89b806a430 0x7f89b806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:00.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.909+0000 7f89dea00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89b806a430 0x7f89b806c8f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f89d0006fd0 tx=0x7f89d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:00.912+0000 7f89d6a00640 1 -- 172.21.15.39:0/2048289224 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f89d8076cf0 con 0x7f89e00b2b10 2024-01-24T17:42:01.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.143+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f89e004dff0 con 0x7f89e00b2b10 2024-01-24T17:42:01.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.144+0000 7f89d6a00640 1 -- 172.21.15.39:0/2048289224 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f89d807a310 con 0x7f89e00b2b10 2024-01-24T17:42:01.144 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89b806a430 msgr2=0x7f89b806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89b806a430 0x7f89b806c8f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f89d0006fd0 tx=0x7f89d0008040 comp rx=0 tx=0).stop 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00b2b10 msgr2=0x7f89e00b0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00b2b10 0x7f89e00b0b00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f89d8036040 tx=0x7f89d8004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 shutdown_connections 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89b806a430 0x7f89b806c8f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e00b2b10 0x7f89e00b0b00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89e015ab00 0x7f89e0154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 --2- 172.21.15.39:0/2048289224 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89e015a720 0x7f89e01f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:01.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 >> 172.21.15.39:0/2048289224 conn(0x7f89e006a090 msgr2=0x7f89e01599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:01.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 shutdown_connections 2024-01-24T17:42:01.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:01.147+0000 7f89dfe00640 1 -- 172.21.15.39:0/2048289224 wait complete. 2024-01-24T17:42:01.166 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:42:01.166 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:42:01.166 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:42:01.166 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-24T17:42:01.166 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:42:01.166 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:42:01.166 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:42:01.167 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:42:01.167 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:42:01.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:01.167 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:42:01.225 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42608sec preferred_lft 42608sec 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:42:01.226 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:01.227 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-24T17:42:01.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:01.227 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:01.227 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:01.227 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link add name ceph-brx type bridge 2024-01-24T17:42:01.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip addr flush dev ceph-brx 2024-01-24T17:42:01.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set ceph-brx up 2024-01-24T17:42:01.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-24T17:42:01.228 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:01.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:01.657 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:42:01.734 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:42:01.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:01.737 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:42:01.791 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:42:01.791 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:42:01.792 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:42:01.792 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:42:01.792 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:42:01.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:01.792 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:01.792 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:01.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:42:01.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:42:01.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:42:01.793 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:02.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:02.490 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:42:02.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:02.546 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list-id 2024-01-24T17:42:02.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:02.600 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:02.601 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:02.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:42:02.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-24T17:42:02.601 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:02.738 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-24T17:42:02.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:02.739 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:02.739 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:02.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-24T17:42:02.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-24T17:42:02.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-24T17:42:02.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-24T17:42:02.739 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-24T17:42:02.739 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:03.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:03.007 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:03.007 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:03.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:42:03.007 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set dev brx.0 master ceph-brx 2024-01-24T17:42:03.007 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:03.172 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-24T17:42:03.173 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:42:03.173 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:42:03.217 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:42:03.218 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:42:03.218 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:42:03.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:42:03.409 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:42:03.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:42:03.495 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:42:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:42:03.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:03.499 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:42:03.548 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:42:03.548 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-24T17:42:03.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:42:03.625 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:42:03.625 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:42:03.710 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:42:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:42:03.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:03.714 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:42:03.777 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:42:03.776+0000 7fa3c2649480 -1 init, newargv = 0x559131676ed0 newargc=15 2024-01-24T17:42:03.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[42500]: starting ceph client 2024-01-24T17:42:03.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[42500]: starting fuse 2024-01-24T17:42:04.861 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:42:04.957 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:42:04.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:42:05.050 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:42:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:42:05.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:05.054 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:42:05.104 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:42:05.105 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-24T17:42:05.105 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:42:05.106 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:42:05.107 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2024-01-24T17:42:05.107 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:42:05.278 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.42500.asok 2024-01-24T17:42:05.459 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42500.asok 2024-01-24T17:42:05.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:05.459 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42500.asok status 2024-01-24T17:42:05.695 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:42:05.695 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "42500", 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 4475, 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:42:05.696 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 4475 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3939708107 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3939708107 2024-01-24T17:42:05.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.4475 192.168.0.1:0/3939708107", 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/3939708107", 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 7, 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 20, 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:42:05.698 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:42:05.710 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:42:05.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.936+0000 7f1f78e00640 1 Processor -- start 2024-01-24T17:42:05.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.937+0000 7f1f78e00640 1 -- start start 2024-01-24T17:42:05.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.938+0000 7f1f78e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1f7415ca30 0x7f1f740cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:05.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.938+0000 7f1f78e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f740c9c10 con 0x7f1f740cd3c0 2024-01-24T17:42:05.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.938+0000 7f1f78e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f740c9650 con 0x7f1f740caf00 2024-01-24T17:42:05.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.938+0000 7f1f78e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f740c9930 con 0x7f1f7415ca30 2024-01-24T17:42:05.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.938+0000 7f1f69200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1f740cd3c0 0x7f1f740c8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55778/0 (socket says 172.21.15.39:55778) 2024-01-24T17:42:05.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.938+0000 7f1f69200640 1 -- 172.21.15.39:0/1812611762 learned_addr learned my addr 172.21.15.39:0/1812611762 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775619914 0 0) 0x7f1f740c9c10 con 0x7f1f740cd3c0 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71e00640 1 --2- 172.21.15.39:0/1812611762 >> v2:172.21.15.100:3300/0 conn(0x7f1f7415ca30 0x7f1f740cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f54003520 con 0x7f1f740cd3c0 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1673912586 0 0) 0x7f1f54003520 con 0x7f1f740cd3c0 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 >> v2:172.21.15.100:3300/0 conn(0x7f1f7415ca30 msgr2=0x7f1f740cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 --2- 172.21.15.39:0/1812611762 >> v2:172.21.15.100:3300/0 conn(0x7f1f7415ca30 0x7f1f740cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 >> v1:172.21.15.100:6789/0 conn(0x7f1f740caf00 legacy=0x7f1f74070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f740cc680 con 0x7f1f740cd3c0 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1f5c001670 con 0x7f1f740cd3c0 2024-01-24T17:42:05.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 >> v1:172.21.15.39:6789/0 conn(0x7f1f740cd3c0 legacy=0x7f1f740c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f54003970 0x7f1f54003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f540047c0 0x7f1f54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740c9930 con 0x7f1f54005180 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5400c020 con 0x7f1f540047c0 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.939+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5400c1e0 con 0x7f1f54003970 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.940+0000 7f1f72800640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.940+0000 7f1f69200640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f540047c0 0x7f1f54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.940+0000 7f1f71e00640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f54003970 0x7f1f54003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.940+0000 7f1f72800640 1 -- 172.21.15.39:0/1812611762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f54003970 msgr2=0x7f1f54003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.940+0000 7f1f72800640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f54003970 0x7f1f54003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.940+0000 7f1f72800640 1 -- 172.21.15.39:0/1812611762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f540047c0 msgr2=0x7f1f54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.940+0000 7f1f72800640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f540047c0 0x7f1f54004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:05.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.940+0000 7f1f72800640 1 -- 172.21.15.39:0/1812611762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f54003520 con 0x7f1f54005180 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.941+0000 7f1f72800640 1 -- 172.21.15.39:0/1812611762 reap_dead start 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.941+0000 7f1f72800640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f5400b980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c001e50 tx=0x7f1f540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44727dc5df75b865 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c003b70 con 0x7f1f54005180 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c003d10 con 0x7f1f54005180 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f71400640 1 -- 172.21.15.39:0/1812611762 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c005310 con 0x7f1f54005180 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f78e00640 1 -- 172.21.15.39:0/1812611762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 msgr2=0x7f1f5400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f78e00640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f5400b980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c001e50 tx=0x7f1f540057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f78e00640 1 -- 172.21.15.39:0/1812611762 shutdown_connections 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f78e00640 1 --2- 172.21.15.39:0/1812611762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f5400b980 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f78e00640 1 -- 172.21.15.39:0/1812611762 >> 172.21.15.39:0/1812611762 conn(0x7f1f7406a3e0 msgr2=0x7f1f7406bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:05.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f78e00640 1 -- 172.21.15.39:0/1812611762 shutdown_connections 2024-01-24T17:42:05.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.942+0000 7f1f78e00640 1 -- 172.21.15.39:0/1812611762 wait complete. 2024-01-24T17:42:05.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.944+0000 7f1f78e00640 1 Processor -- start 2024-01-24T17:42:05.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.944+0000 7f1f78e00640 1 -- start start 2024-01-24T17:42:05.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.945+0000 7f1f78e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f741f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:05.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.945+0000 7f1f78e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f741f2fc0 0x7f1f740c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:05.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.945+0000 7f1f78e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f740c4930 0x7f1f741f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:05.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.945+0000 7f1f78e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740c2960 con 0x7f1f54005180 2024-01-24T17:42:05.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.945+0000 7f1f78e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f741f3440 con 0x7f1f741f2fc0 2024-01-24T17:42:05.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.945+0000 7f1f78e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740c4db0 con 0x7f1f740c4930 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f741f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f741f4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39308/0 (socket says 172.21.15.39:39308) 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 -- 172.21.15.39:0/2376055488 learned_addr learned my addr 172.21.15.39:0/2376055488 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 -- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f740c4930 msgr2=0x7f1f741f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f740c4930 0x7f1f741f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f69200640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f741f2fc0 0x7f1f740c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 -- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f741f2fc0 msgr2=0x7f1f740c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f741f2fc0 0x7f1f740c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 -- 172.21.15.39:0/2376055488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f740c2050 con 0x7f1f54005180 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.946+0000 7f1f71e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f741f4db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1f64010860 tx=0x7f1f64010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.947+0000 7f1f6aa00640 1 -- 172.21.15.39:0/2376055488 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f64009610 con 0x7f1f54005180 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.947+0000 7f1f6aa00640 1 -- 172.21.15.39:0/2376055488 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6400eb00 con 0x7f1f54005180 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.947+0000 7f1f6aa00640 1 -- 172.21.15.39:0/2376055488 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6400ee90 con 0x7f1f54005180 2024-01-24T17:42:05.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.947+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f741f2c50 con 0x7f1f54005180 2024-01-24T17:42:05.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.947+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f740c2580 con 0x7f1f54005180 2024-01-24T17:42:05.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.948+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f740c2bd0 con 0x7f1f54005180 2024-01-24T17:42:05.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.949+0000 7f1f6aa00640 1 -- 172.21.15.39:0/2376055488 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1f6401c400 con 0x7f1f54005180 2024-01-24T17:42:05.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.950+0000 7f1f6aa00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f4c06a430 0x7f1f4c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:05.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.950+0000 7f1f6aa00640 1 -- 172.21.15.39:0/2376055488 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f64088e80 con 0x7f1f54005180 2024-01-24T17:42:05.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.951+0000 7f1f69200640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f4c06a430 0x7f1f4c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:05.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.952+0000 7f1f69200640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f4c06a430 0x7f1f4c06c8f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c001200 tx=0x7f1f5c000f30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:05.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:05.955+0000 7f1f6aa00640 1 -- 172.21.15.39:0/2376055488 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1f64054460 con 0x7f1f54005180 2024-01-24T17:42:06.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.186+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1f74073ef0 con 0x7f1f54005180 2024-01-24T17:42:06.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.187+0000 7f1f6aa00640 1 -- 172.21.15.39:0/2376055488 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1f64057a80 con 0x7f1f54005180 2024-01-24T17:42:06.188 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f4c06a430 msgr2=0x7f1f4c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f4c06a430 0x7f1f4c06c8f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c001200 tx=0x7f1f5c000f30 comp rx=0 tx=0).stop 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 msgr2=0x7f1f741f4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f741f4db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1f64010860 tx=0x7f1f64010d20 comp rx=0 tx=0).stop 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 shutdown_connections 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f740c4930 0x7f1f741f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f741f2fc0 0x7f1f740c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f4c06a430 0x7f1f4c06c8f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 --2- 172.21.15.39:0/2376055488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f54005180 0x7f1f741f4db0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 >> 172.21.15.39:0/2376055488 conn(0x7f1f7406a3e0 msgr2=0x7f1f7415e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 shutdown_connections 2024-01-24T17:42:06.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:06.190+0000 7f1f78e00640 1 -- 172.21.15.39:0/2376055488 wait complete. 2024-01-24T17:42:06.210 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:42:06.210 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:42:06.211 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:42:06.211 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-24T17:42:06.211 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-24T17:42:06.211 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:42:06.211 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:42:06.211 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:42:06.211 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-24T17:42:06.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.211 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:42:06.267 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:42:06.267 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:42:06.268 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42603sec preferred_lft 42603sec 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:42:06.269 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.271 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:42:06.357 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:42:06.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.361 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:42:06.412 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:42:06.413 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:42:06.413 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:42:06.413 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:42:06.413 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:42:06.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.413 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:06.413 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:06.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:42:06.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:42:06.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:42:06.414 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:06.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.592 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:42:06.641 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:42:06.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.642 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list-id 2024-01-24T17:42:06.691 INFO:teuthology.orchestra.run.smithi100.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-24T17:42:06.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.692 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:06.692 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:06.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:42:06.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-24T17:42:06.692 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:06.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.839 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-24T17:42:06.925 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:42:06.925 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ce:34:fd:35:93:9c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::cc34:fdff:fe35:939c/64 scope link 2024-01-24T17:42:06.926 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:06.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:06.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-24T17:42:07.016 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether ce:34:fd:35:93:9c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::cc34:fdff:fe35:939c/64 scope link 2024-01-24T17:42:07.017 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:42:07.021 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-24T17:42:07.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:07.022 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:07.022 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:07.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-24T17:42:07.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-24T17:42:07.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-24T17:42:07.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-24T17:42:07.022 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-24T17:42:07.022 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:07.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:07.275 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:42:07.275 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:42:07.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.1 up 2024-01-24T17:42:07.275 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set dev brx.1 master ceph-brx 2024-01-24T17:42:07.275 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:42:07.432 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-01-24T17:42:07.432 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:42:07.432 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-24T17:42:07.479 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-24T17:42:07.480 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:42:07.480 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-24T17:42:07.575 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:42:07.671 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:42:07.671 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:42:07.762 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:42:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:42:07.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:07.767 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:42:07.821 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:42:07.821 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [39] 2024-01-24T17:42:07.821 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-01-24T17:42:07.826 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:42:07.897 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:42:07.897 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:42:07.964 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:42:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:42:07.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:07.967 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:42:08.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi100.stderr:2024-01-24T17:42:08.000+0000 7f0ad837c480 -1 init, newargv = 0x5597871a8d90 newargc=15 2024-01-24T17:42:08.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi100.stderr:ceph-fuse[43010]: starting ceph client 2024-01-24T17:42:08.009 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:42:08.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi100.stderr:ceph-fuse[43010]: starting fuse 2024-01-24T17:42:09.111 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:42:09.211 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:42:09.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:42:09.303 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:42:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:42:09.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:09.307 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:42:09.360 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:42:09.360 INFO:teuthology.orchestra.run.smithi100.stdout:42 2024-01-24T17:42:09.360 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39, 42] 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:42:09.361 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.3")) 2024-01-24T17:42:09.362 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:42:09.539 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.3.43010.asok 2024-01-24T17:42:09.721 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.43010.asok 2024-01-24T17:42:09.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:09.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.43010.asok status 2024-01-24T17:42:09.956 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:42:09.956 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:42:09.956 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:42:09.956 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:42:09.956 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "3", 2024-01-24T17:42:09.956 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:42:09.956 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-01-24T17:42:09.956 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "43010", 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 4481, 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 4481 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.2:0", 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1731803620 2024-01-24T17:42:09.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.2:0", 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1731803620 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.4481 192.168.0.2:0/1731803620", 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.2:0/1731803620", 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 7, 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 20, 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:42:09.958 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:42:09.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:09.971 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:42:09.971 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:42:10.031 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:42:10.032 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:42:10.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:10.032 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:42:10.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:10.134 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:42:10.134 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.2 2024-01-24T17:42:10.191 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:42:10.191 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-01-24T17:42:10.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:10.191 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-24T17:42:10.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:10.297 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:42:10.297 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:42:10.355 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:42:10.356 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:42:10.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:10.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:42:10.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:10.452 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:42:10.452 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.3 2024-01-24T17:42:10.507 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:42:10.508 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-01-24T17:42:10.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:10.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-24T17:42:10.595 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-24T17:42:10.635 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:10.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.865+0000 7fbff9400640 1 Processor -- start 2024-01-24T17:42:10.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.865+0000 7fbff9400640 1 -- start start 2024-01-24T17:42:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.867+0000 7fbff9400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbff415a950 0x7fbff40ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.867+0000 7fbff9400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40cb660 0x7fbff40b1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.867+0000 7fbff9400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff40c9180 con 0x7fbff40caaa0 2024-01-24T17:42:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.867+0000 7fbff9400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff40b43d0 con 0x7fbff415a950 2024-01-24T17:42:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.867+0000 7fbff9400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c8f70 con 0x7fbff40cb660 2024-01-24T17:42:10.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.867+0000 7fbfebe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fbff40caaa0 0x7fbff40caf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55788/0 (socket says 172.21.15.39:55788) 2024-01-24T17:42:10.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.867+0000 7fbfebe00640 1 -- 172.21.15.39:0/2151578797 learned_addr learned my addr 172.21.15.39:0/2151578797 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff2800640 1 --2- 172.21.15.39:0/2151578797 >> v2:172.21.15.100:3300/0 conn(0x7fbff415a950 0x7fbff40ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff3200640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40cb660 0x7fbff40b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45241190 0 0) 0x7fbff40c9180 con 0x7fbff40caaa0 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd4003520 con 0x7fbff40caaa0 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2643511540 0 0) 0x7fbfd4003520 con 0x7fbff40caaa0 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40cb660 msgr2=0x7fbff40b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40cb660 0x7fbff40b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 >> v2:172.21.15.100:3300/0 conn(0x7fbff415a950 msgr2=0x7fbff40ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 --2- 172.21.15.39:0/2151578797 >> v2:172.21.15.100:3300/0 conn(0x7fbff415a950 0x7fbff40ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff40c9460 con 0x7fbff40caaa0 2024-01-24T17:42:10.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fbfdc001690 con 0x7fbff40caaa0 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.868+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 >> v1:172.21.15.39:6789/0 conn(0x7fbff40caaa0 legacy=0x7fbff40caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff1e00640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd4003970 0x7fbfd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff1e00640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40cb660 0x7fbfd40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff1e00640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbfd4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40b43d0 con 0x7fbfd4004d00 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c8f70 con 0x7fbfd4003970 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd400b980 con 0x7fbff40cb660 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff3200640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbfd4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbfebe00640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40cb660 0x7fbfd40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff2800640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd4003970 0x7fbfd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff3200640 1 -- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40cb660 msgr2=0x7fbfd40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:10.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff3200640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40cb660 0x7fbfd40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff3200640 1 -- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd4003970 msgr2=0x7fbfd4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff3200640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd4003970 0x7fbfd4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff3200640 1 -- 172.21.15.39:0/2151578797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd4003520 con 0x7fbfd4004d00 2024-01-24T17:42:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.869+0000 7fbff2800640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd4003970 0x7fbfd4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:10.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.870+0000 7fbff3200640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbfd4005180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00e170 tx=0x7fbfd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998dd1d1755f441c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:10.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.871+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec02cbf0 con 0x7fbfd4004d00 2024-01-24T17:42:10.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.871+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec00f040 con 0x7fbfd4004d00 2024-01-24T17:42:10.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.871+0000 7fbff1e00640 1 -- 172.21.15.39:0/2151578797 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec034440 con 0x7fbfd4004d00 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.871+0000 7fbff9400640 1 -- 172.21.15.39:0/2151578797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 msgr2=0x7fbfd4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.871+0000 7fbff9400640 1 --2- 172.21.15.39:0/2151578797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbfd4005180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00e170 tx=0x7fbfd40057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.871+0000 7fbff3200640 1 -- 172.21.15.39:0/2151578797 reap_dead start 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.871+0000 7fbff9400640 1 -- 172.21.15.39:0/2151578797 shutdown_connections 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.871+0000 7fbff9400640 1 -- 172.21.15.39:0/2151578797 >> 172.21.15.39:0/2151578797 conn(0x7fbff406a3e0 msgr2=0x7fbff406bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.872+0000 7fbff9400640 1 -- 172.21.15.39:0/2151578797 shutdown_connections 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.872+0000 7fbff9400640 1 -- 172.21.15.39:0/2151578797 wait complete. 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.873+0000 7fbff9400640 1 Processor -- start 2024-01-24T17:42:10.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.873+0000 7fbff9400640 1 -- start start 2024-01-24T17:42:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.874+0000 7fbff9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbff41585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.874+0000 7fbff9400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbff40acb80 0x7fbff40cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.874+0000 7fbff9400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40ad000 0x7fbff40cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.874+0000 7fbff9400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c6670 con 0x7fbfd4004d00 2024-01-24T17:42:10.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.874+0000 7fbff9400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40a8d90 con 0x7fbff40acb80 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.874+0000 7fbff9400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40c59f0 con 0x7fbff40ad000 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbff41585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbff41585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39314/0 (socket says 172.21.15.39:39314) 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff2800640 1 -- 172.21.15.39:0/3964146876 learned_addr learned my addr 172.21.15.39:0/3964146876 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbfebe00640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbff40acb80 0x7fbff40cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff3200640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40ad000 0x7fbff40cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff2800640 1 -- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40ad000 msgr2=0x7fbff40cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff2800640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40ad000 0x7fbff40cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff2800640 1 -- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbff40acb80 msgr2=0x7fbff40cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff2800640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbff40acb80 0x7fbff40cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:10.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.875+0000 7fbff2800640 1 -- 172.21.15.39:0/3964146876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff40c5cd0 con 0x7fbfd4004d00 2024-01-24T17:42:10.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.876+0000 7fbff2800640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbff41585a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4010a00 tx=0x7fbfe4010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:10.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.876+0000 7fbfeaa00640 1 -- 172.21.15.39:0/3964146876 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400ec20 con 0x7fbfd4004d00 2024-01-24T17:42:10.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.876+0000 7fbfeaa00640 1 -- 172.21.15.39:0/3964146876 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe400ed80 con 0x7fbfd4004d00 2024-01-24T17:42:10.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.876+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff40c5fb0 con 0x7fbfd4004d00 2024-01-24T17:42:10.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.876+0000 7fbfeaa00640 1 -- 172.21.15.39:0/3964146876 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe401c620 con 0x7fbfd4004d00 2024-01-24T17:42:10.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.876+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff4158ae0 con 0x7fbfd4004d00 2024-01-24T17:42:10.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.877+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff40c6c80 con 0x7fbfd4004d00 2024-01-24T17:42:10.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.878+0000 7fbfeaa00640 1 -- 172.21.15.39:0/3964146876 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbfe401c780 con 0x7fbfd4004d00 2024-01-24T17:42:10.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.879+0000 7fbfeaa00640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfcc06a430 0x7fbfcc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:10.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.879+0000 7fbfeaa00640 1 -- 172.21.15.39:0/3964146876 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfe4088f80 con 0x7fbfd4004d00 2024-01-24T17:42:10.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.880+0000 7fbfebe00640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfcc06a430 0x7fbfcc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:10.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.881+0000 7fbfebe00640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfcc06a430 0x7fbfcc06c8f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbff4070dc0 tx=0x7fbfdc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:10.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:10.884+0000 7fbfeaa00640 1 -- 172.21.15.39:0/3964146876 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbff40c6c80 con 0x7fbfd4004d00 2024-01-24T17:42:11.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.079+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbff40c6c80 con 0x7fbfd4004d00 2024-01-24T17:42:11.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.082+0000 7fbfeaa00640 1 -- 172.21.15.39:0/3964146876 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7fbff40c6c80 con 0x7fbfd4004d00 2024-01-24T17:42:11.084 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:11.084 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:11.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfcc06a430 msgr2=0x7fbfcc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfcc06a430 0x7fbfcc06c8f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbff4070dc0 tx=0x7fbfdc009040 comp rx=0 tx=0).stop 2024-01-24T17:42:11.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 msgr2=0x7fbff41585a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbff41585a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4010a00 tx=0x7fbfe4010ec0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 shutdown_connections 2024-01-24T17:42:11.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbff40ad000 0x7fbff40cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbff40acb80 0x7fbff40cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfcc06a430 0x7fbfcc06c8f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 --2- 172.21.15.39:0/3964146876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd4004d00 0x7fbff41585a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.085+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 >> 172.21.15.39:0/3964146876 conn(0x7fbff406a3e0 msgr2=0x7fbff4159ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:11.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.086+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 shutdown_connections 2024-01-24T17:42:11.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.086+0000 7fbff9400640 1 -- 172.21.15.39:0/3964146876 wait complete. 2024-01-24T17:42:11.107 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:11.108 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:11.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.322+0000 7f5337000640 1 Processor -- start 2024-01-24T17:42:11.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.323+0000 7f5337000640 1 -- start start 2024-01-24T17:42:11.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.324+0000 7f5337000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f53300b2710 0x7f53300d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.324+0000 7f5337000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53300b38a0 con 0x7f53300d2e80 2024-01-24T17:42:11.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.324+0000 7f5337000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53300d13e0 con 0x7f533015c8d0 2024-01-24T17:42:11.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.324+0000 7f5337000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53300d0ab0 con 0x7f53300b2710 2024-01-24T17:42:11.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.324+0000 7f5335200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f53300d2e80 0x7f53300cd630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55794/0 (socket says 172.21.15.39:55794) 2024-01-24T17:42:11.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.324+0000 7f5335200640 1 -- 172.21.15.39:0/2419050859 learned_addr learned my addr 172.21.15.39:0/2419050859 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:11.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307014017 0 0) 0x7f53300b38a0 con 0x7f53300d2e80 2024-01-24T17:42:11.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5310003520 con 0x7f53300d2e80 2024-01-24T17:42:11.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f5336600640 1 --2- 172.21.15.39:0/2419050859 >> v2:172.21.15.100:3300/0 conn(0x7f53300b2710 0x7f53300d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 484368186 0 0) 0x7f5310003520 con 0x7f53300d2e80 2024-01-24T17:42:11.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 >> v2:172.21.15.100:3300/0 conn(0x7f53300b2710 msgr2=0x7f53300d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:11.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 --2- 172.21.15.39:0/2419050859 >> v2:172.21.15.100:3300/0 conn(0x7f53300b2710 0x7f53300d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 >> v1:172.21.15.100:6789/0 conn(0x7f533015c8d0 legacy=0x7f53300d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:11.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53300d0d90 con 0x7f53300d2e80 2024-01-24T17:42:11.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f5318001670 con 0x7f53300d2e80 2024-01-24T17:42:11.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 >> v1:172.21.15.39:6789/0 conn(0x7f53300d2e80 legacy=0x7f53300cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5310003970 0x7f5310003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53100047c0 0x7f5310004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5310005180 0x7f531000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53300d0ab0 con 0x7f53100047c0 2024-01-24T17:42:11.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531000c0b0 con 0x7f5310003970 2024-01-24T17:42:11.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.325+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531000c270 con 0x7f5310005180 2024-01-24T17:42:11.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5335200640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53100047c0 0x7f5310004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5335c00640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5310003970 0x7f5310003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5336600640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5310005180 0x7f531000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5335200640 1 -- 172.21.15.39:0/2419050859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5310005180 msgr2=0x7f531000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:11.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5335200640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5310005180 0x7f531000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5335200640 1 -- 172.21.15.39:0/2419050859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5310003970 msgr2=0x7f5310003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:11.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5335200640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5310003970 0x7f5310003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5335200640 1 -- 172.21.15.39:0/2419050859 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5310003520 con 0x7f53100047c0 2024-01-24T17:42:11.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.326+0000 7f5336600640 1 -- 172.21.15.39:0/2419050859 reap_dead start 2024-01-24T17:42:11.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.327+0000 7f5335200640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53100047c0 0x7f5310004c40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5318001eb0 tx=0x7f53100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82049e807273a1e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:11.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318005ad0 con 0x7f53100047c0 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5318005c50 con 0x7f53100047c0 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f532fe00640 1 -- 172.21.15.39:0/2419050859 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531802c850 con 0x7f53100047c0 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f5337000640 1 -- 172.21.15.39:0/2419050859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53100047c0 msgr2=0x7f5310004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f5337000640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53100047c0 0x7f5310004c40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5318001eb0 tx=0x7f53100057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f5337000640 1 -- 172.21.15.39:0/2419050859 shutdown_connections 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f5337000640 1 --2- 172.21.15.39:0/2419050859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53100047c0 0x7f5310004c40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f5337000640 1 -- 172.21.15.39:0/2419050859 >> 172.21.15.39:0/2419050859 conn(0x7f5330071dd0 msgr2=0x7f533009bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f5337000640 1 -- 172.21.15.39:0/2419050859 shutdown_connections 2024-01-24T17:42:11.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.328+0000 7f5337000640 1 -- 172.21.15.39:0/2419050859 wait complete. 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.330+0000 7f5337000640 1 Processor -- start 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.330+0000 7f5337000640 1 -- start start 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5337000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53100047c0 0x7f53300ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5337000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53301505e0 0x7f53301515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5337000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5330151b30 0x7f533015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5337000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330150a60 con 0x7f53301505e0 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5337000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53300b0e90 con 0x7f53100047c0 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5337000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53300c98b0 con 0x7f5330151b30 2024-01-24T17:42:11.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5335200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53301505e0 0x7f53301515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5335200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53301505e0 0x7f53301515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39340/0 (socket says 172.21.15.39:39340) 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.331+0000 7f5335200640 1 -- 172.21.15.39:0/975066955 learned_addr learned my addr 172.21.15.39:0/975066955 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f5336600640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5330151b30 0x7f533015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f5335c00640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53100047c0 0x7f53300ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f5335200640 1 -- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5330151b30 msgr2=0x7f533015a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f5335200640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5330151b30 0x7f533015a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f5335200640 1 -- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53100047c0 msgr2=0x7f53300ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f5335200640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53100047c0 0x7f53300ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f5335200640 1 -- 172.21.15.39:0/975066955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53300ca430 con 0x7f53301505e0 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f5335200640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53301505e0 0x7f53301515f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f531802c3f0 tx=0x7f531802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.332+0000 7f532d600640 1 -- 172.21.15.39:0/975066955 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531803c040 con 0x7f53301505e0 2024-01-24T17:42:11.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.333+0000 7f532d600640 1 -- 172.21.15.39:0/975066955 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531802cd20 con 0x7f53301505e0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.333+0000 7f532d600640 1 -- 172.21.15.39:0/975066955 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318007270 con 0x7f53301505e0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.333+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53300c9b00 con 0x7f53301505e0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.333+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53300ca0c0 con 0x7f53301505e0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.334+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533004dff0 con 0x7f53301505e0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.335+0000 7f532d600640 1 -- 172.21.15.39:0/975066955 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5318032070 con 0x7f53301505e0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.336+0000 7f532d600640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f530806a430 0x7f530806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.336+0000 7f532d600640 1 -- 172.21.15.39:0/975066955 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53180a9cc0 con 0x7f53301505e0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.336+0000 7f5335c00640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f530806a430 0x7f530806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.337+0000 7f5335c00640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f530806a430 0x7f530806c8f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5320007fd0 tx=0x7f5320009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:11.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.340+0000 7f532d600640 1 -- 172.21.15.39:0/975066955 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f53180b0080 con 0x7f53301505e0 2024-01-24T17:42:11.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.532+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53300d1fb0 con 0x7f53301505e0 2024-01-24T17:42:11.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.533+0000 7f532d600640 1 -- 172.21.15.39:0/975066955 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5318075320 con 0x7f53301505e0 2024-01-24T17:42:11.535 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:11.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.535+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f530806a430 msgr2=0x7f530806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.535+0000 7f5337000640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f530806a430 0x7f530806c8f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5320007fd0 tx=0x7f5320009040 comp rx=0 tx=0).stop 2024-01-24T17:42:11.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.535+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53301505e0 msgr2=0x7f53301515f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.535+0000 7f5337000640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53301505e0 0x7f53301515f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f531802c3f0 tx=0x7f531802e070 comp rx=0 tx=0).stop 2024-01-24T17:42:11.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.536+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 shutdown_connections 2024-01-24T17:42:11.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.536+0000 7f5337000640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5330151b30 0x7f533015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.536+0000 7f5337000640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f530806a430 0x7f530806c8f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.536+0000 7f5337000640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53301505e0 0x7f53301515f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.536+0000 7f5337000640 1 --2- 172.21.15.39:0/975066955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53100047c0 0x7f53300ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.536+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 >> 172.21.15.39:0/975066955 conn(0x7f5330071dd0 msgr2=0x7f53300b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:11.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.536+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 shutdown_connections 2024-01-24T17:42:11.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.536+0000 7f5337000640 1 -- 172.21.15.39:0/975066955 wait complete. 2024-01-24T17:42:11.558 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:11.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.767+0000 7ffa39600640 1 Processor -- start 2024-01-24T17:42:11.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.767+0000 7ffa39600640 1 -- start start 2024-01-24T17:42:11.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.768+0000 7ffa39600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffa3415c8d0 0x7ffa340d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.768+0000 7ffa39600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffa340b2710 0x7ffa340d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.768+0000 7ffa39600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340d2cb0 0x7ffa340cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.768+0000 7ffa39600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa340d3120 con 0x7ffa3415c8d0 2024-01-24T17:42:11.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.768+0000 7ffa39600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa340b38a0 con 0x7ffa340b2710 2024-01-24T17:42:11.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.768+0000 7ffa39600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa340d13e0 con 0x7ffa340d2cb0 2024-01-24T17:42:11.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.769+0000 7ffa33e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffa3415c8d0 0x7ffa340d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.769+0000 7ffa33e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffa3415c8d0 0x7ffa340d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39348/0 (socket says 172.21.15.39:39348) 2024-01-24T17:42:11.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.769+0000 7ffa33e00640 1 -- 172.21.15.39:0/192719554 learned_addr learned my addr 172.21.15.39:0/192719554 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:11.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.769+0000 7ffa33400640 1 --2- 172.21.15.39:0/192719554 >> v2:172.21.15.100:3300/0 conn(0x7ffa340b2710 0x7ffa340d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.769+0000 7ffa38c00640 1 --2- 172.21.15.39:0/192719554 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340d2cb0 0x7ffa340cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.769+0000 7ffa33e00640 1 -- 172.21.15.39:0/192719554 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340d2cb0 msgr2=0x7ffa340cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:11.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.769+0000 7ffa33e00640 1 --2- 172.21.15.39:0/192719554 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340d2cb0 0x7ffa340cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.770+0000 7ffa33e00640 1 -- 172.21.15.39:0/192719554 >> v2:172.21.15.100:3300/0 conn(0x7ffa340b2710 msgr2=0x7ffa340d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:11.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.770+0000 7ffa33e00640 1 --2- 172.21.15.39:0/192719554 >> v2:172.21.15.100:3300/0 conn(0x7ffa340b2710 0x7ffa340d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.770+0000 7ffa33e00640 1 -- 172.21.15.39:0/192719554 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa340d0ab0 con 0x7ffa3415c8d0 2024-01-24T17:42:11.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.772+0000 7ffa33e00640 1 --2- 172.21.15.39:0/192719554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa3415c8d0 0x7ffa340d03b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ffa340cd0f0 tx=0x7ffa2402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bf3a974442deaa server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:11.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.772+0000 7ffa32a00640 1 -- 172.21.15.39:0/192719554 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24031640 con 0x7ffa3415c8d0 2024-01-24T17:42:11.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.772+0000 7ffa32a00640 1 -- 172.21.15.39:0/192719554 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa240317a0 con 0x7ffa3415c8d0 2024-01-24T17:42:11.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.772+0000 7ffa32a00640 1 -- 172.21.15.39:0/192719554 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24031b30 con 0x7ffa3415c8d0 2024-01-24T17:42:11.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 -- 172.21.15.39:0/192719554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa3415c8d0 msgr2=0x7ffa340d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 --2- 172.21.15.39:0/192719554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa3415c8d0 0x7ffa340d03b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ffa340cd0f0 tx=0x7ffa2402b8f0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 -- 172.21.15.39:0/192719554 shutdown_connections 2024-01-24T17:42:11.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 --2- 172.21.15.39:0/192719554 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340d2cb0 0x7ffa340cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 --2- 172.21.15.39:0/192719554 >> v2:172.21.15.100:3300/0 conn(0x7ffa340b2710 0x7ffa340d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 --2- 172.21.15.39:0/192719554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa3415c8d0 0x7ffa340d03b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 -- 172.21.15.39:0/192719554 >> 172.21.15.39:0/192719554 conn(0x7ffa34071dd0 msgr2=0x7ffa3409bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:11.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 -- 172.21.15.39:0/192719554 shutdown_connections 2024-01-24T17:42:11.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.773+0000 7ffa39600640 1 -- 172.21.15.39:0/192719554 wait complete. 2024-01-24T17:42:11.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.775+0000 7ffa39600640 1 Processor -- start 2024-01-24T17:42:11.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.775+0000 7ffa39600640 1 -- start start 2024-01-24T17:42:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa39600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340b2710 0x7ffa341f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa39600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa340d2cb0 0x7ffa340b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa39600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa3415c8d0 0x7ffa340b7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa39600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3414f940 con 0x7ffa340d2cb0 2024-01-24T17:42:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa39600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa340b6f50 con 0x7ffa3415c8d0 2024-01-24T17:42:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa39600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa341f1fe0 con 0x7ffa340b2710 2024-01-24T17:42:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa33400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa340d2cb0 0x7ffa340b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa33400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa340d2cb0 0x7ffa340b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39360/0 (socket says 172.21.15.39:39360) 2024-01-24T17:42:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa33400640 1 -- 172.21.15.39:0/875985038 learned_addr learned my addr 172.21.15.39:0/875985038 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa33400640 1 -- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340b2710 msgr2=0x7ffa341f1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa33400640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340b2710 0x7ffa341f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa33400640 1 -- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa3415c8d0 msgr2=0x7ffa340b7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa33400640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa3415c8d0 0x7ffa340b7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.776+0000 7ffa33400640 1 -- 172.21.15.39:0/875985038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240094d0 con 0x7ffa340d2cb0 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.777+0000 7ffa33400640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa340d2cb0 0x7ffa340b77b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00d830 tx=0x7ffa1c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.777+0000 7ffa2be00640 1 -- 172.21.15.39:0/875985038 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c00be10 con 0x7ffa340d2cb0 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.777+0000 7ffa2be00640 1 -- 172.21.15.39:0/875985038 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c0098b0 con 0x7ffa340d2cb0 2024-01-24T17:42:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.777+0000 7ffa2be00640 1 -- 172.21.15.39:0/875985038 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c009a40 con 0x7ffa340d2cb0 2024-01-24T17:42:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.777+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa340b6340 con 0x7ffa340d2cb0 2024-01-24T17:42:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.777+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa340b6900 con 0x7ffa340d2cb0 2024-01-24T17:42:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.779+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3404dff0 con 0x7ffa340d2cb0 2024-01-24T17:42:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.779+0000 7ffa2be00640 1 -- 172.21.15.39:0/875985038 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ffa1c009ba0 con 0x7ffa340d2cb0 2024-01-24T17:42:11.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.780+0000 7ffa2be00640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffa0c06a460 0x7ffa0c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:11.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.780+0000 7ffa2be00640 1 -- 172.21.15.39:0/875985038 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa1c088c60 con 0x7ffa340d2cb0 2024-01-24T17:42:11.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.781+0000 7ffa33e00640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffa0c06a460 0x7ffa0c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:11.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.782+0000 7ffa33e00640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffa0c06a460 0x7ffa0c06c920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffa24036000 tx=0x7ffa24030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:11.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.785+0000 7ffa2be00640 1 -- 172.21.15.39:0/875985038 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ffa1c054250 con 0x7ffa340d2cb0 2024-01-24T17:42:11.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.972+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa340b7100 con 0x7ffa340d2cb0 2024-01-24T17:42:11.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.973+0000 7ffa2be00640 1 -- 172.21.15.39:0/875985038 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffa340b7100 con 0x7ffa340d2cb0 2024-01-24T17:42:11.974 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:11.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.975+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffa0c06a460 msgr2=0x7ffa0c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.975+0000 7ffa39600640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffa0c06a460 0x7ffa0c06c920 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffa24036000 tx=0x7ffa24030000 comp rx=0 tx=0).stop 2024-01-24T17:42:11.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.975+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa340d2cb0 msgr2=0x7ffa340b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:11.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.975+0000 7ffa39600640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa340d2cb0 0x7ffa340b77b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00d830 tx=0x7ffa1c00dcf0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.975+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 shutdown_connections 2024-01-24T17:42:11.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.976+0000 7ffa39600640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffa3415c8d0 0x7ffa340b7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.976+0000 7ffa39600640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffa0c06a460 0x7ffa0c06c920 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.976+0000 7ffa39600640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa340d2cb0 0x7ffa340b77b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.976+0000 7ffa39600640 1 --2- 172.21.15.39:0/875985038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffa340b2710 0x7ffa341f1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.976+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 >> 172.21.15.39:0/875985038 conn(0x7ffa34071dd0 msgr2=0x7ffa340af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.976+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 shutdown_connections 2024-01-24T17:42:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:11.976+0000 7ffa39600640 1 -- 172.21.15.39:0/875985038 wait complete. 2024-01-24T17:42:11.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:12.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.213+0000 7f0f69800640 1 Processor -- start 2024-01-24T17:42:12.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.213+0000 7f0f69800640 1 -- start start 2024-01-24T17:42:12.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.215+0000 7f0f69800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0f640cb830 0x7f0f640cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.215+0000 7f0f69800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f64071320 con 0x7f0f6406a380 2024-01-24T17:42:12.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.215+0000 7f0f69800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f64071600 con 0x7f0f6415fed0 2024-01-24T17:42:12.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.215+0000 7f0f69800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f640718e0 con 0x7f0f640cb830 2024-01-24T17:42:12.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.215+0000 7f0f63400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0f6406a380 0x7f0f640c9b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55804/0 (socket says 172.21.15.39:55804) 2024-01-24T17:42:12.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.215+0000 7f0f63400640 1 -- 172.21.15.39:0/825522720 learned_addr learned my addr 172.21.15.39:0/825522720 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:12.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f68e00640 1 --2- 172.21.15.39:0/825522720 >> v2:172.21.15.100:3300/0 conn(0x7f0f640cb830 0x7f0f640cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934625955 0 0) 0x7f0f64071320 con 0x7f0f6406a380 2024-01-24T17:42:12.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f44003520 con 0x7f0f6406a380 2024-01-24T17:42:12.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903702319 0 0) 0x7f0f44003520 con 0x7f0f6406a380 2024-01-24T17:42:12.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 >> v2:172.21.15.100:3300/0 conn(0x7f0f640cb830 msgr2=0x7f0f640cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:12.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 --2- 172.21.15.39:0/825522720 >> v2:172.21.15.100:3300/0 conn(0x7f0f640cb830 0x7f0f640cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 >> v1:172.21.15.100:6789/0 conn(0x7f0f6415fed0 legacy=0x7f0f64069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f64071bc0 con 0x7f0f6406a380 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f0f4c001670 con 0x7f0f6406a380 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 >> v1:172.21.15.39:6789/0 conn(0x7f0f6406a380 legacy=0x7f0f640c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44003970 0x7f0f44003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f440047c0 0x7f0f44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44005180 0x7f0f4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640718e0 con 0x7f0f44003970 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4400c0d0 con 0x7f0f440047c0 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.216+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4400c290 con 0x7f0f44005180 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f63e00640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44003970 0x7f0f44003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f63400640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f440047c0 0x7f0f44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f63e00640 1 -- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44005180 msgr2=0x7f0f4400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:12.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f68e00640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44005180 0x7f0f4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f63e00640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44005180 0x7f0f4400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f63e00640 1 -- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f440047c0 msgr2=0x7f0f44004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:12.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f63e00640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f440047c0 0x7f0f44004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f63e00640 1 -- 172.21.15.39:0/825522720 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f44003520 con 0x7f0f44003970 2024-01-24T17:42:12.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f63400640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f440047c0 0x7f0f44004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:12.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.217+0000 7f0f68e00640 1 -- 172.21.15.39:0/825522720 reap_dead start 2024-01-24T17:42:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.218+0000 7f0f63e00640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44003970 0x7f0f44003d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0f54001e50 tx=0x7f0f440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06ab01f16f9881c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.218+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54003b70 con 0x7f0f44003970 2024-01-24T17:42:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.218+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f54003cd0 con 0x7f0f44003970 2024-01-24T17:42:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.219+0000 7f0f62a00640 1 -- 172.21.15.39:0/825522720 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f54005310 con 0x7f0f44003970 2024-01-24T17:42:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.219+0000 7f0f69800640 1 -- 172.21.15.39:0/825522720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44003970 msgr2=0x7f0f44003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.219+0000 7f0f69800640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44003970 0x7f0f44003d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0f54001e50 tx=0x7f0f440057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.219+0000 7f0f69800640 1 -- 172.21.15.39:0/825522720 shutdown_connections 2024-01-24T17:42:12.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.219+0000 7f0f69800640 1 --2- 172.21.15.39:0/825522720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44003970 0x7f0f44003d90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.219+0000 7f0f69800640 1 -- 172.21.15.39:0/825522720 >> 172.21.15.39:0/825522720 conn(0x7f0f640b10a0 msgr2=0x7f0f640b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:12.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.219+0000 7f0f69800640 1 -- 172.21.15.39:0/825522720 shutdown_connections 2024-01-24T17:42:12.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.219+0000 7f0f69800640 1 -- 172.21.15.39:0/825522720 wait complete. 2024-01-24T17:42:12.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.221+0000 7f0f69800640 1 Processor -- start 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.221+0000 7f0f69800640 1 -- start start 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.222+0000 7f0f69800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44003970 0x7f0f641f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.222+0000 7f0f69800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f640c8a10 0x7f0f641f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.222+0000 7f0f69800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f640b2f80 0x7f0f640afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.222+0000 7f0f69800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f640c76d0 con 0x7f0f640b2f80 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.222+0000 7f0f69800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f641f5b00 con 0x7f0f640c8a10 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.222+0000 7f0f69800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f641f5de0 con 0x7f0f44003970 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f640b2f80 0x7f0f640afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f640b2f80 0x7f0f640afd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39374/0 (socket says 172.21.15.39:39374) 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 -- 172.21.15.39:0/344616501 learned_addr learned my addr 172.21.15.39:0/344616501 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f63e00640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44003970 0x7f0f641f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f63400640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f640c8a10 0x7f0f641f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 -- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44003970 msgr2=0x7f0f641f5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44003970 0x7f0f641f5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 -- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f640c8a10 msgr2=0x7f0f641f17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f640c8a10 0x7f0f641f17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 -- 172.21.15.39:0/344616501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f641f60c0 con 0x7f0f640b2f80 2024-01-24T17:42:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.223+0000 7f0f68e00640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f640b2f80 0x7f0f640afd70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f640cf2f0 tx=0x7f0f5c00e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.224+0000 7f0f5be00640 1 -- 172.21.15.39:0/344616501 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c0042f0 con 0x7f0f640b2f80 2024-01-24T17:42:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.224+0000 7f0f5be00640 1 -- 172.21.15.39:0/344616501 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5c004470 con 0x7f0f640b2f80 2024-01-24T17:42:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.224+0000 7f0f5be00640 1 -- 172.21.15.39:0/344616501 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c0038a0 con 0x7f0f640b2f80 2024-01-24T17:42:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.224+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f641f63a0 con 0x7f0f640b2f80 2024-01-24T17:42:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.224+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f640ae0a0 con 0x7f0f640b2f80 2024-01-24T17:42:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.225+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f64067040 con 0x7f0f640b2f80 2024-01-24T17:42:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.226+0000 7f0f5be00640 1 -- 172.21.15.39:0/344616501 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0f5c003a00 con 0x7f0f640b2f80 2024-01-24T17:42:12.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.227+0000 7f0f5be00640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3c06a460 0x7f0f3c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.227+0000 7f0f5be00640 1 -- 172.21.15.39:0/344616501 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f5c0881e0 con 0x7f0f640b2f80 2024-01-24T17:42:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.228+0000 7f0f63e00640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3c06a460 0x7f0f3c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.229+0000 7f0f63e00640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3c06a460 0x7f0f3c06c920 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0f54034000 tx=0x7f0f54009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:12.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.231+0000 7f0f5be00640 1 -- 172.21.15.39:0/344616501 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0f5c053730 con 0x7f0f640b2f80 2024-01-24T17:42:12.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.427+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f640c95f0 con 0x7f0f640b2f80 2024-01-24T17:42:12.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.427+0000 7f0f5be00640 1 -- 172.21.15.39:0/344616501 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f5c003cb0 con 0x7f0f640b2f80 2024-01-24T17:42:12.429 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:12.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.430+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3c06a460 msgr2=0x7f0f3c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.430+0000 7f0f69800640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3c06a460 0x7f0f3c06c920 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0f54034000 tx=0x7f0f54009030 comp rx=0 tx=0).stop 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.430+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f640b2f80 msgr2=0x7f0f640afd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.430+0000 7f0f69800640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f640b2f80 0x7f0f640afd70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f640cf2f0 tx=0x7f0f5c00e8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.431+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 shutdown_connections 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.431+0000 7f0f69800640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3c06a460 0x7f0f3c06c920 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.431+0000 7f0f69800640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f640b2f80 0x7f0f640afd70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.431+0000 7f0f69800640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f640c8a10 0x7f0f641f17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.431+0000 7f0f69800640 1 --2- 172.21.15.39:0/344616501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f44003970 0x7f0f641f5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.431+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 >> 172.21.15.39:0/344616501 conn(0x7f0f640b10a0 msgr2=0x7f0f64069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.431+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 shutdown_connections 2024-01-24T17:42:12.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.431+0000 7f0f69800640 1 -- 172.21.15.39:0/344616501 wait complete. 2024-01-24T17:42:12.453 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:12.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.670+0000 7f5765000640 1 Processor -- start 2024-01-24T17:42:12.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.670+0000 7f5765000640 1 -- start start 2024-01-24T17:42:12.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.671+0000 7f5765000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f57600cf990 0x7f57600cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.671+0000 7f5765000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f576015fb90 0x7f576015d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.671+0000 7f5765000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57600d02b0 con 0x7f576015d020 2024-01-24T17:42:12.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.671+0000 7f5765000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576015e080 con 0x7f57600cf990 2024-01-24T17:42:12.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.671+0000 7f5765000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576015dac0 con 0x7f576015fb90 2024-01-24T17:42:12.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.672+0000 7f575de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f57600cf990 0x7f57600cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.672+0000 7f575de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f57600cf990 0x7f57600cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39380/0 (socket says 172.21.15.39:39380) 2024-01-24T17:42:12.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.672+0000 7f575de00640 1 -- 172.21.15.39:0/3087780313 learned_addr learned my addr 172.21.15.39:0/3087780313 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:12.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.672+0000 7f575de00640 1 -- 172.21.15.39:0/3087780313 >> v2:172.21.15.100:3300/0 conn(0x7f576015fb90 msgr2=0x7f576015d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:12.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.672+0000 7f575de00640 1 --2- 172.21.15.39:0/3087780313 >> v2:172.21.15.100:3300/0 conn(0x7f576015fb90 0x7f576015d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.672+0000 7f575de00640 1 -- 172.21.15.39:0/3087780313 >> v1:172.21.15.100:6789/0 conn(0x7f576015d020 legacy=0x7f5760160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:12.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.672+0000 7f575de00640 1 -- 172.21.15.39:0/3087780313 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576015dda0 con 0x7f57600cf990 2024-01-24T17:42:12.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.673+0000 7f575de00640 1 --2- 172.21.15.39:0/3087780313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57600cf990 0x7f57600cfd70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f576006bea0 tx=0x7f575002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df41af26a5a10bc5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:12.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.674+0000 7f575ca00640 1 -- 172.21.15.39:0/3087780313 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750031870 con 0x7f57600cf990 2024-01-24T17:42:12.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.674+0000 7f575ca00640 1 -- 172.21.15.39:0/3087780313 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57500319d0 con 0x7f57600cf990 2024-01-24T17:42:12.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.674+0000 7f575ca00640 1 -- 172.21.15.39:0/3087780313 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750031d60 con 0x7f57600cf990 2024-01-24T17:42:12.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.674+0000 7f5765000640 1 -- 172.21.15.39:0/3087780313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57600cf990 msgr2=0x7f57600cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.674+0000 7f5765000640 1 --2- 172.21.15.39:0/3087780313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57600cf990 0x7f57600cfd70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f576006bea0 tx=0x7f575002b910 comp rx=0 tx=0).stop 2024-01-24T17:42:12.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.675+0000 7f5765000640 1 -- 172.21.15.39:0/3087780313 shutdown_connections 2024-01-24T17:42:12.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.675+0000 7f5765000640 1 --2- 172.21.15.39:0/3087780313 >> v2:172.21.15.100:3300/0 conn(0x7f576015fb90 0x7f576015d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.675+0000 7f5765000640 1 --2- 172.21.15.39:0/3087780313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57600cf990 0x7f57600cfd70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.675+0000 7f5765000640 1 -- 172.21.15.39:0/3087780313 >> 172.21.15.39:0/3087780313 conn(0x7f5760071790 msgr2=0x7f5760071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:12.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.675+0000 7f5765000640 1 -- 172.21.15.39:0/3087780313 shutdown_connections 2024-01-24T17:42:12.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.675+0000 7f5765000640 1 -- 172.21.15.39:0/3087780313 wait complete. 2024-01-24T17:42:12.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.677+0000 7f5765000640 1 Processor -- start 2024-01-24T17:42:12.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.677+0000 7f5765000640 1 -- start start 2024-01-24T17:42:12.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f5765000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57600cf990 0x7f5760158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f5765000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f576015d020 0x7f57600c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f5765000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f576015fb90 0x7f57600c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f5765000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57601f3420 con 0x7f576015fb90 2024-01-24T17:42:12.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f5765000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57600cc8c0 con 0x7f576015d020 2024-01-24T17:42:12.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f5765000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57601f3140 con 0x7f57600cf990 2024-01-24T17:42:12.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f575e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f576015fb90 0x7f57600c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f575e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f576015fb90 0x7f57600c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39386/0 (socket says 172.21.15.39:39386) 2024-01-24T17:42:12.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f575e800640 1 -- 172.21.15.39:0/2061985897 learned_addr learned my addr 172.21.15.39:0/2061985897 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:12.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f575de00640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57600cf990 0x7f5760158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f575d400640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f576015d020 0x7f57600c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f575e800640 1 -- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57600cf990 msgr2=0x7f5760158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.678+0000 7f575e800640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57600cf990 0x7f5760158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f575e800640 1 -- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f576015d020 msgr2=0x7f57600c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f575e800640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f576015d020 0x7f57600c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f575e800640 1 -- 172.21.15.39:0/2061985897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57500094d0 con 0x7f576015fb90 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f575d400640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f576015d020 0x7f57600c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f575e800640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f576015fb90 0x7f57600c8f80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f575800b970 tx=0x7f575800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f5756000640 1 -- 172.21.15.39:0/2061985897 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575800cb40 con 0x7f576015fb90 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f5756000640 1 -- 172.21.15.39:0/2061985897 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575800ccc0 con 0x7f576015fb90 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f5756000640 1 -- 172.21.15.39:0/2061985897 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758014680 con 0x7f576015fb90 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57600c6660 con 0x7f576015fb90 2024-01-24T17:42:12.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.679+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57600c6c20 con 0x7f576015fb90 2024-01-24T17:42:12.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.681+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57600c6e40 con 0x7f576015fb90 2024-01-24T17:42:12.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.681+0000 7f5756000640 1 -- 172.21.15.39:0/2061985897 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f575801d3b0 con 0x7f576015fb90 2024-01-24T17:42:12.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.682+0000 7f5756000640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f573806a430 0x7f573806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:12.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.682+0000 7f5756000640 1 -- 172.21.15.39:0/2061985897 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5758088840 con 0x7f576015fb90 2024-01-24T17:42:12.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.683+0000 7f575de00640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f573806a430 0x7f573806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:12.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.684+0000 7f575de00640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f573806a430 0x7f573806c8f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5750036000 tx=0x7f575003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:12.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.687+0000 7f5756000640 1 -- 172.21.15.39:0/2061985897 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5758053e20 con 0x7f576015fb90 2024-01-24T17:42:12.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.882+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5760073ef0 con 0x7f576015fb90 2024-01-24T17:42:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.884+0000 7f5756000640 1 -- 172.21.15.39:0/2061985897 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f5758019090 con 0x7f576015fb90 2024-01-24T17:42:12.886 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:12.886 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f573806a430 msgr2=0x7f573806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f573806a430 0x7f573806c8f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5750036000 tx=0x7f575003c040 comp rx=0 tx=0).stop 2024-01-24T17:42:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f576015fb90 msgr2=0x7f57600c8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f576015fb90 0x7f57600c8f80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f575800b970 tx=0x7f575800be30 comp rx=0 tx=0).stop 2024-01-24T17:42:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 shutdown_connections 2024-01-24T17:42:12.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f573806a430 0x7f573806c8f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f576015fb90 0x7f57600c8f80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f576015d020 0x7f57600c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 --2- 172.21.15.39:0/2061985897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f57600cf990 0x7f5760158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.887+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 >> 172.21.15.39:0/2061985897 conn(0x7f5760071790 msgr2=0x7f576015c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.888+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 shutdown_connections 2024-01-24T17:42:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:12.888+0000 7f5765000640 1 -- 172.21.15.39:0/2061985897 wait complete. 2024-01-24T17:42:12.911 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:12.912 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:13.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.135+0000 7f8ab5200640 1 Processor -- start 2024-01-24T17:42:13.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.136+0000 7f8ab5200640 1 -- start start 2024-01-24T17:42:13.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.137+0000 7f8ab5200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8ab00cb540 0x7f8ab006d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.137+0000 7f8ab5200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab00ad0b0 con 0x7f8ab006c3a0 2024-01-24T17:42:13.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.137+0000 7f8ab5200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab00b2370 con 0x7f8ab00accd0 2024-01-24T17:42:13.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.137+0000 7f8ab5200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab006d840 con 0x7f8ab00cb540 2024-01-24T17:42:13.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.137+0000 7f8aafe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8ab006c3a0 0x7f8ab00ab030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59130/0 (socket says 172.21.15.39:59130) 2024-01-24T17:42:13.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.137+0000 7f8aafe00640 1 -- 172.21.15.39:0/4119866106 learned_addr learned my addr 172.21.15.39:0/4119866106 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:13.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.138+0000 7f8aaea00640 1 --2- 172.21.15.39:0/4119866106 >> v2:172.21.15.100:3300/0 conn(0x7f8ab00cb540 0x7f8ab006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.138+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890348370 0 0) 0x7f8ab00ad0b0 con 0x7f8ab006c3a0 2024-01-24T17:42:13.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.138+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a90003520 con 0x7f8ab006c3a0 2024-01-24T17:42:13.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.138+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 12481261 0 0) 0x7f8a90003520 con 0x7f8ab006c3a0 2024-01-24T17:42:13.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.138+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 >> v2:172.21.15.100:3300/0 conn(0x7f8ab00cb540 msgr2=0x7f8ab006d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:13.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.138+0000 7f8aae000640 1 --2- 172.21.15.39:0/4119866106 >> v2:172.21.15.100:3300/0 conn(0x7f8ab00cb540 0x7f8ab006d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.138+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 >> v1:172.21.15.100:6789/0 conn(0x7f8ab00accd0 legacy=0x7f8ab00cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab00b2650 con 0x7f8ab006c3a0 2024-01-24T17:42:13.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8aa80019f0 con 0x7f8ab006c3a0 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 >> v1:172.21.15.39:6789/0 conn(0x7f8ab006c3a0 legacy=0x7f8ab00ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a90003970 0x7f8a90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a900047c0 0x7f8a90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a90005180 0x7f8a9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab006d840 con 0x7f8a90003970 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9000c020 con 0x7f8a900047c0 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9000c1e0 con 0x7f8a90005180 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aaf400640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a90003970 0x7f8a90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.139+0000 7f8aaea00640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a900047c0 0x7f8a90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.140+0000 7f8aafe00640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a90005180 0x7f8a9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.140+0000 7f8aaf400640 1 -- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a90005180 msgr2=0x7f8a9000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.140+0000 7f8aaf400640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a90005180 0x7f8a9000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.140+0000 7f8aaf400640 1 -- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a900047c0 msgr2=0x7f8a90004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:13.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.140+0000 7f8aaf400640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a900047c0 0x7f8a90004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.140+0000 7f8aaf400640 1 -- 172.21.15.39:0/4119866106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a90003520 con 0x7f8a90003970 2024-01-24T17:42:13.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.140+0000 7f8aafe00640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8a90005180 0x7f8a9000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:13.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.140+0000 7f8aafe00640 1 -- 172.21.15.39:0/4119866106 reap_dead start 2024-01-24T17:42:13.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.141+0000 7f8aaf400640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a90003970 0x7f8a90003d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0001770 tx=0x7f8a900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d1f84fb8c31b78 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:13.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.141+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa00052d0 con 0x7f8a90003970 2024-01-24T17:42:13.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.141+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa002e050 con 0x7f8a90003970 2024-01-24T17:42:13.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.141+0000 7f8aae000640 1 -- 172.21.15.39:0/4119866106 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0003640 con 0x7f8a90003970 2024-01-24T17:42:13.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.142+0000 7f8ab5200640 1 -- 172.21.15.39:0/4119866106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a90003970 msgr2=0x7f8a90003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.142+0000 7f8ab5200640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a90003970 0x7f8a90003d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0001770 tx=0x7f8a900057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.142+0000 7f8ab5200640 1 -- 172.21.15.39:0/4119866106 shutdown_connections 2024-01-24T17:42:13.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.142+0000 7f8ab5200640 1 --2- 172.21.15.39:0/4119866106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a90003970 0x7f8a90003d90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.142+0000 7f8ab5200640 1 -- 172.21.15.39:0/4119866106 >> 172.21.15.39:0/4119866106 conn(0x7f8ab0069e50 msgr2=0x7f8ab006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:13.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.142+0000 7f8ab5200640 1 -- 172.21.15.39:0/4119866106 shutdown_connections 2024-01-24T17:42:13.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.142+0000 7f8ab5200640 1 -- 172.21.15.39:0/4119866106 wait complete. 2024-01-24T17:42:13.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.144+0000 7f8ab5200640 1 Processor -- start 2024-01-24T17:42:13.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.144+0000 7f8ab5200640 1 -- start start 2024-01-24T17:42:13.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8ab5200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a90003970 0x7f8ab0157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8ab5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ab0158480 0x7f8ab015ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8ab5200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ab015b320 0x7f8ab015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8ab5200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab015d590 con 0x7f8ab0158480 2024-01-24T17:42:13.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8ab5200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab00ade80 con 0x7f8a90003970 2024-01-24T17:42:13.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8ab5200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab00ae160 con 0x7f8ab015b320 2024-01-24T17:42:13.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ab0158480 0x7f8ab015ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ab0158480 0x7f8ab015ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58462/0 (socket says 172.21.15.39:58462) 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaea00640 1 -- 172.21.15.39:0/2914700203 learned_addr learned my addr 172.21.15.39:0/2914700203 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaf400640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a90003970 0x7f8ab0157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaea00640 1 -- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ab015b320 msgr2=0x7f8ab015b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaea00640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ab015b320 0x7f8ab015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaea00640 1 -- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a90003970 msgr2=0x7f8ab0157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaea00640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a90003970 0x7f8ab0157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaea00640 1 -- 172.21.15.39:0/2914700203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab00ae440 con 0x7f8ab0158480 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.145+0000 7f8aaf400640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a90003970 0x7f8ab0157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.146+0000 7f8aaea00640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ab0158480 0x7f8ab015ade0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8a980116d0 tx=0x7f8a98011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.146+0000 7f8aa7400640 1 -- 172.21.15.39:0/2914700203 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9800f9c0 con 0x7f8ab0158480 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.146+0000 7f8aa7400640 1 -- 172.21.15.39:0/2914700203 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9800fb20 con 0x7f8ab0158480 2024-01-24T17:42:13.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.146+0000 7f8aa7400640 1 -- 172.21.15.39:0/2914700203 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9800fe90 con 0x7f8ab0158480 2024-01-24T17:42:13.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.146+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab00ae720 con 0x7f8ab0158480 2024-01-24T17:42:13.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.146+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab0158900 con 0x7f8ab0158480 2024-01-24T17:42:13.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.147+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab004dff0 con 0x7f8ab0158480 2024-01-24T17:42:13.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.148+0000 7f8aa7400640 1 -- 172.21.15.39:0/2914700203 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8a98015460 con 0x7f8ab0158480 2024-01-24T17:42:13.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.149+0000 7f8aa7400640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8a8806a480 0x7f8a8806c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.149+0000 7f8aa7400640 1 -- 172.21.15.39:0/2914700203 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a98089190 con 0x7f8ab0158480 2024-01-24T17:42:13.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.150+0000 7f8aaf400640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8a8806a480 0x7f8a8806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.151+0000 7f8aaf400640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8a8806a480 0x7f8a8806c940 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0033000 tx=0x7f8aa0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:13.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.153+0000 7f8aa7400640 1 -- 172.21.15.39:0/2914700203 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8a98054760 con 0x7f8ab0158480 2024-01-24T17:42:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.348+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ab0070d70 con 0x7f8ab0158480 2024-01-24T17:42:13.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.349+0000 7f8aa7400640 1 -- 172.21.15.39:0/2914700203 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a98057d80 con 0x7f8ab0158480 2024-01-24T17:42:13.350 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:13.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8a8806a480 msgr2=0x7f8a8806c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8a8806a480 0x7f8a8806c940 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0033000 tx=0x7f8aa0009030 comp rx=0 tx=0).stop 2024-01-24T17:42:13.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ab0158480 msgr2=0x7f8ab015ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ab0158480 0x7f8ab015ade0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8a980116d0 tx=0x7f8a98011b90 comp rx=0 tx=0).stop 2024-01-24T17:42:13.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 shutdown_connections 2024-01-24T17:42:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ab015b320 0x7f8ab015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8a8806a480 0x7f8a8806c940 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ab0158480 0x7f8ab015ade0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 --2- 172.21.15.39:0/2914700203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8a90003970 0x7f8ab0157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.351+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 >> 172.21.15.39:0/2914700203 conn(0x7f8ab0069e50 msgr2=0x7f8ab015a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.352+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 shutdown_connections 2024-01-24T17:42:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.352+0000 7f8ab5200640 1 -- 172.21.15.39:0/2914700203 wait complete. 2024-01-24T17:42:13.371 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:13.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.580+0000 7f1e53e00640 1 Processor -- start 2024-01-24T17:42:13.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.580+0000 7f1e53e00640 1 -- start start 2024-01-24T17:42:13.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.581+0000 7f1e53e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e5415c8d0 0x7f1e540d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.581+0000 7f1e53e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1e540d2e80 0x7f1e540cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.581+0000 7f1e53e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e540b38a0 con 0x7f1e540b2710 2024-01-24T17:42:13.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.581+0000 7f1e53e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e540d13e0 con 0x7f1e540d2e80 2024-01-24T17:42:13.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.581+0000 7f1e53e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e540d0ab0 con 0x7f1e5415c8d0 2024-01-24T17:42:13.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.582+0000 7f1e52a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e5415c8d0 0x7f1e540d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.582+0000 7f1e53400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1e540d2e80 0x7f1e540cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.582+0000 7f1e4be00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1e540b2710 0x7f1e540d2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:37406/0 (socket says 172.21.15.39:37406) 2024-01-24T17:42:13.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.582+0000 7f1e4be00640 1 -- 172.21.15.39:0/2194423710 learned_addr learned my addr 172.21.15.39:0/2194423710 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:13.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.583+0000 7f1e52000640 1 -- 172.21.15.39:0/2194423710 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127203934 0 0) 0x7f1e540b38a0 con 0x7f1e540b2710 2024-01-24T17:42:13.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.583+0000 7f1e52000640 1 -- 172.21.15.39:0/2194423710 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e34003520 con 0x7f1e540b2710 2024-01-24T17:42:13.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.583+0000 7f1e52a00640 1 -- 172.21.15.39:0/2194423710 >> v2:172.21.15.100:3300/0 conn(0x7f1e540d2e80 msgr2=0x7f1e540cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:13.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.583+0000 7f1e52a00640 1 --2- 172.21.15.39:0/2194423710 >> v2:172.21.15.100:3300/0 conn(0x7f1e540d2e80 0x7f1e540cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.583+0000 7f1e52a00640 1 -- 172.21.15.39:0/2194423710 >> v1:172.21.15.100:6789/0 conn(0x7f1e540b2710 legacy=0x7f1e540d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.583+0000 7f1e52a00640 1 -- 172.21.15.39:0/2194423710 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e540d0d90 con 0x7f1e5415c8d0 2024-01-24T17:42:13.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.584+0000 7f1e52a00640 1 --2- 172.21.15.39:0/2194423710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e5415c8d0 0x7f1e540d03b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1e440097a0 tx=0x7f1e4402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df0498ac5177dc9d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:13.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.585+0000 7f1e52000640 1 -- 172.21.15.39:0/2194423710 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4402d040 con 0x7f1e5415c8d0 2024-01-24T17:42:13.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.585+0000 7f1e52000640 1 -- 172.21.15.39:0/2194423710 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e44031810 con 0x7f1e5415c8d0 2024-01-24T17:42:13.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.585+0000 7f1e52000640 1 -- 172.21.15.39:0/2194423710 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e44031ba0 con 0x7f1e5415c8d0 2024-01-24T17:42:13.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.586+0000 7f1e53e00640 1 -- 172.21.15.39:0/2194423710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e5415c8d0 msgr2=0x7f1e540d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.586+0000 7f1e53e00640 1 --2- 172.21.15.39:0/2194423710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e5415c8d0 0x7f1e540d03b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1e440097a0 tx=0x7f1e4402b8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.586+0000 7f1e53e00640 1 -- 172.21.15.39:0/2194423710 shutdown_connections 2024-01-24T17:42:13.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.586+0000 7f1e53e00640 1 --2- 172.21.15.39:0/2194423710 >> v2:172.21.15.100:3300/0 conn(0x7f1e540d2e80 0x7f1e540cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.586+0000 7f1e53e00640 1 --2- 172.21.15.39:0/2194423710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e5415c8d0 0x7f1e540d03b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.586+0000 7f1e53e00640 1 -- 172.21.15.39:0/2194423710 >> 172.21.15.39:0/2194423710 conn(0x7f1e54071dd0 msgr2=0x7f1e5409bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:13.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.586+0000 7f1e53e00640 1 -- 172.21.15.39:0/2194423710 shutdown_connections 2024-01-24T17:42:13.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.586+0000 7f1e53e00640 1 -- 172.21.15.39:0/2194423710 wait complete. 2024-01-24T17:42:13.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.588+0000 7f1e53e00640 1 Processor -- start 2024-01-24T17:42:13.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.588+0000 7f1e53e00640 1 -- start start 2024-01-24T17:42:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e53e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e540b2710 0x7f1e540cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e53e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e540d2e80 0x7f1e540cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e53e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e5415c8d0 0x7f1e541f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e53e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5415ea50 con 0x7f1e540d2e80 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e53e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e540c9c90 con 0x7f1e5415c8d0 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e53e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e540c9f70 con 0x7f1e540b2710 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e540d2e80 0x7f1e540cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e540d2e80 0x7f1e540cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58464/0 (socket says 172.21.15.39:58464) 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e4be00640 1 -- 172.21.15.39:0/1966578594 learned_addr learned my addr 172.21.15.39:0/1966578594 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e53400640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e5415c8d0 0x7f1e541f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e52a00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e540b2710 0x7f1e540cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e4be00640 1 -- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e540b2710 msgr2=0x7f1e540cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e4be00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e540b2710 0x7f1e540cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e4be00640 1 -- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e5415c8d0 msgr2=0x7f1e541f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.589+0000 7f1e4be00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e5415c8d0 0x7f1e541f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.590+0000 7f1e4be00640 1 -- 172.21.15.39:0/1966578594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e44009400 con 0x7f1e540d2e80 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.590+0000 7f1e4be00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e540d2e80 0x7f1e540cbec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00c430 tx=0x7f1e3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.590+0000 7f1e4aa00640 1 -- 172.21.15.39:0/1966578594 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c005ce0 con 0x7f1e540d2e80 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.590+0000 7f1e4aa00640 1 -- 172.21.15.39:0/1966578594 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c005e40 con 0x7f1e540d2e80 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.590+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e540ca250 con 0x7f1e540d2e80 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.590+0000 7f1e4aa00640 1 -- 172.21.15.39:0/1966578594 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0072f0 con 0x7f1e540d2e80 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.590+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e540ca810 con 0x7f1e540d2e80 2024-01-24T17:42:13.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.591+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5404dff0 con 0x7f1e540d2e80 2024-01-24T17:42:13.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.592+0000 7f1e4aa00640 1 -- 172.21.15.39:0/1966578594 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1e3c007450 con 0x7f1e540d2e80 2024-01-24T17:42:13.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.593+0000 7f1e4aa00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e2c06a430 0x7f1e2c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:13.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.593+0000 7f1e4aa00640 1 -- 172.21.15.39:0/1966578594 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e3c088090 con 0x7f1e540d2e80 2024-01-24T17:42:13.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.594+0000 7f1e52a00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e2c06a430 0x7f1e2c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:13.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.595+0000 7f1e52a00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e2c06a430 0x7f1e2c06c8f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e44036000 tx=0x7f1e4403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:13.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.599+0000 7f1e4aa00640 1 -- 172.21.15.39:0/1966578594 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1e3c0535e0 con 0x7f1e540d2e80 2024-01-24T17:42:13.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.792+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e540d0670 con 0x7f1e540d2e80 2024-01-24T17:42:13.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.793+0000 7f1e4aa00640 1 -- 172.21.15.39:0/1966578594 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e3c056c00 con 0x7f1e540d2e80 2024-01-24T17:42:13.794 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e2c06a430 msgr2=0x7f1e2c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e2c06a430 0x7f1e2c06c8f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e44036000 tx=0x7f1e4403c040 comp rx=0 tx=0).stop 2024-01-24T17:42:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e540d2e80 msgr2=0x7f1e540cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e540d2e80 0x7f1e540cbec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c00c430 tx=0x7f1e3c00c8f0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 shutdown_connections 2024-01-24T17:42:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e5415c8d0 0x7f1e541f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e2c06a430 0x7f1e2c06c8f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e540d2e80 0x7f1e540cbec0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 --2- 172.21.15.39:0/1966578594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e540b2710 0x7f1e540cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.795+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 >> 172.21.15.39:0/1966578594 conn(0x7f1e54071dd0 msgr2=0x7f1e540b4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.796+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 shutdown_connections 2024-01-24T17:42:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:13.796+0000 7f1e53e00640 1 -- 172.21.15.39:0/1966578594 wait complete. 2024-01-24T17:42:13.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:14.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.044+0000 7f46fbe00640 1 Processor -- start 2024-01-24T17:42:14.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.045+0000 7f46fbe00640 1 -- start start 2024-01-24T17:42:14.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.046+0000 7f46fbe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f46fc0cb540 0x7f46fc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.046+0000 7f46fbe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc0ad0b0 con 0x7f46fc0accd0 2024-01-24T17:42:14.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.046+0000 7f46fbe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc0b2370 con 0x7f46fc06c3a0 2024-01-24T17:42:14.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.046+0000 7f46fbe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc06d840 con 0x7f46fc0cb540 2024-01-24T17:42:14.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.046+0000 7f46faa00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f46fc0accd0 0x7f46fc0cae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59140/0 (socket says 172.21.15.39:59140) 2024-01-24T17:42:14.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.046+0000 7f46faa00640 1 -- 172.21.15.39:0/4271154254 learned_addr learned my addr 172.21.15.39:0/4271154254 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:14.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031718600 0 0) 0x7f46fc0ad0b0 con 0x7f46fc0accd0 2024-01-24T17:42:14.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46fa000640 1 --2- 172.21.15.39:0/4271154254 >> v2:172.21.15.100:3300/0 conn(0x7f46fc0cb540 0x7f46fc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc003520 con 0x7f46fc0accd0 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3447241068 0 0) 0x7f46dc003520 con 0x7f46fc0accd0 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 >> v2:172.21.15.100:3300/0 conn(0x7f46fc0cb540 msgr2=0x7f46fc06d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 --2- 172.21.15.39:0/4271154254 >> v2:172.21.15.100:3300/0 conn(0x7f46fc0cb540 0x7f46fc06d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 >> v1:172.21.15.100:6789/0 conn(0x7f46fc06c3a0 legacy=0x7f46fc0ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0b2650 con 0x7f46fc0accd0 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f46ec0017f0 con 0x7f46fc0accd0 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 >> v1:172.21.15.39:6789/0 conn(0x7f46fc0accd0 legacy=0x7f46fc0cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.047+0000 7f46f9600640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc003970 0x7f46dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46f9600640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0047c0 0x7f46dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46f9600640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46dc005180 0x7f46dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc06d840 con 0x7f46dc0047c0 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc00c020 con 0x7f46dc005180 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc00c1e0 con 0x7f46dc003970 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46fa000640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0047c0 0x7f46dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46fb400640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46dc005180 0x7f46dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46faa00640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc003970 0x7f46dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46fa000640 1 -- 172.21.15.39:0/4271154254 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc003970 msgr2=0x7f46dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46fa000640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc003970 0x7f46dc003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46fa000640 1 -- 172.21.15.39:0/4271154254 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46dc005180 msgr2=0x7f46dc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46fa000640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46dc005180 0x7f46dc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46fa000640 1 -- 172.21.15.39:0/4271154254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc003520 con 0x7f46dc0047c0 2024-01-24T17:42:14.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.048+0000 7f46fb400640 1 -- 172.21.15.39:0/4271154254 reap_dead start 2024-01-24T17:42:14.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.049+0000 7f46fa000640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0047c0 0x7f46dc004c40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009ac0 tx=0x7f46dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1295285832c30626 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e402c6b0 con 0x7f46dc0047c0 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e402c810 con 0x7f46dc0047c0 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46f9600640 1 -- 172.21.15.39:0/4271154254 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e402cba0 con 0x7f46dc0047c0 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46fbe00640 1 -- 172.21.15.39:0/4271154254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0047c0 msgr2=0x7f46dc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46fbe00640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0047c0 0x7f46dc004c40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f46e4009ac0 tx=0x7f46dc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46fbe00640 1 -- 172.21.15.39:0/4271154254 shutdown_connections 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46fbe00640 1 --2- 172.21.15.39:0/4271154254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46dc0047c0 0x7f46dc004c40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46fbe00640 1 -- 172.21.15.39:0/4271154254 >> 172.21.15.39:0/4271154254 conn(0x7f46fc069e50 msgr2=0x7f46fc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:14.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46fbe00640 1 -- 172.21.15.39:0/4271154254 shutdown_connections 2024-01-24T17:42:14.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.050+0000 7f46fbe00640 1 -- 172.21.15.39:0/4271154254 wait complete. 2024-01-24T17:42:14.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.052+0000 7f46fbe00640 1 Processor -- start 2024-01-24T17:42:14.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.052+0000 7f46fbe00640 1 -- start start 2024-01-24T17:42:14.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.053+0000 7f46fbe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc0047c0 0x7f46fc1ed660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.053+0000 7f46fbe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46fc1562d0 0x7f46fc158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.053+0000 7f46fbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46fc06a2c0 0x7f46fc159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.053+0000 7f46fbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc15ae80 con 0x7f46fc06a2c0 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.053+0000 7f46fbe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0afcb0 con 0x7f46fc1562d0 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.053+0000 7f46fbe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0b0990 con 0x7f46dc0047c0 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46fc06a2c0 0x7f46fc159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46fc06a2c0 0x7f46fc159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58470/0 (socket says 172.21.15.39:58470) 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 -- 172.21.15.39:0/4089403623 learned_addr learned my addr 172.21.15.39:0/4089403623 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46faa00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc0047c0 0x7f46fc1ed660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fa000640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46fc1562d0 0x7f46fc158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 -- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc0047c0 msgr2=0x7f46fc1ed660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc0047c0 0x7f46fc1ed660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 -- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46fc1562d0 msgr2=0x7f46fc158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46fc1562d0 0x7f46fc158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 -- 172.21.15.39:0/4089403623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc15b0f0 con 0x7f46fc06a2c0 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.054+0000 7f46fb400640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46fc06a2c0 0x7f46fc159210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f46f4007fd0 tx=0x7f46f400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.055+0000 7f46f2a00640 1 -- 172.21.15.39:0/4089403623 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4006d60 con 0x7f46fc06a2c0 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.055+0000 7f46f2a00640 1 -- 172.21.15.39:0/4089403623 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f4003970 con 0x7f46fc06a2c0 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.055+0000 7f46f2a00640 1 -- 172.21.15.39:0/4089403623 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4003d20 con 0x7f46fc06a2c0 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.055+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc0b0c70 con 0x7f46fc06a2c0 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.055+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc1edd80 con 0x7f46fc06a2c0 2024-01-24T17:42:14.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.056+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc04dff0 con 0x7f46fc06a2c0 2024-01-24T17:42:14.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.057+0000 7f46f2a00640 1 -- 172.21.15.39:0/4089403623 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f46f4005070 con 0x7f46fc06a2c0 2024-01-24T17:42:14.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.058+0000 7f46f2a00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46d406a460 0x7f46d406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.058+0000 7f46f2a00640 1 -- 172.21.15.39:0/4089403623 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46f4087d80 con 0x7f46fc06a2c0 2024-01-24T17:42:14.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.058+0000 7f46faa00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46d406a460 0x7f46d406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.059+0000 7f46faa00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46d406a460 0x7f46d406c920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f46ec001920 tx=0x7f46ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.062+0000 7f46f2a00640 1 -- 172.21.15.39:0/4089403623 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f46f40532d0 con 0x7f46fc06a2c0 2024-01-24T17:42:14.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.258+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46fc1568c0 con 0x7f46fc06a2c0 2024-01-24T17:42:14.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.259+0000 7f46f2a00640 1 -- 172.21.15.39:0/4089403623 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f46f40190a0 con 0x7f46fc06a2c0 2024-01-24T17:42:14.261 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:14.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.261+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46d406a460 msgr2=0x7f46d406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.261+0000 7f46fbe00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46d406a460 0x7f46d406c920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f46ec001920 tx=0x7f46ec009040 comp rx=0 tx=0).stop 2024-01-24T17:42:14.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.261+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46fc06a2c0 msgr2=0x7f46fc159210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.261+0000 7f46fbe00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46fc06a2c0 0x7f46fc159210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f46f4007fd0 tx=0x7f46f400e960 comp rx=0 tx=0).stop 2024-01-24T17:42:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.262+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 shutdown_connections 2024-01-24T17:42:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.262+0000 7f46fbe00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f46d406a460 0x7f46d406c920 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.262+0000 7f46fbe00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46fc06a2c0 0x7f46fc159210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.262+0000 7f46fbe00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f46fc1562d0 0x7f46fc158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.262+0000 7f46fbe00640 1 --2- 172.21.15.39:0/4089403623 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f46dc0047c0 0x7f46fc1ed660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.262+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 >> 172.21.15.39:0/4089403623 conn(0x7f46fc069e50 msgr2=0x7f46fc0ab030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.262+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 shutdown_connections 2024-01-24T17:42:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.262+0000 7f46fbe00640 1 -- 172.21.15.39:0/4089403623 wait complete. 2024-01-24T17:42:14.282 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:14.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.493+0000 7ff0f7e00640 1 Processor -- start 2024-01-24T17:42:14.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.493+0000 7ff0f7e00640 1 -- start start 2024-01-24T17:42:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.494+0000 7ff0f7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff0f815a720 0x7ff0f80cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.494+0000 7ff0f7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff0f80ce8e0 0x7ff0f80ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.494+0000 7ff0f7e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f80cf260 con 0x7ff0f815ab00 2024-01-24T17:42:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.494+0000 7ff0f7e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f80ce400 con 0x7ff0f815a720 2024-01-24T17:42:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.494+0000 7ff0f7e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f80ccef0 con 0x7ff0f80ce8e0 2024-01-24T17:42:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f6a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff0f815a720 0x7ff0f80cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f6a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff0f815a720 0x7ff0f80cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58484/0 (socket says 172.21.15.39:58484) 2024-01-24T17:42:14.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f6a00640 1 -- 172.21.15.39:0/1391755755 learned_addr learned my addr 172.21.15.39:0/1391755755 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f7400640 1 --2- 172.21.15.39:0/1391755755 >> v2:172.21.15.100:3300/0 conn(0x7ff0f80ce8e0 0x7ff0f80ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f6a00640 1 -- 172.21.15.39:0/1391755755 >> v2:172.21.15.100:3300/0 conn(0x7ff0f80ce8e0 msgr2=0x7ff0f80ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/1391755755 >> v2:172.21.15.100:3300/0 conn(0x7ff0f80ce8e0 0x7ff0f80ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f6a00640 1 -- 172.21.15.39:0/1391755755 >> v1:172.21.15.100:6789/0 conn(0x7ff0f815ab00 legacy=0x7ff0f80cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f6a00640 1 -- 172.21.15.39:0/1391755755 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f80ce110 con 0x7ff0f815a720 2024-01-24T17:42:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.495+0000 7ff0f7400640 1 --2- 172.21.15.39:0/1391755755 >> v2:172.21.15.100:3300/0 conn(0x7ff0f80ce8e0 0x7ff0f80ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.497+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/1391755755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80cdbd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072270 tx=0x7ff0e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f99d658db28e963 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.497+0000 7ff0f6000640 1 -- 172.21.15.39:0/1391755755 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80317e0 con 0x7ff0f815a720 2024-01-24T17:42:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.498+0000 7ff0f6000640 1 -- 172.21.15.39:0/1391755755 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e8031940 con 0x7ff0f815a720 2024-01-24T17:42:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.498+0000 7ff0f6000640 1 -- 172.21.15.39:0/1391755755 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8031cd0 con 0x7ff0f815a720 2024-01-24T17:42:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.498+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1391755755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 msgr2=0x7ff0f80cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.498+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1391755755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80cdbd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072270 tx=0x7ff0e802b880 comp rx=0 tx=0).stop 2024-01-24T17:42:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.498+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1391755755 shutdown_connections 2024-01-24T17:42:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.498+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1391755755 >> v2:172.21.15.100:3300/0 conn(0x7ff0f80ce8e0 0x7ff0f80ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.498+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1391755755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80cdbd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.498+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1391755755 >> 172.21.15.39:0/1391755755 conn(0x7ff0f806a090 msgr2=0x7ff0f806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:14.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.499+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1391755755 shutdown_connections 2024-01-24T17:42:14.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.499+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1391755755 wait complete. 2024-01-24T17:42:14.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.500+0000 7ff0f7e00640 1 Processor -- start 2024-01-24T17:42:14.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.500+0000 7ff0f7e00640 1 -- start start 2024-01-24T17:42:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.501+0000 7ff0f7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0f80ce8e0 0x7ff0f80cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.501+0000 7ff0f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80c4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.501+0000 7ff0f7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0f815ab00 0x7ff0f80c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.501+0000 7ff0f7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f80ca320 con 0x7ff0f815a720 2024-01-24T17:42:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.501+0000 7ff0f7e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f81521b0 con 0x7ff0f815ab00 2024-01-24T17:42:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.501+0000 7ff0f7e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f80c4b40 con 0x7ff0f80ce8e0 2024-01-24T17:42:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80c4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58486/0 (socket says 172.21.15.39:58486) 2024-01-24T17:42:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 -- 172.21.15.39:0/1971307111 learned_addr learned my addr 172.21.15.39:0/1971307111 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 -- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0f80ce8e0 msgr2=0x7ff0f80cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0f80ce8e0 0x7ff0f80cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 -- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0f815ab00 msgr2=0x7ff0f80c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0f815ab00 0x7ff0f80c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 -- 172.21.15.39:0/1971307111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80094d0 con 0x7ff0f815a720 2024-01-24T17:42:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.502+0000 7ff0ede00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80c4600 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000b3f0 tx=0x7ff0e000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.503+0000 7ff0efe00640 1 -- 172.21.15.39:0/1971307111 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0011830 con 0x7ff0f815a720 2024-01-24T17:42:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.503+0000 7ff0efe00640 1 -- 172.21.15.39:0/1971307111 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e0011990 con 0x7ff0f815a720 2024-01-24T17:42:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.503+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f8153680 con 0x7ff0f815a720 2024-01-24T17:42:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.503+0000 7ff0efe00640 1 -- 172.21.15.39:0/1971307111 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0011d40 con 0x7ff0f815a720 2024-01-24T17:42:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.503+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f8158ef0 con 0x7ff0f815a720 2024-01-24T17:42:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.504+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f80c9de0 con 0x7ff0f815a720 2024-01-24T17:42:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.505+0000 7ff0efe00640 1 -- 172.21.15.39:0/1971307111 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff0e001a400 con 0x7ff0f815a720 2024-01-24T17:42:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.506+0000 7ff0efe00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0d006a430 0x7ff0d006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.506+0000 7ff0efe00640 1 -- 172.21.15.39:0/1971307111 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0e0088840 con 0x7ff0f815a720 2024-01-24T17:42:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.507+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0d006a430 0x7ff0d006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.508+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0d006a430 0x7ff0d006c8f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8036000 tx=0x7ff0e8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.511+0000 7ff0efe00640 1 -- 172.21.15.39:0/1971307111 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff0f80c9de0 con 0x7ff0f815a720 2024-01-24T17:42:14.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.711+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0f80c9de0 con 0x7ff0f815a720 2024-01-24T17:42:14.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.714+0000 7ff0efe00640 1 -- 172.21.15.39:0/1971307111 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16041 (secure 0 0 0) 0x7ff0f80c9de0 con 0x7ff0f815a720 2024-01-24T17:42:14.716 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:14.716 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":20,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:41:34.358862+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:14.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.716+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0d006a430 msgr2=0x7ff0d006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.716+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0d006a430 0x7ff0d006c8f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8036000 tx=0x7ff0e8030000 comp rx=0 tx=0).stop 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.716+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 msgr2=0x7ff0f80c4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.716+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80c4600 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff0e000b3f0 tx=0x7ff0e000b8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.716+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 shutdown_connections 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.716+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0f815ab00 0x7ff0f80c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.717+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0d006a430 0x7ff0d006c8f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.717+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f815a720 0x7ff0f80c4600 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.717+0000 7ff0f7e00640 1 --2- 172.21.15.39:0/1971307111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0f80ce8e0 0x7ff0f80cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.717+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 >> 172.21.15.39:0/1971307111 conn(0x7ff0f806a090 msgr2=0x7ff0f806a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.717+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 shutdown_connections 2024-01-24T17:42:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.717+0000 7ff0f7e00640 1 -- 172.21.15.39:0/1971307111 wait complete. 2024-01-24T17:42:14.737 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:14.737 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:14.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.955+0000 7f13cfe00640 1 Processor -- start 2024-01-24T17:42:14.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.956+0000 7f13cfe00640 1 -- start start 2024-01-24T17:42:14.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.957+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d015c8d0 0x7f13d00d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.957+0000 7f13cfe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d00b38a0 con 0x7f13d00d2e80 2024-01-24T17:42:14.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.957+0000 7f13cfe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d00d13e0 con 0x7f13d00b2710 2024-01-24T17:42:14.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.957+0000 7f13cfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00d0ab0 con 0x7f13d015c8d0 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.957+0000 7f13cf400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f13d00d2e80 0x7f13d00cd630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59146/0 (socket says 172.21.15.39:59146) 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.957+0000 7f13cf400640 1 -- 172.21.15.39:0/3839442057 learned_addr learned my addr 172.21.15.39:0/3839442057 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13cea00640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d015c8d0 0x7f13d00d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271624770 0 0) 0x7f13d00b38a0 con 0x7f13d00d2e80 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b0003520 con 0x7f13d00d2e80 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1134315877 0 0) 0x7f13b0003520 con 0x7f13d00d2e80 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d015c8d0 msgr2=0x7f13d00d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d015c8d0 0x7f13d00d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 >> v1:172.21.15.100:6789/0 conn(0x7f13d00b2710 legacy=0x7f13d00d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d00d0d90 con 0x7f13d00d2e80 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f13c8001670 con 0x7f13d00d2e80 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 >> v1:172.21.15.39:6789/0 conn(0x7f13d00d2e80 legacy=0x7f13d00cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.958+0000 7f13ce000640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b00039a0 0x7f13b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13ce000640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d015c8d0 0x7f13b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13ce000640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b0004d00 0x7f13b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00d0ab0 con 0x7f13b0004d00 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b000ba30 con 0x7f13b00039a0 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b000bbf0 con 0x7f13d015c8d0 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13cf400640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b0004d00 0x7f13b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13cea00640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b00039a0 0x7f13b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13c5e00640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d015c8d0 0x7f13b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13cf400640 1 -- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d015c8d0 msgr2=0x7f13b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13cf400640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d015c8d0 0x7f13b00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13cf400640 1 -- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b00039a0 msgr2=0x7f13b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:14.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13cf400640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b00039a0 0x7f13b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.959+0000 7f13cf400640 1 -- 172.21.15.39:0/3839442057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b0003520 con 0x7f13b0004d00 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.960+0000 7f13cf400640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b0004d00 0x7f13b0005180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f13d008c990 tx=0x7f13b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fc3cfbc620f622 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.961+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8005ad0 con 0x7f13b0004d00 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.961+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c8005c30 con 0x7f13b0004d00 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.961+0000 7f13ce000640 1 -- 172.21.15.39:0/3839442057 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c802c830 con 0x7f13b0004d00 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.961+0000 7f13cfe00640 1 -- 172.21.15.39:0/3839442057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b0004d00 msgr2=0x7f13b0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.961+0000 7f13cfe00640 1 --2- 172.21.15.39:0/3839442057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b0004d00 0x7f13b0005180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f13d008c990 tx=0x7f13b00057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.961+0000 7f13cf400640 1 -- 172.21.15.39:0/3839442057 reap_dead start 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.961+0000 7f13cfe00640 1 -- 172.21.15.39:0/3839442057 shutdown_connections 2024-01-24T17:42:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.961+0000 7f13cfe00640 1 -- 172.21.15.39:0/3839442057 >> 172.21.15.39:0/3839442057 conn(0x7f13d0071dd0 msgr2=0x7f13d009bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:14.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.962+0000 7f13cfe00640 1 -- 172.21.15.39:0/3839442057 shutdown_connections 2024-01-24T17:42:14.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.962+0000 7f13cfe00640 1 -- 172.21.15.39:0/3839442057 wait complete. 2024-01-24T17:42:14.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.963+0000 7f13cfe00640 1 Processor -- start 2024-01-24T17:42:14.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.964+0000 7f13cfe00640 1 -- start start 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13b0004d00 0x7f13d01f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13d014f330 0x7f13d015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d00cbf20 0x7f13d00cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d00cc8e0 con 0x7f13d00cbf20 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cfe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d01497e0 con 0x7f13d014f330 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d0149ac0 con 0x7f13b0004d00 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d00cbf20 0x7f13d00cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d00cbf20 0x7f13d00cc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58500/0 (socket says 172.21.15.39:58500) 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cf400640 1 -- 172.21.15.39:0/465977305 learned_addr learned my addr 172.21.15.39:0/465977305 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cea00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13b0004d00 0x7f13d01f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13c5e00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13d014f330 0x7f13d015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cf400640 1 -- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13b0004d00 msgr2=0x7f13d01f1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cf400640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13b0004d00 0x7f13d01f1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cf400640 1 -- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13d014f330 msgr2=0x7f13d015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cf400640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13d014f330 0x7f13d015d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.965+0000 7f13cf400640 1 -- 172.21.15.39:0/465977305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d01f1fb0 con 0x7f13d00cbf20 2024-01-24T17:42:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.966+0000 7f13cf400640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d00cbf20 0x7f13d00cc3a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f13c800c2d0 tx=0x7f13c802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.966+0000 7f13c7400640 1 -- 172.21.15.39:0/465977305 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c803c040 con 0x7f13d00cbf20 2024-01-24T17:42:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.966+0000 7f13c7400640 1 -- 172.21.15.39:0/465977305 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c802cc40 con 0x7f13d00cbf20 2024-01-24T17:42:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.966+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d015e240 con 0x7f13d00cbf20 2024-01-24T17:42:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.966+0000 7f13c7400640 1 -- 172.21.15.39:0/465977305 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8001740 con 0x7f13d00cbf20 2024-01-24T17:42:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.966+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d015e800 con 0x7f13d00cbf20 2024-01-24T17:42:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.967+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d004dff0 con 0x7f13d00cbf20 2024-01-24T17:42:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.968+0000 7f13c7400640 1 -- 172.21.15.39:0/465977305 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f13c8032070 con 0x7f13d00cbf20 2024-01-24T17:42:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.969+0000 7f13c7400640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13a806a430 0x7f13a806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.969+0000 7f13c7400640 1 -- 172.21.15.39:0/465977305 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13c80a9710 con 0x7f13d00cbf20 2024-01-24T17:42:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.970+0000 7f13cea00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13a806a430 0x7f13a806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:14.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.971+0000 7f13cea00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13a806a430 0x7f13a806c8f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f13c0007380 tx=0x7f13c000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:14.973+0000 7f13c7400640 1 -- 172.21.15.39:0/465977305 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f13c8074c60 con 0x7f13d00cbf20 2024-01-24T17:42:15.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.169+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13d00b1930 con 0x7f13d00cbf20 2024-01-24T17:42:15.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.170+0000 7f13c7400640 1 -- 172.21.15.39:0/465977305 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13c8078280 con 0x7f13d00cbf20 2024-01-24T17:42:15.172 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:15.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.172+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13a806a430 msgr2=0x7f13a806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.172+0000 7f13cfe00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13a806a430 0x7f13a806c8f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f13c0007380 tx=0x7f13c000b040 comp rx=0 tx=0).stop 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.172+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d00cbf20 msgr2=0x7f13d00cc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.172+0000 7f13cfe00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d00cbf20 0x7f13d00cc3a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f13c800c2d0 tx=0x7f13c802e070 comp rx=0 tx=0).stop 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.173+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 shutdown_connections 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.173+0000 7f13cfe00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13a806a430 0x7f13a806c8f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.173+0000 7f13cfe00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d00cbf20 0x7f13d00cc3a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.173+0000 7f13cfe00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13d014f330 0x7f13d015d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.173+0000 7f13cfe00640 1 --2- 172.21.15.39:0/465977305 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13b0004d00 0x7f13d01f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.173+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 >> 172.21.15.39:0/465977305 conn(0x7f13d0071dd0 msgr2=0x7f13d00cfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.173+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 shutdown_connections 2024-01-24T17:42:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.173+0000 7f13cfe00640 1 -- 172.21.15.39:0/465977305 wait complete. 2024-01-24T17:42:15.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:15.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.417+0000 7f4df5600640 1 Processor -- start 2024-01-24T17:42:15.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.417+0000 7f4df5600640 1 -- start start 2024-01-24T17:42:15.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.418+0000 7f4df5600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 0x7f4df00cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.418+0000 7f4df5600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4df00ce390 0x7f4df00ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.418+0000 7f4df5600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4df00cc320 0x7f4df00cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.418+0000 7f4df5600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df00cac20 con 0x7f4df00ce390 2024-01-24T17:42:15.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.418+0000 7f4df5600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df015d090 con 0x7f4df00cc320 2024-01-24T17:42:15.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.418+0000 7f4df5600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df00ced30 con 0x7f4df015ccb0 2024-01-24T17:42:15.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.419+0000 7f4de7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4df00ce390 0x7f4df00ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.419+0000 7f4de7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4df00ce390 0x7f4df00ce7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58512/0 (socket says 172.21.15.39:58512) 2024-01-24T17:42:15.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.419+0000 7f4de7e00640 1 -- 172.21.15.39:0/3716577353 learned_addr learned my addr 172.21.15.39:0/3716577353 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:15.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.419+0000 7f4df4c00640 1 --2- 172.21.15.39:0/3716577353 >> v2:172.21.15.100:3300/0 conn(0x7f4df00cc320 0x7f4df00cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.419+0000 7f4defe00640 1 --2- 172.21.15.39:0/3716577353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 0x7f4df00cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.419+0000 7f4de7e00640 1 -- 172.21.15.39:0/3716577353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 msgr2=0x7f4df00cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:15.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.419+0000 7f4de7e00640 1 --2- 172.21.15.39:0/3716577353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 0x7f4df00cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.419+0000 7f4de7e00640 1 -- 172.21.15.39:0/3716577353 >> v2:172.21.15.100:3300/0 conn(0x7f4df00cc320 msgr2=0x7f4df00cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:15.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.420+0000 7f4de7e00640 1 --2- 172.21.15.39:0/3716577353 >> v2:172.21.15.100:3300/0 conn(0x7f4df00cc320 0x7f4df00cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.420+0000 7f4de7e00640 1 -- 172.21.15.39:0/3716577353 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df006c540 con 0x7f4df00ce390 2024-01-24T17:42:15.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.420+0000 7f4df4c00640 1 --2- 172.21.15.39:0/3716577353 >> v2:172.21.15.100:3300/0 conn(0x7f4df00cc320 0x7f4df00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:15.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.421+0000 7f4de7e00640 1 --2- 172.21.15.39:0/3716577353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00ce390 0x7f4df00ce7f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4df00c8a00 tx=0x7f4dd802db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a17fe74d227d66d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:15.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.421+0000 7f4def400640 1 -- 172.21.15.39:0/3716577353 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd802f8e0 con 0x7f4df00ce390 2024-01-24T17:42:15.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.421+0000 7f4def400640 1 -- 172.21.15.39:0/3716577353 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd802fa40 con 0x7f4df00ce390 2024-01-24T17:42:15.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.421+0000 7f4def400640 1 -- 172.21.15.39:0/3716577353 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd802fdd0 con 0x7f4df00ce390 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 -- 172.21.15.39:0/3716577353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00ce390 msgr2=0x7f4df00ce7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 --2- 172.21.15.39:0/3716577353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00ce390 0x7f4df00ce7f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4df00c8a00 tx=0x7f4dd802db20 comp rx=0 tx=0).stop 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 -- 172.21.15.39:0/3716577353 shutdown_connections 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 --2- 172.21.15.39:0/3716577353 >> v2:172.21.15.100:3300/0 conn(0x7f4df00cc320 0x7f4df00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 --2- 172.21.15.39:0/3716577353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00ce390 0x7f4df00ce7f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 --2- 172.21.15.39:0/3716577353 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 0x7f4df00cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 -- 172.21.15.39:0/3716577353 >> 172.21.15.39:0/3716577353 conn(0x7f4df0069e50 msgr2=0x7f4df006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 -- 172.21.15.39:0/3716577353 shutdown_connections 2024-01-24T17:42:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.422+0000 7f4df5600640 1 -- 172.21.15.39:0/3716577353 wait complete. 2024-01-24T17:42:15.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.424+0000 7f4df5600640 1 Processor -- start 2024-01-24T17:42:15.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.424+0000 7f4df5600640 1 -- start start 2024-01-24T17:42:15.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4df5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00cc320 0x7f4df009bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4df5600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4df00ce390 0x7f4df00c4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4df5600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 0x7f4df00981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4df5600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df009ad20 con 0x7f4df00cc320 2024-01-24T17:42:15.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4df5600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df009b000 con 0x7f4df00ce390 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4df5600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df009b2e0 con 0x7f4df015ccb0 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4defe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00cc320 0x7f4df009bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4defe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00cc320 0x7f4df009bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58522/0 (socket says 172.21.15.39:58522) 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.425+0000 7f4defe00640 1 -- 172.21.15.39:0/2548297699 learned_addr learned my addr 172.21.15.39:0/2548297699 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4df4c00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 0x7f4df00981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4de7e00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4df00ce390 0x7f4df00c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4defe00640 1 -- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 msgr2=0x7f4df00981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4defe00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 0x7f4df00981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4defe00640 1 -- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4df00ce390 msgr2=0x7f4df00c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4defe00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4df00ce390 0x7f4df00c4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4defe00640 1 -- 172.21.15.39:0/2548297699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd80094d0 con 0x7f4df00cc320 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4de7e00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4df00ce390 0x7f4df00c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:15.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4defe00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00cc320 0x7f4df009bdc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4de000d7d0 tx=0x7f4de000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:15.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4decc00640 1 -- 172.21.15.39:0/2548297699 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de00096c0 con 0x7f4df00cc320 2024-01-24T17:42:15.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4decc00640 1 -- 172.21.15.39:0/2548297699 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de0009820 con 0x7f4df00cc320 2024-01-24T17:42:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4decc00640 1 -- 172.21.15.39:0/2548297699 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de0009bd0 con 0x7f4df00cc320 2024-01-24T17:42:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.426+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df009b5c0 con 0x7f4df00cc320 2024-01-24T17:42:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.427+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df00c4870 con 0x7f4df00cc320 2024-01-24T17:42:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.428+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df004dff0 con 0x7f4df00cc320 2024-01-24T17:42:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.429+0000 7f4decc00640 1 -- 172.21.15.39:0/2548297699 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4de0009d30 con 0x7f4df00cc320 2024-01-24T17:42:15.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.430+0000 7f4decc00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4dc806a430 0x7f4dc806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.430+0000 7f4decc00640 1 -- 172.21.15.39:0/2548297699 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4de00890b0 con 0x7f4df00cc320 2024-01-24T17:42:15.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.430+0000 7f4de7e00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4dc806a430 0x7f4dc806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.431+0000 7f4de7e00640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4dc806a430 0x7f4dc806c8f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8038000 tx=0x7f4dd8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:15.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.434+0000 7f4decc00640 1 -- 172.21.15.39:0/2548297699 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4de0054680 con 0x7f4df00cc320 2024-01-24T17:42:15.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.629+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4df006d9a0 con 0x7f4df00cc320 2024-01-24T17:42:15.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.630+0000 7f4decc00640 1 -- 172.21.15.39:0/2548297699 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4de0057ca0 con 0x7f4df00cc320 2024-01-24T17:42:15.632 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4dc806a430 msgr2=0x7f4dc806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4dc806a430 0x7f4dc806c8f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8038000 tx=0x7f4dd8008030 comp rx=0 tx=0).stop 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00cc320 msgr2=0x7f4df009bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00cc320 0x7f4df009bdc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4de000d7d0 tx=0x7f4de000dc90 comp rx=0 tx=0).stop 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 shutdown_connections 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4df015ccb0 0x7f4df00981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4df00ce390 0x7f4df00c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4dc806a430 0x7f4dc806c8f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 --2- 172.21.15.39:0/2548297699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4df00cc320 0x7f4df009bdc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 >> 172.21.15.39:0/2548297699 conn(0x7f4df0069e50 msgr2=0x7f4df00ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:15.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 shutdown_connections 2024-01-24T17:42:15.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.633+0000 7f4df5600640 1 -- 172.21.15.39:0/2548297699 wait complete. 2024-01-24T17:42:15.654 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:15.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.877+0000 7fc701200640 1 Processor -- start 2024-01-24T17:42:15.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.877+0000 7fc701200640 1 -- start start 2024-01-24T17:42:15.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.879+0000 7fc701200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6fc15c8d0 0x7fc6fc0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.879+0000 7fc701200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0d2e80 0x7fc6fc0cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.879+0000 7fc701200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6fc0b38a0 con 0x7fc6fc0b2710 2024-01-24T17:42:15.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.879+0000 7fc701200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6fc0d13e0 con 0x7fc6fc15c8d0 2024-01-24T17:42:15.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.879+0000 7fc701200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc0d0ab0 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.879+0000 7fc6fb400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6fc15c8d0 0x7fc6fc0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.880+0000 7fc6fb400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6fc15c8d0 0x7fc6fc0d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58524/0 (socket says 172.21.15.39:58524) 2024-01-24T17:42:15.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.880+0000 7fc6fb400640 1 -- 172.21.15.39:0/1402807865 learned_addr learned my addr 172.21.15.39:0/1402807865 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:15.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.880+0000 7fc6fb400640 1 -- 172.21.15.39:0/1402807865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0d2e80 msgr2=0x7fc6fc0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:15.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.880+0000 7fc6fbe00640 1 --2- 172.21.15.39:0/1402807865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0d2e80 0x7fc6fc0cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.880+0000 7fc6fb400640 1 --2- 172.21.15.39:0/1402807865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0d2e80 0x7fc6fc0cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.880+0000 7fc6fb400640 1 -- 172.21.15.39:0/1402807865 >> v1:172.21.15.100:6789/0 conn(0x7fc6fc0b2710 legacy=0x7fc6fc0d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.880+0000 7fc6fb400640 1 -- 172.21.15.39:0/1402807865 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc0d0d90 con 0x7fc6fc15c8d0 2024-01-24T17:42:15.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.881+0000 7fc6fb400640 1 --2- 172.21.15.39:0/1402807865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc15c8d0 0x7fc6fc0d03b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009870 tx=0x7fc6ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb8c864bbf8bd08 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:15.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.882+0000 7fc6faa00640 1 -- 172.21.15.39:0/1402807865 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec030840 con 0x7fc6fc15c8d0 2024-01-24T17:42:15.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.882+0000 7fc6faa00640 1 -- 172.21.15.39:0/1402807865 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec0309a0 con 0x7fc6fc15c8d0 2024-01-24T17:42:15.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.882+0000 7fc6faa00640 1 -- 172.21.15.39:0/1402807865 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec030d30 con 0x7fc6fc15c8d0 2024-01-24T17:42:15.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.882+0000 7fc701200640 1 -- 172.21.15.39:0/1402807865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc15c8d0 msgr2=0x7fc6fc0d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.882+0000 7fc701200640 1 --2- 172.21.15.39:0/1402807865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc15c8d0 0x7fc6fc0d03b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009870 tx=0x7fc6ec02b980 comp rx=0 tx=0).stop 2024-01-24T17:42:15.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.883+0000 7fc701200640 1 -- 172.21.15.39:0/1402807865 shutdown_connections 2024-01-24T17:42:15.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.883+0000 7fc701200640 1 --2- 172.21.15.39:0/1402807865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0d2e80 0x7fc6fc0cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.883+0000 7fc701200640 1 --2- 172.21.15.39:0/1402807865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc15c8d0 0x7fc6fc0d03b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.883+0000 7fc701200640 1 -- 172.21.15.39:0/1402807865 >> 172.21.15.39:0/1402807865 conn(0x7fc6fc071dd0 msgr2=0x7fc6fc09bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:15.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.883+0000 7fc701200640 1 -- 172.21.15.39:0/1402807865 shutdown_connections 2024-01-24T17:42:15.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.883+0000 7fc701200640 1 -- 172.21.15.39:0/1402807865 wait complete. 2024-01-24T17:42:15.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.884+0000 7fc701200640 1 Processor -- start 2024-01-24T17:42:15.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.885+0000 7fc701200640 1 -- start start 2024-01-24T17:42:15.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc701200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0b2710 0x7fc6fc0b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc701200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc0d2e80 0x7fc6fc14d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc701200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6fc15c8d0 0x7fc6fc0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc701200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc14e930 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc701200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc14e370 con 0x7fc6fc15c8d0 2024-01-24T17:42:15.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc701200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc14e650 con 0x7fc6fc0b2710 2024-01-24T17:42:15.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6f3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc0d2e80 0x7fc6fc14d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6f3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc0d2e80 0x7fc6fc14d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58528/0 (socket says 172.21.15.39:58528) 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6f3e00640 1 -- 172.21.15.39:0/2791387371 learned_addr learned my addr 172.21.15.39:0/2791387371 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6fb400640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0b2710 0x7fc6fc0b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6fbe00640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6fc15c8d0 0x7fc6fc0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6f3e00640 1 -- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0b2710 msgr2=0x7fc6fc0b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6f3e00640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0b2710 0x7fc6fc0b7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6f3e00640 1 -- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6fc15c8d0 msgr2=0x7fc6fc0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6f3e00640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6fc15c8d0 0x7fc6fc0ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.886+0000 7fc6f3e00640 1 -- 172.21.15.39:0/2791387371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec0094d0 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.887+0000 7fc6f3e00640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc0d2e80 0x7fc6fc14d940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400c400 tx=0x7fc6e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:15.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.887+0000 7fc6f3400640 1 -- 172.21.15.39:0/2791387371 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e40052d0 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.887+0000 7fc6f3400640 1 -- 172.21.15.39:0/2791387371 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e4005450 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.887+0000 7fc6f3400640 1 -- 172.21.15.39:0/2791387371 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4007260 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.887+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc14baa0 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.887+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc0b7a90 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.888+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6fc14b900 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.889+0000 7fc6f3400640 1 -- 172.21.15.39:0/2791387371 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc6e40073c0 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.890+0000 7fc6f3400640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6d406a430 0x7fc6d406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:15.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.890+0000 7fc6f3400640 1 -- 172.21.15.39:0/2791387371 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc6e4087e10 con 0x7fc6fc0d2e80 2024-01-24T17:42:15.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.891+0000 7fc6fb400640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6d406a430 0x7fc6d406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:15.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.892+0000 7fc6fb400640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6d406a430 0x7fc6d406c8f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec036000 tx=0x7fc6ec03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:15.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:15.895+0000 7fc6f3400640 1 -- 172.21.15.39:0/2791387371 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc6fc14b900 con 0x7fc6fc0d2e80 2024-01-24T17:42:16.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.090+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6fc14b900 con 0x7fc6fc0d2e80 2024-01-24T17:42:16.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.091+0000 7fc6f3400640 1 -- 172.21.15.39:0/2791387371 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc6fc14b900 con 0x7fc6fc0d2e80 2024-01-24T17:42:16.093 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6d406a430 msgr2=0x7fc6d406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6d406a430 0x7fc6d406c8f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec036000 tx=0x7fc6ec03b040 comp rx=0 tx=0).stop 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc0d2e80 msgr2=0x7fc6fc14d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc0d2e80 0x7fc6fc14d940 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400c400 tx=0x7fc6e400c8c0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 shutdown_connections 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6fc15c8d0 0x7fc6fc0ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6d406a430 0x7fc6d406c8f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6fc0d2e80 0x7fc6fc14d940 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 --2- 172.21.15.39:0/2791387371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6fc0b2710 0x7fc6fc0b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 >> 172.21.15.39:0/2791387371 conn(0x7fc6fc071dd0 msgr2=0x7fc6fc15f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 shutdown_connections 2024-01-24T17:42:16.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.094+0000 7fc701200640 1 -- 172.21.15.39:0/2791387371 wait complete. 2024-01-24T17:42:16.113 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:16.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.308+0000 7f228f400640 1 Processor -- start 2024-01-24T17:42:16.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.309+0000 7f228f400640 1 -- start start 2024-01-24T17:42:16.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f228815c280 0x7f22880cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 0x7f228806bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228f400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22880c9670 con 0x7f22880c9b00 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228f400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22880adab0 con 0x7f228815c280 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228f400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22880c8a60 con 0x7f22880c9f90 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228e000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f228815c280 0x7f22880cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228e000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f228815c280 0x7f22880cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58534/0 (socket says 172.21.15.39:58534) 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228d600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f22880c9b00 0x7f2288071130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59150/0 (socket says 172.21.15.39:59150) 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.310+0000 7f228e000640 1 -- 172.21.15.39:0/3763988241 learned_addr learned my addr 172.21.15.39:0/3763988241 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.311+0000 7f228ea00640 1 --2- 172.21.15.39:0/3763988241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 0x7f228806bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.311+0000 7f228e000640 1 -- 172.21.15.39:0/3763988241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 msgr2=0x7f228806bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.311+0000 7f228e000640 1 --2- 172.21.15.39:0/3763988241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 0x7f228806bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.311+0000 7f228e000640 1 -- 172.21.15.39:0/3763988241 >> v1:172.21.15.39:6789/0 conn(0x7f22880c9b00 legacy=0x7f2288071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.311+0000 7f228e000640 1 -- 172.21.15.39:0/3763988241 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880c8d40 con 0x7f228815c280 2024-01-24T17:42:16.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.311+0000 7f228cc00640 1 -- 172.21.15.39:0/3763988241 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422014711 0 0) 0x7f22880c9670 con 0x7f22880c9b00 2024-01-24T17:42:16.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.312+0000 7f228e000640 1 --2- 172.21.15.39:0/3763988241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f228815c280 0x7f22880cb2b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f227800bb10 tx=0x7f227802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f5d70dbc1f6cec server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:16.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.312+0000 7f228cc00640 1 -- 172.21.15.39:0/3763988241 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2278036440 con 0x7f228815c280 2024-01-24T17:42:16.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228cc00640 1 -- 172.21.15.39:0/3763988241 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22780365a0 con 0x7f228815c280 2024-01-24T17:42:16.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228cc00640 1 -- 172.21.15.39:0/3763988241 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2278036930 con 0x7f228815c280 2024-01-24T17:42:16.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228f400640 1 -- 172.21.15.39:0/3763988241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f228815c280 msgr2=0x7f22880cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228f400640 1 --2- 172.21.15.39:0/3763988241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f228815c280 0x7f22880cb2b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f227800bb10 tx=0x7f227802e150 comp rx=0 tx=0).stop 2024-01-24T17:42:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228f400640 1 -- 172.21.15.39:0/3763988241 shutdown_connections 2024-01-24T17:42:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228f400640 1 --2- 172.21.15.39:0/3763988241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 0x7f228806bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228f400640 1 --2- 172.21.15.39:0/3763988241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f228815c280 0x7f22880cb2b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228f400640 1 -- 172.21.15.39:0/3763988241 >> 172.21.15.39:0/3763988241 conn(0x7f228806a090 msgr2=0x7f228806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228f400640 1 -- 172.21.15.39:0/3763988241 shutdown_connections 2024-01-24T17:42:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.313+0000 7f228f400640 1 -- 172.21.15.39:0/3763988241 wait complete. 2024-01-24T17:42:16.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.315+0000 7f228f400640 1 Processor -- start 2024-01-24T17:42:16.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.315+0000 7f228f400640 1 -- start start 2024-01-24T17:42:16.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.316+0000 7f228f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22880c9b00 0x7f22880c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.316+0000 7f228f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 0x7f228814ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.316+0000 7f228f400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f228815c280 0x7f22880ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.316+0000 7f228f400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22880c6eb0 con 0x7f22880c9b00 2024-01-24T17:42:16.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.316+0000 7f228f400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22880c62a0 con 0x7f228815c280 2024-01-24T17:42:16.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.316+0000 7f228f400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22880c6580 con 0x7f22880c9f90 2024-01-24T17:42:16.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.316+0000 7f228e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22880c9b00 0x7f22880c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22880c9b00 0x7f22880c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58544/0 (socket says 172.21.15.39:58544) 2024-01-24T17:42:16.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228e000640 1 -- 172.21.15.39:0/3294838641 learned_addr learned my addr 172.21.15.39:0/3294838641 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:16.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228ea00640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f228815c280 0x7f22880ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228d600640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 0x7f228814ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228e000640 1 -- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 msgr2=0x7f228814ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228e000640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 0x7f228814ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228e000640 1 -- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f228815c280 msgr2=0x7f22880ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228e000640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f228815c280 0x7f22880ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228e000640 1 -- 172.21.15.39:0/3294838641 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227800b770 con 0x7f22880c9b00 2024-01-24T17:42:16.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.317+0000 7f228e000640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22880c9b00 0x7f22880c7340 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f227800b2c0 tx=0x7f22780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:16.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.321+0000 7f2286000640 1 -- 172.21.15.39:0/3294838641 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2278036440 con 0x7f22880c9b00 2024-01-24T17:42:16.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.321+0000 7f2286000640 1 -- 172.21.15.39:0/3294838641 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2278034ba0 con 0x7f22880c9b00 2024-01-24T17:42:16.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.321+0000 7f2286000640 1 -- 172.21.15.39:0/3294838641 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2278041440 con 0x7f22880c9b00 2024-01-24T17:42:16.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.321+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22880c6860 con 0x7f22880c9b00 2024-01-24T17:42:16.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.321+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22880ce8a0 con 0x7f22880c9b00 2024-01-24T17:42:16.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.322+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228804dff0 con 0x7f22880c9b00 2024-01-24T17:42:16.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.323+0000 7f2286000640 1 -- 172.21.15.39:0/3294838641 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f22780403e0 con 0x7f22880c9b00 2024-01-24T17:42:16.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.324+0000 7f2286000640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f226006a430 0x7f226006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.324+0000 7f2286000640 1 -- 172.21.15.39:0/3294838641 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f22780aad50 con 0x7f22880c9b00 2024-01-24T17:42:16.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.325+0000 7f228d600640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f226006a430 0x7f226006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.326+0000 7f228d600640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f226006a430 0x7f226006c8f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2270007fd0 tx=0x7f2270009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:16.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.329+0000 7f2286000640 1 -- 172.21.15.39:0/3294838641 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2278076200 con 0x7f22880c9b00 2024-01-24T17:42:16.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.525+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22880b4390 con 0x7f22880c9b00 2024-01-24T17:42:16.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.528+0000 7f2286000640 1 -- 172.21.15.39:0/3294838641 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f2278046090 con 0x7f22880c9b00 2024-01-24T17:42:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:16.530 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:16.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.530+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f226006a430 msgr2=0x7f226006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.530+0000 7f228f400640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f226006a430 0x7f226006c8f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2270007fd0 tx=0x7f2270009040 comp rx=0 tx=0).stop 2024-01-24T17:42:16.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.530+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22880c9b00 msgr2=0x7f22880c7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.530+0000 7f228f400640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22880c9b00 0x7f22880c7340 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f227800b2c0 tx=0x7f22780046c0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.530+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 shutdown_connections 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.531+0000 7f228f400640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f228815c280 0x7f22880ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.531+0000 7f228f400640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f22880c9f90 0x7f228814ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.531+0000 7f228f400640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f226006a430 0x7f226006c8f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.531+0000 7f228f400640 1 --2- 172.21.15.39:0/3294838641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22880c9b00 0x7f22880c7340 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.531+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 >> 172.21.15.39:0/3294838641 conn(0x7f228806a090 msgr2=0x7f228806a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.531+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 shutdown_connections 2024-01-24T17:42:16.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.531+0000 7f228f400640 1 -- 172.21.15.39:0/3294838641 wait complete. 2024-01-24T17:42:16.551 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:16.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.774+0000 7f737d400640 1 Processor -- start 2024-01-24T17:42:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.774+0000 7f737d400640 1 -- start start 2024-01-24T17:42:16.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.775+0000 7f737d400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f73780ce820 0x7f73780c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.775+0000 7f737d400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737815d090 con 0x7f737815ccb0 2024-01-24T17:42:16.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.775+0000 7f737d400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73780ce1a0 con 0x7f73780ce390 2024-01-24T17:42:16.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.775+0000 7f737d400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73780cac20 con 0x7f73780ce820 2024-01-24T17:42:16.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.775+0000 7f7377e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f737815ccb0 0x7f73780cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59158/0 (socket says 172.21.15.39:59158) 2024-01-24T17:42:16.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.775+0000 7f7377e00640 1 -- 172.21.15.39:0/1014405208 learned_addr learned my addr 172.21.15.39:0/1014405208 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:16.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.775+0000 7f737ca00640 1 --2- 172.21.15.39:0/1014405208 >> v2:172.21.15.39:3300/0 conn(0x7f73780ce820 0x7f73780c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.776+0000 7f7376a00640 1 -- 172.21.15.39:0/1014405208 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400548382 0 0) 0x7f737815d090 con 0x7f737815ccb0 2024-01-24T17:42:16.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.776+0000 7f7376a00640 1 -- 172.21.15.39:0/1014405208 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7358003520 con 0x7f737815ccb0 2024-01-24T17:42:16.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.776+0000 7f737ca00640 1 -- 172.21.15.39:0/1014405208 >> v1:172.21.15.100:6789/0 conn(0x7f73780ce390 legacy=0x7f73780c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.776+0000 7f737ca00640 1 -- 172.21.15.39:0/1014405208 >> v1:172.21.15.39:6789/0 conn(0x7f737815ccb0 legacy=0x7f73780cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.776+0000 7f737ca00640 1 -- 172.21.15.39:0/1014405208 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737806c540 con 0x7f73780ce820 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.777+0000 7f737ca00640 1 --2- 172.21.15.39:0/1014405208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce820 0x7f73780c9460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f73700097a0 tx=0x7f737002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1dc7d29bdcc765e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.777+0000 7f7376a00640 1 -- 172.21.15.39:0/1014405208 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370030610 con 0x7f73780ce820 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f7376a00640 1 -- 172.21.15.39:0/1014405208 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7370030770 con 0x7f73780ce820 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f7376a00640 1 -- 172.21.15.39:0/1014405208 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370030b00 con 0x7f73780ce820 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f737d400640 1 -- 172.21.15.39:0/1014405208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce820 msgr2=0x7f73780c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f737d400640 1 --2- 172.21.15.39:0/1014405208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce820 0x7f73780c9460 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f73700097a0 tx=0x7f737002b960 comp rx=0 tx=0).stop 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f737d400640 1 -- 172.21.15.39:0/1014405208 shutdown_connections 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f737d400640 1 --2- 172.21.15.39:0/1014405208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce820 0x7f73780c9460 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f737d400640 1 -- 172.21.15.39:0/1014405208 >> 172.21.15.39:0/1014405208 conn(0x7f7378069e50 msgr2=0x7f737806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f737d400640 1 -- 172.21.15.39:0/1014405208 shutdown_connections 2024-01-24T17:42:16.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.778+0000 7f737d400640 1 -- 172.21.15.39:0/1014405208 wait complete. 2024-01-24T17:42:16.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.780+0000 7f737d400640 1 Processor -- start 2024-01-24T17:42:16.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.780+0000 7f737d400640 1 -- start start 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f737d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce390 0x7f737815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f737d400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780ce820 0x7f73780c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f737d400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f737815ccb0 0x7f73780c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f737d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780c4040 con 0x7f73780ce390 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f737d400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780c4a10 con 0x7f73780ce820 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f737d400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73780c3430 con 0x7f737815ccb0 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f7377e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce390 0x7f737815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f7377e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce390 0x7f737815aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58550/0 (socket says 172.21.15.39:58550) 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.781+0000 7f7377e00640 1 -- 172.21.15.39:0/1011693454 learned_addr learned my addr 172.21.15.39:0/1011693454 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f737ca00640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f737815ccb0 0x7f73780c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f7377400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780ce820 0x7f73780c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f7377e00640 1 -- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f737815ccb0 msgr2=0x7f73780c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f7377e00640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f737815ccb0 0x7f73780c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f7377e00640 1 -- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780ce820 msgr2=0x7f73780c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f7377e00640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780ce820 0x7f73780c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f7377e00640 1 -- 172.21.15.39:0/1011693454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7370009400 con 0x7f73780ce390 2024-01-24T17:42:16.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f7377400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780ce820 0x7f73780c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:16.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.782+0000 7f7377e00640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce390 0x7f737815aea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7368004e80 tx=0x7f736800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:16.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.783+0000 7f736fe00640 1 -- 172.21.15.39:0/1011693454 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736800ce20 con 0x7f73780ce390 2024-01-24T17:42:16.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.783+0000 7f736fe00640 1 -- 172.21.15.39:0/1011693454 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7368012400 con 0x7f73780ce390 2024-01-24T17:42:16.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.783+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73780c3710 con 0x7f73780ce390 2024-01-24T17:42:16.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.783+0000 7f736fe00640 1 -- 172.21.15.39:0/1011693454 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73680127d0 con 0x7f73780ce390 2024-01-24T17:42:16.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.783+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73780c3cd0 con 0x7f73780ce390 2024-01-24T17:42:16.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.784+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73780c4b80 con 0x7f73780ce390 2024-01-24T17:42:16.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.785+0000 7f736fe00640 1 -- 172.21.15.39:0/1011693454 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7368007070 con 0x7f73780ce390 2024-01-24T17:42:16.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.786+0000 7f736fe00640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f735006a460 0x7f735006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:16.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.786+0000 7f736fe00640 1 -- 172.21.15.39:0/1011693454 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7368088990 con 0x7f73780ce390 2024-01-24T17:42:16.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.786+0000 7f7377400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f735006a460 0x7f735006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:16.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.787+0000 7f7377400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f735006a460 0x7f735006c920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f737809c350 tx=0x7f7360009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:16.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.790+0000 7f736fe00640 1 -- 172.21.15.39:0/1011693454 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7368053f50 con 0x7f73780ce390 2024-01-24T17:42:16.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.983+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f73780b2df0 con 0x7f73780ce390 2024-01-24T17:42:16.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.984+0000 7f736fe00640 1 -- 172.21.15.39:0/1011693454 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7368017070 con 0x7f73780ce390 2024-01-24T17:42:16.986 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:16.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f735006a460 msgr2=0x7f735006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f735006a460 0x7f735006c920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f737809c350 tx=0x7f7360009040 comp rx=0 tx=0).stop 2024-01-24T17:42:16.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce390 msgr2=0x7f737815aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:16.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce390 0x7f737815aea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7368004e80 tx=0x7f736800a430 comp rx=0 tx=0).stop 2024-01-24T17:42:16.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 shutdown_connections 2024-01-24T17:42:16.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f737815ccb0 0x7f73780c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73780ce820 0x7f73780c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f735006a460 0x7f735006c920 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 --2- 172.21.15.39:0/1011693454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73780ce390 0x7f737815aea0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:16.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.987+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 >> 172.21.15.39:0/1011693454 conn(0x7f7378069e50 msgr2=0x7f737806a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:16.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.988+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 shutdown_connections 2024-01-24T17:42:16.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:16.988+0000 7f737d400640 1 -- 172.21.15.39:0/1011693454 wait complete. 2024-01-24T17:42:17.009 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:17.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.231+0000 7f6bda600640 1 Processor -- start 2024-01-24T17:42:17.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.232+0000 7f6bda600640 1 -- start start 2024-01-24T17:42:17.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.233+0000 7f6bda600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6bd415fff0 0x7f6bd40b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.233+0000 7f6bda600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6bd40d14c0 0x7f6bd40cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.233+0000 7f6bda600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd40cda90 con 0x7f6bd40b2a10 2024-01-24T17:42:17.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.233+0000 7f6bda600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd40b2e90 con 0x7f6bd40d14c0 2024-01-24T17:42:17.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.233+0000 7f6bda600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd40d32c0 con 0x7f6bd415fff0 2024-01-24T17:42:17.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6bd40d14c0 0x7f6bd40cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6bd40d14c0 0x7f6bd40cdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58560/0 (socket says 172.21.15.39:58560) 2024-01-24T17:42:17.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd3e00640 1 -- 172.21.15.39:0/4290180136 learned_addr learned my addr 172.21.15.39:0/4290180136 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:17.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd3400640 1 --2- 172.21.15.39:0/4290180136 >> v2:172.21.15.100:3300/0 conn(0x7f6bd415fff0 0x7f6bd40b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd3e00640 1 -- 172.21.15.39:0/4290180136 >> v2:172.21.15.100:3300/0 conn(0x7f6bd415fff0 msgr2=0x7f6bd40b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:17.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd3e00640 1 --2- 172.21.15.39:0/4290180136 >> v2:172.21.15.100:3300/0 conn(0x7f6bd415fff0 0x7f6bd40b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd3e00640 1 -- 172.21.15.39:0/4290180136 >> v1:172.21.15.39:6789/0 conn(0x7f6bd40b2a10 legacy=0x7f6bd40d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd2a00640 1 -- 172.21.15.39:0/4290180136 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603975083 0 0) 0x7f6bd40cda90 con 0x7f6bd40b2a10 2024-01-24T17:42:17.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.234+0000 7f6bd3e00640 1 -- 172.21.15.39:0/4290180136 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40b30a0 con 0x7f6bd40d14c0 2024-01-24T17:42:17.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.235+0000 7f6bd3e00640 1 --2- 172.21.15.39:0/4290180136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd40d14c0 0x7f6bd40cdf20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40d28e0 tx=0x7f6bcc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa205f3fbe3ee47 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:17.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bd2a00640 1 -- 172.21.15.39:0/4290180136 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc02cb20 con 0x7f6bd40d14c0 2024-01-24T17:42:17.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bd2a00640 1 -- 172.21.15.39:0/4290180136 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc02cc80 con 0x7f6bd40d14c0 2024-01-24T17:42:17.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bd2a00640 1 -- 172.21.15.39:0/4290180136 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc0343f0 con 0x7f6bd40d14c0 2024-01-24T17:42:17.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bda600640 1 -- 172.21.15.39:0/4290180136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd40d14c0 msgr2=0x7f6bd40cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bda600640 1 --2- 172.21.15.39:0/4290180136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd40d14c0 0x7f6bd40cdf20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40d28e0 tx=0x7f6bcc02bf80 comp rx=0 tx=0).stop 2024-01-24T17:42:17.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bda600640 1 -- 172.21.15.39:0/4290180136 shutdown_connections 2024-01-24T17:42:17.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bda600640 1 --2- 172.21.15.39:0/4290180136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd40d14c0 0x7f6bd40cdf20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bda600640 1 --2- 172.21.15.39:0/4290180136 >> v2:172.21.15.100:3300/0 conn(0x7f6bd415fff0 0x7f6bd40b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.236+0000 7f6bda600640 1 -- 172.21.15.39:0/4290180136 >> 172.21.15.39:0/4290180136 conn(0x7f6bd409bcc0 msgr2=0x7f6bd40d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:17.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.237+0000 7f6bda600640 1 -- 172.21.15.39:0/4290180136 shutdown_connections 2024-01-24T17:42:17.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.237+0000 7f6bda600640 1 -- 172.21.15.39:0/4290180136 wait complete. 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.238+0000 7f6bda600640 1 Processor -- start 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.238+0000 7f6bda600640 1 -- start start 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.239+0000 7f6bda600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd40b2a10 0x7f6bd41f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.239+0000 7f6bda600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bd40d14c0 0x7f6bd415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.239+0000 7f6bda600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd415fff0 0x7f6bd40cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.239+0000 7f6bda600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd41e7000 con 0x7f6bd415fff0 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.239+0000 7f6bda600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd415c8b0 con 0x7f6bd40b2a10 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.239+0000 7f6bda600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40b4180 con 0x7f6bd40d14c0 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd415fff0 0x7f6bd40cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd415fff0 0x7f6bd40cbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58574/0 (socket says 172.21.15.39:58574) 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3e00640 1 -- 172.21.15.39:0/1439233897 learned_addr learned my addr 172.21.15.39:0/1439233897 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:17.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bcbe00640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bd40d14c0 0x7f6bd415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3400640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd40b2a10 0x7f6bd41f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3e00640 1 -- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bd40d14c0 msgr2=0x7f6bd415c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3e00640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bd40d14c0 0x7f6bd415c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3e00640 1 -- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd40b2a10 msgr2=0x7f6bd41f1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3e00640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd40b2a10 0x7f6bd41f1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.240+0000 7f6bd3e00640 1 -- 172.21.15.39:0/1439233897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc009b30 con 0x7f6bd415fff0 2024-01-24T17:42:17.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.241+0000 7f6bd3e00640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd415fff0 0x7f6bd40cbda0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036040 tx=0x7f6bcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:17.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.241+0000 7f6bcb400640 1 -- 172.21.15.39:0/1439233897 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc007e70 con 0x7f6bd415fff0 2024-01-24T17:42:17.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.241+0000 7f6bcb400640 1 -- 172.21.15.39:0/1439233897 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc009510 con 0x7f6bd415fff0 2024-01-24T17:42:17.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.241+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd40b4420 con 0x7f6bd415fff0 2024-01-24T17:42:17.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.241+0000 7f6bcb400640 1 -- 172.21.15.39:0/1439233897 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc03e690 con 0x7f6bd415fff0 2024-01-24T17:42:17.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.241+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd40b49e0 con 0x7f6bd415fff0 2024-01-24T17:42:17.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.242+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd4073ef0 con 0x7f6bd415fff0 2024-01-24T17:42:17.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.243+0000 7f6bcb400640 1 -- 172.21.15.39:0/1439233897 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6bcc03e7f0 con 0x7f6bd415fff0 2024-01-24T17:42:17.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.244+0000 7f6bcb400640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6bac06a430 0x7f6bac06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.244+0000 7f6bcb400640 1 -- 172.21.15.39:0/1439233897 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6bcc0ab8f0 con 0x7f6bd415fff0 2024-01-24T17:42:17.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.245+0000 7f6bd3400640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6bac06a430 0x7f6bac06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.246+0000 7f6bd3400640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6bac06a430 0x7f6bac06c8f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4006fd0 tx=0x7f6bc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:17.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.249+0000 7f6bcb400640 1 -- 172.21.15.39:0/1439233897 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6bcc076eb0 con 0x7f6bd415fff0 2024-01-24T17:42:17.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.444+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6bd40d2ff0 con 0x7f6bd415fff0 2024-01-24T17:42:17.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.445+0000 7f6bcb400640 1 -- 172.21.15.39:0/1439233897 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6bcc043020 con 0x7f6bd415fff0 2024-01-24T17:42:17.446 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:17.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6bac06a430 msgr2=0x7f6bac06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6bac06a430 0x7f6bac06c8f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4006fd0 tx=0x7f6bc4008040 comp rx=0 tx=0).stop 2024-01-24T17:42:17.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd415fff0 msgr2=0x7f6bd40cbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd415fff0 0x7f6bd40cbda0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc036040 tx=0x7f6bcc004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:17.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 shutdown_connections 2024-01-24T17:42:17.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6bac06a430 0x7f6bac06c8f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6bd415fff0 0x7f6bd40cbda0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6bd40d14c0 0x7f6bd415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 --2- 172.21.15.39:0/1439233897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd40b2a10 0x7f6bd41f1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.447+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 >> 172.21.15.39:0/1439233897 conn(0x7f6bd409bcc0 msgr2=0x7f6bd40d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:17.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.448+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 shutdown_connections 2024-01-24T17:42:17.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.448+0000 7f6bda600640 1 -- 172.21.15.39:0/1439233897 wait complete. 2024-01-24T17:42:17.466 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:17.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.686+0000 7faba4e00640 1 Processor -- start 2024-01-24T17:42:17.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.686+0000 7faba4e00640 1 -- start start 2024-01-24T17:42:17.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.687+0000 7faba4e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7faba00accd0 0x7faba006d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.687+0000 7faba4e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faba00cb540 0x7faba00ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.687+0000 7faba4e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba00ad0b0 con 0x7faba006c3a0 2024-01-24T17:42:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.687+0000 7faba4e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba006d840 con 0x7faba00cb540 2024-01-24T17:42:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.687+0000 7faba4e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba00b2370 con 0x7faba00accd0 2024-01-24T17:42:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.688+0000 7fab9fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faba00cb540 0x7faba00ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.688+0000 7fab9fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faba00cb540 0x7faba00ab030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58576/0 (socket says 172.21.15.39:58576) 2024-01-24T17:42:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.688+0000 7fab9fe00640 1 -- 172.21.15.39:0/51610354 learned_addr learned my addr 172.21.15.39:0/51610354 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.688+0000 7fab9f400640 1 --2- 172.21.15.39:0/51610354 >> v2:172.21.15.100:3300/0 conn(0x7faba00accd0 0x7faba006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.689+0000 7fab9fe00640 1 -- 172.21.15.39:0/51610354 >> v2:172.21.15.100:3300/0 conn(0x7faba00accd0 msgr2=0x7faba006d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:17.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.689+0000 7fab9fe00640 1 --2- 172.21.15.39:0/51610354 >> v2:172.21.15.100:3300/0 conn(0x7faba00accd0 0x7faba006d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.689+0000 7fab9fe00640 1 -- 172.21.15.39:0/51610354 >> v1:172.21.15.100:6789/0 conn(0x7faba006c3a0 legacy=0x7faba00cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.689+0000 7fab9fe00640 1 -- 172.21.15.39:0/51610354 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba00b2650 con 0x7faba00cb540 2024-01-24T17:42:17.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.690+0000 7fab9fe00640 1 --2- 172.21.15.39:0/51610354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00cb540 0x7faba00ab030 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7faba006e060 tx=0x7fab9802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63621e22efca0ba8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:17.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.690+0000 7fab9e000640 1 -- 172.21.15.39:0/51610354 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9802cb10 con 0x7faba00cb540 2024-01-24T17:42:17.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.690+0000 7fab9e000640 1 -- 172.21.15.39:0/51610354 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9802cc70 con 0x7faba00cb540 2024-01-24T17:42:17.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.690+0000 7fab9e000640 1 -- 172.21.15.39:0/51610354 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab980343f0 con 0x7faba00cb540 2024-01-24T17:42:17.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.691+0000 7faba4e00640 1 -- 172.21.15.39:0/51610354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00cb540 msgr2=0x7faba00ab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.691+0000 7faba4e00640 1 --2- 172.21.15.39:0/51610354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00cb540 0x7faba00ab030 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7faba006e060 tx=0x7fab9802bf70 comp rx=0 tx=0).stop 2024-01-24T17:42:17.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.691+0000 7faba4e00640 1 -- 172.21.15.39:0/51610354 shutdown_connections 2024-01-24T17:42:17.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.691+0000 7faba4e00640 1 --2- 172.21.15.39:0/51610354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00cb540 0x7faba00ab030 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.691+0000 7faba4e00640 1 --2- 172.21.15.39:0/51610354 >> v2:172.21.15.100:3300/0 conn(0x7faba00accd0 0x7faba006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.691+0000 7faba4e00640 1 -- 172.21.15.39:0/51610354 >> 172.21.15.39:0/51610354 conn(0x7faba0069e50 msgr2=0x7faba006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:17.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.691+0000 7faba4e00640 1 -- 172.21.15.39:0/51610354 shutdown_connections 2024-01-24T17:42:17.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.691+0000 7faba4e00640 1 -- 172.21.15.39:0/51610354 wait complete. 2024-01-24T17:42:17.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.692+0000 7faba4e00640 1 Processor -- start 2024-01-24T17:42:17.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.693+0000 7faba4e00640 1 -- start start 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.693+0000 7faba4e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faba006c3a0 0x7faba00aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7faba4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00accd0 0x7faba00b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7faba4e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faba00cb540 0x7faba015ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7faba4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba01551f0 con 0x7faba00accd0 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7faba4e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba00af410 con 0x7faba00cb540 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7faba4e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba0154ec0 con 0x7faba006c3a0 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7fab9ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00accd0 0x7faba00b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7fab9ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00accd0 0x7faba00b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58592/0 (socket says 172.21.15.39:58592) 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7fab9ea00640 1 -- 172.21.15.39:0/729930838 learned_addr learned my addr 172.21.15.39:0/729930838 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7fab9ea00640 1 -- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faba006c3a0 msgr2=0x7faba00aeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7fab9ea00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faba006c3a0 0x7faba00aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7fab9ea00640 1 -- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faba00cb540 msgr2=0x7faba015ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7fab9ea00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faba00cb540 0x7faba015ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.694+0000 7fab9ea00640 1 -- 172.21.15.39:0/729930838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab98009b30 con 0x7faba00accd0 2024-01-24T17:42:17.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.695+0000 7fab9ea00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00accd0 0x7faba00b06d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab8800c400 tx=0x7fab8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:17.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.695+0000 7fab97400640 1 -- 172.21.15.39:0/729930838 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab880052d0 con 0x7faba00accd0 2024-01-24T17:42:17.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.695+0000 7fab97400640 1 -- 172.21.15.39:0/729930838 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab88005430 con 0x7faba00accd0 2024-01-24T17:42:17.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.695+0000 7fab97400640 1 -- 172.21.15.39:0/729930838 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88003830 con 0x7faba00accd0 2024-01-24T17:42:17.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.695+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba00afec0 con 0x7faba00accd0 2024-01-24T17:42:17.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.695+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba00af900 con 0x7faba00accd0 2024-01-24T17:42:17.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.696+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba0155330 con 0x7faba00accd0 2024-01-24T17:42:17.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.697+0000 7fab97400640 1 -- 172.21.15.39:0/729930838 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fab88003990 con 0x7faba00accd0 2024-01-24T17:42:17.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.698+0000 7fab97400640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab7806a430 0x7fab7806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:17.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.698+0000 7fab97400640 1 -- 172.21.15.39:0/729930838 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fab88087e50 con 0x7faba00accd0 2024-01-24T17:42:17.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.699+0000 7fab9f400640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab7806a430 0x7fab7806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:17.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.700+0000 7fab9f400640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab7806a430 0x7fab7806c8f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fab90006e20 tx=0x7fab90008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:17.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.703+0000 7fab97400640 1 -- 172.21.15.39:0/729930838 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fab880534b0 con 0x7faba00accd0 2024-01-24T17:42:17.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.897+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faba004dff0 con 0x7faba00accd0 2024-01-24T17:42:17.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.898+0000 7fab97400640 1 -- 172.21.15.39:0/729930838 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fab88017020 con 0x7faba00accd0 2024-01-24T17:42:17.900 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab7806a430 msgr2=0x7fab7806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab7806a430 0x7fab7806c8f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fab90006e20 tx=0x7fab90008040 comp rx=0 tx=0).stop 2024-01-24T17:42:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00accd0 msgr2=0x7faba00b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00accd0 0x7faba00b06d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab8800c400 tx=0x7fab8800c8c0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 shutdown_connections 2024-01-24T17:42:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faba00cb540 0x7faba015ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab7806a430 0x7fab7806c8f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faba00accd0 0x7faba00b06d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 --2- 172.21.15.39:0/729930838 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faba006c3a0 0x7faba00aeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:17.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 >> 172.21.15.39:0/729930838 conn(0x7faba0069e50 msgr2=0x7faba00ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:17.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 shutdown_connections 2024-01-24T17:42:17.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:17.901+0000 7faba4e00640 1 -- 172.21.15.39:0/729930838 wait complete. 2024-01-24T17:42:17.930 INFO:tasks.cephfs_test_runner:Starting test: test_mount_conn_close (tasks.cephfs.test_sessionmap.TestSessionMap) That when a client unmounts, the thread count on the MDS goes back 2024-01-24T17:42:17.930 INFO:tasks.cephfs_test_runner:test_mount_conn_close (tasks.cephfs.test_sessionmap.TestSessionMap) 2024-01-24T17:42:17.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close' 2024-01-24T17:42:18.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.150+0000 7f4c2dc00640 1 Processor -- start 2024-01-24T17:42:18.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.150+0000 7f4c2dc00640 1 -- start start 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.151+0000 7f4c2dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4c28153770 0x7f4c28160d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c2dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c280b1740 0x7f4c28070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c2dc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2815d7e0 con 0x7f4c2815cd90 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c2dc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2815dac0 con 0x7f4c28153770 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c2dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c28160500 con 0x7f4c280b1740 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c27e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4c28153770 0x7f4c28160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c27400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4c2815cd90 0x7f4c280b1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59188/0 (socket says 172.21.15.39:59188) 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c27400640 1 -- 172.21.15.39:0/2259824396 learned_addr learned my addr 172.21.15.39:0/2259824396 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c27e00640 1 -- 172.21.15.39:0/2259824396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c280b1740 msgr2=0x7f4c28070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c27e00640 1 --2- 172.21.15.39:0/2259824396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c280b1740 0x7f4c28070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:18.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c27e00640 1 -- 172.21.15.39:0/2259824396 >> v1:172.21.15.39:6789/0 conn(0x7f4c2815cd90 legacy=0x7f4c280b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:18.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c27e00640 1 -- 172.21.15.39:0/2259824396 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c281607e0 con 0x7f4c28153770 2024-01-24T17:42:18.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.152+0000 7f4c26a00640 1 -- 172.21.15.39:0/2259824396 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208058315 0 0) 0x7f4c2815d7e0 con 0x7f4c2815cd90 2024-01-24T17:42:18.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.153+0000 7f4c27e00640 1 --2- 172.21.15.39:0/2259824396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c28153770 0x7f4c28160d00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4c180097a0 tx=0x7f4c1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d092a9115712f2f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:18.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c26a00640 1 -- 172.21.15.39:0/2259824396 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18031830 con 0x7f4c28153770 2024-01-24T17:42:18.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c26a00640 1 -- 172.21.15.39:0/2259824396 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c18031990 con 0x7f4c28153770 2024-01-24T17:42:18.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c26a00640 1 -- 172.21.15.39:0/2259824396 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18031d20 con 0x7f4c28153770 2024-01-24T17:42:18.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c2dc00640 1 -- 172.21.15.39:0/2259824396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c28153770 msgr2=0x7f4c28160d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:18.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/2259824396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c28153770 0x7f4c28160d00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4c180097a0 tx=0x7f4c1802b8d0 comp rx=0 tx=0).stop 2024-01-24T17:42:18.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c2dc00640 1 -- 172.21.15.39:0/2259824396 shutdown_connections 2024-01-24T17:42:18.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/2259824396 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c280b1740 0x7f4c28070eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:18.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/2259824396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c28153770 0x7f4c28160d00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:18.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.154+0000 7f4c2dc00640 1 -- 172.21.15.39:0/2259824396 >> 172.21.15.39:0/2259824396 conn(0x7f4c280cf530 msgr2=0x7f4c280cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:18.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.155+0000 7f4c2dc00640 1 -- 172.21.15.39:0/2259824396 shutdown_connections 2024-01-24T17:42:18.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.155+0000 7f4c2dc00640 1 -- 172.21.15.39:0/2259824396 wait complete. 2024-01-24T17:42:18.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.156+0000 7f4c2dc00640 1 Processor -- start 2024-01-24T17:42:18.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.156+0000 7f4c2dc00640 1 -- start start 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.157+0000 7f4c2dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c280b1740 0x7f4c28152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.157+0000 7f4c2dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c28153770 0x7f4c28158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.157+0000 7f4c2dc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c2815cd90 0x7f4c28156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.157+0000 7f4c2dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c28152fd0 con 0x7f4c280b1740 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.157+0000 7f4c2dc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c281589c0 con 0x7f4c2815cd90 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.157+0000 7f4c2dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c281f2340 con 0x7f4c28153770 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c280b1740 0x7f4c28152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c280b1740 0x7f4c28152a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58606/0 (socket says 172.21.15.39:58606) 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 -- 172.21.15.39:0/1142714316 learned_addr learned my addr 172.21.15.39:0/1142714316 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27400640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c28153770 0x7f4c28158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 -- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c28153770 msgr2=0x7f4c28158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:18.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c28153770 0x7f4c28158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:18.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c2d200640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c2815cd90 0x7f4c28156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:18.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 -- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c2815cd90 msgr2=0x7f4c28156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:18.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c2815cd90 0x7f4c28156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:18.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 -- 172.21.15.39:0/1142714316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c18009400 con 0x7f4c280b1740 2024-01-24T17:42:18.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27400640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c28153770 0x7f4c28158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:18.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c2d200640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c2815cd90 0x7f4c28156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:18.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.158+0000 7f4c27e00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c280b1740 0x7f4c28152a90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4c180098c0 tx=0x7f4c180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:18.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.159+0000 7f4c1fe00640 1 -- 172.21.15.39:0/1142714316 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1803ba70 con 0x7f4c280b1740 2024-01-24T17:42:18.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.159+0000 7f4c1fe00640 1 -- 172.21.15.39:0/1142714316 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1803bbd0 con 0x7f4c280b1740 2024-01-24T17:42:18.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.159+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c281f2620 con 0x7f4c280b1740 2024-01-24T17:42:18.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.159+0000 7f4c1fe00640 1 -- 172.21.15.39:0/1142714316 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c180453f0 con 0x7f4c280b1740 2024-01-24T17:42:18.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.159+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c281f2be0 con 0x7f4c280b1740 2024-01-24T17:42:18.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.160+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c281f55c0 con 0x7f4c280b1740 2024-01-24T17:42:18.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.161+0000 7f4c1fe00640 1 -- 172.21.15.39:0/1142714316 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4c1804e450 con 0x7f4c280b1740 2024-01-24T17:42:18.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.162+0000 7f4c1fe00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c0006a430 0x7f4c0006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:18.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.162+0000 7f4c1fe00640 1 -- 172.21.15.39:0/1142714316 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4c180aabf0 con 0x7f4c280b1740 2024-01-24T17:42:18.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.162+0000 7f4c27400640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c0006a430 0x7f4c0006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:18.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.164+0000 7f4c27400640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c0006a430 0x7f4c0006c8f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4c281f5520 tx=0x7f4c10009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:18.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.166+0000 7f4c1fe00640 1 -- 172.21.15.39:0/1142714316 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4c281f55c0 con 0x7f4c280b1740 2024-01-24T17:42:18.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:18.434+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close"]} v 0) v1 -- 0x7f4c281f55c0 con 0x7f4c280b1740 2024-01-24T17:42:19.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.333+0000 7f4c1fe00640 1 -- 172.21.15.39:0/1142714316 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close"]}]=0 v60) v1 ==== 147+0+0 (secure 0 0 0) 0x7f4c281f55c0 con 0x7f4c280b1740 2024-01-24T17:42:19.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.336+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c0006a430 msgr2=0x7f4c0006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.336+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c0006a430 0x7f4c0006c8f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4c281f5520 tx=0x7f4c10009040 comp rx=0 tx=0).stop 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.336+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c280b1740 msgr2=0x7f4c28152a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.336+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c280b1740 0x7f4c28152a90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4c180098c0 tx=0x7f4c180046c0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.337+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 shutdown_connections 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.337+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c2815cd90 0x7f4c28156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.337+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c28153770 0x7f4c28158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.337+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c0006a430 0x7f4c0006c8f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.337+0000 7f4c2dc00640 1 --2- 172.21.15.39:0/1142714316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c280b1740 0x7f4c28152a90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.337+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 >> 172.21.15.39:0/1142714316 conn(0x7f4c280cf530 msgr2=0x7f4c280b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.337+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 shutdown_connections 2024-01-24T17:42:19.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.337+0000 7f4c2dc00640 1 -- 172.21.15.39:0/1142714316 wait complete. 2024-01-24T17:42:19.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-24T17:42:19.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.584+0000 7fb1eee00640 1 Processor -- start 2024-01-24T17:42:19.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.585+0000 7fb1eee00640 1 -- start start 2024-01-24T17:42:19.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.586+0000 7fb1eee00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb1e815c9e0 0x7fb1e80b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:19.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eee00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb1e80cdd60 0x7fb1e80d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:19.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eee00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80b7e30 con 0x7fb1e80b8250 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eee00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e80b6d90 con 0x7fb1e815c9e0 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eee00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e80b7070 con 0x7fb1e80cdd60 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eda00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb1e815c9e0 0x7fb1e80b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eda00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb1e815c9e0 0x7fb1e80b6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58622/0 (socket says 172.21.15.39:58622) 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eda00640 1 -- 172.21.15.39:0/3225798795 learned_addr learned my addr 172.21.15.39:0/3225798795 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eda00640 1 -- 172.21.15.39:0/3225798795 >> v2:172.21.15.100:3300/0 conn(0x7fb1e80cdd60 msgr2=0x7fb1e80d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eda00640 1 --2- 172.21.15.39:0/3225798795 >> v2:172.21.15.100:3300/0 conn(0x7fb1e80cdd60 0x7fb1e80d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eda00640 1 -- 172.21.15.39:0/3225798795 >> v1:172.21.15.100:6789/0 conn(0x7fb1e80b8250 legacy=0x7fb1e80cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:19.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.587+0000 7fb1eda00640 1 -- 172.21.15.39:0/3225798795 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80b7350 con 0x7fb1e815c9e0 2024-01-24T17:42:19.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.588+0000 7fb1eda00640 1 --2- 172.21.15.39:0/3225798795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e815c9e0 0x7fb1e80b6500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80d3500 tx=0x7fb1d802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9384b2d48be0d1c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.589+0000 7fb1e7e00640 1 -- 172.21.15.39:0/3225798795 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d802cb20 con 0x7fb1e815c9e0 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.589+0000 7fb1e7e00640 1 -- 172.21.15.39:0/3225798795 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d802cc80 con 0x7fb1e815c9e0 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.589+0000 7fb1e7e00640 1 -- 172.21.15.39:0/3225798795 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d80343f0 con 0x7fb1e815c9e0 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.590+0000 7fb1eee00640 1 -- 172.21.15.39:0/3225798795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e815c9e0 msgr2=0x7fb1e80b6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.590+0000 7fb1eee00640 1 --2- 172.21.15.39:0/3225798795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e815c9e0 0x7fb1e80b6500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80d3500 tx=0x7fb1d802bf80 comp rx=0 tx=0).stop 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.590+0000 7fb1eee00640 1 -- 172.21.15.39:0/3225798795 shutdown_connections 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.590+0000 7fb1eee00640 1 --2- 172.21.15.39:0/3225798795 >> v2:172.21.15.100:3300/0 conn(0x7fb1e80cdd60 0x7fb1e80d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.590+0000 7fb1eee00640 1 --2- 172.21.15.39:0/3225798795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e815c9e0 0x7fb1e80b6500 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.590+0000 7fb1eee00640 1 -- 172.21.15.39:0/3225798795 >> 172.21.15.39:0/3225798795 conn(0x7fb1e8071790 msgr2=0x7fb1e8071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.590+0000 7fb1eee00640 1 -- 172.21.15.39:0/3225798795 shutdown_connections 2024-01-24T17:42:19.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.590+0000 7fb1eee00640 1 -- 172.21.15.39:0/3225798795 wait complete. 2024-01-24T17:42:19.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.592+0000 7fb1eee00640 1 Processor -- start 2024-01-24T17:42:19.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.592+0000 7fb1eee00640 1 -- start start 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1eee00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e80b8250 0x7fb1e81f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1eee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e80cdd60 0x7fb1e815f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1eee00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e815c9e0 0x7fb1e8152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1eee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80cab10 con 0x7fb1e80cdd60 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1eee00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80a93d0 con 0x7fb1e80b8250 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1eee00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80cc1b0 con 0x7fb1e815c9e0 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1ed000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e80cdd60 0x7fb1e815f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1ed000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e80cdd60 0x7fb1e815f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58630/0 (socket says 172.21.15.39:58630) 2024-01-24T17:42:19.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.593+0000 7fb1ed000640 1 -- 172.21.15.39:0/1166271923 learned_addr learned my addr 172.21.15.39:0/1166271923 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1ed000640 1 -- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e815c9e0 msgr2=0x7fb1e8152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1ee400640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e815c9e0 0x7fb1e8152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1eda00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e80b8250 0x7fb1e81f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1ed000640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e815c9e0 0x7fb1e8152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1ed000640 1 -- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e80b8250 msgr2=0x7fb1e81f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1ed000640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e80b8250 0x7fb1e81f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1ed000640 1 -- 172.21.15.39:0/1166271923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d8009b30 con 0x7fb1e80cdd60 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1eda00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e80b8250 0x7fb1e81f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:19.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1ee400640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e815c9e0 0x7fb1e8152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:19.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1ed000640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e80cdd60 0x7fb1e815f570 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000b3f0 tx=0x7fb1d000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:19.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1e5600640 1 -- 172.21.15.39:0/1166271923 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0011830 con 0x7fb1e80cdd60 2024-01-24T17:42:19.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1e5600640 1 -- 172.21.15.39:0/1166271923 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d0011990 con 0x7fb1e80cdd60 2024-01-24T17:42:19.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1e5600640 1 -- 172.21.15.39:0/1166271923 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0011d20 con 0x7fb1e80cdd60 2024-01-24T17:42:19.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.594+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e815d150 con 0x7fb1e80cdd60 2024-01-24T17:42:19.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.595+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e815d9f0 con 0x7fb1e80cdd60 2024-01-24T17:42:19.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.596+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e8067040 con 0x7fb1e80cdd60 2024-01-24T17:42:19.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.597+0000 7fb1e5600640 1 -- 172.21.15.39:0/1166271923 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb1d001a400 con 0x7fb1e80cdd60 2024-01-24T17:42:19.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.598+0000 7fb1e5600640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb1c006a430 0x7fb1c006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:19.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.598+0000 7fb1e5600640 1 -- 172.21.15.39:0/1166271923 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb1d0088a40 con 0x7fb1e80cdd60 2024-01-24T17:42:19.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.598+0000 7fb1eda00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb1c006a430 0x7fb1c006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.599+0000 7fb1eda00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb1c006a430 0x7fb1c006c8f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8035000 tx=0x7fb1d8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.602+0000 7fb1e5600640 1 -- 172.21.15.39:0/1166271923 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb1d0053ff0 con 0x7fb1e80cdd60 2024-01-24T17:42:19.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.799+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb1e8158aa0 con 0x7fb1e80cdd60 2024-01-24T17:42:19.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.804+0000 7fb1e5600640 1 -- 172.21.15.39:0/1166271923 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb1d0057610 con 0x7fb1e80cdd60 2024-01-24T17:42:19.808 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb1c006a430 msgr2=0x7fb1c006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb1c006a430 0x7fb1c006c8f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8035000 tx=0x7fb1d8030000 comp rx=0 tx=0).stop 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e80cdd60 msgr2=0x7fb1e815f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e80cdd60 0x7fb1e815f570 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000b3f0 tx=0x7fb1d000b8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 shutdown_connections 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb1e815c9e0 0x7fb1e8152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb1c006a430 0x7fb1c006c8f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1e80cdd60 0x7fb1e815f570 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 --2- 172.21.15.39:0/1166271923 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1e80b8250 0x7fb1e81f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.807+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 >> 172.21.15.39:0/1166271923 conn(0x7fb1e8071790 msgr2=0x7fb1e80cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.808+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 shutdown_connections 2024-01-24T17:42:19.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:19.808+0000 7fb1eee00640 1 -- 172.21.15.39:0/1166271923 wait complete. 2024-01-24T17:42:19.816 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:42:19.830 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:42:19.830 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:42:19.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:42:20.104 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:42:20.104 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:42:20.104 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:20.154 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:42:20.154 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:20.245 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:20.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.470+0000 7f7027800640 1 Processor -- start 2024-01-24T17:42:20.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.471+0000 7f7027800640 1 -- start start 2024-01-24T17:42:20.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f7027800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f70200d0a30 0x7f702006ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f7027800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f702015ddd0 0x7f70200ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f7027800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702006b640 con 0x7f70200d0e10 2024-01-24T17:42:20.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f7027800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702006b920 con 0x7f702015ddd0 2024-01-24T17:42:20.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f7027800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702006bc00 con 0x7f70200d0a30 2024-01-24T17:42:20.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f701fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f702015ddd0 0x7f70200ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f701fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f702015ddd0 0x7f70200ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58640/0 (socket says 172.21.15.39:58640) 2024-01-24T17:42:20.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f701fe00640 1 -- 172.21.15.39:0/362849425 learned_addr learned my addr 172.21.15.39:0/362849425 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:20.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f701fe00640 1 -- 172.21.15.39:0/362849425 >> v2:172.21.15.100:3300/0 conn(0x7f70200d0a30 msgr2=0x7f702006ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:20.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f701fe00640 1 --2- 172.21.15.39:0/362849425 >> v2:172.21.15.100:3300/0 conn(0x7f70200d0a30 0x7f702006ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f701fe00640 1 -- 172.21.15.39:0/362849425 >> v1:172.21.15.100:6789/0 conn(0x7f70200d0e10 legacy=0x7f70200b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:20.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.472+0000 7f701fe00640 1 -- 172.21.15.39:0/362849425 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70200b08b0 con 0x7f702015ddd0 2024-01-24T17:42:20.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.473+0000 7f701fe00640 1 --2- 172.21.15.39:0/362849425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702015ddd0 0x7f70200ceb00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7008009870 tx=0x7f700802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254360d4400722cd server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:20.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f701f400640 1 -- 172.21.15.39:0/362849425 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008031750 con 0x7f702015ddd0 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f701f400640 1 -- 172.21.15.39:0/362849425 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70080318b0 con 0x7f702015ddd0 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f701f400640 1 -- 172.21.15.39:0/362849425 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008031c40 con 0x7f702015ddd0 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f7027800640 1 -- 172.21.15.39:0/362849425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702015ddd0 msgr2=0x7f70200ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f7027800640 1 --2- 172.21.15.39:0/362849425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702015ddd0 0x7f70200ceb00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7008009870 tx=0x7f700802b7f0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f7027800640 1 -- 172.21.15.39:0/362849425 shutdown_connections 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f7027800640 1 --2- 172.21.15.39:0/362849425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f702015ddd0 0x7f70200ceb00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f7027800640 1 --2- 172.21.15.39:0/362849425 >> v2:172.21.15.100:3300/0 conn(0x7f70200d0a30 0x7f702006ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.474+0000 7f7027800640 1 -- 172.21.15.39:0/362849425 >> 172.21.15.39:0/362849425 conn(0x7f7020089fd0 msgr2=0x7f7020099b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.475+0000 7f7027800640 1 -- 172.21.15.39:0/362849425 shutdown_connections 2024-01-24T17:42:20.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.475+0000 7f7027800640 1 -- 172.21.15.39:0/362849425 wait complete. 2024-01-24T17:42:20.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.476+0000 7f7027800640 1 Processor -- start 2024-01-24T17:42:20.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.476+0000 7f7027800640 1 -- start start 2024-01-24T17:42:20.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.477+0000 7f7027800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70200d0a30 0x7f7020157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.477+0000 7f7027800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70200d0e10 0x7f7020155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.477+0000 7f7027800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f702015ddd0 0x7f70200c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.477+0000 7f7027800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70200ca7d0 con 0x7f70200d0a30 2024-01-24T17:42:20.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.477+0000 7f7027800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70201559c0 con 0x7f702015ddd0 2024-01-24T17:42:20.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.477+0000 7f7027800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7020156e30 con 0x7f70200d0e10 2024-01-24T17:42:20.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.477+0000 7f7024e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70200d0a30 0x7f7020157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7024e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70200d0a30 0x7f7020157bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58648/0 (socket says 172.21.15.39:58648) 2024-01-24T17:42:20.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7024e00640 1 -- 172.21.15.39:0/277786297 learned_addr learned my addr 172.21.15.39:0/277786297 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7025800640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f702015ddd0 0x7f70200c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f701fe00640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70200d0e10 0x7f7020155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7024e00640 1 -- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70200d0e10 msgr2=0x7f7020155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7024e00640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70200d0e10 0x7f7020155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7024e00640 1 -- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f702015ddd0 msgr2=0x7f70200c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7024e00640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f702015ddd0 0x7f70200c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7024e00640 1 -- 172.21.15.39:0/277786297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70080094d0 con 0x7f70200d0a30 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7025800640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f702015ddd0 0x7f70200c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.478+0000 7f7024e00640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70200d0a30 0x7f7020157bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f701000b410 tx=0x7f701000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:20.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.479+0000 7f701cc00640 1 -- 172.21.15.39:0/277786297 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010011830 con 0x7f70200d0a30 2024-01-24T17:42:20.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.479+0000 7f701cc00640 1 -- 172.21.15.39:0/277786297 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70100119b0 con 0x7f70200d0a30 2024-01-24T17:42:20.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.479+0000 7f701cc00640 1 -- 172.21.15.39:0/277786297 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010011d60 con 0x7f70200d0a30 2024-01-24T17:42:20.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.479+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7020157110 con 0x7f70200d0a30 2024-01-24T17:42:20.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.479+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70201576d0 con 0x7f70200d0a30 2024-01-24T17:42:20.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.480+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7020067040 con 0x7f70200d0a30 2024-01-24T17:42:20.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.481+0000 7f701cc00640 1 -- 172.21.15.39:0/277786297 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f701001a400 con 0x7f70200d0a30 2024-01-24T17:42:20.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.482+0000 7f701cc00640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff806a430 0x7f6ff806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.482+0000 7f701cc00640 1 -- 172.21.15.39:0/277786297 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7010088b00 con 0x7f70200d0a30 2024-01-24T17:42:20.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.482+0000 7f701fe00640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff806a430 0x7f6ff806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.483+0000 7f701fe00640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff806a430 0x7f6ff806c8f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7008035000 tx=0x7f7008008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:20.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.486+0000 7f701cc00640 1 -- 172.21.15.39:0/277786297 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f70100540d0 con 0x7f70200d0a30 2024-01-24T17:42:20.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.679+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70200b0290 con 0x7f70200d0a30 2024-01-24T17:42:20.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.680+0000 7f701cc00640 1 -- 172.21.15.39:0/277786297 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f7010015020 con 0x7f70200d0a30 2024-01-24T17:42:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:20.682 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff806a430 msgr2=0x7f6ff806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff806a430 0x7f6ff806c8f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7008035000 tx=0x7f7008008030 comp rx=0 tx=0).stop 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70200d0a30 msgr2=0x7f7020157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70200d0a30 0x7f7020157bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f701000b410 tx=0x7f701000b8d0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 shutdown_connections 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f702015ddd0 0x7f70200c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70200d0e10 0x7f7020155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff806a430 0x7f6ff806c8f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 --2- 172.21.15.39:0/277786297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70200d0a30 0x7f7020157bf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 >> 172.21.15.39:0/277786297 conn(0x7f7020089fd0 msgr2=0x7f70201e7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:20.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 shutdown_connections 2024-01-24T17:42:20.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.683+0000 7f7027800640 1 -- 172.21.15.39:0/277786297 wait complete. 2024-01-24T17:42:20.703 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:20.703 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:20.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.906+0000 7f14d0600640 1 Processor -- start 2024-01-24T17:42:20.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.907+0000 7f14d0600640 1 -- start start 2024-01-24T17:42:20.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.908+0000 7f14d0600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 0x7f14c8065cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.908+0000 7f14d0600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f14c80cb2e0 0x7f14c80cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.908+0000 7f14d0600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f14c806c870 0x7f14c806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.908+0000 7f14d0600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c806d240 con 0x7f14c80cb2e0 2024-01-24T17:42:20.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.908+0000 7f14d0600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c80ca160 con 0x7f14c806c870 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.909+0000 7f14d0600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80cb100 con 0x7f14c81614c0 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.909+0000 7f14cd200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f14c80cb2e0 0x7f14c80cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.909+0000 7f14cd200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f14c80cb2e0 0x7f14c80cabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58650/0 (socket says 172.21.15.39:58650) 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.909+0000 7f14cd200640 1 -- 172.21.15.39:0/2079526327 learned_addr learned my addr 172.21.15.39:0/2079526327 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.909+0000 7f14cdc00640 1 --2- 172.21.15.39:0/2079526327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 0x7f14c8065cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.909+0000 7f14ce600640 1 --2- 172.21.15.39:0/2079526327 >> v2:172.21.15.100:3300/0 conn(0x7f14c806c870 0x7f14c806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.910+0000 7f14cd200640 1 -- 172.21.15.39:0/2079526327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 msgr2=0x7f14c8065cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.910+0000 7f14cd200640 1 --2- 172.21.15.39:0/2079526327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 0x7f14c8065cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.910+0000 7f14cd200640 1 -- 172.21.15.39:0/2079526327 >> v2:172.21.15.100:3300/0 conn(0x7f14c806c870 msgr2=0x7f14c806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.910+0000 7f14cd200640 1 --2- 172.21.15.39:0/2079526327 >> v2:172.21.15.100:3300/0 conn(0x7f14c806c870 0x7f14c806cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.910+0000 7f14cd200640 1 -- 172.21.15.39:0/2079526327 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c80ae540 con 0x7f14c80cb2e0 2024-01-24T17:42:20.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.911+0000 7f14cd200640 1 --2- 172.21.15.39:0/2079526327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c80cabc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f14c80ced90 tx=0x7f14b002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1def1b840232e083 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.911+0000 7f14c7e00640 1 -- 172.21.15.39:0/2079526327 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0031740 con 0x7f14c80cb2e0 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.911+0000 7f14c7e00640 1 -- 172.21.15.39:0/2079526327 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b00318a0 con 0x7f14c80cb2e0 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.911+0000 7f14c7e00640 1 -- 172.21.15.39:0/2079526327 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0031c30 con 0x7f14c80cb2e0 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 -- 172.21.15.39:0/2079526327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 msgr2=0x7f14c80cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 --2- 172.21.15.39:0/2079526327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c80cabc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f14c80ced90 tx=0x7f14b002b7e0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 -- 172.21.15.39:0/2079526327 shutdown_connections 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 --2- 172.21.15.39:0/2079526327 >> v2:172.21.15.100:3300/0 conn(0x7f14c806c870 0x7f14c806cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 --2- 172.21.15.39:0/2079526327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c80cabc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 --2- 172.21.15.39:0/2079526327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 0x7f14c8065cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 -- 172.21.15.39:0/2079526327 >> 172.21.15.39:0/2079526327 conn(0x7f14c80b0960 msgr2=0x7f14c80b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 -- 172.21.15.39:0/2079526327 shutdown_connections 2024-01-24T17:42:20.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.912+0000 7f14d0600640 1 -- 172.21.15.39:0/2079526327 wait complete. 2024-01-24T17:42:20.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.913+0000 7f14d0600640 1 Processor -- start 2024-01-24T17:42:20.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.914+0000 7f14d0600640 1 -- start start 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.914+0000 7f14d0600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c806c870 0x7f14c80ac360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.914+0000 7f14d0600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c8153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.914+0000 7f14d0600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 0x7f14c815b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.914+0000 7f14d0600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80c41d0 con 0x7f14c80cb2e0 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.914+0000 7f14d0600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80c44b0 con 0x7f14c806c870 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.914+0000 7f14d0600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80c4790 con 0x7f14c81614c0 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c8153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c8153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58658/0 (socket says 172.21.15.39:58658) 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 -- 172.21.15.39:0/2129894030 learned_addr learned my addr 172.21.15.39:0/2129894030 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cdc00640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c806c870 0x7f14c80ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 -- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 msgr2=0x7f14c815b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:20.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 0x7f14c815b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 -- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c806c870 msgr2=0x7f14c80ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:20.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c806c870 0x7f14c80ac360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:20.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 -- 172.21.15.39:0/2129894030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b00094d0 con 0x7f14c80cb2e0 2024-01-24T17:42:20.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.915+0000 7f14cd200640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c8153220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f14b0000c00 tx=0x7f14b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:20.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.916+0000 7f14c5600640 1 -- 172.21.15.39:0/2129894030 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0033c70 con 0x7f14c80cb2e0 2024-01-24T17:42:20.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.916+0000 7f14c5600640 1 -- 172.21.15.39:0/2129894030 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0033dd0 con 0x7f14c80cb2e0 2024-01-24T17:42:20.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.916+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c80c4a70 con 0x7f14c80cb2e0 2024-01-24T17:42:20.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.916+0000 7f14c5600640 1 -- 172.21.15.39:0/2129894030 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b003d690 con 0x7f14c80cb2e0 2024-01-24T17:42:20.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.916+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c80ac8a0 con 0x7f14c80cb2e0 2024-01-24T17:42:20.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.917+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c80722a0 con 0x7f14c80cb2e0 2024-01-24T17:42:20.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.918+0000 7f14c5600640 1 -- 172.21.15.39:0/2129894030 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f14b003a040 con 0x7f14c80cb2e0 2024-01-24T17:42:20.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.919+0000 7f14c5600640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14a006a430 0x7f14a006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:20.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.919+0000 7f14c5600640 1 -- 172.21.15.39:0/2129894030 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f14b00aa7b0 con 0x7f14c80cb2e0 2024-01-24T17:42:20.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.920+0000 7f14cdc00640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14a006a430 0x7f14a006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:20.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.921+0000 7f14cdc00640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14a006a430 0x7f14a006c8f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f14b8006fd0 tx=0x7f14b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:20.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:20.923+0000 7f14c5600640 1 -- 172.21.15.39:0/2129894030 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f14b0075e90 con 0x7f14c80cb2e0 2024-01-24T17:42:21.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.114+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14c8067040 con 0x7f14c80cb2e0 2024-01-24T17:42:21.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.115+0000 7f14c5600640 1 -- 172.21.15.39:0/2129894030 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14b0038020 con 0x7f14c80cb2e0 2024-01-24T17:42:21.117 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:21.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14a006a430 msgr2=0x7f14a006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14a006a430 0x7f14a006c8f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f14b8006fd0 tx=0x7f14b8008040 comp rx=0 tx=0).stop 2024-01-24T17:42:21.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 msgr2=0x7f14c8153220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c8153220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f14b0000c00 tx=0x7f14b00046c0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 shutdown_connections 2024-01-24T17:42:21.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14c81614c0 0x7f14c815b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14a006a430 0x7f14a006c8f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14c80cb2e0 0x7f14c8153220 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 --2- 172.21.15.39:0/2129894030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14c806c870 0x7f14c80ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 >> 172.21.15.39:0/2129894030 conn(0x7f14c80b0960 msgr2=0x7f14c815f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 shutdown_connections 2024-01-24T17:42:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.118+0000 7f14d0600640 1 -- 172.21.15.39:0/2129894030 wait complete. 2024-01-24T17:42:21.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:21.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.328+0000 7f8d41200640 1 Processor -- start 2024-01-24T17:42:21.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.329+0000 7f8d41200640 1 -- start start 2024-01-24T17:42:21.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.330+0000 7f8d41200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8d3c15c8d0 0x7f8d3c0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.330+0000 7f8d41200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c0b2710 0x7f8d3c0d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.330+0000 7f8d41200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0b38a0 con 0x7f8d3c0d2cb0 2024-01-24T17:42:21.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.330+0000 7f8d41200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c0d3140 con 0x7f8d3c15c8d0 2024-01-24T17:42:21.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.330+0000 7f8d41200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0d13e0 con 0x7f8d3c0b2710 2024-01-24T17:42:21.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8d3c15c8d0 0x7f8d3c0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8d3c15c8d0 0x7f8d3c0d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58662/0 (socket says 172.21.15.39:58662) 2024-01-24T17:42:21.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3b400640 1 -- 172.21.15.39:0/2802638795 learned_addr learned my addr 172.21.15.39:0/2802638795 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:21.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3aa00640 1 --2- 172.21.15.39:0/2802638795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c0b2710 0x7f8d3c0d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3b400640 1 -- 172.21.15.39:0/2802638795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c0b2710 msgr2=0x7f8d3c0d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:21.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3b400640 1 --2- 172.21.15.39:0/2802638795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c0b2710 0x7f8d3c0d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3b400640 1 -- 172.21.15.39:0/2802638795 >> v1:172.21.15.100:6789/0 conn(0x7f8d3c0d2cb0 legacy=0x7f8d3c0cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:21.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3b400640 1 -- 172.21.15.39:0/2802638795 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c0d0ab0 con 0x7f8d3c15c8d0 2024-01-24T17:42:21.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.331+0000 7f8d3aa00640 1 --2- 172.21.15.39:0/2802638795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c0b2710 0x7f8d3c0d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:21.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.332+0000 7f8d3b400640 1 --2- 172.21.15.39:0/2802638795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c15c8d0 0x7f8d3c0d03b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0098a0 tx=0x7f8d2c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34819c61007d9032 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:21.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d3a000640 1 -- 172.21.15.39:0/2802638795 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c031640 con 0x7f8d3c15c8d0 2024-01-24T17:42:21.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d3a000640 1 -- 172.21.15.39:0/2802638795 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c0317a0 con 0x7f8d3c15c8d0 2024-01-24T17:42:21.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d3a000640 1 -- 172.21.15.39:0/2802638795 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c031b30 con 0x7f8d3c15c8d0 2024-01-24T17:42:21.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d41200640 1 -- 172.21.15.39:0/2802638795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c15c8d0 msgr2=0x7f8d3c0d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d41200640 1 --2- 172.21.15.39:0/2802638795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c15c8d0 0x7f8d3c0d03b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0098a0 tx=0x7f8d2c02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d41200640 1 -- 172.21.15.39:0/2802638795 shutdown_connections 2024-01-24T17:42:21.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d41200640 1 --2- 172.21.15.39:0/2802638795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c0b2710 0x7f8d3c0d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d41200640 1 --2- 172.21.15.39:0/2802638795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c15c8d0 0x7f8d3c0d03b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d41200640 1 -- 172.21.15.39:0/2802638795 >> 172.21.15.39:0/2802638795 conn(0x7f8d3c071dd0 msgr2=0x7f8d3c09bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:21.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d41200640 1 -- 172.21.15.39:0/2802638795 shutdown_connections 2024-01-24T17:42:21.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.333+0000 7f8d41200640 1 -- 172.21.15.39:0/2802638795 wait complete. 2024-01-24T17:42:21.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.335+0000 7f8d41200640 1 Processor -- start 2024-01-24T17:42:21.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.336+0000 7f8d41200640 1 -- start start 2024-01-24T17:42:21.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d41200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c0b2710 0x7f8d3c155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d41200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d3c0d2cb0 0x7f8d3c1f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d41200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c15c8d0 0x7f8d3c0cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d41200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c15e080 con 0x7f8d3c0b2710 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d41200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c15e320 con 0x7f8d3c0d2cb0 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d41200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c15e600 con 0x7f8d3c15c8d0 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c0b2710 0x7f8d3c155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c0b2710 0x7f8d3c155260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58678/0 (socket says 172.21.15.39:58678) 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3b400640 1 -- 172.21.15.39:0/865516277 learned_addr learned my addr 172.21.15.39:0/865516277 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3aa00640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d3c0d2cb0 0x7f8d3c1f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3be00640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c15c8d0 0x7f8d3c0cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3b400640 1 -- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c15c8d0 msgr2=0x7f8d3c0cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3b400640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c15c8d0 0x7f8d3c0cc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3b400640 1 -- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d3c0d2cb0 msgr2=0x7f8d3c1f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3b400640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d3c0d2cb0 0x7f8d3c1f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.337+0000 7f8d3b400640 1 -- 172.21.15.39:0/865516277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c0094d0 con 0x7f8d3c0b2710 2024-01-24T17:42:21.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.338+0000 7f8d3b400640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c0b2710 0x7f8d3c155260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0098a0 tx=0x7f8d2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.343+0000 7f8d33400640 1 -- 172.21.15.39:0/865516277 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c03ac60 con 0x7f8d3c0b2710 2024-01-24T17:42:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.343+0000 7f8d33400640 1 -- 172.21.15.39:0/865516277 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c03ade0 con 0x7f8d3c0b2710 2024-01-24T17:42:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.343+0000 7f8d33400640 1 -- 172.21.15.39:0/865516277 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c044630 con 0x7f8d3c0b2710 2024-01-24T17:42:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.343+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c15e8e0 con 0x7f8d3c0b2710 2024-01-24T17:42:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.343+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c15eea0 con 0x7f8d3c0b2710 2024-01-24T17:42:21.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.344+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c15e740 con 0x7f8d3c0b2710 2024-01-24T17:42:21.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.345+0000 7f8d33400640 1 -- 172.21.15.39:0/865516277 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8d2c02f050 con 0x7f8d3c0b2710 2024-01-24T17:42:21.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.346+0000 7f8d33400640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d1406a430 0x7f8d1406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.346+0000 7f8d33400640 1 -- 172.21.15.39:0/865516277 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8d2c0aa9c0 con 0x7f8d3c0b2710 2024-01-24T17:42:21.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.347+0000 7f8d3aa00640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d1406a430 0x7f8d1406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.348+0000 7f8d3aa00640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d1406a430 0x7f8d1406c8f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8d240075b0 tx=0x7f8d24007540 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:21.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.350+0000 7f8d33400640 1 -- 172.21.15.39:0/865516277 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8d3c15e740 con 0x7f8d3c0b2710 2024-01-24T17:42:21.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.549+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8d3c15e740 con 0x7f8d3c0b2710 2024-01-24T17:42:21.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.549+0000 7f8d33400640 1 -- 172.21.15.39:0/865516277 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8d3c15e740 con 0x7f8d3c0b2710 2024-01-24T17:42:21.551 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d1406a430 msgr2=0x7f8d1406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d1406a430 0x7f8d1406c8f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8d240075b0 tx=0x7f8d24007540 comp rx=0 tx=0).stop 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c0b2710 msgr2=0x7f8d3c155260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c0b2710 0x7f8d3c155260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0098a0 tx=0x7f8d2c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 shutdown_connections 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8d3c15c8d0 0x7f8d3c0cc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d3c0d2cb0 0x7f8d3c1f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d1406a430 0x7f8d1406c8f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 --2- 172.21.15.39:0/865516277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d3c0b2710 0x7f8d3c155260 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 >> 172.21.15.39:0/865516277 conn(0x7f8d3c071dd0 msgr2=0x7f8d3c0cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:21.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 shutdown_connections 2024-01-24T17:42:21.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.552+0000 7f8d41200640 1 -- 172.21.15.39:0/865516277 wait complete. 2024-01-24T17:42:21.572 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:21.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.778+0000 7f70f6600640 1 Processor -- start 2024-01-24T17:42:21.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.779+0000 7f70f6600640 1 -- start start 2024-01-24T17:42:21.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.780+0000 7f70f6600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f00d2e80 0x7f70f00d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.780+0000 7f70f6600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f00b38a0 con 0x7f70f00b2710 2024-01-24T17:42:21.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.780+0000 7f70f6600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f00d13e0 con 0x7f70f015c8d0 2024-01-24T17:42:21.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.780+0000 7f70f6600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00d0ab0 con 0x7f70f00d2e80 2024-01-24T17:42:21.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.780+0000 7f70f5c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f70f00b2710 0x7f70f00cd630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59192/0 (socket says 172.21.15.39:59192) 2024-01-24T17:42:21.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.780+0000 7f70f5c00640 1 -- 172.21.15.39:0/313611459 learned_addr learned my addr 172.21.15.39:0/313611459 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70efe00640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f00d2e80 0x7f70f00d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760599836 0 0) 0x7f70f00b38a0 con 0x7f70f00b2710 2024-01-24T17:42:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d0003520 con 0x7f70f00b2710 2024-01-24T17:42:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022462088 0 0) 0x7f70d0003520 con 0x7f70f00b2710 2024-01-24T17:42:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f00d2e80 msgr2=0x7f70f00d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f00d2e80 0x7f70f00d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 >> v1:172.21.15.100:6789/0 conn(0x7f70f015c8d0 legacy=0x7f70f00d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f00d0d90 con 0x7f70f00b2710 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f70e80019f0 con 0x7f70f00b2710 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 >> v1:172.21.15.39:6789/0 conn(0x7f70f00b2710 legacy=0x7f70f00cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70d00039a0 0x7f70d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.781+0000 7f70ef400640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70ef400640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f00d2e80 0x7f70d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00d0ab0 con 0x7f70d00047c0 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d000ba30 con 0x7f70d00039a0 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d000bbf0 con 0x7f70f00d2e80 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70efe00640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70f5c00640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f00d2e80 0x7f70d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70f5200640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70d00039a0 0x7f70d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70efe00640 1 -- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f00d2e80 msgr2=0x7f70d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70efe00640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f00d2e80 0x7f70d0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70efe00640 1 -- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70d00039a0 msgr2=0x7f70d0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70efe00640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70d00039a0 0x7f70d0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70efe00640 1 -- 172.21.15.39:0/313611459 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d0003520 con 0x7f70d00047c0 2024-01-24T17:42:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.782+0000 7f70f5200640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70d00039a0 0x7f70d0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.783+0000 7f70efe00640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70d0004c40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009a90 tx=0x7f70d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7863fb87f69f110c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.783+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d802c680 con 0x7f70d00047c0 2024-01-24T17:42:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.783+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d802c7e0 con 0x7f70d00047c0 2024-01-24T17:42:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.783+0000 7f70ef400640 1 -- 172.21.15.39:0/313611459 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d802cb70 con 0x7f70d00047c0 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.784+0000 7f70f6600640 1 -- 172.21.15.39:0/313611459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 msgr2=0x7f70d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.784+0000 7f70f6600640 1 --2- 172.21.15.39:0/313611459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70d0004c40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009a90 tx=0x7f70d00057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.784+0000 7f70f5c00640 1 -- 172.21.15.39:0/313611459 reap_dead start 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.784+0000 7f70f6600640 1 -- 172.21.15.39:0/313611459 shutdown_connections 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.784+0000 7f70f6600640 1 -- 172.21.15.39:0/313611459 >> 172.21.15.39:0/313611459 conn(0x7f70f0071dd0 msgr2=0x7f70f009bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.784+0000 7f70f6600640 1 -- 172.21.15.39:0/313611459 shutdown_connections 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.784+0000 7f70f6600640 1 -- 172.21.15.39:0/313611459 wait complete. 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.786+0000 7f70f6600640 1 Processor -- start 2024-01-24T17:42:21.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.786+0000 7f70f6600640 1 -- start start 2024-01-24T17:42:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f6600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70f0153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f6600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70f015f660 0x7f70f0154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f6600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f01518d0 0x7f70f0151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f6600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0154450 con 0x7f70d00047c0 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f6600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f01522a0 con 0x7f70f015f660 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f6600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0156a90 con 0x7f70f01518d0 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70f0153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70f0153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58694/0 (socket says 172.21.15.39:58694) 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.787+0000 7f70f5200640 1 -- 172.21.15.39:0/4221300527 learned_addr learned my addr 172.21.15.39:0/4221300527 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70efe00640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70f015f660 0x7f70f0154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70f5200640 1 -- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f01518d0 msgr2=0x7f70f0151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70f5200640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f01518d0 0x7f70f0151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70f5200640 1 -- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70f015f660 msgr2=0x7f70f0154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70f5200640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70f015f660 0x7f70f0154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70f5200640 1 -- 172.21.15.39:0/4221300527 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0153bb0 con 0x7f70d00047c0 2024-01-24T17:42:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70f5200640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70f0153260 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f70e000d7a0 tx=0x7f70e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:21.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70ecc00640 1 -- 172.21.15.39:0/4221300527 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0017070 con 0x7f70d00047c0 2024-01-24T17:42:21.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70ecc00640 1 -- 172.21.15.39:0/4221300527 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e0005070 con 0x7f70d00047c0 2024-01-24T17:42:21.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70ecc00640 1 -- 172.21.15.39:0/4221300527 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0005420 con 0x7f70d00047c0 2024-01-24T17:42:21.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.788+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f0153e90 con 0x7f70d00047c0 2024-01-24T17:42:21.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.789+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f0156d30 con 0x7f70d00047c0 2024-01-24T17:42:21.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.790+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f004dff0 con 0x7f70d00047c0 2024-01-24T17:42:21.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.791+0000 7f70ecc00640 1 -- 172.21.15.39:0/4221300527 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f70e0007030 con 0x7f70d00047c0 2024-01-24T17:42:21.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.791+0000 7f70ecc00640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70c806a430 0x7f70c806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:21.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.791+0000 7f70ecc00640 1 -- 172.21.15.39:0/4221300527 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f70e0088200 con 0x7f70d00047c0 2024-01-24T17:42:21.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.792+0000 7f70efe00640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70c806a430 0x7f70c806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:21.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.793+0000 7f70efe00640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70c806a430 0x7f70c806c8f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f70d8036000 tx=0x7f70d8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:21.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.796+0000 7f70ecc00640 1 -- 172.21.15.39:0/4221300527 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f70e00537d0 con 0x7f70d00047c0 2024-01-24T17:42:21.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.982+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f70f015aca0 con 0x7f70d00047c0 2024-01-24T17:42:21.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.985+0000 7f70ecc00640 1 -- 172.21.15.39:0/4221300527 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f70e0056df0 con 0x7f70d00047c0 2024-01-24T17:42:21.985 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:21.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70c806a430 msgr2=0x7f70c806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70c806a430 0x7f70c806c8f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f70d8036000 tx=0x7f70d8030000 comp rx=0 tx=0).stop 2024-01-24T17:42:21.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 msgr2=0x7f70f0153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:21.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70f0153260 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f70e000d7a0 tx=0x7f70e000dc60 comp rx=0 tx=0).stop 2024-01-24T17:42:21.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 shutdown_connections 2024-01-24T17:42:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70f01518d0 0x7f70f0151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70f015f660 0x7f70f0154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70c806a430 0x7f70c806c8f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 --2- 172.21.15.39:0/4221300527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70d00047c0 0x7f70f0153260 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 >> 172.21.15.39:0/4221300527 conn(0x7f70f0071dd0 msgr2=0x7f70f00b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 shutdown_connections 2024-01-24T17:42:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:21.987+0000 7f70f6600640 1 -- 172.21.15.39:0/4221300527 wait complete. 2024-01-24T17:42:22.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:42:22.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.215+0000 7f0ca0200640 1 Processor -- start 2024-01-24T17:42:22.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.216+0000 7f0ca0200640 1 -- start start 2024-01-24T17:42:22.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.217+0000 7f0ca0200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c980cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.217+0000 7f0ca0200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0c9815fb90 0x7f0c9815d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.217+0000 7f0ca0200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c980d02b0 con 0x7f0c9815d020 2024-01-24T17:42:22.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.217+0000 7f0ca0200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9815e080 con 0x7f0c9815fb90 2024-01-24T17:42:22.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.217+0000 7f0ca0200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9815dac0 con 0x7f0c980cf990 2024-01-24T17:42:22.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9ce00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0c9815d020 0x7f0c98160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59202/0 (socket says 172.21.15.39:59202) 2024-01-24T17:42:22.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9ce00640 1 -- 172.21.15.39:0/556617781 learned_addr learned my addr 172.21.15.39:0/556617781 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9e200640 1 --2- 172.21.15.39:0/556617781 >> v2:172.21.15.39:3300/0 conn(0x7f0c9815fb90 0x7f0c9815d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9d800640 1 --2- 172.21.15.39:0/556617781 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c980cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c97e00640 1 -- 172.21.15.39:0/556617781 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664132388 0 0) 0x7f0c980d02b0 con 0x7f0c9815d020 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c97e00640 1 -- 172.21.15.39:0/556617781 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c78003520 con 0x7f0c9815d020 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9e200640 1 -- 172.21.15.39:0/556617781 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 msgr2=0x7f0c980cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9e200640 1 --2- 172.21.15.39:0/556617781 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c980cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9e200640 1 -- 172.21.15.39:0/556617781 >> v1:172.21.15.39:6789/0 conn(0x7f0c9815d020 legacy=0x7f0c98160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9e200640 1 -- 172.21.15.39:0/556617781 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9815dda0 con 0x7f0c9815fb90 2024-01-24T17:42:22.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.218+0000 7f0c9d800640 1 --2- 172.21.15.39:0/556617781 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c980cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:22.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0c9e200640 1 --2- 172.21.15.39:0/556617781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c9815d460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009d90 tx=0x7f0c9002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87eb24331d80e6b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:22.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0c97e00640 1 -- 172.21.15.39:0/556617781 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9002cc50 con 0x7f0c9815fb90 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0c97e00640 1 -- 172.21.15.39:0/556617781 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9002cdb0 con 0x7f0c9815fb90 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0c97e00640 1 -- 172.21.15.39:0/556617781 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90034660 con 0x7f0c9815fb90 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0ca0200640 1 -- 172.21.15.39:0/556617781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 msgr2=0x7f0c9815d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0ca0200640 1 --2- 172.21.15.39:0/556617781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c9815d460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009d90 tx=0x7f0c9002bea0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0ca0200640 1 -- 172.21.15.39:0/556617781 shutdown_connections 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0ca0200640 1 --2- 172.21.15.39:0/556617781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c9815d460 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0ca0200640 1 --2- 172.21.15.39:0/556617781 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c980cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.220+0000 7f0ca0200640 1 -- 172.21.15.39:0/556617781 >> 172.21.15.39:0/556617781 conn(0x7f0c98071790 msgr2=0x7f0c98071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.221+0000 7f0ca0200640 1 -- 172.21.15.39:0/556617781 shutdown_connections 2024-01-24T17:42:22.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.221+0000 7f0ca0200640 1 -- 172.21.15.39:0/556617781 wait complete. 2024-01-24T17:42:22.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.222+0000 7f0ca0200640 1 Processor -- start 2024-01-24T17:42:22.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.223+0000 7f0ca0200640 1 -- start start 2024-01-24T17:42:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.223+0000 7f0ca0200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c98156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.223+0000 7f0ca0200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0c9815d020 0x7f0c980d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.223+0000 7f0ca0200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c98158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0ca0200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980d1b70 con 0x7f0c9815fb90 2024-01-24T17:42:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0ca0200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980d15b0 con 0x7f0c9815d020 2024-01-24T17:42:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0ca0200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980d1890 con 0x7f0c980cf990 2024-01-24T17:42:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c98158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c98158800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58700/0 (socket says 172.21.15.39:58700) 2024-01-24T17:42:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9e200640 1 -- 172.21.15.39:0/2520104136 learned_addr learned my addr 172.21.15.39:0/2520104136 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9d800640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c98156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9e200640 1 -- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 msgr2=0x7f0c98156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9ce00640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0c9815d020 0x7f0c980d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9e200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c98156ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9e200640 1 -- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0c9815d020 msgr2=0x7f0c980d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9e200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0c9815d020 0x7f0c980d0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.224+0000 7f0c9e200640 1 -- 172.21.15.39:0/2520104136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c90009a70 con 0x7f0c9815fb90 2024-01-24T17:42:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.225+0000 7f0c9e200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c98158800 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009d90 tx=0x7f0c90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.225+0000 7f0c95600640 1 -- 172.21.15.39:0/2520104136 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90040650 con 0x7f0c9815fb90 2024-01-24T17:42:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.225+0000 7f0c95600640 1 -- 172.21.15.39:0/2520104136 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c900407b0 con 0x7f0c9815fb90 2024-01-24T17:42:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.225+0000 7f0c95600640 1 -- 172.21.15.39:0/2520104136 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90040b60 con 0x7f0c9815fb90 2024-01-24T17:42:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.225+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c980c7800 con 0x7f0c9815fb90 2024-01-24T17:42:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.225+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c980d3290 con 0x7f0c9815fb90 2024-01-24T17:42:22.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.226+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c980c7660 con 0x7f0c9815fb90 2024-01-24T17:42:22.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.227+0000 7f0c95600640 1 -- 172.21.15.39:0/2520104136 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0c90040cc0 con 0x7f0c9815fb90 2024-01-24T17:42:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.228+0000 7f0c95600640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0c7006a430 0x7f0c7006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.228+0000 7f0c95600640 1 -- 172.21.15.39:0/2520104136 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0c900ab7d0 con 0x7f0c9815fb90 2024-01-24T17:42:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.229+0000 7f0c9d800640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0c7006a430 0x7f0c7006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.230+0000 7f0c9d800640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0c7006a430 0x7f0c7006c8f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c880096d0 tx=0x7f0c88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.232+0000 7f0c95600640 1 -- 172.21.15.39:0/2520104136 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0c980c7660 con 0x7f0c9815fb90 2024-01-24T17:42:22.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.436+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0c980c7d20 con 0x7f0c9815fb90 2024-01-24T17:42:22.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.437+0000 7f0c95600640 1 -- 172.21.15.39:0/2520104136 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0c980c7d20 con 0x7f0c9815fb90 2024-01-24T17:42:22.438 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:42:22.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.439+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0c7006a430 msgr2=0x7f0c7006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.439+0000 7f0ca0200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0c7006a430 0x7f0c7006c8f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c880096d0 tx=0x7f0c88008040 comp rx=0 tx=0).stop 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.439+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 msgr2=0x7f0c98158800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.439+0000 7f0ca0200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c98158800 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009d90 tx=0x7f0c90004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.440+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 shutdown_connections 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.440+0000 7f0ca0200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0c7006a430 0x7f0c7006c8f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.440+0000 7f0ca0200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c9815fb90 0x7f0c98158800 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.440+0000 7f0ca0200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0c9815d020 0x7f0c980d0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.440+0000 7f0ca0200640 1 --2- 172.21.15.39:0/2520104136 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0c980cf990 0x7f0c98156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.440+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 >> 172.21.15.39:0/2520104136 conn(0x7f0c98071790 msgr2=0x7f0c980b3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.440+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 shutdown_connections 2024-01-24T17:42:22.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.440+0000 7f0ca0200640 1 -- 172.21.15.39:0/2520104136 wait complete. 2024-01-24T17:42:22.457 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:42:22.457 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:22.504 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:22.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.768+0000 7f5d1be00640 1 Processor -- start 2024-01-24T17:42:22.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.768+0000 7f5d1be00640 1 -- start start 2024-01-24T17:42:22.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d1be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5d14159b40 0x7f5d140cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d1be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5d1413f9d0 0x7f5d140713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d1be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d14070f30 con 0x7f5d140aff10 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d1be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1415ac00 con 0x7f5d14159b40 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d1be00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1415a080 con 0x7f5d1413f9d0 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d18a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5d14159b40 0x7f5d140cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d19400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f5d140aff10 0x7f5d140b02f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59210/0 (socket says 172.21.15.39:59210) 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d18a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5d14159b40 0x7f5d140cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58702/0 (socket says 172.21.15.39:58702) 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.769+0000 7f5d19400640 1 -- 172.21.15.39:0/3120468236 learned_addr learned my addr 172.21.15.39:0/3120468236 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.770+0000 7f5d19e00640 1 --2- 172.21.15.39:0/3120468236 >> v2:172.21.15.100:3300/0 conn(0x7f5d1413f9d0 0x7f5d140713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.770+0000 7f5d18a00640 1 -- 172.21.15.39:0/3120468236 >> v2:172.21.15.100:3300/0 conn(0x7f5d1413f9d0 msgr2=0x7f5d140713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.770+0000 7f5d18a00640 1 --2- 172.21.15.39:0/3120468236 >> v2:172.21.15.100:3300/0 conn(0x7f5d1413f9d0 0x7f5d140713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.770+0000 7f5d18a00640 1 -- 172.21.15.39:0/3120468236 >> v1:172.21.15.39:6789/0 conn(0x7f5d140aff10 legacy=0x7f5d140b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.770+0000 7f5d18a00640 1 -- 172.21.15.39:0/3120468236 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1415a360 con 0x7f5d14159b40 2024-01-24T17:42:22.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.770+0000 7f5d13e00640 1 -- 172.21.15.39:0/3120468236 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029519572 0 0) 0x7f5d14070f30 con 0x7f5d140aff10 2024-01-24T17:42:22.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.771+0000 7f5d18a00640 1 --2- 172.21.15.39:0/3120468236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140cc060 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5d1415d190 tx=0x7f5cfc02e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=681e457d1650c02d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:22.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.771+0000 7f5d13e00640 1 -- 172.21.15.39:0/3120468236 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc02ee50 con 0x7f5d14159b40 2024-01-24T17:42:22.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.771+0000 7f5d13e00640 1 -- 172.21.15.39:0/3120468236 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc036440 con 0x7f5d14159b40 2024-01-24T17:42:22.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.771+0000 7f5d13e00640 1 -- 172.21.15.39:0/3120468236 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc0367d0 con 0x7f5d14159b40 2024-01-24T17:42:22.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.772+0000 7f5d1be00640 1 -- 172.21.15.39:0/3120468236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 msgr2=0x7f5d140cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.772+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3120468236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140cc060 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5d1415d190 tx=0x7f5cfc02e0a0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.772+0000 7f5d1be00640 1 -- 172.21.15.39:0/3120468236 shutdown_connections 2024-01-24T17:42:22.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.772+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3120468236 >> v2:172.21.15.100:3300/0 conn(0x7f5d1413f9d0 0x7f5d140713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.772+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3120468236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140cc060 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.772+0000 7f5d1be00640 1 -- 172.21.15.39:0/3120468236 >> 172.21.15.39:0/3120468236 conn(0x7f5d1406a2a0 msgr2=0x7f5d1406bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:22.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.772+0000 7f5d1be00640 1 -- 172.21.15.39:0/3120468236 shutdown_connections 2024-01-24T17:42:22.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.772+0000 7f5d1be00640 1 -- 172.21.15.39:0/3120468236 wait complete. 2024-01-24T17:42:22.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.774+0000 7f5d1be00640 1 Processor -- start 2024-01-24T17:42:22.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.774+0000 7f5d1be00640 1 -- start start 2024-01-24T17:42:22.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d1be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d140aff10 0x7f5d140ad090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d1be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5d1413f9d0 0x7f5d140c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d1be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d1be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140c5dc0 con 0x7f5d14159b40 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d1be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140acc00 con 0x7f5d140aff10 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d1be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140c5560 con 0x7f5d1413f9d0 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d19e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d19e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58714/0 (socket says 172.21.15.39:58714) 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d19e00640 1 -- 172.21.15.39:0/3329648576 learned_addr learned my addr 172.21.15.39:0/3329648576 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d19400640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d140aff10 0x7f5d140ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.775+0000 7f5d18a00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5d1413f9d0 0x7f5d140c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d19e00640 1 -- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5d1413f9d0 msgr2=0x7f5d140c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d19e00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5d1413f9d0 0x7f5d140c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d19e00640 1 -- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d140aff10 msgr2=0x7f5d140ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d19e00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d140aff10 0x7f5d140ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d19e00640 1 -- 172.21.15.39:0/3329648576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc00b6a0 con 0x7f5d14159b40 2024-01-24T17:42:22.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d19e00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140c5f40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c007e40 tx=0x7f5d0c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:22.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d11600640 1 -- 172.21.15.39:0/3329648576 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c014620 con 0x7f5d14159b40 2024-01-24T17:42:22.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d11600640 1 -- 172.21.15.39:0/3329648576 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c00ee90 con 0x7f5d14159b40 2024-01-24T17:42:22.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d11600640 1 -- 172.21.15.39:0/3329648576 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c0149e0 con 0x7f5d14159b40 2024-01-24T17:42:22.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d140c5800 con 0x7f5d14159b40 2024-01-24T17:42:22.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.776+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d140ac890 con 0x7f5d14159b40 2024-01-24T17:42:22.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.777+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d140ac6f0 con 0x7f5d14159b40 2024-01-24T17:42:22.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.778+0000 7f5d11600640 1 -- 172.21.15.39:0/3329648576 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5d0c014b40 con 0x7f5d14159b40 2024-01-24T17:42:22.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.779+0000 7f5d11600640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cec06a430 0x7f5cec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:22.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.779+0000 7f5d11600640 1 -- 172.21.15.39:0/3329648576 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5d0c089380 con 0x7f5d14159b40 2024-01-24T17:42:22.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.780+0000 7f5d19400640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cec06a430 0x7f5cec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:22.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.781+0000 7f5d19400640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cec06a430 0x7f5cec06c8f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5d04000f60 tx=0x7f5d04007b70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:22.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.784+0000 7f5d11600640 1 -- 172.21.15.39:0/3329648576 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5d140ac6f0 con 0x7f5d14159b40 2024-01-24T17:42:22.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.977+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d140ac6f0 con 0x7f5d14159b40 2024-01-24T17:42:22.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.979+0000 7f5d11600640 1 -- 172.21.15.39:0/3329648576 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f5d140ac6f0 con 0x7f5d14159b40 2024-01-24T17:42:22.980 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:22.980 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:22.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.982+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cec06a430 msgr2=0x7f5cec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.982+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cec06a430 0x7f5cec06c8f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5d04000f60 tx=0x7f5d04007b70 comp rx=0 tx=0).stop 2024-01-24T17:42:22.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.982+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 msgr2=0x7f5d140c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.982+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140c5f40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c007e40 tx=0x7f5d0c00e4b0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.983+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 shutdown_connections 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.983+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5cec06a430 0x7f5cec06c8f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.983+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d14159b40 0x7f5d140c5f40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.983+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5d1413f9d0 0x7f5d140c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.983+0000 7f5d1be00640 1 --2- 172.21.15.39:0/3329648576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d140aff10 0x7f5d140ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.983+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 >> 172.21.15.39:0/3329648576 conn(0x7f5d1406a2a0 msgr2=0x7f5d1415b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.983+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 shutdown_connections 2024-01-24T17:42:22.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:22.983+0000 7f5d1be00640 1 -- 172.21.15.39:0/3329648576 wait complete. 2024-01-24T17:42:23.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:23.002 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:23.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.215+0000 7fb5c9000640 1 Processor -- start 2024-01-24T17:42:23.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.215+0000 7fb5c9000640 1 -- start start 2024-01-24T17:42:23.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.216+0000 7fb5c9000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ce390 0x7fb5c40ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.216+0000 7fb5c9000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c415d090 con 0x7fb5c415ccb0 2024-01-24T17:42:23.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.216+0000 7fb5c9000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c40ced30 con 0x7fb5c40c8d50 2024-01-24T17:42:23.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.216+0000 7fb5c9000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40cac20 con 0x7fb5c40ce390 2024-01-24T17:42:23.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.217+0000 7fb5c3400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb5c415ccb0 0x7fb5c40cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47922/0 (socket says 172.21.15.39:47922) 2024-01-24T17:42:23.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.217+0000 7fb5c3400640 1 -- 172.21.15.39:0/790247380 learned_addr learned my addr 172.21.15.39:0/790247380 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:23.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.217+0000 7fb5c2a00640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ce390 0x7fb5c40ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.217+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097030443 0 0) 0x7fb5c415d090 con 0x7fb5c415ccb0 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.217+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a4003520 con 0x7fb5c415ccb0 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1253340480 0 0) 0x7fb5a4003520 con 0x7fb5c415ccb0 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ce390 msgr2=0x7fb5c40ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ce390 0x7fb5c40ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 >> v1:172.21.15.100:6789/0 conn(0x7fb5c40c8d50 legacy=0x7fb5c40c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c40ce1a0 con 0x7fb5c415ccb0 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fb5bc005410 con 0x7fb5c415ccb0 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 >> v1:172.21.15.39:6789/0 conn(0x7fb5c415ccb0 legacy=0x7fb5c40cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5a40039a0 0x7fb5a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4004390 0x7fb5a4004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a4005060 0x7fb5a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40cac20 con 0x7fb5a4005060 2024-01-24T17:42:23.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a400e180 con 0x7fb5a4004390 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a400e340 con 0x7fb5a40039a0 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c3e00640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a4005060 0x7fb5a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.218+0000 7fb5c3400640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5a40039a0 0x7fb5a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.219+0000 7fb5c2a00640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4004390 0x7fb5a4004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.219+0000 7fb5c3e00640 1 -- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5a40039a0 msgr2=0x7fb5a4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.219+0000 7fb5c3e00640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5a40039a0 0x7fb5a4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.219+0000 7fb5c3e00640 1 -- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4004390 msgr2=0x7fb5a4004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.219+0000 7fb5c3e00640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4004390 0x7fb5a4004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.219+0000 7fb5c3e00640 1 -- 172.21.15.39:0/790247380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4003520 con 0x7fb5a4005060 2024-01-24T17:42:23.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.220+0000 7fb5c3e00640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a4005060 0x7fb5a400b980 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4001750 tx=0x7fb5a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7e146c2ae7ea5e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:23.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.220+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40052d0 con 0x7fb5a4005060 2024-01-24T17:42:23.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.220+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b402e050 con 0x7fb5a4005060 2024-01-24T17:42:23.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.220+0000 7fb5c2000640 1 -- 172.21.15.39:0/790247380 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4003830 con 0x7fb5a4005060 2024-01-24T17:42:23.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.221+0000 7fb5c9000640 1 -- 172.21.15.39:0/790247380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a4005060 msgr2=0x7fb5a400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.221+0000 7fb5c9000640 1 --2- 172.21.15.39:0/790247380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5a4005060 0x7fb5a400b980 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4001750 tx=0x7fb5a40057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.221+0000 7fb5c3e00640 1 -- 172.21.15.39:0/790247380 reap_dead start 2024-01-24T17:42:23.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.221+0000 7fb5c9000640 1 -- 172.21.15.39:0/790247380 shutdown_connections 2024-01-24T17:42:23.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.221+0000 7fb5c9000640 1 -- 172.21.15.39:0/790247380 >> 172.21.15.39:0/790247380 conn(0x7fb5c4069e50 msgr2=0x7fb5c406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:23.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.221+0000 7fb5c9000640 1 -- 172.21.15.39:0/790247380 shutdown_connections 2024-01-24T17:42:23.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.221+0000 7fb5c9000640 1 -- 172.21.15.39:0/790247380 wait complete. 2024-01-24T17:42:23.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.223+0000 7fb5c9000640 1 Processor -- start 2024-01-24T17:42:23.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.223+0000 7fb5c9000640 1 -- start start 2024-01-24T17:42:23.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c9000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4005060 0x7fb5c41571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c4157740 0x7fb5c40c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c9000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ac080 0x7fb5c40ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c9000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40ad330 con 0x7fb5c4157740 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c9000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c40ad860 con 0x7fb5a4005060 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c9000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4157bc0 con 0x7fb5c40ac080 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c4157740 0x7fb5c40c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c4157740 0x7fb5c40c3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59292/0 (socket says 172.21.15.39:59292) 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c2a00640 1 -- 172.21.15.39:0/3436416971 learned_addr learned my addr 172.21.15.39:0/3436416971 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.224+0000 7fb5c3e00640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ac080 0x7fb5c40ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c3400640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4005060 0x7fb5c41571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c2a00640 1 -- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ac080 msgr2=0x7fb5c40ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c2a00640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ac080 0x7fb5c40ac500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c2a00640 1 -- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4005060 msgr2=0x7fb5c41571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c2a00640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4005060 0x7fb5c41571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c2a00640 1 -- 172.21.15.39:0/3436416971 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c40aca40 con 0x7fb5c4157740 2024-01-24T17:42:23.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c3400640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4005060 0x7fb5c41571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:23.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c2a00640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c4157740 0x7fb5c40c3f30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac011600 tx=0x7fb5ac011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:23.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5bb400640 1 -- 172.21.15.39:0/3436416971 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac00f920 con 0x7fb5c4157740 2024-01-24T17:42:23.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5bb400640 1 -- 172.21.15.39:0/3436416971 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ac00fa80 con 0x7fb5c4157740 2024-01-24T17:42:23.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5bb400640 1 -- 172.21.15.39:0/3436416971 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac00fe30 con 0x7fb5c4157740 2024-01-24T17:42:23.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.225+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c40ad580 con 0x7fb5c4157740 2024-01-24T17:42:23.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.226+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c40c5900 con 0x7fb5c4157740 2024-01-24T17:42:23.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.227+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c404dff0 con 0x7fb5c4157740 2024-01-24T17:42:23.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.228+0000 7fb5bb400640 1 -- 172.21.15.39:0/3436416971 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb5ac015460 con 0x7fb5c4157740 2024-01-24T17:42:23.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.229+0000 7fb5bb400640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a460 0x7fb59c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.229+0000 7fb5bb400640 1 -- 172.21.15.39:0/3436416971 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb5ac0891e0 con 0x7fb5c4157740 2024-01-24T17:42:23.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.230+0000 7fb5c3400640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a460 0x7fb59c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.231+0000 7fb5c3400640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a460 0x7fb59c06c920 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5c41f14c0 tx=0x7fb5bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:23.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.234+0000 7fb5bb400640 1 -- 172.21.15.39:0/3436416971 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb5ac0547a0 con 0x7fb5c4157740 2024-01-24T17:42:23.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.429+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5c40ca4e0 con 0x7fb5c4157740 2024-01-24T17:42:23.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.430+0000 7fb5bb400640 1 -- 172.21.15.39:0/3436416971 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb5ac057dc0 con 0x7fb5c4157740 2024-01-24T17:42:23.431 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:23.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.432+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a460 msgr2=0x7fb59c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.432+0000 7fb5c9000640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a460 0x7fb59c06c920 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5c41f14c0 tx=0x7fb5bc009040 comp rx=0 tx=0).stop 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.432+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c4157740 msgr2=0x7fb5c40c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.432+0000 7fb5c9000640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c4157740 0x7fb5c40c3f30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac011600 tx=0x7fb5ac011ac0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.433+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 shutdown_connections 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.433+0000 7fb5c9000640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c40ac080 0x7fb5c40ac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.433+0000 7fb5c9000640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a460 0x7fb59c06c920 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.433+0000 7fb5c9000640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c4157740 0x7fb5c40c3f30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.433+0000 7fb5c9000640 1 --2- 172.21.15.39:0/3436416971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5a4005060 0x7fb5c41571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.433+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 >> 172.21.15.39:0/3436416971 conn(0x7fb5c4069e50 msgr2=0x7fb5c40cc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.433+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 shutdown_connections 2024-01-24T17:42:23.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.433+0000 7fb5c9000640 1 -- 172.21.15.39:0/3436416971 wait complete. 2024-01-24T17:42:23.453 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:23.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.679+0000 7f2bc0800640 1 Processor -- start 2024-01-24T17:42:23.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.680+0000 7f2bc0800640 1 -- start start 2024-01-24T17:42:23.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.681+0000 7f2bc0800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb815fff0 0x7f2bb80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.681+0000 7f2bc0800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2bb80d14c0 0x7f2bb80cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.681+0000 7f2bc0800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80cda90 con 0x7f2bb80b2a10 2024-01-24T17:42:23.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.681+0000 7f2bc0800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80b2e90 con 0x7f2bb80d14c0 2024-01-24T17:42:23.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.681+0000 7f2bc0800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80d32c0 con 0x7f2bb815fff0 2024-01-24T17:42:23.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.681+0000 7f2bbd400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2bb80b2a10 0x7f2bb80d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47932/0 (socket says 172.21.15.39:47932) 2024-01-24T17:42:23.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.681+0000 7f2bbd400640 1 -- 172.21.15.39:0/3110746126 learned_addr learned my addr 172.21.15.39:0/3110746126 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972914700 0 0) 0x7f2bb80cda90 con 0x7f2bb80b2a10 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b98003520 con 0x7f2bb80b2a10 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbe800640 1 --2- 172.21.15.39:0/3110746126 >> v2:172.21.15.100:3300/0 conn(0x7f2bb80d14c0 0x7f2bb80cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbde00640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb815fff0 0x7f2bb80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664172765 0 0) 0x7f2b98003520 con 0x7f2bb80b2a10 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb815fff0 msgr2=0x7f2bb80b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb815fff0 0x7f2bb80b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 >> v2:172.21.15.100:3300/0 conn(0x7f2bb80d14c0 msgr2=0x7f2bb80cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 --2- 172.21.15.39:0/3110746126 >> v2:172.21.15.100:3300/0 conn(0x7f2bb80d14c0 0x7f2bb80cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80b30a0 con 0x7f2bb80b2a10 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2ba0001670 con 0x7f2bb80b2a10 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 >> v1:172.21.15.39:6789/0 conn(0x7f2bb80b2a10 legacy=0x7f2bb80d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b98003970 0x7f2b98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b980047c0 0x7f2b98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb815fff0 0x7f2b98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80b2e90 con 0x7f2b980047c0 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80d32c0 con 0x7f2b98003970 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.682+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9800ba80 con 0x7f2bb815fff0 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.683+0000 7f2bbd400640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b980047c0 0x7f2b98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.683+0000 7f2bbde00640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b98003970 0x7f2b98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.683+0000 7f2bbe800640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb815fff0 0x7f2b98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.683+0000 7f2bbd400640 1 -- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb815fff0 msgr2=0x7f2b98005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.683+0000 7f2bbd400640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb815fff0 0x7f2b98005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.683+0000 7f2bbd400640 1 -- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b98003970 msgr2=0x7f2b98003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.683+0000 7f2bbd400640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b98003970 0x7f2b98003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.683+0000 7f2bbd400640 1 -- 172.21.15.39:0/3110746126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b98003520 con 0x7f2b980047c0 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.684+0000 7f2bbd400640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b980047c0 0x7f2b98004c40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000c160 tx=0x7f2b980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846e02c09337a087 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0005ad0 con 0x7f2b980047c0 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba0005c50 con 0x7f2b980047c0 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bbca00640 1 -- 172.21.15.39:0/3110746126 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba002c7c0 con 0x7f2b980047c0 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bc0800640 1 -- 172.21.15.39:0/3110746126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b980047c0 msgr2=0x7f2b98004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bc0800640 1 --2- 172.21.15.39:0/3110746126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b980047c0 0x7f2b98004c40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000c160 tx=0x7f2b980057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bbe800640 1 -- 172.21.15.39:0/3110746126 reap_dead start 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bc0800640 1 -- 172.21.15.39:0/3110746126 shutdown_connections 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bc0800640 1 -- 172.21.15.39:0/3110746126 >> 172.21.15.39:0/3110746126 conn(0x7f2bb809bcc0 msgr2=0x7f2bb80d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bc0800640 1 -- 172.21.15.39:0/3110746126 shutdown_connections 2024-01-24T17:42:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.685+0000 7f2bc0800640 1 -- 172.21.15.39:0/3110746126 wait complete. 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.687+0000 7f2bc0800640 1 Processor -- start 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.687+0000 7f2bc0800640 1 -- start start 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.688+0000 7f2bc0800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b980047c0 0x7f2bb8159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.688+0000 7f2bc0800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8154ea0 0x7f2bb81582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.688+0000 7f2bc0800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb80cba20 0x7f2bb80cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.688+0000 7f2bc0800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80cc3f0 con 0x7f2bb80cba20 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.688+0000 7f2bc0800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8159570 con 0x7f2bb8154ea0 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.688+0000 7f2bc0800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb815a990 con 0x7f2b980047c0 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbe800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb80cba20 0x7f2bb80cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbe800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb80cba20 0x7f2bb80cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59318/0 (socket says 172.21.15.39:59318) 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbe800640 1 -- 172.21.15.39:0/817947211 learned_addr learned my addr 172.21.15.39:0/817947211 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbde00640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b980047c0 0x7f2bb8159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbd400640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8154ea0 0x7f2bb81582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbe800640 1 -- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b980047c0 msgr2=0x7f2bb8159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbe800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b980047c0 0x7f2bb8159030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbe800640 1 -- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8154ea0 msgr2=0x7f2bb81582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbe800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8154ea0 0x7f2bb81582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbe800640 1 -- 172.21.15.39:0/817947211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb8156ce0 con 0x7f2bb80cba20 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.689+0000 7f2bbd400640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8154ea0 0x7f2bb81582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.690+0000 7f2bbe800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb80cba20 0x7f2bb80cbeb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b40b0 tx=0x7f2bb0009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.690+0000 7f2bb6000640 1 -- 172.21.15.39:0/817947211 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0007c90 con 0x7f2bb80cba20 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.690+0000 7f2bb6000640 1 -- 172.21.15.39:0/817947211 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb0007e10 con 0x7f2bb80cba20 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.690+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb8156fc0 con 0x7f2bb80cba20 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.690+0000 7f2bb6000640 1 -- 172.21.15.39:0/817947211 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0014640 con 0x7f2bb80cba20 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.690+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb8157580 con 0x7f2bb80cba20 2024-01-24T17:42:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.691+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb815ab50 con 0x7f2bb80cba20 2024-01-24T17:42:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.692+0000 7f2bb6000640 1 -- 172.21.15.39:0/817947211 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2bb001d3b0 con 0x7f2bb80cba20 2024-01-24T17:42:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.693+0000 7f2bb6000640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.693+0000 7f2bb6000640 1 -- 172.21.15.39:0/817947211 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2bb0088730 con 0x7f2bb80cba20 2024-01-24T17:42:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.694+0000 7f2bbde00640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:23.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.695+0000 7f2bbde00640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8006fd0 tx=0x7f2ba8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:23.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.697+0000 7f2bb6000640 1 -- 172.21.15.39:0/817947211 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2bb0053c60 con 0x7f2bb80cba20 2024-01-24T17:42:23.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.891+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2bb8073ef0 con 0x7f2bb80cba20 2024-01-24T17:42:23.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.892+0000 7f2bb6000640 1 -- 172.21.15.39:0/817947211 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2bb0057280 con 0x7f2bb80cba20 2024-01-24T17:42:23.893 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 msgr2=0x7f2b9006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8006fd0 tx=0x7f2ba8008040 comp rx=0 tx=0).stop 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb80cba20 msgr2=0x7f2bb80cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb80cba20 0x7f2bb80cbeb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b40b0 tx=0x7f2bb0009d30 comp rx=0 tx=0).stop 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 shutdown_connections 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2bb80cba20 0x7f2bb80cbeb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8154ea0 0x7f2bb81582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 --2- 172.21.15.39:0/817947211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b980047c0 0x7f2bb8159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 >> 172.21.15.39:0/817947211 conn(0x7f2bb809bcc0 msgr2=0x7f2bb806c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 shutdown_connections 2024-01-24T17:42:23.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:23.895+0000 7f2bc0800640 1 -- 172.21.15.39:0/817947211 wait complete. 2024-01-24T17:42:23.916 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:24.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.143+0000 7f84b7800640 1 Processor -- start 2024-01-24T17:42:24.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.143+0000 7f84b7800640 1 -- start start 2024-01-24T17:42:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.144+0000 7f84b7800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f84b015fff0 0x7f84b00b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.144+0000 7f84b7800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f84b00d14c0 0x7f84b00cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.144+0000 7f84b7800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b00cda90 con 0x7f84b00b2a10 2024-01-24T17:42:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.144+0000 7f84b7800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b00b2e90 con 0x7f84b00d14c0 2024-01-24T17:42:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.144+0000 7f84b7800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b00d32c0 con 0x7f84b015fff0 2024-01-24T17:42:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84b5800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f84b00d14c0 0x7f84b00cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84afe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f84b00b2a10 0x7f84b00d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47938/0 (socket says 172.21.15.39:47938) 2024-01-24T17:42:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84afe00640 1 -- 172.21.15.39:0/242804194 learned_addr learned my addr 172.21.15.39:0/242804194 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84b4e00640 1 --2- 172.21.15.39:0/242804194 >> v2:172.21.15.100:3300/0 conn(0x7f84b015fff0 0x7f84b00b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84af400640 1 -- 172.21.15.39:0/242804194 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019014843 0 0) 0x7f84b00cda90 con 0x7f84b00b2a10 2024-01-24T17:42:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84af400640 1 -- 172.21.15.39:0/242804194 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8490003520 con 0x7f84b00b2a10 2024-01-24T17:42:24.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84b5800640 1 -- 172.21.15.39:0/242804194 >> v2:172.21.15.100:3300/0 conn(0x7f84b015fff0 msgr2=0x7f84b00b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:24.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84b5800640 1 --2- 172.21.15.39:0/242804194 >> v2:172.21.15.100:3300/0 conn(0x7f84b015fff0 0x7f84b00b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84b5800640 1 -- 172.21.15.39:0/242804194 >> v1:172.21.15.39:6789/0 conn(0x7f84b00b2a10 legacy=0x7f84b00d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.145+0000 7f84b5800640 1 -- 172.21.15.39:0/242804194 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00b30a0 con 0x7f84b00d14c0 2024-01-24T17:42:24.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.146+0000 7f84b5800640 1 --2- 172.21.15.39:0/242804194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b00d14c0 0x7f84b00cdf20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f84b00d28e0 tx=0x7f84a802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902bce64974975fa server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:24.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.147+0000 7f84af400640 1 -- 172.21.15.39:0/242804194 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a802cc70 con 0x7f84b00d14c0 2024-01-24T17:42:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.147+0000 7f84af400640 1 -- 172.21.15.39:0/242804194 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a802cdd0 con 0x7f84b00d14c0 2024-01-24T17:42:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.147+0000 7f84af400640 1 -- 172.21.15.39:0/242804194 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8034640 con 0x7f84b00d14c0 2024-01-24T17:42:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.147+0000 7f84b7800640 1 -- 172.21.15.39:0/242804194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b00d14c0 msgr2=0x7f84b00cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.147+0000 7f84b7800640 1 --2- 172.21.15.39:0/242804194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b00d14c0 0x7f84b00cdf20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f84b00d28e0 tx=0x7f84a802bec0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.147+0000 7f84b7800640 1 -- 172.21.15.39:0/242804194 shutdown_connections 2024-01-24T17:42:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.147+0000 7f84b7800640 1 --2- 172.21.15.39:0/242804194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b00d14c0 0x7f84b00cdf20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.147+0000 7f84b7800640 1 --2- 172.21.15.39:0/242804194 >> v2:172.21.15.100:3300/0 conn(0x7f84b015fff0 0x7f84b00b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.148+0000 7f84b7800640 1 -- 172.21.15.39:0/242804194 >> 172.21.15.39:0/242804194 conn(0x7f84b009bcc0 msgr2=0x7f84b00d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.148+0000 7f84b7800640 1 -- 172.21.15.39:0/242804194 shutdown_connections 2024-01-24T17:42:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.148+0000 7f84b7800640 1 -- 172.21.15.39:0/242804194 wait complete. 2024-01-24T17:42:24.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.149+0000 7f84b7800640 1 Processor -- start 2024-01-24T17:42:24.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.150+0000 7f84b7800640 1 -- start start 2024-01-24T17:42:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b7800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b00b2a10 0x7f84b00b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b7800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b00d14c0 0x7f84b015d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b7800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b015fff0 0x7f84b014d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b7800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00ca970 con 0x7f84b015fff0 2024-01-24T17:42:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b7800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b015d010 con 0x7f84b00d14c0 2024-01-24T17:42:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b7800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00c9cf0 con 0x7f84b00b2a10 2024-01-24T17:42:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b015fff0 0x7f84b014d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b015fff0 0x7f84b014d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59342/0 (socket says 172.21.15.39:59342) 2024-01-24T17:42:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b5800640 1 -- 172.21.15.39:0/3182721172 learned_addr learned my addr 172.21.15.39:0/3182721172 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b4e00640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b00b2a10 0x7f84b00b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b5800640 1 -- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b00b2a10 msgr2=0x7f84b00b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b5800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b00b2a10 0x7f84b00b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b5800640 1 -- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b00d14c0 msgr2=0x7f84b015d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84afe00640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b00d14c0 0x7f84b015d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b5800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b00d14c0 0x7f84b015d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.151+0000 7f84b5800640 1 -- 172.21.15.39:0/3182721172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a8009a70 con 0x7f84b015fff0 2024-01-24T17:42:24.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.152+0000 7f84b5800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b015fff0 0x7f84b014d830 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f84a802c4d0 tx=0x7f84a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:24.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.152+0000 7f84acc00640 1 -- 172.21.15.39:0/3182721172 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a803e610 con 0x7f84b015fff0 2024-01-24T17:42:24.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.152+0000 7f84acc00640 1 -- 172.21.15.39:0/3182721172 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a803e790 con 0x7f84b015fff0 2024-01-24T17:42:24.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.152+0000 7f84acc00640 1 -- 172.21.15.39:0/3182721172 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a803eb40 con 0x7f84b015fff0 2024-01-24T17:42:24.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.152+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b00c9fd0 con 0x7f84b015fff0 2024-01-24T17:42:24.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.152+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b00ca590 con 0x7f84b015fff0 2024-01-24T17:42:24.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.153+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b00caab0 con 0x7f84b015fff0 2024-01-24T17:42:24.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.155+0000 7f84acc00640 1 -- 172.21.15.39:0/3182721172 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f84a803b050 con 0x7f84b015fff0 2024-01-24T17:42:24.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.156+0000 7f84acc00640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f848806a460 0x7f848806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.156+0000 7f84acc00640 1 -- 172.21.15.39:0/3182721172 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f84a80ab3a0 con 0x7f84b015fff0 2024-01-24T17:42:24.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.157+0000 7f84b4e00640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f848806a460 0x7f848806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.158+0000 7f84b4e00640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f848806a460 0x7f848806c920 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f84a0006fd0 tx=0x7f84a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:24.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.162+0000 7f84acc00640 1 -- 172.21.15.39:0/3182721172 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f84b00caab0 con 0x7f84b015fff0 2024-01-24T17:42:24.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.358+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84b00caab0 con 0x7f84b015fff0 2024-01-24T17:42:24.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.365+0000 7f84acc00640 1 -- 172.21.15.39:0/3182721172 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84a8039020 con 0x7f84b015fff0 2024-01-24T17:42:24.366 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:24.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f848806a460 msgr2=0x7f848806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f848806a460 0x7f848806c920 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f84a0006fd0 tx=0x7f84a0008040 comp rx=0 tx=0).stop 2024-01-24T17:42:24.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b015fff0 msgr2=0x7f84b014d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b015fff0 0x7f84b014d830 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f84a802c4d0 tx=0x7f84a8004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:24.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 shutdown_connections 2024-01-24T17:42:24.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f848806a460 0x7f848806c920 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84b015fff0 0x7f84b014d830 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84b00d14c0 0x7f84b015d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 --2- 172.21.15.39:0/3182721172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84b00b2a10 0x7f84b00b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 >> 172.21.15.39:0/3182721172 conn(0x7f84b009bcc0 msgr2=0x7f84b015ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:24.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 shutdown_connections 2024-01-24T17:42:24.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.368+0000 7f84b7800640 1 -- 172.21.15.39:0/3182721172 wait complete. 2024-01-24T17:42:24.388 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:42:24.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.608+0000 7f490f600640 1 Processor -- start 2024-01-24T17:42:24.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.609+0000 7f490f600640 1 -- start start 2024-01-24T17:42:24.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.610+0000 7f490f600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49080cf990 0x7f49080cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.610+0000 7f490f600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815fb90 0x7f490815d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.610+0000 7f490f600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49080d02b0 con 0x7f490815d020 2024-01-24T17:42:24.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.610+0000 7f490f600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490815e080 con 0x7f49080cf990 2024-01-24T17:42:24.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.610+0000 7f490f600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490815dac0 con 0x7f490815fb90 2024-01-24T17:42:24.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.611+0000 7f490cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49080cf990 0x7f49080cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.611+0000 7f490cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49080cf990 0x7f49080cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59346/0 (socket says 172.21.15.39:59346) 2024-01-24T17:42:24.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.611+0000 7f490cc00640 1 -- 172.21.15.39:0/809110216 learned_addr learned my addr 172.21.15.39:0/809110216 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:24.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.611+0000 7f490cc00640 1 -- 172.21.15.39:0/809110216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815fb90 msgr2=0x7f490815d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:24.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.611+0000 7f490cc00640 1 --2- 172.21.15.39:0/809110216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815fb90 0x7f490815d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.611+0000 7f490cc00640 1 -- 172.21.15.39:0/809110216 >> v1:172.21.15.100:6789/0 conn(0x7f490815d020 legacy=0x7f4908160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:24.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.611+0000 7f490cc00640 1 -- 172.21.15.39:0/809110216 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490815dda0 con 0x7f49080cf990 2024-01-24T17:42:24.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.613+0000 7f490cc00640 1 --2- 172.21.15.39:0/809110216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49080cf990 0x7f49080cfd70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009870 tx=0x7f48f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20000b2f45bcecb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:24.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.613+0000 7f4907400640 1 -- 172.21.15.39:0/809110216 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8031640 con 0x7f49080cf990 2024-01-24T17:42:24.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.613+0000 7f4907400640 1 -- 172.21.15.39:0/809110216 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f80317a0 con 0x7f49080cf990 2024-01-24T17:42:24.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.613+0000 7f4907400640 1 -- 172.21.15.39:0/809110216 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8031b30 con 0x7f49080cf990 2024-01-24T17:42:24.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.614+0000 7f490f600640 1 -- 172.21.15.39:0/809110216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49080cf990 msgr2=0x7f49080cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.614+0000 7f490f600640 1 --2- 172.21.15.39:0/809110216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49080cf990 0x7f49080cfd70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009870 tx=0x7f48f802b8f0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.614+0000 7f490f600640 1 -- 172.21.15.39:0/809110216 shutdown_connections 2024-01-24T17:42:24.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.614+0000 7f490f600640 1 --2- 172.21.15.39:0/809110216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815fb90 0x7f490815d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.614+0000 7f490f600640 1 --2- 172.21.15.39:0/809110216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49080cf990 0x7f49080cfd70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.614+0000 7f490f600640 1 -- 172.21.15.39:0/809110216 >> 172.21.15.39:0/809110216 conn(0x7f4908071790 msgr2=0x7f4908071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:24.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.614+0000 7f490f600640 1 -- 172.21.15.39:0/809110216 shutdown_connections 2024-01-24T17:42:24.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.614+0000 7f490f600640 1 -- 172.21.15.39:0/809110216 wait complete. 2024-01-24T17:42:24.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.616+0000 7f490f600640 1 Processor -- start 2024-01-24T17:42:24.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.616+0000 7f490f600640 1 -- start start 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490f600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49080cf990 0x7f490814dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490f600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815d020 0x7f49080d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490f600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f490815fb90 0x7f49080d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490f600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49080d1b30 con 0x7f490815fb90 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490f600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490814e240 con 0x7f49080cf990 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490f600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49080d0eb0 con 0x7f490815d020 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f490815fb90 0x7f49080d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f490815fb90 0x7f49080d1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59360/0 (socket says 172.21.15.39:59360) 2024-01-24T17:42:24.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.617+0000 7f490d600640 1 -- 172.21.15.39:0/1600687894 learned_addr learned my addr 172.21.15.39:0/1600687894 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f490cc00640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49080cf990 0x7f490814dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f4907e00640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815d020 0x7f49080d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f490d600640 1 -- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815d020 msgr2=0x7f49080d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f490d600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815d020 0x7f49080d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f490d600640 1 -- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49080cf990 msgr2=0x7f490814dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f490d600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49080cf990 0x7f490814dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f490d600640 1 -- 172.21.15.39:0/1600687894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f80094d0 con 0x7f490815fb90 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f490d600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f490815fb90 0x7f49080d1f50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f490000b9e0 tx=0x7f490000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f4904c00640 1 -- 172.21.15.39:0/1600687894 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490000cb20 con 0x7f490815fb90 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f4904c00640 1 -- 172.21.15.39:0/1600687894 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490000cc80 con 0x7f490815fb90 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f4904c00640 1 -- 172.21.15.39:0/1600687894 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900014440 con 0x7f490815fb90 2024-01-24T17:42:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.618+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49080d1190 con 0x7f490815fb90 2024-01-24T17:42:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.619+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49080d1750 con 0x7f490815fb90 2024-01-24T17:42:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.620+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49080d1c70 con 0x7f490815fb90 2024-01-24T17:42:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.621+0000 7f4904c00640 1 -- 172.21.15.39:0/1600687894 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f490001d3b0 con 0x7f490815fb90 2024-01-24T17:42:24.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.622+0000 7f4904c00640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f48e006a430 0x7f48e006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:24.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.622+0000 7f4904c00640 1 -- 172.21.15.39:0/1600687894 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f49000888c0 con 0x7f490815fb90 2024-01-24T17:42:24.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.622+0000 7f490cc00640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f48e006a430 0x7f48e006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:24.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.623+0000 7f490cc00640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f48e006a430 0x7f48e006c8f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f48f8036000 tx=0x7f48f803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:24.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.626+0000 7f4904c00640 1 -- 172.21.15.39:0/1600687894 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f49080d1c70 con 0x7f490815fb90 2024-01-24T17:42:24.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.834+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f49080d1c70 con 0x7f490815fb90 2024-01-24T17:42:24.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.834+0000 7f4904c00640 1 -- 172.21.15.39:0/1600687894 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f49080d1c70 con 0x7f490815fb90 2024-01-24T17:42:24.835 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:42:24.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.837+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f48e006a430 msgr2=0x7f48e006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.837+0000 7f490f600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f48e006a430 0x7f48e006c8f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f48f8036000 tx=0x7f48f803c040 comp rx=0 tx=0).stop 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.837+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f490815fb90 msgr2=0x7f49080d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.837+0000 7f490f600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f490815fb90 0x7f49080d1f50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f490000b9e0 tx=0x7f490000bea0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.838+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 shutdown_connections 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.838+0000 7f490f600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f48e006a430 0x7f48e006c8f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.838+0000 7f490f600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f490815fb90 0x7f49080d1f50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.838+0000 7f490f600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f490815d020 0x7f49080d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.838+0000 7f490f600640 1 --2- 172.21.15.39:0/1600687894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49080cf990 0x7f490814dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.838+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 >> 172.21.15.39:0/1600687894 conn(0x7f4908071790 msgr2=0x7f49080b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.838+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 shutdown_connections 2024-01-24T17:42:24.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:24.838+0000 7f490f600640 1 -- 172.21.15.39:0/1600687894 wait complete. 2024-01-24T17:42:24.857 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:42:24.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:24.857 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:42:24.955 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:42:24.955 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:42:24.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:24.955 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:42:25.020 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:42:25.020 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:25.080 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:42:25.080 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:25.132 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:25.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.362+0000 7f2c8b200640 1 Processor -- start 2024-01-24T17:42:25.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.363+0000 7f2c8b200640 1 -- start start 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.364+0000 7f2c8b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c0cf230 0x7f2c8c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.364+0000 7f2c8b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2c8c0cd400 0x7f2c8c0cd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.364+0000 7f2c8b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8c0d0240 con 0x7f2c8c15ca50 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.364+0000 7f2c8b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8c0cff60 con 0x7f2c8c0cd400 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.364+0000 7f2c8b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c0d0660 con 0x7f2c8c0cf230 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.364+0000 7f2c89e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2c8c15ca50 0x7f2c8c0d1140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47948/0 (socket says 172.21.15.39:47948) 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c89e00640 1 -- 172.21.15.39:0/2951683707 learned_addr learned my addr 172.21.15.39:0/2951683707 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c8a800640 1 --2- 172.21.15.39:0/2951683707 >> v2:172.21.15.39:3300/0 conn(0x7f2c8c0cd400 0x7f2c8c0cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c81200640 1 --2- 172.21.15.39:0/2951683707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c0cf230 0x7f2c8c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c89400640 1 -- 172.21.15.39:0/2951683707 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623629669 0 0) 0x7f2c8c0d0240 con 0x7f2c8c15ca50 2024-01-24T17:42:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c89400640 1 -- 172.21.15.39:0/2951683707 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c6c003520 con 0x7f2c8c15ca50 2024-01-24T17:42:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c8a800640 1 -- 172.21.15.39:0/2951683707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c0cf230 msgr2=0x7f2c8c0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c8a800640 1 --2- 172.21.15.39:0/2951683707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c0cf230 0x7f2c8c0cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c8a800640 1 -- 172.21.15.39:0/2951683707 >> v1:172.21.15.39:6789/0 conn(0x7f2c8c15ca50 legacy=0x7f2c8c0d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.365+0000 7f2c8a800640 1 -- 172.21.15.39:0/2951683707 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c0d0940 con 0x7f2c8c0cd400 2024-01-24T17:42:25.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.367+0000 7f2c8a800640 1 --2- 172.21.15.39:0/2951683707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cd400 0x7f2c8c0cd850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2c84009d90 tx=0x7f2c8402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e3a8998b18a03e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.367+0000 7f2c89400640 1 -- 172.21.15.39:0/2951683707 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8402cbc0 con 0x7f2c8c0cd400 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.367+0000 7f2c89400640 1 -- 172.21.15.39:0/2951683707 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8402cd20 con 0x7f2c8c0cd400 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.367+0000 7f2c89400640 1 -- 172.21.15.39:0/2951683707 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c84034640 con 0x7f2c8c0cd400 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.367+0000 7f2c8b200640 1 -- 172.21.15.39:0/2951683707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cd400 msgr2=0x7f2c8c0cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.367+0000 7f2c8b200640 1 --2- 172.21.15.39:0/2951683707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cd400 0x7f2c8c0cd850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2c84009d90 tx=0x7f2c8402be10 comp rx=0 tx=0).stop 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.368+0000 7f2c8b200640 1 -- 172.21.15.39:0/2951683707 shutdown_connections 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.368+0000 7f2c8b200640 1 --2- 172.21.15.39:0/2951683707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cd400 0x7f2c8c0cd850 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.368+0000 7f2c8b200640 1 --2- 172.21.15.39:0/2951683707 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c0cf230 0x7f2c8c0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.368+0000 7f2c8b200640 1 -- 172.21.15.39:0/2951683707 >> 172.21.15.39:0/2951683707 conn(0x7f2c8c071dd0 msgr2=0x7f2c8c09bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.368+0000 7f2c8b200640 1 -- 172.21.15.39:0/2951683707 shutdown_connections 2024-01-24T17:42:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.368+0000 7f2c8b200640 1 -- 172.21.15.39:0/2951683707 wait complete. 2024-01-24T17:42:25.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.370+0000 7f2c8b200640 1 Processor -- start 2024-01-24T17:42:25.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.370+0000 7f2c8b200640 1 -- start start 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c8b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c8c0cd400 0x7f2c8c0afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c8b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cf230 0x7f2c8c153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c8b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c15ca50 0x7f2c8c0b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c8b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c158460 con 0x7f2c8c0cf230 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c8b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c1543e0 con 0x7f2c8c0cd400 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c8b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c158110 con 0x7f2c8c15ca50 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c81200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cf230 0x7f2c8c153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c81200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cf230 0x7f2c8c153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59378/0 (socket says 172.21.15.39:59378) 2024-01-24T17:42:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c81200640 1 -- 172.21.15.39:0/4105845117 learned_addr learned my addr 172.21.15.39:0/4105845117 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c89e00640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c8c0cd400 0x7f2c8c0afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c81200640 1 -- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c15ca50 msgr2=0x7f2c8c0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c8a800640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c15ca50 0x7f2c8c0b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c81200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c15ca50 0x7f2c8c0b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c81200640 1 -- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c8c0cd400 msgr2=0x7f2c8c0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c81200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c8c0cd400 0x7f2c8c0afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c81200640 1 -- 172.21.15.39:0/4105845117 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c84009a70 con 0x7f2c8c0cf230 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.371+0000 7f2c89e00640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c8c0cd400 0x7f2c8c0afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.372+0000 7f2c81200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cf230 0x7f2c8c153ea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400d800 tx=0x7f2c7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.372+0000 7f2c82a00640 1 -- 172.21.15.39:0/4105845117 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7400b6c0 con 0x7f2c8c0cf230 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.372+0000 7f2c82a00640 1 -- 172.21.15.39:0/4105845117 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7400b820 con 0x7f2c8c0cf230 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.372+0000 7f2c82a00640 1 -- 172.21.15.39:0/4105845117 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7400bbb0 con 0x7f2c8c0cf230 2024-01-24T17:42:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.372+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c0b04c0 con 0x7f2c8c0cf230 2024-01-24T17:42:25.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.372+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8c0b1410 con 0x7f2c8c0cf230 2024-01-24T17:42:25.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.374+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c8c0677b0 con 0x7f2c8c0cf230 2024-01-24T17:42:25.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.374+0000 7f2c82a00640 1 -- 172.21.15.39:0/4105845117 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2c7400bd10 con 0x7f2c8c0cf230 2024-01-24T17:42:25.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.375+0000 7f2c82a00640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c6406a430 0x7f2c6406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.375+0000 7f2c82a00640 1 -- 172.21.15.39:0/4105845117 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2c74088e80 con 0x7f2c8c0cf230 2024-01-24T17:42:25.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.376+0000 7f2c89e00640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c6406a430 0x7f2c6406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.377+0000 7f2c89e00640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c6406a430 0x7f2c6406c8f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c005880 tx=0x7f2c7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:25.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.380+0000 7f2c82a00640 1 -- 172.21.15.39:0/4105845117 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2c74054450 con 0x7f2c8c0cf230 2024-01-24T17:42:25.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.577+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c8c0b09e0 con 0x7f2c8c0cf230 2024-01-24T17:42:25.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.580+0000 7f2c82a00640 1 -- 172.21.15.39:0/4105845117 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f2c74057a70 con 0x7f2c8c0cf230 2024-01-24T17:42:25.581 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:25.581 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.582+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c6406a430 msgr2=0x7f2c6406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.582+0000 7f2c8b200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c6406a430 0x7f2c6406c8f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c005880 tx=0x7f2c7c009040 comp rx=0 tx=0).stop 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.582+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cf230 msgr2=0x7f2c8c153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.582+0000 7f2c8b200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cf230 0x7f2c8c153ea0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400d800 tx=0x7f2c7400dcc0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.583+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 shutdown_connections 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.583+0000 7f2c8b200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c8c15ca50 0x7f2c8c0b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.583+0000 7f2c8b200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c6406a430 0x7f2c6406c8f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.583+0000 7f2c8b200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c8c0cf230 0x7f2c8c153ea0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.583+0000 7f2c8b200640 1 --2- 172.21.15.39:0/4105845117 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c8c0cd400 0x7f2c8c0afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.583+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 >> 172.21.15.39:0/4105845117 conn(0x7f2c8c071dd0 msgr2=0x7f2c8c06b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.583+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 shutdown_connections 2024-01-24T17:42:25.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.583+0000 7f2c8b200640 1 -- 172.21.15.39:0/4105845117 wait complete. 2024-01-24T17:42:25.603 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:25.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:25.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.831+0000 7f0aed800640 1 Processor -- start 2024-01-24T17:42:25.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.831+0000 7f0aed800640 1 -- start start 2024-01-24T17:42:25.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.832+0000 7f0aed800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0ae815c8d0 0x7f0ae80d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.833+0000 7f0aed800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae80d2e80 0x7f0ae80cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.833+0000 7f0aed800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae80b38a0 con 0x7f0ae80b2710 2024-01-24T17:42:25.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.833+0000 7f0aed800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae80d13e0 con 0x7f0ae815c8d0 2024-01-24T17:42:25.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.833+0000 7f0aed800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae80d0ab0 con 0x7f0ae80d2e80 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.833+0000 7f0aece00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae80d2e80 0x7f0ae80cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.833+0000 7f0ae7400640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f0ae80b2710 0x7f0ae80d2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:38846/0 (socket says 172.21.15.39:38846) 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.833+0000 7f0ae7400640 1 -- 172.21.15.39:0/4012739078 learned_addr learned my addr 172.21.15.39:0/4012739078 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.833+0000 7f0aece00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae80d2e80 0x7f0ae80cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:36724/0 (socket says 172.21.15.39:36724) 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.834+0000 7f0ae7e00640 1 --2- 172.21.15.39:0/4012739078 >> v2:172.21.15.100:3300/0 conn(0x7f0ae815c8d0 0x7f0ae80d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.834+0000 7f0aece00640 1 -- 172.21.15.39:0/4012739078 >> v2:172.21.15.100:3300/0 conn(0x7f0ae815c8d0 msgr2=0x7f0ae80d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.834+0000 7f0aece00640 1 --2- 172.21.15.39:0/4012739078 >> v2:172.21.15.100:3300/0 conn(0x7f0ae815c8d0 0x7f0ae80d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.834+0000 7f0aece00640 1 -- 172.21.15.39:0/4012739078 >> v1:172.21.15.100:6789/0 conn(0x7f0ae80b2710 legacy=0x7f0ae80d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.834+0000 7f0aece00640 1 -- 172.21.15.39:0/4012739078 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae80d0d90 con 0x7f0ae80d2e80 2024-01-24T17:42:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.834+0000 7f0ae6a00640 1 -- 172.21.15.39:0/4012739078 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830197074 0 0) 0x7f0ae80b38a0 con 0x7f0ae80b2710 2024-01-24T17:42:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.835+0000 7f0aece00640 1 --2- 172.21.15.39:0/4012739078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae80d2e80 0x7f0ae80cd630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00097a0 tx=0x7f0ae002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43f640602821080f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.836+0000 7f0ae6a00640 1 -- 172.21.15.39:0/4012739078 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae002d040 con 0x7f0ae80d2e80 2024-01-24T17:42:25.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.836+0000 7f0ae6a00640 1 -- 172.21.15.39:0/4012739078 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae0031810 con 0x7f0ae80d2e80 2024-01-24T17:42:25.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.836+0000 7f0ae6a00640 1 -- 172.21.15.39:0/4012739078 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0031ba0 con 0x7f0ae80d2e80 2024-01-24T17:42:25.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.837+0000 7f0aed800640 1 -- 172.21.15.39:0/4012739078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae80d2e80 msgr2=0x7f0ae80cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.837+0000 7f0aed800640 1 --2- 172.21.15.39:0/4012739078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae80d2e80 0x7f0ae80cd630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00097a0 tx=0x7f0ae002b8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.837+0000 7f0aed800640 1 -- 172.21.15.39:0/4012739078 shutdown_connections 2024-01-24T17:42:25.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.837+0000 7f0aed800640 1 --2- 172.21.15.39:0/4012739078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae80d2e80 0x7f0ae80cd630 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.837+0000 7f0aed800640 1 --2- 172.21.15.39:0/4012739078 >> v2:172.21.15.100:3300/0 conn(0x7f0ae815c8d0 0x7f0ae80d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.837+0000 7f0aed800640 1 -- 172.21.15.39:0/4012739078 >> 172.21.15.39:0/4012739078 conn(0x7f0ae8071dd0 msgr2=0x7f0ae809bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:25.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.837+0000 7f0aed800640 1 -- 172.21.15.39:0/4012739078 shutdown_connections 2024-01-24T17:42:25.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.837+0000 7f0aed800640 1 -- 172.21.15.39:0/4012739078 wait complete. 2024-01-24T17:42:25.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.839+0000 7f0aed800640 1 Processor -- start 2024-01-24T17:42:25.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.839+0000 7f0aed800640 1 -- start start 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0aed800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ae80b2710 0x7f0ae809d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0aed800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ae80d2e80 0x7f0ae80cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0aed800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae815c8d0 0x7f0ae8158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0aed800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae80cc210 con 0x7f0ae80d2e80 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0aed800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae81f5b30 con 0x7f0ae80b2710 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0aed800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae809e300 con 0x7f0ae815c8d0 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0ae7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ae80d2e80 0x7f0ae80cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0ae7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ae80d2e80 0x7f0ae80cbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59386/0 (socket says 172.21.15.39:59386) 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0ae7400640 1 -- 172.21.15.39:0/3962776488 learned_addr learned my addr 172.21.15.39:0/3962776488 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:25.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.840+0000 7f0ae7e00640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ae80b2710 0x7f0ae809d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0aece00640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae815c8d0 0x7f0ae8158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0ae7400640 1 -- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae815c8d0 msgr2=0x7f0ae8158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0ae7400640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae815c8d0 0x7f0ae8158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0ae7400640 1 -- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ae80b2710 msgr2=0x7f0ae809d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0ae7400640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ae80b2710 0x7f0ae809d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0ae7400640 1 -- 172.21.15.39:0/3962776488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae0009400 con 0x7f0ae80d2e80 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0ae7400640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ae80d2e80 0x7f0ae80cbcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0ad000c890 tx=0x7f0ad000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0adfe00640 1 -- 172.21.15.39:0/3962776488 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad000a640 con 0x7f0ae80d2e80 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0adfe00640 1 -- 172.21.15.39:0/3962776488 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad000a7c0 con 0x7f0ae80d2e80 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0adfe00640 1 -- 172.21.15.39:0/3962776488 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad000ab70 con 0x7f0ae80d2e80 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.841+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae809dd40 con 0x7f0ae80d2e80 2024-01-24T17:42:25.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.842+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae81f57c0 con 0x7f0ae80d2e80 2024-01-24T17:42:25.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.843+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae81f5cc0 con 0x7f0ae80d2e80 2024-01-24T17:42:25.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.844+0000 7f0adfe00640 1 -- 172.21.15.39:0/3962776488 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0ad000acd0 con 0x7f0ae80d2e80 2024-01-24T17:42:25.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.844+0000 7f0adfe00640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ac006a430 0x7f0ac006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:25.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.845+0000 7f0adfe00640 1 -- 172.21.15.39:0/3962776488 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0ad0087f00 con 0x7f0ae80d2e80 2024-01-24T17:42:25.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.845+0000 7f0ae7e00640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ac006a430 0x7f0ac006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:25.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.846+0000 7f0ae7e00640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ac006a430 0x7f0ac006c8f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009a30 tx=0x7f0ad8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:25.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:25.849+0000 7f0adfe00640 1 -- 172.21.15.39:0/3962776488 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0ad0053430 con 0x7f0ae80d2e80 2024-01-24T17:42:26.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.039+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ae804dff0 con 0x7f0ae80d2e80 2024-01-24T17:42:26.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.040+0000 7f0adfe00640 1 -- 172.21.15.39:0/3962776488 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ad00180a0 con 0x7f0ae80d2e80 2024-01-24T17:42:26.040 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:26.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ac006a430 msgr2=0x7f0ac006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ac006a430 0x7f0ac006c8f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009a30 tx=0x7f0ad8008040 comp rx=0 tx=0).stop 2024-01-24T17:42:26.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ae80d2e80 msgr2=0x7f0ae80cbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ae80d2e80 0x7f0ae80cbcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0ad000c890 tx=0x7f0ad000cd50 comp rx=0 tx=0).stop 2024-01-24T17:42:26.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 shutdown_connections 2024-01-24T17:42:26.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ae815c8d0 0x7f0ae8158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ac006a430 0x7f0ac006c8f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ae80d2e80 0x7f0ae80cbcd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 --2- 172.21.15.39:0/3962776488 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ae80b2710 0x7f0ae809d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.042+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 >> 172.21.15.39:0/3962776488 conn(0x7f0ae8071dd0 msgr2=0x7f0ae815eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:26.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.043+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 shutdown_connections 2024-01-24T17:42:26.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.043+0000 7f0aed800640 1 -- 172.21.15.39:0/3962776488 wait complete. 2024-01-24T17:42:26.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:26.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.287+0000 7f67bd000640 1 Processor -- start 2024-01-24T17:42:26.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.287+0000 7f67bd000640 1 -- start start 2024-01-24T17:42:26.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.289+0000 7f67bd000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67b80b0730 0x7f67b815da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.289+0000 7f67bd000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b815e450 con 0x7f67b80b0350 2024-01-24T17:42:26.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.289+0000 7f67bd000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b806d4c0 con 0x7f67b815dfc0 2024-01-24T17:42:26.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.289+0000 7f67bd000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b806cf00 con 0x7f67b80b0730 2024-01-24T17:42:26.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.289+0000 7f67b5e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f67b80b0350 0x7f67b806c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47958/0 (socket says 172.21.15.39:47958) 2024-01-24T17:42:26.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.289+0000 7f67b5e00640 1 -- 172.21.15.39:0/2261910441 learned_addr learned my addr 172.21.15.39:0/2261910441 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:26.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67ad200640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67b80b0730 0x7f67b815da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265762243 0 0) 0x7f67a8001110 con 0x7f67b80b0350 2024-01-24T17:42:26.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6798003520 con 0x7f67b80b0350 2024-01-24T17:42:26.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3491980534 0 0) 0x7f6798003520 con 0x7f67b80b0350 2024-01-24T17:42:26.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67b80b0730 msgr2=0x7f67b815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67b80b0730 0x7f67b815da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 >> v1:172.21.15.100:6789/0 conn(0x7f67b815dfc0 legacy=0x7f67b80b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b806d1e0 con 0x7f67b80b0350 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f67a8001800 con 0x7f67b80b0350 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 >> v1:172.21.15.39:6789/0 conn(0x7f67b80b0350 legacy=0x7f67b806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67980039a0 0x7f6798003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.290+0000 7f67b5400640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67b80b0730 0x7f67980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5400640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6798004d00 0x7f6798005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b806cf00 con 0x7f67980039a0 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679800bb40 con 0x7f6798004d00 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679800bd00 con 0x7f67b80b0730 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5e00640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67980039a0 0x7f6798003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67ad200640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67b80b0730 0x7f67980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b6800640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6798004d00 0x7f6798005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5e00640 1 -- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67b80b0730 msgr2=0x7f67980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5e00640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67b80b0730 0x7f67980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5e00640 1 -- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6798004d00 msgr2=0x7f6798005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5e00640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6798004d00 0x7f6798005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b5e00640 1 -- 172.21.15.39:0/2261910441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6798003520 con 0x7f67980039a0 2024-01-24T17:42:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.291+0000 7f67b6800640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6798004d00 0x7f6798005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.292+0000 7f67b5e00640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67980039a0 0x7f6798003dc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f67a8007fc0 tx=0x7f67980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5775613f58b2fb2a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.293+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8005ad0 con 0x7f67980039a0 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.293+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a8005c50 con 0x7f67980039a0 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.293+0000 7f67b5400640 1 -- 172.21.15.39:0/2261910441 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a802c7e0 con 0x7f67980039a0 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.293+0000 7f67bd000640 1 -- 172.21.15.39:0/2261910441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67980039a0 msgr2=0x7f6798003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.293+0000 7f67bd000640 1 --2- 172.21.15.39:0/2261910441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67980039a0 0x7f6798003dc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f67a8007fc0 tx=0x7f67980057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.293+0000 7f67b6800640 1 -- 172.21.15.39:0/2261910441 reap_dead start 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.293+0000 7f67bd000640 1 -- 172.21.15.39:0/2261910441 shutdown_connections 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.293+0000 7f67bd000640 1 -- 172.21.15.39:0/2261910441 >> 172.21.15.39:0/2261910441 conn(0x7f67b80cb0e0 msgr2=0x7f67b80cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.294+0000 7f67bd000640 1 -- 172.21.15.39:0/2261910441 shutdown_connections 2024-01-24T17:42:26.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.294+0000 7f67bd000640 1 -- 172.21.15.39:0/2261910441 wait complete. 2024-01-24T17:42:26.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.295+0000 7f67bd000640 1 Processor -- start 2024-01-24T17:42:26.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.295+0000 7f67bd000640 1 -- start start 2024-01-24T17:42:26.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.296+0000 7f67bd000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67980039a0 0x7f67b81f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.296+0000 7f67bd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67b80c8230 0x7f67b815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.296+0000 7f67bd000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67b80b1ce0 0x7f67b80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.296+0000 7f67bd000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b80b26b0 con 0x7f67b80c8230 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.296+0000 7f67bd000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b80cc8e0 con 0x7f67b80b1ce0 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.296+0000 7f67bd000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b81f2c20 con 0x7f67980039a0 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67b80c8230 0x7f67b815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67b80c8230 0x7f67b815bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59398/0 (socket says 172.21.15.39:59398) 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 -- 172.21.15.39:0/2443703045 learned_addr learned my addr 172.21.15.39:0/2443703045 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67b5e00640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67980039a0 0x7f67b81f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67b6800640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67b80b1ce0 0x7f67b80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 -- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67980039a0 msgr2=0x7f67b81f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67980039a0 0x7f67b81f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 -- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67b80b1ce0 msgr2=0x7f67b80b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67b80b1ce0 0x7f67b80b2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 -- 172.21.15.39:0/2443703045 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b81f2f00 con 0x7f67b80c8230 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67b5e00640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67980039a0 0x7f67b81f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.297+0000 7f67ad200640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67b80c8230 0x7f67b815bb20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f67a0011600 tx=0x7f67a0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.298+0000 7f67aea00640 1 -- 172.21.15.39:0/2443703045 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a000f920 con 0x7f67b80c8230 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.298+0000 7f67aea00640 1 -- 172.21.15.39:0/2443703045 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a000fa80 con 0x7f67b80c8230 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.298+0000 7f67aea00640 1 -- 172.21.15.39:0/2443703045 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a000fdf0 con 0x7f67b80c8230 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.298+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b81f31e0 con 0x7f67b80c8230 2024-01-24T17:42:26.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.298+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b80c86b0 con 0x7f67b80c8230 2024-01-24T17:42:26.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.299+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b8073ef0 con 0x7f67b80c8230 2024-01-24T17:42:26.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.300+0000 7f67aea00640 1 -- 172.21.15.39:0/2443703045 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f67a0015460 con 0x7f67b80c8230 2024-01-24T17:42:26.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.301+0000 7f67aea00640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f679006a460 0x7f679006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.301+0000 7f67aea00640 1 -- 172.21.15.39:0/2443703045 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f67a0088d70 con 0x7f67b80c8230 2024-01-24T17:42:26.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.302+0000 7f67b5e00640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f679006a460 0x7f679006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.303+0000 7f67b5e00640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f679006a460 0x7f679006c920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f67a8034000 tx=0x7f67a8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:26.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.306+0000 7f67aea00640 1 -- 172.21.15.39:0/2443703045 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f67a0054330 con 0x7f67b80c8230 2024-01-24T17:42:26.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.500+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f67b80cca20 con 0x7f67b80c8230 2024-01-24T17:42:26.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.501+0000 7f67aea00640 1 -- 172.21.15.39:0/2443703045 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f67a0057950 con 0x7f67b80c8230 2024-01-24T17:42:26.502 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:26.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f679006a460 msgr2=0x7f679006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f679006a460 0x7f679006c920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f67a8034000 tx=0x7f67a8009030 comp rx=0 tx=0).stop 2024-01-24T17:42:26.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67b80c8230 msgr2=0x7f67b815bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67b80c8230 0x7f67b815bb20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f67a0011600 tx=0x7f67a0011ac0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 shutdown_connections 2024-01-24T17:42:26.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f67b80b1ce0 0x7f67b80b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f679006a460 0x7f679006c920 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67b80c8230 0x7f67b815bb20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 --2- 172.21.15.39:0/2443703045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f67980039a0 0x7f67b81f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 >> 172.21.15.39:0/2443703045 conn(0x7f67b80cb0e0 msgr2=0x7f67b80c2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:26.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 shutdown_connections 2024-01-24T17:42:26.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.504+0000 7f67bd000640 1 -- 172.21.15.39:0/2443703045 wait complete. 2024-01-24T17:42:26.523 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:26.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.744+0000 7f3a7ee00640 1 Processor -- start 2024-01-24T17:42:26.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.744+0000 7f3a7ee00640 1 -- start start 2024-01-24T17:42:26.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.746+0000 7f3a7ee00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3a7815ccb0 0x7f3a780cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.746+0000 7f3a7ee00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a7815d090 con 0x7f3a780c9d60 2024-01-24T17:42:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.746+0000 7f3a7ee00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a780cd140 con 0x7f3a780ce390 2024-01-24T17:42:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.746+0000 7f3a7ee00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a780cac20 con 0x7f3a7815ccb0 2024-01-24T17:42:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.746+0000 7f3a7e400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3a780c9d60 0x7f3a780cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47964/0 (socket says 172.21.15.39:47964) 2024-01-24T17:42:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.746+0000 7f3a7e400640 1 -- 172.21.15.39:0/3044639137 learned_addr learned my addr 172.21.15.39:0/3044639137 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a7da00640 1 --2- 172.21.15.39:0/3044639137 >> v2:172.21.15.100:3300/0 conn(0x7f3a7815ccb0 0x7f3a780cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376381497 0 0) 0x7f3a7815d090 con 0x7f3a780c9d60 2024-01-24T17:42:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a58003520 con 0x7f3a780c9d60 2024-01-24T17:42:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626645805 0 0) 0x7f3a58003520 con 0x7f3a780c9d60 2024-01-24T17:42:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 >> v2:172.21.15.100:3300/0 conn(0x7f3a7815ccb0 msgr2=0x7f3a780cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 --2- 172.21.15.39:0/3044639137 >> v2:172.21.15.100:3300/0 conn(0x7f3a7815ccb0 0x7f3a780cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 >> v1:172.21.15.100:6789/0 conn(0x7f3a780ce390 legacy=0x7f3a780cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7806c540 con 0x7f3a780c9d60 2024-01-24T17:42:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f3a700019f0 con 0x7f3a780c9d60 2024-01-24T17:42:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 >> v1:172.21.15.39:6789/0 conn(0x7f3a780c9d60 legacy=0x7f3a780cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a58003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a580047c0 0x7f3a58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.747+0000 7f3a77e00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a58005180 0x7f3a5800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a780cac20 con 0x7f3a580039c0 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5800e290 con 0x7f3a58005180 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5800e450 con 0x7f3a580047c0 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7da00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a58003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7e400640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a58005180 0x7f3a5800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7da00640 1 -- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a580047c0 msgr2=0x7f3a58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7d000640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a580047c0 0x7f3a58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7da00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a580047c0 0x7f3a58004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7da00640 1 -- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a58005180 msgr2=0x7f3a5800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7da00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a58005180 0x7f3a5800db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7da00640 1 -- 172.21.15.39:0/3044639137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a58003520 con 0x7f3a580039c0 2024-01-24T17:42:26.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7e400640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a58005180 0x7f3a5800db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:26.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.748+0000 7f3a7e400640 1 -- 172.21.15.39:0/3044639137 reap_dead start 2024-01-24T17:42:26.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.749+0000 7f3a7da00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a58003de0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009ab0 tx=0x7f3a580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591737b5a2399e98 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.750+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68005990 con 0x7f3a580039c0 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.750+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a68005af0 con 0x7f3a580039c0 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.750+0000 7f3a77e00640 1 -- 172.21.15.39:0/3044639137 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68005ea0 con 0x7f3a580039c0 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.750+0000 7f3a7ee00640 1 -- 172.21.15.39:0/3044639137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 msgr2=0x7f3a58003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.750+0000 7f3a7ee00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a58003de0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009ab0 tx=0x7f3a580057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.750+0000 7f3a7ee00640 1 -- 172.21.15.39:0/3044639137 shutdown_connections 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.750+0000 7f3a7ee00640 1 --2- 172.21.15.39:0/3044639137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a58003de0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.750+0000 7f3a7ee00640 1 -- 172.21.15.39:0/3044639137 >> 172.21.15.39:0/3044639137 conn(0x7f3a78069e50 msgr2=0x7f3a7806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.751+0000 7f3a7ee00640 1 -- 172.21.15.39:0/3044639137 shutdown_connections 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.751+0000 7f3a7ee00640 1 -- 172.21.15.39:0/3044639137 wait complete. 2024-01-24T17:42:26.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.752+0000 7f3a7ee00640 1 Processor -- start 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.753+0000 7f3a7ee00640 1 -- start start 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.753+0000 7f3a7ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a780c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.753+0000 7f3a7ee00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a780c3e00 0x7f3a780cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7ee00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a780cc860 0x7f3a7815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a780ccce0 con 0x7f3a580039c0 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7ee00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a780c6070 con 0x7f3a780c3e00 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7ee00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a780c4330 con 0x7f3a780cc860 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a780c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a780c38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59406/0 (socket says 172.21.15.39:59406) 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7da00640 1 -- 172.21.15.39:0/124707864 learned_addr learned my addr 172.21.15.39:0/124707864 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7d000640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a780c3e00 0x7f3a780cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7da00640 1 -- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a780cc860 msgr2=0x7f3a7815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7da00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a780cc860 0x7f3a7815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7da00640 1 -- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a780c3e00 msgr2=0x7f3a780cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7da00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a780c3e00 0x7f3a780cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.754+0000 7f3a7da00640 1 -- 172.21.15.39:0/124707864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a781f0fd0 con 0x7f3a580039c0 2024-01-24T17:42:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.755+0000 7f3a7da00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a780c38c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3a68036040 tx=0x7f3a68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.755+0000 7f3a75600640 1 -- 172.21.15.39:0/124707864 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68005e70 con 0x7f3a580039c0 2024-01-24T17:42:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.755+0000 7f3a75600640 1 -- 172.21.15.39:0/124707864 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a68004020 con 0x7f3a580039c0 2024-01-24T17:42:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.755+0000 7f3a75600640 1 -- 172.21.15.39:0/124707864 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a680043d0 con 0x7f3a580039c0 2024-01-24T17:42:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.755+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a781f12b0 con 0x7f3a580039c0 2024-01-24T17:42:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.755+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a781f1870 con 0x7f3a580039c0 2024-01-24T17:42:26.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.757+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a780ccc80 con 0x7f3a580039c0 2024-01-24T17:42:26.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.757+0000 7f3a75600640 1 -- 172.21.15.39:0/124707864 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3a68003610 con 0x7f3a580039c0 2024-01-24T17:42:26.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.758+0000 7f3a75600640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a5006a430 0x7f3a5006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:26.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.758+0000 7f3a75600640 1 -- 172.21.15.39:0/124707864 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3a680aaaf0 con 0x7f3a580039c0 2024-01-24T17:42:26.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.759+0000 7f3a7d000640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a5006a430 0x7f3a5006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:26.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.760+0000 7f3a7d000640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a5006a430 0x7f3a5006c8f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3a780cb0f0 tx=0x7f3a60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:26.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.762+0000 7f3a75600640 1 -- 172.21.15.39:0/124707864 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3a780ccc80 con 0x7f3a580039c0 2024-01-24T17:42:26.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.953+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a780ccc80 con 0x7f3a580039c0 2024-01-24T17:42:26.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.954+0000 7f3a75600640 1 -- 172.21.15.39:0/124707864 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a68076210 con 0x7f3a580039c0 2024-01-24T17:42:26.955 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.956+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a5006a430 msgr2=0x7f3a5006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.956+0000 7f3a7ee00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a5006a430 0x7f3a5006c8f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3a780cb0f0 tx=0x7f3a60009040 comp rx=0 tx=0).stop 2024-01-24T17:42:26.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.956+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 msgr2=0x7f3a780c38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.956+0000 7f3a7ee00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a780c38c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3a68036040 tx=0x7f3a68004730 comp rx=0 tx=0).stop 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.957+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 shutdown_connections 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.957+0000 7f3a7ee00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a780cc860 0x7f3a7815aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.957+0000 7f3a7ee00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a780c3e00 0x7f3a780cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.957+0000 7f3a7ee00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a5006a430 0x7f3a5006c8f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.957+0000 7f3a7ee00640 1 --2- 172.21.15.39:0/124707864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a580039c0 0x7f3a780c38c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.957+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 >> 172.21.15.39:0/124707864 conn(0x7f3a78069e50 msgr2=0x7f3a780aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.957+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 shutdown_connections 2024-01-24T17:42:26.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:26.957+0000 7f3a7ee00640 1 -- 172.21.15.39:0/124707864 wait complete. 2024-01-24T17:42:26.976 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:42:27.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.193+0000 7f0508800640 1 Processor -- start 2024-01-24T17:42:27.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.194+0000 7f0508800640 1 -- start start 2024-01-24T17:42:27.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.195+0000 7f0508800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05000cf990 0x7f05000cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.195+0000 7f0508800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f050015fb90 0x7f050015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.195+0000 7f0508800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05000d02b0 con 0x7f050015d020 2024-01-24T17:42:27.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.195+0000 7f0508800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050015e080 con 0x7f050015fb90 2024-01-24T17:42:27.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.195+0000 7f0508800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050015dac0 con 0x7f05000cf990 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0506800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f050015fb90 0x7f050015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0505400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f050015d020 0x7f0500160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47972/0 (socket says 172.21.15.39:47972) 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0505400640 1 -- 172.21.15.39:0/4001000674 learned_addr learned my addr 172.21.15.39:0/4001000674 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0505e00640 1 --2- 172.21.15.39:0/4001000674 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05000cf990 0x7f05000cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0504a00640 1 -- 172.21.15.39:0/4001000674 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570280082 0 0) 0x7f05000d02b0 con 0x7f050015d020 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0504a00640 1 -- 172.21.15.39:0/4001000674 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e0003520 con 0x7f050015d020 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0506800640 1 -- 172.21.15.39:0/4001000674 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05000cf990 msgr2=0x7f05000cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0506800640 1 --2- 172.21.15.39:0/4001000674 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05000cf990 0x7f05000cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0506800640 1 -- 172.21.15.39:0/4001000674 >> v1:172.21.15.39:6789/0 conn(0x7f050015d020 legacy=0x7f0500160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.196+0000 7f0506800640 1 -- 172.21.15.39:0/4001000674 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050015dda0 con 0x7f050015fb90 2024-01-24T17:42:27.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.197+0000 7f0506800640 1 --2- 172.21.15.39:0/4001000674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f050015d460 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009d90 tx=0x7f04f802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d789fb61e9aa944 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:27.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0504a00640 1 -- 172.21.15.39:0/4001000674 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f802cc50 con 0x7f050015fb90 2024-01-24T17:42:27.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0504a00640 1 -- 172.21.15.39:0/4001000674 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f802cdb0 con 0x7f050015fb90 2024-01-24T17:42:27.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0504a00640 1 -- 172.21.15.39:0/4001000674 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f8034660 con 0x7f050015fb90 2024-01-24T17:42:27.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0508800640 1 -- 172.21.15.39:0/4001000674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 msgr2=0x7f050015d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0508800640 1 --2- 172.21.15.39:0/4001000674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f050015d460 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009d90 tx=0x7f04f802bea0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0508800640 1 -- 172.21.15.39:0/4001000674 shutdown_connections 2024-01-24T17:42:27.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0508800640 1 --2- 172.21.15.39:0/4001000674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f050015d460 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0508800640 1 --2- 172.21.15.39:0/4001000674 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f05000cf990 0x7f05000cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0508800640 1 -- 172.21.15.39:0/4001000674 >> 172.21.15.39:0/4001000674 conn(0x7f0500071790 msgr2=0x7f0500071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:27.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0508800640 1 -- 172.21.15.39:0/4001000674 shutdown_connections 2024-01-24T17:42:27.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.198+0000 7f0508800640 1 -- 172.21.15.39:0/4001000674 wait complete. 2024-01-24T17:42:27.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.200+0000 7f0508800640 1 Processor -- start 2024-01-24T17:42:27.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.200+0000 7f0508800640 1 -- start start 2024-01-24T17:42:27.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.201+0000 7f0508800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05000cf990 0x7f05000ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.201+0000 7f0508800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f050015d020 0x7f05000c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.201+0000 7f0508800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f05000c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.201+0000 7f0508800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05000cd240 con 0x7f050015fb90 2024-01-24T17:42:27.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.201+0000 7f0508800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0500149150 con 0x7f05000cf990 2024-01-24T17:42:27.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.201+0000 7f0508800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05000ccc80 con 0x7f050015d020 2024-01-24T17:42:27.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0506800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f05000c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0506800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f05000c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59420/0 (socket says 172.21.15.39:59420) 2024-01-24T17:42:27.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0506800640 1 -- 172.21.15.39:0/2343071846 learned_addr learned my addr 172.21.15.39:0/2343071846 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0505e00640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05000cf990 0x7f05000ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0505400640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f050015d020 0x7f05000c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0506800640 1 -- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f050015d020 msgr2=0x7f05000c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0506800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f050015d020 0x7f05000c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0506800640 1 -- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05000cf990 msgr2=0x7f05000ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0506800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05000cf990 0x7f05000ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0506800640 1 -- 172.21.15.39:0/2343071846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8009a70 con 0x7f050015fb90 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.202+0000 7f0505400640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f050015d020 0x7f05000c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:27.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.203+0000 7f0506800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f05000c8570 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f04f802c4b0 tx=0x7f04f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:27.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.203+0000 7f04fe000640 1 -- 172.21.15.39:0/2343071846 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f803e610 con 0x7f050015fb90 2024-01-24T17:42:27.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.203+0000 7f04fe000640 1 -- 172.21.15.39:0/2343071846 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f803e770 con 0x7f050015fb90 2024-01-24T17:42:27.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.203+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05000ccf60 con 0x7f050015fb90 2024-01-24T17:42:27.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.203+0000 7f04fe000640 1 -- 172.21.15.39:0/2343071846 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f803eb60 con 0x7f050015fb90 2024-01-24T17:42:27.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.203+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05000ca820 con 0x7f050015fb90 2024-01-24T17:42:27.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.204+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05001492e0 con 0x7f050015fb90 2024-01-24T17:42:27.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.205+0000 7f04fe000640 1 -- 172.21.15.39:0/2343071846 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f04f803b050 con 0x7f050015fb90 2024-01-24T17:42:27.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.206+0000 7f04fe000640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04d806a430 0x7f04d806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.206+0000 7f04fe000640 1 -- 172.21.15.39:0/2343071846 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f04f80aaa30 con 0x7f050015fb90 2024-01-24T17:42:27.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.207+0000 7f0505e00640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04d806a430 0x7f04d806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.208+0000 7f0505e00640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04d806a430 0x7f04d806c8f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009a30 tx=0x7f04f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:27.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.211+0000 7f04fe000640 1 -- 172.21.15.39:0/2343071846 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f04f8076090 con 0x7f050015fb90 2024-01-24T17:42:27.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.425+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0500073ef0 con 0x7f050015fb90 2024-01-24T17:42:27.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.426+0000 7f04fe000640 1 -- 172.21.15.39:0/2343071846 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f04f8039080 con 0x7f050015fb90 2024-01-24T17:42:27.426 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.428+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04d806a430 msgr2=0x7f04d806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.428+0000 7f0508800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04d806a430 0x7f04d806c8f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009a30 tx=0x7f04f0008040 comp rx=0 tx=0).stop 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.428+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 msgr2=0x7f05000c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.428+0000 7f0508800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f05000c8570 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f04f802c4b0 tx=0x7f04f8004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.429+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 shutdown_connections 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.429+0000 7f0508800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04d806a430 0x7f04d806c8f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.429+0000 7f0508800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f050015fb90 0x7f05000c8570 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.429+0000 7f0508800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f050015d020 0x7f05000c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.429+0000 7f0508800640 1 --2- 172.21.15.39:0/2343071846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05000cf990 0x7f05000ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.429+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 >> 172.21.15.39:0/2343071846 conn(0x7f0500071790 msgr2=0x7f050015ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.429+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 shutdown_connections 2024-01-24T17:42:27.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.429+0000 7f0508800640 1 -- 172.21.15.39:0/2343071846 wait complete. 2024-01-24T17:42:27.449 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:42:27.449 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:27.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:27.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.734+0000 7f0d81a00640 1 Processor -- start 2024-01-24T17:42:27.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.734+0000 7f0d81a00640 1 -- start start 2024-01-24T17:42:27.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.735+0000 7f0d81a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0d7c0d1b30 0x7f0d7c0b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.735+0000 7f0d81a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0b49c0 0x7f0d7c0b59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.735+0000 7f0d81a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c0b7ce0 con 0x7f0d7c15ca40 2024-01-24T17:42:27.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.735+0000 7f0d81a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d7c0b4df0 con 0x7f0d7c0d1b30 2024-01-24T17:42:27.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.735+0000 7f0d81a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c0b70d0 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.736+0000 7f0d7a000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0d7c0d1b30 0x7f0d7c0b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.736+0000 7f0d7aa00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f0d7c15ca40 0x7f0d7c0b8390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:38866/0 (socket says 172.21.15.39:38866) 2024-01-24T17:42:27.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.736+0000 7f0d7b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0b49c0 0x7f0d7c0b59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.736+0000 7f0d7aa00640 1 -- 172.21.15.39:0/668897767 learned_addr learned my addr 172.21.15.39:0/668897767 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:27.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.737+0000 7f0d7a000640 1 -- 172.21.15.39:0/668897767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0b49c0 msgr2=0x7f0d7c0b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:27.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.737+0000 7f0d7a000640 1 --2- 172.21.15.39:0/668897767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0b49c0 0x7f0d7c0b59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.737+0000 7f0d7a000640 1 -- 172.21.15.39:0/668897767 >> v1:172.21.15.100:6789/0 conn(0x7f0d7c15ca40 legacy=0x7f0d7c0b8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.737+0000 7f0d7a000640 1 -- 172.21.15.39:0/668897767 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0b73b0 con 0x7f0d7c0d1b30 2024-01-24T17:42:27.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.738+0000 7f0d7a000640 1 --2- 172.21.15.39:0/668897767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c0d1b30 0x7f0d7c0b4480 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009d90 tx=0x7f0d6c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aaf3160bceb0131 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:27.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.739+0000 7f0d79600640 1 -- 172.21.15.39:0/668897767 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c02cc60 con 0x7f0d7c0d1b30 2024-01-24T17:42:27.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.739+0000 7f0d79600640 1 -- 172.21.15.39:0/668897767 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c02cdc0 con 0x7f0d7c0d1b30 2024-01-24T17:42:27.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.739+0000 7f0d79600640 1 -- 172.21.15.39:0/668897767 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c034660 con 0x7f0d7c0d1b30 2024-01-24T17:42:27.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.740+0000 7f0d81a00640 1 -- 172.21.15.39:0/668897767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c0d1b30 msgr2=0x7f0d7c0b4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.740+0000 7f0d81a00640 1 --2- 172.21.15.39:0/668897767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c0d1b30 0x7f0d7c0b4480 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c009d90 tx=0x7f0d6c02beb0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.740+0000 7f0d81a00640 1 -- 172.21.15.39:0/668897767 shutdown_connections 2024-01-24T17:42:27.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.740+0000 7f0d81a00640 1 --2- 172.21.15.39:0/668897767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c0d1b30 0x7f0d7c0b4480 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.740+0000 7f0d81a00640 1 --2- 172.21.15.39:0/668897767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0b49c0 0x7f0d7c0b59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.740+0000 7f0d81a00640 1 -- 172.21.15.39:0/668897767 >> 172.21.15.39:0/668897767 conn(0x7f0d7c071790 msgr2=0x7f0d7c071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:27.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.740+0000 7f0d81a00640 1 -- 172.21.15.39:0/668897767 shutdown_connections 2024-01-24T17:42:27.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.740+0000 7f0d81a00640 1 -- 172.21.15.39:0/668897767 wait complete. 2024-01-24T17:42:27.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.742+0000 7f0d81a00640 1 Processor -- start 2024-01-24T17:42:27.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.742+0000 7f0d81a00640 1 -- start start 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d81a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d7c0b49c0 0x7f0d7c0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d81a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0d1b30 0x7f0d7c15e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d81a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c15ca40 0x7f0d7c15eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d81a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c15f330 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d81a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c15f610 con 0x7f0d7c15ca40 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d81a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7c15f8f0 con 0x7f0d7c0d1b30 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d7aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d7c0b49c0 0x7f0d7c0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d7aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d7c0b49c0 0x7f0d7c0c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59432/0 (socket says 172.21.15.39:59432) 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d7aa00640 1 -- 172.21.15.39:0/2038231202 learned_addr learned my addr 172.21.15.39:0/2038231202 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d7b400640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c15ca40 0x7f0d7c15eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d7a000640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0d1b30 0x7f0d7c15e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.744+0000 7f0d7aa00640 1 -- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0d1b30 msgr2=0x7f0d7c15e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d7aa00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0d1b30 0x7f0d7c15e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d7aa00640 1 -- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c15ca40 msgr2=0x7f0d7c15eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d7aa00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c15ca40 0x7f0d7c15eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d7aa00640 1 -- 172.21.15.39:0/2038231202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c009a70 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d7aa00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d7c0b49c0 0x7f0d7c0c8260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400c480 tx=0x7f0d6400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d72a00640 1 -- 172.21.15.39:0/2038231202 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64005ce0 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d72a00640 1 -- 172.21.15.39:0/2038231202 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d64005e40 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d72a00640 1 -- 172.21.15.39:0/2038231202 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d640072f0 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7c15fbd0 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.745+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c15feb0 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.746+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d7c073ef0 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.747+0000 7f0d72a00640 1 -- 172.21.15.39:0/2038231202 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0d64007510 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.748+0000 7f0d72a00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0d5406a430 0x7f0d5406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:27.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.748+0000 7f0d72a00640 1 -- 172.21.15.39:0/2038231202 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0d64087e50 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.749+0000 7f0d7a000640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0d5406a430 0x7f0d5406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:27.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.750+0000 7f0d7a000640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0d5406a430 0x7f0d5406c8f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c036000 tx=0x7f0d6c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:27.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.754+0000 7f0d72a00640 1 -- 172.21.15.39:0/2038231202 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0d640534c0 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.948+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d7c0b7940 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.951+0000 7f0d72a00640 1 -- 172.21.15.39:0/2038231202 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f0d64017090 con 0x7f0d7c0b49c0 2024-01-24T17:42:27.952 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:27.952 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:27.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0d5406a430 msgr2=0x7f0d5406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0d5406a430 0x7f0d5406c8f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c036000 tx=0x7f0d6c030000 comp rx=0 tx=0).stop 2024-01-24T17:42:27.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d7c0b49c0 msgr2=0x7f0d7c0c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:27.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d7c0b49c0 0x7f0d7c0c8260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400c480 tx=0x7f0d6400c940 comp rx=0 tx=0).stop 2024-01-24T17:42:27.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 shutdown_connections 2024-01-24T17:42:27.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d7c15ca40 0x7f0d7c15eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d7c0d1b30 0x7f0d7c15e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0d5406a430 0x7f0d5406c8f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 --2- 172.21.15.39:0/2038231202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d7c0b49c0 0x7f0d7c0c8260 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:27.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.953+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 >> 172.21.15.39:0/2038231202 conn(0x7f0d7c071790 msgr2=0x7f0d7c0b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:27.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.954+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 shutdown_connections 2024-01-24T17:42:27.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:27.954+0000 7f0d81a00640 1 -- 172.21.15.39:0/2038231202 wait complete. 2024-01-24T17:42:27.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:27.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.197+0000 7f8ed0e00640 1 Processor -- start 2024-01-24T17:42:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.197+0000 7f8ed0e00640 1 -- start start 2024-01-24T17:42:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.199+0000 7f8ed0e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc06d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.199+0000 7f8ed0e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8ecc06bfa0 0x7f8ecc0b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.199+0000 7f8ed0e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc06c4d0 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.199+0000 7f8ed0e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc0ca0d0 con 0x7f8ecc06bfa0 2024-01-24T17:42:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.199+0000 7f8ed0e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc06c7b0 con 0x7f8ecc1597f0 2024-01-24T17:42:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.200+0000 7f8ec9e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc06d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.200+0000 7f8eca800640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f8ecc0b00e0 0x7f8ecc0ca4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:38880/0 (socket says 172.21.15.39:38880) 2024-01-24T17:42:28.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.200+0000 7f8eca800640 1 -- 172.21.15.39:0/1841609162 learned_addr learned my addr 172.21.15.39:0/1841609162 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:28.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.200+0000 7f8ec1200640 1 --2- 172.21.15.39:0/1841609162 >> v2:172.21.15.100:3300/0 conn(0x7f8ecc06bfa0 0x7f8ecc0b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.201+0000 7f8ec9400640 1 -- 172.21.15.39:0/1841609162 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572535970 0 0) 0x7f8ecc06c4d0 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.201+0000 7f8ec9400640 1 -- 172.21.15.39:0/1841609162 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eac005730 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.201+0000 7f8ec9e00640 1 -- 172.21.15.39:0/1841609162 >> v2:172.21.15.100:3300/0 conn(0x7f8ecc06bfa0 msgr2=0x7f8ecc0b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.201+0000 7f8ec9e00640 1 --2- 172.21.15.39:0/1841609162 >> v2:172.21.15.100:3300/0 conn(0x7f8ecc06bfa0 0x7f8ecc0b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.201+0000 7f8ec9e00640 1 -- 172.21.15.39:0/1841609162 >> v1:172.21.15.100:6789/0 conn(0x7f8ecc0b00e0 legacy=0x7f8ecc0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.201+0000 7f8ec9e00640 1 -- 172.21.15.39:0/1841609162 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc06ca90 con 0x7f8ecc1597f0 2024-01-24T17:42:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.202+0000 7f8ec9e00640 1 --2- 172.21.15.39:0/1841609162 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc06d890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009d90 tx=0x7f8ebc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74f77ed77574b104 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.202+0000 7f8ec9400640 1 -- 172.21.15.39:0/1841609162 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc02cc50 con 0x7f8ecc1597f0 2024-01-24T17:42:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ec9400640 1 -- 172.21.15.39:0/1841609162 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc02cdb0 con 0x7f8ecc1597f0 2024-01-24T17:42:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ec9400640 1 -- 172.21.15.39:0/1841609162 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc034660 con 0x7f8ecc1597f0 2024-01-24T17:42:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1841609162 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 msgr2=0x7f8ecc06d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1841609162 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc06d890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009d90 tx=0x7f8ebc02bea0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1841609162 shutdown_connections 2024-01-24T17:42:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1841609162 >> v2:172.21.15.100:3300/0 conn(0x7f8ecc06bfa0 0x7f8ecc0b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1841609162 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc06d890 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1841609162 >> 172.21.15.39:0/1841609162 conn(0x7f8ecc0cb0e0 msgr2=0x7f8ecc0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.203+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1841609162 shutdown_connections 2024-01-24T17:42:28.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.204+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1841609162 wait complete. 2024-01-24T17:42:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.206+0000 7f8ed0e00640 1 Processor -- start 2024-01-24T17:42:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.206+0000 7f8ed0e00640 1 -- start start 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.207+0000 7f8ed0e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ecc06bfa0 0x7f8ecc14ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.207+0000 7f8ed0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ecc0b00e0 0x7f8ecc1e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.207+0000 7f8ed0e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc15bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.207+0000 7f8ed0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc1e4fb0 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.207+0000 7f8ed0e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc1f5cf0 con 0x7f8ecc06bfa0 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.207+0000 7f8ed0e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc1f5fd0 con 0x7f8ecc1597f0 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ecc0b00e0 0x7f8ecc1e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ecc0b00e0 0x7f8ecc1e4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59448/0 (socket says 172.21.15.39:59448) 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec1200640 1 -- 172.21.15.39:0/1929893301 learned_addr learned my addr 172.21.15.39:0/1929893301 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec9e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ecc06bfa0 0x7f8ecc14ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8eca800640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc15bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec1200640 1 -- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 msgr2=0x7f8ecc15bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec1200640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc15bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec1200640 1 -- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ecc06bfa0 msgr2=0x7f8ecc14ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec1200640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ecc06bfa0 0x7f8ecc14ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8ec1200640 1 -- 172.21.15.39:0/1929893301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc009a70 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.208+0000 7f8eca800640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc15bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:28.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.209+0000 7f8ec1200640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ecc0b00e0 0x7f8ecc1e4a70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400e980 tx=0x7f8eb400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:28.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.209+0000 7f8ec2a00640 1 -- 172.21.15.39:0/1929893301 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4009670 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.209+0000 7f8ec2a00640 1 -- 172.21.15.39:0/1929893301 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb400ca00 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.209+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc1f62b0 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.209+0000 7f8ec2a00640 1 -- 172.21.15.39:0/1929893301 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400cd90 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.209+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc159bd0 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.210+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc073ef0 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.211+0000 7f8ec2a00640 1 -- 172.21.15.39:0/1929893301 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8eb4013470 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.212+0000 7f8ec2a00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8ea406a430 0x7f8ea406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.212+0000 7f8ec2a00640 1 -- 172.21.15.39:0/1929893301 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8eb4088f40 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.213+0000 7f8ec9e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8ea406a430 0x7f8ea406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.214+0000 7f8ec9e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8ea406a430 0x7f8ea406c8f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc036000 tx=0x7f8ebc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:28.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.216+0000 7f8ec2a00640 1 -- 172.21.15.39:0/1929893301 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8eb4054470 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.414+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ecc0c8540 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.415+0000 7f8ec2a00640 1 -- 172.21.15.39:0/1929893301 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8eb4057a90 con 0x7f8ecc0b00e0 2024-01-24T17:42:28.416 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:28.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8ea406a430 msgr2=0x7f8ea406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8ea406a430 0x7f8ea406c8f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc036000 tx=0x7f8ebc030000 comp rx=0 tx=0).stop 2024-01-24T17:42:28.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ecc0b00e0 msgr2=0x7f8ecc1e4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ecc0b00e0 0x7f8ecc1e4a70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400e980 tx=0x7f8eb400ee40 comp rx=0 tx=0).stop 2024-01-24T17:42:28.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 shutdown_connections 2024-01-24T17:42:28.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8ecc1597f0 0x7f8ecc15bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8ea406a430 0x7f8ea406c8f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ecc0b00e0 0x7f8ecc1e4a70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 --2- 172.21.15.39:0/1929893301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8ecc06bfa0 0x7f8ecc14ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.418+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 >> 172.21.15.39:0/1929893301 conn(0x7f8ecc0cb0e0 msgr2=0x7f8ecc1587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:28.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.419+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 shutdown_connections 2024-01-24T17:42:28.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.419+0000 7f8ed0e00640 1 -- 172.21.15.39:0/1929893301 wait complete. 2024-01-24T17:42:28.438 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:28.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.663+0000 7f2f99a00640 1 Processor -- start 2024-01-24T17:42:28.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.663+0000 7f2f99a00640 1 -- start start 2024-01-24T17:42:28.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.665+0000 7f2f99a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2f940cb540 0x7f2f9406d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.665+0000 7f2f99a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f9406c3a0 0x7f2f940ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.665+0000 7f2f99a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f940ad0b0 con 0x7f2f940accd0 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.665+0000 7f2f99a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f9406d840 con 0x7f2f940cb540 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.665+0000 7f2f99a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940b2370 con 0x7f2f9406c3a0 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.665+0000 7f2f93e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2f940accd0 0x7f2f940cae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47986/0 (socket says 172.21.15.39:47986) 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.665+0000 7f2f93e00640 1 -- 172.21.15.39:0/577753426 learned_addr learned my addr 172.21.15.39:0/577753426 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.665+0000 7f2f8be00640 1 --2- 172.21.15.39:0/577753426 >> v2:172.21.15.39:3300/0 conn(0x7f2f940cb540 0x7f2f9406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.666+0000 7f2f99000640 1 --2- 172.21.15.39:0/577753426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f9406c3a0 0x7f2f940ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.666+0000 7f2f93400640 1 -- 172.21.15.39:0/577753426 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622481450 0 0) 0x7f2f940ad0b0 con 0x7f2f940accd0 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.666+0000 7f2f93400640 1 -- 172.21.15.39:0/577753426 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f74003520 con 0x7f2f940accd0 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.666+0000 7f2f8be00640 1 -- 172.21.15.39:0/577753426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f9406c3a0 msgr2=0x7f2f940ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.666+0000 7f2f8be00640 1 --2- 172.21.15.39:0/577753426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f9406c3a0 0x7f2f940ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.666+0000 7f2f8be00640 1 -- 172.21.15.39:0/577753426 >> v1:172.21.15.39:6789/0 conn(0x7f2f940accd0 legacy=0x7f2f940cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.666+0000 7f2f8be00640 1 -- 172.21.15.39:0/577753426 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f940b2650 con 0x7f2f940cb540 2024-01-24T17:42:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.667+0000 7f2f8be00640 1 --2- 172.21.15.39:0/577753426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f940cb540 0x7f2f9406d300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0097a0 tx=0x7f2f7c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69b5be328f1774c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.667+0000 7f2f93400640 1 -- 172.21.15.39:0/577753426 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c02d040 con 0x7f2f940cb540 2024-01-24T17:42:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f93400640 1 -- 172.21.15.39:0/577753426 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7c031810 con 0x7f2f940cb540 2024-01-24T17:42:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f93400640 1 -- 172.21.15.39:0/577753426 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c031ba0 con 0x7f2f940cb540 2024-01-24T17:42:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f99a00640 1 -- 172.21.15.39:0/577753426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f940cb540 msgr2=0x7f2f9406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f99a00640 1 --2- 172.21.15.39:0/577753426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f940cb540 0x7f2f9406d300 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0097a0 tx=0x7f2f7c02b8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f99a00640 1 -- 172.21.15.39:0/577753426 shutdown_connections 2024-01-24T17:42:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f99a00640 1 --2- 172.21.15.39:0/577753426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f940cb540 0x7f2f9406d300 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f99a00640 1 --2- 172.21.15.39:0/577753426 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f9406c3a0 0x7f2f940ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f99a00640 1 -- 172.21.15.39:0/577753426 >> 172.21.15.39:0/577753426 conn(0x7f2f94069e50 msgr2=0x7f2f9406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f99a00640 1 -- 172.21.15.39:0/577753426 shutdown_connections 2024-01-24T17:42:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.668+0000 7f2f99a00640 1 -- 172.21.15.39:0/577753426 wait complete. 2024-01-24T17:42:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.670+0000 7f2f99a00640 1 Processor -- start 2024-01-24T17:42:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.670+0000 7f2f99a00640 1 -- start start 2024-01-24T17:42:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f99a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f9406c3a0 0x7f2f9415d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f99a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f940accd0 0x7f2f940c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f99a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f940cb540 0x7f2f940ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f99a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940b0800 con 0x7f2f9406c3a0 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f99a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940c1d30 con 0x7f2f940accd0 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f99a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940b0d80 con 0x7f2f940cb540 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f9406c3a0 0x7f2f9415d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f9406c3a0 0x7f2f9415d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59472/0 (socket says 172.21.15.39:59472) 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.671+0000 7f2f93e00640 1 -- 172.21.15.39:0/98700645 learned_addr learned my addr 172.21.15.39:0/98700645 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f8be00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f940accd0 0x7f2f940c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f93e00640 1 -- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f940cb540 msgr2=0x7f2f940ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f93e00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f940cb540 0x7f2f940ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f93e00640 1 -- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f940accd0 msgr2=0x7f2f940c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f93e00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f940accd0 0x7f2f940c78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f93e00640 1 -- 172.21.15.39:0/98700645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c009400 con 0x7f2f9406c3a0 2024-01-24T17:42:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f93e00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f9406c3a0 0x7f2f9415d370 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400c4d0 tx=0x7f2f8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f90c00640 1 -- 172.21.15.39:0/98700645 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84007480 con 0x7f2f9406c3a0 2024-01-24T17:42:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f90c00640 1 -- 172.21.15.39:0/98700645 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f840075e0 con 0x7f2f9406c3a0 2024-01-24T17:42:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.672+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9415ad50 con 0x7f2f9406c3a0 2024-01-24T17:42:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.673+0000 7f2f90c00640 1 -- 172.21.15.39:0/98700645 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f840059c0 con 0x7f2f9406c3a0 2024-01-24T17:42:28.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.673+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f940c19c0 con 0x7f2f9406c3a0 2024-01-24T17:42:28.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.674+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9415aee0 con 0x7f2f9406c3a0 2024-01-24T17:42:28.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.676+0000 7f2f90c00640 1 -- 172.21.15.39:0/98700645 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2f84005b20 con 0x7f2f9406c3a0 2024-01-24T17:42:28.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.677+0000 7f2f90c00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f6c06a430 0x7f2f6c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.677+0000 7f2f90c00640 1 -- 172.21.15.39:0/98700645 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2f84087e50 con 0x7f2f9406c3a0 2024-01-24T17:42:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.678+0000 7f2f8be00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f6c06a430 0x7f2f6c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.679+0000 7f2f8be00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f6c06a430 0x7f2f6c06c8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c036000 tx=0x7f2f7c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:28.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.682+0000 7f2f90c00640 1 -- 172.21.15.39:0/98700645 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2f840534b0 con 0x7f2f9406c3a0 2024-01-24T17:42:28.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.877+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f940cca30 con 0x7f2f9406c3a0 2024-01-24T17:42:28.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.878+0000 7f2f90c00640 1 -- 172.21.15.39:0/98700645 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f840170a0 con 0x7f2f9406c3a0 2024-01-24T17:42:28.879 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f6c06a430 msgr2=0x7f2f6c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f6c06a430 0x7f2f6c06c8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c036000 tx=0x7f2f7c03c040 comp rx=0 tx=0).stop 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f9406c3a0 msgr2=0x7f2f9415d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f9406c3a0 0x7f2f9415d370 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400c4d0 tx=0x7f2f8400c990 comp rx=0 tx=0).stop 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 shutdown_connections 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f940cb540 0x7f2f940ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f940accd0 0x7f2f940c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f6c06a430 0x7f2f6c06c8f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 --2- 172.21.15.39:0/98700645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f9406c3a0 0x7f2f9415d370 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 >> 172.21.15.39:0/98700645 conn(0x7f2f94069e50 msgr2=0x7f2f9415eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 shutdown_connections 2024-01-24T17:42:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:28.881+0000 7f2f99a00640 1 -- 172.21.15.39:0/98700645 wait complete. 2024-01-24T17:42:28.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:29.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.123+0000 7f2b1a200640 1 Processor -- start 2024-01-24T17:42:29.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.124+0000 7f2b1a200640 1 -- start start 2024-01-24T17:42:29.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.125+0000 7f2b1a200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b1415fff0 0x7f2b140b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.125+0000 7f2b1a200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2b140b2a10 0x7f2b140d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.125+0000 7f2b1a200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2b140d12f0 0x7f2b140b4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.125+0000 7f2b1a200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b140d1780 con 0x7f2b140b2a10 2024-01-24T17:42:29.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.125+0000 7f2b1a200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b140b2e90 con 0x7f2b140d12f0 2024-01-24T17:42:29.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.125+0000 7f2b1a200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140cda90 con 0x7f2b1415fff0 2024-01-24T17:42:29.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b12a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2b140b2a10 0x7f2b140d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b12a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2b140b2a10 0x7f2b140d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59482/0 (socket says 172.21.15.39:59482) 2024-01-24T17:42:29.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b12a00640 1 -- 172.21.15.39:0/493990832 learned_addr learned my addr 172.21.15.39:0/493990832 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:29.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b13400640 1 --2- 172.21.15.39:0/493990832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b1415fff0 0x7f2b140b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b13e00640 1 --2- 172.21.15.39:0/493990832 >> v2:172.21.15.100:3300/0 conn(0x7f2b140d12f0 0x7f2b140b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b12a00640 1 -- 172.21.15.39:0/493990832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b1415fff0 msgr2=0x7f2b140b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:29.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b12a00640 1 --2- 172.21.15.39:0/493990832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b1415fff0 0x7f2b140b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b12a00640 1 -- 172.21.15.39:0/493990832 >> v2:172.21.15.100:3300/0 conn(0x7f2b140d12f0 msgr2=0x7f2b140b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:29.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b12a00640 1 --2- 172.21.15.39:0/493990832 >> v2:172.21.15.100:3300/0 conn(0x7f2b140d12f0 0x7f2b140b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.126+0000 7f2b12a00640 1 -- 172.21.15.39:0/493990832 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140d32c0 con 0x7f2b140b2a10 2024-01-24T17:42:29.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.127+0000 7f2b12a00640 1 --2- 172.21.15.39:0/493990832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b140b2a10 0x7f2b140d0db0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2b140b40b0 tx=0x7f2afc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac985cd85b905e50 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:29.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b12000640 1 -- 172.21.15.39:0/493990832 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc031870 con 0x7f2b140b2a10 2024-01-24T17:42:29.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b12000640 1 -- 172.21.15.39:0/493990832 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2afc0319d0 con 0x7f2b140b2a10 2024-01-24T17:42:29.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b12000640 1 -- 172.21.15.39:0/493990832 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc031d60 con 0x7f2b140b2a10 2024-01-24T17:42:29.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b1a200640 1 -- 172.21.15.39:0/493990832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b140b2a10 msgr2=0x7f2b140d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b1a200640 1 --2- 172.21.15.39:0/493990832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b140b2a10 0x7f2b140d0db0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2b140b40b0 tx=0x7f2afc02b910 comp rx=0 tx=0).stop 2024-01-24T17:42:29.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b1a200640 1 -- 172.21.15.39:0/493990832 shutdown_connections 2024-01-24T17:42:29.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b1a200640 1 --2- 172.21.15.39:0/493990832 >> v2:172.21.15.100:3300/0 conn(0x7f2b140d12f0 0x7f2b140b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b1a200640 1 --2- 172.21.15.39:0/493990832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b140b2a10 0x7f2b140d0db0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b1a200640 1 --2- 172.21.15.39:0/493990832 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b1415fff0 0x7f2b140b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.128+0000 7f2b1a200640 1 -- 172.21.15.39:0/493990832 >> 172.21.15.39:0/493990832 conn(0x7f2b1409bcc0 msgr2=0x7f2b140d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:29.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.129+0000 7f2b1a200640 1 -- 172.21.15.39:0/493990832 shutdown_connections 2024-01-24T17:42:29.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.129+0000 7f2b1a200640 1 -- 172.21.15.39:0/493990832 wait complete. 2024-01-24T17:42:29.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.130+0000 7f2b1a200640 1 Processor -- start 2024-01-24T17:42:29.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.131+0000 7f2b1a200640 1 -- start start 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.131+0000 7f2b1a200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b140b2a10 0x7f2b1415b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.131+0000 7f2b1a200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b140d12f0 0x7f2b140c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b1a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b1415fff0 0x7f2b140c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b1a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1415a5a0 con 0x7f2b1415fff0 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b1a200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1415a880 con 0x7f2b140d12f0 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b1a200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1415ab60 con 0x7f2b140b2a10 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b1415fff0 0x7f2b140c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b1415fff0 0x7f2b140c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59496/0 (socket says 172.21.15.39:59496) 2024-01-24T17:42:29.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13e00640 1 -- 172.21.15.39:0/469653190 learned_addr learned my addr 172.21.15.39:0/469653190 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:29.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13400640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b140b2a10 0x7f2b1415b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b12a00640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b140d12f0 0x7f2b140c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13e00640 1 -- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b140b2a10 msgr2=0x7f2b1415b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13e00640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b140b2a10 0x7f2b1415b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13e00640 1 -- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b140d12f0 msgr2=0x7f2b140c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13e00640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b140d12f0 0x7f2b140c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.132+0000 7f2b13e00640 1 -- 172.21.15.39:0/469653190 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc0094d0 con 0x7f2b1415fff0 2024-01-24T17:42:29.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.133+0000 7f2b13e00640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b1415fff0 0x7f2b140c9f00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c00d800 tx=0x7f2b0c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:29.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.133+0000 7f2b0b400640 1 -- 172.21.15.39:0/469653190 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c00b6c0 con 0x7f2b1415fff0 2024-01-24T17:42:29.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.133+0000 7f2b0b400640 1 -- 172.21.15.39:0/469653190 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c00b840 con 0x7f2b1415fff0 2024-01-24T17:42:29.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.133+0000 7f2b0b400640 1 -- 172.21.15.39:0/469653190 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c00bbf0 con 0x7f2b1415fff0 2024-01-24T17:42:29.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.133+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1415ae40 con 0x7f2b1415fff0 2024-01-24T17:42:29.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.133+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1415a110 con 0x7f2b1415fff0 2024-01-24T17:42:29.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.134+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1415b360 con 0x7f2b1415fff0 2024-01-24T17:42:29.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.135+0000 7f2b0b400640 1 -- 172.21.15.39:0/469653190 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2b0c00bd50 con 0x7f2b1415fff0 2024-01-24T17:42:29.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.136+0000 7f2b0b400640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aec06a430 0x7f2aec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.136+0000 7f2b0b400640 1 -- 172.21.15.39:0/469653190 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2b0c088e40 con 0x7f2b1415fff0 2024-01-24T17:42:29.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.137+0000 7f2b13400640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aec06a430 0x7f2aec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.138+0000 7f2b13400640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aec06a430 0x7f2aec06c8f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009940 tx=0x7f2b04003cd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:29.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.141+0000 7f2b0b400640 1 -- 172.21.15.39:0/469653190 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2b1415b360 con 0x7f2b1415fff0 2024-01-24T17:42:29.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.331+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b1415b360 con 0x7f2b1415fff0 2024-01-24T17:42:29.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.331+0000 7f2b0b400640 1 -- 172.21.15.39:0/469653190 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2b1415b360 con 0x7f2b1415fff0 2024-01-24T17:42:29.332 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:29.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aec06a430 msgr2=0x7f2aec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aec06a430 0x7f2aec06c8f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009940 tx=0x7f2b04003cd0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b1415fff0 msgr2=0x7f2b140c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b1415fff0 0x7f2b140c9f00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c00d800 tx=0x7f2b0c00dcc0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 shutdown_connections 2024-01-24T17:42:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aec06a430 0x7f2aec06c8f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b1415fff0 0x7f2b140c9f00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b140d12f0 0x7f2b140c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 --2- 172.21.15.39:0/469653190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b140b2a10 0x7f2b1415b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 >> 172.21.15.39:0/469653190 conn(0x7f2b1409bcc0 msgr2=0x7f2b140b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 shutdown_connections 2024-01-24T17:42:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.334+0000 7f2b1a200640 1 -- 172.21.15.39:0/469653190 wait complete. 2024-01-24T17:42:29.352 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:42:29.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.562+0000 7efdab200640 1 Processor -- start 2024-01-24T17:42:29.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.562+0000 7efdab200640 1 -- start start 2024-01-24T17:42:29.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.563+0000 7efdab200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efdac15c900 0x7efdac0d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.563+0000 7efdab200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac06c2e0 0x7efdac0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.563+0000 7efdab200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdac0d03e0 con 0x7efdac0d2640 2024-01-24T17:42:29.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.563+0000 7efdab200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdac0d0630 con 0x7efdac15c900 2024-01-24T17:42:29.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.563+0000 7efdab200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdac0d0910 con 0x7efdac06c2e0 2024-01-24T17:42:29.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.563+0000 7efdaa800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efdac0d2640 0x7efdac0d2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48002/0 (socket says 172.21.15.39:48002) 2024-01-24T17:42:29.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.563+0000 7efdaa800640 1 -- 172.21.15.39:0/3030780049 learned_addr learned my addr 172.21.15.39:0/3030780049 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:29.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda9400640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac06c2e0 0x7efdac0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda9e00640 1 --2- 172.21.15.39:0/3030780049 >> v2:172.21.15.100:3300/0 conn(0x7efdac15c900 0x7efdac0d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418547283 0 0) 0x7efdac0d03e0 con 0x7efdac0d2640 2024-01-24T17:42:29.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c003520 con 0x7efdac0d2640 2024-01-24T17:42:29.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4216768715 0 0) 0x7efd8c003520 con 0x7efdac0d2640 2024-01-24T17:42:29.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac06c2e0 msgr2=0x7efdac0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:29.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac06c2e0 0x7efdac0cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 >> v2:172.21.15.100:3300/0 conn(0x7efdac15c900 msgr2=0x7efdac0d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 --2- 172.21.15.39:0/3030780049 >> v2:172.21.15.100:3300/0 conn(0x7efdac15c900 0x7efdac0d13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac0d0bf0 con 0x7efdac0d2640 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7efda40019f0 con 0x7efdac0d2640 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 >> v1:172.21.15.39:6789/0 conn(0x7efdac0d2640 legacy=0x7efdac0d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efd8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd8c0047c0 0x7efd8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac06c2e0 0x7efd8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac0d0630 con 0x7efd8c003970 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac0d0910 con 0x7efd8c0047c0 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.564+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c00b980 con 0x7efdac06c2e0 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.565+0000 7efda9e00640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efd8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.565+0000 7efda9e00640 1 -- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac06c2e0 msgr2=0x7efd8c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:29.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.565+0000 7efda9e00640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac06c2e0 0x7efd8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.565+0000 7efda9e00640 1 -- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd8c0047c0 msgr2=0x7efd8c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:29.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.565+0000 7efda9e00640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd8c0047c0 0x7efd8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.565+0000 7efda9e00640 1 -- 172.21.15.39:0/3030780049 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c003520 con 0x7efd8c003970 2024-01-24T17:42:29.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.566+0000 7efda9e00640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efd8c003d90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efd9c009ab0 tx=0x7efd8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8916bede44411e55 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:29.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.566+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c005990 con 0x7efd8c003970 2024-01-24T17:42:29.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.566+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9c005af0 con 0x7efd8c003970 2024-01-24T17:42:29.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.566+0000 7efda8a00640 1 -- 172.21.15.39:0/3030780049 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c005ea0 con 0x7efd8c003970 2024-01-24T17:42:29.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.567+0000 7efdab200640 1 -- 172.21.15.39:0/3030780049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 msgr2=0x7efd8c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.567+0000 7efdab200640 1 --2- 172.21.15.39:0/3030780049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efd8c003d90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efd9c009ab0 tx=0x7efd8c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.567+0000 7efdaa800640 1 -- 172.21.15.39:0/3030780049 reap_dead start 2024-01-24T17:42:29.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.567+0000 7efdab200640 1 -- 172.21.15.39:0/3030780049 shutdown_connections 2024-01-24T17:42:29.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.567+0000 7efdab200640 1 -- 172.21.15.39:0/3030780049 >> 172.21.15.39:0/3030780049 conn(0x7efdac09bb30 msgr2=0x7efdac089fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:29.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.567+0000 7efdab200640 1 -- 172.21.15.39:0/3030780049 shutdown_connections 2024-01-24T17:42:29.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.567+0000 7efdab200640 1 -- 172.21.15.39:0/3030780049 wait complete. 2024-01-24T17:42:29.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.569+0000 7efdab200640 1 Processor -- start 2024-01-24T17:42:29.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.569+0000 7efdab200640 1 -- start start 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efdab200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efdac1e6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efdab200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac1e5220 0x7efdac15a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efdab200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdac15b070 0x7efdac15a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efdab200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac15aea0 con 0x7efd8c003970 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efdab200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac1e5720 con 0x7efdac15b070 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efdab200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdac0b6570 con 0x7efdac1e5220 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efdac1e6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efdac1e6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59510/0 (socket says 172.21.15.39:59510) 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9e00640 1 -- 172.21.15.39:0/4017663537 learned_addr learned my addr 172.21.15.39:0/4017663537 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9400640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac1e5220 0x7efdac15a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9e00640 1 -- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac1e5220 msgr2=0x7efdac15a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9e00640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac1e5220 0x7efdac15a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efdaa800640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdac15b070 0x7efdac15a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9e00640 1 -- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdac15b070 msgr2=0x7efdac15a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9e00640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdac15b070 0x7efdac15a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.570+0000 7efda9e00640 1 -- 172.21.15.39:0/4017663537 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac0b6850 con 0x7efd8c003970 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.571+0000 7efda9e00640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efdac1e6160 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efd9c036040 tx=0x7efd9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.571+0000 7efda2000640 1 -- 172.21.15.39:0/4017663537 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c005e70 con 0x7efd8c003970 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.571+0000 7efda2000640 1 -- 172.21.15.39:0/4017663537 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9c004020 con 0x7efd8c003970 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.571+0000 7efda2000640 1 -- 172.21.15.39:0/4017663537 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9c0043d0 con 0x7efd8c003970 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.571+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdac0b6b30 con 0x7efd8c003970 2024-01-24T17:42:29.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.571+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdac15b780 con 0x7efd8c003970 2024-01-24T17:42:29.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.572+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdac15b910 con 0x7efd8c003970 2024-01-24T17:42:29.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.573+0000 7efda2000640 1 -- 172.21.15.39:0/4017663537 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efd9c003610 con 0x7efd8c003970 2024-01-24T17:42:29.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.574+0000 7efda2000640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd8406a460 0x7efd8406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:29.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.574+0000 7efda2000640 1 -- 172.21.15.39:0/4017663537 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efd9c0aab50 con 0x7efd8c003970 2024-01-24T17:42:29.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.575+0000 7efda9400640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd8406a460 0x7efd8406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:29.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.576+0000 7efda9400640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd8406a460 0x7efd8406c920 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efdac0c9360 tx=0x7efd94008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:29.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.579+0000 7efda2000640 1 -- 172.21.15.39:0/4017663537 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efd9c076120 con 0x7efd8c003970 2024-01-24T17:42:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.776+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efdac15aea0 con 0x7efd8c003970 2024-01-24T17:42:29.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.777+0000 7efda2000640 1 -- 172.21.15.39:0/4017663537 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7efd9c079740 con 0x7efd8c003970 2024-01-24T17:42:29.778 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:42:29.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.779+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd8406a460 msgr2=0x7efd8406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.779+0000 7efdab200640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd8406a460 0x7efd8406c920 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efdac0c9360 tx=0x7efd94008040 comp rx=0 tx=0).stop 2024-01-24T17:42:29.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.779+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 msgr2=0x7efdac1e6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:29.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.779+0000 7efdab200640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efdac1e6160 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efd9c036040 tx=0x7efd9c004730 comp rx=0 tx=0).stop 2024-01-24T17:42:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.780+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 shutdown_connections 2024-01-24T17:42:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.780+0000 7efdab200640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdac15b070 0x7efdac15a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.780+0000 7efdab200640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdac1e5220 0x7efdac15a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.780+0000 7efdab200640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd8406a460 0x7efd8406c920 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.780+0000 7efdab200640 1 --2- 172.21.15.39:0/4017663537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd8c003970 0x7efdac1e6160 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.780+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 >> 172.21.15.39:0/4017663537 conn(0x7efdac09bb30 msgr2=0x7efdac0c6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.780+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 shutdown_connections 2024-01-24T17:42:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:29.780+0000 7efdab200640 1 -- 172.21.15.39:0/4017663537 wait complete. 2024-01-24T17:42:29.798 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:42:29.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:29.798 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:42:29.898 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:42:29.898 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:42:29.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:29.899 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:42:29.959 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:42:29.959 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:30.015 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:42:30.015 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:30.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:30.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.271+0000 7f0b5aa00640 1 Processor -- start 2024-01-24T17:42:30.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.271+0000 7f0b5aa00640 1 -- start start 2024-01-24T17:42:30.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.272+0000 7f0b5aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.272+0000 7f0b5aa00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0b540710e0 0x7f0b540affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.272+0000 7f0b5aa00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b540cad00 con 0x7f0b54159a00 2024-01-24T17:42:30.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.272+0000 7f0b5aa00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b54071650 con 0x7f0b540710e0 2024-01-24T17:42:30.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.272+0000 7f0b5aa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b54071930 con 0x7f0b54159620 2024-01-24T17:42:30.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.273+0000 7f0b53e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.273+0000 7f0b53e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540ca7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:36916/0 (socket says 172.21.15.39:36916) 2024-01-24T17:42:30.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.273+0000 7f0b53e00640 1 -- 172.21.15.39:0/3528364893 learned_addr learned my addr 172.21.15.39:0/3528364893 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:30.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.273+0000 7f0b53400640 1 --2- 172.21.15.39:0/3528364893 >> v2:172.21.15.100:3300/0 conn(0x7f0b540710e0 0x7f0b540affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.274+0000 7f0b53e00640 1 -- 172.21.15.39:0/3528364893 >> v2:172.21.15.100:3300/0 conn(0x7f0b540710e0 msgr2=0x7f0b540affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:30.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.274+0000 7f0b53e00640 1 --2- 172.21.15.39:0/3528364893 >> v2:172.21.15.100:3300/0 conn(0x7f0b540710e0 0x7f0b540affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.274+0000 7f0b53e00640 1 -- 172.21.15.39:0/3528364893 >> v1:172.21.15.100:6789/0 conn(0x7f0b54159a00 legacy=0x7f0b540c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.274+0000 7f0b53e00640 1 -- 172.21.15.39:0/3528364893 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b54071c10 con 0x7f0b54159620 2024-01-24T17:42:30.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.275+0000 7f0b53e00640 1 --2- 172.21.15.39:0/3528364893 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540ca7c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b44009e50 tx=0x7f0b4402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53cca7b4f2fb75e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:30.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.276+0000 7f0b52a00640 1 -- 172.21.15.39:0/3528364893 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4402cb00 con 0x7f0b54159620 2024-01-24T17:42:30.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.276+0000 7f0b52a00640 1 -- 172.21.15.39:0/3528364893 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4402cc60 con 0x7f0b54159620 2024-01-24T17:42:30.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.276+0000 7f0b52a00640 1 -- 172.21.15.39:0/3528364893 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b440343f0 con 0x7f0b54159620 2024-01-24T17:42:30.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.277+0000 7f0b5aa00640 1 -- 172.21.15.39:0/3528364893 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 msgr2=0x7f0b540ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.277+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/3528364893 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540ca7c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0b44009e50 tx=0x7f0b4402bf60 comp rx=0 tx=0).stop 2024-01-24T17:42:30.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.277+0000 7f0b5aa00640 1 -- 172.21.15.39:0/3528364893 shutdown_connections 2024-01-24T17:42:30.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.277+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/3528364893 >> v2:172.21.15.100:3300/0 conn(0x7f0b540710e0 0x7f0b540affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.277+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/3528364893 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540ca7c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.277+0000 7f0b5aa00640 1 -- 172.21.15.39:0/3528364893 >> 172.21.15.39:0/3528364893 conn(0x7f0b5406a3e0 msgr2=0x7f0b5406bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:30.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.277+0000 7f0b5aa00640 1 -- 172.21.15.39:0/3528364893 shutdown_connections 2024-01-24T17:42:30.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.277+0000 7f0b5aa00640 1 -- 172.21.15.39:0/3528364893 wait complete. 2024-01-24T17:42:30.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.279+0000 7f0b5aa00640 1 Processor -- start 2024-01-24T17:42:30.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.279+0000 7f0b5aa00640 1 -- start start 2024-01-24T17:42:30.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.280+0000 7f0b5aa00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b540710e0 0x7f0b541ef720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.280+0000 7f0b5aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540c2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.280+0000 7f0b5aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b54159a00 0x7f0b540c2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.280+0000 7f0b5aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b541f1080 con 0x7f0b54159a00 2024-01-24T17:42:30.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.280+0000 7f0b5aa00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b541f0d10 con 0x7f0b540710e0 2024-01-24T17:42:30.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.280+0000 7f0b5aa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b541f12d0 con 0x7f0b54159620 2024-01-24T17:42:30.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.280+0000 7f0b58a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b54159a00 0x7f0b540c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b58a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b54159a00 0x7f0b540c2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59526/0 (socket says 172.21.15.39:59526) 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b58a00640 1 -- 172.21.15.39:0/993917944 learned_addr learned my addr 172.21.15.39:0/993917944 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b53e00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b540710e0 0x7f0b541ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b53400640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b58a00640 1 -- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 msgr2=0x7f0b540c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b58a00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540c2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b58a00640 1 -- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b540710e0 msgr2=0x7f0b541ef720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b58a00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b540710e0 0x7f0b541ef720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b58a00640 1 -- 172.21.15.39:0/993917944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b44009b30 con 0x7f0b54159a00 2024-01-24T17:42:30.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b53400640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:30.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b53e00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b540710e0 0x7f0b541ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:30.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.281+0000 7f0b58a00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b54159a00 0x7f0b540c2570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00c390 tx=0x7f0b4c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:30.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.282+0000 7f0b4be00640 1 -- 172.21.15.39:0/993917944 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c0052d0 con 0x7f0b54159a00 2024-01-24T17:42:30.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.282+0000 7f0b4be00640 1 -- 172.21.15.39:0/993917944 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c005430 con 0x7f0b54159a00 2024-01-24T17:42:30.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.282+0000 7f0b4be00640 1 -- 172.21.15.39:0/993917944 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c003810 con 0x7f0b54159a00 2024-01-24T17:42:30.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.282+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b541f15b0 con 0x7f0b54159a00 2024-01-24T17:42:30.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.282+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b541efc60 con 0x7f0b54159a00 2024-01-24T17:42:30.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.283+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5406c840 con 0x7f0b54159a00 2024-01-24T17:42:30.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.284+0000 7f0b4be00640 1 -- 172.21.15.39:0/993917944 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0b4c003970 con 0x7f0b54159a00 2024-01-24T17:42:30.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.285+0000 7f0b4be00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b2c06a430 0x7f0b2c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.285+0000 7f0b4be00640 1 -- 172.21.15.39:0/993917944 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0b4c087dc0 con 0x7f0b54159a00 2024-01-24T17:42:30.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.286+0000 7f0b53e00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b2c06a430 0x7f0b2c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.287+0000 7f0b53e00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b2c06a430 0x7f0b2c06c8f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0b44035000 tx=0x7f0b44030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.289+0000 7f0b4be00640 1 -- 172.21.15.39:0/993917944 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0b4c053420 con 0x7f0b54159a00 2024-01-24T17:42:30.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.486+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b541f1ad0 con 0x7f0b54159a00 2024-01-24T17:42:30.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.488+0000 7f0b4be00640 1 -- 172.21.15.39:0/993917944 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f0b541f1ad0 con 0x7f0b54159a00 2024-01-24T17:42:30.489 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:30.489 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:30.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.490+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b2c06a430 msgr2=0x7f0b2c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.490+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b2c06a430 0x7f0b2c06c8f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0b44035000 tx=0x7f0b44030000 comp rx=0 tx=0).stop 2024-01-24T17:42:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.490+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b54159a00 msgr2=0x7f0b540c2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.490+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b54159a00 0x7f0b540c2570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00c390 tx=0x7f0b4c00c850 comp rx=0 tx=0).stop 2024-01-24T17:42:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.491+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 shutdown_connections 2024-01-24T17:42:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.491+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b2c06a430 0x7f0b2c06c8f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.491+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b54159a00 0x7f0b540c2570 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.491+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0b54159620 0x7f0b540c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.491+0000 7f0b5aa00640 1 --2- 172.21.15.39:0/993917944 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0b540710e0 0x7f0b541ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.491+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 >> 172.21.15.39:0/993917944 conn(0x7f0b5406a3e0 msgr2=0x7f0b540adea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:30.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.491+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 shutdown_connections 2024-01-24T17:42:30.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.491+0000 7f0b5aa00640 1 -- 172.21.15.39:0/993917944 wait complete. 2024-01-24T17:42:30.511 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:30.513 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:30.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.739+0000 7fa7b6000640 1 Processor -- start 2024-01-24T17:42:30.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.739+0000 7fa7b6000640 1 -- start start 2024-01-24T17:42:30.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.740+0000 7fa7b6000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa7b015ccb0 0x7fa7b00cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.740+0000 7fa7b6000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa7b00c9d60 0x7fa7b00ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.740+0000 7fa7b6000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b015d090 con 0x7fa7b00ce390 2024-01-24T17:42:30.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.740+0000 7fa7b6000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b00cac20 con 0x7fa7b00c9d60 2024-01-24T17:42:30.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.740+0000 7fa7b6000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b00ced60 con 0x7fa7b015ccb0 2024-01-24T17:42:30.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7b5600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa7b00c9d60 0x7fa7b00ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7a7c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa7b00ce390 0x7fa7b00cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48014/0 (socket says 172.21.15.39:48014) 2024-01-24T17:42:30.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7a7c00640 1 -- 172.21.15.39:0/532566017 learned_addr learned my addr 172.21.15.39:0/532566017 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:30.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7b4c00640 1 --2- 172.21.15.39:0/532566017 >> v2:172.21.15.100:3300/0 conn(0x7fa7b015ccb0 0x7fa7b00cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7b5600640 1 -- 172.21.15.39:0/532566017 >> v2:172.21.15.100:3300/0 conn(0x7fa7b015ccb0 msgr2=0x7fa7b00cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:30.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7b5600640 1 --2- 172.21.15.39:0/532566017 >> v2:172.21.15.100:3300/0 conn(0x7fa7b015ccb0 0x7fa7b00cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7b5600640 1 -- 172.21.15.39:0/532566017 >> v1:172.21.15.39:6789/0 conn(0x7fa7b00ce390 legacy=0x7fa7b00cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7b5600640 1 -- 172.21.15.39:0/532566017 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b006c540 con 0x7fa7b00c9d60 2024-01-24T17:42:30.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.741+0000 7fa7afe00640 1 -- 172.21.15.39:0/532566017 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10583456 0 0) 0x7fa7b015d090 con 0x7fa7b00ce390 2024-01-24T17:42:30.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.742+0000 7fa7b5600640 1 --2- 172.21.15.39:0/532566017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b00c9d60 0x7fa7b00ce820 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00cda20 tx=0x7fa7a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bab8930c6cc2663 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:30.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.743+0000 7fa7afe00640 1 -- 172.21.15.39:0/532566017 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a802cc80 con 0x7fa7b00c9d60 2024-01-24T17:42:30.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.743+0000 7fa7afe00640 1 -- 172.21.15.39:0/532566017 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a802cde0 con 0x7fa7b00c9d60 2024-01-24T17:42:30.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.743+0000 7fa7afe00640 1 -- 172.21.15.39:0/532566017 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8034660 con 0x7fa7b00c9d60 2024-01-24T17:42:30.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.743+0000 7fa7b6000640 1 -- 172.21.15.39:0/532566017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b00c9d60 msgr2=0x7fa7b00ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.743+0000 7fa7b6000640 1 --2- 172.21.15.39:0/532566017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b00c9d60 0x7fa7b00ce820 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00cda20 tx=0x7fa7a802bed0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.743+0000 7fa7b6000640 1 -- 172.21.15.39:0/532566017 shutdown_connections 2024-01-24T17:42:30.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.743+0000 7fa7b6000640 1 --2- 172.21.15.39:0/532566017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b00c9d60 0x7fa7b00ce820 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.743+0000 7fa7b6000640 1 --2- 172.21.15.39:0/532566017 >> v2:172.21.15.100:3300/0 conn(0x7fa7b015ccb0 0x7fa7b00cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.744+0000 7fa7b6000640 1 -- 172.21.15.39:0/532566017 >> 172.21.15.39:0/532566017 conn(0x7fa7b0069e50 msgr2=0x7fa7b006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:30.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.744+0000 7fa7b6000640 1 -- 172.21.15.39:0/532566017 shutdown_connections 2024-01-24T17:42:30.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.744+0000 7fa7b6000640 1 -- 172.21.15.39:0/532566017 wait complete. 2024-01-24T17:42:30.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.745+0000 7fa7b6000640 1 Processor -- start 2024-01-24T17:42:30.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.745+0000 7fa7b6000640 1 -- start start 2024-01-24T17:42:30.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.746+0000 7fa7b6000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7b00c9d60 0x7fa7b00c2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.746+0000 7fa7b6000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7b00ce390 0x7fa7b00caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.746+0000 7fa7b6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b015ccb0 0x7fa7b00cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.746+0000 7fa7b6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00c5980 con 0x7fa7b015ccb0 2024-01-24T17:42:30.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.746+0000 7fa7b6000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00c5c60 con 0x7fa7b00ce390 2024-01-24T17:42:30.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.746+0000 7fa7b6000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00c5f40 con 0x7fa7b00c9d60 2024-01-24T17:42:30.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b015ccb0 0x7fa7b00cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b015ccb0 0x7fa7b00cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59556/0 (socket says 172.21.15.39:59556) 2024-01-24T17:42:30.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b5600640 1 -- 172.21.15.39:0/2763674091 learned_addr learned my addr 172.21.15.39:0/2763674091 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7a7c00640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7b00ce390 0x7fa7b00caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b4c00640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7b00c9d60 0x7fa7b00c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b5600640 1 -- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7b00c9d60 msgr2=0x7fa7b00c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b5600640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7b00c9d60 0x7fa7b00c2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b5600640 1 -- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7b00ce390 msgr2=0x7fa7b00caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b5600640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7b00ce390 0x7fa7b00caea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.747+0000 7fa7b5600640 1 -- 172.21.15.39:0/2763674091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8009a70 con 0x7fa7b015ccb0 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.748+0000 7fa7b5600640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b015ccb0 0x7fa7b00cb3e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009e80 tx=0x7fa7a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:30.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.748+0000 7fa7ad600640 1 -- 172.21.15.39:0/2763674091 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8040650 con 0x7fa7b015ccb0 2024-01-24T17:42:30.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.748+0000 7fa7ad600640 1 -- 172.21.15.39:0/2763674091 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a80407b0 con 0x7fa7b015ccb0 2024-01-24T17:42:30.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.748+0000 7fa7ad600640 1 -- 172.21.15.39:0/2763674091 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8040b60 con 0x7fa7b015ccb0 2024-01-24T17:42:30.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.748+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b00c6220 con 0x7fa7b015ccb0 2024-01-24T17:42:30.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.748+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b00c69d0 con 0x7fa7b015ccb0 2024-01-24T17:42:30.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.749+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b00c6b60 con 0x7fa7b015ccb0 2024-01-24T17:42:30.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.750+0000 7fa7ad600640 1 -- 172.21.15.39:0/2763674091 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa7a8040cc0 con 0x7fa7b015ccb0 2024-01-24T17:42:30.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.751+0000 7fa7ad600640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78806a430 0x7fa78806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:30.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.751+0000 7fa7ad600640 1 -- 172.21.15.39:0/2763674091 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa7a80ab740 con 0x7fa7b015ccb0 2024-01-24T17:42:30.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.752+0000 7fa7b4c00640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78806a430 0x7fa78806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:30.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.753+0000 7fa7b4c00640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78806a430 0x7fa78806c8f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00096d0 tx=0x7fa7a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:30.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.755+0000 7fa7ad600640 1 -- 172.21.15.39:0/2763674091 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa7a8076d00 con 0x7fa7b015ccb0 2024-01-24T17:42:30.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.949+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa7b004dff0 con 0x7fa7b015ccb0 2024-01-24T17:42:30.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.950+0000 7fa7ad600640 1 -- 172.21.15.39:0/2763674091 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa7a8009450 con 0x7fa7b015ccb0 2024-01-24T17:42:30.951 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78806a430 msgr2=0x7fa78806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78806a430 0x7fa78806c8f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00096d0 tx=0x7fa7a0008040 comp rx=0 tx=0).stop 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b015ccb0 msgr2=0x7fa7b00cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b015ccb0 0x7fa7b00cb3e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009e80 tx=0x7fa7a8004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 shutdown_connections 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78806a430 0x7fa78806c8f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b015ccb0 0x7fa7b00cb3e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7b00ce390 0x7fa7b00caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 --2- 172.21.15.39:0/2763674091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7b00c9d60 0x7fa7b00c2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 >> 172.21.15.39:0/2763674091 conn(0x7fa7b0069e50 msgr2=0x7fa7b00ca140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 shutdown_connections 2024-01-24T17:42:30.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:30.953+0000 7fa7b6000640 1 -- 172.21.15.39:0/2763674091 wait complete. 2024-01-24T17:42:30.972 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:31.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.190+0000 7f19c6c00640 1 Processor -- start 2024-01-24T17:42:31.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.191+0000 7f19c6c00640 1 -- start start 2024-01-24T17:42:31.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.192+0000 7f19c6c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f19c015c9e0 0x7f19c00b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.192+0000 7f19c6c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f19c00b8250 0x7f19c00cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.192+0000 7f19c6c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c00b7e30 con 0x7f19c00cdb90 2024-01-24T17:42:31.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.192+0000 7f19c6c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c00ce040 con 0x7f19c00b8250 2024-01-24T17:42:31.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.192+0000 7f19c6c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c00b6d90 con 0x7f19c015c9e0 2024-01-24T17:42:31.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.193+0000 7f19c4e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f19c00b8250 0x7f19c00cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.193+0000 7f19c4e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f19c00b8250 0x7f19c00cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59568/0 (socket says 172.21.15.39:59568) 2024-01-24T17:42:31.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.193+0000 7f19c4e00640 1 -- 172.21.15.39:0/1365214227 learned_addr learned my addr 172.21.15.39:0/1365214227 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:31.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.193+0000 7f19c5800640 1 --2- 172.21.15.39:0/1365214227 >> v2:172.21.15.100:3300/0 conn(0x7f19c015c9e0 0x7f19c00b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.194+0000 7f19c4e00640 1 -- 172.21.15.39:0/1365214227 >> v2:172.21.15.100:3300/0 conn(0x7f19c015c9e0 msgr2=0x7f19c00b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:31.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.194+0000 7f19c4e00640 1 --2- 172.21.15.39:0/1365214227 >> v2:172.21.15.100:3300/0 conn(0x7f19c015c9e0 0x7f19c00b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.194+0000 7f19c4e00640 1 -- 172.21.15.39:0/1365214227 >> v1:172.21.15.100:6789/0 conn(0x7f19c00cdb90 legacy=0x7f19c00b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.194+0000 7f19c4e00640 1 -- 172.21.15.39:0/1365214227 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c00b7070 con 0x7f19c00b8250 2024-01-24T17:42:31.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.194+0000 7f19c5800640 1 --2- 172.21.15.39:0/1365214227 >> v2:172.21.15.100:3300/0 conn(0x7f19c015c9e0 0x7f19c00b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:31.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.195+0000 7f19c4e00640 1 --2- 172.21.15.39:0/1365214227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00cd650 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f19c00cfe40 tx=0x7f19b002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1b0ca52d4dc994 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:31.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.195+0000 7f19bfe00640 1 -- 172.21.15.39:0/1365214227 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b002cbb0 con 0x7f19c00b8250 2024-01-24T17:42:31.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.195+0000 7f19bfe00640 1 -- 172.21.15.39:0/1365214227 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b002cd10 con 0x7f19c00b8250 2024-01-24T17:42:31.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.195+0000 7f19bfe00640 1 -- 172.21.15.39:0/1365214227 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0034610 con 0x7f19c00b8250 2024-01-24T17:42:31.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.196+0000 7f19c6c00640 1 -- 172.21.15.39:0/1365214227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 msgr2=0x7f19c00cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.196+0000 7f19c6c00640 1 --2- 172.21.15.39:0/1365214227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00cd650 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f19c00cfe40 tx=0x7f19b002c010 comp rx=0 tx=0).stop 2024-01-24T17:42:31.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.196+0000 7f19c6c00640 1 -- 172.21.15.39:0/1365214227 shutdown_connections 2024-01-24T17:42:31.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.196+0000 7f19c6c00640 1 --2- 172.21.15.39:0/1365214227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00cd650 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.196+0000 7f19c6c00640 1 --2- 172.21.15.39:0/1365214227 >> v2:172.21.15.100:3300/0 conn(0x7f19c015c9e0 0x7f19c00b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.196+0000 7f19c6c00640 1 -- 172.21.15.39:0/1365214227 >> 172.21.15.39:0/1365214227 conn(0x7f19c0071790 msgr2=0x7f19c0071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:31.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.196+0000 7f19c6c00640 1 -- 172.21.15.39:0/1365214227 shutdown_connections 2024-01-24T17:42:31.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.196+0000 7f19c6c00640 1 -- 172.21.15.39:0/1365214227 wait complete. 2024-01-24T17:42:31.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.198+0000 7f19c6c00640 1 Processor -- start 2024-01-24T17:42:31.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.198+0000 7f19c6c00640 1 -- start start 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c6c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c6c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19c00cdb90 0x7f19c00d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c6c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19c015c9e0 0x7f19c00d1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c6c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c014ccf0 con 0x7f19c00b8250 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c6c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0156f30 con 0x7f19c00cdb90 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c6c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c014c0e0 con 0x7f19c015c9e0 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59574/0 (socket says 172.21.15.39:59574) 2024-01-24T17:42:31.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.199+0000 7f19c5800640 1 -- 172.21.15.39:0/3319837254 learned_addr learned my addr 172.21.15.39:0/3319837254 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c4e00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19c00cdb90 0x7f19c00d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c6200640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19c015c9e0 0x7f19c00d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c5800640 1 -- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19c015c9e0 msgr2=0x7f19c00d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c5800640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19c015c9e0 0x7f19c00d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c5800640 1 -- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19c00cdb90 msgr2=0x7f19c00d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c5800640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19c00cdb90 0x7f19c00d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c5800640 1 -- 172.21.15.39:0/3319837254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b0009b30 con 0x7f19c00b8250 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c5800640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00c8cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f19a800d7d0 tx=0x7f19a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19bd600640 1 -- 172.21.15.39:0/3319837254 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a8004030 con 0x7f19c00b8250 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19bd600640 1 -- 172.21.15.39:0/3319837254 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a8004190 con 0x7f19c00b8250 2024-01-24T17:42:31.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19bd600640 1 -- 172.21.15.39:0/3319837254 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a8004520 con 0x7f19c00b8250 2024-01-24T17:42:31.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.200+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c014c3c0 con 0x7f19c00b8250 2024-01-24T17:42:31.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.201+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c014c980 con 0x7f19c00b8250 2024-01-24T17:42:31.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.202+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c0067040 con 0x7f19c00b8250 2024-01-24T17:42:31.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.203+0000 7f19bd600640 1 -- 172.21.15.39:0/3319837254 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f19a800b6c0 con 0x7f19c00b8250 2024-01-24T17:42:31.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.204+0000 7f19bd600640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f199806a430 0x7f199806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.204+0000 7f19bd600640 1 -- 172.21.15.39:0/3319837254 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f19a8088b70 con 0x7f19c00b8250 2024-01-24T17:42:31.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.204+0000 7f19c4e00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f199806a430 0x7f199806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.205+0000 7f19c4e00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f199806a430 0x7f199806c8f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f19b0035000 tx=0x7f19b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:31.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.208+0000 7f19bd600640 1 -- 172.21.15.39:0/3319837254 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f19a8054140 con 0x7f19c00b8250 2024-01-24T17:42:31.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.405+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f19c00b7320 con 0x7f19c00b8250 2024-01-24T17:42:31.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.410+0000 7f19bd600640 1 -- 172.21.15.39:0/3319837254 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f19a8057760 con 0x7f19c00b8250 2024-01-24T17:42:31.411 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:31.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f199806a430 msgr2=0x7f199806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f199806a430 0x7f199806c8f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f19b0035000 tx=0x7f19b0030000 comp rx=0 tx=0).stop 2024-01-24T17:42:31.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 msgr2=0x7f19c00c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00c8cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f19a800d7d0 tx=0x7f19a800dc90 comp rx=0 tx=0).stop 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 shutdown_connections 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19c015c9e0 0x7f19c00d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19c00cdb90 0x7f19c00d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f199806a430 0x7f199806c8f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 --2- 172.21.15.39:0/3319837254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00b8250 0x7f19c00c8cc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.413+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 >> 172.21.15.39:0/3319837254 conn(0x7f19c0071790 msgr2=0x7f19c006ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.414+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 shutdown_connections 2024-01-24T17:42:31.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.414+0000 7f19c6c00640 1 -- 172.21.15.39:0/3319837254 wait complete. 2024-01-24T17:42:31.435 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:31.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.659+0000 7fdf7aa00640 1 Processor -- start 2024-01-24T17:42:31.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.659+0000 7fdf7aa00640 1 -- start start 2024-01-24T17:42:31.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf7aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf740cb540 0x7fdf7406d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf7aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdf7406c3a0 0x7fdf740ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf7aa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740ad0b0 con 0x7fdf740accd0 2024-01-24T17:42:31.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf7aa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7406d840 con 0x7fdf7406c3a0 2024-01-24T17:42:31.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf7aa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740b2370 con 0x7fdf740cb540 2024-01-24T17:42:31.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf79600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fdf740accd0 0x7fdf740cae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48022/0 (socket says 172.21.15.39:48022) 2024-01-24T17:42:31.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf79600640 1 -- 172.21.15.39:0/2776346438 learned_addr learned my addr 172.21.15.39:0/2776346438 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:31.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf7a000640 1 --2- 172.21.15.39:0/2776346438 >> v2:172.21.15.39:3300/0 conn(0x7fdf7406c3a0 0x7fdf740ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf78c00640 1 --2- 172.21.15.39:0/2776346438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf740cb540 0x7fdf7406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.661+0000 7fdf73e00640 1 -- 172.21.15.39:0/2776346438 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501390714 0 0) 0x7fdf740ad0b0 con 0x7fdf740accd0 2024-01-24T17:42:31.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.662+0000 7fdf73e00640 1 -- 172.21.15.39:0/2776346438 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf54003520 con 0x7fdf740accd0 2024-01-24T17:42:31.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.662+0000 7fdf7a000640 1 -- 172.21.15.39:0/2776346438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf740cb540 msgr2=0x7fdf7406d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:31.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.662+0000 7fdf7a000640 1 --2- 172.21.15.39:0/2776346438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf740cb540 0x7fdf7406d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.662+0000 7fdf7a000640 1 -- 172.21.15.39:0/2776346438 >> v1:172.21.15.39:6789/0 conn(0x7fdf740accd0 legacy=0x7fdf740cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.662+0000 7fdf7a000640 1 -- 172.21.15.39:0/2776346438 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740b2650 con 0x7fdf7406c3a0 2024-01-24T17:42:31.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.662+0000 7fdf78c00640 1 --2- 172.21.15.39:0/2776346438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf740cb540 0x7fdf7406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.663+0000 7fdf7a000640 1 --2- 172.21.15.39:0/2776346438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf7406c3a0 0x7fdf740ab030 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009d90 tx=0x7fdf6c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df74c5bf58fe974 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.663+0000 7fdf73e00640 1 -- 172.21.15.39:0/2776346438 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c02cc50 con 0x7fdf7406c3a0 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.663+0000 7fdf73e00640 1 -- 172.21.15.39:0/2776346438 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c02cdb0 con 0x7fdf7406c3a0 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.663+0000 7fdf73e00640 1 -- 172.21.15.39:0/2776346438 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c034660 con 0x7fdf7406c3a0 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.664+0000 7fdf7aa00640 1 -- 172.21.15.39:0/2776346438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf7406c3a0 msgr2=0x7fdf740ab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.664+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/2776346438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf7406c3a0 0x7fdf740ab030 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009d90 tx=0x7fdf6c02bea0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.664+0000 7fdf7aa00640 1 -- 172.21.15.39:0/2776346438 shutdown_connections 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.664+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/2776346438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf7406c3a0 0x7fdf740ab030 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.664+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/2776346438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf740cb540 0x7fdf7406d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.664+0000 7fdf7aa00640 1 -- 172.21.15.39:0/2776346438 >> 172.21.15.39:0/2776346438 conn(0x7fdf74069e50 msgr2=0x7fdf7406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.664+0000 7fdf7aa00640 1 -- 172.21.15.39:0/2776346438 shutdown_connections 2024-01-24T17:42:31.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.664+0000 7fdf7aa00640 1 -- 172.21.15.39:0/2776346438 wait complete. 2024-01-24T17:42:31.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.666+0000 7fdf7aa00640 1 Processor -- start 2024-01-24T17:42:31.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.666+0000 7fdf7aa00640 1 -- start start 2024-01-24T17:42:31.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.667+0000 7fdf7aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf7406c3a0 0x7fdf740cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.667+0000 7fdf7aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf740accd0 0x7fdf741f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.667+0000 7fdf7aa00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdf740cb540 0x7fdf740c2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.667+0000 7fdf7aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740c8ba0 con 0x7fdf740accd0 2024-01-24T17:42:31.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.667+0000 7fdf7aa00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740c8e80 con 0x7fdf740cb540 2024-01-24T17:42:31.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.667+0000 7fdf7aa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740c9160 con 0x7fdf7406c3a0 2024-01-24T17:42:31.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.667+0000 7fdf78c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf740accd0 0x7fdf741f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf78c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf740accd0 0x7fdf741f15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59590/0 (socket says 172.21.15.39:59590) 2024-01-24T17:42:31.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf78c00640 1 -- 172.21.15.39:0/1070229186 learned_addr learned my addr 172.21.15.39:0/1070229186 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:31.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf79600640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf7406c3a0 0x7fdf740cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf7a000640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdf740cb540 0x7fdf740c2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf78c00640 1 -- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf7406c3a0 msgr2=0x7fdf740cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf78c00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf7406c3a0 0x7fdf740cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf78c00640 1 -- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdf740cb540 msgr2=0x7fdf740c2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf78c00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdf740cb540 0x7fdf740c2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf78c00640 1 -- 172.21.15.39:0/1070229186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c009a70 con 0x7fdf740accd0 2024-01-24T17:42:31.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf7a000640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdf740cb540 0x7fdf740c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:31.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.668+0000 7fdf78c00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf740accd0 0x7fdf741f15e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c00e7e0 tx=0x7fdf5c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:31.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.669+0000 7fdf71600640 1 -- 172.21.15.39:0/1070229186 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c004030 con 0x7fdf740accd0 2024-01-24T17:42:31.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.669+0000 7fdf71600640 1 -- 172.21.15.39:0/1070229186 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5c004190 con 0x7fdf740accd0 2024-01-24T17:42:31.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.669+0000 7fdf71600640 1 -- 172.21.15.39:0/1070229186 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c004520 con 0x7fdf740accd0 2024-01-24T17:42:31.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.669+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf740c9440 con 0x7fdf740accd0 2024-01-24T17:42:31.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.669+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf740cf120 con 0x7fdf740accd0 2024-01-24T17:42:31.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.670+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7404dff0 con 0x7fdf740accd0 2024-01-24T17:42:31.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.671+0000 7fdf71600640 1 -- 172.21.15.39:0/1070229186 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdf5c00c6d0 con 0x7fdf740accd0 2024-01-24T17:42:31.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.672+0000 7fdf71600640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdf4c06a430 0x7fdf4c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:31.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.672+0000 7fdf71600640 1 -- 172.21.15.39:0/1070229186 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdf5c088540 con 0x7fdf740accd0 2024-01-24T17:42:31.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.673+0000 7fdf79600640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdf4c06a430 0x7fdf4c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:31.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.674+0000 7fdf79600640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdf4c06a430 0x7fdf4c06c8f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdf640010e0 tx=0x7fdf64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:31.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.676+0000 7fdf71600640 1 -- 172.21.15.39:0/1070229186 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdf5c053ba0 con 0x7fdf740accd0 2024-01-24T17:42:31.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.874+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdf740cca30 con 0x7fdf740accd0 2024-01-24T17:42:31.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.874+0000 7fdf71600640 1 -- 172.21.15.39:0/1070229186 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdf5c0571c0 con 0x7fdf740accd0 2024-01-24T17:42:31.875 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:31.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdf4c06a430 msgr2=0x7fdf4c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdf4c06a430 0x7fdf4c06c8f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdf640010e0 tx=0x7fdf64009040 comp rx=0 tx=0).stop 2024-01-24T17:42:31.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf740accd0 msgr2=0x7fdf741f15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:31.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf740accd0 0x7fdf741f15e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c00e7e0 tx=0x7fdf5c00eca0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 shutdown_connections 2024-01-24T17:42:31.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdf740cb540 0x7fdf740c2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdf4c06a430 0x7fdf4c06c8f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdf740accd0 0x7fdf741f15e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 --2- 172.21.15.39:0/1070229186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdf7406c3a0 0x7fdf740cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:31.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 >> 172.21.15.39:0/1070229186 conn(0x7fdf74069e50 msgr2=0x7fdf740cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:31.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 shutdown_connections 2024-01-24T17:42:31.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:31.878+0000 7fdf7aa00640 1 -- 172.21.15.39:0/1070229186 wait complete. 2024-01-24T17:42:31.898 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:42:32.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.121+0000 7f2986e00640 1 Processor -- start 2024-01-24T17:42:32.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.121+0000 7f2986e00640 1 -- start start 2024-01-24T17:42:32.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.122+0000 7f2986e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f29800cb020 0x7f298015ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.123+0000 7f2986e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298015b240 con 0x7f29800c8c10 2024-01-24T17:42:32.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.123+0000 7f2986e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298015b520 con 0x7f29800cb400 2024-01-24T17:42:32.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.123+0000 7f2986e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f298015b800 con 0x7f29800cb020 2024-01-24T17:42:32.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.123+0000 7f2986400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f29800c8c10 0x7f298015e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48036/0 (socket says 172.21.15.39:48036) 2024-01-24T17:42:32.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.123+0000 7f2986400640 1 -- 172.21.15.39:0/3338797808 learned_addr learned my addr 172.21.15.39:0/3338797808 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.123+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428958869 0 0) 0x7f298015b240 con 0x7f29800c8c10 2024-01-24T17:42:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.123+0000 7f2985a00640 1 --2- 172.21.15.39:0/3338797808 >> v2:172.21.15.100:3300/0 conn(0x7f29800cb020 0x7f298015ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.123+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2960003520 con 0x7f29800c8c10 2024-01-24T17:42:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238886167 0 0) 0x7f2960003520 con 0x7f29800c8c10 2024-01-24T17:42:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 >> v2:172.21.15.100:3300/0 conn(0x7f29800cb020 msgr2=0x7f298015ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 --2- 172.21.15.39:0/3338797808 >> v2:172.21.15.100:3300/0 conn(0x7f29800cb020 0x7f298015ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 >> v1:172.21.15.100:6789/0 conn(0x7f29800cb400 legacy=0x7f29800ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298015bae0 con 0x7f29800c8c10 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f29780019f0 con 0x7f29800c8c10 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 >> v1:172.21.15.39:6789/0 conn(0x7f29800c8c10 legacy=0x7f298015e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29600039c0 0x7f2960003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29600047c0 0x7f2960004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2960005180 0x7f296000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298015b800 con 0x7f29600047c0 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296000c080 con 0x7f2960005180 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296000c240 con 0x7f29600039c0 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.124+0000 7f2985000640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29600047c0 0x7f2960004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2985a00640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29600039c0 0x7f2960003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2986400640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2960005180 0x7f296000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2985000640 1 -- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29600039c0 msgr2=0x7f2960003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2985000640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29600039c0 0x7f2960003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2985000640 1 -- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2960005180 msgr2=0x7f296000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2985000640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2960005180 0x7f296000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2985000640 1 -- 172.21.15.39:0/3338797808 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2960003520 con 0x7f29600047c0 2024-01-24T17:42:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2985a00640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29600039c0 0x7f2960003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2986400640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2960005180 0x7f296000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:32.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.125+0000 7f2986400640 1 -- 172.21.15.39:0/3338797808 reap_dead start 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.126+0000 7f2985000640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29600047c0 0x7f2960004c40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2968001750 tx=0x7f29600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9a5b4f0158934c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.126+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29680052d0 con 0x7f29600047c0 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.126+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296802e050 con 0x7f29600047c0 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.126+0000 7f297fe00640 1 -- 172.21.15.39:0/3338797808 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968003830 con 0x7f29600047c0 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.127+0000 7f2986e00640 1 -- 172.21.15.39:0/3338797808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29600047c0 msgr2=0x7f2960004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.127+0000 7f2986e00640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29600047c0 0x7f2960004c40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2968001750 tx=0x7f29600057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.127+0000 7f2986e00640 1 -- 172.21.15.39:0/3338797808 shutdown_connections 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.127+0000 7f2986e00640 1 --2- 172.21.15.39:0/3338797808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29600047c0 0x7f2960004c40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.127+0000 7f2986e00640 1 -- 172.21.15.39:0/3338797808 >> 172.21.15.39:0/3338797808 conn(0x7f298006a2a0 msgr2=0x7f29800afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:32.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.127+0000 7f2986e00640 1 -- 172.21.15.39:0/3338797808 shutdown_connections 2024-01-24T17:42:32.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.127+0000 7f2986e00640 1 -- 172.21.15.39:0/3338797808 wait complete. 2024-01-24T17:42:32.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.129+0000 7f2986e00640 1 Processor -- start 2024-01-24T17:42:32.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.129+0000 7f2986e00640 1 -- start start 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.130+0000 7f2986e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29600047c0 0x7f29801ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.130+0000 7f2986e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2980159970 0x7f2980159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.130+0000 7f2986e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29800b32b0 0x7f29800b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.130+0000 7f2986e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298015a330 con 0x7f29800b32b0 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.130+0000 7f2986e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29801efc20 con 0x7f29600047c0 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.130+0000 7f2986e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29800c79e0 con 0x7f2980159970 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29800b32b0 0x7f29800b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29800b32b0 0x7f29800b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59612/0 (socket says 172.21.15.39:59612) 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 -- 172.21.15.39:0/314074095 learned_addr learned my addr 172.21.15.39:0/314074095 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2985a00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29600047c0 0x7f29801ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2985000640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2980159970 0x7f2980159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 -- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2980159970 msgr2=0x7f2980159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2980159970 0x7f2980159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 -- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29600047c0 msgr2=0x7f29801ef6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29600047c0 0x7f29801ef6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 -- 172.21.15.39:0/314074095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29800c7cc0 con 0x7f29800b32b0 2024-01-24T17:42:32.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2985000640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2980159970 0x7f2980159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:32.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.131+0000 7f2986400640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29800b32b0 0x7f29800b3760 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f29800b0b50 tx=0x7f297800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:32.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.132+0000 7f297d600640 1 -- 172.21.15.39:0/314074095 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978004d60 con 0x7f29800b32b0 2024-01-24T17:42:32.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.132+0000 7f297d600640 1 -- 172.21.15.39:0/314074095 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297800bb80 con 0x7f29800b32b0 2024-01-24T17:42:32.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.132+0000 7f297d600640 1 -- 172.21.15.39:0/314074095 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978007070 con 0x7f29800b32b0 2024-01-24T17:42:32.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.132+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29800c7fa0 con 0x7f29800b32b0 2024-01-24T17:42:32.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.132+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29800c8560 con 0x7f29800b32b0 2024-01-24T17:42:32.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.133+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29800b0fc0 con 0x7f29800b32b0 2024-01-24T17:42:32.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.134+0000 7f297d600640 1 -- 172.21.15.39:0/314074095 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f29780071d0 con 0x7f29800b32b0 2024-01-24T17:42:32.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.135+0000 7f297d600640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f295806a430 0x7f295806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.135+0000 7f297d600640 1 -- 172.21.15.39:0/314074095 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2978088d70 con 0x7f29800b32b0 2024-01-24T17:42:32.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.135+0000 7f2985a00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f295806a430 0x7f295806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.137+0000 7f2985a00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f295806a430 0x7f295806c8f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f298004f260 tx=0x7f2970008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:32.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.139+0000 7f297d600640 1 -- 172.21.15.39:0/314074095 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f29800b0fc0 con 0x7f29800b32b0 2024-01-24T17:42:32.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.347+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f29800b0fc0 con 0x7f29800b32b0 2024-01-24T17:42:32.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.348+0000 7f297d600640 1 -- 172.21.15.39:0/314074095 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f29800b0fc0 con 0x7f29800b32b0 2024-01-24T17:42:32.348 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:42:32.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f295806a430 msgr2=0x7f295806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f295806a430 0x7f295806c8f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f298004f260 tx=0x7f2970008040 comp rx=0 tx=0).stop 2024-01-24T17:42:32.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29800b32b0 msgr2=0x7f29800b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29800b32b0 0x7f29800b3760 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f29800b0b50 tx=0x7f297800b4a0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 shutdown_connections 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f295806a430 0x7f295806c8f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29800b32b0 0x7f29800b3760 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2980159970 0x7f2980159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 --2- 172.21.15.39:0/314074095 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29600047c0 0x7f29801ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.351+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 >> 172.21.15.39:0/314074095 conn(0x7f298006a2a0 msgr2=0x7f298015d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.352+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 shutdown_connections 2024-01-24T17:42:32.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.352+0000 7f2986e00640 1 -- 172.21.15.39:0/314074095 wait complete. 2024-01-24T17:42:32.371 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:42:32.371 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:32.427 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:32.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.638+0000 7f3337e00640 1 Processor -- start 2024-01-24T17:42:32.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.638+0000 7f3337e00640 1 -- start start 2024-01-24T17:42:32.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.639+0000 7f3337e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f333815a720 0x7f33380cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.639+0000 7f3337e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33380ce400 con 0x7f33380ce8e0 2024-01-24T17:42:32.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.639+0000 7f3337e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33380ccef0 con 0x7f333815ab00 2024-01-24T17:42:32.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.639+0000 7f3337e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33380ce110 con 0x7f333815a720 2024-01-24T17:42:32.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.640+0000 7f3337400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f33380ce8e0 0x7f33380cb2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48038/0 (socket says 172.21.15.39:48038) 2024-01-24T17:42:32.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.640+0000 7f3337400640 1 -- 172.21.15.39:0/3226293229 learned_addr learned my addr 172.21.15.39:0/3226293229 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.640+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377625356 0 0) 0x7f33380ce400 con 0x7f33380ce8e0 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.640+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3318003520 con 0x7f33380ce8e0 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.640+0000 7f3336a00640 1 --2- 172.21.15.39:0/3226293229 >> v2:172.21.15.100:3300/0 conn(0x7f333815a720 0x7f33380cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3102350552 0 0) 0x7f3318003520 con 0x7f33380ce8e0 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 >> v2:172.21.15.100:3300/0 conn(0x7f333815a720 msgr2=0x7f33380cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 --2- 172.21.15.39:0/3226293229 >> v2:172.21.15.100:3300/0 conn(0x7f333815a720 0x7f33380cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 >> v1:172.21.15.100:6789/0 conn(0x7f333815ab00 legacy=0x7f33380cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333815b3b0 con 0x7f33380ce8e0 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f33300019f0 con 0x7f33380ce8e0 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 >> v1:172.21.15.39:6789/0 conn(0x7f33380ce8e0 legacy=0x7f33380cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180039c0 0x7f3318003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33180047c0 0x7f3318004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33180052a0 0x7f331800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33380ce110 con 0x7f33180047c0 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331800c1c0 con 0x7f33180039c0 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331800c380 con 0x7f33180052a0 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3336000640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33180047c0 0x7f3318004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.641+0000 7f3336a00640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180039c0 0x7f3318003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.642+0000 7f3337400640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33180052a0 0x7f331800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.642+0000 7f3336000640 1 -- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33180052a0 msgr2=0x7f331800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.642+0000 7f3336000640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33180052a0 0x7f331800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.642+0000 7f3336000640 1 -- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180039c0 msgr2=0x7f3318003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.642+0000 7f3336000640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180039c0 0x7f3318003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.642+0000 7f3336000640 1 -- 172.21.15.39:0/3226293229 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3318003520 con 0x7f33180047c0 2024-01-24T17:42:32.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.642+0000 7f3336a00640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180039c0 0x7f3318003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:32.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.642+0000 7f3337400640 1 -- 172.21.15.39:0/3226293229 reap_dead start 2024-01-24T17:42:32.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.643+0000 7f3336000640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33180047c0 0x7f3318004c40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3320001750 tx=0x7f33180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7bec6d79d130bb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:32.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.643+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33200052d0 con 0x7f33180047c0 2024-01-24T17:42:32.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.643+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332002e050 con 0x7f33180047c0 2024-01-24T17:42:32.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.643+0000 7f3335600640 1 -- 172.21.15.39:0/3226293229 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320003830 con 0x7f33180047c0 2024-01-24T17:42:32.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.644+0000 7f3337e00640 1 -- 172.21.15.39:0/3226293229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33180047c0 msgr2=0x7f3318004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.644+0000 7f3337e00640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33180047c0 0x7f3318004c40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3320001750 tx=0x7f33180057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.644+0000 7f3337e00640 1 -- 172.21.15.39:0/3226293229 shutdown_connections 2024-01-24T17:42:32.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.644+0000 7f3337e00640 1 --2- 172.21.15.39:0/3226293229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33180047c0 0x7f3318004c40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.644+0000 7f3337e00640 1 -- 172.21.15.39:0/3226293229 >> 172.21.15.39:0/3226293229 conn(0x7f333806a090 msgr2=0x7f333806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:32.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.644+0000 7f3337e00640 1 -- 172.21.15.39:0/3226293229 shutdown_connections 2024-01-24T17:42:32.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.644+0000 7f3337e00640 1 -- 172.21.15.39:0/3226293229 wait complete. 2024-01-24T17:42:32.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.646+0000 7f3337e00640 1 Processor -- start 2024-01-24T17:42:32.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.646+0000 7f3337e00640 1 -- start start 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3337e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180047c0 0x7f3338156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3337e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33381577d0 0x7f33381570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3337e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33380b0fa0 0x7f33380b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3337e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33380b2810 con 0x7f33381577d0 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3337e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33380c4bb0 con 0x7f33180047c0 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3337e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33380b21c0 con 0x7f33380b0fa0 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3336000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33381577d0 0x7f33381570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3336000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33381577d0 0x7f33381570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59628/0 (socket says 172.21.15.39:59628) 2024-01-24T17:42:32.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3336000640 1 -- 172.21.15.39:0/1132546708 learned_addr learned my addr 172.21.15.39:0/1132546708 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3336a00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180047c0 0x7f3338156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3337400640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33380b0fa0 0x7f33380b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3336000640 1 -- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33380b0fa0 msgr2=0x7f33380b1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3336000640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33380b0fa0 0x7f33380b1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3336000640 1 -- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180047c0 msgr2=0x7f3338156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.647+0000 7f3336000640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180047c0 0x7f3338156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.648+0000 7f3336000640 1 -- 172.21.15.39:0/1132546708 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33380b24a0 con 0x7f33381577d0 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.648+0000 7f3337400640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33380b0fa0 0x7f33380b1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.648+0000 7f3336000640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33381577d0 0x7f33381570c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f332002bc30 tx=0x7f3320001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.648+0000 7f332ea00640 1 -- 172.21.15.39:0/1132546708 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33200076c0 con 0x7f33381577d0 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.648+0000 7f332ea00640 1 -- 172.21.15.39:0/1132546708 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3320007a30 con 0x7f33381577d0 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.648+0000 7f332ea00640 1 -- 172.21.15.39:0/1132546708 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320006c20 con 0x7f33381577d0 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.648+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33380b1990 con 0x7f33381577d0 2024-01-24T17:42:32.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.648+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33380b1e50 con 0x7f33381577d0 2024-01-24T17:42:32.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.650+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3338070700 con 0x7f33381577d0 2024-01-24T17:42:32.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.650+0000 7f332ea00640 1 -- 172.21.15.39:0/1132546708 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f33200035f0 con 0x7f33381577d0 2024-01-24T17:42:32.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.651+0000 7f332ea00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f331006a460 0x7f331006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:32.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.651+0000 7f332ea00640 1 -- 172.21.15.39:0/1132546708 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f33200aa5a0 con 0x7f33381577d0 2024-01-24T17:42:32.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.652+0000 7f3336a00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f331006a460 0x7f331006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:32.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.653+0000 7f3336a00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f331006a460 0x7f331006c920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3328006fd0 tx=0x7f3328008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:32.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.656+0000 7f332ea00640 1 -- 172.21.15.39:0/1132546708 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3320075b60 con 0x7f33381577d0 2024-01-24T17:42:32.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.852+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f333804dff0 con 0x7f33381577d0 2024-01-24T17:42:32.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.855+0000 7f332ea00640 1 -- 172.21.15.39:0/1132546708 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f3320079180 con 0x7f33381577d0 2024-01-24T17:42:32.856 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:32.856 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:32.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f331006a460 msgr2=0x7f331006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f331006a460 0x7f331006c920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3328006fd0 tx=0x7f3328008040 comp rx=0 tx=0).stop 2024-01-24T17:42:32.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33381577d0 msgr2=0x7f33381570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:32.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33381577d0 0x7f33381570c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f332002bc30 tx=0x7f3320001120 comp rx=0 tx=0).stop 2024-01-24T17:42:32.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 shutdown_connections 2024-01-24T17:42:32.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f33380b0fa0 0x7f33380b1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f331006a460 0x7f331006c920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33381577d0 0x7f33381570c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 --2- 172.21.15.39:0/1132546708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33180047c0 0x7f3338156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:32.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.857+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 >> 172.21.15.39:0/1132546708 conn(0x7f333806a090 msgr2=0x7f3338072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:32.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.858+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 shutdown_connections 2024-01-24T17:42:32.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:32.858+0000 7f3337e00640 1 -- 172.21.15.39:0/1132546708 wait complete. 2024-01-24T17:42:32.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:32.877 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:33.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.088+0000 7fea9f200640 1 Processor -- start 2024-01-24T17:42:33.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.089+0000 7fea9f200640 1 -- start start 2024-01-24T17:42:33.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.090+0000 7fea9f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fea9815c8d0 0x7fea980d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.090+0000 7fea9f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980d2e80 0x7fea980cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.090+0000 7fea9f200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea980b38a0 con 0x7fea980b2710 2024-01-24T17:42:33.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.090+0000 7fea9f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea980d13e0 con 0x7fea9815c8d0 2024-01-24T17:42:33.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.090+0000 7fea9f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea980d0ab0 con 0x7fea980d2e80 2024-01-24T17:42:33.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.091+0000 7fea9de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fea9815c8d0 0x7fea980d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.091+0000 7fea9de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fea9815c8d0 0x7fea980d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44524/0 (socket says 172.21.15.39:44524) 2024-01-24T17:42:33.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.091+0000 7fea9de00640 1 -- 172.21.15.39:0/1418662961 learned_addr learned my addr 172.21.15.39:0/1418662961 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:33.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.091+0000 7fea9d400640 1 --1- 172.21.15.39:0/1418662961 >> v1:172.21.15.100:6789/0 conn(0x7fea980b2710 0x7fea980d2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:52302/0 (socket says 172.21.15.39:52302) 2024-01-24T17:42:33.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.091+0000 7fea9e800640 1 --2- 172.21.15.39:0/1418662961 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980d2e80 0x7fea980cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.092+0000 7fea9de00640 1 -- 172.21.15.39:0/1418662961 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980d2e80 msgr2=0x7fea980cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:33.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.092+0000 7fea9de00640 1 --2- 172.21.15.39:0/1418662961 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980d2e80 0x7fea980cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.092+0000 7fea9de00640 1 -- 172.21.15.39:0/1418662961 >> v1:172.21.15.100:6789/0 conn(0x7fea980b2710 legacy=0x7fea980d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.092+0000 7fea9de00640 1 -- 172.21.15.39:0/1418662961 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980d0d90 con 0x7fea9815c8d0 2024-01-24T17:42:33.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.093+0000 7fea9de00640 1 --2- 172.21.15.39:0/1418662961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea980d03b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fea88009870 tx=0x7fea8802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9296ab397c72cf server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:33.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9ca00640 1 -- 172.21.15.39:0/1418662961 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88030840 con 0x7fea9815c8d0 2024-01-24T17:42:33.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9ca00640 1 -- 172.21.15.39:0/1418662961 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea880309a0 con 0x7fea9815c8d0 2024-01-24T17:42:33.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9ca00640 1 -- 172.21.15.39:0/1418662961 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88030d30 con 0x7fea9815c8d0 2024-01-24T17:42:33.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9f200640 1 -- 172.21.15.39:0/1418662961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 msgr2=0x7fea980d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9f200640 1 --2- 172.21.15.39:0/1418662961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea980d03b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fea88009870 tx=0x7fea8802b980 comp rx=0 tx=0).stop 2024-01-24T17:42:33.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9f200640 1 -- 172.21.15.39:0/1418662961 shutdown_connections 2024-01-24T17:42:33.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9f200640 1 --2- 172.21.15.39:0/1418662961 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980d2e80 0x7fea980cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9f200640 1 --2- 172.21.15.39:0/1418662961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea980d03b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.094+0000 7fea9f200640 1 -- 172.21.15.39:0/1418662961 >> 172.21.15.39:0/1418662961 conn(0x7fea98071dd0 msgr2=0x7fea9809bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:33.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.095+0000 7fea9f200640 1 -- 172.21.15.39:0/1418662961 shutdown_connections 2024-01-24T17:42:33.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.095+0000 7fea9f200640 1 -- 172.21.15.39:0/1418662961 wait complete. 2024-01-24T17:42:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.096+0000 7fea9f200640 1 Processor -- start 2024-01-24T17:42:33.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.096+0000 7fea9f200640 1 -- start start 2024-01-24T17:42:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.097+0000 7fea9f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980b2710 0x7fea98151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.097+0000 7fea9f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea980d2e80 0x7fea9809d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.097+0000 7fea9f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea9809f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.097+0000 7fea9f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9809ebe0 con 0x7fea9815c8d0 2024-01-24T17:42:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.097+0000 7fea9f200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea981520b0 con 0x7fea980d2e80 2024-01-24T17:42:33.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.097+0000 7fea9f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9809df60 con 0x7fea980b2710 2024-01-24T17:42:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea9809f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea9809f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44534/0 (socket says 172.21.15.39:44534) 2024-01-24T17:42:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9e800640 1 -- 172.21.15.39:0/4193254423 learned_addr learned my addr 172.21.15.39:0/4193254423 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9de00640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980b2710 0x7fea98151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9d400640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea980d2e80 0x7fea9809d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9e800640 1 -- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980b2710 msgr2=0x7fea98151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9e800640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980b2710 0x7fea98151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9e800640 1 -- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea980d2e80 msgr2=0x7fea9809d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9e800640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea980d2e80 0x7fea9809d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.098+0000 7fea9e800640 1 -- 172.21.15.39:0/4193254423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880094d0 con 0x7fea9815c8d0 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.099+0000 7fea9e800640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea9809f3b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fea9000efc0 tx=0x7fea9000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.099+0000 7fea96000640 1 -- 172.21.15.39:0/4193254423 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea90009ca0 con 0x7fea9815c8d0 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.099+0000 7fea96000640 1 -- 172.21.15.39:0/4193254423 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea9000f040 con 0x7fea9815c8d0 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.099+0000 7fea96000640 1 -- 172.21.15.39:0/4193254423 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea900146c0 con 0x7fea9815c8d0 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.099+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9809e240 con 0x7fea9815c8d0 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.099+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9809e800 con 0x7fea9815c8d0 2024-01-24T17:42:33.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.100+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea980ccce0 con 0x7fea9815c8d0 2024-01-24T17:42:33.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.102+0000 7fea96000640 1 -- 172.21.15.39:0/4193254423 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fea90014820 con 0x7fea9815c8d0 2024-01-24T17:42:33.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.103+0000 7fea96000640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a430 0x7fea7006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.103+0000 7fea96000640 1 -- 172.21.15.39:0/4193254423 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fea90089850 con 0x7fea9815c8d0 2024-01-24T17:42:33.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.104+0000 7fea9de00640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a430 0x7fea7006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.105+0000 7fea9de00640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a430 0x7fea7006c8f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fea88036000 tx=0x7fea8803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:33.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.108+0000 7fea96000640 1 -- 172.21.15.39:0/4193254423 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fea90054e10 con 0x7fea9815c8d0 2024-01-24T17:42:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.303+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea9804dff0 con 0x7fea9815c8d0 2024-01-24T17:42:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.304+0000 7fea96000640 1 -- 172.21.15.39:0/4193254423 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea90018020 con 0x7fea9815c8d0 2024-01-24T17:42:33.305 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a430 msgr2=0x7fea7006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a430 0x7fea7006c8f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fea88036000 tx=0x7fea8803b040 comp rx=0 tx=0).stop 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 msgr2=0x7fea9809f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea9809f3b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fea9000efc0 tx=0x7fea9000c490 comp rx=0 tx=0).stop 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 shutdown_connections 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a430 0x7fea7006c8f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea9815c8d0 0x7fea9809f3b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea980d2e80 0x7fea9809d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 --2- 172.21.15.39:0/4193254423 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea980b2710 0x7fea98151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 >> 172.21.15.39:0/4193254423 conn(0x7fea98071dd0 msgr2=0x7fea9815be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 shutdown_connections 2024-01-24T17:42:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.307+0000 7fea9f200640 1 -- 172.21.15.39:0/4193254423 wait complete. 2024-01-24T17:42:33.327 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:33.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.552+0000 7f9bdec00640 1 Processor -- start 2024-01-24T17:42:33.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.552+0000 7f9bdec00640 1 -- start start 2024-01-24T17:42:33.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.553+0000 7f9bdec00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9bd815ccb0 0x7f9bd80cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.553+0000 7f9bdec00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c9d60 0x7f9bd80ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.553+0000 7f9bdec00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd815d090 con 0x7f9bd80ce390 2024-01-24T17:42:33.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.553+0000 7f9bdec00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd80cac20 con 0x7f9bd815ccb0 2024-01-24T17:42:33.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.554+0000 7f9bdec00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80ced60 con 0x7f9bd80c9d60 2024-01-24T17:42:33.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.554+0000 7f9bdce00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9bd80ce390 0x7f9bd80cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47144/0 (socket says 172.21.15.39:47144) 2024-01-24T17:42:33.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.554+0000 7f9bdce00640 1 -- 172.21.15.39:0/3591963167 learned_addr learned my addr 172.21.15.39:0/3591963167 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:33.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.554+0000 7f9bde200640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c9d60 0x7f9bd80ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.554+0000 7f9bdd800640 1 --2- 172.21.15.39:0/3591963167 >> v2:172.21.15.100:3300/0 conn(0x7f9bd815ccb0 0x7f9bd80cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.554+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879977017 0 0) 0x7f9bd815d090 con 0x7f9bd80ce390 2024-01-24T17:42:33.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.554+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb8003520 con 0x7f9bd80ce390 2024-01-24T17:42:33.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179953874 0 0) 0x7f9bb8003520 con 0x7f9bd80ce390 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c9d60 msgr2=0x7f9bd80ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c9d60 0x7f9bd80ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 >> v2:172.21.15.100:3300/0 conn(0x7f9bd815ccb0 msgr2=0x7f9bd80cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 --2- 172.21.15.39:0/3591963167 >> v2:172.21.15.100:3300/0 conn(0x7f9bd815ccb0 0x7f9bd80cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd806c540 con 0x7f9bd80ce390 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f9bc0001670 con 0x7f9bd80ce390 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 >> v1:172.21.15.39:6789/0 conn(0x7f9bd80ce390 legacy=0x7f9bd80cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bb8003970 0x7f9bb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c9d60 0x7f9bb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80cac20 con 0x7f9bb8004d00 2024-01-24T17:42:33.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80ced60 con 0x7f9bb8003970 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb800b980 con 0x7f9bd80c9d60 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.555+0000 7f9bde200640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.556+0000 7f9bdd800640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bb8003970 0x7f9bb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.556+0000 7f9bdce00640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c9d60 0x7f9bb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.556+0000 7f9bde200640 1 -- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c9d60 msgr2=0x7f9bb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.556+0000 7f9bde200640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c9d60 0x7f9bb80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.556+0000 7f9bde200640 1 -- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bb8003970 msgr2=0x7f9bb8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.556+0000 7f9bde200640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bb8003970 0x7f9bb8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.556+0000 7f9bde200640 1 -- 172.21.15.39:0/3591963167 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8003520 con 0x7f9bb8004d00 2024-01-24T17:42:33.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.557+0000 7f9bde200640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bb8005180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009f90 tx=0x7f9bb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf8ad7f2aad3569 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:33.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.557+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd002cbd0 con 0x7f9bb8004d00 2024-01-24T17:42:33.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.557+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd000f040 con 0x7f9bb8004d00 2024-01-24T17:42:33.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.557+0000 7f9bd7e00640 1 -- 172.21.15.39:0/3591963167 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0034440 con 0x7f9bb8004d00 2024-01-24T17:42:33.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.558+0000 7f9bdec00640 1 -- 172.21.15.39:0/3591963167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 msgr2=0x7f9bb8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.558+0000 7f9bdec00640 1 --2- 172.21.15.39:0/3591963167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bb8005180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009f90 tx=0x7f9bb80057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.558+0000 7f9bde200640 1 -- 172.21.15.39:0/3591963167 reap_dead start 2024-01-24T17:42:33.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.558+0000 7f9bdec00640 1 -- 172.21.15.39:0/3591963167 shutdown_connections 2024-01-24T17:42:33.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.558+0000 7f9bdec00640 1 -- 172.21.15.39:0/3591963167 >> 172.21.15.39:0/3591963167 conn(0x7f9bd8069e50 msgr2=0x7f9bd806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:33.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.558+0000 7f9bdec00640 1 -- 172.21.15.39:0/3591963167 shutdown_connections 2024-01-24T17:42:33.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.558+0000 7f9bdec00640 1 -- 172.21.15.39:0/3591963167 wait complete. 2024-01-24T17:42:33.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.560+0000 7f9bdec00640 1 Processor -- start 2024-01-24T17:42:33.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.560+0000 7f9bdec00640 1 -- start start 2024-01-24T17:42:33.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bd8159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdec00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bd80b1430 0x7f9bd80c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdec00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c54f0 0x7f9bd80adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd809bc70 con 0x7f9bb8004d00 2024-01-24T17:42:33.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdec00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd809bf50 con 0x7f9bd80b1430 2024-01-24T17:42:33.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdec00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80c7490 con 0x7f9bd80c54f0 2024-01-24T17:42:33.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdd800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bd8159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdd800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bd8159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44556/0 (socket says 172.21.15.39:44556) 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.561+0000 7f9bdd800640 1 -- 172.21.15.39:0/2199113003 learned_addr learned my addr 172.21.15.39:0/2199113003 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bdce00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bd80b1430 0x7f9bd80c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bde200640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c54f0 0x7f9bd80adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bdd800640 1 -- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c54f0 msgr2=0x7f9bd80adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bdd800640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c54f0 0x7f9bd80adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bdd800640 1 -- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bd80b1430 msgr2=0x7f9bd80c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bdd800640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bd80b1430 0x7f9bd80c7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bdd800640 1 -- 172.21.15.39:0/2199113003 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80c7770 con 0x7f9bb8004d00 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bde200640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c54f0 0x7f9bd80adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.562+0000 7f9bdd800640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bd8159eb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80115b0 tx=0x7f9bc8011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.563+0000 7f9bd5600640 1 -- 172.21.15.39:0/2199113003 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8005da0 con 0x7f9bb8004d00 2024-01-24T17:42:33.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.563+0000 7f9bd5600640 1 -- 172.21.15.39:0/2199113003 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc8006050 con 0x7f9bb8004d00 2024-01-24T17:42:33.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.563+0000 7f9bd5600640 1 -- 172.21.15.39:0/2199113003 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc80063c0 con 0x7f9bb8004d00 2024-01-24T17:42:33.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.563+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd809b8b0 con 0x7f9bb8004d00 2024-01-24T17:42:33.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.563+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd815a3f0 con 0x7f9bb8004d00 2024-01-24T17:42:33.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.564+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd80c7290 con 0x7f9bb8004d00 2024-01-24T17:42:33.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.565+0000 7f9bd5600640 1 -- 172.21.15.39:0/2199113003 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9bc8006520 con 0x7f9bb8004d00 2024-01-24T17:42:33.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.566+0000 7f9bd5600640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bb006a430 0x7f9bb006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:33.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.566+0000 7f9bd5600640 1 -- 172.21.15.39:0/2199113003 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9bc80890a0 con 0x7f9bb8004d00 2024-01-24T17:42:33.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.566+0000 7f9bdce00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bb006a430 0x7f9bb006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:33.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.567+0000 7f9bdce00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bb006a430 0x7f9bb006c8f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9bd80b1850 tx=0x7f9bc0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:33.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.570+0000 7f9bd5600640 1 -- 172.21.15.39:0/2199113003 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9bc80545d0 con 0x7f9bb8004d00 2024-01-24T17:42:33.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.763+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9bd804dff0 con 0x7f9bb8004d00 2024-01-24T17:42:33.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.764+0000 7f9bd5600640 1 -- 172.21.15.39:0/2199113003 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9bc8057bf0 con 0x7f9bb8004d00 2024-01-24T17:42:33.765 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bb006a430 msgr2=0x7f9bb006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bb006a430 0x7f9bb006c8f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9bd80b1850 tx=0x7f9bc0009040 comp rx=0 tx=0).stop 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 msgr2=0x7f9bd8159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bd8159eb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80115b0 tx=0x7f9bc8011a70 comp rx=0 tx=0).stop 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 shutdown_connections 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9bd80c54f0 0x7f9bd80adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9bd80b1430 0x7f9bd80c7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9bb006a430 0x7f9bb006c8f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 --2- 172.21.15.39:0/2199113003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bb8004d00 0x7f9bd8159eb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 >> 172.21.15.39:0/2199113003 conn(0x7f9bd8069e50 msgr2=0x7f9bd80ca220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:33.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 shutdown_connections 2024-01-24T17:42:33.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:33.767+0000 7f9bdec00640 1 -- 172.21.15.39:0/2199113003 wait complete. 2024-01-24T17:42:33.789 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:34.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.009+0000 7fcb1ca00640 1 Processor -- start 2024-01-24T17:42:34.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.010+0000 7fcb1ca00640 1 -- start start 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.011+0000 7fcb1ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcb1815c9e0 0x7fcb180b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.011+0000 7fcb1ca00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcb180b8250 0x7fcb180cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.011+0000 7fcb1ca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 0x7fcb180b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.011+0000 7fcb1ca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb180b7e30 con 0x7fcb1815c9e0 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.011+0000 7fcb1ca00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb180ce040 con 0x7fcb180b8250 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.011+0000 7fcb1ca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb180b6d90 con 0x7fcb180cdb90 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb16800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcb1815c9e0 0x7fcb180b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb16800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcb1815c9e0 0x7fcb180b6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44568/0 (socket says 172.21.15.39:44568) 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb16800640 1 -- 172.21.15.39:0/2764618148 learned_addr learned my addr 172.21.15.39:0/2764618148 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb15e00640 1 --2- 172.21.15.39:0/2764618148 >> v2:172.21.15.100:3300/0 conn(0x7fcb180b8250 0x7fcb180cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb17200640 1 --2- 172.21.15.39:0/2764618148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 0x7fcb180b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb16800640 1 -- 172.21.15.39:0/2764618148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 msgr2=0x7fcb180b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:34.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb16800640 1 --2- 172.21.15.39:0/2764618148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 0x7fcb180b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb16800640 1 -- 172.21.15.39:0/2764618148 >> v2:172.21.15.100:3300/0 conn(0x7fcb180b8250 msgr2=0x7fcb180cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:34.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb16800640 1 --2- 172.21.15.39:0/2764618148 >> v2:172.21.15.100:3300/0 conn(0x7fcb180b8250 0x7fcb180cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.012+0000 7fcb16800640 1 -- 172.21.15.39:0/2764618148 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb180b7070 con 0x7fcb1815c9e0 2024-01-24T17:42:34.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.013+0000 7fcb16800640 1 --2- 172.21.15.39:0/2764618148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb180b6500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcb18072300 tx=0x7fcb0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acc52af2d61cd80 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:34.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.014+0000 7fcb15400640 1 -- 172.21.15.39:0/2764618148 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08030860 con 0x7fcb1815c9e0 2024-01-24T17:42:34.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.014+0000 7fcb15400640 1 -- 172.21.15.39:0/2764618148 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb080309c0 con 0x7fcb1815c9e0 2024-01-24T17:42:34.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.014+0000 7fcb15400640 1 -- 172.21.15.39:0/2764618148 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08030d50 con 0x7fcb1815c9e0 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.014+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2764618148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 msgr2=0x7fcb180b6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.014+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2764618148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb180b6500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcb18072300 tx=0x7fcb0802b9a0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.015+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2764618148 shutdown_connections 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.015+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2764618148 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 0x7fcb180b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.015+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2764618148 >> v2:172.21.15.100:3300/0 conn(0x7fcb180b8250 0x7fcb180cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.015+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2764618148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb180b6500 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.015+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2764618148 >> 172.21.15.39:0/2764618148 conn(0x7fcb18071790 msgr2=0x7fcb18071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.015+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2764618148 shutdown_connections 2024-01-24T17:42:34.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.015+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2764618148 wait complete. 2024-01-24T17:42:34.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.016+0000 7fcb1ca00640 1 Processor -- start 2024-01-24T17:42:34.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.017+0000 7fcb1ca00640 1 -- start start 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.017+0000 7fcb1ca00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcb180b8250 0x7fcb181f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.017+0000 7fcb1ca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 0x7fcb1814e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.017+0000 7fcb1ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb181f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.017+0000 7fcb1ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1814f4f0 con 0x7fcb1815c9e0 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.017+0000 7fcb1ca00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb181f60e0 con 0x7fcb180b8250 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.017+0000 7fcb1ca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb181f63c0 con 0x7fcb180cdb90 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb17200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb181f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb17200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb181f5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44570/0 (socket says 172.21.15.39:44570) 2024-01-24T17:42:34.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb17200640 1 -- 172.21.15.39:0/2574747198 learned_addr learned my addr 172.21.15.39:0/2574747198 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:34.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb16800640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcb180b8250 0x7fcb181f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb15e00640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 0x7fcb1814e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb17200640 1 -- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 msgr2=0x7fcb1814e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb17200640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 0x7fcb1814e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb17200640 1 -- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcb180b8250 msgr2=0x7fcb181f2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb17200640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcb180b8250 0x7fcb181f2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.018+0000 7fcb17200640 1 -- 172.21.15.39:0/2574747198 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb080094d0 con 0x7fcb1815c9e0 2024-01-24T17:42:34.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.019+0000 7fcb17200640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb181f5570 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcb10007d90 tx=0x7fcb1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:34.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.019+0000 7fcb0ea00640 1 -- 172.21.15.39:0/2574747198 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10014420 con 0x7fcb1815c9e0 2024-01-24T17:42:34.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.019+0000 7fcb0ea00640 1 -- 172.21.15.39:0/2574747198 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb100145a0 con 0x7fcb1815c9e0 2024-01-24T17:42:34.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.019+0000 7fcb0ea00640 1 -- 172.21.15.39:0/2574747198 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10014950 con 0x7fcb1815c9e0 2024-01-24T17:42:34.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.019+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb181f1ff0 con 0x7fcb1815c9e0 2024-01-24T17:42:34.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.019+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb181f25b0 con 0x7fcb1815c9e0 2024-01-24T17:42:34.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.020+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb180b7330 con 0x7fcb1815c9e0 2024-01-24T17:42:34.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.021+0000 7fcb0ea00640 1 -- 172.21.15.39:0/2574747198 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcb10014ab0 con 0x7fcb1815c9e0 2024-01-24T17:42:34.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.022+0000 7fcb0ea00640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcaf006a430 0x7fcaf006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.022+0000 7fcb0ea00640 1 -- 172.21.15.39:0/2574747198 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcb10089170 con 0x7fcb1815c9e0 2024-01-24T17:42:34.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.023+0000 7fcb16800640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcaf006a430 0x7fcaf006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.024+0000 7fcb16800640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcaf006a430 0x7fcaf006c8f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcb08036000 tx=0x7fcb08008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:34.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.027+0000 7fcb0ea00640 1 -- 172.21.15.39:0/2574747198 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcb180b7330 con 0x7fcb1815c9e0 2024-01-24T17:42:34.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.224+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb180b7330 con 0x7fcb1815c9e0 2024-01-24T17:42:34.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.226+0000 7fcb0ea00640 1 -- 172.21.15.39:0/2574747198 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb180b7330 con 0x7fcb1815c9e0 2024-01-24T17:42:34.227 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:34.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcaf006a430 msgr2=0x7fcaf006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcaf006a430 0x7fcaf006c8f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcb08036000 tx=0x7fcb08008030 comp rx=0 tx=0).stop 2024-01-24T17:42:34.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 msgr2=0x7fcb181f5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb181f5570 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcb10007d90 tx=0x7fcb1000e4b0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 shutdown_connections 2024-01-24T17:42:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcaf006a430 0x7fcaf006c8f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcb1815c9e0 0x7fcb181f5570 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcb180cdb90 0x7fcb1814e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 --2- 172.21.15.39:0/2574747198 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcb180b8250 0x7fcb181f2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 >> 172.21.15.39:0/2574747198 conn(0x7fcb18071790 msgr2=0x7fcb180d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 shutdown_connections 2024-01-24T17:42:34.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.229+0000 7fcb1ca00640 1 -- 172.21.15.39:0/2574747198 wait complete. 2024-01-24T17:42:34.250 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:42:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.472+0000 7f8cabe00640 1 Processor -- start 2024-01-24T17:42:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.472+0000 7f8cabe00640 1 -- start start 2024-01-24T17:42:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.474+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cac0cb400 0x7f8cac15ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.474+0000 7f8cabe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac15b240 con 0x7f8cac0cb020 2024-01-24T17:42:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.474+0000 7f8cabe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac15b520 con 0x7f8cac0c8c10 2024-01-24T17:42:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.474+0000 7f8cabe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac15b800 con 0x7f8cac0cb400 2024-01-24T17:42:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.474+0000 7f8caaa00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8cac0cb020 0x7f8cac0ca4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47148/0 (socket says 172.21.15.39:47148) 2024-01-24T17:42:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.474+0000 7f8caaa00640 1 -- 172.21.15.39:0/2179392275 learned_addr learned my addr 172.21.15.39:0/2179392275 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601591745 0 0) 0x7f8cac15b240 con 0x7f8cac0cb020 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c003520 con 0x7f8cac0cb020 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8ca3e00640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cac0cb400 0x7f8cac15ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1987342217 0 0) 0x7f8c8c003520 con 0x7f8cac0cb020 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cac0cb400 msgr2=0x7f8cac15ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cac0cb400 0x7f8cac15ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 >> v1:172.21.15.100:6789/0 conn(0x7f8cac0c8c10 legacy=0x7f8cac15e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac15bae0 con 0x7f8cac0cb020 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8c9c001670 con 0x7f8cac0cb020 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 >> v1:172.21.15.39:6789/0 conn(0x7f8cac0cb020 legacy=0x7f8cac0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c8c0039a0 0x7f8c8c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c8c0047c0 0x7f8c8c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cac0cb400 0x7f8c8c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac15b800 con 0x7f8c8c0039a0 2024-01-24T17:42:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c00bb10 con 0x7f8c8c0047c0 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8c00bcd0 con 0x7f8cac0cb400 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.475+0000 7f8caaa00640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c8c0039a0 0x7f8c8c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.476+0000 7f8ca3e00640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c8c0047c0 0x7f8c8c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.476+0000 7f8caaa00640 1 -- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cac0cb400 msgr2=0x7f8c8c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.476+0000 7f8cab400640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cac0cb400 0x7f8c8c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.476+0000 7f8caaa00640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cac0cb400 0x7f8c8c005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.476+0000 7f8caaa00640 1 -- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c8c0047c0 msgr2=0x7f8c8c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.476+0000 7f8caaa00640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c8c0047c0 0x7f8c8c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.476+0000 7f8caaa00640 1 -- 172.21.15.39:0/2179392275 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c003520 con 0x7f8c8c0039a0 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.476+0000 7f8ca3e00640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c8c0047c0 0x7f8c8c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.477+0000 7f8caaa00640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c8c0039a0 0x7f8c8c003dc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00c0e0 tx=0x7f8c8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f22059328b2e87 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:34.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.477+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c005ad0 con 0x7f8c8c0039a0 2024-01-24T17:42:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.477+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c005c30 con 0x7f8c8c0039a0 2024-01-24T17:42:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.477+0000 7f8caa000640 1 -- 172.21.15.39:0/2179392275 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c02c810 con 0x7f8c8c0039a0 2024-01-24T17:42:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.478+0000 7f8cabe00640 1 -- 172.21.15.39:0/2179392275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c8c0039a0 msgr2=0x7f8c8c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.478+0000 7f8cabe00640 1 --2- 172.21.15.39:0/2179392275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c8c0039a0 0x7f8c8c003dc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00c0e0 tx=0x7f8c8c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.478+0000 7f8cab400640 1 -- 172.21.15.39:0/2179392275 reap_dead start 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.478+0000 7f8cabe00640 1 -- 172.21.15.39:0/2179392275 shutdown_connections 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.478+0000 7f8cabe00640 1 -- 172.21.15.39:0/2179392275 >> 172.21.15.39:0/2179392275 conn(0x7f8cac06a2a0 msgr2=0x7f8cac0afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.478+0000 7f8cabe00640 1 -- 172.21.15.39:0/2179392275 shutdown_connections 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.478+0000 7f8cabe00640 1 -- 172.21.15.39:0/2179392275 wait complete. 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.480+0000 7f8cabe00640 1 Processor -- start 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.480+0000 7f8cabe00640 1 -- start start 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c8c0039a0 0x7f8cac1550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cac0c5470 0x7f8cac14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cac0af410 0x7f8cac0af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cabe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac0c58d0 con 0x7f8cac0af410 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cabe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac14cf00 con 0x7f8cac0c5470 2024-01-24T17:42:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cabe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac0afdd0 con 0x7f8c8c0039a0 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cab400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cac0af410 0x7f8cac0af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cab400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cac0af410 0x7f8cac0af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44588/0 (socket says 172.21.15.39:44588) 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.481+0000 7f8cab400640 1 -- 172.21.15.39:0/4024224357 learned_addr learned my addr 172.21.15.39:0/4024224357 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8ca3e00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cac0c5470 0x7f8cac14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8caaa00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c8c0039a0 0x7f8cac1550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8cab400640 1 -- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c8c0039a0 msgr2=0x7f8cac1550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8cab400640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c8c0039a0 0x7f8cac1550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8cab400640 1 -- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cac0c5470 msgr2=0x7f8cac14c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8cab400640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cac0c5470 0x7f8cac14c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8cab400640 1 -- 172.21.15.39:0/4024224357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac0c82b0 con 0x7f8cac0af410 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8cab400640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cac0af410 0x7f8cac0af890 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0b4490 tx=0x7f8ca400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:34.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8ca2a00640 1 -- 172.21.15.39:0/4024224357 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4003b70 con 0x7f8cac0af410 2024-01-24T17:42:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8ca2a00640 1 -- 172.21.15.39:0/4024224357 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca4003cd0 con 0x7f8cac0af410 2024-01-24T17:42:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8ca2a00640 1 -- 172.21.15.39:0/4024224357 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40052b0 con 0x7f8cac0af410 2024-01-24T17:42:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.482+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac0cd800 con 0x7f8cac0af410 2024-01-24T17:42:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.483+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac0cddc0 con 0x7f8cac0af410 2024-01-24T17:42:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.484+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac0c9390 con 0x7f8cac0af410 2024-01-24T17:42:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.485+0000 7f8ca2a00640 1 -- 172.21.15.39:0/4024224357 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8ca4005410 con 0x7f8cac0af410 2024-01-24T17:42:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.485+0000 7f8ca2a00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8406a430 0x7f8c8406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.486+0000 7f8ca2a00640 1 -- 172.21.15.39:0/4024224357 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8ca40881b0 con 0x7f8cac0af410 2024-01-24T17:42:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.486+0000 7f8caaa00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8406a430 0x7f8c8406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:34.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.487+0000 7f8caaa00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8406a430 0x7f8c8406c8f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00c0e0 tx=0x7f8c9c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:34.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.490+0000 7f8ca2a00640 1 -- 172.21.15.39:0/4024224357 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8ca4053770 con 0x7f8cac0af410 2024-01-24T17:42:34.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.696+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8cac0c89f0 con 0x7f8cac0af410 2024-01-24T17:42:34.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.697+0000 7f8ca2a00640 1 -- 172.21.15.39:0/4024224357 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8ca4056d90 con 0x7f8cac0af410 2024-01-24T17:42:34.698 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8406a430 msgr2=0x7f8c8406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8406a430 0x7f8c8406c8f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00c0e0 tx=0x7f8c9c009030 comp rx=0 tx=0).stop 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cac0af410 msgr2=0x7f8cac0af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cac0af410 0x7f8cac0af890 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0b4490 tx=0x7f8ca400dc00 comp rx=0 tx=0).stop 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 shutdown_connections 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8406a430 0x7f8c8406c8f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cac0af410 0x7f8cac0af890 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cac0c5470 0x7f8cac14c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 --2- 172.21.15.39:0/4024224357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c8c0039a0 0x7f8cac1550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 >> 172.21.15.39:0/4024224357 conn(0x7f8cac06a2a0 msgr2=0x7f8cac15bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:34.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 shutdown_connections 2024-01-24T17:42:34.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:34.700+0000 7f8cabe00640 1 -- 172.21.15.39:0/4024224357 wait complete. 2024-01-24T17:42:34.720 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:42:34.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:34.720 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-01-24T17:42:34.821 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:42:34.821 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:42:34.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:34.822 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-24T17:42:34.878 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:42:34.878 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:34.935 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:42:34.935 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:34.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:35.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.216+0000 7f701f000640 1 Processor -- start 2024-01-24T17:42:35.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.217+0000 7f701f000640 1 -- start start 2024-01-24T17:42:35.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.218+0000 7f701f000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7018070970 0x7f70180c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.218+0000 7f701f000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f70180c9d80 0x7f70180cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.218+0000 7f701f000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180ca1b0 con 0x7f701815c4e0 2024-01-24T17:42:35.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.218+0000 7f701f000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7018070df0 con 0x7f7018070970 2024-01-24T17:42:35.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.218+0000 7f701f000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701815c8c0 con 0x7f70180c9d80 2024-01-24T17:42:35.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.218+0000 7f7017400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7018070970 0x7f70180c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.219+0000 7f7017400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7018070970 0x7f70180c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44602/0 (socket says 172.21.15.39:44602) 2024-01-24T17:42:35.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.219+0000 7f7017400640 1 -- 172.21.15.39:0/314550738 learned_addr learned my addr 172.21.15.39:0/314550738 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:35.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.219+0000 7f7017400640 1 -- 172.21.15.39:0/314550738 >> v2:172.21.15.100:3300/0 conn(0x7f70180c9d80 msgr2=0x7f70180cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:35.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.219+0000 7f7017400640 1 --2- 172.21.15.39:0/314550738 >> v2:172.21.15.100:3300/0 conn(0x7f70180c9d80 0x7f70180cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.219+0000 7f7017400640 1 -- 172.21.15.39:0/314550738 >> v1:172.21.15.100:6789/0 conn(0x7f701815c4e0 legacy=0x7f70180c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:35.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.219+0000 7f7017400640 1 -- 172.21.15.39:0/314550738 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701815da80 con 0x7f7018070970 2024-01-24T17:42:35.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.220+0000 7f7017400640 1 --2- 172.21.15.39:0/314550738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7018070970 0x7f70180c9840 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f701806e0d0 tx=0x7f700002cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9754faf73aff98d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:35.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.220+0000 7f7016a00640 1 -- 172.21.15.39:0/314550738 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700002dad0 con 0x7f7018070970 2024-01-24T17:42:35.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f7016a00640 1 -- 172.21.15.39:0/314550738 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700002dc30 con 0x7f7018070970 2024-01-24T17:42:35.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f7016a00640 1 -- 172.21.15.39:0/314550738 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000035440 con 0x7f7018070970 2024-01-24T17:42:35.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f701f000640 1 -- 172.21.15.39:0/314550738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7018070970 msgr2=0x7f70180c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f701f000640 1 --2- 172.21.15.39:0/314550738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7018070970 0x7f70180c9840 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f701806e0d0 tx=0x7f700002cd20 comp rx=0 tx=0).stop 2024-01-24T17:42:35.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f701f000640 1 -- 172.21.15.39:0/314550738 shutdown_connections 2024-01-24T17:42:35.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f701f000640 1 --2- 172.21.15.39:0/314550738 >> v2:172.21.15.100:3300/0 conn(0x7f70180c9d80 0x7f70180cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f701f000640 1 --2- 172.21.15.39:0/314550738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7018070970 0x7f70180c9840 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f701f000640 1 -- 172.21.15.39:0/314550738 >> 172.21.15.39:0/314550738 conn(0x7f701806a3e0 msgr2=0x7f701806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:35.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f701f000640 1 -- 172.21.15.39:0/314550738 shutdown_connections 2024-01-24T17:42:35.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.221+0000 7f701f000640 1 -- 172.21.15.39:0/314550738 wait complete. 2024-01-24T17:42:35.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.223+0000 7f701f000640 1 Processor -- start 2024-01-24T17:42:35.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.223+0000 7f701f000640 1 -- start start 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.224+0000 7f701f000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7018070970 0x7f70180afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.224+0000 7f701f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70180c9d80 0x7f701815a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.224+0000 7f701f000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f701815c4e0 0x7f701815a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.224+0000 7f701f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7018148220 con 0x7f70180c9d80 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.224+0000 7f701f000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70181f4ad0 con 0x7f701815c4e0 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.224+0000 7f701f000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7018148da0 con 0x7f7018070970 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70180c9d80 0x7f701815a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70180c9d80 0x7f701815a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44618/0 (socket says 172.21.15.39:44618) 2024-01-24T17:42:35.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017400640 1 -- 172.21.15.39:0/3554309088 learned_addr learned my addr 172.21.15.39:0/3554309088 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017e00640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7018070970 0x7f70180afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017400640 1 -- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7018070970 msgr2=0x7f70180afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017400640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7018070970 0x7f70180afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017400640 1 -- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f701815c4e0 msgr2=0x7f701815a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f701d000640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f701815c4e0 0x7f701815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017400640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f701815c4e0 0x7f701815a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017400640 1 -- 172.21.15.39:0/3554309088 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700000a670 con 0x7f70180c9d80 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f7017e00640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7018070970 0x7f70180afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.225+0000 7f701d000640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f701815c4e0 0x7f701815a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:35.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.226+0000 7f7017400640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70180c9d80 0x7f701815a0c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f700000a1c0 tx=0x7f7000004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:35.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.226+0000 7f700fe00640 1 -- 172.21.15.39:0/3554309088 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000040860 con 0x7f70180c9d80 2024-01-24T17:42:35.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.226+0000 7f700fe00640 1 -- 172.21.15.39:0/3554309088 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70000409e0 con 0x7f70180c9d80 2024-01-24T17:42:35.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.226+0000 7f700fe00640 1 -- 172.21.15.39:0/3554309088 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000040d90 con 0x7f70180c9d80 2024-01-24T17:42:35.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.226+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70181f3d10 con 0x7f70180c9d80 2024-01-24T17:42:35.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.226+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70181f42d0 con 0x7f70180c9d80 2024-01-24T17:42:35.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.227+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7018073ef0 con 0x7f70180c9d80 2024-01-24T17:42:35.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.228+0000 7f700fe00640 1 -- 172.21.15.39:0/3554309088 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f700003c070 con 0x7f70180c9d80 2024-01-24T17:42:35.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.229+0000 7f700fe00640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff006a430 0x7f6ff006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.230+0000 7f700fe00640 1 -- 172.21.15.39:0/3554309088 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f70000ab370 con 0x7f70180c9d80 2024-01-24T17:42:35.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.230+0000 7f7017e00640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff006a430 0x7f6ff006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.231+0000 7f7017e00640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff006a430 0x7f6ff006c8f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f700800a830 tx=0x7f7008008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:35.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.234+0000 7f700fe00640 1 -- 172.21.15.39:0/3554309088 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f70000769d0 con 0x7f70180c9d80 2024-01-24T17:42:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.432+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70180af760 con 0x7f70180c9d80 2024-01-24T17:42:35.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.434+0000 7f700fe00640 1 -- 172.21.15.39:0/3554309088 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f7000079ff0 con 0x7f70180c9d80 2024-01-24T17:42:35.435 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:35.436 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:35.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff006a430 msgr2=0x7f6ff006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff006a430 0x7f6ff006c8f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f700800a830 tx=0x7f7008008040 comp rx=0 tx=0).stop 2024-01-24T17:42:35.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70180c9d80 msgr2=0x7f701815a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70180c9d80 0x7f701815a0c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f700000a1c0 tx=0x7f7000004730 comp rx=0 tx=0).stop 2024-01-24T17:42:35.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 shutdown_connections 2024-01-24T17:42:35.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f701815c4e0 0x7f701815a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ff006a430 0x7f6ff006c8f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70180c9d80 0x7f701815a0c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 --2- 172.21.15.39:0/3554309088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7018070970 0x7f70180afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.437+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 >> 172.21.15.39:0/3554309088 conn(0x7f701806a3e0 msgr2=0x7f7018151c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:35.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.438+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 shutdown_connections 2024-01-24T17:42:35.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.438+0000 7f701f000640 1 -- 172.21.15.39:0/3554309088 wait complete. 2024-01-24T17:42:35.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:35.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:35.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.672+0000 7fa6eb200640 1 Processor -- start 2024-01-24T17:42:35.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.672+0000 7fa6eb200640 1 -- start start 2024-01-24T17:42:35.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.673+0000 7fa6eb200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa6ec06a380 0x7fa6ec0b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.673+0000 7fa6eb200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec071370 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.674+0000 7fa6eb200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec0ccea0 con 0x7fa6ec15a830 2024-01-24T17:42:35.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.674+0000 7fa6eb200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ec071f10 con 0x7fa6ec06a380 2024-01-24T17:42:35.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.674+0000 7fa6e9400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa6ec0b4b30 0x7fa6ec069c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47156/0 (socket says 172.21.15.39:47156) 2024-01-24T17:42:35.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.674+0000 7fa6e9400640 1 -- 172.21.15.39:0/15258267 learned_addr learned my addr 172.21.15.39:0/15258267 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:35.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.674+0000 7fa6ea800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa6ec06a380 0x7fa6ec0b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.674+0000 7fa6e8a00640 1 -- 172.21.15.39:0/15258267 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376492300 0 0) 0x7fa6ec071370 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.675+0000 7fa6e8a00640 1 -- 172.21.15.39:0/15258267 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6cc003520 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.675+0000 7fa6ea800640 1 -- 172.21.15.39:0/15258267 >> v1:172.21.15.100:6789/0 conn(0x7fa6ec15a830 legacy=0x7fa6ec071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.675+0000 7fa6ea800640 1 -- 172.21.15.39:0/15258267 >> v1:172.21.15.39:6789/0 conn(0x7fa6ec0b4b30 legacy=0x7fa6ec069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.675+0000 7fa6ea800640 1 -- 172.21.15.39:0/15258267 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0cc8e0 con 0x7fa6ec06a380 2024-01-24T17:42:35.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.676+0000 7fa6ea800640 1 --2- 172.21.15.39:0/15258267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec06a380 0x7fa6ec0b23a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009d90 tx=0x7fa6e402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f2262d158ad9e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:35.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.676+0000 7fa6e8a00640 1 -- 172.21.15.39:0/15258267 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e402cd00 con 0x7fa6ec06a380 2024-01-24T17:42:35.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.676+0000 7fa6e8a00640 1 -- 172.21.15.39:0/15258267 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e402ce60 con 0x7fa6ec06a380 2024-01-24T17:42:35.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.676+0000 7fa6e8a00640 1 -- 172.21.15.39:0/15258267 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4034660 con 0x7fa6ec06a380 2024-01-24T17:42:35.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.677+0000 7fa6eb200640 1 -- 172.21.15.39:0/15258267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec06a380 msgr2=0x7fa6ec0b23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.677+0000 7fa6eb200640 1 --2- 172.21.15.39:0/15258267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec06a380 0x7fa6ec0b23a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009d90 tx=0x7fa6e402bf50 comp rx=0 tx=0).stop 2024-01-24T17:42:35.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.677+0000 7fa6eb200640 1 -- 172.21.15.39:0/15258267 shutdown_connections 2024-01-24T17:42:35.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.677+0000 7fa6eb200640 1 --2- 172.21.15.39:0/15258267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec06a380 0x7fa6ec0b23a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.677+0000 7fa6eb200640 1 -- 172.21.15.39:0/15258267 >> 172.21.15.39:0/15258267 conn(0x7fa6ec0b10a0 msgr2=0x7fa6ec0b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:35.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.677+0000 7fa6eb200640 1 -- 172.21.15.39:0/15258267 shutdown_connections 2024-01-24T17:42:35.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.677+0000 7fa6eb200640 1 -- 172.21.15.39:0/15258267 wait complete. 2024-01-24T17:42:35.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.679+0000 7fa6eb200640 1 Processor -- start 2024-01-24T17:42:35.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.679+0000 7fa6eb200640 1 -- start start 2024-01-24T17:42:35.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.680+0000 7fa6eb200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6ec06a380 0x7fa6ec0cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.680+0000 7fa6eb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec0b4b30 0x7fa6ec0ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.680+0000 7fa6eb200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6ec15a830 0x7fa6ec0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.680+0000 7fa6eb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1f6260 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.680+0000 7fa6eb200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1f5370 con 0x7fa6ec15a830 2024-01-24T17:42:35.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.680+0000 7fa6eb200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec1f5650 con 0x7fa6ec06a380 2024-01-24T17:42:35.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.680+0000 7fa6e9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec0b4b30 0x7fa6ec0ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec0b4b30 0x7fa6ec0ada70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44624/0 (socket says 172.21.15.39:44624) 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9400640 1 -- 172.21.15.39:0/4144221793 learned_addr learned my addr 172.21.15.39:0/4144221793 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6ea800640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6ec15a830 0x7fa6ec0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9e00640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6ec06a380 0x7fa6ec0cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9400640 1 -- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6ec06a380 msgr2=0x7fa6ec0cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9400640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6ec06a380 0x7fa6ec0cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9400640 1 -- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6ec15a830 msgr2=0x7fa6ec0adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9400640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6ec15a830 0x7fa6ec0adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9400640 1 -- 172.21.15.39:0/4144221793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e4009a70 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6ea800640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6ec15a830 0x7fa6ec0adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.681+0000 7fa6e9400640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec0b4b30 0x7fa6ec0ada70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400c500 tx=0x7fa6d400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.682+0000 7fa6e2000640 1 -- 172.21.15.39:0/4144221793 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4005ce0 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.682+0000 7fa6e2000640 1 -- 172.21.15.39:0/4144221793 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d4005e40 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.682+0000 7fa6e2000640 1 -- 172.21.15.39:0/4144221793 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4007330 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.682+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec1f5930 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.682+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec1f5ef0 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.683+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec067040 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.684+0000 7fa6e2000640 1 -- 172.21.15.39:0/4144221793 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa6d4007550 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.685+0000 7fa6e2000640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6c406a430 0x7fa6c406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:35.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.685+0000 7fa6e2000640 1 -- 172.21.15.39:0/4144221793 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa6d4087ff0 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.686+0000 7fa6e9e00640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6c406a430 0x7fa6c406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:35.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.687+0000 7fa6e9e00640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6c406a430 0x7fa6c406c8f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000f60 tx=0x7fa6dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:35.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.690+0000 7fa6e2000640 1 -- 172.21.15.39:0/4144221793 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa6d4053670 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.881+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6ec0c4090 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.882+0000 7fa6e2000640 1 -- 172.21.15.39:0/4144221793 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa6d40170a0 con 0x7fa6ec0b4b30 2024-01-24T17:42:35.882 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:35.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.884+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6c406a430 msgr2=0x7fa6c406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.884+0000 7fa6eb200640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6c406a430 0x7fa6c406c8f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000f60 tx=0x7fa6dc009040 comp rx=0 tx=0).stop 2024-01-24T17:42:35.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.884+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec0b4b30 msgr2=0x7fa6ec0ada70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.884+0000 7fa6eb200640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec0b4b30 0x7fa6ec0ada70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400c500 tx=0x7fa6d400c9c0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.885+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 shutdown_connections 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.885+0000 7fa6eb200640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6ec15a830 0x7fa6ec0adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.885+0000 7fa6eb200640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6c406a430 0x7fa6c406c8f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.885+0000 7fa6eb200640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6ec0b4b30 0x7fa6ec0ada70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.885+0000 7fa6eb200640 1 --2- 172.21.15.39:0/4144221793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6ec06a380 0x7fa6ec0cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.885+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 >> 172.21.15.39:0/4144221793 conn(0x7fa6ec0b10a0 msgr2=0x7fa6ec069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.885+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 shutdown_connections 2024-01-24T17:42:35.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:35.885+0000 7fa6eb200640 1 -- 172.21.15.39:0/4144221793 wait complete. 2024-01-24T17:42:35.903 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:36.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.128+0000 7fca51400640 1 Processor -- start 2024-01-24T17:42:36.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.129+0000 7fca51400640 1 -- start start 2024-01-24T17:42:36.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.130+0000 7fca51400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fca4c161040 0x7fca4c06c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.130+0000 7fca51400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0c13f0 0x7fca4c0c1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.130+0000 7fca51400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4c06c430 con 0x7fca4c0ad530 2024-01-24T17:42:36.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.130+0000 7fca51400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca4c0c1dc0 con 0x7fca4c161040 2024-01-24T17:42:36.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.130+0000 7fca51400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c161420 con 0x7fca4c0c13f0 2024-01-24T17:42:36.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.130+0000 7fca43e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fca4c0ad530 0x7fca4c0b2410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47168/0 (socket says 172.21.15.39:47168) 2024-01-24T17:42:36.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.130+0000 7fca43e00640 1 -- 172.21.15.39:0/1793667733 learned_addr learned my addr 172.21.15.39:0/1793667733 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:36.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.130+0000 7fca4be00640 1 --2- 172.21.15.39:0/1793667733 >> v2:172.21.15.39:3300/0 conn(0x7fca4c161040 0x7fca4c06c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.131+0000 7fca50a00640 1 --2- 172.21.15.39:0/1793667733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0c13f0 0x7fca4c0c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.131+0000 7fca4be00640 1 -- 172.21.15.39:0/1793667733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0c13f0 msgr2=0x7fca4c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:36.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.131+0000 7fca4be00640 1 --2- 172.21.15.39:0/1793667733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0c13f0 0x7fca4c0c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.131+0000 7fca4be00640 1 -- 172.21.15.39:0/1793667733 >> v1:172.21.15.39:6789/0 conn(0x7fca4c0ad530 legacy=0x7fca4c0b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.131+0000 7fca4b400640 1 -- 172.21.15.39:0/1793667733 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891808699 0 0) 0x7fca4c06c430 con 0x7fca4c0ad530 2024-01-24T17:42:36.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.131+0000 7fca4be00640 1 -- 172.21.15.39:0/1793667733 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c0cf320 con 0x7fca4c161040 2024-01-24T17:42:36.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.132+0000 7fca4be00640 1 --2- 172.21.15.39:0/1793667733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c161040 0x7fca4c06c750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009790 tx=0x7fca3c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2cb0ea19732aa5d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:36.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.132+0000 7fca4b400640 1 -- 172.21.15.39:0/1793667733 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c0316f0 con 0x7fca4c161040 2024-01-24T17:42:36.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.132+0000 7fca4b400640 1 -- 172.21.15.39:0/1793667733 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c031850 con 0x7fca4c161040 2024-01-24T17:42:36.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca4b400640 1 -- 172.21.15.39:0/1793667733 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c031be0 con 0x7fca4c161040 2024-01-24T17:42:36.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca51400640 1 -- 172.21.15.39:0/1793667733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c161040 msgr2=0x7fca4c06c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca51400640 1 --2- 172.21.15.39:0/1793667733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c161040 0x7fca4c06c750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009790 tx=0x7fca3c02b790 comp rx=0 tx=0).stop 2024-01-24T17:42:36.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca51400640 1 -- 172.21.15.39:0/1793667733 shutdown_connections 2024-01-24T17:42:36.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca51400640 1 --2- 172.21.15.39:0/1793667733 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0c13f0 0x7fca4c0c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca51400640 1 --2- 172.21.15.39:0/1793667733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c161040 0x7fca4c06c750 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca51400640 1 -- 172.21.15.39:0/1793667733 >> 172.21.15.39:0/1793667733 conn(0x7fca4c069e50 msgr2=0x7fca4c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:36.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca51400640 1 -- 172.21.15.39:0/1793667733 shutdown_connections 2024-01-24T17:42:36.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.133+0000 7fca51400640 1 -- 172.21.15.39:0/1793667733 wait complete. 2024-01-24T17:42:36.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.135+0000 7fca51400640 1 Processor -- start 2024-01-24T17:42:36.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.135+0000 7fca51400640 1 -- start start 2024-01-24T17:42:36.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.136+0000 7fca51400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0ad530 0x7fca4c15d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.136+0000 7fca51400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c0c13f0 0x7fca4c163ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.136+0000 7fca51400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca4c161040 0x7fca4c0c2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.136+0000 7fca51400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c163a40 con 0x7fca4c0c13f0 2024-01-24T17:42:36.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.136+0000 7fca51400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c0c3830 con 0x7fca4c161040 2024-01-24T17:42:36.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.136+0000 7fca51400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c0c3550 con 0x7fca4c0ad530 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c0c13f0 0x7fca4c163ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c0c13f0 0x7fca4c163ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44632/0 (socket says 172.21.15.39:44632) 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 -- 172.21.15.39:0/4279164585 learned_addr learned my addr 172.21.15.39:0/4279164585 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca4be00640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0ad530 0x7fca4c15d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca50a00640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca4c161040 0x7fca4c0c2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 -- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0ad530 msgr2=0x7fca4c15d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0ad530 0x7fca4c15d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 -- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca4c161040 msgr2=0x7fca4c0c2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca4c161040 0x7fca4c0c2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 -- 172.21.15.39:0/4279164585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c0093a0 con 0x7fca4c0c13f0 2024-01-24T17:42:36.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.137+0000 7fca43e00640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c0c13f0 0x7fca4c163ed0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fca3400c430 tx=0x7fca3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:36.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.138+0000 7fca48c00640 1 -- 172.21.15.39:0/4279164585 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca340074e0 con 0x7fca4c0c13f0 2024-01-24T17:42:36.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.138+0000 7fca48c00640 1 -- 172.21.15.39:0/4279164585 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca34007660 con 0x7fca4c0c13f0 2024-01-24T17:42:36.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.138+0000 7fca48c00640 1 -- 172.21.15.39:0/4279164585 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca34004a20 con 0x7fca4c0c13f0 2024-01-24T17:42:36.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.138+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4c1633f0 con 0x7fca4c0c13f0 2024-01-24T17:42:36.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.138+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4c0c3190 con 0x7fca4c0c13f0 2024-01-24T17:42:36.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.139+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4c163250 con 0x7fca4c0c13f0 2024-01-24T17:42:36.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.140+0000 7fca48c00640 1 -- 172.21.15.39:0/4279164585 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fca34004b80 con 0x7fca4c0c13f0 2024-01-24T17:42:36.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.141+0000 7fca48c00640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2406a430 0x7fca2406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.141+0000 7fca48c00640 1 -- 172.21.15.39:0/4279164585 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fca340887f0 con 0x7fca4c0c13f0 2024-01-24T17:42:36.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.142+0000 7fca4be00640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2406a430 0x7fca2406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.143+0000 7fca4be00640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2406a430 0x7fca2406c8f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fca3c036000 tx=0x7fca3c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:36.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.145+0000 7fca48c00640 1 -- 172.21.15.39:0/4279164585 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fca4c163250 con 0x7fca4c0c13f0 2024-01-24T17:42:36.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.339+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca4c163250 con 0x7fca4c0c13f0 2024-01-24T17:42:36.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.340+0000 7fca48c00640 1 -- 172.21.15.39:0/4279164585 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fca4c163250 con 0x7fca4c0c13f0 2024-01-24T17:42:36.340 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:36.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.342+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2406a430 msgr2=0x7fca2406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.342+0000 7fca51400640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2406a430 0x7fca2406c8f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fca3c036000 tx=0x7fca3c008030 comp rx=0 tx=0).stop 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.342+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c0c13f0 msgr2=0x7fca4c163ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.342+0000 7fca51400640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c0c13f0 0x7fca4c163ed0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fca3400c430 tx=0x7fca3400c8f0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.343+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 shutdown_connections 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.343+0000 7fca51400640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca4c161040 0x7fca4c0c2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.343+0000 7fca51400640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2406a430 0x7fca2406c8f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.343+0000 7fca51400640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca4c0c13f0 0x7fca4c163ed0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.343+0000 7fca51400640 1 --2- 172.21.15.39:0/4279164585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4c0ad530 0x7fca4c15d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.343+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 >> 172.21.15.39:0/4279164585 conn(0x7fca4c069e50 msgr2=0x7fca4c06a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.343+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 shutdown_connections 2024-01-24T17:42:36.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.343+0000 7fca51400640 1 -- 172.21.15.39:0/4279164585 wait complete. 2024-01-24T17:42:36.363 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:36.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.592+0000 7f31db200640 1 Processor -- start 2024-01-24T17:42:36.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.592+0000 7f31db200640 1 -- start start 2024-01-24T17:42:36.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.593+0000 7f31db200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc0accd0 0x7f31dc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.593+0000 7f31db200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f31dc0cb540 0x7f31dc0ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.593+0000 7f31db200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc0ad0b0 con 0x7f31dc06c3a0 2024-01-24T17:42:36.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.593+0000 7f31db200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc06d840 con 0x7f31dc0cb540 2024-01-24T17:42:36.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.593+0000 7f31db200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0b2370 con 0x7f31dc0accd0 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.593+0000 7f31d9e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc0accd0 0x7f31dc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.593+0000 7f31d9e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc0accd0 0x7f31dc06d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:35412/0 (socket says 172.21.15.39:35412) 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.594+0000 7f31d9e00640 1 -- 172.21.15.39:0/220757134 learned_addr learned my addr 172.21.15.39:0/220757134 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.594+0000 7f31da800640 1 --2- 172.21.15.39:0/220757134 >> v2:172.21.15.100:3300/0 conn(0x7f31dc0cb540 0x7f31dc0ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.594+0000 7f31d9e00640 1 -- 172.21.15.39:0/220757134 >> v2:172.21.15.100:3300/0 conn(0x7f31dc0cb540 msgr2=0x7f31dc0ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.594+0000 7f31d9e00640 1 --2- 172.21.15.39:0/220757134 >> v2:172.21.15.100:3300/0 conn(0x7f31dc0cb540 0x7f31dc0ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.594+0000 7f31d9e00640 1 -- 172.21.15.39:0/220757134 >> v1:172.21.15.100:6789/0 conn(0x7f31dc06c3a0 legacy=0x7f31dc0cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.594+0000 7f31d9e00640 1 -- 172.21.15.39:0/220757134 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc0b2650 con 0x7f31dc0accd0 2024-01-24T17:42:36.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.594+0000 7f31da800640 1 --2- 172.21.15.39:0/220757134 >> v2:172.21.15.100:3300/0 conn(0x7f31dc0cb540 0x7f31dc0ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:36.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.596+0000 7f31d9e00640 1 --2- 172.21.15.39:0/220757134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc0accd0 0x7f31dc06d300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009850 tx=0x7f31cc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c582ec73172469 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:36.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.596+0000 7f31d8a00640 1 -- 172.21.15.39:0/220757134 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc004230 con 0x7f31dc0accd0 2024-01-24T17:42:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.596+0000 7f31d8a00640 1 -- 172.21.15.39:0/220757134 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc004390 con 0x7f31dc0accd0 2024-01-24T17:42:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.596+0000 7f31d8a00640 1 -- 172.21.15.39:0/220757134 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc003810 con 0x7f31dc0accd0 2024-01-24T17:42:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.597+0000 7f31db200640 1 -- 172.21.15.39:0/220757134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc0accd0 msgr2=0x7f31dc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.597+0000 7f31db200640 1 --2- 172.21.15.39:0/220757134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc0accd0 0x7f31dc06d300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009850 tx=0x7f31cc02b960 comp rx=0 tx=0).stop 2024-01-24T17:42:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.597+0000 7f31db200640 1 -- 172.21.15.39:0/220757134 shutdown_connections 2024-01-24T17:42:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.597+0000 7f31db200640 1 --2- 172.21.15.39:0/220757134 >> v2:172.21.15.100:3300/0 conn(0x7f31dc0cb540 0x7f31dc0ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.597+0000 7f31db200640 1 --2- 172.21.15.39:0/220757134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc0accd0 0x7f31dc06d300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.597+0000 7f31db200640 1 -- 172.21.15.39:0/220757134 >> 172.21.15.39:0/220757134 conn(0x7f31dc069e50 msgr2=0x7f31dc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:36.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.597+0000 7f31db200640 1 -- 172.21.15.39:0/220757134 shutdown_connections 2024-01-24T17:42:36.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.597+0000 7f31db200640 1 -- 172.21.15.39:0/220757134 wait complete. 2024-01-24T17:42:36.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.599+0000 7f31db200640 1 Processor -- start 2024-01-24T17:42:36.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.599+0000 7f31db200640 1 -- start start 2024-01-24T17:42:36.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31db200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc06c3a0 0x7f31dc0c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31db200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31dc0accd0 0x7f31dc0c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31db200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31dc0cb540 0x7f31dc15b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31db200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc15d380 con 0x7f31dc0cb540 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31db200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc15c950 con 0x7f31dc0accd0 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31db200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc15ce80 con 0x7f31dc06c3a0 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31da800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31dc0cb540 0x7f31dc15b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31da800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31dc0cb540 0x7f31dc15b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44648/0 (socket says 172.21.15.39:44648) 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31da800640 1 -- 172.21.15.39:0/3832072072 learned_addr learned my addr 172.21.15.39:0/3832072072 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.600+0000 7f31d9e00640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc06c3a0 0x7f31dc0c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31d9400640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31dc0accd0 0x7f31dc0c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31da800640 1 -- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc06c3a0 msgr2=0x7f31dc0c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31da800640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc06c3a0 0x7f31dc0c8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31da800640 1 -- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31dc0accd0 msgr2=0x7f31dc0c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31da800640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31dc0accd0 0x7f31dc0c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31da800640 1 -- 172.21.15.39:0/3832072072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc009460 con 0x7f31dc0cb540 2024-01-24T17:42:36.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31d9e00640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc06c3a0 0x7f31dc0c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:36.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31da800640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31dc0cb540 0x7f31dc15b8d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f31d4007df0 tx=0x7f31d400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:36.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31d2000640 1 -- 172.21.15.39:0/3832072072 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4003990 con 0x7f31dc0cb540 2024-01-24T17:42:36.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31d2000640 1 -- 172.21.15.39:0/3832072072 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d400f040 con 0x7f31dc0cb540 2024-01-24T17:42:36.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31d2000640 1 -- 172.21.15.39:0/3832072072 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4003d30 con 0x7f31dc0cb540 2024-01-24T17:42:36.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.601+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc0c4890 con 0x7f31dc0cb540 2024-01-24T17:42:36.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.602+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc15d6d0 con 0x7f31dc0cb540 2024-01-24T17:42:36.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.603+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc04dff0 con 0x7f31dc0cb540 2024-01-24T17:42:36.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.604+0000 7f31d2000640 1 -- 172.21.15.39:0/3832072072 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f31d4004030 con 0x7f31dc0cb540 2024-01-24T17:42:36.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.605+0000 7f31d2000640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f31b406a430 0x7f31b406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:36.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.605+0000 7f31d2000640 1 -- 172.21.15.39:0/3832072072 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f31d4088b80 con 0x7f31dc0cb540 2024-01-24T17:42:36.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.606+0000 7f31d9e00640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f31b406a430 0x7f31b406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:36.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.606+0000 7f31d9e00640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f31b406a430 0x7f31b406c8f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f31cc035000 tx=0x7f31cc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:36.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.610+0000 7f31d2000640 1 -- 172.21.15.39:0/3832072072 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f31d40541e0 con 0x7f31dc0cb540 2024-01-24T17:42:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.808+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f31dc0cca30 con 0x7f31dc0cb540 2024-01-24T17:42:36.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.809+0000 7f31d2000640 1 -- 172.21.15.39:0/3832072072 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f31d401a0a0 con 0x7f31dc0cb540 2024-01-24T17:42:36.810 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f31b406a430 msgr2=0x7f31b406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f31b406a430 0x7f31b406c8f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f31cc035000 tx=0x7f31cc008030 comp rx=0 tx=0).stop 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31dc0cb540 msgr2=0x7f31dc15b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31dc0cb540 0x7f31dc15b8d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f31d4007df0 tx=0x7f31d400d4a0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 shutdown_connections 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f31b406a430 0x7f31b406c8f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31dc0cb540 0x7f31dc15b8d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31dc0accd0 0x7f31dc0c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 --2- 172.21.15.39:0/3832072072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31dc06c3a0 0x7f31dc0c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 >> 172.21.15.39:0/3832072072 conn(0x7f31dc069e50 msgr2=0x7f31dc15a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 shutdown_connections 2024-01-24T17:42:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:36.812+0000 7f31db200640 1 -- 172.21.15.39:0/3832072072 wait complete. 2024-01-24T17:42:36.832 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:42:37.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.058+0000 7efc7b600640 1 Processor -- start 2024-01-24T17:42:37.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.058+0000 7efc7b600640 1 -- start start 2024-01-24T17:42:37.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.059+0000 7efc7b600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efc7415cc90 0x7efc740d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.059+0000 7efc7b600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.059+0000 7efc7b600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc740ce160 con 0x7efc740b5b10 2024-01-24T17:42:37.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.059+0000 7efc7b600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7415d070 con 0x7efc7415cc90 2024-01-24T17:42:37.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.059+0000 7efc7b600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc740d03a0 con 0x7efc740cf9e0 2024-01-24T17:42:37.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.060+0000 7efc6be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.060+0000 7efc79600640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7efc740b5b10 0x7efc740b5fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:52340/0 (socket says 172.21.15.39:52340) 2024-01-24T17:42:37.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.060+0000 7efc78c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efc7415cc90 0x7efc740d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.060+0000 7efc79600640 1 -- 172.21.15.39:0/3699193624 learned_addr learned my addr 172.21.15.39:0/3699193624 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:37.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.061+0000 7efc6be00640 1 -- 172.21.15.39:0/3699193624 >> v2:172.21.15.100:3300/0 conn(0x7efc7415cc90 msgr2=0x7efc740d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:37.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.061+0000 7efc6be00640 1 --2- 172.21.15.39:0/3699193624 >> v2:172.21.15.100:3300/0 conn(0x7efc7415cc90 0x7efc740d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.061+0000 7efc6be00640 1 -- 172.21.15.39:0/3699193624 >> v1:172.21.15.100:6789/0 conn(0x7efc740b5b10 legacy=0x7efc740b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.061+0000 7efc6be00640 1 -- 172.21.15.39:0/3699193624 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc740d2500 con 0x7efc740cf9e0 2024-01-24T17:42:37.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.062+0000 7efc6be00640 1 --2- 172.21.15.39:0/3699193624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740cfe60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0097a0 tx=0x7efc5c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba186f6fcc195274 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:37.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.063+0000 7efc73e00640 1 -- 172.21.15.39:0/3699193624 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c031830 con 0x7efc740cf9e0 2024-01-24T17:42:37.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.063+0000 7efc73e00640 1 -- 172.21.15.39:0/3699193624 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5c031990 con 0x7efc740cf9e0 2024-01-24T17:42:37.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.063+0000 7efc73e00640 1 -- 172.21.15.39:0/3699193624 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c031d20 con 0x7efc740cf9e0 2024-01-24T17:42:37.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.063+0000 7efc7b600640 1 -- 172.21.15.39:0/3699193624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 msgr2=0x7efc740cfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.063+0000 7efc7b600640 1 --2- 172.21.15.39:0/3699193624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740cfe60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0097a0 tx=0x7efc5c02b8d0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.064+0000 7efc7b600640 1 -- 172.21.15.39:0/3699193624 shutdown_connections 2024-01-24T17:42:37.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.064+0000 7efc7b600640 1 --2- 172.21.15.39:0/3699193624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740cfe60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.064+0000 7efc7b600640 1 --2- 172.21.15.39:0/3699193624 >> v2:172.21.15.100:3300/0 conn(0x7efc7415cc90 0x7efc740d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.064+0000 7efc7b600640 1 -- 172.21.15.39:0/3699193624 >> 172.21.15.39:0/3699193624 conn(0x7efc74071790 msgr2=0x7efc74071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:37.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.064+0000 7efc7b600640 1 -- 172.21.15.39:0/3699193624 shutdown_connections 2024-01-24T17:42:37.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.064+0000 7efc7b600640 1 -- 172.21.15.39:0/3699193624 wait complete. 2024-01-24T17:42:37.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.066+0000 7efc7b600640 1 Processor -- start 2024-01-24T17:42:37.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.066+0000 7efc7b600640 1 -- start start 2024-01-24T17:42:37.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.067+0000 7efc7b600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740b5b10 0x7efc741f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.067+0000 7efc7b600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740b7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.067+0000 7efc7b600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc7415cc90 0x7efc7415f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.067+0000 7efc7b600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740cbb70 con 0x7efc740b5b10 2024-01-24T17:42:37.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.067+0000 7efc7b600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc741f32a0 con 0x7efc7415cc90 2024-01-24T17:42:37.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.067+0000 7efc7b600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc741f30a0 con 0x7efc740cf9e0 2024-01-24T17:42:37.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc78c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740b5b10 0x7efc741f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc78c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740b5b10 0x7efc741f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44662/0 (socket says 172.21.15.39:44662) 2024-01-24T17:42:37.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc78c00640 1 -- 172.21.15.39:0/2221888475 learned_addr learned my addr 172.21.15.39:0/2221888475 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:37.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc6be00640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc79600640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc7415cc90 0x7efc7415f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc78c00640 1 -- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 msgr2=0x7efc740b7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc78c00640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740b7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc78c00640 1 -- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc7415cc90 msgr2=0x7efc7415f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc78c00640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc7415cc90 0x7efc7415f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.068+0000 7efc78c00640 1 -- 172.21.15.39:0/2221888475 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5c009400 con 0x7efc740b5b10 2024-01-24T17:42:37.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.069+0000 7efc78c00640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740b5b10 0x7efc741f2b60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efc6400e9e0 tx=0x7efc6400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:37.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.069+0000 7efc71600640 1 -- 172.21.15.39:0/2221888475 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64009670 con 0x7efc740b5b10 2024-01-24T17:42:37.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.069+0000 7efc71600640 1 -- 172.21.15.39:0/2221888475 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6400c9d0 con 0x7efc740b5b10 2024-01-24T17:42:37.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.069+0000 7efc71600640 1 -- 172.21.15.39:0/2221888475 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6400cd80 con 0x7efc740b5b10 2024-01-24T17:42:37.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.069+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc740d0f40 con 0x7efc740b5b10 2024-01-24T17:42:37.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.069+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc740cb2d0 con 0x7efc740b5b10 2024-01-24T17:42:37.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.070+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc740c7540 con 0x7efc740b5b10 2024-01-24T17:42:37.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.071+0000 7efc71600640 1 -- 172.21.15.39:0/2221888475 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efc64013470 con 0x7efc740b5b10 2024-01-24T17:42:37.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.072+0000 7efc71600640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a430 0x7efc4c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.072+0000 7efc71600640 1 -- 172.21.15.39:0/2221888475 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efc64088bc0 con 0x7efc740b5b10 2024-01-24T17:42:37.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.073+0000 7efc6be00640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a430 0x7efc4c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.074+0000 7efc6be00640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a430 0x7efc4c06c8f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efc5c036000 tx=0x7efc5c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:37.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.076+0000 7efc71600640 1 -- 172.21.15.39:0/2221888475 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efc740c7540 con 0x7efc740b5b10 2024-01-24T17:42:37.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.285+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efc740c7540 con 0x7efc740b5b10 2024-01-24T17:42:37.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.286+0000 7efc71600640 1 -- 172.21.15.39:0/2221888475 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7efc740c7540 con 0x7efc740b5b10 2024-01-24T17:42:37.286 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:42:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a430 msgr2=0x7efc4c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a430 0x7efc4c06c8f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efc5c036000 tx=0x7efc5c03c040 comp rx=0 tx=0).stop 2024-01-24T17:42:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740b5b10 msgr2=0x7efc741f2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740b5b10 0x7efc741f2b60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efc6400e9e0 tx=0x7efc6400eea0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 shutdown_connections 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc7415cc90 0x7efc7415f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc740cf9e0 0x7efc740b7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a430 0x7efc4c06c8f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 --2- 172.21.15.39:0/2221888475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740b5b10 0x7efc741f2b60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 >> 172.21.15.39:0/2221888475 conn(0x7efc74071790 msgr2=0x7efc740b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 shutdown_connections 2024-01-24T17:42:37.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.288+0000 7efc7b600640 1 -- 172.21.15.39:0/2221888475 wait complete. 2024-01-24T17:42:37.307 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:42:37.307 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:42:37.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:37.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.571+0000 7f4f6ea00640 1 Processor -- start 2024-01-24T17:42:37.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.571+0000 7f4f6ea00640 1 -- start start 2024-01-24T17:42:37.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.572+0000 7f4f6ea00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4f6815ca30 0x7f4f680cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.573+0000 7f4f6ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680cd3c0 0x7f4f680c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.573+0000 7f4f6ea00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680c9c10 con 0x7f4f680c90d0 2024-01-24T17:42:37.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.573+0000 7f4f6ea00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f680c9650 con 0x7f4f6815ca30 2024-01-24T17:42:37.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.573+0000 7f4f6ea00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f680c9930 con 0x7f4f680cd3c0 2024-01-24T17:42:37.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f67e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4f6815ca30 0x7f4f680cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f5fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680cd3c0 0x7f4f680c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f67e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4f6815ca30 0x7f4f680cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:33694/0 (socket says 172.21.15.39:33694) 2024-01-24T17:42:37.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f67e00640 1 -- 172.21.15.39:0/2517146427 learned_addr learned my addr 172.21.15.39:0/2517146427 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f6ca00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f4f680c90d0 0x7f4f680caf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:52346/0 (socket says 172.21.15.39:52346) 2024-01-24T17:42:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f67e00640 1 -- 172.21.15.39:0/2517146427 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680cd3c0 msgr2=0x7f4f680c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f67e00640 1 --2- 172.21.15.39:0/2517146427 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680cd3c0 0x7f4f680c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f67e00640 1 -- 172.21.15.39:0/2517146427 >> v1:172.21.15.100:6789/0 conn(0x7f4f680c90d0 legacy=0x7f4f680caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.574+0000 7f4f67e00640 1 -- 172.21.15.39:0/2517146427 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f680cc680 con 0x7f4f6815ca30 2024-01-24T17:42:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.576+0000 7f4f67e00640 1 --2- 172.21.15.39:0/2517146427 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f6815ca30 0x7f4f680cce80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009870 tx=0x7f4f5802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c1b606c2241c359 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.576+0000 7f4f67400640 1 -- 172.21.15.39:0/2517146427 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58030840 con 0x7f4f6815ca30 2024-01-24T17:42:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.576+0000 7f4f67400640 1 -- 172.21.15.39:0/2517146427 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f580309a0 con 0x7f4f6815ca30 2024-01-24T17:42:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.576+0000 7f4f67400640 1 -- 172.21.15.39:0/2517146427 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58030d30 con 0x7f4f6815ca30 2024-01-24T17:42:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.577+0000 7f4f6ea00640 1 -- 172.21.15.39:0/2517146427 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f6815ca30 msgr2=0x7f4f680cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.577+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/2517146427 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f6815ca30 0x7f4f680cce80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009870 tx=0x7f4f5802b980 comp rx=0 tx=0).stop 2024-01-24T17:42:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.577+0000 7f4f6ea00640 1 -- 172.21.15.39:0/2517146427 shutdown_connections 2024-01-24T17:42:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.577+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/2517146427 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680cd3c0 0x7f4f680c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.577+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/2517146427 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f6815ca30 0x7f4f680cce80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.577+0000 7f4f6ea00640 1 -- 172.21.15.39:0/2517146427 >> 172.21.15.39:0/2517146427 conn(0x7f4f6806a3e0 msgr2=0x7f4f6806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.577+0000 7f4f6ea00640 1 -- 172.21.15.39:0/2517146427 shutdown_connections 2024-01-24T17:42:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.577+0000 7f4f6ea00640 1 -- 172.21.15.39:0/2517146427 wait complete. 2024-01-24T17:42:37.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.579+0000 7f4f6ea00640 1 Processor -- start 2024-01-24T17:42:37.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.579+0000 7f4f6ea00640 1 -- start start 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f6ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680c90d0 0x7f4f681f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f6ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6815ca30 0x7f4f6806c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f6ea00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f681f2d90 0x7f4f681f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f6ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6815ee40 con 0x7f4f6815ca30 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f6ea00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68143160 con 0x7f4f681f2d90 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f6ea00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f681f3760 con 0x7f4f680c90d0 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f5fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6815ca30 0x7f4f6806c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f5fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6815ca30 0x7f4f6806c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44672/0 (socket says 172.21.15.39:44672) 2024-01-24T17:42:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f5fe00640 1 -- 172.21.15.39:0/1023062865 learned_addr learned my addr 172.21.15.39:0/1023062865 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f67e00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680c90d0 0x7f4f681f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.580+0000 7f4f6ca00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f681f2d90 0x7f4f681f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f5fe00640 1 -- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680c90d0 msgr2=0x7f4f681f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f5fe00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680c90d0 0x7f4f681f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f5fe00640 1 -- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f681f2d90 msgr2=0x7f4f681f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f5fe00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f681f2d90 0x7f4f681f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f5fe00640 1 -- 172.21.15.39:0/1023062865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f580094d0 con 0x7f4f6815ca30 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f6ca00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f681f2d90 0x7f4f681f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f5fe00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6815ca30 0x7f4f6806c170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000e9b0 tx=0x7f4f5000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f64c00640 1 -- 172.21.15.39:0/1023062865 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f50009670 con 0x7f4f6815ca30 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f64c00640 1 -- 172.21.15.39:0/1023062865 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5000c9a0 con 0x7f4f6815ca30 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f64c00640 1 -- 172.21.15.39:0/1023062865 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5000cd50 con 0x7f4f6815ca30 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.581+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f681f5d90 con 0x7f4f6815ca30 2024-01-24T17:42:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.582+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f681f6310 con 0x7f4f6815ca30 2024-01-24T17:42:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.583+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6815ef80 con 0x7f4f6815ca30 2024-01-24T17:42:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.584+0000 7f4f64c00640 1 -- 172.21.15.39:0/1023062865 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4f50013470 con 0x7f4f6815ca30 2024-01-24T17:42:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.585+0000 7f4f64c00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4f4006a430 0x7f4f4006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.585+0000 7f4f64c00640 1 -- 172.21.15.39:0/1023062865 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4f50088b20 con 0x7f4f6815ca30 2024-01-24T17:42:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.585+0000 7f4f67e00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4f4006a430 0x7f4f4006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:37.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.586+0000 7f4f67e00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4f4006a430 0x7f4f4006c8f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f58036000 tx=0x7f4f5803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.589+0000 7f4f64c00640 1 -- 172.21.15.39:0/1023062865 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4f6815ef80 con 0x7f4f6815ca30 2024-01-24T17:42:37.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.782+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f6815ef80 con 0x7f4f6815ca30 2024-01-24T17:42:37.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.784+0000 7f4f64c00640 1 -- 172.21.15.39:0/1023062865 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f4f50085ef0 con 0x7f4f6815ca30 2024-01-24T17:42:37.785 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:37.786 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.787+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4f4006a430 msgr2=0x7f4f4006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.787+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4f4006a430 0x7f4f4006c8f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f58036000 tx=0x7f4f5803b040 comp rx=0 tx=0).stop 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.787+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6815ca30 msgr2=0x7f4f6806c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.787+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6815ca30 0x7f4f6806c170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000e9b0 tx=0x7f4f5000ee70 comp rx=0 tx=0).stop 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.788+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 shutdown_connections 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.788+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4f681f2d90 0x7f4f681f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.788+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4f4006a430 0x7f4f4006c8f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.788+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4f6815ca30 0x7f4f6806c170 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.788+0000 7f4f6ea00640 1 --2- 172.21.15.39:0/1023062865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4f680c90d0 0x7f4f681f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:37.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.788+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 >> 172.21.15.39:0/1023062865 conn(0x7f4f6806a3e0 msgr2=0x7f4f680acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:37.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.788+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 shutdown_connections 2024-01-24T17:42:37.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:37.788+0000 7f4f6ea00640 1 -- 172.21.15.39:0/1023062865 wait complete. 2024-01-24T17:42:37.807 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:37.807 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:38.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.025+0000 7f1b2b600640 1 Processor -- start 2024-01-24T17:42:38.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.025+0000 7f1b2b600640 1 -- start start 2024-01-24T17:42:38.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.027+0000 7f1b2b600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1b240b07f0 0x7f1b24160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.027+0000 7f1b2b600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b24070660 con 0x7f1b241607d0 2024-01-24T17:42:38.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.027+0000 7f1b2b600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b24071200 con 0x7f1b241600d0 2024-01-24T17:42:38.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.027+0000 7f1b2b600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b241612d0 con 0x7f1b240b07f0 2024-01-24T17:42:38.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.027+0000 7f1b1be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1b241607d0 0x7f1b240b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47176/0 (socket says 172.21.15.39:47176) 2024-01-24T17:42:38.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.027+0000 7f1b1be00640 1 -- 172.21.15.39:0/1540642654 learned_addr learned my addr 172.21.15.39:0/1540642654 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:38.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.027+0000 7f1b29600640 1 --2- 172.21.15.39:0/1540642654 >> v2:172.21.15.39:3300/0 conn(0x7f1b240b07f0 0x7f1b24160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.028+0000 7f1b23e00640 1 -- 172.21.15.39:0/1540642654 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395964408 0 0) 0x7f1b24070660 con 0x7f1b241607d0 2024-01-24T17:42:38.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.028+0000 7f1b23e00640 1 -- 172.21.15.39:0/1540642654 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04003520 con 0x7f1b241607d0 2024-01-24T17:42:38.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.028+0000 7f1b29600640 1 -- 172.21.15.39:0/1540642654 >> v1:172.21.15.100:6789/0 conn(0x7f1b241600d0 legacy=0x7f1b24070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.028+0000 7f1b29600640 1 -- 172.21.15.39:0/1540642654 >> v1:172.21.15.39:6789/0 conn(0x7f1b241607d0 legacy=0x7f1b240b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.028+0000 7f1b29600640 1 -- 172.21.15.39:0/1540642654 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b241615b0 con 0x7f1b240b07f0 2024-01-24T17:42:38.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.029+0000 7f1b29600640 1 --2- 172.21.15.39:0/1540642654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b240b07f0 0x7f1b24160c60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009d90 tx=0x7f1b1c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36caa49788f9afb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b23e00640 1 -- 172.21.15.39:0/1540642654 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c02cd00 con 0x7f1b240b07f0 2024-01-24T17:42:38.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b23e00640 1 -- 172.21.15.39:0/1540642654 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1c02ce60 con 0x7f1b240b07f0 2024-01-24T17:42:38.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b23e00640 1 -- 172.21.15.39:0/1540642654 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c034660 con 0x7f1b240b07f0 2024-01-24T17:42:38.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b2b600640 1 -- 172.21.15.39:0/1540642654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b240b07f0 msgr2=0x7f1b24160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b2b600640 1 --2- 172.21.15.39:0/1540642654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b240b07f0 0x7f1b24160c60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009d90 tx=0x7f1b1c02bf50 comp rx=0 tx=0).stop 2024-01-24T17:42:38.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b2b600640 1 -- 172.21.15.39:0/1540642654 shutdown_connections 2024-01-24T17:42:38.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b2b600640 1 --2- 172.21.15.39:0/1540642654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b240b07f0 0x7f1b24160c60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b2b600640 1 -- 172.21.15.39:0/1540642654 >> 172.21.15.39:0/1540642654 conn(0x7f1b240cb0e0 msgr2=0x7f1b240cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:38.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b2b600640 1 -- 172.21.15.39:0/1540642654 shutdown_connections 2024-01-24T17:42:38.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.030+0000 7f1b2b600640 1 -- 172.21.15.39:0/1540642654 wait complete. 2024-01-24T17:42:38.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.032+0000 7f1b2b600640 1 Processor -- start 2024-01-24T17:42:38.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.032+0000 7f1b2b600640 1 -- start start 2024-01-24T17:42:38.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b2b600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1b240b07f0 0x7f1b240b1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b2b600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b241600d0 0x7f1b24150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b2b600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1b241607d0 0x7f1b240b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b2b600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b241f6840 con 0x7f1b241600d0 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b2b600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b240b1a80 con 0x7f1b241607d0 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b2b600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b24150580 con 0x7f1b240b07f0 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b1be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b241600d0 0x7f1b24150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b1be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b241600d0 0x7f1b24150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44688/0 (socket says 172.21.15.39:44688) 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.033+0000 7f1b1be00640 1 -- 172.21.15.39:0/3546402930 learned_addr learned my addr 172.21.15.39:0/3546402930 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b1be00640 1 -- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1b240b07f0 msgr2=0x7f1b240b1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b28c00640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1b240b07f0 0x7f1b240b1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b29600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1b241607d0 0x7f1b240b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b1be00640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1b240b07f0 0x7f1b240b1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b1be00640 1 -- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1b241607d0 msgr2=0x7f1b240b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b1be00640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1b241607d0 0x7f1b240b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b1be00640 1 -- 172.21.15.39:0/3546402930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c009a70 con 0x7f1b241600d0 2024-01-24T17:42:38.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b29600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1b241607d0 0x7f1b240b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:38.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b28c00640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1b240b07f0 0x7f1b240b1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:38.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b1be00640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b241600d0 0x7f1b24150040 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00c500 tx=0x7f1b0c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.034+0000 7f1b21600640 1 -- 172.21.15.39:0/3546402930 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c005ce0 con 0x7f1b241600d0 2024-01-24T17:42:38.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.035+0000 7f1b21600640 1 -- 172.21.15.39:0/3546402930 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0c005e40 con 0x7f1b241600d0 2024-01-24T17:42:38.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.035+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b241f5cc0 con 0x7f1b241600d0 2024-01-24T17:42:38.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.035+0000 7f1b21600640 1 -- 172.21.15.39:0/3546402930 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c007330 con 0x7f1b241600d0 2024-01-24T17:42:38.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.035+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b241f6280 con 0x7f1b241600d0 2024-01-24T17:42:38.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.036+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b24073ef0 con 0x7f1b241600d0 2024-01-24T17:42:38.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.037+0000 7f1b21600640 1 -- 172.21.15.39:0/3546402930 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1b0c007490 con 0x7f1b241600d0 2024-01-24T17:42:38.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.038+0000 7f1b21600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1afc06a460 0x7f1afc06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.038+0000 7f1b21600640 1 -- 172.21.15.39:0/3546402930 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1b0c087fe0 con 0x7f1b241600d0 2024-01-24T17:42:38.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.039+0000 7f1b28c00640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1afc06a460 0x7f1afc06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.040+0000 7f1b28c00640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1afc06a460 0x7f1afc06c920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1b14000f60 tx=0x7f1b14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.043+0000 7f1b21600640 1 -- 172.21.15.39:0/3546402930 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1b0c053660 con 0x7f1b241600d0 2024-01-24T17:42:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.237+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b240cbe80 con 0x7f1b241600d0 2024-01-24T17:42:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.238+0000 7f1b21600640 1 -- 172.21.15.39:0/3546402930 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b0c0170a0 con 0x7f1b241600d0 2024-01-24T17:42:38.238 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.240+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1afc06a460 msgr2=0x7f1afc06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.240+0000 7f1b2b600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1afc06a460 0x7f1afc06c920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1b14000f60 tx=0x7f1b14009040 comp rx=0 tx=0).stop 2024-01-24T17:42:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.240+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b241600d0 msgr2=0x7f1b24150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.240+0000 7f1b2b600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b241600d0 0x7f1b24150040 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00c500 tx=0x7f1b0c00c9c0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.241+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 shutdown_connections 2024-01-24T17:42:38.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.241+0000 7f1b2b600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1b241607d0 0x7f1b240b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.241+0000 7f1b2b600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1afc06a460 0x7f1afc06c920 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.241+0000 7f1b2b600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b241600d0 0x7f1b24150040 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.241+0000 7f1b2b600640 1 --2- 172.21.15.39:0/3546402930 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1b240b07f0 0x7f1b240b1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.241+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 >> 172.21.15.39:0/3546402930 conn(0x7f1b240cb0e0 msgr2=0x7f1b240acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:38.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.241+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 shutdown_connections 2024-01-24T17:42:38.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.241+0000 7f1b2b600640 1 -- 172.21.15.39:0/3546402930 wait complete. 2024-01-24T17:42:38.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:38.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.475+0000 7f40b6000640 1 Processor -- start 2024-01-24T17:42:38.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.475+0000 7f40b6000640 1 -- start start 2024-01-24T17:42:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.476+0000 7f40b6000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f40b00b2230 0x7f40b00b2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.476+0000 7f40b6000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b00b2b90 con 0x7f40b015ab30 2024-01-24T17:42:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.476+0000 7f40b6000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b00cebf0 con 0x7f40b00ce760 2024-01-24T17:42:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.476+0000 7f40b6000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b00b6770 con 0x7f40b00b2230 2024-01-24T17:42:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.476+0000 7f40afe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f40b00b2230 0x7f40b00b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.476+0000 7f40af400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f40b015ab30 0x7f40b00cfc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47188/0 (socket says 172.21.15.39:47188) 2024-01-24T17:42:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.476+0000 7f40af400640 1 -- 172.21.15.39:0/1278600447 learned_addr learned my addr 172.21.15.39:0/1278600447 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:38.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.477+0000 7f40afe00640 1 -- 172.21.15.39:0/1278600447 >> v1:172.21.15.100:6789/0 conn(0x7f40b00ce760 legacy=0x7f40b00b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:38.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.477+0000 7f40afe00640 1 -- 172.21.15.39:0/1278600447 >> v1:172.21.15.39:6789/0 conn(0x7f40b015ab30 legacy=0x7f40b00cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.477+0000 7f40afe00640 1 -- 172.21.15.39:0/1278600447 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b00ceed0 con 0x7f40b00b2230 2024-01-24T17:42:38.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.478+0000 7f40afe00640 1 --2- 172.21.15.39:0/1278600447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b00b2650 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009d90 tx=0x7f40a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff94e2dd1159987 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.478+0000 7f40ae000640 1 -- 172.21.15.39:0/1278600447 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a802cc60 con 0x7f40b00b2230 2024-01-24T17:42:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.478+0000 7f40ae000640 1 -- 172.21.15.39:0/1278600447 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a802cdc0 con 0x7f40b00b2230 2024-01-24T17:42:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.478+0000 7f40ae000640 1 -- 172.21.15.39:0/1278600447 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a8034660 con 0x7f40b00b2230 2024-01-24T17:42:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.479+0000 7f40b6000640 1 -- 172.21.15.39:0/1278600447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 msgr2=0x7f40b00b2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.479+0000 7f40b6000640 1 --2- 172.21.15.39:0/1278600447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b00b2650 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009d90 tx=0x7f40a802beb0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.479+0000 7f40b6000640 1 -- 172.21.15.39:0/1278600447 shutdown_connections 2024-01-24T17:42:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.479+0000 7f40b6000640 1 --2- 172.21.15.39:0/1278600447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b00b2650 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.479+0000 7f40b6000640 1 -- 172.21.15.39:0/1278600447 >> 172.21.15.39:0/1278600447 conn(0x7f40b00cd230 msgr2=0x7f40b00cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.479+0000 7f40b6000640 1 -- 172.21.15.39:0/1278600447 shutdown_connections 2024-01-24T17:42:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.479+0000 7f40b6000640 1 -- 172.21.15.39:0/1278600447 wait complete. 2024-01-24T17:42:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.481+0000 7f40b6000640 1 Processor -- start 2024-01-24T17:42:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.481+0000 7f40b6000640 1 -- start start 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40b6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b01f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40b6000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40b015ab30 0x7f40b015d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40b6000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40b01f1f10 0x7f40b01f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40b6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b015dd60 con 0x7f40b00b2230 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40b6000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b01f5510 con 0x7f40b01f1f10 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40b6000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b01f2a00 con 0x7f40b015ab30 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40af400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b01f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40af400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b01f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44714/0 (socket says 172.21.15.39:44714) 2024-01-24T17:42:38.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.482+0000 7f40af400640 1 -- 172.21.15.39:0/960639327 learned_addr learned my addr 172.21.15.39:0/960639327 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40aea00640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40b015ab30 0x7f40b015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40afe00640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40b01f1f10 0x7f40b01f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40af400640 1 -- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40b015ab30 msgr2=0x7f40b015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40af400640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40b015ab30 0x7f40b015d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40af400640 1 -- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40b01f1f10 msgr2=0x7f40b01f23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40af400640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40b01f1f10 0x7f40b01f23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40af400640 1 -- 172.21.15.39:0/960639327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a8009a70 con 0x7f40b00b2230 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40aea00640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40b015ab30 0x7f40b015d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40afe00640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40b01f1f10 0x7f40b01f23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40af400640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b01f5f60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c480 tx=0x7f40a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40a7400640 1 -- 172.21.15.39:0/960639327 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0005ce0 con 0x7f40b00b2230 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40a7400640 1 -- 172.21.15.39:0/960639327 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a0005e40 con 0x7f40b00b2230 2024-01-24T17:42:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40a7400640 1 -- 172.21.15.39:0/960639327 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a00072f0 con 0x7f40b00b2230 2024-01-24T17:42:38.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.483+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b01f2ce0 con 0x7f40b00b2230 2024-01-24T17:42:38.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.484+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b01f5a40 con 0x7f40b00b2230 2024-01-24T17:42:38.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.485+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b00cf190 con 0x7f40b00b2230 2024-01-24T17:42:38.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.486+0000 7f40a7400640 1 -- 172.21.15.39:0/960639327 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f40a0007510 con 0x7f40b00b2230 2024-01-24T17:42:38.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.487+0000 7f40a7400640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f408806a430 0x7f408806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.487+0000 7f40a7400640 1 -- 172.21.15.39:0/960639327 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f40a0087d90 con 0x7f40b00b2230 2024-01-24T17:42:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.487+0000 7f40aea00640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f408806a430 0x7f408806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.489+0000 7f40aea00640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f408806a430 0x7f408806c8f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40b00d1210 tx=0x7f4098009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.492+0000 7f40a7400640 1 -- 172.21.15.39:0/960639327 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f40b00cf190 con 0x7f40b00b2230 2024-01-24T17:42:38.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.686+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40b00cf190 con 0x7f40b00b2230 2024-01-24T17:42:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.687+0000 7f40a7400640 1 -- 172.21.15.39:0/960639327 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f40b00cf190 con 0x7f40b00b2230 2024-01-24T17:42:38.688 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:38.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f408806a430 msgr2=0x7f408806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f408806a430 0x7f408806c8f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40b00d1210 tx=0x7f4098009040 comp rx=0 tx=0).stop 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 msgr2=0x7f40b01f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b01f5f60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c480 tx=0x7f40a000c940 comp rx=0 tx=0).stop 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 shutdown_connections 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40b01f1f10 0x7f40b01f23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40b015ab30 0x7f40b015d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f408806a430 0x7f408806c8f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 --2- 172.21.15.39:0/960639327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40b00b2230 0x7f40b01f5f60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.690+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 >> 172.21.15.39:0/960639327 conn(0x7f40b00cd230 msgr2=0x7f40b00b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.691+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 shutdown_connections 2024-01-24T17:42:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.691+0000 7f40b6000640 1 -- 172.21.15.39:0/960639327 wait complete. 2024-01-24T17:42:38.709 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:38.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.932+0000 7fedbac00640 1 Processor -- start 2024-01-24T17:42:38.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.932+0000 7fedbac00640 1 -- start start 2024-01-24T17:42:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.933+0000 7fedbac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fedb40d2e80 0x7fedb40d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.933+0000 7fedbac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb40b38a0 con 0x7fedb415c8d0 2024-01-24T17:42:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.933+0000 7fedbac00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb40d13e0 con 0x7fedb40b2710 2024-01-24T17:42:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.933+0000 7fedbac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb40d0ab0 con 0x7fedb40d2e80 2024-01-24T17:42:38.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.934+0000 7fedb9800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fedb415c8d0 0x7fedb40d2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47200/0 (socket says 172.21.15.39:47200) 2024-01-24T17:42:38.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.934+0000 7fedb9800640 1 -- 172.21.15.39:0/2684802528 learned_addr learned my addr 172.21.15.39:0/2684802528 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:38.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.934+0000 7fedb8e00640 1 --2- 172.21.15.39:0/2684802528 >> v2:172.21.15.39:3300/0 conn(0x7fedb40d2e80 0x7fedb40d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.934+0000 7fedb8e00640 1 -- 172.21.15.39:0/2684802528 >> v1:172.21.15.100:6789/0 conn(0x7fedb40b2710 legacy=0x7fedb40cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:38.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.934+0000 7fedb8e00640 1 -- 172.21.15.39:0/2684802528 >> v1:172.21.15.39:6789/0 conn(0x7fedb415c8d0 legacy=0x7fedb40d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.934+0000 7fedb8e00640 1 -- 172.21.15.39:0/2684802528 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb40d0d90 con 0x7fedb40d2e80 2024-01-24T17:42:38.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.935+0000 7fedb8e00640 1 --2- 172.21.15.39:0/2684802528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb40d2e80 0x7fedb40d03b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0097a0 tx=0x7fed9c02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d8f2ead304e47b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.936+0000 7fedb3e00640 1 -- 172.21.15.39:0/2684802528 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c030710 con 0x7fedb40d2e80 2024-01-24T17:42:38.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.936+0000 7fedb3e00640 1 -- 172.21.15.39:0/2684802528 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c02e050 con 0x7fedb40d2e80 2024-01-24T17:42:38.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.936+0000 7fedb3e00640 1 -- 172.21.15.39:0/2684802528 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c030a90 con 0x7fedb40d2e80 2024-01-24T17:42:38.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.936+0000 7fedbac00640 1 -- 172.21.15.39:0/2684802528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb40d2e80 msgr2=0x7fedb40d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.936+0000 7fedbac00640 1 --2- 172.21.15.39:0/2684802528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb40d2e80 0x7fedb40d03b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0097a0 tx=0x7fed9c02b710 comp rx=0 tx=0).stop 2024-01-24T17:42:38.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.937+0000 7fedbac00640 1 -- 172.21.15.39:0/2684802528 shutdown_connections 2024-01-24T17:42:38.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.937+0000 7fedbac00640 1 --2- 172.21.15.39:0/2684802528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb40d2e80 0x7fedb40d03b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.937+0000 7fedbac00640 1 -- 172.21.15.39:0/2684802528 >> 172.21.15.39:0/2684802528 conn(0x7fedb4071dd0 msgr2=0x7fedb409bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:38.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.937+0000 7fedbac00640 1 -- 172.21.15.39:0/2684802528 shutdown_connections 2024-01-24T17:42:38.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.937+0000 7fedbac00640 1 -- 172.21.15.39:0/2684802528 wait complete. 2024-01-24T17:42:38.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.939+0000 7fedbac00640 1 Processor -- start 2024-01-24T17:42:38.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.939+0000 7fedbac00640 1 -- start start 2024-01-24T17:42:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedbac00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedb40b2710 0x7fedb41f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedbac00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedb40d2e80 0x7fedb4154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedbac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb415c8d0 0x7fedb40ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedbac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40cb380 con 0x7fedb415c8d0 2024-01-24T17:42:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedbac00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb41f5c90 con 0x7fedb40b2710 2024-01-24T17:42:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedbac00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb41576c0 con 0x7fedb40d2e80 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedba200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb415c8d0 0x7fedb40ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedba200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb415c8d0 0x7fedb40ca730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44730/0 (socket says 172.21.15.39:44730) 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.940+0000 7fedba200640 1 -- 172.21.15.39:0/2965732431 learned_addr learned my addr 172.21.15.39:0/2965732431 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedb8e00640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedb40d2e80 0x7fedb4154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedb9800640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedb40b2710 0x7fedb41f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedba200640 1 -- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedb40d2e80 msgr2=0x7fedb4154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedba200640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedb40d2e80 0x7fedb4154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedba200640 1 -- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedb40b2710 msgr2=0x7fedb41f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedba200640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedb40b2710 0x7fedb41f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:38.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedba200640 1 -- 172.21.15.39:0/2965732431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c009400 con 0x7fedb415c8d0 2024-01-24T17:42:38.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedba200640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb415c8d0 0x7fedb40ca730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fedac00c410 tx=0x7fedac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedb1600640 1 -- 172.21.15.39:0/2965732431 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac011790 con 0x7fedb415c8d0 2024-01-24T17:42:38.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedb1600640 1 -- 172.21.15.39:0/2965732431 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac0118f0 con 0x7fedb415c8d0 2024-01-24T17:42:38.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.941+0000 7fedb1600640 1 -- 172.21.15.39:0/2965732431 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac011ce0 con 0x7fedb415c8d0 2024-01-24T17:42:38.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.942+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb4154d90 con 0x7fedb415c8d0 2024-01-24T17:42:38.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.942+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb41578d0 con 0x7fedb415c8d0 2024-01-24T17:42:38.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.943+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb41f2e40 con 0x7fedb415c8d0 2024-01-24T17:42:38.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.944+0000 7fedb1600640 1 -- 172.21.15.39:0/2965732431 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fedac01c460 con 0x7fedb415c8d0 2024-01-24T17:42:38.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.945+0000 7fedb1600640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fed8c06a430 0x7fed8c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:38.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.945+0000 7fedb1600640 1 -- 172.21.15.39:0/2965732431 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fedac088600 con 0x7fedb415c8d0 2024-01-24T17:42:38.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.946+0000 7fedb9800640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fed8c06a430 0x7fed8c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:38.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.947+0000 7fedb9800640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fed8c06a430 0x7fed8c06c8f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7feda40012e0 tx=0x7feda4005b70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:38.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:38.949+0000 7fedb1600640 1 -- 172.21.15.39:0/2965732431 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fedac053bc0 con 0x7fedb415c8d0 2024-01-24T17:42:39.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.147+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fedb404dff0 con 0x7fedb415c8d0 2024-01-24T17:42:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.147+0000 7fedb1600640 1 -- 172.21.15.39:0/2965732431 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fedac0571e0 con 0x7fedb415c8d0 2024-01-24T17:42:39.148 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:39.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fed8c06a430 msgr2=0x7fed8c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:39.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fed8c06a430 0x7fed8c06c8f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7feda40012e0 tx=0x7feda4005b70 comp rx=0 tx=0).stop 2024-01-24T17:42:39.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb415c8d0 msgr2=0x7fedb40ca730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:39.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb415c8d0 0x7fedb40ca730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fedac00c410 tx=0x7fedac00c8d0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 shutdown_connections 2024-01-24T17:42:39.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fed8c06a430 0x7fed8c06c8f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedb415c8d0 0x7fedb40ca730 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedb40d2e80 0x7fedb4154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 --2- 172.21.15.39:0/2965732431 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedb40b2710 0x7fedb41f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 >> 172.21.15.39:0/2965732431 conn(0x7fedb4071dd0 msgr2=0x7fedb40d2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 shutdown_connections 2024-01-24T17:42:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.150+0000 7fedbac00640 1 -- 172.21.15.39:0/2965732431 wait complete. 2024-01-24T17:42:39.169 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:42:39.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.378+0000 7f607e200640 1 Processor -- start 2024-01-24T17:42:39.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.378+0000 7f607e200640 1 -- start start 2024-01-24T17:42:39.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.379+0000 7f607e200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb020 0x7f607815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:39.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.379+0000 7f607e200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f60780cb400 0x7f60780ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:39.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.379+0000 7f607e200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607815b240 con 0x7f60780c8c10 2024-01-24T17:42:39.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.380+0000 7f607e200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607815b520 con 0x7f60780cb400 2024-01-24T17:42:39.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.380+0000 7f607e200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607815b800 con 0x7f60780cb020 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.380+0000 7f607ce00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb020 0x7f607815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.380+0000 7f607d800640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f60780c8c10 0x7f607815e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:52386/0 (socket says 172.21.15.39:52386) 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.380+0000 7f607d800640 1 -- 172.21.15.39:0/1305745505 learned_addr learned my addr 172.21.15.39:0/1305745505 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.380+0000 7f607ce00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb020 0x7f607815ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:35506/0 (socket says 172.21.15.39:35506) 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.380+0000 7f6077e00640 1 --2- 172.21.15.39:0/1305745505 >> v2:172.21.15.100:3300/0 conn(0x7f60780cb400 0x7f60780ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.381+0000 7f6077400640 1 -- 172.21.15.39:0/1305745505 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772330976 0 0) 0x7f607815b240 con 0x7f60780c8c10 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.381+0000 7f6077400640 1 -- 172.21.15.39:0/1305745505 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6058003520 con 0x7f60780c8c10 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.381+0000 7f607ce00640 1 -- 172.21.15.39:0/1305745505 >> v2:172.21.15.100:3300/0 conn(0x7f60780cb400 msgr2=0x7f60780ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.381+0000 7f607ce00640 1 --2- 172.21.15.39:0/1305745505 >> v2:172.21.15.100:3300/0 conn(0x7f60780cb400 0x7f60780ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.381+0000 7f607ce00640 1 -- 172.21.15.39:0/1305745505 >> v1:172.21.15.100:6789/0 conn(0x7f60780c8c10 legacy=0x7f607815e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:39.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.381+0000 7f607ce00640 1 -- 172.21.15.39:0/1305745505 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607815bae0 con 0x7f60780cb020 2024-01-24T17:42:39.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.383+0000 7f607ce00640 1 --2- 172.21.15.39:0/1305745505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb020 0x7f607815ac60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60680097a0 tx=0x7f606802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ce64f2fb519d9c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:39.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.383+0000 7f6077400640 1 -- 172.21.15.39:0/1305745505 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068031830 con 0x7f60780cb020 2024-01-24T17:42:39.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.383+0000 7f6077400640 1 -- 172.21.15.39:0/1305745505 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6068031990 con 0x7f60780cb020 2024-01-24T17:42:39.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.383+0000 7f6077400640 1 -- 172.21.15.39:0/1305745505 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068031d00 con 0x7f60780cb020 2024-01-24T17:42:39.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.384+0000 7f607e200640 1 -- 172.21.15.39:0/1305745505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb020 msgr2=0x7f607815ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:39.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.384+0000 7f607e200640 1 --2- 172.21.15.39:0/1305745505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb020 0x7f607815ac60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f60680097a0 tx=0x7f606802b8d0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.384+0000 7f607e200640 1 -- 172.21.15.39:0/1305745505 shutdown_connections 2024-01-24T17:42:39.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.384+0000 7f607e200640 1 --2- 172.21.15.39:0/1305745505 >> v2:172.21.15.100:3300/0 conn(0x7f60780cb400 0x7f60780ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.384+0000 7f607e200640 1 --2- 172.21.15.39:0/1305745505 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb020 0x7f607815ac60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.384+0000 7f607e200640 1 -- 172.21.15.39:0/1305745505 >> 172.21.15.39:0/1305745505 conn(0x7f607806a2a0 msgr2=0x7f60780afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:39.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.384+0000 7f607e200640 1 -- 172.21.15.39:0/1305745505 shutdown_connections 2024-01-24T17:42:39.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.384+0000 7f607e200640 1 -- 172.21.15.39:0/1305745505 wait complete. 2024-01-24T17:42:39.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.386+0000 7f607e200640 1 Processor -- start 2024-01-24T17:42:39.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.386+0000 7f607e200640 1 -- start start 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f607e200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60780c8c10 0x7f60780aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f607e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60780cb020 0x7f60780b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f607e200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb400 0x7f60781f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f607e200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60780c5630 con 0x7f60780cb020 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f607e200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60780afd80 con 0x7f60780c8c10 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f607e200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60780af4e0 con 0x7f60780cb400 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f6077e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60780cb020 0x7f60780b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f6077e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60780cb020 0x7f60780b3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44736/0 (socket says 172.21.15.39:44736) 2024-01-24T17:42:39.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.387+0000 7f6077e00640 1 -- 172.21.15.39:0/975361634 learned_addr learned my addr 172.21.15.39:0/975361634 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:39.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f607d800640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb400 0x7f60781f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6077e00640 1 -- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb400 msgr2=0x7f60781f2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f607ce00640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60780c8c10 0x7f60780aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6077e00640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb400 0x7f60781f2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6077e00640 1 -- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60780c8c10 msgr2=0x7f60780aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6077e00640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60780c8c10 0x7f60780aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6077e00640 1 -- 172.21.15.39:0/975361634 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6068009400 con 0x7f60780cb020 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f607d800640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb400 0x7f60781f2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f607ce00640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60780c8c10 0x7f60780aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6077e00640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60780cb020 0x7f60780b3680 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f606000e9b0 tx=0x7f606000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6074c00640 1 -- 172.21.15.39:0/975361634 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060009670 con 0x7f60780cb020 2024-01-24T17:42:39.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6074c00640 1 -- 172.21.15.39:0/975361634 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606000c9a0 con 0x7f60780cb020 2024-01-24T17:42:39.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f6074c00640 1 -- 172.21.15.39:0/975361634 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606000cd50 con 0x7f60780cb020 2024-01-24T17:42:39.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.388+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60780af7c0 con 0x7f60780cb020 2024-01-24T17:42:39.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.389+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60780c6d00 con 0x7f60780cb020 2024-01-24T17:42:39.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.390+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607804dff0 con 0x7f60780cb020 2024-01-24T17:42:39.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.391+0000 7f6074c00640 1 -- 172.21.15.39:0/975361634 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6060013470 con 0x7f60780cb020 2024-01-24T17:42:39.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.392+0000 7f6074c00640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f605006a460 0x7f605006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:39.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.392+0000 7f6074c00640 1 -- 172.21.15.39:0/975361634 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6060088b10 con 0x7f60780cb020 2024-01-24T17:42:39.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.392+0000 7f607ce00640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f605006a460 0x7f605006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:39.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.394+0000 7f607ce00640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f605006a460 0x7f605006c920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6068036000 tx=0x7f6068008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:39.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.397+0000 7f6074c00640 1 -- 172.21.15.39:0/975361634 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6060054170 con 0x7f60780cb020 2024-01-24T17:42:39.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.607+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f60780c89f0 con 0x7f60780cb020 2024-01-24T17:42:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.608+0000 7f6074c00640 1 -- 172.21.15.39:0/975361634 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6060017080 con 0x7f60780cb020 2024-01-24T17:42:39.609 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f605006a460 msgr2=0x7f605006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f605006a460 0x7f605006c920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6068036000 tx=0x7f6068008030 comp rx=0 tx=0).stop 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60780cb020 msgr2=0x7f60780b3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60780cb020 0x7f60780b3680 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f606000e9b0 tx=0x7f606000ee70 comp rx=0 tx=0).stop 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 shutdown_connections 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60780cb400 0x7f60781f2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f605006a460 0x7f605006c920 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60780cb020 0x7f60780b3680 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 --2- 172.21.15.39:0/975361634 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60780c8c10 0x7f60780aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 >> 172.21.15.39:0/975361634 conn(0x7f607806a2a0 msgr2=0x7f6078159c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 shutdown_connections 2024-01-24T17:42:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:39.611+0000 7f607e200640 1 -- 172.21.15.39:0/975361634 wait complete. 2024-01-24T17:42:39.630 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:42:39.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:39.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-01-24T17:42:39.732 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:42:39.732 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:42:39.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:42:39.732 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-24T17:42:39.783 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-24T17:42:39.783 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:42:40.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.009+0000 7fc970200640 1 Processor -- start 2024-01-24T17:42:40.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.009+0000 7fc970200640 1 -- start start 2024-01-24T17:42:40.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.011+0000 7fc970200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9681654d0 0x7fc9680c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.011+0000 7fc970200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680cc630 con 0x7fc9680c90d0 2024-01-24T17:42:40.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.011+0000 7fc970200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680c9570 con 0x7fc9681658b0 2024-01-24T17:42:40.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.011+0000 7fc970200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680cc9d0 con 0x7fc9681654d0 2024-01-24T17:42:40.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.011+0000 7fc96ce00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc9680c90d0 0x7fc968070030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47216/0 (socket says 172.21.15.39:47216) 2024-01-24T17:42:40.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.011+0000 7fc96ce00640 1 -- 172.21.15.39:0/23958528 learned_addr learned my addr 172.21.15.39:0/23958528 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc96d800640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9681654d0 0x7fc9680c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048778690 0 0) 0x7fc9680cc630 con 0x7fc9680c90d0 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc948003520 con 0x7fc9680c90d0 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2473895004 0 0) 0x7fc948003520 con 0x7fc9680c90d0 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9681654d0 msgr2=0x7fc9680c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9681654d0 0x7fc9680c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 >> v1:172.21.15.100:6789/0 conn(0x7fc9681658b0 legacy=0x7fc9680caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680c9850 con 0x7fc9680c90d0 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc950001670 con 0x7fc9680c90d0 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 >> v1:172.21.15.39:6789/0 conn(0x7fc9680c90d0 legacy=0x7fc968070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9480039a0 0x7fc948003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.012+0000 7fc967e00640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9681654d0 0x7fc9480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc967e00640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc948005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680cc9d0 con 0x7fc948004d00 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94800bb10 con 0x7fc9480039a0 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94800bcd0 con 0x7fc9681654d0 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc96e200640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc948005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc96d800640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9480039a0 0x7fc948003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc96ce00640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9681654d0 0x7fc9480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc96e200640 1 -- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9681654d0 msgr2=0x7fc9480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc96e200640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9681654d0 0x7fc9480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc96e200640 1 -- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9480039a0 msgr2=0x7fc948003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc96e200640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9480039a0 0x7fc948003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.013+0000 7fc96e200640 1 -- 172.21.15.39:0/23958528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948003520 con 0x7fc948004d00 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.014+0000 7fc96e200640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc948005180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc960001f60 tx=0x7fc9480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990bf2565a78b909 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96002d6f0 con 0x7fc948004d00 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96002d870 con 0x7fc948004d00 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc967e00640 1 -- 172.21.15.39:0/23958528 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96002dc20 con 0x7fc948004d00 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc970200640 1 -- 172.21.15.39:0/23958528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 msgr2=0x7fc948005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc970200640 1 --2- 172.21.15.39:0/23958528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc948005180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc960001f60 tx=0x7fc9480057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc96e200640 1 -- 172.21.15.39:0/23958528 reap_dead start 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc970200640 1 -- 172.21.15.39:0/23958528 shutdown_connections 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc970200640 1 -- 172.21.15.39:0/23958528 >> 172.21.15.39:0/23958528 conn(0x7fc96806a3e0 msgr2=0x7fc96806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.015+0000 7fc970200640 1 -- 172.21.15.39:0/23958528 shutdown_connections 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.016+0000 7fc970200640 1 -- 172.21.15.39:0/23958528 wait complete. 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.017+0000 7fc970200640 1 Processor -- start 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.017+0000 7fc970200640 1 -- start start 2024-01-24T17:42:40.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.018+0000 7fc970200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc9681f9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.018+0000 7fc970200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9681f7580 0x7fc968162930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.018+0000 7fc970200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9680c7da0 0x7fc9680c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.018+0000 7fc970200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681fae00 con 0x7fc948004d00 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.018+0000 7fc970200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681fa840 con 0x7fc9681f7580 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.018+0000 7fc970200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9681fab20 con 0x7fc9680c7da0 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc9681f9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc9681f9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44750/0 (socket says 172.21.15.39:44750) 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 -- 172.21.15.39:0/3774305778 learned_addr learned my addr 172.21.15.39:0/3774305778 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96e200640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9680c7da0 0x7fc9680c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96ce00640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9681f7580 0x7fc968162930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 -- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9680c7da0 msgr2=0x7fc9680c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9680c7da0 0x7fc9680c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 -- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9681f7580 msgr2=0x7fc968162930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9681f7580 0x7fc968162930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 -- 172.21.15.39:0/3774305778 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680c75a0 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.019+0000 7fc96d800640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc9681f9ef0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc958011600 tx=0x7fc958011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.020+0000 7fc965600640 1 -- 172.21.15.39:0/3774305778 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800f920 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.020+0000 7fc965600640 1 -- 172.21.15.39:0/3774305778 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95800fa80 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.020+0000 7fc965600640 1 -- 172.21.15.39:0/3774305778 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800fe10 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.020+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9680c7880 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.020+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9681fa610 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.021+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc968073ef0 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.022+0000 7fc965600640 1 -- 172.21.15.39:0/3774305778 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc958015460 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.023+0000 7fc965600640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc94006a430 0x7fc94006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.023+0000 7fc965600640 1 -- 172.21.15.39:0/3774305778 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc9580891d0 con 0x7fc948004d00 2024-01-24T17:42:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.024+0000 7fc96ce00640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc94006a430 0x7fc94006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.025+0000 7fc96ce00640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc94006a430 0x7fc94006c8f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc9681f7980 tx=0x7fc950009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.027+0000 7fc965600640 1 -- 172.21.15.39:0/3774305778 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc958054790 con 0x7fc948004d00 2024-01-24T17:42:40.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.258+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9680b0b40 con 0x7fc948004d00 2024-01-24T17:42:40.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.258+0000 7fc965600640 1 -- 172.21.15.39:0/3774305778 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7fc958057db0 con 0x7fc948004d00 2024-01-24T17:42:40.259 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:40.259 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:41:33.354808+0000","modified":"2024-01-24T17:41:36.370204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-24T17:42:40.260 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-01-24T17:42:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc94006a430 msgr2=0x7fc94006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc94006a430 0x7fc94006c8f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc9681f7980 tx=0x7fc950009040 comp rx=0 tx=0).stop 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 msgr2=0x7fc9681f9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc9681f9ef0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc958011600 tx=0x7fc958011ac0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 shutdown_connections 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9680c7da0 0x7fc9680c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9681f7580 0x7fc968162930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc94006a430 0x7fc94006c8f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 --2- 172.21.15.39:0/3774305778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc948004d00 0x7fc9681f9ef0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.261+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 >> 172.21.15.39:0/3774305778 conn(0x7fc96806a3e0 msgr2=0x7fc96815fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.262+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 shutdown_connections 2024-01-24T17:42:40.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.262+0000 7fc970200640 1 -- 172.21.15.39:0/3774305778 wait complete. 2024-01-24T17:42:40.279 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:40.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.483+0000 7ff776200640 1 Processor -- start 2024-01-24T17:42:40.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.483+0000 7ff776200640 1 -- start start 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff776200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff77015a950 0x7ff7700ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff776200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700caaa0 0x7ff7700caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff776200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7700c9180 con 0x7ff7700cb490 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff776200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7700b43d0 con 0x7ff77015a950 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff776200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7700c8f70 con 0x7ff7700caaa0 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff77015a950 0x7ff7700ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff77015a950 0x7ff7700ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44758/0 (socket says 172.21.15.39:44758) 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76f400640 1 -- 172.21.15.39:0/2584790794 learned_addr learned my addr 172.21.15.39:0/2584790794 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76ea00640 1 --2- 172.21.15.39:0/2584790794 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700caaa0 0x7ff7700caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76f400640 1 -- 172.21.15.39:0/2584790794 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700caaa0 msgr2=0x7ff7700caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76f400640 1 --2- 172.21.15.39:0/2584790794 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700caaa0 0x7ff7700caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76f400640 1 -- 172.21.15.39:0/2584790794 >> v1:172.21.15.39:6789/0 conn(0x7ff7700cb490 legacy=0x7ff7700affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76f400640 1 -- 172.21.15.39:0/2584790794 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7700c9460 con 0x7ff77015a950 2024-01-24T17:42:40.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.485+0000 7ff76ea00640 1 --2- 172.21.15.39:0/2584790794 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700caaa0 0x7ff7700caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:40.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.487+0000 7ff76f400640 1 --2- 172.21.15.39:0/2584790794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff77015a950 0x7ff7700ca560 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff760009870 tx=0x7ff76002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2bebd8126411d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.487+0000 7ff76e000640 1 -- 172.21.15.39:0/2584790794 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760030840 con 0x7ff77015a950 2024-01-24T17:42:40.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.487+0000 7ff76e000640 1 -- 172.21.15.39:0/2584790794 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7600309a0 con 0x7ff77015a950 2024-01-24T17:42:40.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.487+0000 7ff76e000640 1 -- 172.21.15.39:0/2584790794 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760030d30 con 0x7ff77015a950 2024-01-24T17:42:40.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.488+0000 7ff776200640 1 -- 172.21.15.39:0/2584790794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff77015a950 msgr2=0x7ff7700ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.488+0000 7ff776200640 1 --2- 172.21.15.39:0/2584790794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff77015a950 0x7ff7700ca560 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff760009870 tx=0x7ff76002b980 comp rx=0 tx=0).stop 2024-01-24T17:42:40.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.488+0000 7ff776200640 1 -- 172.21.15.39:0/2584790794 shutdown_connections 2024-01-24T17:42:40.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.488+0000 7ff776200640 1 --2- 172.21.15.39:0/2584790794 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700caaa0 0x7ff7700caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.488+0000 7ff776200640 1 --2- 172.21.15.39:0/2584790794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff77015a950 0x7ff7700ca560 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.488+0000 7ff776200640 1 -- 172.21.15.39:0/2584790794 >> 172.21.15.39:0/2584790794 conn(0x7ff77006a3e0 msgr2=0x7ff77006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:40.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.488+0000 7ff776200640 1 -- 172.21.15.39:0/2584790794 shutdown_connections 2024-01-24T17:42:40.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.488+0000 7ff776200640 1 -- 172.21.15.39:0/2584790794 wait complete. 2024-01-24T17:42:40.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.490+0000 7ff776200640 1 Processor -- start 2024-01-24T17:42:40.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.490+0000 7ff776200640 1 -- start start 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff776200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7700caaa0 0x7ff77014cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff776200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700cb490 0x7ff7700c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff776200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff77015a950 0x7ff7700b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff776200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7700cd100 con 0x7ff7700caaa0 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff776200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7700cc4f0 con 0x7ff77015a950 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff776200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7700cc7d0 con 0x7ff7700cb490 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff76f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7700caaa0 0x7ff77014cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff76f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7700caaa0 0x7ff77014cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44760/0 (socket says 172.21.15.39:44760) 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff76f400640 1 -- 172.21.15.39:0/3473427257 learned_addr learned my addr 172.21.15.39:0/3473427257 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff76ea00640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700cb490 0x7ff7700c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff76f400640 1 -- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700cb490 msgr2=0x7ff7700c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff76f400640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700cb490 0x7ff7700c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff76fe00640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff77015a950 0x7ff7700b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.491+0000 7ff76f400640 1 -- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff77015a950 msgr2=0x7ff7700b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff76f400640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff77015a950 0x7ff7700b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff76f400640 1 -- 172.21.15.39:0/3473427257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7600094d0 con 0x7ff7700caaa0 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff76ea00640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700cb490 0x7ff7700c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff76fe00640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff77015a950 0x7ff7700b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff76f400640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7700caaa0 0x7ff77014cac0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff760000c00 tx=0x7ff760004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff767400640 1 -- 172.21.15.39:0/3473427257 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760030810 con 0x7ff7700caaa0 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff767400640 1 -- 172.21.15.39:0/3473427257 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff760039d90 con 0x7ff7700caaa0 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff767400640 1 -- 172.21.15.39:0/3473427257 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760043670 con 0x7ff7700caaa0 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7700ccab0 con 0x7ff7700caaa0 2024-01-24T17:42:40.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.492+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7700cd590 con 0x7ff7700caaa0 2024-01-24T17:42:40.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.494+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff770073ef0 con 0x7ff7700caaa0 2024-01-24T17:42:40.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.494+0000 7ff767400640 1 -- 172.21.15.39:0/3473427257 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff76004c3b0 con 0x7ff7700caaa0 2024-01-24T17:42:40.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.495+0000 7ff767400640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff74806a480 0x7ff74806c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.495+0000 7ff767400640 1 -- 172.21.15.39:0/3473427257 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff7600ab030 con 0x7ff7700caaa0 2024-01-24T17:42:40.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.496+0000 7ff76ea00640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff74806a480 0x7ff74806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.497+0000 7ff76ea00640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff74806a480 0x7ff74806c940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff758006fd0 tx=0x7ff758008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.500+0000 7ff767400640 1 -- 172.21.15.39:0/3473427257 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff7600766e0 con 0x7ff7700caaa0 2024-01-24T17:42:40.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.693+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7700b1950 con 0x7ff7700caaa0 2024-01-24T17:42:40.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.696+0000 7ff767400640 1 -- 172.21.15.39:0/3473427257 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7ff760079d00 con 0x7ff7700caaa0 2024-01-24T17:42:40.697 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:40.697 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff74806a480 msgr2=0x7ff74806c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff74806a480 0x7ff74806c940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff758006fd0 tx=0x7ff758008040 comp rx=0 tx=0).stop 2024-01-24T17:42:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7700caaa0 msgr2=0x7ff77014cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7700caaa0 0x7ff77014cac0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff760000c00 tx=0x7ff760004730 comp rx=0 tx=0).stop 2024-01-24T17:42:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 shutdown_connections 2024-01-24T17:42:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff77015a950 0x7ff7700b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7700cb490 0x7ff7700c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff74806a480 0x7ff74806c940 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 --2- 172.21.15.39:0/3473427257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7700caaa0 0x7ff77014cac0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.699+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 >> 172.21.15.39:0/3473427257 conn(0x7ff77006a3e0 msgr2=0x7ff770159c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:40.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.700+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 shutdown_connections 2024-01-24T17:42:40.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.700+0000 7ff776200640 1 -- 172.21.15.39:0/3473427257 wait complete. 2024-01-24T17:42:40.719 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:41:31.818248+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:41:32.355179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:40.719 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:40.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.918+0000 7fc7b7000640 1 Processor -- start 2024-01-24T17:42:40.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.918+0000 7fc7b7000640 1 -- start start 2024-01-24T17:42:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.920+0000 7fc7b7000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc7b015e190 0x7fc7b01656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.920+0000 7fc7b7000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b006d4c0 con 0x7fc7b00b0350 2024-01-24T17:42:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.920+0000 7fc7b7000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b006cf00 con 0x7fc7b00b0730 2024-01-24T17:42:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.920+0000 7fc7b7000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b006d1e0 con 0x7fc7b015e190 2024-01-24T17:42:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.920+0000 7fc7afe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc7b00b0350 0x7fc7b006c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47222/0 (socket says 172.21.15.39:47222) 2024-01-24T17:42:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.920+0000 7fc7afe00640 1 -- 172.21.15.39:0/2535282796 learned_addr learned my addr 172.21.15.39:0/2535282796 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.920+0000 7fc7b5000640 1 --2- 172.21.15.39:0/2535282796 >> v2:172.21.15.39:3300/0 conn(0x7fc7b015e190 0x7fc7b01656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.920+0000 7fc7aea00640 1 -- 172.21.15.39:0/2535282796 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297379148 0 0) 0x7fc7b006d4c0 con 0x7fc7b00b0350 2024-01-24T17:42:40.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.921+0000 7fc7aea00640 1 -- 172.21.15.39:0/2535282796 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc790003520 con 0x7fc7b00b0350 2024-01-24T17:42:40.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.921+0000 7fc7b5000640 1 -- 172.21.15.39:0/2535282796 >> v1:172.21.15.100:6789/0 conn(0x7fc7b00b0730 legacy=0x7fc7b015da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.921+0000 7fc7b5000640 1 -- 172.21.15.39:0/2535282796 >> v1:172.21.15.39:6789/0 conn(0x7fc7b00b0350 legacy=0x7fc7b006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.921+0000 7fc7b5000640 1 -- 172.21.15.39:0/2535282796 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b00b15e0 con 0x7fc7b015e190 2024-01-24T17:42:40.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.922+0000 7fc7b5000640 1 --2- 172.21.15.39:0/2535282796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b015e190 0x7fc7b01656a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009d90 tx=0x7fc7a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b179788c75cf8159 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7aea00640 1 -- 172.21.15.39:0/2535282796 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a802cc60 con 0x7fc7b015e190 2024-01-24T17:42:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7aea00640 1 -- 172.21.15.39:0/2535282796 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a802cdc0 con 0x7fc7b015e190 2024-01-24T17:42:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7aea00640 1 -- 172.21.15.39:0/2535282796 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8034640 con 0x7fc7b015e190 2024-01-24T17:42:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7b7000640 1 -- 172.21.15.39:0/2535282796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b015e190 msgr2=0x7fc7b01656a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7b7000640 1 --2- 172.21.15.39:0/2535282796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b015e190 0x7fc7b01656a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009d90 tx=0x7fc7a802beb0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7b7000640 1 -- 172.21.15.39:0/2535282796 shutdown_connections 2024-01-24T17:42:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7b7000640 1 --2- 172.21.15.39:0/2535282796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b015e190 0x7fc7b01656a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7b7000640 1 -- 172.21.15.39:0/2535282796 >> 172.21.15.39:0/2535282796 conn(0x7fc7b00cb0e0 msgr2=0x7fc7b00cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7b7000640 1 -- 172.21.15.39:0/2535282796 shutdown_connections 2024-01-24T17:42:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.923+0000 7fc7b7000640 1 -- 172.21.15.39:0/2535282796 wait complete. 2024-01-24T17:42:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.925+0000 7fc7b7000640 1 Processor -- start 2024-01-24T17:42:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.925+0000 7fc7b7000640 1 -- start start 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.926+0000 7fc7b7000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7b00b0350 0x7fc7b01568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.926+0000 7fc7b7000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b00b0730 0x7fc7b01f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.926+0000 7fc7b7000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7b015e190 0x7fc7b00c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.926+0000 7fc7b7000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0070280 con 0x7fc7b00b0730 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.926+0000 7fc7b7000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0070560 con 0x7fc7b00b0350 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.926+0000 7fc7b7000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0070840 con 0x7fc7b015e190 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.926+0000 7fc7af400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b00b0730 0x7fc7b01f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7af400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b00b0730 0x7fc7b01f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44788/0 (socket says 172.21.15.39:44788) 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7af400640 1 -- 172.21.15.39:0/3517293551 learned_addr learned my addr 172.21.15.39:0/3517293551 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7afe00640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7b00b0350 0x7fc7b01568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7b5000640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7b015e190 0x7fc7b00c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7af400640 1 -- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7b015e190 msgr2=0x7fc7b00c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7af400640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7b015e190 0x7fc7b00c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7af400640 1 -- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7b00b0350 msgr2=0x7fc7b01568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7af400640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7b00b0350 0x7fc7b01568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7af400640 1 -- 172.21.15.39:0/3517293551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a8009a70 con 0x7fc7b00b0730 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7b5000640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7b015e190 0x7fc7b00c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7af400640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b00b0730 0x7fc7b01f1af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc79800c450 tx=0x7fc79800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7a7e00640 1 -- 172.21.15.39:0/3517293551 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7980052d0 con 0x7fc7b00b0730 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7a7e00640 1 -- 172.21.15.39:0/3517293551 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc798005430 con 0x7fc7b00b0730 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7a7e00640 1 -- 172.21.15.39:0/3517293551 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798012870 con 0x7fc7b00b0730 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.927+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b00ab960 con 0x7fc7b00b0730 2024-01-24T17:42:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.928+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b0070b20 con 0x7fc7b00b0730 2024-01-24T17:42:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.929+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b0073ef0 con 0x7fc7b00b0730 2024-01-24T17:42:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.930+0000 7fc7a7e00640 1 -- 172.21.15.39:0/3517293551 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc7980129d0 con 0x7fc7b00b0730 2024-01-24T17:42:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.931+0000 7fc7a7e00640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc78806a430 0x7fc78806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.931+0000 7fc7a7e00640 1 -- 172.21.15.39:0/3517293551 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc7980884d0 con 0x7fc7b00b0730 2024-01-24T17:42:40.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.931+0000 7fc7afe00640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc78806a430 0x7fc78806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:40.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.932+0000 7fc7afe00640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc78806a430 0x7fc78806c8f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00011e0 tx=0x7fc7a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:40.935+0000 7fc7a7e00640 1 -- 172.21.15.39:0/3517293551 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc798053a00 con 0x7fc7b00b0730 2024-01-24T17:42:41.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.118+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7b00b0f80 con 0x7fc7b00b0730 2024-01-24T17:42:41.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.119+0000 7fc7a7e00640 1 -- 172.21.15.39:0/3517293551 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc798057020 con 0x7fc7b00b0730 2024-01-24T17:42:41.120 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:41.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc78806a430 msgr2=0x7fc78806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc78806a430 0x7fc78806c8f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00011e0 tx=0x7fc7a0009040 comp rx=0 tx=0).stop 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b00b0730 msgr2=0x7fc7b01f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b00b0730 0x7fc7b01f1af0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc79800c450 tx=0x7fc79800c910 comp rx=0 tx=0).stop 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 shutdown_connections 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7b015e190 0x7fc7b00c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc78806a430 0x7fc78806c8f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7b00b0730 0x7fc7b01f1af0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 --2- 172.21.15.39:0/3517293551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7b00b0350 0x7fc7b01568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 >> 172.21.15.39:0/3517293551 conn(0x7fc7b00cb0e0 msgr2=0x7fc7b0152bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 shutdown_connections 2024-01-24T17:42:41.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.122+0000 7fc7b7000640 1 -- 172.21.15.39:0/3517293551 wait complete. 2024-01-24T17:42:41.142 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:42:41.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.359+0000 7f6697200640 1 Processor -- start 2024-01-24T17:42:41.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.359+0000 7f6697200640 1 -- start start 2024-01-24T17:42:41.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.360+0000 7f6697200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900cb540 0x7f669006d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.360+0000 7f6697200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f669006c3a0 0x7f66900ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.360+0000 7f6697200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66900ad0b0 con 0x7f66900accd0 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.360+0000 7f6697200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669006d840 con 0x7f669006c3a0 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.360+0000 7f6697200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66900b2370 con 0x7f66900cb540 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.361+0000 7f6696800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f669006c3a0 0x7f66900ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.361+0000 7f6696800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f669006c3a0 0x7f66900ab030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44790/0 (socket says 172.21.15.39:44790) 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.361+0000 7f6696800640 1 -- 172.21.15.39:0/2822599089 learned_addr learned my addr 172.21.15.39:0/2822599089 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.361+0000 7f6695400640 1 --2- 172.21.15.39:0/2822599089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900cb540 0x7f669006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.361+0000 7f6696800640 1 -- 172.21.15.39:0/2822599089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900cb540 msgr2=0x7f669006d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.361+0000 7f6696800640 1 --2- 172.21.15.39:0/2822599089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900cb540 0x7f669006d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.361+0000 7f6696800640 1 -- 172.21.15.39:0/2822599089 >> v1:172.21.15.100:6789/0 conn(0x7f66900accd0 legacy=0x7f66900cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:41.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.361+0000 7f6696800640 1 -- 172.21.15.39:0/2822599089 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66900b2650 con 0x7f669006c3a0 2024-01-24T17:42:41.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.362+0000 7f6696800640 1 --2- 172.21.15.39:0/2822599089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f669006c3a0 0x7f66900ab030 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6688009e50 tx=0x7f668802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f332ee675cbbac04 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:41.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6694a00640 1 -- 172.21.15.39:0/2822599089 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668802ca70 con 0x7f669006c3a0 2024-01-24T17:42:41.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6694a00640 1 -- 172.21.15.39:0/2822599089 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668802cbd0 con 0x7f669006c3a0 2024-01-24T17:42:41.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6694a00640 1 -- 172.21.15.39:0/2822599089 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66880343f0 con 0x7f669006c3a0 2024-01-24T17:42:41.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6697200640 1 -- 172.21.15.39:0/2822599089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f669006c3a0 msgr2=0x7f66900ab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6697200640 1 --2- 172.21.15.39:0/2822599089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f669006c3a0 0x7f66900ab030 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6688009e50 tx=0x7f668802bed0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6697200640 1 -- 172.21.15.39:0/2822599089 shutdown_connections 2024-01-24T17:42:41.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6697200640 1 --2- 172.21.15.39:0/2822599089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f669006c3a0 0x7f66900ab030 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6697200640 1 --2- 172.21.15.39:0/2822599089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900cb540 0x7f669006d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6697200640 1 -- 172.21.15.39:0/2822599089 >> 172.21.15.39:0/2822599089 conn(0x7f6690069e50 msgr2=0x7f669006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:41.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6697200640 1 -- 172.21.15.39:0/2822599089 shutdown_connections 2024-01-24T17:42:41.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.363+0000 7f6697200640 1 -- 172.21.15.39:0/2822599089 wait complete. 2024-01-24T17:42:41.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.365+0000 7f6697200640 1 Processor -- start 2024-01-24T17:42:41.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.365+0000 7f6697200640 1 -- start start 2024-01-24T17:42:41.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6697200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f669006c3a0 0x7f66900c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6697200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900accd0 0x7f669015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6697200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66900cb540 0x7f66900b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6697200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669015cac0 con 0x7f66900cb540 2024-01-24T17:42:41.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6697200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669015bd10 con 0x7f669006c3a0 2024-01-24T17:42:41.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6697200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669015bf40 con 0x7f66900accd0 2024-01-24T17:42:41.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6696800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66900cb540 0x7f66900b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6696800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66900cb540 0x7f66900b02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44792/0 (socket says 172.21.15.39:44792) 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.366+0000 7f6696800640 1 -- 172.21.15.39:0/1583065762 learned_addr learned my addr 172.21.15.39:0/1583065762 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6695400640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900accd0 0x7f669015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6695e00640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f669006c3a0 0x7f66900c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6696800640 1 -- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900accd0 msgr2=0x7f669015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6696800640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900accd0 0x7f669015b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6696800640 1 -- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f669006c3a0 msgr2=0x7f66900c3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6696800640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f669006c3a0 0x7f66900c3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6696800640 1 -- 172.21.15.39:0/1583065762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6688009b30 con 0x7f66900cb540 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6695400640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900accd0 0x7f669015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f6696800640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66900cb540 0x7f66900b02c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6688009e50 tx=0x7f6688004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.367+0000 7f668e000640 1 -- 172.21.15.39:0/1583065762 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688009510 con 0x7f66900cb540 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.368+0000 7f668e000640 1 -- 172.21.15.39:0/1583065762 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668803e450 con 0x7f66900cb540 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.368+0000 7f668e000640 1 -- 172.21.15.39:0/1583065762 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668803e800 con 0x7f66900cb540 2024-01-24T17:42:41.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.368+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669015c220 con 0x7f66900cb540 2024-01-24T17:42:41.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.368+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669015c7e0 con 0x7f66900cb540 2024-01-24T17:42:41.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.369+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669004dff0 con 0x7f66900cb540 2024-01-24T17:42:41.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.370+0000 7f668e000640 1 -- 172.21.15.39:0/1583065762 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f668803e960 con 0x7f66900cb540 2024-01-24T17:42:41.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.371+0000 7f668e000640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666806a430 0x7f666806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.371+0000 7f668e000640 1 -- 172.21.15.39:0/1583065762 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f66880ab850 con 0x7f66900cb540 2024-01-24T17:42:41.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.371+0000 7f6695e00640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666806a430 0x7f666806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.372+0000 7f6695e00640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666806a430 0x7f666806c8f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6680003e00 tx=0x7f6680009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:41.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.375+0000 7f668e000640 1 -- 172.21.15.39:0/1583065762 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6688076e30 con 0x7f66900cb540 2024-01-24T17:42:41.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.560+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f66900cca30 con 0x7f66900cb540 2024-01-24T17:42:41.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.561+0000 7f668e000640 1 -- 172.21.15.39:0/1583065762 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f668807a450 con 0x7f66900cb540 2024-01-24T17:42:41.561 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:42:41.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666806a430 msgr2=0x7f666806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666806a430 0x7f666806c8f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6680003e00 tx=0x7f6680009040 comp rx=0 tx=0).stop 2024-01-24T17:42:41.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66900cb540 msgr2=0x7f66900b02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66900cb540 0x7f66900b02c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6688009e50 tx=0x7f6688004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:41.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 shutdown_connections 2024-01-24T17:42:41.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666806a430 0x7f666806c8f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66900cb540 0x7f66900b02c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66900accd0 0x7f669015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 --2- 172.21.15.39:0/1583065762 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f669006c3a0 0x7f66900c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 >> 172.21.15.39:0/1583065762 conn(0x7f6690069e50 msgr2=0x7f669006a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:41.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 shutdown_connections 2024-01-24T17:42:41.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.563+0000 7f6697200640 1 -- 172.21.15.39:0/1583065762 wait complete. 2024-01-24T17:42:41.582 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:42:41.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.800+0000 7f4c56a00640 1 Processor -- start 2024-01-24T17:42:41.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.800+0000 7f4c56a00640 1 -- start start 2024-01-24T17:42:41.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.802+0000 7f4c56a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4c5015ccb0 0x7f4c500cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.802+0000 7f4c56a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5015d090 con 0x7f4c500ce390 2024-01-24T17:42:41.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.802+0000 7f4c56a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c500cd140 con 0x7f4c500c9d60 2024-01-24T17:42:41.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.802+0000 7f4c56a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c500cac20 con 0x7f4c5015ccb0 2024-01-24T17:42:41.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.802+0000 7f4c55600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4c5015ccb0 0x7f4c500cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.802+0000 7f4c54c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4c500ce390 0x7f4c500cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47230/0 (socket says 172.21.15.39:47230) 2024-01-24T17:42:41.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.802+0000 7f4c54c00640 1 -- 172.21.15.39:0/1215356678 learned_addr learned my addr 172.21.15.39:0/1215356678 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:41.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.803+0000 7f4c55600640 1 -- 172.21.15.39:0/1215356678 >> v1:172.21.15.100:6789/0 conn(0x7f4c500c9d60 legacy=0x7f4c500cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:41.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.803+0000 7f4c55600640 1 -- 172.21.15.39:0/1215356678 >> v1:172.21.15.39:6789/0 conn(0x7f4c500ce390 legacy=0x7f4c500cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.803+0000 7f4c55600640 1 -- 172.21.15.39:0/1215356678 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5006c540 con 0x7f4c5015ccb0 2024-01-24T17:42:41.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.803+0000 7f4c4fe00640 1 -- 172.21.15.39:0/1215356678 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544866849 0 0) 0x7f4c5015d090 con 0x7f4c500ce390 2024-01-24T17:42:41.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.804+0000 7f4c55600640 1 --2- 172.21.15.39:0/1215356678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5015ccb0 0x7f4c500cda90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009790 tx=0x7f4c4002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80dbc284341215e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:41.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.804+0000 7f4c4fe00640 1 -- 172.21.15.39:0/1215356678 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4002d040 con 0x7f4c5015ccb0 2024-01-24T17:42:41.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.804+0000 7f4c4fe00640 1 -- 172.21.15.39:0/1215356678 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c40031810 con 0x7f4c5015ccb0 2024-01-24T17:42:41.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.804+0000 7f4c4fe00640 1 -- 172.21.15.39:0/1215356678 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40031ba0 con 0x7f4c5015ccb0 2024-01-24T17:42:41.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.805+0000 7f4c56a00640 1 -- 172.21.15.39:0/1215356678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5015ccb0 msgr2=0x7f4c500cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.805+0000 7f4c56a00640 1 --2- 172.21.15.39:0/1215356678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5015ccb0 0x7f4c500cda90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009790 tx=0x7f4c4002b8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.805+0000 7f4c56a00640 1 -- 172.21.15.39:0/1215356678 shutdown_connections 2024-01-24T17:42:41.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.805+0000 7f4c56a00640 1 --2- 172.21.15.39:0/1215356678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c5015ccb0 0x7f4c500cda90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.805+0000 7f4c56a00640 1 -- 172.21.15.39:0/1215356678 >> 172.21.15.39:0/1215356678 conn(0x7f4c50069e50 msgr2=0x7f4c5006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:41.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.805+0000 7f4c56a00640 1 -- 172.21.15.39:0/1215356678 shutdown_connections 2024-01-24T17:42:41.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.805+0000 7f4c56a00640 1 -- 172.21.15.39:0/1215356678 wait complete. 2024-01-24T17:42:41.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.806+0000 7f4c56a00640 1 Processor -- start 2024-01-24T17:42:41.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.807+0000 7f4c56a00640 1 -- start start 2024-01-24T17:42:41.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.807+0000 7f4c56a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c500c9d60 0x7f4c50159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.807+0000 7f4c56a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c500ce390 0x7f4c500707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.807+0000 7f4c56a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c5015ccb0 0x7f4c501f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.807+0000 7f4c56a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c50070e40 con 0x7f4c500ce390 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.807+0000 7f4c56a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c50071120 con 0x7f4c5015ccb0 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.807+0000 7f4c56a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c50071400 con 0x7f4c500c9d60 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c54c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c500ce390 0x7f4c500707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c54c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c500ce390 0x7f4c500707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44804/0 (socket says 172.21.15.39:44804) 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c54c00640 1 -- 172.21.15.39:0/3180034444 learned_addr learned my addr 172.21.15.39:0/3180034444 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c55600640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c500c9d60 0x7f4c50159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c56000640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c5015ccb0 0x7f4c501f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c54c00640 1 -- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c500c9d60 msgr2=0x7f4c50159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c54c00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c500c9d60 0x7f4c50159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c54c00640 1 -- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c5015ccb0 msgr2=0x7f4c501f1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c54c00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c5015ccb0 0x7f4c501f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.808+0000 7f4c54c00640 1 -- 172.21.15.39:0/3180034444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400093a0 con 0x7f4c500ce390 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.809+0000 7f4c54c00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c500ce390 0x7f4c500707d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800c4d0 tx=0x7f4c3800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:41.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.809+0000 7f4c4d600640 1 -- 172.21.15.39:0/3180034444 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c380074e0 con 0x7f4c500ce390 2024-01-24T17:42:41.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.809+0000 7f4c4d600640 1 -- 172.21.15.39:0/3180034444 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c38007660 con 0x7f4c500ce390 2024-01-24T17:42:41.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.809+0000 7f4c4d600640 1 -- 172.21.15.39:0/3180034444 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c38004a20 con 0x7f4c500ce390 2024-01-24T17:42:41.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.809+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c500716e0 con 0x7f4c500ce390 2024-01-24T17:42:41.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.809+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c50159ac0 con 0x7f4c500ce390 2024-01-24T17:42:41.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.810+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5006cdc0 con 0x7f4c500ce390 2024-01-24T17:42:41.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.811+0000 7f4c4d600640 1 -- 172.21.15.39:0/3180034444 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4c38004b80 con 0x7f4c500ce390 2024-01-24T17:42:41.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.812+0000 7f4c4d600640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c2806a430 0x7f4c2806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:41.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.812+0000 7f4c4d600640 1 -- 172.21.15.39:0/3180034444 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4c38087b20 con 0x7f4c500ce390 2024-01-24T17:42:41.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.813+0000 7f4c55600640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c2806a430 0x7f4c2806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:41.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.814+0000 7f4c55600640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c2806a430 0x7f4c2806c8f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c40036000 tx=0x7f4c4003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:41.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:41.817+0000 7f4c4d600640 1 -- 172.21.15.39:0/3180034444 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4c5006cdc0 con 0x7f4c500ce390 2024-01-24T17:42:42.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.003+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4c500c6130 con 0x7f4c500ce390 2024-01-24T17:42:42.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.004+0000 7f4c4d600640 1 -- 172.21.15.39:0/3180034444 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4c500c6130 con 0x7f4c500ce390 2024-01-24T17:42:42.004 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c2806a430 msgr2=0x7f4c2806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c2806a430 0x7f4c2806c8f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c40036000 tx=0x7f4c4003c040 comp rx=0 tx=0).stop 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c500ce390 msgr2=0x7f4c500707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c500ce390 0x7f4c500707d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800c4d0 tx=0x7f4c3800c990 comp rx=0 tx=0).stop 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 shutdown_connections 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c5015ccb0 0x7f4c501f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c2806a430 0x7f4c2806c8f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c500ce390 0x7f4c500707d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 --2- 172.21.15.39:0/3180034444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c500c9d60 0x7f4c50159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 >> 172.21.15.39:0/3180034444 conn(0x7f4c50069e50 msgr2=0x7f4c500ade80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:42.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 shutdown_connections 2024-01-24T17:42:42.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.006+0000 7f4c56a00640 1 -- 172.21.15.39:0/3180034444 wait complete. 2024-01-24T17:42:42.025 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:42:42.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.231+0000 7f213f200640 1 Processor -- start 2024-01-24T17:42:42.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.232+0000 7f213f200640 1 -- start start 2024-01-24T17:42:42.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.233+0000 7f213f200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f213806c7b0 0x7f21380cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.233+0000 7f213f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21381615b0 con 0x7f213815c9b0 2024-01-24T17:42:42.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.233+0000 7f213f200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138160ff0 con 0x7f213815cd90 2024-01-24T17:42:42.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.233+0000 7f213f200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2138160e00 con 0x7f213806c7b0 2024-01-24T17:42:42.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.233+0000 7f213de00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f213815c9b0 0x7f21381606f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47246/0 (socket says 172.21.15.39:47246) 2024-01-24T17:42:42.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.233+0000 7f213de00640 1 -- 172.21.15.39:0/789368315 learned_addr learned my addr 172.21.15.39:0/789368315 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213e800640 1 --2- 172.21.15.39:0/789368315 >> v2:172.21.15.100:3300/0 conn(0x7f213806c7b0 0x7f21380cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197153233 0 0) 0x7f21381615b0 con 0x7f213815c9b0 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2118003520 con 0x7f213815c9b0 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2277387109 0 0) 0x7f2118003520 con 0x7f213815c9b0 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 >> v2:172.21.15.100:3300/0 conn(0x7f213806c7b0 msgr2=0x7f21380cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 --2- 172.21.15.39:0/789368315 >> v2:172.21.15.100:3300/0 conn(0x7f213806c7b0 0x7f21380cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 >> v1:172.21.15.100:6789/0 conn(0x7f213815cd90 legacy=0x7f213806b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21381612d0 con 0x7f213815c9b0 2024-01-24T17:42:42.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2128001670 con 0x7f213815c9b0 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 >> v1:172.21.15.39:6789/0 conn(0x7f213815c9b0 legacy=0x7f21381606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2118003970 0x7f2118003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21180047c0 0x7f2118004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21180052a0 0x7f211800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211800c140 con 0x7f21180052a0 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211800c2b0 con 0x7f21180047c0 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.234+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211800c470 con 0x7f2118003970 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213e800640 1 --2- 172.21.15.39:0/789368315 >> v2:172.21.15.100:3300/0 conn(0x7f213806c7b0 0x7f21380cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213e800640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21180052a0 0x7f211800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213de00640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2118003970 0x7f2118003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213e800640 1 -- 172.21.15.39:0/789368315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2118003970 msgr2=0x7f2118003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213d400640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21180047c0 0x7f2118004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213e800640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2118003970 0x7f2118003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213e800640 1 -- 172.21.15.39:0/789368315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21180047c0 msgr2=0x7f2118004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:42.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213e800640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21180047c0 0x7f2118004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.235+0000 7f213e800640 1 -- 172.21.15.39:0/789368315 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2118003520 con 0x7f21180052a0 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.236+0000 7f213e800640 1 -- 172.21.15.39:0/789368315 reap_dead start 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.236+0000 7f213e800640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21180052a0 0x7f211800b980 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f21300073b0 tx=0x7f21180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6cf8141874c36 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213002cd90 con 0x7f21180052a0 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2130034440 con 0x7f21180052a0 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213ca00640 1 -- 172.21.15.39:0/789368315 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21300347f0 con 0x7f21180052a0 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213f200640 1 -- 172.21.15.39:0/789368315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21180052a0 msgr2=0x7f211800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213f200640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21180052a0 0x7f211800b980 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f21300073b0 tx=0x7f21180057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213f200640 1 -- 172.21.15.39:0/789368315 shutdown_connections 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213f200640 1 --2- 172.21.15.39:0/789368315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21180052a0 0x7f211800b980 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213f200640 1 -- 172.21.15.39:0/789368315 >> 172.21.15.39:0/789368315 conn(0x7f2138069c70 msgr2=0x7f213806a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:42.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213f200640 1 -- 172.21.15.39:0/789368315 shutdown_connections 2024-01-24T17:42:42.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.237+0000 7f213f200640 1 -- 172.21.15.39:0/789368315 wait complete. 2024-01-24T17:42:42.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.239+0000 7f213f200640 1 Processor -- start 2024-01-24T17:42:42.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.239+0000 7f213f200640 1 -- start start 2024-01-24T17:42:42.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21180052a0 0x7f21381f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21380ab8c0 0x7f213806b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21380ad4c0 0x7f2138155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138157720 con 0x7f21380ab8c0 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213f200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138154f90 con 0x7f21380ad4c0 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2138155250 con 0x7f21180052a0 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21380ab8c0 0x7f213806b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21380ab8c0 0x7f213806b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44816/0 (socket says 172.21.15.39:44816) 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.240+0000 7f213d400640 1 -- 172.21.15.39:0/875423797 learned_addr learned my addr 172.21.15.39:0/875423797 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f213d400640 1 -- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21180052a0 msgr2=0x7f21381f6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f213d400640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21180052a0 0x7f21381f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f213d400640 1 -- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21380ad4c0 msgr2=0x7f2138155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f213d400640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21380ad4c0 0x7f2138155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f213d400640 1 -- 172.21.15.39:0/875423797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138155530 con 0x7f21380ab8c0 2024-01-24T17:42:42.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f213d400640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21380ab8c0 0x7f213806b8f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f212000d480 tx=0x7f212000d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:42.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f2136000640 1 -- 172.21.15.39:0/875423797 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2120005480 con 0x7f21380ab8c0 2024-01-24T17:42:42.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f2136000640 1 -- 172.21.15.39:0/875423797 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2120003610 con 0x7f21380ab8c0 2024-01-24T17:42:42.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f2136000640 1 -- 172.21.15.39:0/875423797 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21200039e0 con 0x7f21380ab8c0 2024-01-24T17:42:42.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.241+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2138155810 con 0x7f21380ab8c0 2024-01-24T17:42:42.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.242+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213806be30 con 0x7f21380ab8c0 2024-01-24T17:42:42.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.243+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213804dff0 con 0x7f21380ab8c0 2024-01-24T17:42:42.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.244+0000 7f2136000640 1 -- 172.21.15.39:0/875423797 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2120003b40 con 0x7f21380ab8c0 2024-01-24T17:42:42.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.244+0000 7f2136000640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211006a430 0x7f211006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.245+0000 7f2136000640 1 -- 172.21.15.39:0/875423797 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f21200888a0 con 0x7f21380ab8c0 2024-01-24T17:42:42.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.245+0000 7f213de00640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211006a430 0x7f211006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.246+0000 7f213de00640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211006a430 0x7f211006c8f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f21280017a0 tx=0x7f2128009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:42.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.249+0000 7f2136000640 1 -- 172.21.15.39:0/875423797 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2120053e90 con 0x7f21380ab8c0 2024-01-24T17:42:42.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.484+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f213806bff0 con 0x7f21380ab8c0 2024-01-24T17:42:42.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.493+0000 7f2136000640 1 -- 172.21.15.39:0/875423797 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7f21200574b0 con 0x7f21380ab8c0 2024-01-24T17:42:42.493 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:42.494 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:41:33.354808+0000","modified":"2024-01-24T17:41:36.370204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-24T17:42:42.494 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-01-24T17:42:42.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.495+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211006a430 msgr2=0x7f211006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.495+0000 7f213f200640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211006a430 0x7f211006c8f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f21280017a0 tx=0x7f2128009040 comp rx=0 tx=0).stop 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.495+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21380ab8c0 msgr2=0x7f213806b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.495+0000 7f213f200640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21380ab8c0 0x7f213806b8f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f212000d480 tx=0x7f212000d940 comp rx=0 tx=0).stop 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.496+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 shutdown_connections 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.496+0000 7f213f200640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21380ad4c0 0x7f2138155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.496+0000 7f213f200640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f211006a430 0x7f211006c8f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.496+0000 7f213f200640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21380ab8c0 0x7f213806b8f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.496+0000 7f213f200640 1 --2- 172.21.15.39:0/875423797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21180052a0 0x7f21381f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.496+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 >> 172.21.15.39:0/875423797 conn(0x7f2138069c70 msgr2=0x7f213815fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.496+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 shutdown_connections 2024-01-24T17:42:42.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.496+0000 7f213f200640 1 -- 172.21.15.39:0/875423797 wait complete. 2024-01-24T17:42:42.515 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:42.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.738+0000 7fc254c00640 1 Processor -- start 2024-01-24T17:42:42.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.738+0000 7fc254c00640 1 -- start start 2024-01-24T17:42:42.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.740+0000 7fc254c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc25015cdf0 0x7fc2500cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.740+0000 7fc254c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 0x7fc2500d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.740+0000 7fc254c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2500cf5a0 con 0x7fc2500d0d30 2024-01-24T17:42:42.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.740+0000 7fc254c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2500cf3c0 con 0x7fc25015cdf0 2024-01-24T17:42:42.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.740+0000 7fc254c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2500d11b0 con 0x7fc2500cef60 2024-01-24T17:42:42.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.740+0000 7fc24de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc25015cdf0 0x7fc2500cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.740+0000 7fc24de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc25015cdf0 0x7fc2500cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44826/0 (socket says 172.21.15.39:44826) 2024-01-24T17:42:42.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.740+0000 7fc24de00640 1 -- 172.21.15.39:0/1583827900 learned_addr learned my addr 172.21.15.39:0/1583827900 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:42.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.741+0000 7fc24d400640 1 --2- 172.21.15.39:0/1583827900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 0x7fc2500d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.741+0000 7fc24de00640 1 -- 172.21.15.39:0/1583827900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 msgr2=0x7fc2500d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:42.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.741+0000 7fc24de00640 1 --2- 172.21.15.39:0/1583827900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 0x7fc2500d07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.741+0000 7fc24de00640 1 -- 172.21.15.39:0/1583827900 >> v1:172.21.15.100:6789/0 conn(0x7fc2500d0d30 legacy=0x7fc2500cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:42.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.741+0000 7fc24de00640 1 -- 172.21.15.39:0/1583827900 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2500cf880 con 0x7fc25015cdf0 2024-01-24T17:42:42.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.742+0000 7fc24de00640 1 --2- 172.21.15.39:0/1583827900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc2500cea20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc240009e50 tx=0x7fc24002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31895f5d1d82ddfe server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:42.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.743+0000 7fc24ca00640 1 -- 172.21.15.39:0/1583827900 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24002cb00 con 0x7fc25015cdf0 2024-01-24T17:42:42.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.743+0000 7fc24ca00640 1 -- 172.21.15.39:0/1583827900 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24002cc60 con 0x7fc25015cdf0 2024-01-24T17:42:42.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.743+0000 7fc24ca00640 1 -- 172.21.15.39:0/1583827900 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2400343f0 con 0x7fc25015cdf0 2024-01-24T17:42:42.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.743+0000 7fc254c00640 1 -- 172.21.15.39:0/1583827900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 msgr2=0x7fc2500cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.743+0000 7fc254c00640 1 --2- 172.21.15.39:0/1583827900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc2500cea20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc240009e50 tx=0x7fc24002bf60 comp rx=0 tx=0).stop 2024-01-24T17:42:42.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.744+0000 7fc254c00640 1 -- 172.21.15.39:0/1583827900 shutdown_connections 2024-01-24T17:42:42.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.744+0000 7fc254c00640 1 --2- 172.21.15.39:0/1583827900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 0x7fc2500d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.744+0000 7fc254c00640 1 --2- 172.21.15.39:0/1583827900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc2500cea20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.744+0000 7fc254c00640 1 -- 172.21.15.39:0/1583827900 >> 172.21.15.39:0/1583827900 conn(0x7fc250099b60 msgr2=0x7fc250089fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:42.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.744+0000 7fc254c00640 1 -- 172.21.15.39:0/1583827900 shutdown_connections 2024-01-24T17:42:42.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.744+0000 7fc254c00640 1 -- 172.21.15.39:0/1583827900 wait complete. 2024-01-24T17:42:42.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.745+0000 7fc254c00640 1 Processor -- start 2024-01-24T17:42:42.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.745+0000 7fc254c00640 1 -- start start 2024-01-24T17:42:42.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.746+0000 7fc254c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 0x7fc2500b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.746+0000 7fc254c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2500d0d30 0x7fc2500b58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.746+0000 7fc254c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc25015d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.746+0000 7fc254c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc250154b00 con 0x7fc25015cdf0 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.746+0000 7fc254c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc250150660 con 0x7fc2500d0d30 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.746+0000 7fc254c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2501582f0 con 0x7fc2500cef60 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc25015d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc25015d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44828/0 (socket says 172.21.15.39:44828) 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24e800640 1 -- 172.21.15.39:0/220135029 learned_addr learned my addr 172.21.15.39:0/220135029 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24de00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 0x7fc2500b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24d400640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2500d0d30 0x7fc2500b58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24e800640 1 -- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 msgr2=0x7fc2500b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24e800640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 0x7fc2500b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24e800640 1 -- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2500d0d30 msgr2=0x7fc2500b58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24e800640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2500d0d30 0x7fc2500b58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.747+0000 7fc24e800640 1 -- 172.21.15.39:0/220135029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc240009b30 con 0x7fc25015cdf0 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.748+0000 7fc24e800640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc25015d1d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc24800c3c0 tx=0x7fc24800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.748+0000 7fc246000640 1 -- 172.21.15.39:0/220135029 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2480052d0 con 0x7fc25015cdf0 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.748+0000 7fc246000640 1 -- 172.21.15.39:0/220135029 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc248005430 con 0x7fc25015cdf0 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.748+0000 7fc246000640 1 -- 172.21.15.39:0/220135029 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc248003880 con 0x7fc25015cdf0 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.748+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc250157770 con 0x7fc25015cdf0 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.748+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc250157d30 con 0x7fc25015cdf0 2024-01-24T17:42:42.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.749+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc250154c70 con 0x7fc25015cdf0 2024-01-24T17:42:42.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.750+0000 7fc246000640 1 -- 172.21.15.39:0/220135029 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc2480039e0 con 0x7fc25015cdf0 2024-01-24T17:42:42.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.751+0000 7fc246000640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22806a460 0x7fc22806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:42.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.751+0000 7fc246000640 1 -- 172.21.15.39:0/220135029 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc248087ea0 con 0x7fc25015cdf0 2024-01-24T17:42:42.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.752+0000 7fc24de00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22806a460 0x7fc22806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:42.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.753+0000 7fc24de00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22806a460 0x7fc22806c920 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc240035000 tx=0x7fc240030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:42.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.755+0000 7fc246000640 1 -- 172.21.15.39:0/220135029 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc248053470 con 0x7fc25015cdf0 2024-01-24T17:42:42.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.950+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc250073ef0 con 0x7fc25015cdf0 2024-01-24T17:42:42.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.951+0000 7fc246000640 1 -- 172.21.15.39:0/220135029 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7fc248056a90 con 0x7fc25015cdf0 2024-01-24T17:42:42.952 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:42.953 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22806a460 msgr2=0x7fc22806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22806a460 0x7fc22806c920 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc240035000 tx=0x7fc240030000 comp rx=0 tx=0).stop 2024-01-24T17:42:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 msgr2=0x7fc25015d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc25015d1d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc24800c3c0 tx=0x7fc24800c880 comp rx=0 tx=0).stop 2024-01-24T17:42:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 shutdown_connections 2024-01-24T17:42:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22806a460 0x7fc22806c920 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc25015cdf0 0x7fc25015d1d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2500d0d30 0x7fc2500b58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 --2- 172.21.15.39:0/220135029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2500cef60 0x7fc2500b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:42.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 >> 172.21.15.39:0/220135029 conn(0x7fc250099b60 msgr2=0x7fc25015adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:42.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 shutdown_connections 2024-01-24T17:42:42.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:42.954+0000 7fc254c00640 1 -- 172.21.15.39:0/220135029 wait complete. 2024-01-24T17:42:42.973 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-24T17:42:42.973 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:42:43.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.193+0000 7fca88400640 1 Processor -- start 2024-01-24T17:42:43.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.193+0000 7fca88400640 1 -- start start 2024-01-24T17:42:43.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.195+0000 7fca88400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fca8015ca30 0x7fca800cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.195+0000 7fca88400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fca800cd3c0 0x7fca800c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.195+0000 7fca88400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca800c9c10 con 0x7fca800c90d0 2024-01-24T17:42:43.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.195+0000 7fca88400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca800c9650 con 0x7fca800cd3c0 2024-01-24T17:42:43.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.195+0000 7fca88400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca800c9930 con 0x7fca8015ca30 2024-01-24T17:42:43.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.195+0000 7fca85000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fca800cd3c0 0x7fca800c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.195+0000 7fca85000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fca800cd3c0 0x7fca800c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40454/0 (socket says 172.21.15.39:40454) 2024-01-24T17:42:43.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.195+0000 7fca85000640 1 -- 172.21.15.39:0/2624505260 learned_addr learned my addr 172.21.15.39:0/2624505260 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:43.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.196+0000 7fca85a00640 1 --2- 172.21.15.39:0/2624505260 >> v2:172.21.15.100:3300/0 conn(0x7fca8015ca30 0x7fca800cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.196+0000 7fca85000640 1 -- 172.21.15.39:0/2624505260 >> v2:172.21.15.100:3300/0 conn(0x7fca8015ca30 msgr2=0x7fca800cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:43.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.196+0000 7fca85000640 1 --2- 172.21.15.39:0/2624505260 >> v2:172.21.15.100:3300/0 conn(0x7fca8015ca30 0x7fca800cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.196+0000 7fca85000640 1 -- 172.21.15.39:0/2624505260 >> v1:172.21.15.100:6789/0 conn(0x7fca800c90d0 legacy=0x7fca800caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:43.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.196+0000 7fca85000640 1 -- 172.21.15.39:0/2624505260 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca800cc680 con 0x7fca800cd3c0 2024-01-24T17:42:43.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.197+0000 7fca85000640 1 --2- 172.21.15.39:0/2624505260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca800cd3c0 0x7fca800c8b90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fca800cc2a0 tx=0x7fca6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6849fab000d606d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:43.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.197+0000 7fca7fe00640 1 -- 172.21.15.39:0/2624505260 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68031870 con 0x7fca800cd3c0 2024-01-24T17:42:43.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.197+0000 7fca7fe00640 1 -- 172.21.15.39:0/2624505260 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca680319d0 con 0x7fca800cd3c0 2024-01-24T17:42:43.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.197+0000 7fca7fe00640 1 -- 172.21.15.39:0/2624505260 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68031d60 con 0x7fca800cd3c0 2024-01-24T17:42:43.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.198+0000 7fca88400640 1 -- 172.21.15.39:0/2624505260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca800cd3c0 msgr2=0x7fca800c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.198+0000 7fca88400640 1 --2- 172.21.15.39:0/2624505260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca800cd3c0 0x7fca800c8b90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fca800cc2a0 tx=0x7fca6802b910 comp rx=0 tx=0).stop 2024-01-24T17:42:43.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.198+0000 7fca88400640 1 -- 172.21.15.39:0/2624505260 shutdown_connections 2024-01-24T17:42:43.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.198+0000 7fca88400640 1 --2- 172.21.15.39:0/2624505260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca800cd3c0 0x7fca800c8b90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.198+0000 7fca88400640 1 --2- 172.21.15.39:0/2624505260 >> v2:172.21.15.100:3300/0 conn(0x7fca8015ca30 0x7fca800cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.198+0000 7fca88400640 1 -- 172.21.15.39:0/2624505260 >> 172.21.15.39:0/2624505260 conn(0x7fca8006a3e0 msgr2=0x7fca8006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:43.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.198+0000 7fca88400640 1 -- 172.21.15.39:0/2624505260 shutdown_connections 2024-01-24T17:42:43.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.198+0000 7fca88400640 1 -- 172.21.15.39:0/2624505260 wait complete. 2024-01-24T17:42:43.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.200+0000 7fca88400640 1 Processor -- start 2024-01-24T17:42:43.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.200+0000 7fca88400640 1 -- start start 2024-01-24T17:42:43.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca88400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca800c90d0 0x7fca8015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca88400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca8015ca30 0x7fca800c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca88400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca80156e80 0x7fca8015ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca88400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca80157f30 con 0x7fca8015ca30 2024-01-24T17:42:43.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca88400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8015d600 con 0x7fca80156e80 2024-01-24T17:42:43.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca88400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca800b02c0 con 0x7fca800c90d0 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca85000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca8015ca30 0x7fca800c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca85000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca8015ca30 0x7fca800c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40456/0 (socket says 172.21.15.39:40456) 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.201+0000 7fca85000640 1 -- 172.21.15.39:0/2782327484 learned_addr learned my addr 172.21.15.39:0/2782327484 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca85a00640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca800c90d0 0x7fca8015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca86400640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca80156e80 0x7fca8015ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca85000640 1 -- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca800c90d0 msgr2=0x7fca8015d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca85000640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca800c90d0 0x7fca8015d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca85000640 1 -- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca80156e80 msgr2=0x7fca8015ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca85000640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca80156e80 0x7fca8015ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca85000640 1 -- 172.21.15.39:0/2782327484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca680094d0 con 0x7fca8015ca30 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca85a00640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca800c90d0 0x7fca8015d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca86400640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca80156e80 0x7fca8015ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:43.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca85000640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca8015ca30 0x7fca800c8120 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fca68000c00 tx=0x7fca680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:43.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.202+0000 7fca7d600640 1 -- 172.21.15.39:0/2782327484 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6803abe0 con 0x7fca8015ca30 2024-01-24T17:42:43.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.203+0000 7fca7d600640 1 -- 172.21.15.39:0/2782327484 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6803ad40 con 0x7fca8015ca30 2024-01-24T17:42:43.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.203+0000 7fca7d600640 1 -- 172.21.15.39:0/2782327484 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68044610 con 0x7fca8015ca30 2024-01-24T17:42:43.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.203+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca800b05a0 con 0x7fca8015ca30 2024-01-24T17:42:43.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.203+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca80157310 con 0x7fca8015ca30 2024-01-24T17:42:43.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.204+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca80073ef0 con 0x7fca8015ca30 2024-01-24T17:42:43.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.205+0000 7fca7d600640 1 -- 172.21.15.39:0/2782327484 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fca6804d450 con 0x7fca8015ca30 2024-01-24T17:42:43.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.206+0000 7fca7d600640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca5806a430 0x7fca5806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.206+0000 7fca7d600640 1 -- 172.21.15.39:0/2782327484 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fca6802d040 con 0x7fca8015ca30 2024-01-24T17:42:43.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.206+0000 7fca85a00640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca5806a430 0x7fca5806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.207+0000 7fca85a00640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca5806a430 0x7fca5806c8f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca70006fd0 tx=0x7fca70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:43.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.210+0000 7fca7d600640 1 -- 172.21.15.39:0/2782327484 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fca680ae030 con 0x7fca8015ca30 2024-01-24T17:42:43.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.443+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca800b44c0 con 0x7fca8015ca30 2024-01-24T17:42:43.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.444+0000 7fca7d600640 1 -- 172.21.15.39:0/2782327484 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7fca68076570 con 0x7fca8015ca30 2024-01-24T17:42:43.444 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:43.444 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:41:33.354808+0000","modified":"2024-01-24T17:41:36.370204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-24T17:42:43.445 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-01-24T17:42:43.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca5806a430 msgr2=0x7fca5806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca5806a430 0x7fca5806c8f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca70006fd0 tx=0x7fca70008040 comp rx=0 tx=0).stop 2024-01-24T17:42:43.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca8015ca30 msgr2=0x7fca800c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca8015ca30 0x7fca800c8120 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fca68000c00 tx=0x7fca680046c0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 shutdown_connections 2024-01-24T17:42:43.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca80156e80 0x7fca8015ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca5806a430 0x7fca5806c8f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca8015ca30 0x7fca800c8120 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 --2- 172.21.15.39:0/2782327484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca800c90d0 0x7fca8015d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.446+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 >> 172.21.15.39:0/2782327484 conn(0x7fca8006a3e0 msgr2=0x7fca800caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:43.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.447+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 shutdown_connections 2024-01-24T17:42:43.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.447+0000 7fca88400640 1 -- 172.21.15.39:0/2782327484 wait complete. 2024-01-24T17:42:43.466 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:42:43.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.686+0000 7f29e5800640 1 Processor -- start 2024-01-24T17:42:43.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.687+0000 7f29e5800640 1 -- start start 2024-01-24T17:42:43.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.688+0000 7f29e5800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f29e00cf990 0x7f29e00cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.688+0000 7f29e5800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e015d020 0x7f29e015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.688+0000 7f29e5800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e00d02b0 con 0x7f29e0160610 2024-01-24T17:42:43.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.688+0000 7f29e5800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e015e080 con 0x7f29e00cf990 2024-01-24T17:42:43.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.688+0000 7f29e5800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e015dac0 con 0x7f29e015d020 2024-01-24T17:42:43.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.688+0000 7f29df400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f29e0160610 0x7f29e0160aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:46910/0 (socket says 172.21.15.39:46910) 2024-01-24T17:42:43.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.688+0000 7f29df400640 1 -- 172.21.15.39:0/1243399020 learned_addr learned my addr 172.21.15.39:0/1243399020 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:43.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e015d020 0x7f29e015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29dea00640 1 --2- 172.21.15.39:0/1243399020 >> v2:172.21.15.100:3300/0 conn(0x7f29e00cf990 0x7f29e00cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863107155 0 0) 0x7f29e00d02b0 con 0x7f29e0160610 2024-01-24T17:42:43.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c0003520 con 0x7f29e0160610 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 529073860 0 0) 0x7f29c0003520 con 0x7f29e0160610 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e015d020 msgr2=0x7f29e015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e015d020 0x7f29e015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 >> v2:172.21.15.100:3300/0 conn(0x7f29e00cf990 msgr2=0x7f29e00cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 --2- 172.21.15.39:0/1243399020 >> v2:172.21.15.100:3300/0 conn(0x7f29e00cf990 0x7f29e00cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e015dda0 con 0x7f29e0160610 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f29d80019f0 con 0x7f29e0160610 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 >> v1:172.21.15.39:6789/0 conn(0x7f29e0160610 legacy=0x7f29e0160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29c0003970 0x7f29c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29c00047c0 0x7f29c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c0005180 0x7f29c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e015e080 con 0x7f29c00047c0 2024-01-24T17:42:43.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e015dac0 con 0x7f29c0003970 2024-01-24T17:42:43.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c000bef0 con 0x7f29c0005180 2024-01-24T17:42:43.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.689+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e015d020 0x7f29e015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:43.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29c00047c0 0x7f29c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29dea00640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29c0003970 0x7f29c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29df400640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c0005180 0x7f29c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29d7e00640 1 -- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c0005180 msgr2=0x7f29c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:43.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c0005180 0x7f29c000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29d7e00640 1 -- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29c0003970 msgr2=0x7f29c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:43.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29c0003970 0x7f29c0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29d7e00640 1 -- 172.21.15.39:0/1243399020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c0003520 con 0x7f29c00047c0 2024-01-24T17:42:43.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.690+0000 7f29dea00640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29c0003970 0x7f29c0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.691+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29c00047c0 0x7f29c0004c40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f29c8006fd0 tx=0x7f29c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bd5764d877be59 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8034680 con 0x7f29c00047c0 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c80347e0 con 0x7f29c00047c0 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29de000640 1 -- 172.21.15.39:0/1243399020 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8034b70 con 0x7f29c00047c0 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29e5800640 1 -- 172.21.15.39:0/1243399020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29c00047c0 msgr2=0x7f29c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29e5800640 1 --2- 172.21.15.39:0/1243399020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29c00047c0 0x7f29c0004c40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f29c8006fd0 tx=0x7f29c00057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29df400640 1 -- 172.21.15.39:0/1243399020 reap_dead start 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29e5800640 1 -- 172.21.15.39:0/1243399020 shutdown_connections 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29e5800640 1 -- 172.21.15.39:0/1243399020 >> 172.21.15.39:0/1243399020 conn(0x7f29e0071790 msgr2=0x7f29e0071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29e5800640 1 -- 172.21.15.39:0/1243399020 shutdown_connections 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.692+0000 7f29e5800640 1 -- 172.21.15.39:0/1243399020 wait complete. 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.694+0000 7f29e5800640 1 Processor -- start 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.694+0000 7f29e5800640 1 -- start start 2024-01-24T17:42:43.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29e5800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c00047c0 0x7f29e0148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29e5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e00d06b0 0x7f29e00b7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29e5800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e015aa30 0x7f29e015aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29e5800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e015b400 con 0x7f29e00d06b0 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29e5800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00b79d0 con 0x7f29e015aa30 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29e5800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00c6fb0 con 0x7f29c00047c0 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29d7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e00d06b0 0x7f29e00b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29d7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e00d06b0 0x7f29e00b7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40472/0 (socket says 172.21.15.39:40472) 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.695+0000 7f29d7e00640 1 -- 172.21.15.39:0/1696530836 learned_addr learned my addr 172.21.15.39:0/1696530836 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29dea00640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c00047c0 0x7f29e0148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29df400640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e015aa30 0x7f29e015aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29d7e00640 1 -- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c00047c0 msgr2=0x7f29e0148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c00047c0 0x7f29e0148f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29d7e00640 1 -- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e015aa30 msgr2=0x7f29e015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e015aa30 0x7f29e015aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29d7e00640 1 -- 172.21.15.39:0/1696530836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e00c7290 con 0x7f29e00d06b0 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29df400640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e015aa30 0x7f29e015aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.696+0000 7f29d7e00640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e00d06b0 0x7f29e00b7490 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f29c802bd30 tx=0x7f29c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.697+0000 7f29d6a00640 1 -- 172.21.15.39:0/1696530836 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c803d810 con 0x7f29e00d06b0 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.697+0000 7f29d6a00640 1 -- 172.21.15.39:0/1696530836 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c803d970 con 0x7f29e00d06b0 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.697+0000 7f29d6a00640 1 -- 172.21.15.39:0/1696530836 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c803dd00 con 0x7f29e00d06b0 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.697+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e00c7570 con 0x7f29e00d06b0 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.697+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e00c7b30 con 0x7f29e00d06b0 2024-01-24T17:42:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.698+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e015b400 con 0x7f29e00d06b0 2024-01-24T17:42:43.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.699+0000 7f29d6a00640 1 -- 172.21.15.39:0/1696530836 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f29c803b040 con 0x7f29e00d06b0 2024-01-24T17:42:43.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.700+0000 7f29d6a00640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29b806a430 0x7f29b806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:43.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.700+0000 7f29d6a00640 1 -- 172.21.15.39:0/1696530836 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f29c80a9ef0 con 0x7f29e00d06b0 2024-01-24T17:42:43.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.701+0000 7f29dea00640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29b806a430 0x7f29b806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:43.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.702+0000 7f29dea00640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29b806a430 0x7f29b806c8f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f29e0072070 tx=0x7f29d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:43.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.705+0000 7f29d6a00640 1 -- 172.21.15.39:0/1696530836 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f29c8075560 con 0x7f29e00d06b0 2024-01-24T17:42:43.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.945+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29e00c7cc0 con 0x7f29e00d06b0 2024-01-24T17:42:43.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.947+0000 7f29d6a00640 1 -- 172.21.15.39:0/1696530836 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4734 (secure 0 0 0) 0x7f29c803de60 con 0x7f29e00d06b0 2024-01-24T17:42:43.947 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:43.947 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3369914597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3369914597},{"type":"v1","addr":"172.21.15.39:6835","nonce":3369914597}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2668043277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":2668043277},{"type":"v1","addr":"172.21.15.100:6837","nonce":2668043277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/1900123081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1900123081},{"type":"v1","addr":"172.21.15.39:6834","nonce":1900123081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:41:33.354808+0000","modified":"2024-01-24T17:41:36.370204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6835/3015590938","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3015590938},{"type":"v1","addr":"172.21.15.100:6835","nonce":3015590938}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-24T17:42:43.948 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-01-24T17:42:43.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.949+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29b806a430 msgr2=0x7f29b806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.949+0000 7f29e5800640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29b806a430 0x7f29b806c8f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f29e0072070 tx=0x7f29d0008040 comp rx=0 tx=0).stop 2024-01-24T17:42:43.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.949+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e00d06b0 msgr2=0x7f29e00b7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:43.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.949+0000 7f29e5800640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e00d06b0 0x7f29e00b7490 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f29c802bd30 tx=0x7f29c8004730 comp rx=0 tx=0).stop 2024-01-24T17:42:43.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.950+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 shutdown_connections 2024-01-24T17:42:43.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.950+0000 7f29e5800640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e015aa30 0x7f29e015aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.950+0000 7f29e5800640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29b806a430 0x7f29b806c8f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.950+0000 7f29e5800640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e00d06b0 0x7f29e00b7490 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.950+0000 7f29e5800640 1 --2- 172.21.15.39:0/1696530836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29c00047c0 0x7f29e0148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:43.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.950+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 >> 172.21.15.39:0/1696530836 conn(0x7f29e0071790 msgr2=0x7f29e015b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:43.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.950+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 shutdown_connections 2024-01-24T17:42:43.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:43.950+0000 7f29e5800640 1 -- 172.21.15.39:0/1696530836 wait complete. 2024-01-24T17:42:43.970 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:44.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.195+0000 7f1f3e000640 1 Processor -- start 2024-01-24T17:42:44.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.196+0000 7f1f3e000640 1 -- start start 2024-01-24T17:42:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.198+0000 7f1f3e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380c3d80 0x7f1f3806ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.198+0000 7f1f3e000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380cc010 con 0x7f1f380b0ea0 2024-01-24T17:42:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.198+0000 7f1f3e000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380cb700 con 0x7f1f380b1b80 2024-01-24T17:42:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.198+0000 7f1f3e000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3806b360 con 0x7f1f380c3d80 2024-01-24T17:42:44.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.198+0000 7f1f3cc00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1f380b0ea0 0x7f1f380cc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:46918/0 (socket says 172.21.15.39:46918) 2024-01-24T17:42:44.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.198+0000 7f1f3cc00640 1 -- 172.21.15.39:0/1621851045 learned_addr learned my addr 172.21.15.39:0/1621851045 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:44.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.198+0000 7f1f3d600640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380c3d80 0x7f1f3806ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.198+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508993318 0 0) 0x7f1f380cc010 con 0x7f1f380b0ea0 2024-01-24T17:42:44.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f18003520 con 0x7f1f380b0ea0 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151557946 0 0) 0x7f1f18003520 con 0x7f1f380b0ea0 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380c3d80 msgr2=0x7f1f3806ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380c3d80 0x7f1f3806ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 >> v1:172.21.15.100:6789/0 conn(0x7f1f380b1b80 legacy=0x7f1f380c3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380cb9c0 con 0x7f1f380b0ea0 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1f28001670 con 0x7f1f380b0ea0 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 >> v1:172.21.15.39:6789/0 conn(0x7f1f380b0ea0 legacy=0x7f1f380cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f18003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f180047c0 0x7f1f18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380c3d80 0x7f1f18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3806b360 con 0x7f1f180039a0 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1800ba90 con 0x7f1f180047c0 2024-01-24T17:42:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1800bc50 con 0x7f1f380c3d80 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.199+0000 7f1f3cc00640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f18003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.200+0000 7f1f3d600640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380c3d80 0x7f1f18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.200+0000 7f1f37e00640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f180047c0 0x7f1f18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.200+0000 7f1f3cc00640 1 -- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380c3d80 msgr2=0x7f1f18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.200+0000 7f1f3cc00640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380c3d80 0x7f1f18005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.200+0000 7f1f3cc00640 1 -- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f180047c0 msgr2=0x7f1f18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.200+0000 7f1f3cc00640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f180047c0 0x7f1f18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.200+0000 7f1f3cc00640 1 -- 172.21.15.39:0/1621851045 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f18003520 con 0x7f1f180039a0 2024-01-24T17:42:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.200+0000 7f1f37e00640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f180047c0 0x7f1f18004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:44.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.201+0000 7f1f3cc00640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f18003dc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1f28001eb0 tx=0x7f1f180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ccb6af1ca9b510 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:44.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.202+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28005ad0 con 0x7f1f180039a0 2024-01-24T17:42:44.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.202+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f28005c30 con 0x7f1f180039a0 2024-01-24T17:42:44.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.202+0000 7f1f37400640 1 -- 172.21.15.39:0/1621851045 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2802c7c0 con 0x7f1f180039a0 2024-01-24T17:42:44.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.202+0000 7f1f3e000640 1 -- 172.21.15.39:0/1621851045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 msgr2=0x7f1f18003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.202+0000 7f1f3e000640 1 --2- 172.21.15.39:0/1621851045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f18003dc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1f28001eb0 tx=0x7f1f180057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.202+0000 7f1f3d600640 1 -- 172.21.15.39:0/1621851045 reap_dead start 2024-01-24T17:42:44.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.203+0000 7f1f3e000640 1 -- 172.21.15.39:0/1621851045 shutdown_connections 2024-01-24T17:42:44.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.203+0000 7f1f3e000640 1 -- 172.21.15.39:0/1621851045 >> 172.21.15.39:0/1621851045 conn(0x7f1f38099a00 msgr2=0x7f1f380d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:44.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.203+0000 7f1f3e000640 1 -- 172.21.15.39:0/1621851045 shutdown_connections 2024-01-24T17:42:44.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.203+0000 7f1f3e000640 1 -- 172.21.15.39:0/1621851045 wait complete. 2024-01-24T17:42:44.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.204+0000 7f1f3e000640 1 Processor -- start 2024-01-24T17:42:44.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.205+0000 7f1f3e000640 1 -- start start 2024-01-24T17:42:44.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f381f7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3e000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f380c87c0 0x7f1f380ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380aea50 0x7f1f38161360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3e000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f380aef00 con 0x7f1f180039a0 2024-01-24T17:42:44.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3e000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f380ca270 con 0x7f1f380c87c0 2024-01-24T17:42:44.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3e000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f381f7570 con 0x7f1f380aea50 2024-01-24T17:42:44.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f381f7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f381f7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40486/0 (socket says 172.21.15.39:40486) 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3cc00640 1 -- 172.21.15.39:0/2454059214 learned_addr learned my addr 172.21.15.39:0/2454059214 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f37e00640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f380c87c0 0x7f1f380ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3d600640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380aea50 0x7f1f38161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3cc00640 1 -- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380aea50 msgr2=0x7f1f38161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3cc00640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380aea50 0x7f1f38161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3cc00640 1 -- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f380c87c0 msgr2=0x7f1f380ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3cc00640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f380c87c0 0x7f1f380ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3cc00640 1 -- 172.21.15.39:0/2454059214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380c8c80 con 0x7f1f180039a0 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f37e00640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f380c87c0 0x7f1f380ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.206+0000 7f1f3d600640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380aea50 0x7f1f38161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:44.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.207+0000 7f1f3cc00640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f381f7030 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1f28001eb0 tx=0x7f1f28001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:44.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.207+0000 7f1f34c00640 1 -- 172.21.15.39:0/2454059214 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2803c040 con 0x7f1f180039a0 2024-01-24T17:42:44.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.207+0000 7f1f34c00640 1 -- 172.21.15.39:0/2454059214 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2802dc60 con 0x7f1f180039a0 2024-01-24T17:42:44.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.207+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f380c8f60 con 0x7f1f180039a0 2024-01-24T17:42:44.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.207+0000 7f1f34c00640 1 -- 172.21.15.39:0/2454059214 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28007070 con 0x7f1f180039a0 2024-01-24T17:42:44.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.207+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f380c9520 con 0x7f1f180039a0 2024-01-24T17:42:44.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.208+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f38067040 con 0x7f1f180039a0 2024-01-24T17:42:44.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.209+0000 7f1f34c00640 1 -- 172.21.15.39:0/2454059214 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1f28032070 con 0x7f1f180039a0 2024-01-24T17:42:44.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.210+0000 7f1f34c00640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f1006a430 0x7f1f1006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.210+0000 7f1f34c00640 1 -- 172.21.15.39:0/2454059214 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1f280aa280 con 0x7f1f180039a0 2024-01-24T17:42:44.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.211+0000 7f1f37e00640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f1006a430 0x7f1f1006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.212+0000 7f1f37e00640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f1006a430 0x7f1f1006c8f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1f3806f8f0 tx=0x7f1f20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:44.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.215+0000 7f1f34c00640 1 -- 172.21.15.39:0/2454059214 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1f280758e0 con 0x7f1f180039a0 2024-01-24T17:42:44.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.411+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f380c9d20 con 0x7f1f180039a0 2024-01-24T17:42:44.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.413+0000 7f1f34c00640 1 -- 172.21.15.39:0/2454059214 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16090 (secure 0 0 0) 0x7f1f28078f00 con 0x7f1f180039a0 2024-01-24T17:42:44.414 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:44.414 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:16.189929+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:41:31.818248+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T17:41:32.355179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3b","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f1006a430 msgr2=0x7f1f1006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f1006a430 0x7f1f1006c8f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1f3806f8f0 tx=0x7f1f20009040 comp rx=0 tx=0).stop 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 msgr2=0x7f1f381f7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f381f7030 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1f28001eb0 tx=0x7f1f28001930 comp rx=0 tx=0).stop 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 shutdown_connections 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f380aea50 0x7f1f38161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f380c87c0 0x7f1f380ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f1006a430 0x7f1f1006c8f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 --2- 172.21.15.39:0/2454059214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f180039a0 0x7f1f381f7030 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 >> 172.21.15.39:0/2454059214 conn(0x7f1f38099a00 msgr2=0x7f1f380cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:44.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 shutdown_connections 2024-01-24T17:42:44.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.416+0000 7f1f3e000640 1 -- 172.21.15.39:0/2454059214 wait complete. 2024-01-24T17:42:44.438 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-24T17:42:44.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.667+0000 7fa48dc00640 1 Processor -- start 2024-01-24T17:42:44.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.667+0000 7fa48dc00640 1 -- start start 2024-01-24T17:42:44.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.669+0000 7fa48dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48806bdb0 0x7fa48806c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.669+0000 7fa48dc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4880ccd90 con 0x7fa4880cd830 2024-01-24T17:42:44.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.669+0000 7fa48dc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48815e010 con 0x7fa488071cf0 2024-01-24T17:42:44.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.669+0000 7fa48dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48806c6d0 con 0x7fa48806bdb0 2024-01-24T17:42:44.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.669+0000 7fa47fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa4880cd830 0x7fa4880cbf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:46926/0 (socket says 172.21.15.39:46926) 2024-01-24T17:42:44.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.669+0000 7fa47fe00640 1 -- 172.21.15.39:0/1247750875 learned_addr learned my addr 172.21.15.39:0/1247750875 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:44.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487e00640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48806bdb0 0x7fa48806c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333086600 0 0) 0x7fa4880ccd90 con 0x7fa4880cd830 2024-01-24T17:42:44.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa468005730 con 0x7fa4880cd830 2024-01-24T17:42:44.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899833139 0 0) 0x7fa468005730 con 0x7fa4880cd830 2024-01-24T17:42:44.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48806bdb0 msgr2=0x7fa48806c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:44.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48806bdb0 0x7fa48806c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 >> v1:172.21.15.100:6789/0 conn(0x7fa488071cf0 legacy=0x7fa4880cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa488159d50 con 0x7fa4880cd830 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fa470001670 con 0x7fa4880cd830 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 >> v1:172.21.15.39:6789/0 conn(0x7fa4880cd830 legacy=0x7fa4880cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa468005bb0 0x7fa468005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4680069d0 0x7fa468006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48806bdb0 0x7fa468007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48806c6d0 con 0x7fa4680069d0 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46800dc40 con 0x7fa468005bb0 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.670+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46800de00 con 0x7fa48806bdb0 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa47fe00640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4680069d0 0x7fa468006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa48d200640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48806bdb0 0x7fa468007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa487e00640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa468005bb0 0x7fa468005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa47fe00640 1 -- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48806bdb0 msgr2=0x7fa468007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:44.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa47fe00640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48806bdb0 0x7fa468007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa47fe00640 1 -- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa468005bb0 msgr2=0x7fa468005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:44.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa47fe00640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa468005bb0 0x7fa468005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa47fe00640 1 -- 172.21.15.39:0/1247750875 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa468005730 con 0x7fa4680069d0 2024-01-24T17:42:44.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.671+0000 7fa487e00640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa468005bb0 0x7fa468005fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:44.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.672+0000 7fa47fe00640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4680069d0 0x7fa468006e50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa47000c210 tx=0x7fa468007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e85e419566d5cd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:44.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470005ad0 con 0x7fa4680069d0 2024-01-24T17:42:44.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa470005c50 con 0x7fa4680069d0 2024-01-24T17:42:44.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa487400640 1 -- 172.21.15.39:0/1247750875 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47002c870 con 0x7fa4680069d0 2024-01-24T17:42:44.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa48dc00640 1 -- 172.21.15.39:0/1247750875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4680069d0 msgr2=0x7fa468006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa48dc00640 1 --2- 172.21.15.39:0/1247750875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4680069d0 0x7fa468006e50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa47000c210 tx=0x7fa468007a00 comp rx=0 tx=0).stop 2024-01-24T17:42:44.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa48d200640 1 -- 172.21.15.39:0/1247750875 reap_dead start 2024-01-24T17:42:44.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa48dc00640 1 -- 172.21.15.39:0/1247750875 shutdown_connections 2024-01-24T17:42:44.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa48dc00640 1 -- 172.21.15.39:0/1247750875 >> 172.21.15.39:0/1247750875 conn(0x7fa48806a2a0 msgr2=0x7fa4880afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:44.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa48dc00640 1 -- 172.21.15.39:0/1247750875 shutdown_connections 2024-01-24T17:42:44.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.673+0000 7fa48dc00640 1 -- 172.21.15.39:0/1247750875 wait complete. 2024-01-24T17:42:44.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.675+0000 7fa48dc00640 1 Processor -- start 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.675+0000 7fa48dc00640 1 -- start start 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.676+0000 7fa48dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4680069d0 0x7fa4880c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.676+0000 7fa48dc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4881487c0 0x7fa48815c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.676+0000 7fa48dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa488148c40 0x7fa48815c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.676+0000 7fa48dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880c5470 con 0x7fa488148c40 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.676+0000 7fa48dc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880c5750 con 0x7fa4881487c0 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.676+0000 7fa48dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4880c5a30 con 0x7fa4680069d0 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa488148c40 0x7fa48815c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa488148c40 0x7fa48815c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40506/0 (socket says 172.21.15.39:40506) 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 -- 172.21.15.39:0/2363607363 learned_addr learned my addr 172.21.15.39:0/2363607363 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa487e00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4680069d0 0x7fa4880c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa47fe00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4881487c0 0x7fa48815c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 -- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4680069d0 msgr2=0x7fa4880c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4680069d0 0x7fa4880c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 -- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4881487c0 msgr2=0x7fa48815c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4881487c0 0x7fa48815c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 -- 172.21.15.39:0/2363607363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4880c5d10 con 0x7fa488148c40 2024-01-24T17:42:44.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa487e00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4680069d0 0x7fa4880c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.677+0000 7fa48d200640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa488148c40 0x7fa48815c6e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa48815d080 tx=0x7fa48000ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.678+0000 7fa484c00640 1 -- 172.21.15.39:0/2363607363 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480003b80 con 0x7fa488148c40 2024-01-24T17:42:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.678+0000 7fa484c00640 1 -- 172.21.15.39:0/2363607363 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480003ce0 con 0x7fa488148c40 2024-01-24T17:42:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.678+0000 7fa484c00640 1 -- 172.21.15.39:0/2363607363 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480005310 con 0x7fa488148c40 2024-01-24T17:42:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.678+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4880c5ff0 con 0x7fa488148c40 2024-01-24T17:42:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.678+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4880c7410 con 0x7fa488148c40 2024-01-24T17:42:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.679+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4880b0bd0 con 0x7fa488148c40 2024-01-24T17:42:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.680+0000 7fa484c00640 1 -- 172.21.15.39:0/2363607363 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa480005470 con 0x7fa488148c40 2024-01-24T17:42:44.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.681+0000 7fa484c00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa46006a430 0x7fa46006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:44.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.681+0000 7fa484c00640 1 -- 172.21.15.39:0/2363607363 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa480088760 con 0x7fa488148c40 2024-01-24T17:42:44.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.681+0000 7fa487e00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa46006a430 0x7fa46006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:44.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.682+0000 7fa487e00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa46006a430 0x7fa46006c8f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa478009b60 tx=0x7fa478008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:44.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.685+0000 7fa484c00640 1 -- 172.21.15.39:0/2363607363 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa480053c90 con 0x7fa488148c40 2024-01-24T17:42:44.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.923+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa4880b4320 con 0x7fa488148c40 2024-01-24T17:42:44.926 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:42:44.925+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:42:44.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.939+0000 7fa484c00640 1 -- 172.21.15.39:0/2363607363 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa4800572b0 con 0x7fa488148c40 2024-01-24T17:42:44.940 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-24T17:42:44.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa46006a430 msgr2=0x7fa46006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa46006a430 0x7fa46006c8f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa478009b60 tx=0x7fa478008040 comp rx=0 tx=0).stop 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa488148c40 msgr2=0x7fa48815c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa488148c40 0x7fa48815c6e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa48815d080 tx=0x7fa48000ea00 comp rx=0 tx=0).stop 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 shutdown_connections 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa46006a430 0x7fa46006c8f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa488148c40 0x7fa48815c6e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4881487c0 0x7fa48815c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 --2- 172.21.15.39:0/2363607363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4680069d0 0x7fa4880c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 >> 172.21.15.39:0/2363607363 conn(0x7fa48806a2a0 msgr2=0x7fa4880cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 shutdown_connections 2024-01-24T17:42:44.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:44.942+0000 7fa48dc00640 1 -- 172.21.15.39:0/2363607363 wait complete. 2024-01-24T17:42:44.961 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-24T17:42:44.965 INFO:tasks.ceph.mds.b.smithi100.stderr: -4089> 2024-01-24T17:41:34.743+0000 7ff3f2e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:42:45.059 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:42:45.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.184+0000 7f9fc8000640 1 Processor -- start 2024-01-24T17:42:45.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.184+0000 7f9fc8000640 1 -- start start 2024-01-24T17:42:45.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.185+0000 7f9fc8000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc015fff0 0x7f9fc00b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:45.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.185+0000 7f9fc8000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9fc00d14c0 0x7f9fc00cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:45.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.185+0000 7f9fc8000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc00cda90 con 0x7f9fc00b2a10 2024-01-24T17:42:45.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc8000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc00b2e90 con 0x7f9fc00d14c0 2024-01-24T17:42:45.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc8000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00d32c0 con 0x7f9fc015fff0 2024-01-24T17:42:45.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc6000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9fc00d14c0 0x7f9fc00cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:45.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc4c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9fc00b2a10 0x7f9fc00d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:46940/0 (socket says 172.21.15.39:46940) 2024-01-24T17:42:45.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc4c00640 1 -- 172.21.15.39:0/658283817 learned_addr learned my addr 172.21.15.39:0/658283817 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:45.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc5600640 1 --2- 172.21.15.39:0/658283817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc015fff0 0x7f9fc00b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:45.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc6000640 1 -- 172.21.15.39:0/658283817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc015fff0 msgr2=0x7f9fc00b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:45.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc6000640 1 --2- 172.21.15.39:0/658283817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc015fff0 0x7f9fc00b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc6000640 1 -- 172.21.15.39:0/658283817 >> v1:172.21.15.39:6789/0 conn(0x7f9fc00b2a10 legacy=0x7f9fc00d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.186+0000 7f9fc6000640 1 -- 172.21.15.39:0/658283817 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc00b30a0 con 0x7f9fc00d14c0 2024-01-24T17:42:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.187+0000 7f9fc6000640 1 --2- 172.21.15.39:0/658283817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc00d14c0 0x7f9fc00cdf20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9fb00097a0 tx=0x7f9fb002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931727d85e17b081 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.188+0000 7f9fbfe00640 1 -- 172.21.15.39:0/658283817 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb002d040 con 0x7f9fc00d14c0 2024-01-24T17:42:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.188+0000 7f9fbfe00640 1 -- 172.21.15.39:0/658283817 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb0031810 con 0x7f9fc00d14c0 2024-01-24T17:42:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.188+0000 7f9fbfe00640 1 -- 172.21.15.39:0/658283817 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0031ba0 con 0x7f9fc00d14c0 2024-01-24T17:42:45.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.188+0000 7f9fc8000640 1 -- 172.21.15.39:0/658283817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc00d14c0 msgr2=0x7f9fc00cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:45.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.188+0000 7f9fc8000640 1 --2- 172.21.15.39:0/658283817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc00d14c0 0x7f9fc00cdf20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9fb00097a0 tx=0x7f9fb002b8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.189+0000 7f9fc8000640 1 -- 172.21.15.39:0/658283817 shutdown_connections 2024-01-24T17:42:45.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.189+0000 7f9fc8000640 1 --2- 172.21.15.39:0/658283817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc00d14c0 0x7f9fc00cdf20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.189+0000 7f9fc8000640 1 --2- 172.21.15.39:0/658283817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc015fff0 0x7f9fc00b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.189+0000 7f9fc8000640 1 -- 172.21.15.39:0/658283817 >> 172.21.15.39:0/658283817 conn(0x7f9fc009bcc0 msgr2=0x7f9fc00d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.189+0000 7f9fc8000640 1 -- 172.21.15.39:0/658283817 shutdown_connections 2024-01-24T17:42:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.189+0000 7f9fc8000640 1 -- 172.21.15.39:0/658283817 wait complete. 2024-01-24T17:42:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.190+0000 7f9fc8000640 1 Processor -- start 2024-01-24T17:42:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.191+0000 7f9fc8000640 1 -- start start 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.191+0000 7f9fc8000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9fc00b2a10 0x7f9fc00c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc8000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc00d14c0 0x7f9fc00ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc8000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc015fff0 0x7f9fc015beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc8000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00cbc50 con 0x7f9fc015fff0 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc8000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00ca910 con 0x7f9fc00b2a10 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc8000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc00cabb0 con 0x7f9fc00d14c0 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc015fff0 0x7f9fc015beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc015fff0 0x7f9fc015beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40532/0 (socket says 172.21.15.39:40532) 2024-01-24T17:42:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc6000640 1 -- 172.21.15.39:0/2327889613 learned_addr learned my addr 172.21.15.39:0/2327889613 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc5600640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9fc00b2a10 0x7f9fc00c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc6000640 1 -- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc00d14c0 msgr2=0x7f9fc00ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc6000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc00d14c0 0x7f9fc00ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc6000640 1 -- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9fc00b2a10 msgr2=0x7f9fc00c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc6000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9fc00b2a10 0x7f9fc00c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.192+0000 7f9fc6000640 1 -- 172.21.15.39:0/2327889613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb0009400 con 0x7f9fc015fff0 2024-01-24T17:42:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.193+0000 7f9fc6000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc015fff0 0x7f9fc015beb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9fb002b8f0 tx=0x7f9fb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.193+0000 7f9fbd600640 1 -- 172.21.15.39:0/2327889613 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb003ace0 con 0x7f9fc015fff0 2024-01-24T17:42:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.193+0000 7f9fbd600640 1 -- 172.21.15.39:0/2327889613 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb003ae60 con 0x7f9fc015fff0 2024-01-24T17:42:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.193+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc00cae90 con 0x7f9fc015fff0 2024-01-24T17:42:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.193+0000 7f9fbd600640 1 -- 172.21.15.39:0/2327889613 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0044630 con 0x7f9fc015fff0 2024-01-24T17:42:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.193+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc00cb450 con 0x7f9fc015fff0 2024-01-24T17:42:45.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.194+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc0073ef0 con 0x7f9fc015fff0 2024-01-24T17:42:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.196+0000 7f9fbd600640 1 -- 172.21.15.39:0/2327889613 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9fb0044790 con 0x7f9fc015fff0 2024-01-24T17:42:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.197+0000 7f9fbd600640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f9806a430 0x7f9f9806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.197+0000 7f9fbd600640 1 -- 172.21.15.39:0/2327889613 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f9fb00aac40 con 0x7f9fc015fff0 2024-01-24T17:42:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.197+0000 7f9fc5600640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f9806a430 0x7f9f9806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:45.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.199+0000 7f9fc5600640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f9806a430 0x7f9f9806c8f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8006fd0 tx=0x7f9fb8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:45.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.202+0000 7f9fbd600640 1 -- 172.21.15.39:0/2327889613 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9fb00761d0 con 0x7f9fc015fff0 2024-01-24T17:42:45.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.441+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9fc006b280 con 0x7f9fc015fff0 2024-01-24T17:42:45.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.944+0000 7f9fbd600640 1 -- 172.21.15.39:0/2327889613 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9fb00797f0 con 0x7f9fc015fff0 2024-01-24T17:42:45.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f9806a430 msgr2=0x7f9f9806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f9806a430 0x7f9f9806c8f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8006fd0 tx=0x7f9fb8008040 comp rx=0 tx=0).stop 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc015fff0 msgr2=0x7f9fc015beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc015fff0 0x7f9fc015beb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9fb002b8f0 tx=0x7f9fb00046c0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 shutdown_connections 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f9806a430 0x7f9f9806c8f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fc015fff0 0x7f9fc015beb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9fc00d14c0 0x7f9fc00ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 --2- 172.21.15.39:0/2327889613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9fc00b2a10 0x7f9fc00c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 >> 172.21.15.39:0/2327889613 conn(0x7f9fc009bcc0 msgr2=0x7f9fc015df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 shutdown_connections 2024-01-24T17:42:45.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:45.949+0000 7f9fc8000640 1 -- 172.21.15.39:0/2327889613 wait complete. 2024-01-24T17:42:45.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-24T17:42:46.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.204+0000 7f384dc00640 1 Processor -- start 2024-01-24T17:42:46.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.204+0000 7f384dc00640 1 -- start start 2024-01-24T17:42:46.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.205+0000 7f384dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f38480b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:46.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.205+0000 7f384dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f38480b2a10 0x7f38480cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:46.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.205+0000 7f384dc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38480cda90 con 0x7f384815fff0 2024-01-24T17:42:46.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.205+0000 7f384dc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38480b2ec0 con 0x7f38480b2a10 2024-01-24T17:42:46.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.205+0000 7f384dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38480d32c0 con 0x7f38480d14c0 2024-01-24T17:42:46.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.205+0000 7f3846a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f384815fff0 0x7f38480d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:46956/0 (socket says 172.21.15.39:46956) 2024-01-24T17:42:46.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.205+0000 7f3846a00640 1 -- 172.21.15.39:0/1204002185 learned_addr learned my addr 172.21.15.39:0/1204002185 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:46.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.205+0000 7f3847400640 1 --2- 172.21.15.39:0/1204002185 >> v2:172.21.15.39:3300/0 conn(0x7f38480b2a10 0x7f38480cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:46.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.206+0000 7f3845600640 1 -- 172.21.15.39:0/1204002185 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96590433 0 0) 0x7f38480cda90 con 0x7f384815fff0 2024-01-24T17:42:46.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.206+0000 7f3846000640 1 --2- 172.21.15.39:0/1204002185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f38480b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:46.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.206+0000 7f3845600640 1 -- 172.21.15.39:0/1204002185 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3828003520 con 0x7f384815fff0 2024-01-24T17:42:46.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.206+0000 7f3847400640 1 -- 172.21.15.39:0/1204002185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 msgr2=0x7f38480b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:46.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.206+0000 7f3847400640 1 --2- 172.21.15.39:0/1204002185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f38480b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.206+0000 7f3847400640 1 -- 172.21.15.39:0/1204002185 >> v1:172.21.15.39:6789/0 conn(0x7f384815fff0 legacy=0x7f38480d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:46.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.206+0000 7f3847400640 1 -- 172.21.15.39:0/1204002185 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38480b30a0 con 0x7f38480b2a10 2024-01-24T17:42:46.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.206+0000 7f3846000640 1 --2- 172.21.15.39:0/1204002185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f38480b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:46.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.207+0000 7f3847400640 1 --2- 172.21.15.39:0/1204002185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38480b2a10 0x7f38480cdf20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3840009d90 tx=0x7f384002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c39595b61773e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.207+0000 7f3845600640 1 -- 172.21.15.39:0/1204002185 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384002cc50 con 0x7f38480b2a10 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.207+0000 7f3845600640 1 -- 172.21.15.39:0/1204002185 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384002cdb0 con 0x7f38480b2a10 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.207+0000 7f3845600640 1 -- 172.21.15.39:0/1204002185 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840034660 con 0x7f38480b2a10 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.208+0000 7f384dc00640 1 -- 172.21.15.39:0/1204002185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38480b2a10 msgr2=0x7f38480cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.208+0000 7f384dc00640 1 --2- 172.21.15.39:0/1204002185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38480b2a10 0x7f38480cdf20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3840009d90 tx=0x7f384002bea0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.208+0000 7f384dc00640 1 -- 172.21.15.39:0/1204002185 shutdown_connections 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.208+0000 7f384dc00640 1 --2- 172.21.15.39:0/1204002185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38480b2a10 0x7f38480cdf20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.208+0000 7f384dc00640 1 --2- 172.21.15.39:0/1204002185 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f38480b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.208+0000 7f384dc00640 1 -- 172.21.15.39:0/1204002185 >> 172.21.15.39:0/1204002185 conn(0x7f384809bcc0 msgr2=0x7f38480d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.208+0000 7f384dc00640 1 -- 172.21.15.39:0/1204002185 shutdown_connections 2024-01-24T17:42:46.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.208+0000 7f384dc00640 1 -- 172.21.15.39:0/1204002185 wait complete. 2024-01-24T17:42:46.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.210+0000 7f384dc00640 1 Processor -- start 2024-01-24T17:42:46.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.210+0000 7f384dc00640 1 -- start start 2024-01-24T17:42:46.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f384dc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f38480b2a10 0x7f38480b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:46.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f384dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f3848157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:46.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f384dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f384815fff0 0x7f38480af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:46.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f384dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38480b1610 con 0x7f384815fff0 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f384dc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3848157cd0 con 0x7f38480b2a10 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f384dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3848157f20 con 0x7f38480d14c0 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f3847400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f384815fff0 0x7f38480af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f3847400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f384815fff0 0x7f38480af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40564/0 (socket says 172.21.15.39:40564) 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.211+0000 7f3847400640 1 -- 172.21.15.39:0/2930017640 learned_addr learned my addr 172.21.15.39:0/2930017640 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3846a00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f38480b2a10 0x7f38480b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3846000640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f3848157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3847400640 1 -- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 msgr2=0x7f3848157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3847400640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f3848157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3847400640 1 -- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f38480b2a10 msgr2=0x7f38480b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3847400640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f38480b2a10 0x7f38480b10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3847400640 1 -- 172.21.15.39:0/2930017640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3840009a70 con 0x7f384815fff0 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3846000640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f3848157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:46.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f3847400640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f384815fff0 0x7f38480af690 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3840009d90 tx=0x7f3840004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:46.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.212+0000 7f383ea00640 1 -- 172.21.15.39:0/2930017640 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384003e610 con 0x7f384815fff0 2024-01-24T17:42:46.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.213+0000 7f383ea00640 1 -- 172.21.15.39:0/2930017640 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384003e770 con 0x7f384815fff0 2024-01-24T17:42:46.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.213+0000 7f383ea00640 1 -- 172.21.15.39:0/2930017640 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384003eb20 con 0x7f384815fff0 2024-01-24T17:42:46.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.213+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3848158200 con 0x7f384815fff0 2024-01-24T17:42:46.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.213+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38481587c0 con 0x7f384815fff0 2024-01-24T17:42:46.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.214+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3848073ef0 con 0x7f384815fff0 2024-01-24T17:42:46.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.215+0000 7f383ea00640 1 -- 172.21.15.39:0/2930017640 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f384003b050 con 0x7f384815fff0 2024-01-24T17:42:46.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.216+0000 7f383ea00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f382006a430 0x7f382006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:46.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.216+0000 7f383ea00640 1 -- 172.21.15.39:0/2930017640 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f38400ab330 con 0x7f384815fff0 2024-01-24T17:42:46.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.216+0000 7f3846a00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f382006a430 0x7f382006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:46.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.218+0000 7f3846a00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f382006a430 0x7f382006c8f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f38380011e0 tx=0x7f3838009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:46.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.221+0000 7f383ea00640 1 -- 172.21.15.39:0/2930017640 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3840076a90 con 0x7f384815fff0 2024-01-24T17:42:46.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.420+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f38480ceae0 con 0x7f384815fff0 2024-01-24T17:42:46.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.956+0000 7f383ea00640 1 -- 172.21.15.39:0/2930017640 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3840009450 con 0x7f384815fff0 2024-01-24T17:42:46.957 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f382006a430 msgr2=0x7f382006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f382006a430 0x7f382006c8f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f38380011e0 tx=0x7f3838009040 comp rx=0 tx=0).stop 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f384815fff0 msgr2=0x7f38480af690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f384815fff0 0x7f38480af690 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3840009d90 tx=0x7f3840004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 shutdown_connections 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f382006a430 0x7f382006c8f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f384815fff0 0x7f38480af690 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f38480d14c0 0x7f3848157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 --2- 172.21.15.39:0/2930017640 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f38480b2a10 0x7f38480b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.961+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 >> 172.21.15.39:0/2930017640 conn(0x7f384809bcc0 msgr2=0x7f38480c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:46.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.962+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 shutdown_connections 2024-01-24T17:42:46.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:46.962+0000 7f384dc00640 1 -- 172.21.15.39:0/2930017640 wait complete. 2024-01-24T17:42:46.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-24T17:42:47.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.206+0000 7f2cc2e00640 1 Processor -- start 2024-01-24T17:42:47.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.207+0000 7f2cc2e00640 1 -- start start 2024-01-24T17:42:47.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.208+0000 7f2cc2e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2cbc071020 0x7f2cbc16c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:47.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.208+0000 7f2cc2e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cbc0ccee0 con 0x7f2cbc0b0730 2024-01-24T17:42:47.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.208+0000 7f2cc2e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cbc0c8d60 con 0x7f2cbc0b0350 2024-01-24T17:42:47.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.208+0000 7f2cc2e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cbc0cc2d0 con 0x7f2cbc071020 2024-01-24T17:42:47.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.208+0000 7f2cb3200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2cbc0b0730 0x7f2cbc0c1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:46970/0 (socket says 172.21.15.39:46970) 2024-01-24T17:42:47.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.208+0000 7f2cb3200640 1 -- 172.21.15.39:0/4012750459 learned_addr learned my addr 172.21.15.39:0/4012750459 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:47.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875483476 0 0) 0x7f2cbc0ccee0 con 0x7f2cbc0b0730 2024-01-24T17:42:47.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c9c003520 con 0x7f2cbc0b0730 2024-01-24T17:42:47.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cc0e00640 1 --2- 172.21.15.39:0/4012750459 >> v2:172.21.15.100:3300/0 conn(0x7f2cbc071020 0x7f2cbc16c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:47.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956206725 0 0) 0x7f2c9c003520 con 0x7f2cbc0b0730 2024-01-24T17:42:47.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 >> v2:172.21.15.100:3300/0 conn(0x7f2cbc071020 msgr2=0x7f2cbc16c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:47.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 --2- 172.21.15.39:0/4012750459 >> v2:172.21.15.100:3300/0 conn(0x7f2cbc071020 0x7f2cbc16c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 >> v1:172.21.15.100:6789/0 conn(0x7f2cbc0b0350 legacy=0x7f2cbc0ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:47.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc0cc5b0 con 0x7f2cbc0b0730 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2ca4001670 con 0x7f2cbc0b0730 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 >> v1:172.21.15.39:6789/0 conn(0x7f2cbc0b0730 legacy=0x7f2cbc0c1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0039f0 0x7f2c9c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c9c0047c0 0x7f2c9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.209+0000 7f2cbb400640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c9c0052a0 0x7f2c9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc0cc2d0 con 0x7f2c9c0052a0 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c00c140 con 0x7f2c9c0047c0 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c00c300 con 0x7f2c9c0039f0 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cc0e00640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c9c0052a0 0x7f2c9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cbbe00640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0039f0 0x7f2c9c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cb3200640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c9c0047c0 0x7f2c9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cc0e00640 1 -- 172.21.15.39:0/4012750459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0039f0 msgr2=0x7f2c9c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:47.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cc0e00640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0039f0 0x7f2c9c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cc0e00640 1 -- 172.21.15.39:0/4012750459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c9c0047c0 msgr2=0x7f2c9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cc0e00640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c9c0047c0 0x7f2c9c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.210+0000 7f2cc0e00640 1 -- 172.21.15.39:0/4012750459 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c003520 con 0x7f2c9c0052a0 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.211+0000 7f2cc0e00640 1 -- 172.21.15.39:0/4012750459 reap_dead start 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.211+0000 7f2cc0e00640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c9c0052a0 0x7f2c9c00b980 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4007330 tx=0x7f2c9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b292be74b5f3977 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb402ccd0 con 0x7f2c9c0052a0 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb402ce30 con 0x7f2c9c0052a0 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cbb400640 1 -- 172.21.15.39:0/4012750459 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4003bd0 con 0x7f2c9c0052a0 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cc2e00640 1 -- 172.21.15.39:0/4012750459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c9c0052a0 msgr2=0x7f2c9c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cc2e00640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c9c0052a0 0x7f2c9c00b980 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4007330 tx=0x7f2c9c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cc2e00640 1 -- 172.21.15.39:0/4012750459 shutdown_connections 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cc2e00640 1 --2- 172.21.15.39:0/4012750459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c9c0052a0 0x7f2c9c00b980 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cc2e00640 1 -- 172.21.15.39:0/4012750459 >> 172.21.15.39:0/4012750459 conn(0x7f2cbc0cb0e0 msgr2=0x7f2cbc0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:47.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cc2e00640 1 -- 172.21.15.39:0/4012750459 shutdown_connections 2024-01-24T17:42:47.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.212+0000 7f2cc2e00640 1 -- 172.21.15.39:0/4012750459 wait complete. 2024-01-24T17:42:47.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.214+0000 7f2cc2e00640 1 Processor -- start 2024-01-24T17:42:47.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.214+0000 7f2cc2e00640 1 -- start start 2024-01-24T17:42:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.215+0000 7f2cc2e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0052a0 0x7f2cbc0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.215+0000 7f2cc2e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cbc161690 0x7f2cbc162e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.215+0000 7f2cc2e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc163370 0x7f2cbc1637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.215+0000 7f2cc2e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc156050 con 0x7f2cbc163370 2024-01-24T17:42:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.215+0000 7f2cc2e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc163d30 con 0x7f2cbc161690 2024-01-24T17:42:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.215+0000 7f2cc2e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc157130 con 0x7f2c9c0052a0 2024-01-24T17:42:47.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc163370 0x7f2cbc1637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc163370 0x7f2cbc1637f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40588/0 (socket says 172.21.15.39:40588) 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 -- 172.21.15.39:0/2419962037 learned_addr learned my addr 172.21.15.39:0/2419962037 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cbbe00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0052a0 0x7f2cbc0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cb3200640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cbc161690 0x7f2cbc162e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 -- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0052a0 msgr2=0x7f2cbc0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0052a0 0x7f2cbc0b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 -- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cbc161690 msgr2=0x7f2cbc162e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cbc161690 0x7f2cbc162e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 -- 172.21.15.39:0/2419962037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc157410 con 0x7f2cbc163370 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cbbe00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0052a0 0x7f2cbc0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cb3200640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cbc161690 0x7f2cbc162e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:47.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.216+0000 7f2cc0e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc163370 0x7f2cbc1637f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4007450 tx=0x7f2cb4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:47.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.217+0000 7f2cb8c00640 1 -- 172.21.15.39:0/2419962037 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40044d0 con 0x7f2cbc163370 2024-01-24T17:42:47.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.217+0000 7f2cb8c00640 1 -- 172.21.15.39:0/2419962037 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb4004650 con 0x7f2cbc163370 2024-01-24T17:42:47.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.217+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cbc1576f0 con 0x7f2cbc163370 2024-01-24T17:42:47.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.217+0000 7f2cb8c00640 1 -- 172.21.15.39:0/2419962037 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4003e30 con 0x7f2cbc163370 2024-01-24T17:42:47.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.217+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cbc156e20 con 0x7f2cbc163370 2024-01-24T17:42:47.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.218+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc073ef0 con 0x7f2cbc163370 2024-01-24T17:42:47.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.220+0000 7f2cb8c00640 1 -- 172.21.15.39:0/2419962037 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2cb403b070 con 0x7f2cbc163370 2024-01-24T17:42:47.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.222+0000 7f2cb8c00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c9406a480 0x7f2c9406c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:47.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.222+0000 7f2cb8c00640 1 -- 172.21.15.39:0/2419962037 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5386+0+0 (secure 0 0 0) 0x7f2cb40a9ff0 con 0x7f2cbc163370 2024-01-24T17:42:47.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.223+0000 7f2cbbe00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c9406a480 0x7f2c9406c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:47.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.228+0000 7f2cbbe00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c9406a480 0x7f2c9406c940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2cac0015b0 tx=0x7f2cac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:47.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.231+0000 7f2cb8c00640 1 -- 172.21.15.39:0/2419962037 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2cb4075720 con 0x7f2cbc163370 2024-01-24T17:42:47.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.430+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2cbc0cbe80 con 0x7f2cbc163370 2024-01-24T17:42:47.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.961+0000 7f2cb8c00640 1 -- 172.21.15.39:0/2419962037 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2cb4078d40 con 0x7f2cbc163370 2024-01-24T17:42:47.962 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-01-24T17:42:47.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.965+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c9406a480 msgr2=0x7f2c9406c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:47.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.965+0000 7f2cc2e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c9406a480 0x7f2c9406c940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2cac0015b0 tx=0x7f2cac009040 comp rx=0 tx=0).stop 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.965+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc163370 msgr2=0x7f2cbc1637f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.965+0000 7f2cc2e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc163370 0x7f2cbc1637f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4007450 tx=0x7f2cb4004ab0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.966+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 shutdown_connections 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.966+0000 7f2cc2e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c9406a480 0x7f2c9406c940 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.966+0000 7f2cc2e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cbc163370 0x7f2cbc1637f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.966+0000 7f2cc2e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2cbc161690 0x7f2cbc162e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.966+0000 7f2cc2e00640 1 --2- 172.21.15.39:0/2419962037 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c9c0052a0 0x7f2cbc0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.966+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 >> 172.21.15.39:0/2419962037 conn(0x7f2cbc0cb0e0 msgr2=0x7f2cbc158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.966+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 shutdown_connections 2024-01-24T17:42:47.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:47.966+0000 7f2cc2e00640 1 -- 172.21.15.39:0/2419962037 wait complete. 2024-01-24T17:42:47.990 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-24T17:42:47.991 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-24T17:42:47.991 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-24T17:42:47.991 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:42:47.991 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-24T17:42:47.991 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-24T17:42:47.991 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-24T17:42:47.992 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:42:47.992 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-24T17:42:47.992 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-24T17:42:47.992 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-24T17:42:47.992 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:42:47.992 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-24T17:42:47.992 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-24T17:42:47.993 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-24T17:42:47.993 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:42:48.009 INFO:tasks.ceph.mds.d:Stopped 2024-01-24T17:42:48.010 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-24T17:42:48.011 INFO:tasks.ceph.mds.b:Stopped 2024-01-24T17:42:48.011 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-24T17:42:48.013 INFO:tasks.ceph.mds.c:Stopped 2024-01-24T17:42:48.013 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-24T17:42:48.015 INFO:tasks.ceph.mds.a:Stopped 2024-01-24T17:42:48.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-24T17:42:48.057 INFO:tasks.ceph.mds.d:Started 2024-01-24T17:42:48.057 DEBUG:teuthology.parallel:result is None 2024-01-24T17:42:48.061 INFO:tasks.ceph.mds.c:Started 2024-01-24T17:42:48.062 DEBUG:teuthology.parallel:result is None 2024-01-24T17:42:48.062 INFO:tasks.ceph.mds.b:Started 2024-01-24T17:42:48.063 DEBUG:teuthology.parallel:result is None 2024-01-24T17:42:48.068 INFO:tasks.ceph.mds.a:Started 2024-01-24T17:42:48.068 DEBUG:teuthology.parallel:result is None 2024-01-24T17:42:48.068 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-24T17:42:48.205 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:42:48.211 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:42:48.235 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:42:48.247 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:42:48.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.284+0000 7fcaef200640 1 Processor -- start 2024-01-24T17:42:48.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.284+0000 7fcaef200640 1 -- start start 2024-01-24T17:42:48.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.285+0000 7fcaef200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcae80cf990 0x7fcae80cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:48.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.285+0000 7fcaef200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae815fb90 0x7fcae815d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:48.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.285+0000 7fcaef200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae80d02b0 con 0x7fcae815d020 2024-01-24T17:42:48.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.285+0000 7fcaef200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae815e080 con 0x7fcae80cf990 2024-01-24T17:42:48.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.285+0000 7fcaef200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae815dac0 con 0x7fcae815fb90 2024-01-24T17:42:48.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.285+0000 7fcae7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcae80cf990 0x7fcae80cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:48.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.285+0000 7fcae7400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fcae815d020 0x7fcae8160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:46982/0 (socket says 172.21.15.39:46982) 2024-01-24T17:42:48.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.285+0000 7fcae7400640 1 -- 172.21.15.39:0/1675005184 learned_addr learned my addr 172.21.15.39:0/1675005184 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.286+0000 7fcaed200640 1 --2- 172.21.15.39:0/1675005184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae815fb90 0x7fcae815d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.286+0000 7fcae7e00640 1 -- 172.21.15.39:0/1675005184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae815fb90 msgr2=0x7fcae815d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.286+0000 7fcae7e00640 1 --2- 172.21.15.39:0/1675005184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae815fb90 0x7fcae815d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.286+0000 7fcae7e00640 1 -- 172.21.15.39:0/1675005184 >> v1:172.21.15.39:6789/0 conn(0x7fcae815d020 legacy=0x7fcae8160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.286+0000 7fcae7e00640 1 -- 172.21.15.39:0/1675005184 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae815dda0 con 0x7fcae80cf990 2024-01-24T17:42:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.286+0000 7fcae6a00640 1 -- 172.21.15.39:0/1675005184 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827669823 0 0) 0x7fcae80d02b0 con 0x7fcae815d020 2024-01-24T17:42:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.286+0000 7fcaed200640 1 --2- 172.21.15.39:0/1675005184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae815fb90 0x7fcae815d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.287+0000 7fcae7e00640 1 --2- 172.21.15.39:0/1675005184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae80cf990 0x7fcae80cfd70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcad80097a0 tx=0x7fcad802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4245e5feefa565 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.287+0000 7fcae6a00640 1 -- 172.21.15.39:0/1675005184 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad802d040 con 0x7fcae80cf990 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.287+0000 7fcae6a00640 1 -- 172.21.15.39:0/1675005184 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad8031810 con 0x7fcae80cf990 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.287+0000 7fcae6a00640 1 -- 172.21.15.39:0/1675005184 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8031ba0 con 0x7fcae80cf990 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.288+0000 7fcaef200640 1 -- 172.21.15.39:0/1675005184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae80cf990 msgr2=0x7fcae80cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.288+0000 7fcaef200640 1 --2- 172.21.15.39:0/1675005184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae80cf990 0x7fcae80cfd70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcad80097a0 tx=0x7fcad802b8b0 comp rx=0 tx=0).stop 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.288+0000 7fcaef200640 1 -- 172.21.15.39:0/1675005184 shutdown_connections 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.288+0000 7fcaef200640 1 --2- 172.21.15.39:0/1675005184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae815fb90 0x7fcae815d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.288+0000 7fcaef200640 1 --2- 172.21.15.39:0/1675005184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae80cf990 0x7fcae80cfd70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.288+0000 7fcaef200640 1 -- 172.21.15.39:0/1675005184 >> 172.21.15.39:0/1675005184 conn(0x7fcae8071790 msgr2=0x7fcae8071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.288+0000 7fcaef200640 1 -- 172.21.15.39:0/1675005184 shutdown_connections 2024-01-24T17:42:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.288+0000 7fcaef200640 1 -- 172.21.15.39:0/1675005184 wait complete. 2024-01-24T17:42:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.290+0000 7fcaef200640 1 Processor -- start 2024-01-24T17:42:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.290+0000 7fcaef200640 1 -- start start 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcaef200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae80cf990 0x7fcae8157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcaef200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae815d020 0x7fcae80c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcaef200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae815fb90 0x7fcae81607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcaef200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae80c78e0 con 0x7fcae815d020 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcaef200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81576d0 con 0x7fcae815fb90 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcaef200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae80c6ff0 con 0x7fcae80cf990 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcae7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae815d020 0x7fcae80c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcae7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae815d020 0x7fcae80c6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40642/0 (socket says 172.21.15.39:40642) 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcae7400640 1 -- 172.21.15.39:0/860262803 learned_addr learned my addr 172.21.15.39:0/860262803 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcaed200640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae815fb90 0x7fcae81607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:48.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.291+0000 7fcae7e00640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae80cf990 0x7fcae8157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:48.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcae7400640 1 -- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae80cf990 msgr2=0x7fcae8157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:48.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcae7400640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae80cf990 0x7fcae8157190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcae7400640 1 -- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae815fb90 msgr2=0x7fcae81607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcae7400640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae815fb90 0x7fcae81607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcae7400640 1 -- 172.21.15.39:0/860262803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad8009400 con 0x7fcae815d020 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcae7400640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae815d020 0x7fcae80c6530 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcad000c430 tx=0x7fcad000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcadfe00640 1 -- 172.21.15.39:0/860262803 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad00074e0 con 0x7fcae815d020 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcadfe00640 1 -- 172.21.15.39:0/860262803 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad0007660 con 0x7fcae815d020 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcadfe00640 1 -- 172.21.15.39:0/860262803 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0004a20 con 0x7fcae815d020 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.292+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae80c72d0 con 0x7fcae815d020 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.293+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae80c7e10 con 0x7fcae815d020 2024-01-24T17:42:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.294+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae8073ef0 con 0x7fcae815d020 2024-01-24T17:42:48.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.295+0000 7fcadfe00640 1 -- 172.21.15.39:0/860262803 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcad0004b80 con 0x7fcae815d020 2024-01-24T17:42:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.295+0000 7fcadfe00640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcac006a430 0x7fcac006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.296+0000 7fcadfe00640 1 -- 172.21.15.39:0/860262803 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcad0088450 con 0x7fcae815d020 2024-01-24T17:42:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.296+0000 7fcae7e00640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcac006a430 0x7fcac006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:48.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.297+0000 7fcae7e00640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcac006a430 0x7fcac006c8f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcad8036000 tx=0x7fcad803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:48.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.300+0000 7fcadfe00640 1 -- 172.21.15.39:0/860262803 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcad0053d30 con 0x7fcae815d020 2024-01-24T17:42:48.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:48.504+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcae80b2140 con 0x7fcae815d020 2024-01-24T17:42:49.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.286+0000 7fcadfe00640 1 -- 172.21.15.39:0/860262803 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcad0057350 con 0x7fcae815d020 2024-01-24T17:42:49.288 INFO:teuthology.orchestra.run.smithi039.stderr: removed all blocklist entries 2024-01-24T17:42:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcac006a430 msgr2=0x7fcac006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcac006a430 0x7fcac006c8f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcad8036000 tx=0x7fcad803c040 comp rx=0 tx=0).stop 2024-01-24T17:42:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae815d020 msgr2=0x7fcae80c6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae815d020 0x7fcae80c6530 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcad000c430 tx=0x7fcad000c8f0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 shutdown_connections 2024-01-24T17:42:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae815fb90 0x7fcae81607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcac006a430 0x7fcac006c8f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae815d020 0x7fcae80c6530 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 --2- 172.21.15.39:0/860262803 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae80cf990 0x7fcae8157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 >> 172.21.15.39:0/860262803 conn(0x7fcae8071790 msgr2=0x7fcae80b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:49.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.290+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 shutdown_connections 2024-01-24T17:42:49.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.291+0000 7fcaef200640 1 -- 172.21.15.39:0/860262803 wait complete. 2024-01-24T17:42:49.313 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-24T17:42:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.531+0000 7f07c4a00640 1 Processor -- start 2024-01-24T17:42:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.531+0000 7f07c4a00640 1 -- start start 2024-01-24T17:42:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.533+0000 7f07c4a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f07bc0cb190 0x7f07bc0c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.533+0000 7f07c4a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07bc06d5a0 con 0x7f07bc0cd560 2024-01-24T17:42:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.533+0000 7f07c4a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07bc06b5a0 con 0x7f07bc1613a0 2024-01-24T17:42:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.533+0000 7f07c4a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07bc065a40 con 0x7f07bc0cb190 2024-01-24T17:42:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.533+0000 7f07c1600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f07bc0cd560 0x7f07bc0c9530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:46988/0 (socket says 172.21.15.39:46988) 2024-01-24T17:42:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.533+0000 7f07c1600640 1 -- 172.21.15.39:0/3179716327 learned_addr learned my addr 172.21.15.39:0/3179716327 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c2a00640 1 --2- 172.21.15.39:0/3179716327 >> v2:172.21.15.100:3300/0 conn(0x7f07bc0cb190 0x7f07bc0c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766704650 0 0) 0x7f07bc06d5a0 con 0x7f07bc0cd560 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f079c003520 con 0x7f07bc0cd560 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1946568540 0 0) 0x7f079c003520 con 0x7f07bc0cd560 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 >> v2:172.21.15.100:3300/0 conn(0x7f07bc0cb190 msgr2=0x7f07bc0c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 --2- 172.21.15.39:0/3179716327 >> v2:172.21.15.100:3300/0 conn(0x7f07bc0cb190 0x7f07bc0c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 >> v1:172.21.15.100:6789/0 conn(0x7f07bc1613a0 legacy=0x7f07bc0cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc065d20 con 0x7f07bc0cd560 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f07a4001670 con 0x7f07bc0cd560 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 >> v1:172.21.15.39:6789/0 conn(0x7f07bc0cd560 legacy=0x7f07bc0c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f079c003970 0x7f079c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f079c0047c0 0x7f079c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:49.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f079c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc065a40 con 0x7f079c005180 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c00c020 con 0x7f079c003970 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.534+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079c00c1e0 con 0x7f079c0047c0 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.535+0000 7f07c2a00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f079c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.535+0000 7f07c2000640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f079c003970 0x7f079c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.535+0000 7f07c1600640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f079c0047c0 0x7f079c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.535+0000 7f07c2a00640 1 -- 172.21.15.39:0/3179716327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f079c0047c0 msgr2=0x7f079c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.535+0000 7f07c2a00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f079c0047c0 0x7f079c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.535+0000 7f07c2a00640 1 -- 172.21.15.39:0/3179716327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f079c003970 msgr2=0x7f079c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.535+0000 7f07c2a00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f079c003970 0x7f079c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.535+0000 7f07c2a00640 1 -- 172.21.15.39:0/3179716327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c003520 con 0x7f079c005180 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.536+0000 7f07c2a00640 1 -- 172.21.15.39:0/3179716327 reap_dead start 2024-01-24T17:42:49.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.536+0000 7f07c2a00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f079c00b980 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f07b4009fc0 tx=0x7f079c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a44acd39c4b826a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.537+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b402cc60 con 0x7f079c005180 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.537+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b400f040 con 0x7f079c005180 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.537+0000 7f07c0c00640 1 -- 172.21.15.39:0/3179716327 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4003990 con 0x7f079c005180 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.537+0000 7f07c4a00640 1 -- 172.21.15.39:0/3179716327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 msgr2=0x7f079c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.537+0000 7f07c4a00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f079c00b980 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f07b4009fc0 tx=0x7f079c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.537+0000 7f07c4a00640 1 -- 172.21.15.39:0/3179716327 shutdown_connections 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.537+0000 7f07c4a00640 1 --2- 172.21.15.39:0/3179716327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f079c00b980 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.537+0000 7f07c4a00640 1 -- 172.21.15.39:0/3179716327 >> 172.21.15.39:0/3179716327 conn(0x7f07bc0b0980 msgr2=0x7f07bc0b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.538+0000 7f07c4a00640 1 -- 172.21.15.39:0/3179716327 shutdown_connections 2024-01-24T17:42:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.538+0000 7f07c4a00640 1 -- 172.21.15.39:0/3179716327 wait complete. 2024-01-24T17:42:49.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.539+0000 7f07c4a00640 1 Processor -- start 2024-01-24T17:42:49.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.539+0000 7f07c4a00640 1 -- start start 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.540+0000 7f07c4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f07bc15cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.540+0000 7f07c4a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07bc0b4180 0x7f07bc15d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.540+0000 7f07c4a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07bc0c9a40 0x7f07bc0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.540+0000 7f07c4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0ca400 con 0x7f079c005180 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.540+0000 7f07c4a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc15d070 con 0x7f07bc0c9a40 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.540+0000 7f07c4a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc15daa0 con 0x7f07bc0b4180 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f07bc15cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f07bc15cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40648/0 (socket says 172.21.15.39:40648) 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 -- 172.21.15.39:0/944858642 learned_addr learned my addr 172.21.15.39:0/944858642 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c1600640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07bc0b4180 0x7f07bc15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 -- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07bc0b4180 msgr2=0x7f07bc15d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07bc0b4180 0x7f07bc15d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2a00640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07bc0c9a40 0x7f07bc0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 -- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07bc0c9a40 msgr2=0x7f07bc0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07bc0c9a40 0x7f07bc0c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 -- 172.21.15.39:0/944858642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc0c43a0 con 0x7f079c005180 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.541+0000 7f07c2000640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f07bc15cb30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00e4d0 tx=0x7f07ac00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.542+0000 7f07ba000640 1 -- 172.21.15.39:0/944858642 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0052e0 con 0x7f079c005180 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.542+0000 7f07ba000640 1 -- 172.21.15.39:0/944858642 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac005440 con 0x7f079c005180 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.542+0000 7f07ba000640 1 -- 172.21.15.39:0/944858642 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac003810 con 0x7f079c005180 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.542+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07bc0c4680 con 0x7f079c005180 2024-01-24T17:42:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.542+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07bc0c4c40 con 0x7f079c005180 2024-01-24T17:42:49.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.543+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07bc0c4e00 con 0x7f079c005180 2024-01-24T17:42:49.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.544+0000 7f07ba000640 1 -- 172.21.15.39:0/944858642 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f07ac003970 con 0x7f079c005180 2024-01-24T17:42:49.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.545+0000 7f07ba000640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f079406a460 0x7f079406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:49.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.545+0000 7f07ba000640 1 -- 172.21.15.39:0/944858642 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f07ac088310 con 0x7f079c005180 2024-01-24T17:42:49.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.545+0000 7f07c1600640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f079406a460 0x7f079406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:49.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.547+0000 7f07c1600640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f079406a460 0x7f079406c920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07a4001200 tx=0x7f07a4000f30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:49.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.549+0000 7f07ba000640 1 -- 172.21.15.39:0/944858642 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f07ac053c50 con 0x7f079c005180 2024-01-24T17:42:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.810+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f07bc073ef0 con 0x7f079c005180 2024-01-24T17:42:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.811+0000 7f07ba000640 1 -- 172.21.15.39:0/944858642 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f07ac057270 con 0x7f079c005180 2024-01-24T17:42:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:49.813 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:42:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_dump": [ 2024-01-24T17:42:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.a", 2024-01-24T17:42:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlZZEtLBAAUxf4chL3tgy3zHlZoraXlg==", 2024-01-24T17:42:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:42:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:42:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:42:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.b", 2024-01-24T17:42:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Fln4utEhAA52ngJFqrdfa9bx56mVhRVA==", 2024-01-24T17:42:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:42:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:42:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:42:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.c", 2024-01-24T17:42:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flo8iGARAAf+29RXWr4RHkILIz7em3mA==", 2024-01-24T17:42:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:42:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:42:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:42:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.818 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.818 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.d", 2024-01-24T17:42:49.819 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flh+08IRAARhZxy//j8c9b3Hg9uV9ajg==", 2024-01-24T17:42:49.819 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.819 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:42:49.819 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:42:49.819 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:42:49.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.820 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.820 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.0", 2024-01-24T17:42:49.820 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCQS7FlOj/pLBAAzi89tAacOfDKdlU0BUJvvA==", 2024-01-24T17:42:49.820 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.821 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:42:49.821 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:42:49.821 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.821 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.822 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.1", 2024-01-24T17:42:49.822 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCUS7FlbRAKDRAAAuAy47MDTtvmh8qNA3ZLTg==", 2024-01-24T17:42:49.822 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.822 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:42:49.822 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:42:49.822 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.823 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.823 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.2", 2024-01-24T17:42:49.823 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCXS7Fl7QmRGRAAVECH7oYTdvDPSHBy9WmHwA==", 2024-01-24T17:42:49.823 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.824 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:42:49.824 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:42:49.824 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.824 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.824 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.3", 2024-01-24T17:42:49.824 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCaS7FlCQHIMxAA9OchsMzsKFxs110SeJ3Atw==", 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.4", 2024-01-24T17:42:49.825 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCiS7FlDsBKBRAAXF2HafraT5nRbB5tBRUgWQ==", 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.5", 2024-01-24T17:42:49.826 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQClS7FlfrXBHhAAy8GP3TLX/iok3OYPGTP5Mw==", 2024-01-24T17:42:49.827 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.827 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:42:49.827 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:42:49.827 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.827 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.6", 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCoS7Fla11hNhAA/3oMP7cFtPSTdJl7eSnPTA==", 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.7", 2024-01-24T17:42:49.828 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCsS7FlKFMtEhAAPmaFzr5XHVQaKZYkJiIarA==", 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.0", 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlTi/3LhAAc8FwXtLH2XjlFMO7wDMcSQ==", 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:42:49.829 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.1", 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlH5s7ChAAlFfFDziTgfv4ibWO08N+4Q==", 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.2", 2024-01-24T17:42:49.830 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlIISlOhAAoVxxez7UjXDxyHQNNPnn2g==", 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.3", 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlO/U/FBAA2pI9aGSS2cZ3AMnn+W1fQA==", 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:42:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.admin", 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCGS7FlxKo7HhAAlu037DIc1WmWkz81BG3j7Q==", 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.bootstrap-mds", 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQC1S7Fliz0qCBAAjpgY73JIUJbrQiIx3g9+Og==", 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile bootstrap-mds" 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.bootstrap-mgr", 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQC1S7FlhkcqCBAA9mU9/1aS0jcdZ/2syLV40A==", 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-24T17:42:49.833 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.bootstrap-osd", 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQC1S7FlIFAqCBAA+ZO14B8Y0kBAngncVuZqUA==", 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile bootstrap-osd" 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.bootstrap-rbd", 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQC1S7Fl6FoqCBAAzjmFnpF6vIOluVi5VGLqiQ==", 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQC1S7FleWQqCBAAZaqnNvM6luS9AxWNf1zCbw==", 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.bootstrap-rgw", 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQC1S7Fldm0qCBAAshUKtIc/PuMAP/T4qCGE7A==", 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.x", 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlRbF6HxAAoFjlyWb5oqIkC3L1iPRmHw==", 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.y", 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7Fl512kFRAAdCpU+QjRmGJmMD5uTdFmHQ==", 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:42:49.836 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:42:49.837 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:42:49.837 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:42:49.837 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.837 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:49.837 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:42:49.837 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.813+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f079406a460 msgr2=0x7f079406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.813+0000 7f07c4a00640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f079406a460 0x7f079406c920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07a4001200 tx=0x7f07a4000f30 comp rx=0 tx=0).stop 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 msgr2=0x7f07bc15cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f07bc15cb30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00e4d0 tx=0x7f07ac00e990 comp rx=0 tx=0).stop 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 shutdown_connections 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07bc0c9a40 0x7f07bc0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07bc0b4180 0x7f07bc15d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f079406a460 0x7f079406c920 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 --2- 172.21.15.39:0/944858642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f079c005180 0x7f07bc15cb30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 >> 172.21.15.39:0/944858642 conn(0x7f07bc0b0980 msgr2=0x7f07bc0ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:49.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 shutdown_connections 2024-01-24T17:42:49.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:49.814+0000 7f07c4a00640 1 -- 172.21.15.39:0/944858642 wait complete. 2024-01-24T17:42:49.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-24T17:42:49.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:42:49.949 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:42:50.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.063+0000 7f8156a00640 1 Processor -- start 2024-01-24T17:42:50.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.063+0000 7f8156a00640 1 -- start start 2024-01-24T17:42:50.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f8156a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f815015d660 0x7f81500cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f8156a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815015d1d0 con 0x7f81500cdd90 2024-01-24T17:42:50.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f8156a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815015ced0 con 0x7f81500cadb0 2024-01-24T17:42:50.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f8156a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8150160be0 con 0x7f815015d660 2024-01-24T17:42:50.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f8154c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f81500cdd90 0x7f81501603b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47002/0 (socket says 172.21.15.39:47002) 2024-01-24T17:42:50.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f8154c00640 1 -- 172.21.15.39:0/1166120616 learned_addr learned my addr 172.21.15.39:0/1166120616 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:50.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937977766 0 0) 0x7f815015d1d0 con 0x7f81500cdd90 2024-01-24T17:42:50.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8130003520 con 0x7f81500cdd90 2024-01-24T17:42:50.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.065+0000 7f8156000640 1 --2- 172.21.15.39:0/1166120616 >> v2:172.21.15.100:3300/0 conn(0x7f815015d660 0x7f81500cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2212982954 0 0) 0x7f8130003520 con 0x7f81500cdd90 2024-01-24T17:42:50.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 >> v2:172.21.15.100:3300/0 conn(0x7f815015d660 msgr2=0x7f81500cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:50.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 --2- 172.21.15.39:0/1166120616 >> v2:172.21.15.100:3300/0 conn(0x7f815015d660 0x7f81500cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 >> v1:172.21.15.100:6789/0 conn(0x7f81500cadb0 legacy=0x7f81500cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8150160ec0 con 0x7f81500cdd90 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8138001670 con 0x7f81500cdd90 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 >> v1:172.21.15.39:6789/0 conn(0x7f81500cdd90 legacy=0x7f81501603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8130003970 0x7f8130003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81300047c0 0x7f8130004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81300052a0 0x7f813000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8150160be0 con 0x7f81300052a0 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813000c150 con 0x7f8130003970 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813000c310 con 0x7f81300047c0 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f8156000640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81300052a0 0x7f813000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.066+0000 7f8154c00640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81300047c0 0x7f8130004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.067+0000 7f8155600640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8130003970 0x7f8130003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.067+0000 7f8156000640 1 -- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81300047c0 msgr2=0x7f8130004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:50.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.067+0000 7f8156000640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81300047c0 0x7f8130004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.067+0000 7f8156000640 1 -- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8130003970 msgr2=0x7f8130003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:50.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.067+0000 7f8156000640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8130003970 0x7f8130003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.067+0000 7f8156000640 1 -- 172.21.15.39:0/1166120616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8130003520 con 0x7f81300052a0 2024-01-24T17:42:50.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.067+0000 7f8155600640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8130003970 0x7f8130003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:50.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.068+0000 7f8156000640 1 -- 172.21.15.39:0/1166120616 reap_dead start 2024-01-24T17:42:50.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.068+0000 7f8156000640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81300052a0 0x7f813000b980 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8148007330 tx=0x7f81300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e588bf5d5381a5c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.068+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814802ccd0 con 0x7f81300052a0 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.068+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814802ce30 con 0x7f81300052a0 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.068+0000 7f814fe00640 1 -- 172.21.15.39:0/1166120616 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148003bd0 con 0x7f81300052a0 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.069+0000 7f8156a00640 1 -- 172.21.15.39:0/1166120616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81300052a0 msgr2=0x7f813000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.069+0000 7f8156a00640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81300052a0 0x7f813000b980 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8148007330 tx=0x7f81300057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.069+0000 7f8156a00640 1 -- 172.21.15.39:0/1166120616 shutdown_connections 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.069+0000 7f8156a00640 1 --2- 172.21.15.39:0/1166120616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81300052a0 0x7f813000b980 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.069+0000 7f8156a00640 1 -- 172.21.15.39:0/1166120616 >> 172.21.15.39:0/1166120616 conn(0x7f815006a2a0 msgr2=0x7f81500afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.069+0000 7f8156a00640 1 -- 172.21.15.39:0/1166120616 shutdown_connections 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.069+0000 7f8156a00640 1 -- 172.21.15.39:0/1166120616 wait complete. 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.071+0000 7f8156a00640 1 Processor -- start 2024-01-24T17:42:50.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.071+0000 7f8156a00640 1 -- start start 2024-01-24T17:42:50.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8156a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81300052a0 0x7f81500b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8156a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81500c2660 0x7f815014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8156a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81500b26f0 0x7f81500b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8156a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81500b30b0 con 0x7f81500c2660 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8156a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81500c2bf0 con 0x7f81300052a0 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8156a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81500c2ed0 con 0x7f81500b26f0 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8154c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81500c2660 0x7f815014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8154c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81500c2660 0x7f815014b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40660/0 (socket says 172.21.15.39:40660) 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.072+0000 7f8154c00640 1 -- 172.21.15.39:0/2608972303 learned_addr learned my addr 172.21.15.39:0/2608972303 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8155600640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81300052a0 0x7f81500b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8156000640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81500b26f0 0x7f81500b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8154c00640 1 -- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81500b26f0 msgr2=0x7f81500b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8154c00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81500b26f0 0x7f81500b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8154c00640 1 -- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81300052a0 msgr2=0x7f81500b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8154c00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81300052a0 0x7f81500b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8154c00640 1 -- 172.21.15.39:0/2608972303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81500c31b0 con 0x7f81500c2660 2024-01-24T17:42:50.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8156000640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81500b26f0 0x7f81500b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:50.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8154c00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81500c2660 0x7f815014b9b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f813800e710 tx=0x7f813800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:50.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f814d600640 1 -- 172.21.15.39:0/2608972303 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813800ee20 con 0x7f81500c2660 2024-01-24T17:42:50.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f814d600640 1 -- 172.21.15.39:0/2608972303 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813800c8d0 con 0x7f81500c2660 2024-01-24T17:42:50.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f814d600640 1 -- 172.21.15.39:0/2608972303 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813800cc80 con 0x7f81500c2660 2024-01-24T17:42:50.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81500c3490 con 0x7f81500c2660 2024-01-24T17:42:50.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.073+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815014bef0 con 0x7f81500c2660 2024-01-24T17:42:50.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.075+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815004dff0 con 0x7f81500c2660 2024-01-24T17:42:50.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.075+0000 7f814d600640 1 -- 172.21.15.39:0/2608972303 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f813801d460 con 0x7f81500c2660 2024-01-24T17:42:50.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.076+0000 7f814d600640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f812806a460 0x7f812806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.076+0000 7f814d600640 1 -- 172.21.15.39:0/2608972303 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8138088550 con 0x7f81500c2660 2024-01-24T17:42:50.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.077+0000 7f8155600640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f812806a460 0x7f812806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.078+0000 7f8155600640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f812806a460 0x7f812806c920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8140004e70 tx=0x7f8140009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:50.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.081+0000 7f814d600640 1 -- 172.21.15.39:0/2608972303 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8138053e90 con 0x7f81500c2660 2024-01-24T17:42:50.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.350+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f81500b0ac0 con 0x7f81500c2660 2024-01-24T17:42:50.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.358+0000 7f814d600640 1 -- 172.21.15.39:0/2608972303 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f81380574b0 con 0x7f81500c2660 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.361+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f812806a460 msgr2=0x7f812806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.361+0000 7f8156a00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f812806a460 0x7f812806c920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8140004e70 tx=0x7f8140009040 comp rx=0 tx=0).stop 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81500c2660 msgr2=0x7f815014b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81500c2660 0x7f815014b9b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f813800e710 tx=0x7f813800ebd0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 shutdown_connections 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f81500b26f0 0x7f81500b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f812806a460 0x7f812806c920 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81500c2660 0x7f815014b9b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 --2- 172.21.15.39:0/2608972303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81300052a0 0x7f81500b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 >> 172.21.15.39:0/2608972303 conn(0x7f815006a2a0 msgr2=0x7f81500ac040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 shutdown_connections 2024-01-24T17:42:50.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.362+0000 7f8156a00640 1 -- 172.21.15.39:0/2608972303 wait complete. 2024-01-24T17:42:50.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-24T17:42:50.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.640+0000 7f83b6400640 1 Processor -- start 2024-01-24T17:42:50.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.641+0000 7f83b6400640 1 -- start start 2024-01-24T17:42:50.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.642+0000 7f83b6400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b00cb020 0x7f83b015ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.642+0000 7f83b6400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f83b00c8c10 0x7f83b00c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.642+0000 7f83b6400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b015b240 con 0x7f83b00cb400 2024-01-24T17:42:50.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.642+0000 7f83b6400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b015b520 con 0x7f83b00c8c10 2024-01-24T17:42:50.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.642+0000 7f83b6400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b015b800 con 0x7f83b00cb020 2024-01-24T17:42:50.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83a7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f83b00cb400 0x7f83b00ca4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47006/0 (socket says 172.21.15.39:47006) 2024-01-24T17:42:50.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83a7e00640 1 -- 172.21.15.39:0/1726073292 learned_addr learned my addr 172.21.15.39:0/1726073292 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83b5000640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b00cb020 0x7f83b015ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83b5a00640 1 --2- 172.21.15.39:0/1726073292 >> v2:172.21.15.100:3300/0 conn(0x7f83b00c8c10 0x7f83b00c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813138580 0 0) 0x7f83b015b240 con 0x7f83b00cb400 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8390003520 con 0x7f83b00cb400 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735272046 0 0) 0x7f8390003520 con 0x7f83b00cb400 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b00cb020 msgr2=0x7f83b015ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b00cb020 0x7f83b015ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 >> v2:172.21.15.100:3300/0 conn(0x7f83b00c8c10 msgr2=0x7f83b00c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 --2- 172.21.15.39:0/1726073292 >> v2:172.21.15.100:3300/0 conn(0x7f83b00c8c10 0x7f83b00c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b015bae0 con 0x7f83b00cb400 2024-01-24T17:42:50.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.643+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8398001670 con 0x7f83b00cb400 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 >> v1:172.21.15.39:6789/0 conn(0x7f83b00cb400 legacy=0x7f83b00ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83afe00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b00cb020 0x7f8390003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83afe00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83afe00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8390004d00 0x7f8390005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b015b520 con 0x7f8390003f40 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b015b800 con 0x7f8390004d00 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839000b980 con 0x7f83b00cb020 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83a7e00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83b5a00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8390004d00 0x7f8390005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83b5000640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b00cb020 0x7f8390003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83a7e00640 1 -- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b00cb020 msgr2=0x7f8390003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83a7e00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b00cb020 0x7f8390003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83a7e00640 1 -- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8390004d00 msgr2=0x7f8390005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:50.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83a7e00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8390004d00 0x7f8390005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.644+0000 7f83a7e00640 1 -- 172.21.15.39:0/1726073292 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390003520 con 0x7f8390003f40 2024-01-24T17:42:50.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.645+0000 7f83a7e00640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83900047c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f839800c190 tx=0x7f83900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f4e447b39d9dac server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:50.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.646+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398005ad0 con 0x7f8390003f40 2024-01-24T17:42:50.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.646+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8398005c30 con 0x7f8390003f40 2024-01-24T17:42:50.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.646+0000 7f83afe00640 1 -- 172.21.15.39:0/1726073292 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839802c810 con 0x7f8390003f40 2024-01-24T17:42:50.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.646+0000 7f83b6400640 1 -- 172.21.15.39:0/1726073292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 msgr2=0x7f83900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.646+0000 7f83b6400640 1 --2- 172.21.15.39:0/1726073292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83900047c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f839800c190 tx=0x7f83900057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.646+0000 7f83b5a00640 1 -- 172.21.15.39:0/1726073292 reap_dead start 2024-01-24T17:42:50.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.646+0000 7f83b6400640 1 -- 172.21.15.39:0/1726073292 shutdown_connections 2024-01-24T17:42:50.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.646+0000 7f83b6400640 1 -- 172.21.15.39:0/1726073292 >> 172.21.15.39:0/1726073292 conn(0x7f83b006a2a0 msgr2=0x7f83b00afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:50.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.647+0000 7f83b6400640 1 -- 172.21.15.39:0/1726073292 shutdown_connections 2024-01-24T17:42:50.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.647+0000 7f83b6400640 1 -- 172.21.15.39:0/1726073292 wait complete. 2024-01-24T17:42:50.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.648+0000 7f83b6400640 1 Processor -- start 2024-01-24T17:42:50.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.648+0000 7f83b6400640 1 -- start start 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.649+0000 7f83b6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83b01f02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.649+0000 7f83b6400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b00c8030 0x7f83b0152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.649+0000 7f83b6400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b01f38e0 0x7f83b01f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.649+0000 7f83b6400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b01530b0 con 0x7f8390003f40 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.649+0000 7f83b6400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00b10f0 con 0x7f83b00c8030 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.649+0000 7f83b6400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00c84b0 con 0x7f83b01f38e0 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83b01f02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83b01f02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40686/0 (socket says 172.21.15.39:40686) 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 -- 172.21.15.39:0/2434073447 learned_addr learned my addr 172.21.15.39:0/2434073447 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 -- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b01f38e0 msgr2=0x7f83b01f3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b01f38e0 0x7f83b01f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83a7e00640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b00c8030 0x7f83b0152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 -- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b00c8030 msgr2=0x7f83b0152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b00c8030 0x7f83b0152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 -- 172.21.15.39:0/2434073447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00c7c00 con 0x7f8390003f40 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83a7e00640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b00c8030 0x7f83b0152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.650+0000 7f83b5000640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83b01f02e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f83a0010810 tx=0x7f83a0010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.651+0000 7f83ad600640 1 -- 172.21.15.39:0/2434073447 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0009610 con 0x7f8390003f40 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.651+0000 7f83ad600640 1 -- 172.21.15.39:0/2434073447 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a000eb60 con 0x7f8390003f40 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.651+0000 7f83ad600640 1 -- 172.21.15.39:0/2434073447 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a001c400 con 0x7f8390003f40 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.651+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b01f08d0 con 0x7f8390003f40 2024-01-24T17:42:50.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.651+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b01f0e90 con 0x7f8390003f40 2024-01-24T17:42:50.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.652+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b004dff0 con 0x7f8390003f40 2024-01-24T17:42:50.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.653+0000 7f83ad600640 1 -- 172.21.15.39:0/2434073447 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f83a001c560 con 0x7f8390003f40 2024-01-24T17:42:50.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.654+0000 7f83ad600640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f838806a460 0x7f838806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:50.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.654+0000 7f83ad600640 1 -- 172.21.15.39:0/2434073447 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f83a0088ac0 con 0x7f8390003f40 2024-01-24T17:42:50.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.654+0000 7f83a7e00640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f838806a460 0x7f838806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:50.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.656+0000 7f83a7e00640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f838806a460 0x7f838806c920 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f839800c160 tx=0x7f8398007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:50.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.658+0000 7f83ad600640 1 -- 172.21.15.39:0/2434073447 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f83a0054410 con 0x7f8390003f40 2024-01-24T17:42:50.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.923+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f83b0071f00 con 0x7f8390003f40 2024-01-24T17:42:50.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.933+0000 7f83ad600640 1 -- 172.21.15.39:0/2434073447 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f83a0057a30 con 0x7f8390003f40 2024-01-24T17:42:50.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f838806a460 msgr2=0x7f838806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f838806a460 0x7f838806c920 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f839800c160 tx=0x7f8398007000 comp rx=0 tx=0).stop 2024-01-24T17:42:50.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 msgr2=0x7f83b01f02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:50.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83b01f02e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f83a0010810 tx=0x7f83a0010cd0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 shutdown_connections 2024-01-24T17:42:50.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f83b01f38e0 0x7f83b01f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f83b00c8030 0x7f83b0152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f838806a460 0x7f838806c920 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 --2- 172.21.15.39:0/2434073447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8390003f40 0x7f83b01f02e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:50.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 >> 172.21.15.39:0/2434073447 conn(0x7f83b006a2a0 msgr2=0x7f83b015a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:50.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 shutdown_connections 2024-01-24T17:42:50.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:50.936+0000 7f83b6400640 1 -- 172.21.15.39:0/2434073447 wait complete. 2024-01-24T17:42:50.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-24T17:42:51.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.173+0000 7f7e75600640 1 Processor -- start 2024-01-24T17:42:51.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.173+0000 7f7e75600640 1 -- start start 2024-01-24T17:42:51.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.175+0000 7f7e75600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 0x7f7e7015ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.175+0000 7f7e75600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e700cb400 0x7f7e700ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.175+0000 7f7e75600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7015b240 con 0x7f7e700c8c10 2024-01-24T17:42:51.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.175+0000 7f7e75600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7015b520 con 0x7f7e700cb400 2024-01-24T17:42:51.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.175+0000 7f7e75600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7015b800 con 0x7f7e700cb020 2024-01-24T17:42:51.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.176+0000 7f7e67200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e700cb400 0x7f7e700ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.176+0000 7f7e6fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 0x7f7e7015ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.176+0000 7f7e67200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e700cb400 0x7f7e700ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:41490/0 (socket says 172.21.15.39:41490) 2024-01-24T17:42:51.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.176+0000 7f7e67200640 1 -- 172.21.15.39:0/3659895632 learned_addr learned my addr 172.21.15.39:0/3659895632 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:51.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.176+0000 7f7e67200640 1 -- 172.21.15.39:0/3659895632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 msgr2=0x7f7e7015ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:51.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.176+0000 7f7e67200640 1 --2- 172.21.15.39:0/3659895632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 0x7f7e7015ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.176+0000 7f7e67200640 1 -- 172.21.15.39:0/3659895632 >> v1:172.21.15.100:6789/0 conn(0x7f7e700c8c10 legacy=0x7f7e7015e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.176+0000 7f7e67200640 1 -- 172.21.15.39:0/3659895632 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7015bae0 con 0x7f7e700cb400 2024-01-24T17:42:51.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.178+0000 7f7e67200640 1 --2- 172.21.15.39:0/3659895632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 0x7f7e700ca4e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009870 tx=0x7f7e5802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4b670bf78714c00 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:51.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.178+0000 7f7e6f400640 1 -- 172.21.15.39:0/3659895632 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58030840 con 0x7f7e700cb400 2024-01-24T17:42:51.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.178+0000 7f7e6f400640 1 -- 172.21.15.39:0/3659895632 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e580309a0 con 0x7f7e700cb400 2024-01-24T17:42:51.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.178+0000 7f7e6f400640 1 -- 172.21.15.39:0/3659895632 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58030d30 con 0x7f7e700cb400 2024-01-24T17:42:51.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.179+0000 7f7e75600640 1 -- 172.21.15.39:0/3659895632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 msgr2=0x7f7e700ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.179+0000 7f7e75600640 1 --2- 172.21.15.39:0/3659895632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 0x7f7e700ca4e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009870 tx=0x7f7e5802b980 comp rx=0 tx=0).stop 2024-01-24T17:42:51.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.179+0000 7f7e75600640 1 -- 172.21.15.39:0/3659895632 shutdown_connections 2024-01-24T17:42:51.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.179+0000 7f7e75600640 1 --2- 172.21.15.39:0/3659895632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 0x7f7e700ca4e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.179+0000 7f7e75600640 1 --2- 172.21.15.39:0/3659895632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 0x7f7e7015ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.179+0000 7f7e75600640 1 -- 172.21.15.39:0/3659895632 >> 172.21.15.39:0/3659895632 conn(0x7f7e7006a2a0 msgr2=0x7f7e700afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:51.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.179+0000 7f7e75600640 1 -- 172.21.15.39:0/3659895632 shutdown_connections 2024-01-24T17:42:51.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.179+0000 7f7e75600640 1 -- 172.21.15.39:0/3659895632 wait complete. 2024-01-24T17:42:51.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.181+0000 7f7e75600640 1 Processor -- start 2024-01-24T17:42:51.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.181+0000 7f7e75600640 1 -- start start 2024-01-24T17:42:51.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e75600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e700c8c10 0x7f7e700c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e75600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 0x7f7e70146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e75600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 0x7f7e70148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e75600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70146950 con 0x7f7e700c8c10 2024-01-24T17:42:51.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e75600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700c5b70 con 0x7f7e700cb400 2024-01-24T17:42:51.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e75600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e701587d0 con 0x7f7e700cb020 2024-01-24T17:42:51.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e6fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e700c8c10 0x7f7e700c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e6fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e700c8c10 0x7f7e700c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40688/0 (socket says 172.21.15.39:40688) 2024-01-24T17:42:51.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e6fe00640 1 -- 172.21.15.39:0/4179939844 learned_addr learned my addr 172.21.15.39:0/4179939844 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:51.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e74c00640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 0x7f7e70148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.182+0000 7f7e67200640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 0x7f7e70146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6fe00640 1 -- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 msgr2=0x7f7e70146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6fe00640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 0x7f7e70146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6fe00640 1 -- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 msgr2=0x7f7e70148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6fe00640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 0x7f7e70148440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6fe00640 1 -- 172.21.15.39:0/4179939844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e580094d0 con 0x7f7e700c8c10 2024-01-24T17:42:51.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6fe00640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e700c8c10 0x7f7e700c5470 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000e9b0 tx=0x7f7e6000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:51.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6cc00640 1 -- 172.21.15.39:0/4179939844 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60009670 con 0x7f7e700c8c10 2024-01-24T17:42:51.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6cc00640 1 -- 172.21.15.39:0/4179939844 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6000c9a0 con 0x7f7e700c8c10 2024-01-24T17:42:51.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e6cc00640 1 -- 172.21.15.39:0/4179939844 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000cd50 con 0x7f7e700c8c10 2024-01-24T17:42:51.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e70158ab0 con 0x7f7e700c8c10 2024-01-24T17:42:51.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.183+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e70159070 con 0x7f7e700c8c10 2024-01-24T17:42:51.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.185+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7004dff0 con 0x7f7e700c8c10 2024-01-24T17:42:51.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.186+0000 7f7e6cc00640 1 -- 172.21.15.39:0/4179939844 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7e60013470 con 0x7f7e700c8c10 2024-01-24T17:42:51.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.186+0000 7f7e6cc00640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e4806a430 0x7f7e4806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.187+0000 7f7e6cc00640 1 -- 172.21.15.39:0/4179939844 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e60088890 con 0x7f7e700c8c10 2024-01-24T17:42:51.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.187+0000 7f7e67200640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e4806a430 0x7f7e4806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.188+0000 7f7e67200640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e4806a430 0x7f7e4806c8f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7e58036000 tx=0x7f7e58008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:51.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.191+0000 7f7e6cc00640 1 -- 172.21.15.39:0/4179939844 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7e600541d0 con 0x7f7e700c8c10 2024-01-24T17:42:51.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.448+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f7e700c89f0 con 0x7f7e700c8c10 2024-01-24T17:42:51.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.453+0000 7f7e6cc00640 1 -- 172.21.15.39:0/4179939844 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7e600170a0 con 0x7f7e700c8c10 2024-01-24T17:42:51.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e4806a430 msgr2=0x7f7e4806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e4806a430 0x7f7e4806c8f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7e58036000 tx=0x7f7e58008030 comp rx=0 tx=0).stop 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e700c8c10 msgr2=0x7f7e700c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e700c8c10 0x7f7e700c5470 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000e9b0 tx=0x7f7e6000ee70 comp rx=0 tx=0).stop 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 shutdown_connections 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e700cb400 0x7f7e70148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e700cb020 0x7f7e70146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e4806a430 0x7f7e4806c8f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 --2- 172.21.15.39:0/4179939844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e700c8c10 0x7f7e700c5470 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.456+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 >> 172.21.15.39:0/4179939844 conn(0x7f7e7006a2a0 msgr2=0x7f7e7006bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.457+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 shutdown_connections 2024-01-24T17:42:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.457+0000 7f7e75600640 1 -- 172.21.15.39:0/4179939844 wait complete. 2024-01-24T17:42:51.474 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.683+0000 7f0bda800640 1 Processor -- start 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.684+0000 7f0bda800640 1 -- start start 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bda800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0bd40cb020 0x7f0bd415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bda800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0bd40c8c10 0x7f0bd40c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bda800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd415b240 con 0x7f0bd40cb400 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bda800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd415b520 con 0x7f0bd40cb020 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bda800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd415b800 con 0x7f0bd40c8c10 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bd9400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0bd40cb020 0x7f0bd415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bd9400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0bd40cb020 0x7f0bd415ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40692/0 (socket says 172.21.15.39:40692) 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bd9400640 1 -- 172.21.15.39:0/1788397035 learned_addr learned my addr 172.21.15.39:0/1788397035 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.685+0000 7f0bd9e00640 1 --2- 172.21.15.39:0/1788397035 >> v2:172.21.15.100:3300/0 conn(0x7f0bd40c8c10 0x7f0bd40c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.686+0000 7f0bd9400640 1 -- 172.21.15.39:0/1788397035 >> v2:172.21.15.100:3300/0 conn(0x7f0bd40c8c10 msgr2=0x7f0bd40c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.686+0000 7f0bd9400640 1 --2- 172.21.15.39:0/1788397035 >> v2:172.21.15.100:3300/0 conn(0x7f0bd40c8c10 0x7f0bd40c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.686+0000 7f0bd9400640 1 -- 172.21.15.39:0/1788397035 >> v1:172.21.15.100:6789/0 conn(0x7f0bd40cb400 legacy=0x7f0bd40ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.686+0000 7f0bd9400640 1 -- 172.21.15.39:0/1788397035 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd415bae0 con 0x7f0bd40cb020 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.687+0000 7f0bd9400640 1 --2- 172.21.15.39:0/1788397035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb020 0x7f0bd415ac60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40c95e0 tx=0x7f0bc402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11033aeb12159c08 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.687+0000 7f0bd3e00640 1 -- 172.21.15.39:0/1788397035 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4031870 con 0x7f0bd40cb020 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.687+0000 7f0bd3e00640 1 -- 172.21.15.39:0/1788397035 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc40319d0 con 0x7f0bd40cb020 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.687+0000 7f0bd3e00640 1 -- 172.21.15.39:0/1788397035 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4031d60 con 0x7f0bd40cb020 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.688+0000 7f0bda800640 1 -- 172.21.15.39:0/1788397035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb020 msgr2=0x7f0bd415ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.688+0000 7f0bda800640 1 --2- 172.21.15.39:0/1788397035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb020 0x7f0bd415ac60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40c95e0 tx=0x7f0bc402b910 comp rx=0 tx=0).stop 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.688+0000 7f0bda800640 1 -- 172.21.15.39:0/1788397035 shutdown_connections 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.688+0000 7f0bda800640 1 --2- 172.21.15.39:0/1788397035 >> v2:172.21.15.100:3300/0 conn(0x7f0bd40c8c10 0x7f0bd40c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.688+0000 7f0bda800640 1 --2- 172.21.15.39:0/1788397035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb020 0x7f0bd415ac60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.688+0000 7f0bda800640 1 -- 172.21.15.39:0/1788397035 >> 172.21.15.39:0/1788397035 conn(0x7f0bd406a2a0 msgr2=0x7f0bd40afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:51.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.688+0000 7f0bda800640 1 -- 172.21.15.39:0/1788397035 shutdown_connections 2024-01-24T17:42:51.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.688+0000 7f0bda800640 1 -- 172.21.15.39:0/1788397035 wait complete. 2024-01-24T17:42:51.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.690+0000 7f0bda800640 1 Processor -- start 2024-01-24T17:42:51.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.690+0000 7f0bda800640 1 -- start start 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bda800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd40c8c10 0x7f0bd4154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bda800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd40cb020 0x7f0bd415d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bda800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb400 0x7f0bd40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bda800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4157160 con 0x7f0bd40cb400 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bda800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4154ff0 con 0x7f0bd40c8c10 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bda800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4155720 con 0x7f0bd40cb020 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bd9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb400 0x7f0bd40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bd9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb400 0x7f0bd40ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40696/0 (socket says 172.21.15.39:40696) 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.691+0000 7f0bd9e00640 1 -- 172.21.15.39:0/3242136136 learned_addr learned my addr 172.21.15.39:0/3242136136 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd9400640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd40c8c10 0x7f0bd4154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd8a00640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd40cb020 0x7f0bd415d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd9e00640 1 -- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd40cb020 msgr2=0x7f0bd415d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd9e00640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd40cb020 0x7f0bd415d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd9e00640 1 -- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd40c8c10 msgr2=0x7f0bd4154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd9e00640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd40c8c10 0x7f0bd4154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd9e00640 1 -- 172.21.15.39:0/3242136136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc40094d0 con 0x7f0bd40cb400 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd9e00640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb400 0x7f0bd40ce270 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc007e40 tx=0x7f0bcc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.692+0000 7f0bd1600640 1 -- 172.21.15.39:0/3242136136 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc014620 con 0x7f0bd40cb400 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.693+0000 7f0bd1600640 1 -- 172.21.15.39:0/3242136136 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc0147a0 con 0x7f0bd40cb400 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.693+0000 7f0bd1600640 1 -- 172.21.15.39:0/3242136136 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc014b50 con 0x7f0bd40cb400 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.693+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd4155a00 con 0x7f0bd40cb400 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.693+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd4155fc0 con 0x7f0bd40cb400 2024-01-24T17:42:51.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.694+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd41564e0 con 0x7f0bd40cb400 2024-01-24T17:42:51.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.695+0000 7f0bd1600640 1 -- 172.21.15.39:0/3242136136 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0bcc014cb0 con 0x7f0bd40cb400 2024-01-24T17:42:51.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.696+0000 7f0bd1600640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bac06a430 0x7f0bac06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:51.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.696+0000 7f0bd1600640 1 -- 172.21.15.39:0/3242136136 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0bcc088e60 con 0x7f0bd40cb400 2024-01-24T17:42:51.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.696+0000 7f0bd9400640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bac06a430 0x7f0bac06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:51.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.697+0000 7f0bd9400640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bac06a430 0x7f0bac06c8f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4036000 tx=0x7f0bc403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:51.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.700+0000 7f0bd1600640 1 -- 172.21.15.39:0/3242136136 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0bd41564e0 con 0x7f0bd40cb400 2024-01-24T17:42:51.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.966+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f0bd41564e0 con 0x7f0bd40cb400 2024-01-24T17:42:51.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.975+0000 7f0bd1600640 1 -- 172.21.15.39:0/3242136136 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0bd41564e0 con 0x7f0bd40cb400 2024-01-24T17:42:51.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bac06a430 msgr2=0x7f0bac06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bac06a430 0x7f0bac06c8f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4036000 tx=0x7f0bc403c040 comp rx=0 tx=0).stop 2024-01-24T17:42:51.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb400 msgr2=0x7f0bd40ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:51.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb400 0x7f0bd40ce270 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc007e40 tx=0x7f0bcc00e4b0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 shutdown_connections 2024-01-24T17:42:51.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bac06a430 0x7f0bac06c8f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd40cb400 0x7f0bd40ce270 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd40cb020 0x7f0bd415d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 --2- 172.21.15.39:0/3242136136 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd40c8c10 0x7f0bd4154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 >> 172.21.15.39:0/3242136136 conn(0x7f0bd406a2a0 msgr2=0x7f0bd40cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 shutdown_connections 2024-01-24T17:42:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:51.978+0000 7f0bda800640 1 -- 172.21.15.39:0/3242136136 wait complete. 2024-01-24T17:42:51.997 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-24T17:42:52.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.219+0000 7f7e0a400640 1 Processor -- start 2024-01-24T17:42:52.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.220+0000 7f7e0a400640 1 -- start start 2024-01-24T17:42:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.221+0000 7f7e0a400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e04158600 0x7f7e040cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.221+0000 7f7e0a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0415b030 con 0x7f7e040cb900 2024-01-24T17:42:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.221+0000 7f7e0a400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0415b310 con 0x7f7e040b1370 2024-01-24T17:42:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.221+0000 7f7e0a400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0415bcb0 con 0x7f7e04158600 2024-01-24T17:42:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.222+0000 7f7e03e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7e040cb900 0x7f7e0415e580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47008/0 (socket says 172.21.15.39:47008) 2024-01-24T17:42:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.222+0000 7f7e03e00640 1 -- 172.21.15.39:0/2275212664 learned_addr learned my addr 172.21.15.39:0/2275212664 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:52.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.222+0000 7f7e03400640 1 --2- 172.21.15.39:0/2275212664 >> v2:172.21.15.100:3300/0 conn(0x7f7e04158600 0x7f7e040cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.222+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103788600 0 0) 0x7f7e0415b030 con 0x7f7e040cb900 2024-01-24T17:42:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.222+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de4003520 con 0x7f7e040cb900 2024-01-24T17:42:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.222+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 295098369 0 0) 0x7f7de4003520 con 0x7f7e040cb900 2024-01-24T17:42:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.222+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 >> v2:172.21.15.100:3300/0 conn(0x7f7e04158600 msgr2=0x7f7e040cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 --2- 172.21.15.39:0/2275212664 >> v2:172.21.15.100:3300/0 conn(0x7f7e04158600 0x7f7e040cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 >> v1:172.21.15.100:6789/0 conn(0x7f7e040b1370 legacy=0x7f7e040b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0415b5f0 con 0x7f7e040cb900 2024-01-24T17:42:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7dfc0019f0 con 0x7f7e040cb900 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 >> v1:172.21.15.39:6789/0 conn(0x7f7e040cb900 legacy=0x7f7e0415e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7de4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de40047c0 0x7f7de4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7de40052a0 0x7f7de400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0415bcb0 con 0x7f7de4003970 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de400c170 con 0x7f7de40052a0 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de400c330 con 0x7f7de40047c0 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e03400640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7de4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e03400640 1 -- 172.21.15.39:0/2275212664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de40047c0 msgr2=0x7f7de4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7dfbe00640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de40047c0 0x7f7de4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e03400640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de40047c0 0x7f7de4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e03400640 1 -- 172.21.15.39:0/2275212664 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7de40052a0 msgr2=0x7f7de400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e03400640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7de40052a0 0x7f7de400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e03400640 1 -- 172.21.15.39:0/2275212664 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de4003520 con 0x7f7de4003970 2024-01-24T17:42:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7e03e00640 1 -- 172.21.15.39:0/2275212664 reap_dead start 2024-01-24T17:42:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.223+0000 7f7dfbe00640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7de40047c0 0x7f7de4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:52.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.225+0000 7f7e03400640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7de4003d90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009a80 tx=0x7f7de40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1451e339d5d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:52.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.225+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4005990 con 0x7f7de4003970 2024-01-24T17:42:52.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.225+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df4005af0 con 0x7f7de4003970 2024-01-24T17:42:52.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.225+0000 7f7e02a00640 1 -- 172.21.15.39:0/2275212664 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4005ea0 con 0x7f7de4003970 2024-01-24T17:42:52.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.226+0000 7f7e0a400640 1 -- 172.21.15.39:0/2275212664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 msgr2=0x7f7de4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.226+0000 7f7e0a400640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7de4003d90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009a80 tx=0x7f7de40057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.226+0000 7f7e0a400640 1 -- 172.21.15.39:0/2275212664 shutdown_connections 2024-01-24T17:42:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.226+0000 7f7e0a400640 1 --2- 172.21.15.39:0/2275212664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7de4003d90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.226+0000 7f7e0a400640 1 -- 172.21.15.39:0/2275212664 >> 172.21.15.39:0/2275212664 conn(0x7f7e0406a090 msgr2=0x7f7e0406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.226+0000 7f7e0a400640 1 -- 172.21.15.39:0/2275212664 shutdown_connections 2024-01-24T17:42:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.226+0000 7f7e0a400640 1 -- 172.21.15.39:0/2275212664 wait complete. 2024-01-24T17:42:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.228+0000 7f7e0a400640 1 Processor -- start 2024-01-24T17:42:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.228+0000 7f7e0a400640 1 -- start start 2024-01-24T17:42:52.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e0a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7e040cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e0a400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e04151fd0 0x7f7e04152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e0a400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e040ae140 0x7f7e040ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e0a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e04158a80 con 0x7f7de4003970 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e0a400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e04158d60 con 0x7f7e04151fd0 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e0a400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e04159040 con 0x7f7e040ae140 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e03400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7e040cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e03400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7e040cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40712/0 (socket says 172.21.15.39:40712) 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e03400640 1 -- 172.21.15.39:0/141134080 learned_addr learned my addr 172.21.15.39:0/141134080 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7e03e00640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e040ae140 0x7f7e040ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.229+0000 7f7dfbe00640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e04151fd0 0x7f7e04152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7e03400640 1 -- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e040ae140 msgr2=0x7f7e040ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7e03400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e040ae140 0x7f7e040ae5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7e03400640 1 -- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e04151fd0 msgr2=0x7f7e04152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7e03400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e04151fd0 0x7f7e04152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7e03400640 1 -- 172.21.15.39:0/141134080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04159320 con 0x7f7de4003970 2024-01-24T17:42:52.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7e03400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7e040cd470 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7df4036040 tx=0x7f7df4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:52.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7dfb400640 1 -- 172.21.15.39:0/141134080 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4005e70 con 0x7f7de4003970 2024-01-24T17:42:52.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7dfb400640 1 -- 172.21.15.39:0/141134080 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df4004020 con 0x7f7de4003970 2024-01-24T17:42:52.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e04159600 con 0x7f7de4003970 2024-01-24T17:42:52.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7dfb400640 1 -- 172.21.15.39:0/141134080 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df40043d0 con 0x7f7de4003970 2024-01-24T17:42:52.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.230+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e040cd9b0 con 0x7f7de4003970 2024-01-24T17:42:52.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.232+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e040cdba0 con 0x7f7de4003970 2024-01-24T17:42:52.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.232+0000 7f7dfb400640 1 -- 172.21.15.39:0/141134080 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7df4003610 con 0x7f7de4003970 2024-01-24T17:42:52.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.233+0000 7f7dfb400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ddc06a430 0x7f7ddc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.233+0000 7f7dfb400640 1 -- 172.21.15.39:0/141134080 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7df40aa7e0 con 0x7f7de4003970 2024-01-24T17:42:52.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.234+0000 7f7dfbe00640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ddc06a430 0x7f7ddc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.235+0000 7f7dfbe00640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ddc06a430 0x7f7ddc06c8f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7e0409c420 tx=0x7f7dec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:52.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.238+0000 7f7dfb400640 1 -- 172.21.15.39:0/141134080 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7df4076120 con 0x7f7de4003970 2024-01-24T17:42:52.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.511+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f7e0404dff0 con 0x7f7de4003970 2024-01-24T17:42:52.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.518+0000 7f7dfb400640 1 -- 172.21.15.39:0/141134080 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7df4079740 con 0x7f7de4003970 2024-01-24T17:42:52.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ddc06a430 msgr2=0x7f7ddc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ddc06a430 0x7f7ddc06c8f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7e0409c420 tx=0x7f7dec009040 comp rx=0 tx=0).stop 2024-01-24T17:42:52.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 msgr2=0x7f7e040cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7e040cd470 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7df4036040 tx=0x7f7df4004730 comp rx=0 tx=0).stop 2024-01-24T17:42:52.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 shutdown_connections 2024-01-24T17:42:52.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e040ae140 0x7f7e040ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e04151fd0 0x7f7e04152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ddc06a430 0x7f7ddc06c8f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 --2- 172.21.15.39:0/141134080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7de4003970 0x7f7e040cd470 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.520+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 >> 172.21.15.39:0/141134080 conn(0x7f7e0406a090 msgr2=0x7f7e04156780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:52.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.521+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 shutdown_connections 2024-01-24T17:42:52.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.521+0000 7f7e0a400640 1 -- 172.21.15.39:0/141134080 wait complete. 2024-01-24T17:42:52.539 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-24T17:42:52.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.748+0000 7f0477400640 1 Processor -- start 2024-01-24T17:42:52.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.748+0000 7f0477400640 1 -- start start 2024-01-24T17:42:52.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.749+0000 7f0477400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f047006b630 0x7f04700b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.749+0000 7f0477400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700d2470 0x7f04700b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0477400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04700d0e50 con 0x7f047015c860 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0477400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04700d1130 con 0x7f047006b630 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0477400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04700d1410 con 0x7f04700d2470 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0476000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f047015c860 0x7f047006af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47018/0 (socket says 172.21.15.39:47018) 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0476000640 1 -- 172.21.15.39:0/1996782211 learned_addr learned my addr 172.21.15.39:0/1996782211 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0475600640 1 --2- 172.21.15.39:0/1996782211 >> v2:172.21.15.100:3300/0 conn(0x7f047006b630 0x7f04700b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0476a00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700d2470 0x7f04700b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236316732 0 0) 0x7f04700d0e50 con 0x7f047015c860 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.750+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0450003520 con 0x7f047015c860 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378487646 0 0) 0x7f0450003520 con 0x7f047015c860 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700d2470 msgr2=0x7f04700b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700d2470 0x7f04700b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 >> v2:172.21.15.100:3300/0 conn(0x7f047006b630 msgr2=0x7f04700b5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 --2- 172.21.15.39:0/1996782211 >> v2:172.21.15.100:3300/0 conn(0x7f047006b630 0x7f04700b5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04700d16f0 con 0x7f047015c860 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f0458001690 con 0x7f047015c860 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 >> v1:172.21.15.39:6789/0 conn(0x7f047015c860 legacy=0x7f047006af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450003970 0x7f0450003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700d2470 0x7f04500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0450004d00 0x7f0450005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04700d1130 con 0x7f0450004d00 2024-01-24T17:42:52.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04700d1410 con 0x7f0450003970 2024-01-24T17:42:52.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045000b980 con 0x7f04700d2470 2024-01-24T17:42:52.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.751+0000 7f0476a00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0450004d00 0x7f0450005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.752+0000 7f0475600640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700d2470 0x7f04500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.752+0000 7f0476000640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450003970 0x7f0450003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.752+0000 7f0476a00640 1 -- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700d2470 msgr2=0x7f04500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:52.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.752+0000 7f0476a00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700d2470 0x7f04500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.752+0000 7f0476a00640 1 -- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450003970 msgr2=0x7f0450003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:52.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.752+0000 7f0476a00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450003970 0x7f0450003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.752+0000 7f0476a00640 1 -- 172.21.15.39:0/1996782211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450003520 con 0x7f0450004d00 2024-01-24T17:42:52.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.752+0000 7f0476000640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450003970 0x7f0450003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:52.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.753+0000 7f0476a00640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0450004d00 0x7f0450005180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0468009af0 tx=0x7f04500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3366e20501669c78 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:52.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.753+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046802c6b0 con 0x7f0450004d00 2024-01-24T17:42:52.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.753+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046802c810 con 0x7f0450004d00 2024-01-24T17:42:52.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.753+0000 7f0474c00640 1 -- 172.21.15.39:0/1996782211 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046802cba0 con 0x7f0450004d00 2024-01-24T17:42:52.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.753+0000 7f0477400640 1 -- 172.21.15.39:0/1996782211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0450004d00 msgr2=0x7f0450005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.753+0000 7f0477400640 1 --2- 172.21.15.39:0/1996782211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0450004d00 0x7f0450005180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0468009af0 tx=0x7f04500057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.754+0000 7f0476a00640 1 -- 172.21.15.39:0/1996782211 reap_dead start 2024-01-24T17:42:52.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.754+0000 7f0477400640 1 -- 172.21.15.39:0/1996782211 shutdown_connections 2024-01-24T17:42:52.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.754+0000 7f0477400640 1 -- 172.21.15.39:0/1996782211 >> 172.21.15.39:0/1996782211 conn(0x7f04700d2a50 msgr2=0x7f04700d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:52.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.754+0000 7f0477400640 1 -- 172.21.15.39:0/1996782211 shutdown_connections 2024-01-24T17:42:52.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.754+0000 7f0477400640 1 -- 172.21.15.39:0/1996782211 wait complete. 2024-01-24T17:42:52.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.755+0000 7f0477400640 1 Processor -- start 2024-01-24T17:42:52.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.756+0000 7f0477400640 1 -- start start 2024-01-24T17:42:52.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.756+0000 7f0477400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450004d00 0x7f0470150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.756+0000 7f0477400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700b3b70 0x7f047015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0477400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04700b3ff0 0x7f047015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0477400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047015d780 con 0x7f04700b3ff0 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0477400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047015fb60 con 0x7f0450004d00 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0477400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047015d0c0 con 0x7f04700b3b70 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04700b3ff0 0x7f047015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04700b3ff0 0x7f047015b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40718/0 (socket says 172.21.15.39:40718) 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476a00640 1 -- 172.21.15.39:0/2877440401 learned_addr learned my addr 172.21.15.39:0/2877440401 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0475600640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700b3b70 0x7f047015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476000640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450004d00 0x7f0470150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476a00640 1 -- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700b3b70 msgr2=0x7f047015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476a00640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700b3b70 0x7f047015f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476a00640 1 -- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450004d00 msgr2=0x7f0470150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476a00640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450004d00 0x7f0470150e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.757+0000 7f0476a00640 1 -- 172.21.15.39:0/2877440401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047015d3a0 con 0x7f04700b3ff0 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.758+0000 7f0476a00640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04700b3ff0 0x7f047015b310 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0468009af0 tx=0x7f0468004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.758+0000 7f046e000640 1 -- 172.21.15.39:0/2877440401 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046802cd90 con 0x7f04700b3ff0 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.758+0000 7f046e000640 1 -- 172.21.15.39:0/2877440401 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0468003610 con 0x7f04700b3ff0 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.758+0000 7f046e000640 1 -- 172.21.15.39:0/2877440401 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04680039c0 con 0x7f04700b3ff0 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.758+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04700cbb50 con 0x7f04700b3ff0 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.758+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047015f5e0 con 0x7f04700b3ff0 2024-01-24T17:42:52.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.759+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047015d530 con 0x7f04700b3ff0 2024-01-24T17:42:52.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.760+0000 7f046e000640 1 -- 172.21.15.39:0/2877440401 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0468003b20 con 0x7f04700b3ff0 2024-01-24T17:42:52.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.761+0000 7f046e000640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f044806a460 0x7f044806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:52.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.761+0000 7f046e000640 1 -- 172.21.15.39:0/2877440401 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f04680a9580 con 0x7f04700b3ff0 2024-01-24T17:42:52.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.761+0000 7f0476000640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f044806a460 0x7f044806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:52.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.762+0000 7f0476000640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f044806a460 0x7f044806c920 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f04700d2920 tx=0x7f0458009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:52.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:52.765+0000 7f046e000640 1 -- 172.21.15.39:0/2877440401 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0468074e20 con 0x7f04700b3ff0 2024-01-24T17:42:53.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.029+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f04700c9890 con 0x7f04700b3ff0 2024-01-24T17:42:53.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.036+0000 7f046e000640 1 -- 172.21.15.39:0/2877440401 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0468078440 con 0x7f04700b3ff0 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f044806a460 msgr2=0x7f044806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f044806a460 0x7f044806c920 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f04700d2920 tx=0x7f0458009040 comp rx=0 tx=0).stop 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04700b3ff0 msgr2=0x7f047015b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04700b3ff0 0x7f047015b310 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0468009af0 tx=0x7f0468004730 comp rx=0 tx=0).stop 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 shutdown_connections 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f044806a460 0x7f044806c920 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04700b3ff0 0x7f047015b310 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04700b3b70 0x7f047015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 --2- 172.21.15.39:0/2877440401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0450004d00 0x7f0470150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 >> 172.21.15.39:0/2877440401 conn(0x7f04700d2a50 msgr2=0x7f04700b8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:53.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 shutdown_connections 2024-01-24T17:42:53.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.039+0000 7f0477400640 1 -- 172.21.15.39:0/2877440401 wait complete. 2024-01-24T17:42:53.059 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:53.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.257+0000 7f121be00640 1 Processor -- start 2024-01-24T17:42:53.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.257+0000 7f121be00640 1 -- start start 2024-01-24T17:42:53.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f121c15ab00 0x7f121c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f121c0ccd20 0x7f121c0ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121c0ce400 con 0x7f121c15a720 2024-01-24T17:42:53.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121c15b3b0 con 0x7f121c0ccd20 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121be00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121c15b690 con 0x7f121c15ab00 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f121c0ccd20 0x7f121c0ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f121c0ccd20 0x7f121c0ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34730/0 (socket says 172.21.15.39:34730) 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121aa00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f121c15a720 0x7f121c0cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40070/0 (socket says 172.21.15.39:40070) 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121b400640 1 -- 172.21.15.39:0/3982008342 learned_addr learned my addr 172.21.15.39:0/3982008342 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121a000640 1 --2- 172.21.15.39:0/3982008342 >> v2:172.21.15.100:3300/0 conn(0x7f121c15ab00 0x7f121c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121b400640 1 -- 172.21.15.39:0/3982008342 >> v2:172.21.15.100:3300/0 conn(0x7f121c15ab00 msgr2=0x7f121c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121b400640 1 --2- 172.21.15.39:0/3982008342 >> v2:172.21.15.100:3300/0 conn(0x7f121c15ab00 0x7f121c0cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.259+0000 7f121b400640 1 -- 172.21.15.39:0/3982008342 >> v1:172.21.15.39:6789/0 conn(0x7f121c15a720 legacy=0x7f121c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.260+0000 7f1219600640 1 -- 172.21.15.39:0/3982008342 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066403311 0 0) 0x7f121c0ce400 con 0x7f121c15a720 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.260+0000 7f121b400640 1 -- 172.21.15.39:0/3982008342 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c15b970 con 0x7f121c0ccd20 2024-01-24T17:42:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.260+0000 7f121a000640 1 --2- 172.21.15.39:0/3982008342 >> v2:172.21.15.100:3300/0 conn(0x7f121c15ab00 0x7f121c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f121b400640 1 --2- 172.21.15.39:0/3982008342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c0ccd20 0x7f121c0ce8e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f121c071d80 tx=0x7f121402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48cf6c703073f9a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f1219600640 1 -- 172.21.15.39:0/3982008342 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121402ca80 con 0x7f121c0ccd20 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f1219600640 1 -- 172.21.15.39:0/3982008342 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121402cbe0 con 0x7f121c0ccd20 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f1219600640 1 -- 172.21.15.39:0/3982008342 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12140343f0 con 0x7f121c0ccd20 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f121be00640 1 -- 172.21.15.39:0/3982008342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c0ccd20 msgr2=0x7f121c0ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f121be00640 1 --2- 172.21.15.39:0/3982008342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c0ccd20 0x7f121c0ce8e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f121c071d80 tx=0x7f121402bee0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f121be00640 1 -- 172.21.15.39:0/3982008342 shutdown_connections 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f121be00640 1 --2- 172.21.15.39:0/3982008342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c0ccd20 0x7f121c0ce8e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f121be00640 1 --2- 172.21.15.39:0/3982008342 >> v2:172.21.15.100:3300/0 conn(0x7f121c15ab00 0x7f121c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.261+0000 7f121be00640 1 -- 172.21.15.39:0/3982008342 >> 172.21.15.39:0/3982008342 conn(0x7f121c06a090 msgr2=0x7f121c06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.262+0000 7f121be00640 1 -- 172.21.15.39:0/3982008342 shutdown_connections 2024-01-24T17:42:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.262+0000 7f121be00640 1 -- 172.21.15.39:0/3982008342 wait complete. 2024-01-24T17:42:53.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.263+0000 7f121be00640 1 Processor -- start 2024-01-24T17:42:53.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.263+0000 7f121be00640 1 -- start start 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.264+0000 7f121be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f121c15a720 0x7f121c0acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.264+0000 7f121be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c15ab00 0x7f121c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.264+0000 7f121be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f121c0c9f50 0x7f121c1f1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.264+0000 7f121be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c0c48c0 con 0x7f121c15ab00 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.264+0000 7f121be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c157480 con 0x7f121c0c9f50 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.264+0000 7f121be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121c0c8f40 con 0x7f121c15a720 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.264+0000 7f121a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c15ab00 0x7f121c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c15ab00 0x7f121c0c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34744/0 (socket says 172.21.15.39:34744) 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121a000640 1 -- 172.21.15.39:0/260383722 learned_addr learned my addr 172.21.15.39:0/260383722 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121aa00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f121c15a720 0x7f121c0acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121b400640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f121c0c9f50 0x7f121c1f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121a000640 1 -- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f121c15a720 msgr2=0x7f121c0acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121a000640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f121c15a720 0x7f121c0acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121a000640 1 -- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f121c0c9f50 msgr2=0x7f121c1f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121a000640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f121c0c9f50 0x7f121c1f1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121a000640 1 -- 172.21.15.39:0/260383722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1214009b30 con 0x7f121c15ab00 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121b400640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f121c0c9f50 0x7f121c1f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121aa00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f121c15a720 0x7f121c0acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f121a000640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c15ab00 0x7f121c0c9a10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f120400c7f0 tx=0x7f120400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.265+0000 7f1212a00640 1 -- 172.21.15.39:0/260383722 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204007a30 con 0x7f121c15ab00 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.266+0000 7f1212a00640 1 -- 172.21.15.39:0/260383722 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1204007b90 con 0x7f121c15ab00 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.266+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121c0c3f90 con 0x7f121c15ab00 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.266+0000 7f1212a00640 1 -- 172.21.15.39:0/260383722 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120400ce40 con 0x7f121c15ab00 2024-01-24T17:42:53.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.266+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121c0c4550 con 0x7f121c15ab00 2024-01-24T17:42:53.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.267+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121c04dff0 con 0x7f121c15ab00 2024-01-24T17:42:53.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.268+0000 7f1212a00640 1 -- 172.21.15.39:0/260383722 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f12040113f0 con 0x7f121c15ab00 2024-01-24T17:42:53.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.268+0000 7f1212a00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f11f406a430 0x7f11f406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.269+0000 7f1212a00640 1 -- 172.21.15.39:0/260383722 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1204088770 con 0x7f121c15ab00 2024-01-24T17:42:53.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.269+0000 7f121aa00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f11f406a430 0x7f11f406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.270+0000 7f121aa00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f11f406a430 0x7f11f406c8f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f120c000f60 tx=0x7f120c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:53.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.273+0000 7f1212a00640 1 -- 172.21.15.39:0/260383722 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f12040540a0 con 0x7f121c15ab00 2024-01-24T17:42:53.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.466+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f121c0b3ff0 con 0x7f121c15ab00 2024-01-24T17:42:53.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.468+0000 7f1212a00640 1 -- 172.21.15.39:0/260383722 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f12040576c0 con 0x7f121c15ab00 2024-01-24T17:42:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:53.469 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":30,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:49.277681+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:53.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f11f406a430 msgr2=0x7f11f406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f11f406a430 0x7f11f406c8f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f120c000f60 tx=0x7f120c009040 comp rx=0 tx=0).stop 2024-01-24T17:42:53.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c15ab00 msgr2=0x7f121c0c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c15ab00 0x7f121c0c9a10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f120400c7f0 tx=0x7f120400cbf0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 shutdown_connections 2024-01-24T17:42:53.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f121c0c9f50 0x7f121c1f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f11f406a430 0x7f11f406c8f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f121c15ab00 0x7f121c0c9a10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 --2- 172.21.15.39:0/260383722 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f121c15a720 0x7f121c0acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 >> 172.21.15.39:0/260383722 conn(0x7f121c06a090 msgr2=0x7f121c0cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:53.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.470+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 shutdown_connections 2024-01-24T17:42:53.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.471+0000 7f121be00640 1 -- 172.21.15.39:0/260383722 wait complete. 2024-01-24T17:42:53.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:42:53.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:42:53.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.707+0000 7f0e3be00640 1 Processor -- start 2024-01-24T17:42:53.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.707+0000 7f0e3be00640 1 -- start start 2024-01-24T17:42:53.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.709+0000 7f0e3be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c15ccb0 0x7f0e3c0cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.709+0000 7f0e3be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0e3c0c9d60 0x7f0e3c0ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.709+0000 7f0e3be00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3c15d090 con 0x7f0e3c0ce390 2024-01-24T17:42:53.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.709+0000 7f0e3be00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3c0cac20 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.709+0000 7f0e3be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c06c540 con 0x7f0e3c15ccb0 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.709+0000 7f0e3b400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0e3c0c9d60 0x7f0e3c0ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.709+0000 7f0e3aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c15ccb0 0x7f0e3c0cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.710+0000 7f0e3aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c15ccb0 0x7f0e3c0cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:57114/0 (socket says 172.21.15.39:57114) 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.710+0000 7f0e3aa00640 1 -- 172.21.15.39:0/3605063120 learned_addr learned my addr 172.21.15.39:0/3605063120 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.710+0000 7f0e3aa00640 1 -- 172.21.15.39:0/3605063120 >> v2:172.21.15.100:3300/0 conn(0x7f0e3c0c9d60 msgr2=0x7f0e3c0ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.710+0000 7f0e3aa00640 1 --2- 172.21.15.39:0/3605063120 >> v2:172.21.15.100:3300/0 conn(0x7f0e3c0c9d60 0x7f0e3c0ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.710+0000 7f0e3aa00640 1 -- 172.21.15.39:0/3605063120 >> v1:172.21.15.100:6789/0 conn(0x7f0e3c0ce390 legacy=0x7f0e3c0cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.710+0000 7f0e3aa00640 1 -- 172.21.15.39:0/3605063120 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c0ced30 con 0x7f0e3c15ccb0 2024-01-24T17:42:53.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.710+0000 7f0e3a000640 1 -- 172.21.15.39:0/3605063120 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577046210 0 0) 0x7f0e3c15d090 con 0x7f0e3c0ce390 2024-01-24T17:42:53.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.711+0000 7f0e3aa00640 1 --2- 172.21.15.39:0/3605063120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c15ccb0 0x7f0e3c0cda90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009ea0 tx=0x7f0e2c02bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afc746a5c8c7335 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:53.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.712+0000 7f0e3a000640 1 -- 172.21.15.39:0/3605063120 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c02cb40 con 0x7f0e3c15ccb0 2024-01-24T17:42:53.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.712+0000 7f0e3a000640 1 -- 172.21.15.39:0/3605063120 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c02cca0 con 0x7f0e3c15ccb0 2024-01-24T17:42:53.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.712+0000 7f0e3a000640 1 -- 172.21.15.39:0/3605063120 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c034610 con 0x7f0e3c15ccb0 2024-01-24T17:42:53.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.712+0000 7f0e3be00640 1 -- 172.21.15.39:0/3605063120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c15ccb0 msgr2=0x7f0e3c0cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.712+0000 7f0e3be00640 1 --2- 172.21.15.39:0/3605063120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c15ccb0 0x7f0e3c0cda90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009ea0 tx=0x7f0e2c02bfa0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.713+0000 7f0e3be00640 1 -- 172.21.15.39:0/3605063120 shutdown_connections 2024-01-24T17:42:53.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.713+0000 7f0e3be00640 1 --2- 172.21.15.39:0/3605063120 >> v2:172.21.15.100:3300/0 conn(0x7f0e3c0c9d60 0x7f0e3c0ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.713+0000 7f0e3be00640 1 --2- 172.21.15.39:0/3605063120 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c15ccb0 0x7f0e3c0cda90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.713+0000 7f0e3be00640 1 -- 172.21.15.39:0/3605063120 >> 172.21.15.39:0/3605063120 conn(0x7f0e3c069e50 msgr2=0x7f0e3c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:53.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.713+0000 7f0e3be00640 1 -- 172.21.15.39:0/3605063120 shutdown_connections 2024-01-24T17:42:53.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.713+0000 7f0e3be00640 1 -- 172.21.15.39:0/3605063120 wait complete. 2024-01-24T17:42:53.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.714+0000 7f0e3be00640 1 Processor -- start 2024-01-24T17:42:53.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.715+0000 7f0e3be00640 1 -- start start 2024-01-24T17:42:53.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.715+0000 7f0e3be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e3c0c9d60 0x7f0e3c097a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.715+0000 7f0e3be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c0ce390 0x7f0e3c1f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e3c15ccb0 0x7f0e3c0ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c09aa30 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c1f5980 con 0x7f0e3c15ccb0 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c09a6b0 con 0x7f0e3c0ce390 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e3c0c9d60 0x7f0e3c097a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e3c0c9d60 0x7f0e3c097a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34754/0 (socket says 172.21.15.39:34754) 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3aa00640 1 -- 172.21.15.39:0/2644296647 learned_addr learned my addr 172.21.15.39:0/2644296647 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3b400640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e3c15ccb0 0x7f0e3c0ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3aa00640 1 -- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c0ce390 msgr2=0x7f0e3c1f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3aa00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c0ce390 0x7f0e3c1f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3aa00640 1 -- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e3c15ccb0 msgr2=0x7f0e3c0ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3aa00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e3c15ccb0 0x7f0e3c0ae990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.716+0000 7f0e3aa00640 1 -- 172.21.15.39:0/2644296647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c009b30 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.717+0000 7f0e3aa00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e3c0c9d60 0x7f0e3c097a60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c036040 tx=0x7f0e2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:53.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.717+0000 7f0e33400640 1 -- 172.21.15.39:0/2644296647 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c040670 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.717+0000 7f0e33400640 1 -- 172.21.15.39:0/2644296647 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c0407d0 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.717+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c1f5bb0 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.717+0000 7f0e33400640 1 -- 172.21.15.39:0/2644296647 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c040b80 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.717+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c1f6170 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.718+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c09ab70 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.720+0000 7f0e33400640 1 -- 172.21.15.39:0/2644296647 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0e2c040ce0 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.720+0000 7f0e33400640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0e1406a480 0x7f0e1406c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:53.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.721+0000 7f0e33400640 1 -- 172.21.15.39:0/2644296647 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e2c0ab500 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.721+0000 7f0e31e00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0e1406a480 0x7f0e1406c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:53.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.722+0000 7f0e31e00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0e1406a480 0x7f0e1406c940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0e24001600 tx=0x7f0e24007910 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:53.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.725+0000 7f0e33400640 1 -- 172.21.15.39:0/2644296647 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0e3c09ab70 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.917+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e3c09ab70 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.917+0000 7f0e33400640 1 -- 172.21.15.39:0/2644296647 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e2c076ff0 con 0x7f0e3c0c9d60 2024-01-24T17:42:53.918 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:42:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0e1406a480 msgr2=0x7f0e1406c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0e1406a480 0x7f0e1406c940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0e24001600 tx=0x7f0e24007910 comp rx=0 tx=0).stop 2024-01-24T17:42:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e3c0c9d60 msgr2=0x7f0e3c097a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:53.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e3c0c9d60 0x7f0e3c097a60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c036040 tx=0x7f0e2c004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 shutdown_connections 2024-01-24T17:42:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e3c15ccb0 0x7f0e3c0ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e3c0ce390 0x7f0e3c1f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0e1406a480 0x7f0e1406c940 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 --2- 172.21.15.39:0/2644296647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e3c0c9d60 0x7f0e3c097a60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 >> 172.21.15.39:0/2644296647 conn(0x7f0e3c069e50 msgr2=0x7f0e3c15bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 shutdown_connections 2024-01-24T17:42:53.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:53.920+0000 7f0e3be00640 1 -- 172.21.15.39:0/2644296647 wait complete. 2024-01-24T17:42:53.941 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T17:42:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.159+0000 7ff069c00640 1 Processor -- start 2024-01-24T17:42:54.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.160+0000 7ff069c00640 1 -- start start 2024-01-24T17:42:54.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.161+0000 7ff069c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff0640cf990 0x7ff0640cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.161+0000 7ff069c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff06415fb90 0x7ff06415d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.161+0000 7ff069c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0640d02b0 con 0x7ff06415d020 2024-01-24T17:42:54.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.161+0000 7ff069c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06415e080 con 0x7ff0640cf990 2024-01-24T17:42:54.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.161+0000 7ff069c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06415dac0 con 0x7ff06415fb90 2024-01-24T17:42:54.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.161+0000 7ff05be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff06415d020 0x7ff064160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40080/0 (socket says 172.21.15.39:40080) 2024-01-24T17:42:54.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.161+0000 7ff05be00640 1 -- 172.21.15.39:0/4202558468 learned_addr learned my addr 172.21.15.39:0/4202558468 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:54.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062a00640 1 --2- 172.21.15.39:0/4202558468 >> v2:172.21.15.100:3300/0 conn(0x7ff0640cf990 0x7ff0640cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff063400640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff06415fb90 0x7ff06415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153234140 0 0) 0x7ff0640d02b0 con 0x7ff06415d020 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff044003520 con 0x7ff06415d020 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4258420162 0 0) 0x7ff044003520 con 0x7ff06415d020 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff06415fb90 msgr2=0x7ff06415d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff06415fb90 0x7ff06415d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 >> v2:172.21.15.100:3300/0 conn(0x7ff0640cf990 msgr2=0x7ff0640cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 --2- 172.21.15.39:0/4202558468 >> v2:172.21.15.100:3300/0 conn(0x7ff0640cf990 0x7ff0640cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06415dda0 con 0x7ff06415d020 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff04c001670 con 0x7ff06415d020 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 >> v1:172.21.15.39:6789/0 conn(0x7ff06415d020 legacy=0x7ff064160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff044003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0440047c0 0x7ff044004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff044005180 0x7ff04400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06415e080 con 0x7ff044003970 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06415dac0 con 0x7ff044005180 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.162+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04400bef0 con 0x7ff0440047c0 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff063400640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff06415fb90 0x7ff06415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff062a00640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff044003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff05be00640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0440047c0 0x7ff044004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff062a00640 1 -- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0440047c0 msgr2=0x7ff044004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff062a00640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0440047c0 0x7ff044004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff062a00640 1 -- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff044005180 msgr2=0x7ff04400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff063400640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff044005180 0x7ff04400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff062a00640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff044005180 0x7ff04400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.163+0000 7ff062a00640 1 -- 172.21.15.39:0/4202558468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044003520 con 0x7ff044003970 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.164+0000 7ff062a00640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff044003d90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff054009ac0 tx=0x7ff0440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e24f6c6a123650 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05402c710 con 0x7ff044003970 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05402c870 con 0x7ff044003970 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff062000640 1 -- 172.21.15.39:0/4202558468 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05402cc00 con 0x7ff044003970 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff069c00640 1 -- 172.21.15.39:0/4202558468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 msgr2=0x7ff044003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff069c00640 1 --2- 172.21.15.39:0/4202558468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff044003d90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff054009ac0 tx=0x7ff0440057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff063400640 1 -- 172.21.15.39:0/4202558468 reap_dead start 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff069c00640 1 -- 172.21.15.39:0/4202558468 shutdown_connections 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff069c00640 1 -- 172.21.15.39:0/4202558468 >> 172.21.15.39:0/4202558468 conn(0x7ff064071790 msgr2=0x7ff064071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff069c00640 1 -- 172.21.15.39:0/4202558468 shutdown_connections 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.165+0000 7ff069c00640 1 -- 172.21.15.39:0/4202558468 wait complete. 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.167+0000 7ff069c00640 1 Processor -- start 2024-01-24T17:42:54.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.167+0000 7ff069c00640 1 -- start start 2024-01-24T17:42:54.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.168+0000 7ff069c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff064148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.168+0000 7ff069c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff06414b6b0 0x7ff06414bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.168+0000 7ff069c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0640c9ad0 0x7ff0640c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.168+0000 7ff069c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640cc950 con 0x7ff044003970 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.168+0000 7ff069c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640ca4c0 con 0x7ff06414b6b0 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.168+0000 7ff069c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640cba10 con 0x7ff0640c9ad0 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.168+0000 7ff062a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff064148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff062a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff064148c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34774/0 (socket says 172.21.15.39:34774) 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff062a00640 1 -- 172.21.15.39:0/1981108232 learned_addr learned my addr 172.21.15.39:0/1981108232 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff05be00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff06414b6b0 0x7ff06414bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff063400640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0640c9ad0 0x7ff0640c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff062a00640 1 -- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0640c9ad0 msgr2=0x7ff0640c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff062a00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0640c9ad0 0x7ff0640c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff062a00640 1 -- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff06414b6b0 msgr2=0x7ff06414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff062a00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff06414b6b0 0x7ff06414bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff062a00640 1 -- 172.21.15.39:0/1981108232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640cbcd0 con 0x7ff044003970 2024-01-24T17:42:54.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff063400640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0640c9ad0 0x7ff0640c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:54.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.169+0000 7ff062a00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff064148c60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff054037040 tx=0x7ff054004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:54.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.170+0000 7ff05aa00640 1 -- 172.21.15.39:0/1981108232 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054003810 con 0x7ff044003970 2024-01-24T17:42:54.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.170+0000 7ff05aa00640 1 -- 172.21.15.39:0/1981108232 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff054003970 con 0x7ff044003970 2024-01-24T17:42:54.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.170+0000 7ff05aa00640 1 -- 172.21.15.39:0/1981108232 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054003d00 con 0x7ff044003970 2024-01-24T17:42:54.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.170+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0640cbfb0 con 0x7ff044003970 2024-01-24T17:42:54.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.170+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0640cc570 con 0x7ff044003970 2024-01-24T17:42:54.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.171+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff064073ef0 con 0x7ff044003970 2024-01-24T17:42:54.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.172+0000 7ff05aa00640 1 -- 172.21.15.39:0/1981108232 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff054040450 con 0x7ff044003970 2024-01-24T17:42:54.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.173+0000 7ff05aa00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff03c06a460 0x7ff03c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.173+0000 7ff05aa00640 1 -- 172.21.15.39:0/1981108232 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff0540aa410 con 0x7ff044003970 2024-01-24T17:42:54.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.173+0000 7ff05be00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff03c06a460 0x7ff03c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.174+0000 7ff05be00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff03c06a460 0x7ff03c06c920 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff06415cc20 tx=0x7ff04c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:54.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.177+0000 7ff05aa00640 1 -- 172.21.15.39:0/1981108232 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff054075d50 con 0x7ff044003970 2024-01-24T17:42:54.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.373+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff0640b2900 con 0x7ff044003970 2024-01-24T17:42:54.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.373+0000 7ff05aa00640 1 -- 172.21.15.39:0/1981108232 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff054003e60 con 0x7ff044003970 2024-01-24T17:42:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:54.374 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-01-24T17:42:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:42:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-01-24T17:42:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-01-24T17:42:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:42:54.375 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-01-24T17:42:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff03c06a460 msgr2=0x7ff03c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff03c06a460 0x7ff03c06c920 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff06415cc20 tx=0x7ff04c009040 comp rx=0 tx=0).stop 2024-01-24T17:42:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 msgr2=0x7ff064148c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff064148c60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff054037040 tx=0x7ff054004730 comp rx=0 tx=0).stop 2024-01-24T17:42:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 shutdown_connections 2024-01-24T17:42:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0640c9ad0 0x7ff0640c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff06414b6b0 0x7ff06414bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff03c06a460 0x7ff03c06c920 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 --2- 172.21.15.39:0/1981108232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff044003970 0x7ff064148c60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 >> 172.21.15.39:0/1981108232 conn(0x7ff064071790 msgr2=0x7ff0640c5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 shutdown_connections 2024-01-24T17:42:54.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.376+0000 7ff069c00640 1 -- 172.21.15.39:0/1981108232 wait complete. 2024-01-24T17:42:54.397 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T17:42:54.397 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T17:42:54.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.621+0000 7fa7adc00640 1 Processor -- start 2024-01-24T17:42:54.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.621+0000 7fa7adc00640 1 -- start start 2024-01-24T17:42:54.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.623+0000 7fa7adc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 0x7fa7a8070f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.623+0000 7fa7adc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa7a806bec0 0x7fa7a806c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.623+0000 7fa7adc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a80c8a80 con 0x7fa7a8158ea0 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.623+0000 7fa7adc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a80714b0 con 0x7fa7a806bec0 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.623+0000 7fa7adc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a806c870 con 0x7fa7a8071cb0 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.623+0000 7fa7ad200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa7a806bec0 0x7fa7a806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.623+0000 7fa7a7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa7a8158ea0 0x7fa7a80c97b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40092/0 (socket says 172.21.15.39:40092) 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.624+0000 7fa7a7e00640 1 -- 172.21.15.39:0/2038306327 learned_addr learned my addr 172.21.15.39:0/2038306327 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.624+0000 7fa7ad200640 1 -- 172.21.15.39:0/2038306327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 msgr2=0x7fa7a8070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.624+0000 7fa79fe00640 1 --2- 172.21.15.39:0/2038306327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 0x7fa7a8070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.624+0000 7fa7ad200640 1 --2- 172.21.15.39:0/2038306327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 0x7fa7a8070f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.624+0000 7fa7ad200640 1 -- 172.21.15.39:0/2038306327 >> v1:172.21.15.39:6789/0 conn(0x7fa7a8158ea0 legacy=0x7fa7a80c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.624+0000 7fa7ad200640 1 -- 172.21.15.39:0/2038306327 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8072110 con 0x7fa7a806bec0 2024-01-24T17:42:54.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.624+0000 7fa79fe00640 1 --2- 172.21.15.39:0/2038306327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 0x7fa7a8070f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:54.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.624+0000 7fa7a7400640 1 -- 172.21.15.39:0/2038306327 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618455224 0 0) 0x7fa7a80c8a80 con 0x7fa7a8158ea0 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.625+0000 7fa7ad200640 1 --2- 172.21.15.39:0/2038306327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a806bec0 0x7fa7a806c330 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009d90 tx=0x7fa7a002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7844f42a0d0a9be server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.625+0000 7fa7a7400640 1 -- 172.21.15.39:0/2038306327 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a002cbd0 con 0x7fa7a806bec0 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.625+0000 7fa7a7400640 1 -- 172.21.15.39:0/2038306327 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a002cd30 con 0x7fa7a806bec0 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.625+0000 7fa7a7400640 1 -- 172.21.15.39:0/2038306327 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a002cea0 con 0x7fa7a806bec0 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.626+0000 7fa7adc00640 1 -- 172.21.15.39:0/2038306327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a806bec0 msgr2=0x7fa7a806c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.626+0000 7fa7adc00640 1 --2- 172.21.15.39:0/2038306327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a806bec0 0x7fa7a806c330 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009d90 tx=0x7fa7a002be20 comp rx=0 tx=0).stop 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.626+0000 7fa7adc00640 1 -- 172.21.15.39:0/2038306327 shutdown_connections 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.626+0000 7fa7adc00640 1 --2- 172.21.15.39:0/2038306327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a806bec0 0x7fa7a806c330 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.626+0000 7fa7adc00640 1 --2- 172.21.15.39:0/2038306327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 0x7fa7a8070f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.626+0000 7fa7adc00640 1 -- 172.21.15.39:0/2038306327 >> 172.21.15.39:0/2038306327 conn(0x7fa7a80caeb0 msgr2=0x7fa7a80cb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:54.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.626+0000 7fa7adc00640 1 -- 172.21.15.39:0/2038306327 shutdown_connections 2024-01-24T17:42:54.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.626+0000 7fa7adc00640 1 -- 172.21.15.39:0/2038306327 wait complete. 2024-01-24T17:42:54.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.628+0000 7fa7adc00640 1 Processor -- start 2024-01-24T17:42:54.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.628+0000 7fa7adc00640 1 -- start start 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7adc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7a806bec0 0x7fa7a81478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7adc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 0x7fa7a80cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7adc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a8158ea0 0x7fa7a80cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7adc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a8167470 con 0x7fa7a8158ea0 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7adc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a80ce390 con 0x7fa7a806bec0 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7adc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a80ce670 con 0x7fa7a8071cb0 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7ad200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a8158ea0 0x7fa7a80cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7ad200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a8158ea0 0x7fa7a80cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34802/0 (socket says 172.21.15.39:34802) 2024-01-24T17:42:54.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.629+0000 7fa7ad200640 1 -- 172.21.15.39:0/809159616 learned_addr learned my addr 172.21.15.39:0/809159616 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:54.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7ad200640 1 -- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 msgr2=0x7fa7a80cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7ad200640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 0x7fa7a80cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7ad200640 1 -- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7a806bec0 msgr2=0x7fa7a81478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7ad200640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7a806bec0 0x7fa7a81478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7ad200640 1 -- 172.21.15.39:0/809159616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a0009a70 con 0x7fa7a8158ea0 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7ad200640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a8158ea0 0x7fa7a80cdd60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009d90 tx=0x7fa7a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7a4c00640 1 -- 172.21.15.39:0/809159616 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a002cea0 con 0x7fa7a8158ea0 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7a4c00640 1 -- 172.21.15.39:0/809159616 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a0003b90 con 0x7fa7a8158ea0 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7a4c00640 1 -- 172.21.15.39:0/809159616 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0004030 con 0x7fa7a8158ea0 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.630+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a80ce950 con 0x7fa7a8158ea0 2024-01-24T17:42:54.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.631+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a80cef10 con 0x7fa7a8158ea0 2024-01-24T17:42:54.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.631+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a80cf0a0 con 0x7fa7a8158ea0 2024-01-24T17:42:54.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.632+0000 7fa7a4c00640 1 -- 172.21.15.39:0/809159616 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa7a003b050 con 0x7fa7a8158ea0 2024-01-24T17:42:54.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.634+0000 7fa7a4c00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78006a430 0x7fa78006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:54.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.634+0000 7fa7a4c00640 1 -- 172.21.15.39:0/809159616 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa7a00aabd0 con 0x7fa7a8158ea0 2024-01-24T17:42:54.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.635+0000 7fa7a7e00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78006a430 0x7fa78006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:54.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.636+0000 7fa7a7e00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78006a430 0x7fa78006c8f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa798007fd0 tx=0x7fa798007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:54.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.639+0000 7fa7a4c00640 1 -- 172.21.15.39:0/809159616 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa7a0076470 con 0x7fa7a8158ea0 2024-01-24T17:42:54.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:54.833+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa7a80ca240 con 0x7fa7a8158ea0 2024-01-24T17:42:55.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.211+0000 7fa7a4c00640 1 -- 172.21.15.39:0/809159616 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa7a0079a90 con 0x7fa7a8158ea0 2024-01-24T17:42:55.213 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-01-24T17:42:55.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78006a430 msgr2=0x7fa78006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:55.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78006a430 0x7fa78006c8f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa798007fd0 tx=0x7fa798007920 comp rx=0 tx=0).stop 2024-01-24T17:42:55.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a8158ea0 msgr2=0x7fa7a80cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:55.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a8158ea0 0x7fa7a80cdd60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009d90 tx=0x7fa7a0004a40 comp rx=0 tx=0).stop 2024-01-24T17:42:55.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 shutdown_connections 2024-01-24T17:42:55.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa78006a430 0x7fa78006c8f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7a8158ea0 0x7fa7a80cdd60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7a8071cb0 0x7fa7a80cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 --2- 172.21.15.39:0/809159616 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7a806bec0 0x7fa7a81478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 >> 172.21.15.39:0/809159616 conn(0x7fa7a80caeb0 msgr2=0x7fa7a815ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:55.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.216+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 shutdown_connections 2024-01-24T17:42:55.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.217+0000 7fa7adc00640 1 -- 172.21.15.39:0/809159616 wait complete. 2024-01-24T17:42:55.240 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T17:42:55.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.460+0000 7ff7e1800640 1 Processor -- start 2024-01-24T17:42:55.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.460+0000 7ff7e1800640 1 -- start start 2024-01-24T17:42:55.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.461+0000 7ff7e1800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7dc15c9b0 0x7ff7dc0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:55.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.461+0000 7ff7e1800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff7dc15cd90 0x7ff7dc0d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:55.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.462+0000 7ff7e1800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7dc06b6f0 con 0x7ff7dc0d1350 2024-01-24T17:42:55.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.462+0000 7ff7e1800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7dc0d1780 con 0x7ff7dc15c9b0 2024-01-24T17:42:55.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.462+0000 7ff7e1800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7dc0af3a0 con 0x7ff7dc15cd90 2024-01-24T17:42:55.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.462+0000 7ff7e0e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff7dc0d1350 0x7ff7dc0cd3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40100/0 (socket says 172.21.15.39:40100) 2024-01-24T17:42:55.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.462+0000 7ff7e0e00640 1 -- 172.21.15.39:0/757092631 learned_addr learned my addr 172.21.15.39:0/757092631 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:55.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.462+0000 7ff7dbe00640 1 --2- 172.21.15.39:0/757092631 >> v2:172.21.15.39:3300/0 conn(0x7ff7dc15c9b0 0x7ff7dc0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:55.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.462+0000 7ff7d3e00640 1 --2- 172.21.15.39:0/757092631 >> v2:172.21.15.100:3300/0 conn(0x7ff7dc15cd90 0x7ff7dc0d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:55.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.463+0000 7ff7db400640 1 -- 172.21.15.39:0/757092631 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080272041 0 0) 0x7ff7dc06b6f0 con 0x7ff7dc0d1350 2024-01-24T17:42:55.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.463+0000 7ff7db400640 1 -- 172.21.15.39:0/757092631 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7bc003520 con 0x7ff7dc0d1350 2024-01-24T17:42:55.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.463+0000 7ff7dbe00640 1 -- 172.21.15.39:0/757092631 >> v2:172.21.15.100:3300/0 conn(0x7ff7dc15cd90 msgr2=0x7ff7dc0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:55.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.463+0000 7ff7dbe00640 1 --2- 172.21.15.39:0/757092631 >> v2:172.21.15.100:3300/0 conn(0x7ff7dc15cd90 0x7ff7dc0d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.463+0000 7ff7dbe00640 1 -- 172.21.15.39:0/757092631 >> v1:172.21.15.39:6789/0 conn(0x7ff7dc0d1350 legacy=0x7ff7dc0cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:55.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.463+0000 7ff7dbe00640 1 -- 172.21.15.39:0/757092631 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc0d0a60 con 0x7ff7dc15c9b0 2024-01-24T17:42:55.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.464+0000 7ff7dbe00640 1 --2- 172.21.15.39:0/757092631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc15c9b0 0x7ff7dc0cf870 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0b88e0 tx=0x7ff7cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af99d38af9ba087e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:55.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.464+0000 7ff7db400640 1 -- 172.21.15.39:0/757092631 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc02e070 con 0x7ff7dc15c9b0 2024-01-24T17:42:55.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.464+0000 7ff7db400640 1 -- 172.21.15.39:0/757092631 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc0315a0 con 0x7ff7dc15c9b0 2024-01-24T17:42:55.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.464+0000 7ff7db400640 1 -- 172.21.15.39:0/757092631 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc031930 con 0x7ff7dc15c9b0 2024-01-24T17:42:55.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.465+0000 7ff7e1800640 1 -- 172.21.15.39:0/757092631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc15c9b0 msgr2=0x7ff7dc0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:55.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.465+0000 7ff7e1800640 1 --2- 172.21.15.39:0/757092631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc15c9b0 0x7ff7dc0cf870 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0b88e0 tx=0x7ff7cc02b840 comp rx=0 tx=0).stop 2024-01-24T17:42:55.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.465+0000 7ff7e1800640 1 -- 172.21.15.39:0/757092631 shutdown_connections 2024-01-24T17:42:55.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.465+0000 7ff7e1800640 1 --2- 172.21.15.39:0/757092631 >> v2:172.21.15.100:3300/0 conn(0x7ff7dc15cd90 0x7ff7dc0d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.465+0000 7ff7e1800640 1 --2- 172.21.15.39:0/757092631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc15c9b0 0x7ff7dc0cf870 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.465+0000 7ff7e1800640 1 -- 172.21.15.39:0/757092631 >> 172.21.15.39:0/757092631 conn(0x7ff7dc0d2830 msgr2=0x7ff7dc0d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:55.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.465+0000 7ff7e1800640 1 -- 172.21.15.39:0/757092631 shutdown_connections 2024-01-24T17:42:55.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.465+0000 7ff7e1800640 1 -- 172.21.15.39:0/757092631 wait complete. 2024-01-24T17:42:55.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.467+0000 7ff7e1800640 1 Processor -- start 2024-01-24T17:42:55.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.467+0000 7ff7e1800640 1 -- start start 2024-01-24T17:42:55.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.468+0000 7ff7e1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc0d1350 0x7ff7dc0c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:55.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.468+0000 7ff7e1800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7dc15c9b0 0x7ff7dc0c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:55.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.468+0000 7ff7e1800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7dc15cd90 0x7ff7dc0c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:55.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.468+0000 7ff7e1800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc15e140 con 0x7ff7dc0d1350 2024-01-24T17:42:55.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.468+0000 7ff7e1800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc15db10 con 0x7ff7dc15c9b0 2024-01-24T17:42:55.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.468+0000 7ff7e1800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0b4450 con 0x7ff7dc15cd90 2024-01-24T17:42:55.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.468+0000 7ff7dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc0d1350 0x7ff7dc0c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:55.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc0d1350 0x7ff7dc0c8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34828/0 (socket says 172.21.15.39:34828) 2024-01-24T17:42:55.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7dbe00640 1 -- 172.21.15.39:0/3961266531 learned_addr learned my addr 172.21.15.39:0/3961266531 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:55.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7e0e00640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7dc15cd90 0x7ff7dc0c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:55.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7d3e00640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7dc15c9b0 0x7ff7dc0c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:55.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7dbe00640 1 -- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7dc15cd90 msgr2=0x7ff7dc0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7dbe00640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7dc15cd90 0x7ff7dc0c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7dbe00640 1 -- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7dc15c9b0 msgr2=0x7ff7dc0c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7dbe00640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7dc15c9b0 0x7ff7dc0c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7dbe00640 1 -- 172.21.15.39:0/3961266531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc009400 con 0x7ff7dc0d1350 2024-01-24T17:42:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.469+0000 7ff7dbe00640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc0d1350 0x7ff7dc0c8ad0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009890 tx=0x7ff7cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.470+0000 7ff7d8c00640 1 -- 172.21.15.39:0/3961266531 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc032630 con 0x7ff7dc0d1350 2024-01-24T17:42:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.470+0000 7ff7d8c00640 1 -- 172.21.15.39:0/3961266531 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7cc032790 con 0x7ff7dc0d1350 2024-01-24T17:42:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.470+0000 7ff7d8c00640 1 -- 172.21.15.39:0/3961266531 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc032b40 con 0x7ff7dc0d1350 2024-01-24T17:42:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.470+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7dc15ddd0 con 0x7ff7dc0d1350 2024-01-24T17:42:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.470+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7dc0b3670 con 0x7ff7dc0d1350 2024-01-24T17:42:55.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.471+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7dc04dff0 con 0x7ff7dc0d1350 2024-01-24T17:42:55.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.473+0000 7ff7d8c00640 1 -- 172.21.15.39:0/3961266531 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff7cc04d3f0 con 0x7ff7dc0d1350 2024-01-24T17:42:55.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.474+0000 7ff7d8c00640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff7b406a4b0 0x7ff7b406c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:55.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.474+0000 7ff7d8c00640 1 -- 172.21.15.39:0/3961266531 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff7cc0aa360 con 0x7ff7dc0d1350 2024-01-24T17:42:55.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.475+0000 7ff7d3e00640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff7b406a4b0 0x7ff7b406c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:55.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.476+0000 7ff7d3e00640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff7b406a4b0 0x7ff7b406c970 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc1f50d0 tx=0x7ff7c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:55.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.478+0000 7ff7d8c00640 1 -- 172.21.15.39:0/3961266531 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff7cc075ca0 con 0x7ff7dc0d1350 2024-01-24T17:42:55.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:55.676+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff7dc0d10e0 con 0x7ff7dc0d1350 2024-01-24T17:42:56.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.227+0000 7ff7d8c00640 1 -- 172.21.15.39:0/3961266531 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff7cc0792c0 con 0x7ff7dc0d1350 2024-01-24T17:42:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-01-24T17:42:56.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.238+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff7b406a4b0 msgr2=0x7ff7b406c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.238+0000 7ff7e1800640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff7b406a4b0 0x7ff7b406c970 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc1f50d0 tx=0x7ff7c4008040 comp rx=0 tx=0).stop 2024-01-24T17:42:56.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.238+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc0d1350 msgr2=0x7ff7dc0c8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.238+0000 7ff7e1800640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc0d1350 0x7ff7dc0c8ad0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009890 tx=0x7ff7cc0046c0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.239+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 shutdown_connections 2024-01-24T17:42:56.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.239+0000 7ff7e1800640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7dc15cd90 0x7ff7dc0c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.239+0000 7ff7e1800640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7dc15c9b0 0x7ff7dc0c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.239+0000 7ff7e1800640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff7b406a4b0 0x7ff7b406c970 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.239+0000 7ff7e1800640 1 --2- 172.21.15.39:0/3961266531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7dc0d1350 0x7ff7dc0c8ad0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.239+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 >> 172.21.15.39:0/3961266531 conn(0x7ff7dc0d2830 msgr2=0x7ff7dc0b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:56.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.239+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 shutdown_connections 2024-01-24T17:42:56.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.240+0000 7ff7e1800640 1 -- 172.21.15.39:0/3961266531 wait complete. 2024-01-24T17:42:56.269 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T17:42:56.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.453+0000 7fc6dbe00640 1 Processor -- start 2024-01-24T17:42:56.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.453+0000 7fc6dbe00640 1 -- start start 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.455+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d40b0350 0x7fc6d406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.455+0000 7fc6dbe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc6d415e190 0x7fc6d40b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.455+0000 7fc6dbe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d406d4c0 con 0x7fc6d406c9f0 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.455+0000 7fc6dbe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d406cf00 con 0x7fc6d415e190 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.455+0000 7fc6dbe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d406d1e0 con 0x7fc6d40b0350 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.456+0000 7fc6d9e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc6d415e190 0x7fc6d40b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.456+0000 7fc6d9400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d40b0350 0x7fc6d406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.456+0000 7fc6d9e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc6d415e190 0x7fc6d40b1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:54672/0 (socket says 172.21.15.39:54672) 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.456+0000 7fc6d9400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d40b0350 0x7fc6d406c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:57212/0 (socket says 172.21.15.39:57212) 2024-01-24T17:42:56.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.456+0000 7fc6d9e00640 1 -- 172.21.15.39:0/4134849498 learned_addr learned my addr 172.21.15.39:0/4134849498 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:56.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.457+0000 7fc6d9e00640 1 -- 172.21.15.39:0/4134849498 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d40b0350 msgr2=0x7fc6d406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:56.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.457+0000 7fc6d9e00640 1 --2- 172.21.15.39:0/4134849498 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d40b0350 0x7fc6d406c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.457+0000 7fc6d9e00640 1 -- 172.21.15.39:0/4134849498 >> v1:172.21.15.100:6789/0 conn(0x7fc6d406c9f0 legacy=0x7fc6d415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.457+0000 7fc6d9e00640 1 -- 172.21.15.39:0/4134849498 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d40b15e0 con 0x7fc6d415e190 2024-01-24T17:42:56.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.458+0000 7fc6d9e00640 1 --2- 172.21.15.39:0/4134849498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d415e190 0x7fc6d40b1de0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009870 tx=0x7fc6cc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e978e6cc5b249b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:56.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.458+0000 7fc6d3e00640 1 -- 172.21.15.39:0/4134849498 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc031640 con 0x7fc6d415e190 2024-01-24T17:42:56.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6d3e00640 1 -- 172.21.15.39:0/4134849498 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc0317a0 con 0x7fc6d415e190 2024-01-24T17:42:56.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6d3e00640 1 -- 172.21.15.39:0/4134849498 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc031b30 con 0x7fc6d415e190 2024-01-24T17:42:56.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6dbe00640 1 -- 172.21.15.39:0/4134849498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d415e190 msgr2=0x7fc6d40b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/4134849498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d415e190 0x7fc6d40b1de0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009870 tx=0x7fc6cc02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6dbe00640 1 -- 172.21.15.39:0/4134849498 shutdown_connections 2024-01-24T17:42:56.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/4134849498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d415e190 0x7fc6d40b1de0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/4134849498 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d40b0350 0x7fc6d406c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6dbe00640 1 -- 172.21.15.39:0/4134849498 >> 172.21.15.39:0/4134849498 conn(0x7fc6d40cb0e0 msgr2=0x7fc6d40cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:56.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6dbe00640 1 -- 172.21.15.39:0/4134849498 shutdown_connections 2024-01-24T17:42:56.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.459+0000 7fc6dbe00640 1 -- 172.21.15.39:0/4134849498 wait complete. 2024-01-24T17:42:56.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.461+0000 7fc6dbe00640 1 Processor -- start 2024-01-24T17:42:56.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.461+0000 7fc6dbe00640 1 -- start start 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.462+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6d406c9f0 0x7fc6d4158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.462+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d40b0350 0x7fc6d414e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.462+0000 7fc6dbe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d415e190 0x7fc6d415b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.462+0000 7fc6dbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d41efea0 con 0x7fc6d406c9f0 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.462+0000 7fc6dbe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d41594c0 con 0x7fc6d40b0350 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.462+0000 7fc6dbe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d41597a0 con 0x7fc6d415e190 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.462+0000 7fc6d9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6d406c9f0 0x7fc6d4158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6d406c9f0 0x7fc6d4158ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34838/0 (socket says 172.21.15.39:34838) 2024-01-24T17:42:56.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9400640 1 -- 172.21.15.39:0/304599319 learned_addr learned my addr 172.21.15.39:0/304599319 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d8a00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d40b0350 0x7fc6d414e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9e00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d415e190 0x7fc6d415b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9400640 1 -- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d415e190 msgr2=0x7fc6d415b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9400640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d415e190 0x7fc6d415b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9400640 1 -- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d40b0350 msgr2=0x7fc6d414e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9400640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d40b0350 0x7fc6d414e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9400640 1 -- 172.21.15.39:0/304599319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0094d0 con 0x7fc6d406c9f0 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.463+0000 7fc6d9400640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6d406c9f0 0x7fc6d4158ec0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400efc0 tx=0x7fc6c400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:56.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.464+0000 7fc6d1600640 1 -- 172.21.15.39:0/304599319 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4014480 con 0x7fc6d406c9f0 2024-01-24T17:42:56.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.464+0000 7fc6d1600640 1 -- 172.21.15.39:0/304599319 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c400f040 con 0x7fc6d406c9f0 2024-01-24T17:42:56.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.464+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d4159a80 con 0x7fc6d406c9f0 2024-01-24T17:42:56.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.464+0000 7fc6d1600640 1 -- 172.21.15.39:0/304599319 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4014800 con 0x7fc6d406c9f0 2024-01-24T17:42:56.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.464+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d41577d0 con 0x7fc6d406c9f0 2024-01-24T17:42:56.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.465+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d4073ef0 con 0x7fc6d406c9f0 2024-01-24T17:42:56.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.466+0000 7fc6d1600640 1 -- 172.21.15.39:0/304599319 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc6c4014960 con 0x7fc6d406c9f0 2024-01-24T17:42:56.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.467+0000 7fc6d1600640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6ac06a430 0x7fc6ac06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.467+0000 7fc6d1600640 1 -- 172.21.15.39:0/304599319 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc6c4089d90 con 0x7fc6d406c9f0 2024-01-24T17:42:56.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.468+0000 7fc6d8a00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6ac06a430 0x7fc6ac06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.469+0000 7fc6d8a00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6ac06a430 0x7fc6ac06c8f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40ac790 tx=0x7fc6bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:56.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.471+0000 7fc6d1600640 1 -- 172.21.15.39:0/304599319 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc6c4055340 con 0x7fc6d406c9f0 2024-01-24T17:42:56.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.698+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc6d414ea10 con 0x7fc6d406c9f0 2024-01-24T17:42:56.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.713+0000 7fc6d1600640 1 -- 172.21.15.39:0/304599319 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc6c4058960 con 0x7fc6d406c9f0 2024-01-24T17:42:56.714 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 4 and data pool 5 2024-01-24T17:42:56.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.721+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6ac06a430 msgr2=0x7fc6ac06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.721+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6ac06a430 0x7fc6ac06c8f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc6d40ac790 tx=0x7fc6bc009040 comp rx=0 tx=0).stop 2024-01-24T17:42:56.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.725+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6d406c9f0 msgr2=0x7fc6d4158ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.725+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6d406c9f0 0x7fc6d4158ec0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400efc0 tx=0x7fc6c400c490 comp rx=0 tx=0).stop 2024-01-24T17:42:56.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.727+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 shutdown_connections 2024-01-24T17:42:56.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.727+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6d415e190 0x7fc6d415b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.727+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6d40b0350 0x7fc6d414e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.727+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6ac06a430 0x7fc6ac06c8f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.727+0000 7fc6dbe00640 1 --2- 172.21.15.39:0/304599319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6d406c9f0 0x7fc6d4158ec0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.727+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 >> 172.21.15.39:0/304599319 conn(0x7fc6d40cb0e0 msgr2=0x7fc6d4152fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:56.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.728+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 shutdown_connections 2024-01-24T17:42:56.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.728+0000 7fc6dbe00640 1 -- 172.21.15.39:0/304599319 wait complete. 2024-01-24T17:42:56.754 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:56.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.933+0000 7f4d91a00640 1 Processor -- start 2024-01-24T17:42:56.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.933+0000 7f4d91a00640 1 -- start start 2024-01-24T17:42:56.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.934+0000 7f4d91a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4d8c0cd210 0x7f4d8c0b0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.934+0000 7f4d91a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c15e490 con 0x7f4d8c0d07c0 2024-01-24T17:42:56.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.934+0000 7f4d91a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c0cf8a0 con 0x7f4d8c0ce400 2024-01-24T17:42:56.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.934+0000 7f4d91a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d8c158990 con 0x7f4d8c0cd210 2024-01-24T17:42:56.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d8aa00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4d8c0d07c0 0x7f4d8c0d0ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40114/0 (socket says 172.21.15.39:40114) 2024-01-24T17:42:56.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d8aa00640 1 -- 172.21.15.39:0/2450352865 learned_addr learned my addr 172.21.15.39:0/2450352865 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:56.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d8b400640 1 --2- 172.21.15.39:0/2450352865 >> v2:172.21.15.100:3300/0 conn(0x7f4d8c0cd210 0x7f4d8c0b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356253275 0 0) 0x7f4d8c15e490 con 0x7f4d8c0d07c0 2024-01-24T17:42:56.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d6c003520 con 0x7f4d8c0d07c0 2024-01-24T17:42:56.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 133013060 0 0) 0x7f4d6c003520 con 0x7f4d8c0d07c0 2024-01-24T17:42:56.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 >> v2:172.21.15.100:3300/0 conn(0x7f4d8c0cd210 msgr2=0x7f4d8c0b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:56.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d89600640 1 --2- 172.21.15.39:0/2450352865 >> v2:172.21.15.100:3300/0 conn(0x7f4d8c0cd210 0x7f4d8c0b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 >> v1:172.21.15.100:6789/0 conn(0x7f4d8c0ce400 legacy=0x7f4d8c15ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.935+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c15f310 con 0x7f4d8c0d07c0 2024-01-24T17:42:56.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f4d7c001670 con 0x7f4d8c0d07c0 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 >> v1:172.21.15.39:6789/0 conn(0x7f4d8c0d07c0 legacy=0x7f4d8c0d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d89600640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6c003970 0x7f4d6c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d89600640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c0047c0 0x7f4d6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d89600640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d6c005180 0x7f4d6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c158990 con 0x7f4d6c003970 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c00c030 con 0x7f4d6c005180 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c00c1f0 con 0x7f4d6c0047c0 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8aa00640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6c003970 0x7f4d6c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8aa00640 1 -- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c0047c0 msgr2=0x7f4d6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8a000640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c0047c0 0x7f4d6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8b400640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d6c005180 0x7f4d6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8aa00640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c0047c0 0x7f4d6c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8aa00640 1 -- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d6c005180 msgr2=0x7f4d6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:56.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8aa00640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d6c005180 0x7f4d6c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8aa00640 1 -- 172.21.15.39:0/2450352865 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c003520 con 0x7f4d6c003970 2024-01-24T17:42:56.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8b400640 1 -- 172.21.15.39:0/2450352865 reap_dead start 2024-01-24T17:42:56.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.936+0000 7f4d8a000640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c0047c0 0x7f4d6c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:56.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.937+0000 7f4d8aa00640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6c003970 0x7f4d6c003d90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00c240 tx=0x7f4d6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb4b3fc4e590c45 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:56.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.938+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c005ad0 con 0x7f4d6c003970 2024-01-24T17:42:56.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.938+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c005c30 con 0x7f4d6c003970 2024-01-24T17:42:56.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.938+0000 7f4d89600640 1 -- 172.21.15.39:0/2450352865 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c02ca70 con 0x7f4d6c003970 2024-01-24T17:42:56.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.938+0000 7f4d91a00640 1 -- 172.21.15.39:0/2450352865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6c003970 msgr2=0x7f4d6c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.938+0000 7f4d91a00640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6c003970 0x7f4d6c003d90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00c240 tx=0x7f4d6c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.938+0000 7f4d91a00640 1 -- 172.21.15.39:0/2450352865 shutdown_connections 2024-01-24T17:42:56.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.938+0000 7f4d91a00640 1 --2- 172.21.15.39:0/2450352865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d6c003970 0x7f4d6c003d90 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.938+0000 7f4d91a00640 1 -- 172.21.15.39:0/2450352865 >> 172.21.15.39:0/2450352865 conn(0x7f4d8c089fd0 msgr2=0x7f4d8c099b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:56.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.939+0000 7f4d91a00640 1 -- 172.21.15.39:0/2450352865 shutdown_connections 2024-01-24T17:42:56.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.939+0000 7f4d91a00640 1 -- 172.21.15.39:0/2450352865 wait complete. 2024-01-24T17:42:56.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.940+0000 7f4d91a00640 1 Processor -- start 2024-01-24T17:42:56.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.940+0000 7f4d91a00640 1 -- start start 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.941+0000 7f4d91a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c003970 0x7f4d8c157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.941+0000 7f4d91a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d8c158090 0x7f4d8c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.941+0000 7f4d91a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d8c0b3960 0x7f4d8c0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.941+0000 7f4d91a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0b4a60 con 0x7f4d8c158090 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.941+0000 7f4d91a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0b4e80 con 0x7f4d8c0b3960 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.941+0000 7f4d91a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0b3de0 con 0x7f4d6c003970 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d8c158090 0x7f4d8c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d8c158090 0x7f4d8c0b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34858/0 (socket says 172.21.15.39:34858) 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8a000640 1 -- 172.21.15.39:0/1633398624 learned_addr learned my addr 172.21.15.39:0/1633398624 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8aa00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c003970 0x7f4d8c157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8a000640 1 -- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c003970 msgr2=0x7f4d8c157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8a000640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c003970 0x7f4d8c157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8a000640 1 -- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d8c0b3960 msgr2=0x7f4d8c0c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8b400640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d8c0b3960 0x7f4d8c0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8a000640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d8c0b3960 0x7f4d8c0c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.942+0000 7f4d8a000640 1 -- 172.21.15.39:0/1633398624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c0b40c0 con 0x7f4d8c158090 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.943+0000 7f4d8a000640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d8c158090 0x7f4d8c0b3420 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400d400 tx=0x7f4d7400d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.943+0000 7f4d82a00640 1 -- 172.21.15.39:0/1633398624 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7400b070 con 0x7f4d8c158090 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.943+0000 7f4d82a00640 1 -- 172.21.15.39:0/1633398624 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d74007030 con 0x7f4d8c158090 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.943+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c0b43a0 con 0x7f4d8c158090 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.943+0000 7f4d82a00640 1 -- 172.21.15.39:0/1633398624 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d74007420 con 0x7f4d8c158090 2024-01-24T17:42:56.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.943+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c157ca0 con 0x7f4d8c158090 2024-01-24T17:42:56.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.944+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8c067040 con 0x7f4d8c158090 2024-01-24T17:42:56.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.945+0000 7f4d82a00640 1 -- 172.21.15.39:0/1633398624 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4d74007580 con 0x7f4d8c158090 2024-01-24T17:42:56.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.946+0000 7f4d82a00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4d6406a430 0x7f4d6406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:56.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.946+0000 7f4d82a00640 1 -- 172.21.15.39:0/1633398624 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d740886a0 con 0x7f4d8c158090 2024-01-24T17:42:56.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.947+0000 7f4d8aa00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4d6406a430 0x7f4d6406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:56.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.948+0000 7f4d8aa00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4d6406a430 0x7f4d6406c8f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c034000 tx=0x7f4d7c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:56.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:56.950+0000 7f4d82a00640 1 -- 172.21.15.39:0/1633398624 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4d74053c80 con 0x7f4d8c158090 2024-01-24T17:42:57.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.144+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d8c0d12d0 con 0x7f4d8c158090 2024-01-24T17:42:57.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.147+0000 7f4d82a00640 1 -- 172.21.15.39:0/1633398624 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f4d740572a0 con 0x7f4d8c158090 2024-01-24T17:42:57.149 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:57.150 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":33,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:56.700521+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4d6406a430 msgr2=0x7f4d6406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4d6406a430 0x7f4d6406c8f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c034000 tx=0x7f4d7c009030 comp rx=0 tx=0).stop 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d8c158090 msgr2=0x7f4d8c0b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d8c158090 0x7f4d8c0b3420 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400d400 tx=0x7f4d7400d8c0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 shutdown_connections 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d8c0b3960 0x7f4d8c0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4d6406a430 0x7f4d6406c8f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d8c158090 0x7f4d8c0b3420 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 --2- 172.21.15.39:0/1633398624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4d6c003970 0x7f4d8c157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 >> 172.21.15.39:0/1633398624 conn(0x7f4d8c089fd0 msgr2=0x7f4d8c06e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 shutdown_connections 2024-01-24T17:42:57.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.150+0000 7f4d91a00640 1 -- 172.21.15.39:0/1633398624 wait complete. 2024-01-24T17:42:57.170 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:42:57.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.392+0000 7fc7c2800640 1 Processor -- start 2024-01-24T17:42:57.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.392+0000 7fc7c2800640 1 -- start start 2024-01-24T17:42:57.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.394+0000 7fc7c2800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1654d0 0x7fc7bc0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.394+0000 7fc7c2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc7bc0c90d0 0x7fc7bc070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.394+0000 7fc7c2800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc0cc630 con 0x7fc7bc1658b0 2024-01-24T17:42:57.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.394+0000 7fc7c2800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cc9d0 con 0x7fc7bc0c90d0 2024-01-24T17:42:57.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.394+0000 7fc7c2800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0c9570 con 0x7fc7bc1654d0 2024-01-24T17:42:57.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.394+0000 7fc7baa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc7bc0c90d0 0x7fc7bc070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.394+0000 7fc7baa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc7bc0c90d0 0x7fc7bc070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34872/0 (socket says 172.21.15.39:34872) 2024-01-24T17:42:57.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.394+0000 7fc7baa00640 1 -- 172.21.15.39:0/348314602 learned_addr learned my addr 172.21.15.39:0/348314602 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:57.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.395+0000 7fc7bb400640 1 --2- 172.21.15.39:0/348314602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1654d0 0x7fc7bc0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.395+0000 7fc7baa00640 1 -- 172.21.15.39:0/348314602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1654d0 msgr2=0x7fc7bc0c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:57.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.395+0000 7fc7baa00640 1 --2- 172.21.15.39:0/348314602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1654d0 0x7fc7bc0c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.395+0000 7fc7baa00640 1 -- 172.21.15.39:0/348314602 >> v1:172.21.15.100:6789/0 conn(0x7fc7bc1658b0 legacy=0x7fc7bc0caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.395+0000 7fc7baa00640 1 -- 172.21.15.39:0/348314602 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc0c9850 con 0x7fc7bc0c90d0 2024-01-24T17:42:57.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.396+0000 7fc7baa00640 1 --2- 172.21.15.39:0/348314602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc0c90d0 0x7fc7bc070030 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009870 tx=0x7fc7a402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7edc98e38e3ad15 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.396+0000 7fc7ba000640 1 -- 172.21.15.39:0/348314602 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40317c0 con 0x7fc7bc0c90d0 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.396+0000 7fc7ba000640 1 -- 172.21.15.39:0/348314602 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a4031920 con 0x7fc7bc0c90d0 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.396+0000 7fc7ba000640 1 -- 172.21.15.39:0/348314602 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4031cb0 con 0x7fc7bc0c90d0 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.397+0000 7fc7c2800640 1 -- 172.21.15.39:0/348314602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc0c90d0 msgr2=0x7fc7bc070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.397+0000 7fc7c2800640 1 --2- 172.21.15.39:0/348314602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc0c90d0 0x7fc7bc070030 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009870 tx=0x7fc7a402b860 comp rx=0 tx=0).stop 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.397+0000 7fc7c2800640 1 -- 172.21.15.39:0/348314602 shutdown_connections 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.397+0000 7fc7c2800640 1 --2- 172.21.15.39:0/348314602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc0c90d0 0x7fc7bc070030 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.397+0000 7fc7c2800640 1 --2- 172.21.15.39:0/348314602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1654d0 0x7fc7bc0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.397+0000 7fc7c2800640 1 -- 172.21.15.39:0/348314602 >> 172.21.15.39:0/348314602 conn(0x7fc7bc06a3e0 msgr2=0x7fc7bc06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.397+0000 7fc7c2800640 1 -- 172.21.15.39:0/348314602 shutdown_connections 2024-01-24T17:42:57.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.397+0000 7fc7c2800640 1 -- 172.21.15.39:0/348314602 wait complete. 2024-01-24T17:42:57.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.399+0000 7fc7c2800640 1 Processor -- start 2024-01-24T17:42:57.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.399+0000 7fc7c2800640 1 -- start start 2024-01-24T17:42:57.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7c2800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0c90d0 0x7fc7bc0c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7c2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc1654d0 0x7fc7bc15ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7c2800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1658b0 0x7fc7bc0b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7c2800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0c1df0 con 0x7fc7bc1654d0 2024-01-24T17:42:57.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7c2800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0b2140 con 0x7fc7bc0c90d0 2024-01-24T17:42:57.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7c2800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0c20b0 con 0x7fc7bc1658b0 2024-01-24T17:42:57.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7baa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc1654d0 0x7fc7bc15ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7baa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc1654d0 0x7fc7bc15ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34886/0 (socket says 172.21.15.39:34886) 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.400+0000 7fc7baa00640 1 -- 172.21.15.39:0/3201304960 learned_addr learned my addr 172.21.15.39:0/3201304960 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7bbe00640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1658b0 0x7fc7bc0b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7bb400640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0c90d0 0x7fc7bc0c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7baa00640 1 -- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1658b0 msgr2=0x7fc7bc0b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7baa00640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1658b0 0x7fc7bc0b2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7baa00640 1 -- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0c90d0 msgr2=0x7fc7bc0c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7baa00640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0c90d0 0x7fc7bc0c6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7baa00640 1 -- 172.21.15.39:0/3201304960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40094d0 con 0x7fc7bc1654d0 2024-01-24T17:42:57.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7baa00640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc1654d0 0x7fc7bc15ea30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc7a402b8a0 tx=0x7fc7a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:57.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7b3400640 1 -- 172.21.15.39:0/3201304960 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4032860 con 0x7fc7bc1654d0 2024-01-24T17:42:57.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7b3400640 1 -- 172.21.15.39:0/3201304960 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a4032a00 con 0x7fc7bc1654d0 2024-01-24T17:42:57.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc0c2390 con 0x7fc7bc1654d0 2024-01-24T17:42:57.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.401+0000 7fc7b3400640 1 -- 172.21.15.39:0/3201304960 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4032db0 con 0x7fc7bc1654d0 2024-01-24T17:42:57.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.402+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc0c6740 con 0x7fc7bc1654d0 2024-01-24T17:42:57.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.403+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc073ef0 con 0x7fc7bc1654d0 2024-01-24T17:42:57.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.404+0000 7fc7b3400640 1 -- 172.21.15.39:0/3201304960 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc7a404e450 con 0x7fc7bc1654d0 2024-01-24T17:42:57.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.404+0000 7fc7b3400640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc79406a460 0x7fc79406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.404+0000 7fc7b3400640 1 -- 172.21.15.39:0/3201304960 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7a40aab00 con 0x7fc7bc1654d0 2024-01-24T17:42:57.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.405+0000 7fc7bb400640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc79406a460 0x7fc79406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.406+0000 7fc7bb400640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc79406a460 0x7fc79406c920 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac006fd0 tx=0x7fc7ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:57.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.409+0000 7fc7b3400640 1 -- 172.21.15.39:0/3201304960 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc7a40ace60 con 0x7fc7bc1654d0 2024-01-24T17:42:57.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.600+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7bc0b44c0 con 0x7fc7bc1654d0 2024-01-24T17:42:57.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.603+0000 7fc7b3400640 1 -- 172.21.15.39:0/3201304960 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16054 (secure 0 0 0) 0x7fc7a40761a0 con 0x7fc7bc1654d0 2024-01-24T17:42:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:42:57.605 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":33,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:56.700521+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:42:57.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc79406a460 msgr2=0x7fc79406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc79406a460 0x7fc79406c920 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac006fd0 tx=0x7fc7ac008040 comp rx=0 tx=0).stop 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc1654d0 msgr2=0x7fc7bc15ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc1654d0 0x7fc7bc15ea30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc7a402b8a0 tx=0x7fc7a40046c0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 shutdown_connections 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7bc1658b0 0x7fc7bc0b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc79406a460 0x7fc79406c920 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7bc1654d0 0x7fc7bc15ea30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 --2- 172.21.15.39:0/3201304960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7bc0c90d0 0x7fc7bc0c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 >> 172.21.15.39:0/3201304960 conn(0x7fc7bc06a3e0 msgr2=0x7fc7bc162700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 shutdown_connections 2024-01-24T17:42:57.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.606+0000 7fc7c2800640 1 -- 172.21.15.39:0/3201304960 wait complete. 2024-01-24T17:42:57.628 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T17:42:57.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.836+0000 7fb0fd000640 1 Processor -- start 2024-01-24T17:42:57.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.837+0000 7fb0fd000640 1 -- start start 2024-01-24T17:42:57.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.838+0000 7fb0fd000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0f80d14c0 0x7fb0f80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.838+0000 7fb0fd000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f80cda90 con 0x7fb0f80b2a10 2024-01-24T17:42:57.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.838+0000 7fb0fd000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f80d32c0 con 0x7fb0f815fff0 2024-01-24T17:42:57.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.838+0000 7fb0fd000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f80b2e80 con 0x7fb0f80d14c0 2024-01-24T17:42:57.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.839+0000 7fb0f5400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0f80d14c0 0x7fb0f80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.839+0000 7fb0f6800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb0f80b2a10 0x7fb0f80cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40120/0 (socket says 172.21.15.39:40120) 2024-01-24T17:42:57.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.839+0000 7fb0f6800640 1 -- 172.21.15.39:0/2868708956 learned_addr learned my addr 172.21.15.39:0/2868708956 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:57.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.839+0000 7fb0f5400640 1 -- 172.21.15.39:0/2868708956 >> v1:172.21.15.100:6789/0 conn(0x7fb0f815fff0 legacy=0x7fb0f80d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.839+0000 7fb0f5400640 1 -- 172.21.15.39:0/2868708956 >> v1:172.21.15.39:6789/0 conn(0x7fb0f80b2a10 legacy=0x7fb0f80cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.839+0000 7fb0f5400640 1 -- 172.21.15.39:0/2868708956 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f80b30a0 con 0x7fb0f80d14c0 2024-01-24T17:42:57.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.839+0000 7fb0f4a00640 1 -- 172.21.15.39:0/2868708956 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560883239 0 0) 0x7fb0f80cda90 con 0x7fb0f80b2a10 2024-01-24T17:42:57.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.840+0000 7fb0f5400640 1 --2- 172.21.15.39:0/2868708956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80d14c0 0x7fb0f80b3660 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009d90 tx=0x7fb0e002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb8eab49791992f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:57.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.840+0000 7fb0f4a00640 1 -- 172.21.15.39:0/2868708956 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e002cd00 con 0x7fb0f80d14c0 2024-01-24T17:42:57.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.840+0000 7fb0f4a00640 1 -- 172.21.15.39:0/2868708956 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e002ce60 con 0x7fb0f80d14c0 2024-01-24T17:42:57.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.840+0000 7fb0f4a00640 1 -- 172.21.15.39:0/2868708956 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0034660 con 0x7fb0f80d14c0 2024-01-24T17:42:57.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.841+0000 7fb0fd000640 1 -- 172.21.15.39:0/2868708956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80d14c0 msgr2=0x7fb0f80b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.841+0000 7fb0fd000640 1 --2- 172.21.15.39:0/2868708956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80d14c0 0x7fb0f80b3660 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009d90 tx=0x7fb0e002bf50 comp rx=0 tx=0).stop 2024-01-24T17:42:57.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.841+0000 7fb0fd000640 1 -- 172.21.15.39:0/2868708956 shutdown_connections 2024-01-24T17:42:57.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.841+0000 7fb0fd000640 1 --2- 172.21.15.39:0/2868708956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80d14c0 0x7fb0f80b3660 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.841+0000 7fb0fd000640 1 -- 172.21.15.39:0/2868708956 >> 172.21.15.39:0/2868708956 conn(0x7fb0f809bcc0 msgr2=0x7fb0f80d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:57.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.841+0000 7fb0fd000640 1 -- 172.21.15.39:0/2868708956 shutdown_connections 2024-01-24T17:42:57.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.841+0000 7fb0fd000640 1 -- 172.21.15.39:0/2868708956 wait complete. 2024-01-24T17:42:57.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.843+0000 7fb0fd000640 1 Processor -- start 2024-01-24T17:42:57.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.843+0000 7fb0fd000640 1 -- start start 2024-01-24T17:42:57.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0fd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80b2a10 0x7fb0f8153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0fd000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0f80d14c0 0x7fb0f809f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0fd000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0f815fff0 0x7fb0f809c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0fd000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f80a06b0 con 0x7fb0f80b2a10 2024-01-24T17:42:57.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0fd000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f809faa0 con 0x7fb0f80d14c0 2024-01-24T17:42:57.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0fd000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f809fd80 con 0x7fb0f815fff0 2024-01-24T17:42:57.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0f5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80b2a10 0x7fb0f8153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0f5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80b2a10 0x7fb0f8153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34906/0 (socket says 172.21.15.39:34906) 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0f5e00640 1 -- 172.21.15.39:0/963201106 learned_addr learned my addr 172.21.15.39:0/963201106 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0f5e00640 1 -- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0f815fff0 msgr2=0x7fb0f809c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0f5e00640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0f815fff0 0x7fb0f809c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0f5e00640 1 -- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0f80d14c0 msgr2=0x7fb0f809f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0f5e00640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0f80d14c0 0x7fb0f809f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.844+0000 7fb0f5e00640 1 -- 172.21.15.39:0/963201106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0009a70 con 0x7fb0f80b2a10 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.845+0000 7fb0f5e00640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80b2a10 0x7fb0f8153590 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800c400 tx=0x7fb0e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.845+0000 7fb0ee000640 1 -- 172.21.15.39:0/963201106 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e80052d0 con 0x7fb0f80b2a10 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.845+0000 7fb0ee000640 1 -- 172.21.15.39:0/963201106 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e8005430 con 0x7fb0f80b2a10 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.845+0000 7fb0ee000640 1 -- 172.21.15.39:0/963201106 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8003830 con 0x7fb0f80b2a10 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.845+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f80a0060 con 0x7fb0f80b2a10 2024-01-24T17:42:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.845+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f809f790 con 0x7fb0f80b2a10 2024-01-24T17:42:57.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.846+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f8073ef0 con 0x7fb0f80b2a10 2024-01-24T17:42:57.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.847+0000 7fb0ee000640 1 -- 172.21.15.39:0/963201106 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb0e8003990 con 0x7fb0f80b2a10 2024-01-24T17:42:57.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.848+0000 7fb0ee000640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d006a430 0x7fb0d006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:57.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.848+0000 7fb0ee000640 1 -- 172.21.15.39:0/963201106 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0e8087f00 con 0x7fb0f80b2a10 2024-01-24T17:42:57.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.848+0000 7fb0f5400640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d006a430 0x7fb0d006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:57.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.849+0000 7fb0f5400640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d006a430 0x7fb0d006c8f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0036000 tx=0x7fb0e002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:57.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:57.852+0000 7fb0ee000640 1 -- 172.21.15.39:0/963201106 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb0e80534e0 con 0x7fb0f80b2a10 2024-01-24T17:42:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.090+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb0f80cccb0 con 0x7fb0f80b2a10 2024-01-24T17:42:58.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.743+0000 7fb0ee000640 1 -- 172.21.15.39:0/963201106 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb0e8056b00 con 0x7fb0f80b2a10 2024-01-24T17:42:58.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d006a430 msgr2=0x7fb0d006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d006a430 0x7fb0d006c8f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0036000 tx=0x7fb0e002e000 comp rx=0 tx=0).stop 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80b2a10 msgr2=0x7fb0f8153590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80b2a10 0x7fb0f8153590 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800c400 tx=0x7fb0e800c8c0 comp rx=0 tx=0).stop 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 shutdown_connections 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0f815fff0 0x7fb0f809c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0f80d14c0 0x7fb0f809f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d006a430 0x7fb0d006c8f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 --2- 172.21.15.39:0/963201106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0f80b2a10 0x7fb0f8153590 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 >> 172.21.15.39:0/963201106 conn(0x7fb0f809bcc0 msgr2=0x7fb0f80af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 shutdown_connections 2024-01-24T17:42:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.747+0000 7fb0fd000640 1 -- 172.21.15.39:0/963201106 wait complete. 2024-01-24T17:42:58.773 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-24T17:42:58.773 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-24T17:42:58.986 INFO:tasks.ceph.mds.a.smithi039.stderr:2024-01-24T17:42:58.984+0000 7f6ff1200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:42:59.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.998+0000 7fbef9000640 1 Processor -- start 2024-01-24T17:42:59.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:58.998+0000 7fbef9000640 1 -- start start 2024-01-24T17:42:59.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.000+0000 7fbef9000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef415e190 0x7fbef41656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:59.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.000+0000 7fbef9000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef406d4c0 con 0x7fbef40b0350 2024-01-24T17:42:59.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.000+0000 7fbef9000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef406cf00 con 0x7fbef40b0730 2024-01-24T17:42:59.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.000+0000 7fbef9000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef406d1e0 con 0x7fbef415e190 2024-01-24T17:42:59.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.000+0000 7fbef1e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fbef40b0350 0x7fbef406c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40130/0 (socket says 172.21.15.39:40130) 2024-01-24T17:42:59.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.000+0000 7fbef1e00640 1 -- 172.21.15.39:0/3893693973 learned_addr learned my addr 172.21.15.39:0/3893693973 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:42:59.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef2800640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef415e190 0x7fbef41656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:59.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695832600 0 0) 0x7fbef406d4c0 con 0x7fbef40b0350 2024-01-24T17:42:59.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed4003520 con 0x7fbef40b0350 2024-01-24T17:42:59.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2927550272 0 0) 0x7fbed4003520 con 0x7fbef40b0350 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef415e190 msgr2=0x7fbef41656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef415e190 0x7fbef41656a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 >> v1:172.21.15.100:6789/0 conn(0x7fbef40b0730 legacy=0x7fbef415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef40b15e0 con 0x7fbef40b0350 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fbee4001670 con 0x7fbef40b0350 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 >> v1:172.21.15.39:6789/0 conn(0x7fbef40b0350 legacy=0x7fbef406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbed4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef415e190 0x7fbed40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.001+0000 7fbef1400640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbed4004d00 0x7fbed4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef406d1e0 con 0x7fbed40039a0 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed400bb10 con 0x7fbed4004d00 2024-01-24T17:42:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed400bcd0 con 0x7fbef415e190 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1e00640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbed4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbeebe00640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef415e190 0x7fbed40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef2800640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbed4004d00 0x7fbed4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1e00640 1 -- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef415e190 msgr2=0x7fbed40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1e00640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef415e190 0x7fbed40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1e00640 1 -- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbed4004d00 msgr2=0x7fbed4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1e00640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbed4004d00 0x7fbed4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef1e00640 1 -- 172.21.15.39:0/3893693973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed4003520 con 0x7fbed40039a0 2024-01-24T17:42:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.002+0000 7fbef2800640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbed4004d00 0x7fbed4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.004+0000 7fbef1e00640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbed4003dc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbee400c0e0 tx=0x7fbed40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132526fc0bb08e22 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.004+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4005ad0 con 0x7fbed40039a0 2024-01-24T17:42:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.004+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee4005c30 con 0x7fbed40039a0 2024-01-24T17:42:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.004+0000 7fbef1400640 1 -- 172.21.15.39:0/3893693973 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee402c7d0 con 0x7fbed40039a0 2024-01-24T17:42:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.005+0000 7fbef9000640 1 -- 172.21.15.39:0/3893693973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 msgr2=0x7fbed4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.005+0000 7fbef9000640 1 --2- 172.21.15.39:0/3893693973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbed4003dc0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbee400c0e0 tx=0x7fbed40057f0 comp rx=0 tx=0).stop 2024-01-24T17:42:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.005+0000 7fbef2800640 1 -- 172.21.15.39:0/3893693973 reap_dead start 2024-01-24T17:42:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.005+0000 7fbef9000640 1 -- 172.21.15.39:0/3893693973 shutdown_connections 2024-01-24T17:42:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.005+0000 7fbef9000640 1 -- 172.21.15.39:0/3893693973 >> 172.21.15.39:0/3893693973 conn(0x7fbef40cb0e0 msgr2=0x7fbef40cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:42:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.005+0000 7fbef9000640 1 -- 172.21.15.39:0/3893693973 shutdown_connections 2024-01-24T17:42:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.005+0000 7fbef9000640 1 -- 172.21.15.39:0/3893693973 wait complete. 2024-01-24T17:42:59.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.007+0000 7fbef9000640 1 Processor -- start 2024-01-24T17:42:59.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.008+0000 7fbef9000640 1 -- start start 2024-01-24T17:42:59.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbef40c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:59.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef9000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef41554c0 0x7fbef41535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef9000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbef4153b30 0x7fbef41e47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef9000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef4153fb0 con 0x7fbed40039a0 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef9000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef41f36b0 con 0x7fbef4153b30 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef9000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40c5210 con 0x7fbef41554c0 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbef40c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbef40c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34926/0 (socket says 172.21.15.39:34926) 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.009+0000 7fbef1e00640 1 -- 172.21.15.39:0/3716667222 learned_addr learned my addr 172.21.15.39:0/3716667222 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbef2800640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbef4153b30 0x7fbef41e47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbeebe00640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef41554c0 0x7fbef41535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbef1e00640 1 -- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef41554c0 msgr2=0x7fbef41535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbef1e00640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef41554c0 0x7fbef41535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbef1e00640 1 -- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbef4153b30 msgr2=0x7fbef41e47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbef1e00640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbef4153b30 0x7fbef41e47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:42:59.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbef1e00640 1 -- 172.21.15.39:0/3716667222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef41f5d80 con 0x7fbed40039a0 2024-01-24T17:42:59.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbef2800640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbef4153b30 0x7fbef41e47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:42:59.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.010+0000 7fbef1e00640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbef40c4cd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbee400c200 tx=0x7fbef40b0ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:59.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.011+0000 7fbeea000640 1 -- 172.21.15.39:0/3716667222 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee402cbe0 con 0x7fbed40039a0 2024-01-24T17:42:59.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.011+0000 7fbeea000640 1 -- 172.21.15.39:0/3716667222 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee402cd40 con 0x7fbed40039a0 2024-01-24T17:42:59.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.011+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef41f6020 con 0x7fbed40039a0 2024-01-24T17:42:59.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.011+0000 7fbeea000640 1 -- 172.21.15.39:0/3716667222 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4007270 con 0x7fbed40039a0 2024-01-24T17:42:59.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.011+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef41f65e0 con 0x7fbed40039a0 2024-01-24T17:42:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.012+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef4073ef0 con 0x7fbed40039a0 2024-01-24T17:42:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.013+0000 7fbeea000640 1 -- 172.21.15.39:0/3716667222 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbee4032070 con 0x7fbed40039a0 2024-01-24T17:42:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.014+0000 7fbeea000640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbecc06a460 0x7fbecc06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:42:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.014+0000 7fbeea000640 1 -- 172.21.15.39:0/3716667222 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbee40a9be0 con 0x7fbed40039a0 2024-01-24T17:42:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.014+0000 7fbeebe00640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbecc06a460 0x7fbecc06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:42:59.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.016+0000 7fbeebe00640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbecc06a460 0x7fbecc06c920 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbef40c9120 tx=0x7fbedc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:42:59.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.018+0000 7fbeea000640 1 -- 172.21.15.39:0/3716667222 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbee40751e0 con 0x7fbed40039a0 2024-01-24T17:42:59.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:42:59.251+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbef40ac8d0 con 0x7fbed40039a0 2024-01-24T17:43:00.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.261+0000 7fbeea000640 1 -- 172.21.15.39:0/3716667222 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbee4045070 con 0x7fbed40039a0 2024-01-24T17:43:00.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbecc06a460 msgr2=0x7fbecc06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbecc06a460 0x7fbecc06c920 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbef40c9120 tx=0x7fbedc009040 comp rx=0 tx=0).stop 2024-01-24T17:43:00.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 msgr2=0x7fbef40c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbef40c4cd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbee400c200 tx=0x7fbef40b0ba0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 shutdown_connections 2024-01-24T17:43:00.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbef4153b30 0x7fbef41e47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbef41554c0 0x7fbef41535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbecc06a460 0x7fbecc06c920 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 --2- 172.21.15.39:0/3716667222 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbed40039a0 0x7fbef40c4cd0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.265+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 >> 172.21.15.39:0/3716667222 conn(0x7fbef40cb0e0 msgr2=0x7fbef40af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:00.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.266+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 shutdown_connections 2024-01-24T17:43:00.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.266+0000 7fbef9000640 1 -- 172.21.15.39:0/3716667222 wait complete. 2024-01-24T17:43:00.289 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-24T17:43:00.289 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:43:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.512+0000 7f88cb400640 1 Processor -- start 2024-01-24T17:43:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.512+0000 7f88cb400640 1 -- start start 2024-01-24T17:43:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88cb400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 0x7f88c4071590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88cb400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c40c9670 con 0x7f88c415c280 2024-01-24T17:43:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88cb400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c40ca210 con 0x7f88c406bde0 2024-01-24T17:43:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88cb400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40adab0 con 0x7f88c4071130 2024-01-24T17:43:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88ca000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f88c415c280 0x7f88c40c9b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40146/0 (socket says 172.21.15.39:40146) 2024-01-24T17:43:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88ca000640 1 -- 172.21.15.39:0/216842105 learned_addr learned my addr 172.21.15.39:0/216842105 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:00.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287340547 0 0) 0x7f88c40c9670 con 0x7f88c415c280 2024-01-24T17:43:00.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88c9600640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 0x7f88c4071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:00.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.514+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a4003520 con 0x7f88c415c280 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168144810 0 0) 0x7f88a4003520 con 0x7f88c415c280 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 msgr2=0x7f88c4071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 0x7f88c4071590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 >> v1:172.21.15.100:6789/0 conn(0x7f88c406bde0 legacy=0x7f88c40cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c40c8a60 con 0x7f88c415c280 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f88b4001670 con 0x7f88c415c280 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c9600640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 0x7f88c4071590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 >> v1:172.21.15.39:6789/0 conn(0x7f88c415c280 legacy=0x7f88c40c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 0x7f88a4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:00.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88a4004160 0x7f88a400a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a400b020 0x7f88a400d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40adab0 con 0x7f88a4004160 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a400dcf0 con 0x7f88a400b020 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a400deb0 con 0x7f88c4071130 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.515+0000 7f88c9600640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88a4004160 0x7f88a400a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.516+0000 7f88ca000640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 0x7f88a4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.516+0000 7f88caa00640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a400b020 0x7f88a400d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.516+0000 7f88c9600640 1 -- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 msgr2=0x7f88a4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.516+0000 7f88c9600640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c4071130 0x7f88a4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.516+0000 7f88c9600640 1 -- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a400b020 msgr2=0x7f88a400d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:00.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.516+0000 7f88c9600640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88a400b020 0x7f88a400d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.516+0000 7f88c9600640 1 -- 172.21.15.39:0/216842105 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4003520 con 0x7f88a4004160 2024-01-24T17:43:00.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.517+0000 7f88c9600640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88a4004160 0x7f88a400a820 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0073e0 tx=0x7f88a40037d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f2d548f9565d58 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:00.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.517+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac02cd30 con 0x7f88a4004160 2024-01-24T17:43:00.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.517+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac02ce90 con 0x7f88a4004160 2024-01-24T17:43:00.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.517+0000 7f88c8c00640 1 -- 172.21.15.39:0/216842105 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac034640 con 0x7f88a4004160 2024-01-24T17:43:00.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.518+0000 7f88cb400640 1 -- 172.21.15.39:0/216842105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88a4004160 msgr2=0x7f88a400a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.518+0000 7f88cb400640 1 --2- 172.21.15.39:0/216842105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88a4004160 0x7f88a400a820 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0073e0 tx=0x7f88a40037d0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.518+0000 7f88caa00640 1 -- 172.21.15.39:0/216842105 reap_dead start 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.518+0000 7f88cb400640 1 -- 172.21.15.39:0/216842105 shutdown_connections 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.518+0000 7f88cb400640 1 -- 172.21.15.39:0/216842105 >> 172.21.15.39:0/216842105 conn(0x7f88c406a090 msgr2=0x7f88c406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.518+0000 7f88cb400640 1 -- 172.21.15.39:0/216842105 shutdown_connections 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.518+0000 7f88cb400640 1 -- 172.21.15.39:0/216842105 wait complete. 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.520+0000 7f88cb400640 1 Processor -- start 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.520+0000 7f88cb400640 1 -- start start 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88cb400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c40c46c0 0x7f88c40c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88cb400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88c40b39d0 0x7f88c40b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88cb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88c40b3270 0x7f88c40aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:00.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88cb400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40b3e30 con 0x7f88c40b3270 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88cb400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40ade50 con 0x7f88c40b39d0 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88cb400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40ae6f0 con 0x7f88c40c46c0 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88c40b3270 0x7f88c40aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88c40b3270 0x7f88c40aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34936/0 (socket says 172.21.15.39:34936) 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.521+0000 7f88caa00640 1 -- 172.21.15.39:0/3846368787 learned_addr learned my addr 172.21.15.39:0/3846368787 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88ca000640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c40c46c0 0x7f88c40c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88c9600640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88c40b39d0 0x7f88c40b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88caa00640 1 -- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c40c46c0 msgr2=0x7f88c40c4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88caa00640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c40c46c0 0x7f88c40c4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88caa00640 1 -- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88c40b39d0 msgr2=0x7f88c40b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88caa00640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88c40b39d0 0x7f88c40b2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88caa00640 1 -- 172.21.15.39:0/3846368787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c40afe30 con 0x7f88c40b3270 2024-01-24T17:43:00.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88caa00640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88c40b3270 0x7f88c40aeb80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f88c406ceb0 tx=0x7f88bc00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:00.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88c2000640 1 -- 172.21.15.39:0/3846368787 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc00b070 con 0x7f88c40b3270 2024-01-24T17:43:00.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88c2000640 1 -- 172.21.15.39:0/3846368787 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc0054e0 con 0x7f88c40b3270 2024-01-24T17:43:00.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88c2000640 1 -- 172.21.15.39:0/3846368787 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc003810 con 0x7f88c40b3270 2024-01-24T17:43:00.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.522+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c40ad040 con 0x7f88c40b3270 2024-01-24T17:43:00.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.523+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c40ae380 con 0x7f88c40b3270 2024-01-24T17:43:00.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.524+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c404dff0 con 0x7f88c40b3270 2024-01-24T17:43:00.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.525+0000 7f88c2000640 1 -- 172.21.15.39:0/3846368787 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f88bc003970 con 0x7f88c40b3270 2024-01-24T17:43:00.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.526+0000 7f88c2000640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f889c06a460 0x7f889c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:00.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.526+0000 7f88c2000640 1 -- 172.21.15.39:0/3846368787 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88bc087fd0 con 0x7f88c40b3270 2024-01-24T17:43:00.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.526+0000 7f88ca000640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f889c06a460 0x7f889c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:00.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.527+0000 7f88ca000640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f889c06a460 0x7f889c06c920 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f88b4001050 tx=0x7f88b4001d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:00.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.530+0000 7f88c2000640 1 -- 172.21.15.39:0/3846368787 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f88bc0535b0 con 0x7f88c40b3270 2024-01-24T17:43:00.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.767+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88c40ae830 con 0x7f88c40b3270 2024-01-24T17:43:00.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.768+0000 7f88c2000640 1 -- 172.21.15.39:0/3846368787 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f88bc018090 con 0x7f88c40b3270 2024-01-24T17:43:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:00.770 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4688,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/648542221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":648542221},{"type":"v1","addr":"172.21.15.100:6837","nonce":648542221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/188830546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":188830546},{"type":"v1","addr":"172.21.15.100:6835","nonce":188830546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/4294097622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4294097622},{"type":"v1","addr":"172.21.15.39:6834","nonce":4294097622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:42:56.701344+0000","modified":"2024-01-24T17:43:00.247932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5175":{"gid":5175,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/3577904963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3577904963},{"type":"v1","addr":"172.21.15.39:6835","nonce":3577904963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-24T17:43:00.770 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-01-24T17:43:00.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.770+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f889c06a460 msgr2=0x7f889c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.770+0000 7f88cb400640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f889c06a460 0x7f889c06c920 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f88b4001050 tx=0x7f88b4001d40 comp rx=0 tx=0).stop 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.770+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88c40b3270 msgr2=0x7f88c40aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.770+0000 7f88cb400640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88c40b3270 0x7f88c40aeb80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f88c406ceb0 tx=0x7f88bc00d890 comp rx=0 tx=0).stop 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.771+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 shutdown_connections 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.771+0000 7f88cb400640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f889c06a460 0x7f889c06c920 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.771+0000 7f88cb400640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f88c40b3270 0x7f88c40aeb80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.771+0000 7f88cb400640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f88c40b39d0 0x7f88c40b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.771+0000 7f88cb400640 1 --2- 172.21.15.39:0/3846368787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f88c40c46c0 0x7f88c40c4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.771+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 >> 172.21.15.39:0/3846368787 conn(0x7f88c406a090 msgr2=0x7f88c40c5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:00.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.771+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 shutdown_connections 2024-01-24T17:43:00.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:00.771+0000 7f88cb400640 1 -- 172.21.15.39:0/3846368787 wait complete. 2024-01-24T17:43:00.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:01.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.011+0000 7f6dc0600640 1 Processor -- start 2024-01-24T17:43:01.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.011+0000 7f6dc0600640 1 -- start start 2024-01-24T17:43:01.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dc0600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db8159620 0x7f6db80ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dc0600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6db80b06d0 0x7f6db80cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dc0600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db80cad00 con 0x7f6db80710e0 2024-01-24T17:43:01.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dc0600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db8071650 con 0x7f6db80b06d0 2024-01-24T17:43:01.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dc0600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8071930 con 0x7f6db8159620 2024-01-24T17:43:01.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dbe600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6db80b06d0 0x7f6db80cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dbe600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6db80b06d0 0x7f6db80cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34942/0 (socket says 172.21.15.39:34942) 2024-01-24T17:43:01.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dbe600640 1 -- 172.21.15.39:0/1482174239 learned_addr learned my addr 172.21.15.39:0/1482174239 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:01.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.013+0000 7f6dbdc00640 1 --2- 172.21.15.39:0/1482174239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db8159620 0x7f6db80ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.014+0000 7f6dbe600640 1 -- 172.21.15.39:0/1482174239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db8159620 msgr2=0x7f6db80ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:01.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.014+0000 7f6dbe600640 1 --2- 172.21.15.39:0/1482174239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db8159620 0x7f6db80ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.014+0000 7f6dbe600640 1 -- 172.21.15.39:0/1482174239 >> v1:172.21.15.100:6789/0 conn(0x7f6db80710e0 legacy=0x7f6db80affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.014+0000 7f6dbe600640 1 -- 172.21.15.39:0/1482174239 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8071c10 con 0x7f6db80b06d0 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.015+0000 7f6dbe600640 1 --2- 172.21.15.39:0/1482174239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db80cc380 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009e50 tx=0x7f6db002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed28b36661cf8c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.015+0000 7f6db7e00640 1 -- 172.21.15.39:0/1482174239 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db002cb00 con 0x7f6db80b06d0 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.015+0000 7f6db7e00640 1 -- 172.21.15.39:0/1482174239 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db002cc60 con 0x7f6db80b06d0 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.015+0000 7f6db7e00640 1 -- 172.21.15.39:0/1482174239 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db00343f0 con 0x7f6db80b06d0 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.016+0000 7f6dc0600640 1 -- 172.21.15.39:0/1482174239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 msgr2=0x7f6db80cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.016+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1482174239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db80cc380 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009e50 tx=0x7f6db002bf60 comp rx=0 tx=0).stop 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.016+0000 7f6dc0600640 1 -- 172.21.15.39:0/1482174239 shutdown_connections 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.016+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1482174239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db80cc380 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.016+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1482174239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db8159620 0x7f6db80ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.016+0000 7f6dc0600640 1 -- 172.21.15.39:0/1482174239 >> 172.21.15.39:0/1482174239 conn(0x7f6db806a3e0 msgr2=0x7f6db806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:01.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.016+0000 7f6dc0600640 1 -- 172.21.15.39:0/1482174239 shutdown_connections 2024-01-24T17:43:01.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.016+0000 7f6dc0600640 1 -- 172.21.15.39:0/1482174239 wait complete. 2024-01-24T17:43:01.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.018+0000 7f6dc0600640 1 Processor -- start 2024-01-24T17:43:01.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.018+0000 7f6dc0600640 1 -- start start 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dc0600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db80710e0 0x7f6db81550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dc0600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db8153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dc0600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6db8159620 0x7f6db815a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dc0600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8155630 con 0x7f6db80b06d0 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dc0600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80c2c40 con 0x7f6db8159620 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dc0600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8154290 con 0x7f6db80710e0 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dbd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db8153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dbd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db8153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34944/0 (socket says 172.21.15.39:34944) 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.019+0000 7f6dbd200640 1 -- 172.21.15.39:0/1831202173 learned_addr learned my addr 172.21.15.39:0/1831202173 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6dbe600640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6db8159620 0x7f6db815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6dbdc00640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db80710e0 0x7f6db81550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6dbd200640 1 -- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db80710e0 msgr2=0x7f6db81550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6dbd200640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db80710e0 0x7f6db81550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6dbd200640 1 -- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6db8159620 msgr2=0x7f6db815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6dbd200640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6db8159620 0x7f6db815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6dbd200640 1 -- 172.21.15.39:0/1831202173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db0009b30 con 0x7f6db80b06d0 2024-01-24T17:43:01.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6dbd200640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db8153d50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6da000c400 tx=0x7f6da000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:01.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.020+0000 7f6db5600640 1 -- 172.21.15.39:0/1831202173 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da00052d0 con 0x7f6db80b06d0 2024-01-24T17:43:01.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.021+0000 7f6db5600640 1 -- 172.21.15.39:0/1831202173 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da0005430 con 0x7f6db80b06d0 2024-01-24T17:43:01.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.021+0000 7f6db5600640 1 -- 172.21.15.39:0/1831202173 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da0003830 con 0x7f6db80b06d0 2024-01-24T17:43:01.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.021+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db80c2030 con 0x7f6db80b06d0 2024-01-24T17:43:01.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.021+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db80c25f0 con 0x7f6db80b06d0 2024-01-24T17:43:01.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.022+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db806c840 con 0x7f6db80b06d0 2024-01-24T17:43:01.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.023+0000 7f6db5600640 1 -- 172.21.15.39:0/1831202173 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6da0003990 con 0x7f6db80b06d0 2024-01-24T17:43:01.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.024+0000 7f6db5600640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6d9006a460 0x7f6d9006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.024+0000 7f6db5600640 1 -- 172.21.15.39:0/1831202173 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6da0087eb0 con 0x7f6db80b06d0 2024-01-24T17:43:01.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.025+0000 7f6dbdc00640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6d9006a460 0x7f6d9006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.026+0000 7f6dbdc00640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6d9006a460 0x7f6d9006c920 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6da8006fd0 tx=0x7f6da8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:01.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.029+0000 7f6db5600640 1 -- 172.21.15.39:0/1831202173 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6da00534a0 con 0x7f6db80b06d0 2024-01-24T17:43:01.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.224+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6db80721a0 con 0x7f6db80b06d0 2024-01-24T17:43:01.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.227+0000 7f6db5600640 1 -- 172.21.15.39:0/1831202173 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f6da0056ac0 con 0x7f6db80b06d0 2024-01-24T17:43:01.229 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:01.229 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:57.705789+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:01.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6d9006a460 msgr2=0x7f6d9006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6d9006a460 0x7f6d9006c920 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6da8006fd0 tx=0x7f6da8008040 comp rx=0 tx=0).stop 2024-01-24T17:43:01.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 msgr2=0x7f6db8153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db8153d50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6da000c400 tx=0x7f6da000c8c0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 shutdown_connections 2024-01-24T17:43:01.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6db8159620 0x7f6db815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6d9006a460 0x7f6d9006c920 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6db80b06d0 0x7f6db8153d50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 --2- 172.21.15.39:0/1831202173 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6db80710e0 0x7f6db81550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 >> 172.21.15.39:0/1831202173 conn(0x7f6db806a3e0 msgr2=0x7f6db8158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:01.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 shutdown_connections 2024-01-24T17:43:01.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.230+0000 7f6dc0600640 1 -- 172.21.15.39:0/1831202173 wait complete. 2024-01-24T17:43:01.252 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-24T17:43:01.252 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:43:01.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.471+0000 7fc839000640 1 Processor -- start 2024-01-24T17:43:01.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.471+0000 7fc839000640 1 -- start start 2024-01-24T17:43:01.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc839000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc839000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc83415db60 0x7fc8340ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc839000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83415dfe0 con 0x7fc8340d0da0 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc839000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8340d0a90 con 0x7fc83415db60 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc839000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340cd760 con 0x7fc83415cbf0 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc832a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc83415db60 0x7fc8340ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc832a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc83415db60 0x7fc8340ce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34958/0 (socket says 172.21.15.39:34958) 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc832a00640 1 -- 172.21.15.39:0/1482977042 learned_addr learned my addr 172.21.15.39:0/1482977042 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.472+0000 7fc833e00640 1 --1- 172.21.15.39:0/1482977042 >> v1:172.21.15.39:6789/0 conn(0x7fc8340d0da0 0x7fc83415e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40156/0 (socket says 172.21.15.39:40156) 2024-01-24T17:43:01.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.473+0000 7fc833400640 1 --2- 172.21.15.39:0/1482977042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.473+0000 7fc832a00640 1 -- 172.21.15.39:0/1482977042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 msgr2=0x7fc8340d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:01.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.473+0000 7fc832a00640 1 --2- 172.21.15.39:0/1482977042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.473+0000 7fc832a00640 1 -- 172.21.15.39:0/1482977042 >> v1:172.21.15.39:6789/0 conn(0x7fc8340d0da0 legacy=0x7fc83415e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.473+0000 7fc832a00640 1 -- 172.21.15.39:0/1482977042 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340cda40 con 0x7fc83415db60 2024-01-24T17:43:01.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.474+0000 7fc832a00640 1 --2- 172.21.15.39:0/1482977042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc8340ce240 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009870 tx=0x7fc81c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a352ae7f0534226e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:01.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.474+0000 7fc832000640 1 -- 172.21.15.39:0/1482977042 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c030840 con 0x7fc83415db60 2024-01-24T17:43:01.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.474+0000 7fc832000640 1 -- 172.21.15.39:0/1482977042 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81c0309a0 con 0x7fc83415db60 2024-01-24T17:43:01.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.474+0000 7fc832000640 1 -- 172.21.15.39:0/1482977042 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c030d30 con 0x7fc83415db60 2024-01-24T17:43:01.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.475+0000 7fc839000640 1 -- 172.21.15.39:0/1482977042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 msgr2=0x7fc8340ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.475+0000 7fc839000640 1 --2- 172.21.15.39:0/1482977042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc8340ce240 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009870 tx=0x7fc81c02b980 comp rx=0 tx=0).stop 2024-01-24T17:43:01.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.475+0000 7fc839000640 1 -- 172.21.15.39:0/1482977042 shutdown_connections 2024-01-24T17:43:01.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.475+0000 7fc839000640 1 --2- 172.21.15.39:0/1482977042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc8340ce240 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.475+0000 7fc839000640 1 --2- 172.21.15.39:0/1482977042 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.475+0000 7fc839000640 1 -- 172.21.15.39:0/1482977042 >> 172.21.15.39:0/1482977042 conn(0x7fc83409bb90 msgr2=0x7fc8340d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:01.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.475+0000 7fc839000640 1 -- 172.21.15.39:0/1482977042 shutdown_connections 2024-01-24T17:43:01.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.475+0000 7fc839000640 1 -- 172.21.15.39:0/1482977042 wait complete. 2024-01-24T17:43:01.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.477+0000 7fc839000640 1 Processor -- start 2024-01-24T17:43:01.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.477+0000 7fc839000640 1 -- start start 2024-01-24T17:43:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc839000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8340d0da0 0x7fc834158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc839000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc839000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc83415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc839000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341f28f0 con 0x7fc83415db60 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc839000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834158e60 con 0x7fc8340d0da0 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc839000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340c9d90 con 0x7fc83415cbf0 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc833e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc83415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc833e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc83415b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34972/0 (socket says 172.21.15.39:34972) 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc833e00640 1 -- 172.21.15.39:0/2923651584 learned_addr learned my addr 172.21.15.39:0/2923651584 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.478+0000 7fc833400640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8340d0da0 0x7fc834158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc832a00640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc833e00640 1 -- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 msgr2=0x7fc8340c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc833e00640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc833e00640 1 -- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8340d0da0 msgr2=0x7fc834158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc833e00640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8340d0da0 0x7fc834158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc833e00640 1 -- 172.21.15.39:0/2923651584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c0094d0 con 0x7fc83415db60 2024-01-24T17:43:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc832a00640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc833e00640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc83415b970 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c730 tx=0x7fc82c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc82b400640 1 -- 172.21.15.39:0/2923651584 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c00ce40 con 0x7fc83415db60 2024-01-24T17:43:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc82b400640 1 -- 172.21.15.39:0/2923651584 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c00a640 con 0x7fc83415db60 2024-01-24T17:43:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8340ca070 con 0x7fc83415db60 2024-01-24T17:43:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.479+0000 7fc82b400640 1 -- 172.21.15.39:0/2923651584 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c00a9f0 con 0x7fc83415db60 2024-01-24T17:43:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.480+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8340ca630 con 0x7fc83415db60 2024-01-24T17:43:01.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.481+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8340ca820 con 0x7fc83415db60 2024-01-24T17:43:01.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.482+0000 7fc82b400640 1 -- 172.21.15.39:0/2923651584 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc82c00ab50 con 0x7fc83415db60 2024-01-24T17:43:01.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.482+0000 7fc82b400640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc80c06a430 0x7fc80c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:01.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.483+0000 7fc82b400640 1 -- 172.21.15.39:0/2923651584 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc82c087ff0 con 0x7fc83415db60 2024-01-24T17:43:01.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.483+0000 7fc833400640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc80c06a430 0x7fc80c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:01.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.484+0000 7fc833400640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc80c06a430 0x7fc80c06c8f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc82400b9f0 tx=0x7fc824008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:01.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.487+0000 7fc82b400640 1 -- 172.21.15.39:0/2923651584 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc82c0bacb0 con 0x7fc83415db60 2024-01-24T17:43:01.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.681+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc8340ce780 con 0x7fc80c06a430 2024-01-24T17:43:01.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.709+0000 7fc82b400640 1 -- 172.21.15.39:0/2923651584 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286965 (secure 0 0 0) 0x7fc82400b690 con 0x7fc80c06a430 2024-01-24T17:43:01.715 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:01.716 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:43:01.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.715+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc80c06a430 msgr2=0x7fc80c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.715+0000 7fc839000640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc80c06a430 0x7fc80c06c8f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc82400b9f0 tx=0x7fc824008040 comp rx=0 tx=0).stop 2024-01-24T17:43:01.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.715+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 msgr2=0x7fc83415b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:01.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.715+0000 7fc839000640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc83415b970 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c730 tx=0x7fc82c00cbf0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.716+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 shutdown_connections 2024-01-24T17:43:01.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.716+0000 7fc839000640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc80c06a430 0x7fc80c06c8f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.716+0000 7fc839000640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83415db60 0x7fc83415b970 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.716+0000 7fc839000640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc83415cbf0 0x7fc8340c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.716+0000 7fc839000640 1 --2- 172.21.15.39:0/2923651584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8340d0da0 0x7fc834158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:01.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.716+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 >> 172.21.15.39:0/2923651584 conn(0x7fc83409bb90 msgr2=0x7fc83406ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:01.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.716+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 shutdown_connections 2024-01-24T17:43:01.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:01.716+0000 7fc839000640 1 -- 172.21.15.39:0/2923651584 wait complete. 2024-01-24T17:43:01.738 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-01-24T17:43:00.239946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236680,"kb_used_data":4104,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754738040,"statfs":{"total":773094113280,"available":772851752960,"internally_reserved":0,"allocated":4202496,"data_stored":2332662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.028804"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743848+0000","last_change":"2024-01-24T17:42:56.737394+0000","last_active":"2024-01-24T17:42:56.743848+0000","last_peered":"2024-01-24T17:42:56.743848+0000","last_clean":"2024-01-24T17:42:56.743848+0000","last_became_active":"2024-01-24T17:42:56.734708+0000","last_became_peered":"2024-01-24T17:42:56.734708+0000","last_unstale":"2024-01-24T17:42:56.743848+0000","last_undegraded":"2024-01-24T17:42:56.743848+0000","last_fullsized":"2024-01-24T17:42:56.743848+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:55:06.270349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757692+0000","last_change":"2024-01-24T17:42:56.755719+0000","last_active":"2024-01-24T17:42:56.757692+0000","last_peered":"2024-01-24T17:42:56.757692+0000","last_clean":"2024-01-24T17:42:56.757692+0000","last_became_active":"2024-01-24T17:42:56.755372+0000","last_became_peered":"2024-01-24T17:42:56.755372+0000","last_unstale":"2024-01-24T17:42:56.757692+0000","last_undegraded":"2024-01-24T17:42:56.757692+0000","last_fullsized":"2024-01-24T17:42:56.757692+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:31:42.965801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743682+0000","last_change":"2024-01-24T17:42:56.740954+0000","last_active":"2024-01-24T17:42:56.743682+0000","last_peered":"2024-01-24T17:42:56.743682+0000","last_clean":"2024-01-24T17:42:56.743682+0000","last_became_active":"2024-01-24T17:42:56.740645+0000","last_became_peered":"2024-01-24T17:42:56.740645+0000","last_unstale":"2024-01-24T17:42:56.743682+0000","last_undegraded":"2024-01-24T17:42:56.743682+0000","last_fullsized":"2024-01-24T17:42:56.743682+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:47:00.411097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759386+0000","last_change":"2024-01-24T17:42:56.750080+0000","last_active":"2024-01-24T17:42:56.759386+0000","last_peered":"2024-01-24T17:42:56.759386+0000","last_clean":"2024-01-24T17:42:56.759386+0000","last_became_active":"2024-01-24T17:42:56.749778+0000","last_became_peered":"2024-01-24T17:42:56.749778+0000","last_unstale":"2024-01-24T17:42:56.759386+0000","last_undegraded":"2024-01-24T17:42:56.759386+0000","last_fullsized":"2024-01-24T17:42:56.759386+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:13:30.543960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.746981+0000","last_change":"2024-01-24T17:42:56.741092+0000","last_active":"2024-01-24T17:42:56.746981+0000","last_peered":"2024-01-24T17:42:56.746981+0000","last_clean":"2024-01-24T17:42:56.746981+0000","last_became_active":"2024-01-24T17:42:56.740425+0000","last_became_peered":"2024-01-24T17:42:56.740425+0000","last_unstale":"2024-01-24T17:42:56.746981+0000","last_undegraded":"2024-01-24T17:42:56.746981+0000","last_fullsized":"2024-01-24T17:42:56.746981+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:36.527103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752868+0000","last_change":"2024-01-24T17:42:56.748144+0000","last_active":"2024-01-24T17:42:56.752868+0000","last_peered":"2024-01-24T17:42:56.752868+0000","last_clean":"2024-01-24T17:42:56.752868+0000","last_became_active":"2024-01-24T17:42:56.747802+0000","last_became_peered":"2024-01-24T17:42:56.747802+0000","last_unstale":"2024-01-24T17:42:56.752868+0000","last_undegraded":"2024-01-24T17:42:56.752868+0000","last_fullsized":"2024-01-24T17:42:56.752868+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:20:51.894139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743963+0000","last_change":"2024-01-24T17:42:56.737351+0000","last_active":"2024-01-24T17:42:56.743963+0000","last_peered":"2024-01-24T17:42:56.743963+0000","last_clean":"2024-01-24T17:42:56.743963+0000","last_became_active":"2024-01-24T17:42:56.734410+0000","last_became_peered":"2024-01-24T17:42:56.734410+0000","last_unstale":"2024-01-24T17:42:56.743963+0000","last_undegraded":"2024-01-24T17:42:56.743963+0000","last_fullsized":"2024-01-24T17:42:56.743963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:51:25.559716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745609+0000","last_change":"2024-01-24T17:42:56.742905+0000","last_active":"2024-01-24T17:42:56.745609+0000","last_peered":"2024-01-24T17:42:56.745609+0000","last_clean":"2024-01-24T17:42:56.745609+0000","last_became_active":"2024-01-24T17:42:56.742634+0000","last_became_peered":"2024-01-24T17:42:56.742634+0000","last_unstale":"2024-01-24T17:42:56.745609+0000","last_undegraded":"2024-01-24T17:42:56.745609+0000","last_fullsized":"2024-01-24T17:42:56.745609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:14:23.221888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.755913+0000","last_change":"2024-01-24T17:42:56.755229+0000","last_active":"2024-01-24T17:42:56.755913+0000","last_peered":"2024-01-24T17:42:56.755913+0000","last_clean":"2024-01-24T17:42:56.755913+0000","last_became_active":"2024-01-24T17:42:56.754883+0000","last_became_peered":"2024-01-24T17:42:56.754883+0000","last_unstale":"2024-01-24T17:42:56.755913+0000","last_undegraded":"2024-01-24T17:42:56.755913+0000","last_fullsized":"2024-01-24T17:42:56.755913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:54:18.631844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757615+0000","last_change":"2024-01-24T17:42:56.749550+0000","last_active":"2024-01-24T17:42:56.757615+0000","last_peered":"2024-01-24T17:42:56.757615+0000","last_clean":"2024-01-24T17:42:56.757615+0000","last_became_active":"2024-01-24T17:42:56.749171+0000","last_became_peered":"2024-01-24T17:42:56.749171+0000","last_unstale":"2024-01-24T17:42:56.757615+0000","last_undegraded":"2024-01-24T17:42:56.757615+0000","last_fullsized":"2024-01-24T17:42:56.757615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:02:23.409598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752712+0000","last_change":"2024-01-24T17:42:56.751514+0000","last_active":"2024-01-24T17:42:56.752712+0000","last_peered":"2024-01-24T17:42:56.752712+0000","last_clean":"2024-01-24T17:42:56.752712+0000","last_became_active":"2024-01-24T17:42:56.751328+0000","last_became_peered":"2024-01-24T17:42:56.751328+0000","last_unstale":"2024-01-24T17:42:56.752712+0000","last_undegraded":"2024-01-24T17:42:56.752712+0000","last_fullsized":"2024-01-24T17:42:56.752712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:59:43.195369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745239+0000","last_change":"2024-01-24T17:42:56.743213+0000","last_active":"2024-01-24T17:42:56.745239+0000","last_peered":"2024-01-24T17:42:56.745239+0000","last_clean":"2024-01-24T17:42:56.745239+0000","last_became_active":"2024-01-24T17:42:56.742931+0000","last_became_peered":"2024-01-24T17:42:56.742931+0000","last_unstale":"2024-01-24T17:42:56.745239+0000","last_undegraded":"2024-01-24T17:42:56.745239+0000","last_fullsized":"2024-01-24T17:42:56.745239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:03:24.612012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760439+0000","last_change":"2024-01-24T17:42:56.758827+0000","last_active":"2024-01-24T17:42:56.760439+0000","last_peered":"2024-01-24T17:42:56.760439+0000","last_clean":"2024-01-24T17:42:56.760439+0000","last_became_active":"2024-01-24T17:42:56.758580+0000","last_became_peered":"2024-01-24T17:42:56.758580+0000","last_unstale":"2024-01-24T17:42:56.760439+0000","last_undegraded":"2024-01-24T17:42:56.760439+0000","last_fullsized":"2024-01-24T17:42:56.760439+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:14:23.221888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757223+0000","last_change":"2024-01-24T17:42:56.749418+0000","last_active":"2024-01-24T17:42:56.757223+0000","last_peered":"2024-01-24T17:42:56.757223+0000","last_clean":"2024-01-24T17:42:56.757223+0000","last_became_active":"2024-01-24T17:42:56.749066+0000","last_became_peered":"2024-01-24T17:42:56.749066+0000","last_unstale":"2024-01-24T17:42:56.757223+0000","last_undegraded":"2024-01-24T17:42:56.757223+0000","last_fullsized":"2024-01-24T17:42:56.757223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:12:29.477504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.737771+0000","last_change":"2024-01-24T17:42:56.736296+0000","last_active":"2024-01-24T17:42:56.737771+0000","last_peered":"2024-01-24T17:42:56.737771+0000","last_clean":"2024-01-24T17:42:56.737771+0000","last_became_active":"2024-01-24T17:42:56.735854+0000","last_became_peered":"2024-01-24T17:42:56.735854+0000","last_unstale":"2024-01-24T17:42:56.737771+0000","last_undegraded":"2024-01-24T17:42:56.737771+0000","last_fullsized":"2024-01-24T17:42:56.737771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:36.527103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.755662+0000","last_change":"2024-01-24T17:42:56.753426+0000","last_active":"2024-01-24T17:42:56.755662+0000","last_peered":"2024-01-24T17:42:56.755662+0000","last_clean":"2024-01-24T17:42:56.755662+0000","last_became_active":"2024-01-24T17:42:56.753112+0000","last_became_peered":"2024-01-24T17:42:56.753112+0000","last_unstale":"2024-01-24T17:42:56.755662+0000","last_undegraded":"2024-01-24T17:42:56.755662+0000","last_fullsized":"2024-01-24T17:42:56.755662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:53:28.617889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743412+0000","last_change":"2024-01-24T17:42:56.740102+0000","last_active":"2024-01-24T17:42:56.743412+0000","last_peered":"2024-01-24T17:42:56.743412+0000","last_clean":"2024-01-24T17:42:56.743412+0000","last_became_active":"2024-01-24T17:42:56.739843+0000","last_became_peered":"2024-01-24T17:42:56.739843+0000","last_unstale":"2024-01-24T17:42:56.743412+0000","last_undegraded":"2024-01-24T17:42:56.743412+0000","last_fullsized":"2024-01-24T17:42:56.743412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:33:44.616167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757751+0000","last_change":"2024-01-24T17:42:56.753010+0000","last_active":"2024-01-24T17:42:56.757751+0000","last_peered":"2024-01-24T17:42:56.757751+0000","last_clean":"2024-01-24T17:42:56.757751+0000","last_became_active":"2024-01-24T17:42:56.752673+0000","last_became_peered":"2024-01-24T17:42:56.752673+0000","last_unstale":"2024-01-24T17:42:56.757751+0000","last_undegraded":"2024-01-24T17:42:56.757751+0000","last_fullsized":"2024-01-24T17:42:56.757751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:43:09.446749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752509+0000","last_change":"2024-01-24T17:42:56.744035+0000","last_active":"2024-01-24T17:42:56.752509+0000","last_peered":"2024-01-24T17:42:56.752509+0000","last_clean":"2024-01-24T17:42:56.752509+0000","last_became_active":"2024-01-24T17:42:56.742467+0000","last_became_peered":"2024-01-24T17:42:56.742467+0000","last_unstale":"2024-01-24T17:42:56.752509+0000","last_undegraded":"2024-01-24T17:42:56.752509+0000","last_fullsized":"2024-01-24T17:42:56.752509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:21.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.741265+0000","last_change":"2024-01-24T17:42:56.737215+0000","last_active":"2024-01-24T17:42:56.741265+0000","last_peered":"2024-01-24T17:42:56.741265+0000","last_clean":"2024-01-24T17:42:56.741265+0000","last_became_active":"2024-01-24T17:42:56.736944+0000","last_became_peered":"2024-01-24T17:42:56.736944+0000","last_unstale":"2024-01-24T17:42:56.741265+0000","last_undegraded":"2024-01-24T17:42:56.741265+0000","last_fullsized":"2024-01-24T17:42:56.741265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:22:25.447077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.744334+0000","last_change":"2024-01-24T17:42:56.740365+0000","last_active":"2024-01-24T17:42:56.744334+0000","last_peered":"2024-01-24T17:42:56.744334+0000","last_clean":"2024-01-24T17:42:56.744334+0000","last_became_active":"2024-01-24T17:42:56.740091+0000","last_became_peered":"2024-01-24T17:42:56.740091+0000","last_unstale":"2024-01-24T17:42:56.744334+0000","last_undegraded":"2024-01-24T17:42:56.744334+0000","last_fullsized":"2024-01-24T17:42:56.744334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:38.514073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.756317+0000","last_change":"2024-01-24T17:42:56.755448+0000","last_active":"2024-01-24T17:42:56.756317+0000","last_peered":"2024-01-24T17:42:56.756317+0000","last_clean":"2024-01-24T17:42:56.756317+0000","last_became_active":"2024-01-24T17:42:56.755090+0000","last_became_peered":"2024-01-24T17:42:56.755090+0000","last_unstale":"2024-01-24T17:42:56.756317+0000","last_undegraded":"2024-01-24T17:42:56.756317+0000","last_fullsized":"2024-01-24T17:42:56.756317+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:31:42.965801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.747027+0000","last_change":"2024-01-24T17:42:56.740112+0000","last_active":"2024-01-24T17:42:56.747027+0000","last_peered":"2024-01-24T17:42:56.747027+0000","last_clean":"2024-01-24T17:42:56.747027+0000","last_became_active":"2024-01-24T17:42:56.738970+0000","last_became_peered":"2024-01-24T17:42:56.738970+0000","last_unstale":"2024-01-24T17:42:56.747027+0000","last_undegraded":"2024-01-24T17:42:56.747027+0000","last_fullsized":"2024-01-24T17:42:56.747027+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:51:25.559716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759298+0000","last_change":"2024-01-24T17:42:56.758311+0000","last_active":"2024-01-24T17:42:56.759298+0000","last_peered":"2024-01-24T17:42:56.759298+0000","last_clean":"2024-01-24T17:42:56.759298+0000","last_became_active":"2024-01-24T17:42:56.758101+0000","last_became_peered":"2024-01-24T17:42:56.758101+0000","last_unstale":"2024-01-24T17:42:56.759298+0000","last_undegraded":"2024-01-24T17:42:56.759298+0000","last_fullsized":"2024-01-24T17:42:56.759298+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:12:29.477504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.761072+0000","last_change":"2024-01-24T17:42:56.756728+0000","last_active":"2024-01-24T17:42:56.761072+0000","last_peered":"2024-01-24T17:42:56.761072+0000","last_clean":"2024-01-24T17:42:56.761072+0000","last_became_active":"2024-01-24T17:42:56.755944+0000","last_became_peered":"2024-01-24T17:42:56.755944+0000","last_unstale":"2024-01-24T17:42:56.761072+0000","last_undegraded":"2024-01-24T17:42:56.761072+0000","last_fullsized":"2024-01-24T17:42:56.761072+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:59:32.121764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757253+0000","last_change":"2024-01-24T17:42:56.748302+0000","last_active":"2024-01-24T17:42:56.757253+0000","last_peered":"2024-01-24T17:42:56.757253+0000","last_clean":"2024-01-24T17:42:56.757253+0000","last_became_active":"2024-01-24T17:42:56.748010+0000","last_became_peered":"2024-01-24T17:42:56.748010+0000","last_unstale":"2024-01-24T17:42:56.757253+0000","last_undegraded":"2024-01-24T17:42:56.757253+0000","last_fullsized":"2024-01-24T17:42:56.757253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:55.658075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.734484+0000","last_change":"2024-01-24T17:42:56.726873+0000","last_active":"2024-01-24T17:42:56.734484+0000","last_peered":"2024-01-24T17:42:56.734484+0000","last_clean":"2024-01-24T17:42:56.734484+0000","last_became_active":"2024-01-24T17:42:56.726133+0000","last_became_peered":"2024-01-24T17:42:56.726133+0000","last_unstale":"2024-01-24T17:42:56.734484+0000","last_undegraded":"2024-01-24T17:42:56.734484+0000","last_fullsized":"2024-01-24T17:42:56.734484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:55:06.270349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.755925+0000","last_change":"2024-01-24T17:42:56.754441+0000","last_active":"2024-01-24T17:42:56.755925+0000","last_peered":"2024-01-24T17:42:56.755925+0000","last_clean":"2024-01-24T17:42:56.755925+0000","last_became_active":"2024-01-24T17:42:56.754160+0000","last_became_peered":"2024-01-24T17:42:56.754160+0000","last_unstale":"2024-01-24T17:42:56.755925+0000","last_undegraded":"2024-01-24T17:42:56.755925+0000","last_fullsized":"2024-01-24T17:42:56.755925+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:36.375174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.755741+0000","last_change":"2024-01-24T17:42:56.753764+0000","last_active":"2024-01-24T17:42:56.755741+0000","last_peered":"2024-01-24T17:42:56.755741+0000","last_clean":"2024-01-24T17:42:56.755741+0000","last_became_active":"2024-01-24T17:42:56.753457+0000","last_became_peered":"2024-01-24T17:42:56.753457+0000","last_unstale":"2024-01-24T17:42:56.755741+0000","last_undegraded":"2024-01-24T17:42:56.755741+0000","last_fullsized":"2024-01-24T17:42:56.755741+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:06.149355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.731999+0000","last_change":"2024-01-24T17:42:56.728192+0000","last_active":"2024-01-24T17:42:56.731999+0000","last_peered":"2024-01-24T17:42:56.731999+0000","last_clean":"2024-01-24T17:42:56.731999+0000","last_became_active":"2024-01-24T17:42:56.727906+0000","last_became_peered":"2024-01-24T17:42:56.727906+0000","last_unstale":"2024-01-24T17:42:56.731999+0000","last_undegraded":"2024-01-24T17:42:56.731999+0000","last_fullsized":"2024-01-24T17:42:56.731999+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:14:23.221888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.744966+0000","last_change":"2024-01-24T17:42:56.742346+0000","last_active":"2024-01-24T17:42:56.744966+0000","last_peered":"2024-01-24T17:42:56.744966+0000","last_clean":"2024-01-24T17:42:56.744966+0000","last_became_active":"2024-01-24T17:42:56.742099+0000","last_became_peered":"2024-01-24T17:42:56.742099+0000","last_unstale":"2024-01-24T17:42:56.744966+0000","last_undegraded":"2024-01-24T17:42:56.744966+0000","last_fullsized":"2024-01-24T17:42:56.744966+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:36.527103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.737081+0000","last_change":"2024-01-24T17:42:56.734581+0000","last_active":"2024-01-24T17:42:56.737081+0000","last_peered":"2024-01-24T17:42:56.737081+0000","last_clean":"2024-01-24T17:42:56.737081+0000","last_became_active":"2024-01-24T17:42:56.734355+0000","last_became_peered":"2024-01-24T17:42:56.734355+0000","last_unstale":"2024-01-24T17:42:56.737081+0000","last_undegraded":"2024-01-24T17:42:56.737081+0000","last_fullsized":"2024-01-24T17:42:56.737081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:02:23.409598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.753220+0000","last_change":"2024-01-24T17:42:56.748649+0000","last_active":"2024-01-24T17:42:56.753220+0000","last_peered":"2024-01-24T17:42:56.753220+0000","last_clean":"2024-01-24T17:42:56.753220+0000","last_became_active":"2024-01-24T17:42:56.746870+0000","last_became_peered":"2024-01-24T17:42:56.746870+0000","last_unstale":"2024-01-24T17:42:56.753220+0000","last_undegraded":"2024-01-24T17:42:56.753220+0000","last_fullsized":"2024-01-24T17:42:56.753220+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:39.993826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.739499+0000","last_change":"2024-01-24T17:42:56.235391+0000","last_active":"2024-01-24T17:42:56.739499+0000","last_peered":"2024-01-24T17:42:56.739499+0000","last_clean":"2024-01-24T17:42:56.739499+0000","last_became_active":"2024-01-24T17:42:56.235025+0000","last_became_peered":"2024-01-24T17:42:56.235025+0000","last_unstale":"2024-01-24T17:42:56.739499+0000","last_undegraded":"2024-01-24T17:42:56.739499+0000","last_fullsized":"2024-01-24T17:42:56.739499+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:55:05.258720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.749616+0000","last_change":"2024-01-24T17:42:56.747675+0000","last_active":"2024-01-24T17:42:56.749616+0000","last_peered":"2024-01-24T17:42:56.749616+0000","last_clean":"2024-01-24T17:42:56.749616+0000","last_became_active":"2024-01-24T17:42:56.747420+0000","last_became_peered":"2024-01-24T17:42:56.747420+0000","last_unstale":"2024-01-24T17:42:56.749616+0000","last_undegraded":"2024-01-24T17:42:56.749616+0000","last_fullsized":"2024-01-24T17:42:56.749616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:53:28.617889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757175+0000","last_change":"2024-01-24T17:42:56.252947+0000","last_active":"2024-01-24T17:42:56.757175+0000","last_peered":"2024-01-24T17:42:56.757175+0000","last_clean":"2024-01-24T17:42:56.757175+0000","last_became_active":"2024-01-24T17:42:56.252591+0000","last_became_peered":"2024-01-24T17:42:56.252591+0000","last_unstale":"2024-01-24T17:42:56.757175+0000","last_undegraded":"2024-01-24T17:42:56.757175+0000","last_fullsized":"2024-01-24T17:42:56.757175+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:23:49.193498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759294+0000","last_change":"2024-01-24T17:42:56.758374+0000","last_active":"2024-01-24T17:42:56.759294+0000","last_peered":"2024-01-24T17:42:56.759294+0000","last_clean":"2024-01-24T17:42:56.759294+0000","last_became_active":"2024-01-24T17:42:56.758130+0000","last_became_peered":"2024-01-24T17:42:56.758130+0000","last_unstale":"2024-01-24T17:42:56.759294+0000","last_undegraded":"2024-01-24T17:42:56.759294+0000","last_fullsized":"2024-01-24T17:42:56.759294+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:55.658075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752440+0000","last_change":"2024-01-24T17:42:56.259106+0000","last_active":"2024-01-24T17:42:56.752440+0000","last_peered":"2024-01-24T17:42:56.752440+0000","last_clean":"2024-01-24T17:42:56.752440+0000","last_became_active":"2024-01-24T17:42:56.256653+0000","last_became_peered":"2024-01-24T17:42:56.256653+0000","last_unstale":"2024-01-24T17:42:56.752440+0000","last_undegraded":"2024-01-24T17:42:56.752440+0000","last_fullsized":"2024-01-24T17:42:56.752440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:43:56.262799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760147+0000","last_change":"2024-01-24T17:42:56.754905+0000","last_active":"2024-01-24T17:42:56.760147+0000","last_peered":"2024-01-24T17:42:56.760147+0000","last_clean":"2024-01-24T17:42:56.760147+0000","last_became_active":"2024-01-24T17:42:56.754600+0000","last_became_peered":"2024-01-24T17:42:56.754600+0000","last_unstale":"2024-01-24T17:42:56.760147+0000","last_undegraded":"2024-01-24T17:42:56.760147+0000","last_fullsized":"2024-01-24T17:42:56.760147+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:59:01.230583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759707+0000","last_change":"2024-01-24T17:42:56.256942+0000","last_active":"2024-01-24T17:42:56.759707+0000","last_peered":"2024-01-24T17:42:56.759707+0000","last_clean":"2024-01-24T17:42:56.759707+0000","last_became_active":"2024-01-24T17:42:56.255777+0000","last_became_peered":"2024-01-24T17:42:56.255777+0000","last_unstale":"2024-01-24T17:42:56.759707+0000","last_undegraded":"2024-01-24T17:42:56.759707+0000","last_fullsized":"2024-01-24T17:42:56.759707+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:20.984980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.750667+0000","last_change":"2024-01-24T17:42:56.749671+0000","last_active":"2024-01-24T17:42:56.750667+0000","last_peered":"2024-01-24T17:42:56.750667+0000","last_clean":"2024-01-24T17:42:56.750667+0000","last_became_active":"2024-01-24T17:42:56.749389+0000","last_became_peered":"2024-01-24T17:42:56.749389+0000","last_unstale":"2024-01-24T17:42:56.750667+0000","last_undegraded":"2024-01-24T17:42:56.750667+0000","last_fullsized":"2024-01-24T17:42:56.750667+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:20:51.894139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.739469+0000","last_change":"2024-01-24T17:42:56.252212+0000","last_active":"2024-01-24T17:42:56.739469+0000","last_peered":"2024-01-24T17:42:56.739469+0000","last_clean":"2024-01-24T17:42:56.739469+0000","last_became_active":"2024-01-24T17:42:56.251588+0000","last_became_peered":"2024-01-24T17:42:56.251588+0000","last_unstale":"2024-01-24T17:42:56.739469+0000","last_undegraded":"2024-01-24T17:42:56.739469+0000","last_fullsized":"2024-01-24T17:42:56.739469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:46:59.399468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.749681+0000","last_change":"2024-01-24T17:42:56.748278+0000","last_active":"2024-01-24T17:42:56.749681+0000","last_peered":"2024-01-24T17:42:56.749681+0000","last_clean":"2024-01-24T17:42:56.749681+0000","last_became_active":"2024-01-24T17:42:56.748053+0000","last_became_peered":"2024-01-24T17:42:56.748053+0000","last_unstale":"2024-01-24T17:42:56.749681+0000","last_undegraded":"2024-01-24T17:42:56.749681+0000","last_fullsized":"2024-01-24T17:42:56.749681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:06.149355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.748827+0000","last_change":"2024-01-24T17:42:56.247421+0000","last_active":"2024-01-24T17:42:56.748827+0000","last_peered":"2024-01-24T17:42:56.748827+0000","last_clean":"2024-01-24T17:42:56.748827+0000","last_became_active":"2024-01-24T17:42:56.247167+0000","last_became_peered":"2024-01-24T17:42:56.247167+0000","last_unstale":"2024-01-24T17:42:56.748827+0000","last_undegraded":"2024-01-24T17:42:56.748827+0000","last_fullsized":"2024-01-24T17:42:56.748827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:40:00.835921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.735330+0000","last_change":"2024-01-24T17:42:56.730877+0000","last_active":"2024-01-24T17:42:56.735330+0000","last_peered":"2024-01-24T17:42:56.735330+0000","last_clean":"2024-01-24T17:42:56.735330+0000","last_became_active":"2024-01-24T17:42:56.730600+0000","last_became_peered":"2024-01-24T17:42:56.730600+0000","last_unstale":"2024-01-24T17:42:56.735330+0000","last_undegraded":"2024-01-24T17:42:56.735330+0000","last_fullsized":"2024-01-24T17:42:56.735330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:03.834363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.731813+0000","last_change":"2024-01-24T17:42:56.251161+0000","last_active":"2024-01-24T17:42:56.731813+0000","last_peered":"2024-01-24T17:42:56.731813+0000","last_clean":"2024-01-24T17:42:56.731813+0000","last_became_active":"2024-01-24T17:42:56.250185+0000","last_became_peered":"2024-01-24T17:42:56.250185+0000","last_unstale":"2024-01-24T17:42:56.731813+0000","last_undegraded":"2024-01-24T17:42:56.731813+0000","last_fullsized":"2024-01-24T17:42:56.731813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:00.768831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757285+0000","last_change":"2024-01-24T17:42:56.745233+0000","last_active":"2024-01-24T17:42:56.757285+0000","last_peered":"2024-01-24T17:42:56.757285+0000","last_clean":"2024-01-24T17:42:56.757285+0000","last_became_active":"2024-01-24T17:42:56.744945+0000","last_became_peered":"2024-01-24T17:42:56.744945+0000","last_unstale":"2024-01-24T17:42:56.757285+0000","last_undegraded":"2024-01-24T17:42:56.757285+0000","last_fullsized":"2024-01-24T17:42:56.757285+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:39.993826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.722840+0000","last_change":"2024-01-24T17:42:56.258443+0000","last_active":"2024-01-24T17:42:56.722840+0000","last_peered":"2024-01-24T17:42:56.722840+0000","last_clean":"2024-01-24T17:42:56.722840+0000","last_became_active":"2024-01-24T17:42:56.257953+0000","last_became_peered":"2024-01-24T17:42:56.257953+0000","last_unstale":"2024-01-24T17:42:56.722840+0000","last_undegraded":"2024-01-24T17:42:56.722840+0000","last_fullsized":"2024-01-24T17:42:56.722840+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:43:56.262799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.750582+0000","last_change":"2024-01-24T17:42:56.748283+0000","last_active":"2024-01-24T17:42:56.750582+0000","last_peered":"2024-01-24T17:42:56.750582+0000","last_clean":"2024-01-24T17:42:56.750582+0000","last_became_active":"2024-01-24T17:42:56.747512+0000","last_became_peered":"2024-01-24T17:42:56.747512+0000","last_unstale":"2024-01-24T17:42:56.750582+0000","last_undegraded":"2024-01-24T17:42:56.750582+0000","last_fullsized":"2024-01-24T17:42:56.750582+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:43:17.406752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.715634+0000","last_change":"2024-01-24T17:42:56.238679+0000","last_active":"2024-01-24T17:42:56.715634+0000","last_peered":"2024-01-24T17:42:56.715634+0000","last_clean":"2024-01-24T17:42:56.715634+0000","last_became_active":"2024-01-24T17:42:56.237459+0000","last_became_peered":"2024-01-24T17:42:56.237459+0000","last_unstale":"2024-01-24T17:42:56.715634+0000","last_undegraded":"2024-01-24T17:42:56.715634+0000","last_fullsized":"2024-01-24T17:42:56.715634+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:20.984980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760091+0000","last_change":"2024-01-24T17:42:56.753985+0000","last_active":"2024-01-24T17:42:56.760091+0000","last_peered":"2024-01-24T17:42:56.760091+0000","last_clean":"2024-01-24T17:42:56.760091+0000","last_became_active":"2024-01-24T17:42:56.750382+0000","last_became_peered":"2024-01-24T17:42:56.750382+0000","last_unstale":"2024-01-24T17:42:56.760091+0000","last_undegraded":"2024-01-24T17:42:56.760091+0000","last_fullsized":"2024-01-24T17:42:56.760091+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:20:00.525823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.718485+0000","last_change":"2024-01-24T17:42:56.254962+0000","last_active":"2024-01-24T17:42:56.718485+0000","last_peered":"2024-01-24T17:42:56.718485+0000","last_clean":"2024-01-24T17:42:56.718485+0000","last_became_active":"2024-01-24T17:42:56.254502+0000","last_became_peered":"2024-01-24T17:42:56.254502+0000","last_unstale":"2024-01-24T17:42:56.718485+0000","last_undegraded":"2024-01-24T17:42:56.718485+0000","last_fullsized":"2024-01-24T17:42:56.718485+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:22:24.435448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759365+0000","last_change":"2024-01-24T17:42:56.749886+0000","last_active":"2024-01-24T17:42:56.759365+0000","last_peered":"2024-01-24T17:42:56.759365+0000","last_clean":"2024-01-24T17:42:56.759365+0000","last_became_active":"2024-01-24T17:42:56.749447+0000","last_became_peered":"2024-01-24T17:42:56.749447+0000","last_unstale":"2024-01-24T17:42:56.759365+0000","last_undegraded":"2024-01-24T17:42:56.759365+0000","last_fullsized":"2024-01-24T17:42:56.759365+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:43:38.170351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.729204+0000","last_change":"2024-01-24T17:42:56.249420+0000","last_active":"2024-01-24T17:42:56.729204+0000","last_peered":"2024-01-24T17:42:56.729204+0000","last_clean":"2024-01-24T17:42:56.729204+0000","last_became_active":"2024-01-24T17:42:56.248932+0000","last_became_peered":"2024-01-24T17:42:56.248932+0000","last_unstale":"2024-01-24T17:42:56.729204+0000","last_undegraded":"2024-01-24T17:42:56.729204+0000","last_fullsized":"2024-01-24T17:42:56.729204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:35.515474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752677+0000","last_change":"2024-01-24T17:42:56.751187+0000","last_active":"2024-01-24T17:42:56.752677+0000","last_peered":"2024-01-24T17:42:56.752677+0000","last_clean":"2024-01-24T17:42:56.752677+0000","last_became_active":"2024-01-24T17:42:56.750933+0000","last_became_peered":"2024-01-24T17:42:56.750933+0000","last_unstale":"2024-01-24T17:42:56.752677+0000","last_undegraded":"2024-01-24T17:42:56.752677+0000","last_fullsized":"2024-01-24T17:42:56.752677+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:12:25.015751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.766983+0000","last_change":"2024-01-24T17:42:56.258040+0000","last_active":"2024-01-24T17:42:56.766983+0000","last_peered":"2024-01-24T17:42:56.766983+0000","last_clean":"2024-01-24T17:42:56.766983+0000","last_became_active":"2024-01-24T17:42:56.256207+0000","last_became_peered":"2024-01-24T17:42:56.256207+0000","last_unstale":"2024-01-24T17:42:56.766983+0000","last_undegraded":"2024-01-24T17:42:56.766983+0000","last_fullsized":"2024-01-24T17:42:56.766983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:40:00.835921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.741117+0000","last_change":"2024-01-24T17:42:56.736381+0000","last_active":"2024-01-24T17:42:56.741117+0000","last_peered":"2024-01-24T17:42:56.741117+0000","last_clean":"2024-01-24T17:42:56.741117+0000","last_became_active":"2024-01-24T17:42:56.735687+0000","last_became_peered":"2024-01-24T17:42:56.735687+0000","last_unstale":"2024-01-24T17:42:56.741117+0000","last_undegraded":"2024-01-24T17:42:56.741117+0000","last_fullsized":"2024-01-24T17:42:56.741117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:15:06.409283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.740007+0000","last_change":"2024-01-24T17:42:56.249175+0000","last_active":"2024-01-24T17:42:56.740007+0000","last_peered":"2024-01-24T17:42:56.740007+0000","last_clean":"2024-01-24T17:42:56.740007+0000","last_became_active":"2024-01-24T17:42:56.248569+0000","last_became_peered":"2024-01-24T17:42:56.248569+0000","last_unstale":"2024-01-24T17:42:56.740007+0000","last_undegraded":"2024-01-24T17:42:56.740007+0000","last_fullsized":"2024-01-24T17:42:56.740007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:54:17.620215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.761127+0000","last_change":"2024-01-24T17:42:56.756710+0000","last_active":"2024-01-24T17:42:56.761127+0000","last_peered":"2024-01-24T17:42:56.761127+0000","last_clean":"2024-01-24T17:42:56.761127+0000","last_became_active":"2024-01-24T17:42:56.756115+0000","last_became_peered":"2024-01-24T17:42:56.756115+0000","last_unstale":"2024-01-24T17:42:56.761127+0000","last_undegraded":"2024-01-24T17:42:56.761127+0000","last_fullsized":"2024-01-24T17:42:56.761127+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:17:06.165401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.734588+0000","last_change":"2024-01-24T17:42:56.259428+0000","last_active":"2024-01-24T17:42:56.734588+0000","last_peered":"2024-01-24T17:42:56.734588+0000","last_clean":"2024-01-24T17:42:56.734588+0000","last_became_active":"2024-01-24T17:42:56.259025+0000","last_became_peered":"2024-01-24T17:42:56.259025+0000","last_unstale":"2024-01-24T17:42:56.734588+0000","last_undegraded":"2024-01-24T17:42:56.734588+0000","last_fullsized":"2024-01-24T17:42:56.734588+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:46:24.456601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.737431+0000","last_change":"2024-01-24T17:42:56.730653+0000","last_active":"2024-01-24T17:42:56.737431+0000","last_peered":"2024-01-24T17:42:56.737431+0000","last_clean":"2024-01-24T17:42:56.737431+0000","last_became_active":"2024-01-24T17:42:56.730364+0000","last_became_peered":"2024-01-24T17:42:56.730364+0000","last_unstale":"2024-01-24T17:42:56.737431+0000","last_undegraded":"2024-01-24T17:42:56.737431+0000","last_fullsized":"2024-01-24T17:42:56.737431+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:22:25.447077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.741047+0000","last_change":"2024-01-24T17:42:56.255206+0000","last_active":"2024-01-24T17:42:56.741047+0000","last_peered":"2024-01-24T17:42:56.741047+0000","last_clean":"2024-01-24T17:42:56.741047+0000","last_became_active":"2024-01-24T17:42:56.254920+0000","last_became_peered":"2024-01-24T17:42:56.254920+0000","last_unstale":"2024-01-24T17:42:56.741047+0000","last_undegraded":"2024-01-24T17:42:56.741047+0000","last_fullsized":"2024-01-24T17:42:56.741047+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:21:39.218288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.737110+0000","last_change":"2024-01-24T17:42:56.734296+0000","last_active":"2024-01-24T17:42:56.737110+0000","last_peered":"2024-01-24T17:42:56.737110+0000","last_clean":"2024-01-24T17:42:56.737110+0000","last_became_active":"2024-01-24T17:42:56.734049+0000","last_became_peered":"2024-01-24T17:42:56.734049+0000","last_unstale":"2024-01-24T17:42:56.737110+0000","last_undegraded":"2024-01-24T17:42:56.737110+0000","last_fullsized":"2024-01-24T17:42:56.737110+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:17:06.165401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.758398+0000","last_change":"2024-01-24T17:42:56.252287+0000","last_active":"2024-01-24T17:42:56.758398+0000","last_peered":"2024-01-24T17:42:56.758398+0000","last_clean":"2024-01-24T17:42:56.758398+0000","last_became_active":"2024-01-24T17:42:56.251984+0000","last_became_peered":"2024-01-24T17:42:56.251984+0000","last_unstale":"2024-01-24T17:42:56.758398+0000","last_undegraded":"2024-01-24T17:42:56.758398+0000","last_fullsized":"2024-01-24T17:42:56.758398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:18.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757428+0000","last_change":"2024-01-24T17:42:56.756812+0000","last_active":"2024-01-24T17:42:56.757428+0000","last_peered":"2024-01-24T17:42:56.757428+0000","last_clean":"2024-01-24T17:42:56.757428+0000","last_became_active":"2024-01-24T17:42:56.756607+0000","last_became_peered":"2024-01-24T17:42:56.756607+0000","last_unstale":"2024-01-24T17:42:56.757428+0000","last_undegraded":"2024-01-24T17:42:56.757428+0000","last_fullsized":"2024-01-24T17:42:56.757428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:58:31.111806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757462+0000","last_change":"2024-01-24T17:42:56.261245+0000","last_active":"2024-01-24T17:42:56.757462+0000","last_peered":"2024-01-24T17:42:56.757462+0000","last_clean":"2024-01-24T17:42:56.757462+0000","last_became_active":"2024-01-24T17:42:56.260901+0000","last_became_peered":"2024-01-24T17:42:56.260901+0000","last_unstale":"2024-01-24T17:42:56.757462+0000","last_undegraded":"2024-01-24T17:42:56.757462+0000","last_fullsized":"2024-01-24T17:42:56.757462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:10:44.352326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.735591+0000","last_change":"2024-01-24T17:42:56.733725+0000","last_active":"2024-01-24T17:42:56.735591+0000","last_peered":"2024-01-24T17:42:56.735591+0000","last_clean":"2024-01-24T17:42:56.735591+0000","last_became_active":"2024-01-24T17:42:56.732597+0000","last_became_peered":"2024-01-24T17:42:56.732597+0000","last_unstale":"2024-01-24T17:42:56.735591+0000","last_undegraded":"2024-01-24T17:42:56.735591+0000","last_fullsized":"2024-01-24T17:42:56.735591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:40:01.847550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.739214+0000","last_change":"2024-01-24T17:42:56.251773+0000","last_active":"2024-01-24T17:42:56.739214+0000","last_peered":"2024-01-24T17:42:56.739214+0000","last_clean":"2024-01-24T17:42:56.739214+0000","last_became_active":"2024-01-24T17:42:56.250308+0000","last_became_peered":"2024-01-24T17:42:56.250308+0000","last_unstale":"2024-01-24T17:42:56.739214+0000","last_undegraded":"2024-01-24T17:42:56.739214+0000","last_fullsized":"2024-01-24T17:42:56.739214+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:02.822734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745594+0000","last_change":"2024-01-24T17:42:56.737465+0000","last_active":"2024-01-24T17:42:56.745594+0000","last_peered":"2024-01-24T17:42:56.745594+0000","last_clean":"2024-01-24T17:42:56.745594+0000","last_became_active":"2024-01-24T17:42:56.733921+0000","last_became_peered":"2024-01-24T17:42:56.733921+0000","last_unstale":"2024-01-24T17:42:56.745594+0000","last_undegraded":"2024-01-24T17:42:56.745594+0000","last_fullsized":"2024-01-24T17:42:56.745594+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:53:28.617889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.725636+0000","last_change":"2024-01-24T17:42:56.243862+0000","last_active":"2024-01-24T17:42:56.725636+0000","last_peered":"2024-01-24T17:42:56.725636+0000","last_clean":"2024-01-24T17:42:56.725636+0000","last_became_active":"2024-01-24T17:42:56.243553+0000","last_became_peered":"2024-01-24T17:42:56.243553+0000","last_unstale":"2024-01-24T17:42:56.725636+0000","last_undegraded":"2024-01-24T17:42:56.725636+0000","last_fullsized":"2024-01-24T17:42:56.725636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:53:27.606261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752530+0000","last_change":"2024-01-24T17:42:56.744370+0000","last_active":"2024-01-24T17:42:56.752530+0000","last_peered":"2024-01-24T17:42:56.752530+0000","last_clean":"2024-01-24T17:42:56.752530+0000","last_became_active":"2024-01-24T17:42:56.743596+0000","last_became_peered":"2024-01-24T17:42:56.743596+0000","last_unstale":"2024-01-24T17:42:56.752530+0000","last_undegraded":"2024-01-24T17:42:56.752530+0000","last_fullsized":"2024-01-24T17:42:56.752530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:21:36.607556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.713804+0000","last_change":"2024-01-24T17:42:56.255970+0000","last_active":"2024-01-24T17:42:56.713804+0000","last_peered":"2024-01-24T17:42:56.713804+0000","last_clean":"2024-01-24T17:42:56.713804+0000","last_became_active":"2024-01-24T17:42:56.253488+0000","last_became_peered":"2024-01-24T17:42:56.253488+0000","last_unstale":"2024-01-24T17:42:56.713804+0000","last_undegraded":"2024-01-24T17:42:56.713804+0000","last_fullsized":"2024-01-24T17:42:56.713804+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:46:24.456601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.741170+0000","last_change":"2024-01-24T17:42:56.736521+0000","last_active":"2024-01-24T17:42:56.741170+0000","last_peered":"2024-01-24T17:42:56.741170+0000","last_clean":"2024-01-24T17:42:56.741170+0000","last_became_active":"2024-01-24T17:42:56.736055+0000","last_became_peered":"2024-01-24T17:42:56.736055+0000","last_unstale":"2024-01-24T17:42:56.741170+0000","last_undegraded":"2024-01-24T17:42:56.741170+0000","last_fullsized":"2024-01-24T17:42:56.741170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:56:25.512110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.758196+0000","last_change":"2024-01-24T17:42:56.258651+0000","last_active":"2024-01-24T17:42:56.758196+0000","last_peered":"2024-01-24T17:42:56.758196+0000","last_clean":"2024-01-24T17:42:56.758196+0000","last_became_active":"2024-01-24T17:42:56.257290+0000","last_became_peered":"2024-01-24T17:42:56.257290+0000","last_unstale":"2024-01-24T17:42:56.758196+0000","last_undegraded":"2024-01-24T17:42:56.758196+0000","last_fullsized":"2024-01-24T17:42:56.758196+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:43:08.435120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745470+0000","last_change":"2024-01-24T17:42:56.737159+0000","last_active":"2024-01-24T17:42:56.745470+0000","last_peered":"2024-01-24T17:42:56.745470+0000","last_clean":"2024-01-24T17:42:56.745470+0000","last_became_active":"2024-01-24T17:42:56.736595+0000","last_became_peered":"2024-01-24T17:42:56.736595+0000","last_unstale":"2024-01-24T17:42:56.745470+0000","last_undegraded":"2024-01-24T17:42:56.745470+0000","last_fullsized":"2024-01-24T17:42:56.745470+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:06.149355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.724401+0000","last_change":"2024-01-24T17:42:56.250980+0000","last_active":"2024-01-24T17:42:56.724401+0000","last_peered":"2024-01-24T17:42:56.724401+0000","last_clean":"2024-01-24T17:42:56.724401+0000","last_became_active":"2024-01-24T17:42:56.250686+0000","last_became_peered":"2024-01-24T17:42:56.250686+0000","last_unstale":"2024-01-24T17:42:56.724401+0000","last_undegraded":"2024-01-24T17:42:56.724401+0000","last_fullsized":"2024-01-24T17:42:56.724401+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:03:23.600383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760432+0000","last_change":"2024-01-24T17:42:56.758705+0000","last_active":"2024-01-24T17:42:56.760432+0000","last_peered":"2024-01-24T17:42:56.760432+0000","last_clean":"2024-01-24T17:42:56.760432+0000","last_became_active":"2024-01-24T17:42:56.758285+0000","last_became_peered":"2024-01-24T17:42:56.758285+0000","last_unstale":"2024-01-24T17:42:56.760432+0000","last_undegraded":"2024-01-24T17:42:56.760432+0000","last_fullsized":"2024-01-24T17:42:56.760432+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:36.375174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.727977+0000","last_change":"2024-01-24T17:42:56.244187+0000","last_active":"2024-01-24T17:42:56.727977+0000","last_peered":"2024-01-24T17:42:56.727977+0000","last_clean":"2024-01-24T17:42:56.727977+0000","last_became_active":"2024-01-24T17:42:56.243927+0000","last_became_peered":"2024-01-24T17:42:56.243927+0000","last_unstale":"2024-01-24T17:42:56.727977+0000","last_undegraded":"2024-01-24T17:42:56.727977+0000","last_fullsized":"2024-01-24T17:42:56.727977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:22:00.592365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.736062+0000","last_change":"2024-01-24T17:42:56.732222+0000","last_active":"2024-01-24T17:42:56.736062+0000","last_peered":"2024-01-24T17:42:56.736062+0000","last_clean":"2024-01-24T17:42:56.736062+0000","last_became_active":"2024-01-24T17:42:56.731328+0000","last_became_peered":"2024-01-24T17:42:56.731328+0000","last_unstale":"2024-01-24T17:42:56.736062+0000","last_undegraded":"2024-01-24T17:42:56.736062+0000","last_fullsized":"2024-01-24T17:42:56.736062+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:42:14.183902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.726998+0000","last_change":"2024-01-24T17:42:56.239354+0000","last_active":"2024-01-24T17:42:56.726998+0000","last_peered":"2024-01-24T17:42:56.726998+0000","last_clean":"2024-01-24T17:42:56.726998+0000","last_became_active":"2024-01-24T17:42:56.238986+0000","last_became_peered":"2024-01-24T17:42:56.238986+0000","last_unstale":"2024-01-24T17:42:56.726998+0000","last_undegraded":"2024-01-24T17:42:56.726998+0000","last_fullsized":"2024-01-24T17:42:56.726998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:37.502444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.753123+0000","last_change":"2024-01-24T17:42:56.747157+0000","last_active":"2024-01-24T17:42:56.753123+0000","last_peered":"2024-01-24T17:42:56.753123+0000","last_clean":"2024-01-24T17:42:56.753123+0000","last_became_active":"2024-01-24T17:42:56.746856+0000","last_became_peered":"2024-01-24T17:42:56.746856+0000","last_unstale":"2024-01-24T17:42:56.753123+0000","last_undegraded":"2024-01-24T17:42:56.753123+0000","last_fullsized":"2024-01-24T17:42:56.753123+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:46:25.468230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.748731+0000","last_change":"2024-01-24T17:42:56.254369+0000","last_active":"2024-01-24T17:42:56.748731+0000","last_peered":"2024-01-24T17:42:56.748731+0000","last_clean":"2024-01-24T17:42:56.748731+0000","last_became_active":"2024-01-24T17:42:56.253957+0000","last_became_peered":"2024-01-24T17:42:56.253957+0000","last_unstale":"2024-01-24T17:42:56.748731+0000","last_undegraded":"2024-01-24T17:42:56.748731+0000","last_fullsized":"2024-01-24T17:42:56.748731+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:03:23.600383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745740+0000","last_change":"2024-01-24T17:42:56.741877+0000","last_active":"2024-01-24T17:42:56.745740+0000","last_peered":"2024-01-24T17:42:56.745740+0000","last_clean":"2024-01-24T17:42:56.745740+0000","last_became_active":"2024-01-24T17:42:56.741614+0000","last_became_peered":"2024-01-24T17:42:56.741614+0000","last_unstale":"2024-01-24T17:42:56.745740+0000","last_undegraded":"2024-01-24T17:42:56.745740+0000","last_fullsized":"2024-01-24T17:42:56.745740+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:55:00.594223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.718739+0000","last_change":"2024-01-24T17:42:56.254749+0000","last_active":"2024-01-24T17:42:56.718739+0000","last_peered":"2024-01-24T17:42:56.718739+0000","last_clean":"2024-01-24T17:42:56.718739+0000","last_became_active":"2024-01-24T17:42:56.254146+0000","last_became_peered":"2024-01-24T17:42:56.254146+0000","last_unstale":"2024-01-24T17:42:56.718739+0000","last_undegraded":"2024-01-24T17:42:56.718739+0000","last_fullsized":"2024-01-24T17:42:56.718739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:56:24.500482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757705+0000","last_change":"2024-01-24T17:42:56.755210+0000","last_active":"2024-01-24T17:42:56.757705+0000","last_peered":"2024-01-24T17:42:56.757705+0000","last_clean":"2024-01-24T17:42:56.757705+0000","last_became_active":"2024-01-24T17:42:56.754010+0000","last_became_peered":"2024-01-24T17:42:56.754010+0000","last_unstale":"2024-01-24T17:42:56.757705+0000","last_undegraded":"2024-01-24T17:42:56.757705+0000","last_fullsized":"2024-01-24T17:42:56.757705+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:56:25.512110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.736194+0000","last_change":"2024-01-24T17:42:56.252599+0000","last_active":"2024-01-24T17:42:56.736194+0000","last_peered":"2024-01-24T17:42:56.736194+0000","last_clean":"2024-01-24T17:42:56.736194+0000","last_became_active":"2024-01-24T17:42:56.252343+0000","last_became_peered":"2024-01-24T17:42:56.252343+0000","last_unstale":"2024-01-24T17:42:56.736194+0000","last_undegraded":"2024-01-24T17:42:56.736194+0000","last_fullsized":"2024-01-24T17:42:56.736194+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:42:13.172274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.736883+0000","last_change":"2024-01-24T17:42:56.733135+0000","last_active":"2024-01-24T17:42:56.736883+0000","last_peered":"2024-01-24T17:42:56.736883+0000","last_clean":"2024-01-24T17:42:56.736883+0000","last_became_active":"2024-01-24T17:42:56.732855+0000","last_became_peered":"2024-01-24T17:42:56.732855+0000","last_unstale":"2024-01-24T17:42:56.736883+0000","last_undegraded":"2024-01-24T17:42:56.736883+0000","last_fullsized":"2024-01-24T17:42:56.736883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:55:06.270349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.729280+0000","last_change":"2024-01-24T17:42:56.249135+0000","last_active":"2024-01-24T17:42:56.729280+0000","last_peered":"2024-01-24T17:42:56.729280+0000","last_clean":"2024-01-24T17:42:56.729280+0000","last_became_active":"2024-01-24T17:42:56.248826+0000","last_became_peered":"2024-01-24T17:42:56.248826+0000","last_unstale":"2024-01-24T17:42:56.729280+0000","last_undegraded":"2024-01-24T17:42:56.729280+0000","last_fullsized":"2024-01-24T17:42:56.729280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:46:59.399468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760435+0000","last_change":"2024-01-24T17:42:56.755020+0000","last_active":"2024-01-24T17:42:56.760435+0000","last_peered":"2024-01-24T17:42:56.760435+0000","last_clean":"2024-01-24T17:42:56.760435+0000","last_became_active":"2024-01-24T17:42:56.754652+0000","last_became_peered":"2024-01-24T17:42:56.754652+0000","last_unstale":"2024-01-24T17:42:56.760435+0000","last_undegraded":"2024-01-24T17:42:56.760435+0000","last_fullsized":"2024-01-24T17:42:56.760435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:03:05.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.726275+0000","last_change":"2024-01-24T17:42:56.244865+0000","last_active":"2024-01-24T17:42:56.726275+0000","last_peered":"2024-01-24T17:42:56.726275+0000","last_clean":"2024-01-24T17:42:56.726275+0000","last_became_active":"2024-01-24T17:42:56.244578+0000","last_became_peered":"2024-01-24T17:42:56.244578+0000","last_unstale":"2024-01-24T17:42:56.726275+0000","last_undegraded":"2024-01-24T17:42:56.726275+0000","last_fullsized":"2024-01-24T17:42:56.726275+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:02.822734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759402+0000","last_change":"2024-01-24T17:42:56.750138+0000","last_active":"2024-01-24T17:42:56.759402+0000","last_peered":"2024-01-24T17:42:56.759402+0000","last_clean":"2024-01-24T17:42:56.759402+0000","last_became_active":"2024-01-24T17:42:56.749804+0000","last_became_peered":"2024-01-24T17:42:56.749804+0000","last_unstale":"2024-01-24T17:42:56.759402+0000","last_undegraded":"2024-01-24T17:42:56.759402+0000","last_fullsized":"2024-01-24T17:42:56.759402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:06.149355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.724479+0000","last_change":"2024-01-24T17:42:56.250644+0000","last_active":"2024-01-24T17:42:56.724479+0000","last_peered":"2024-01-24T17:42:56.724479+0000","last_clean":"2024-01-24T17:42:56.724479+0000","last_became_active":"2024-01-24T17:42:56.250358+0000","last_became_peered":"2024-01-24T17:42:56.250358+0000","last_unstale":"2024-01-24T17:42:56.724479+0000","last_undegraded":"2024-01-24T17:42:56.724479+0000","last_fullsized":"2024-01-24T17:42:56.724479+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:42:13.172274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743501+0000","last_change":"2024-01-24T17:42:56.740628+0000","last_active":"2024-01-24T17:42:56.743501+0000","last_peered":"2024-01-24T17:42:56.743501+0000","last_clean":"2024-01-24T17:42:56.743501+0000","last_became_active":"2024-01-24T17:42:56.739664+0000","last_became_peered":"2024-01-24T17:42:56.739664+0000","last_unstale":"2024-01-24T17:42:56.743501+0000","last_undegraded":"2024-01-24T17:42:56.743501+0000","last_fullsized":"2024-01-24T17:42:56.743501+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:55:01.851549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.736198+0000","last_change":"2024-01-24T17:42:56.252929+0000","last_active":"2024-01-24T17:42:56.736198+0000","last_peered":"2024-01-24T17:42:56.736198+0000","last_clean":"2024-01-24T17:42:56.736198+0000","last_became_active":"2024-01-24T17:42:56.252574+0000","last_became_peered":"2024-01-24T17:42:56.252574+0000","last_unstale":"2024-01-24T17:42:56.736198+0000","last_undegraded":"2024-01-24T17:42:56.736198+0000","last_fullsized":"2024-01-24T17:42:56.736198+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:20:50.882511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.746777+0000","last_change":"2024-01-24T17:42:56.740881+0000","last_active":"2024-01-24T17:42:56.746777+0000","last_peered":"2024-01-24T17:42:56.746777+0000","last_clean":"2024-01-24T17:42:56.746777+0000","last_became_active":"2024-01-24T17:42:56.738865+0000","last_became_peered":"2024-01-24T17:42:56.738865+0000","last_unstale":"2024-01-24T17:42:56.746777+0000","last_undegraded":"2024-01-24T17:42:56.746777+0000","last_fullsized":"2024-01-24T17:42:56.746777+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:36.375174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.718176+0000","last_change":"2024-01-24T17:42:56.242233+0000","last_active":"2024-01-24T17:42:56.718176+0000","last_peered":"2024-01-24T17:42:56.718176+0000","last_clean":"2024-01-24T17:42:56.718176+0000","last_became_active":"2024-01-24T17:42:56.241454+0000","last_became_peered":"2024-01-24T17:42:56.241454+0000","last_unstale":"2024-01-24T17:42:56.718176+0000","last_undegraded":"2024-01-24T17:42:56.718176+0000","last_fullsized":"2024-01-24T17:42:56.718176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:14:22.210259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.233645+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:42:56.233645+0000","last_peered":"2024-01-24T17:42:56.233645+0000","last_clean":"2024-01-24T17:42:56.233645+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:42:56.233645+0000","last_undegraded":"2024-01-24T17:42:56.233645+0000","last_fullsized":"2024-01-24T17:42:56.233645+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.099}]},{"osd":1,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.349},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.82199999999999995}]},{"osd":3,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.5760000000000001,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":1.0780000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.333,"15min":1.333},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.6459999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.585}]},{"osd":5,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.6739999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.175},{"interface":"front","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3719999999999999,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.0009999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.6960000000000002,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.157},{"interface":"front","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4830000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.675}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":385024,"data_stored":146665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.256},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.83499999999999996}]},{"osd":1,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.506}]},{"osd":2,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.0940000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.399}]},{"osd":4,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.446},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":1.4790000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.4849999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.3360000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.419},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.3680000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":736051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":1.123},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.232}]},{"osd":1,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.94399999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.282},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.1349999999999998,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.3979999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":3.036,"5min":3.036,"15min":3.036},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.345,"15min":1.345},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.361}]},{"osd":4,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":1.073}]},{"osd":6,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.302},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.4430000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.345},{"interface":"front","average":{"1min":1.657,"5min":1.657,"15min":1.657},"min":{"1min":1.262,"5min":1.262,"15min":1.262},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.4690000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":146687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.877}]},{"osd":1,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.75},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.1559999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.83599999999999997}]},{"osd":3,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.9409999999999998,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.365}]},{"osd":5,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.248},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.117}]},{"osd":6,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":3.0840000000000001,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":1.498},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.331}]},{"osd":7,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.29},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":4.3250000000000002,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.448}]}]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":389120,"data_stored":145877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":5.1929999999999996},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":8.6989999999999998}]},{"osd":1,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":3.7240000000000002}]},{"osd":2,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":3.4630000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":4.2919999999999998}]},{"osd":4,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":6.7469999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":3.4049999999999998}]},{"osd":5,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":4.3789999999999996},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":2.2149999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":2.0110000000000001,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":9.5440000000000005},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":4.8170000000000002}]},{"osd":7,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":6.6529999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31472,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":360448,"data_stored":138326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.94299999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.044},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":1.081}]},{"osd":4,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.3839999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.155}]},{"osd":5,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.101},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.532}]},{"osd":6,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.5900000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.617,"15min":1.617},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":3.343,"5min":3.343,"15min":3.343},"last":1.214},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.6970000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":970752,"data_stored":735539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.877}]},{"osd":2,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.61899999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":3.149,"5min":3.149,"15min":3.149},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.377},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.423}]},{"osd":5,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.9809999999999999,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":0.92300000000000004}]},{"osd":6,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.2850000000000001,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.153},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":0.94599999999999995}]},{"osd":7,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.484,"15min":1.484},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.8439999999999999,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.244},{"interface":"front","average":{"1min":1.581,"5min":1.581,"15min":1.581},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":3.4350000000000001,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.212}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31488,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340352,"statfs":{"total":96636764160,"available":96604520448,"internally_reserved":0,"allocated":376832,"data_stored":145191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.53900000000000003}]},{"osd":2,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":1.294},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.0529999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.1819999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.0880000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.9300000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.401}]},{"osd":6,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.552},{"interface":"front","average":{"1min":1.484,"5min":1.484,"15min":1.484},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.4430000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.159},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":3.1600000000000001,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.476}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:43:01.743 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:43:02.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.002+0000 7fbfa7e00640 1 Processor -- start 2024-01-24T17:43:02.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.003+0000 7fbfa7e00640 1 -- start start 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.004+0000 7fbfa7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbfa815cbf0 0x7fbfa80d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.004+0000 7fbfa7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbfa80d0da0 0x7fbfa80d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.004+0000 7fbfa7e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa815e010 con 0x7fbfa815db60 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.004+0000 7fbfa7e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa80d1750 con 0x7fbfa80d0da0 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.004+0000 7fbfa7e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa80d0a90 con 0x7fbfa815cbf0 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.005+0000 7fbfa7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbfa80d0da0 0x7fbfa80d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.005+0000 7fbfa7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbfa80d0da0 0x7fbfa80d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34984/0 (socket says 172.21.15.39:34984) 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.005+0000 7fbfa7400640 1 -- 172.21.15.39:0/2781252385 learned_addr learned my addr 172.21.15.39:0/2781252385 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.005+0000 7fbfa6a00640 1 --2- 172.21.15.39:0/2781252385 >> v2:172.21.15.100:3300/0 conn(0x7fbfa815cbf0 0x7fbfa80d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.005+0000 7fbfa7400640 1 -- 172.21.15.39:0/2781252385 >> v2:172.21.15.100:3300/0 conn(0x7fbfa815cbf0 msgr2=0x7fbfa80d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:02.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.005+0000 7fbfa7400640 1 --2- 172.21.15.39:0/2781252385 >> v2:172.21.15.100:3300/0 conn(0x7fbfa815cbf0 0x7fbfa80d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.005+0000 7fbfa7400640 1 -- 172.21.15.39:0/2781252385 >> v1:172.21.15.100:6789/0 conn(0x7fbfa815db60 legacy=0x7fbfa80ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:02.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.005+0000 7fbfa7400640 1 -- 172.21.15.39:0/2781252385 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa80cd760 con 0x7fbfa80d0da0 2024-01-24T17:43:02.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.006+0000 7fbfa7400640 1 --2- 172.21.15.39:0/2781252385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa80d1210 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbfa808d010 tx=0x7fbfa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f429cdc0f8630960 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:02.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa5600640 1 -- 172.21.15.39:0/2781252385 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0031830 con 0x7fbfa80d0da0 2024-01-24T17:43:02.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa5600640 1 -- 172.21.15.39:0/2781252385 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa0031990 con 0x7fbfa80d0da0 2024-01-24T17:43:02.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa5600640 1 -- 172.21.15.39:0/2781252385 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0031d20 con 0x7fbfa80d0da0 2024-01-24T17:43:02.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2781252385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 msgr2=0x7fbfa80d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2781252385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa80d1210 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbfa808d010 tx=0x7fbfa002b8d0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2781252385 shutdown_connections 2024-01-24T17:43:02.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2781252385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa80d1210 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2781252385 >> v2:172.21.15.100:3300/0 conn(0x7fbfa815cbf0 0x7fbfa80d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2781252385 >> 172.21.15.39:0/2781252385 conn(0x7fbfa809bb90 msgr2=0x7fbfa80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:02.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.007+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2781252385 shutdown_connections 2024-01-24T17:43:02.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.008+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2781252385 wait complete. 2024-01-24T17:43:02.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.009+0000 7fbfa7e00640 1 Processor -- start 2024-01-24T17:43:02.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.009+0000 7fbfa7e00640 1 -- start start 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.010+0000 7fbfa7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa8153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.010+0000 7fbfa7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfa815cbf0 0x7fbfa80c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.010+0000 7fbfa7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfa815db60 0x7fbfa8153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.010+0000 7fbfa7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80c9400 con 0x7fbfa80d0da0 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.010+0000 7fbfa7e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80b56d0 con 0x7fbfa815db60 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.010+0000 7fbfa7e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80c91f0 con 0x7fbfa815cbf0 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa8153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa8153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34990/0 (socket says 172.21.15.39:34990) 2024-01-24T17:43:02.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 -- 172.21.15.39:0/2018629624 learned_addr learned my addr 172.21.15.39:0/2018629624 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6000640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfa815cbf0 0x7fbfa80c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 -- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfa815cbf0 msgr2=0x7fbfa80c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa7400640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfa815db60 0x7fbfa8153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfa815cbf0 0x7fbfa80c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 -- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfa815db60 msgr2=0x7fbfa8153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfa815db60 0x7fbfa8153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 -- 172.21.15.39:0/2018629624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa00094d0 con 0x7fbfa80d0da0 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6000640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfa815cbf0 0x7fbfa80c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.011+0000 7fbfa6a00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa8153a90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800d800 tx=0x7fbf9800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.012+0000 7fbf9ea00640 1 -- 172.21.15.39:0/2018629624 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9800b6c0 con 0x7fbfa80d0da0 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.012+0000 7fbf9ea00640 1 -- 172.21.15.39:0/2018629624 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9800b820 con 0x7fbfa80d0da0 2024-01-24T17:43:02.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.012+0000 7fbf9ea00640 1 -- 172.21.15.39:0/2018629624 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9800bbd0 con 0x7fbfa80d0da0 2024-01-24T17:43:02.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.012+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa815b700 con 0x7fbfa80d0da0 2024-01-24T17:43:02.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.012+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa81538c0 con 0x7fbfa80d0da0 2024-01-24T17:43:02.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.013+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa804dff0 con 0x7fbfa80d0da0 2024-01-24T17:43:02.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.014+0000 7fbf9ea00640 1 -- 172.21.15.39:0/2018629624 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbf9800bd30 con 0x7fbfa80d0da0 2024-01-24T17:43:02.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.015+0000 7fbf9ea00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf8006a430 0x7fbf8006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.015+0000 7fbf9ea00640 1 -- 172.21.15.39:0/2018629624 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf98088ec0 con 0x7fbfa80d0da0 2024-01-24T17:43:02.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.016+0000 7fbfa6000640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf8006a430 0x7fbf8006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.017+0000 7fbfa6000640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf8006a430 0x7fbf8006c8f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbfa815fc60 tx=0x7fbf90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:02.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.020+0000 7fbf9ea00640 1 -- 172.21.15.39:0/2018629624 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbf98054410 con 0x7fbfa80d0da0 2024-01-24T17:43:02.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.211+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbfa80cdd20 con 0x7fbf8006a430 2024-01-24T17:43:02.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.233+0000 7fbf9ea00640 1 -- 172.21.15.39:0/2018629624 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286965 (secure 0 0 0) 0x7fbf9000d920 con 0x7fbf8006a430 2024-01-24T17:43:02.239 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:02.241 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf8006a430 msgr2=0x7fbf8006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf8006a430 0x7fbf8006c8f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbfa815fc60 tx=0x7fbf90009040 comp rx=0 tx=0).stop 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 msgr2=0x7fbfa8153a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa8153a90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800d800 tx=0x7fbf9800dcc0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 shutdown_connections 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfa815db60 0x7fbfa8153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfa815cbf0 0x7fbfa80c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf8006a430 0x7fbf8006c8f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 --2- 172.21.15.39:0/2018629624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfa80d0da0 0x7fbfa8153a90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 >> 172.21.15.39:0/2018629624 conn(0x7fbfa809bb90 msgr2=0x7fbfa81597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:02.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 shutdown_connections 2024-01-24T17:43:02.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.240+0000 7fbfa7e00640 1 -- 172.21.15.39:0/2018629624 wait complete. 2024-01-24T17:43:02.262 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-01-24T17:43:00.239946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236680,"kb_used_data":4104,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754738040,"statfs":{"total":773094113280,"available":772851752960,"internally_reserved":0,"allocated":4202496,"data_stored":2332662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.028804"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743848+0000","last_change":"2024-01-24T17:42:56.737394+0000","last_active":"2024-01-24T17:42:56.743848+0000","last_peered":"2024-01-24T17:42:56.743848+0000","last_clean":"2024-01-24T17:42:56.743848+0000","last_became_active":"2024-01-24T17:42:56.734708+0000","last_became_peered":"2024-01-24T17:42:56.734708+0000","last_unstale":"2024-01-24T17:42:56.743848+0000","last_undegraded":"2024-01-24T17:42:56.743848+0000","last_fullsized":"2024-01-24T17:42:56.743848+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:55:06.270349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757692+0000","last_change":"2024-01-24T17:42:56.755719+0000","last_active":"2024-01-24T17:42:56.757692+0000","last_peered":"2024-01-24T17:42:56.757692+0000","last_clean":"2024-01-24T17:42:56.757692+0000","last_became_active":"2024-01-24T17:42:56.755372+0000","last_became_peered":"2024-01-24T17:42:56.755372+0000","last_unstale":"2024-01-24T17:42:56.757692+0000","last_undegraded":"2024-01-24T17:42:56.757692+0000","last_fullsized":"2024-01-24T17:42:56.757692+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:31:42.965801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743682+0000","last_change":"2024-01-24T17:42:56.740954+0000","last_active":"2024-01-24T17:42:56.743682+0000","last_peered":"2024-01-24T17:42:56.743682+0000","last_clean":"2024-01-24T17:42:56.743682+0000","last_became_active":"2024-01-24T17:42:56.740645+0000","last_became_peered":"2024-01-24T17:42:56.740645+0000","last_unstale":"2024-01-24T17:42:56.743682+0000","last_undegraded":"2024-01-24T17:42:56.743682+0000","last_fullsized":"2024-01-24T17:42:56.743682+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:47:00.411097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759386+0000","last_change":"2024-01-24T17:42:56.750080+0000","last_active":"2024-01-24T17:42:56.759386+0000","last_peered":"2024-01-24T17:42:56.759386+0000","last_clean":"2024-01-24T17:42:56.759386+0000","last_became_active":"2024-01-24T17:42:56.749778+0000","last_became_peered":"2024-01-24T17:42:56.749778+0000","last_unstale":"2024-01-24T17:42:56.759386+0000","last_undegraded":"2024-01-24T17:42:56.759386+0000","last_fullsized":"2024-01-24T17:42:56.759386+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:13:30.543960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.746981+0000","last_change":"2024-01-24T17:42:56.741092+0000","last_active":"2024-01-24T17:42:56.746981+0000","last_peered":"2024-01-24T17:42:56.746981+0000","last_clean":"2024-01-24T17:42:56.746981+0000","last_became_active":"2024-01-24T17:42:56.740425+0000","last_became_peered":"2024-01-24T17:42:56.740425+0000","last_unstale":"2024-01-24T17:42:56.746981+0000","last_undegraded":"2024-01-24T17:42:56.746981+0000","last_fullsized":"2024-01-24T17:42:56.746981+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:36.527103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752868+0000","last_change":"2024-01-24T17:42:56.748144+0000","last_active":"2024-01-24T17:42:56.752868+0000","last_peered":"2024-01-24T17:42:56.752868+0000","last_clean":"2024-01-24T17:42:56.752868+0000","last_became_active":"2024-01-24T17:42:56.747802+0000","last_became_peered":"2024-01-24T17:42:56.747802+0000","last_unstale":"2024-01-24T17:42:56.752868+0000","last_undegraded":"2024-01-24T17:42:56.752868+0000","last_fullsized":"2024-01-24T17:42:56.752868+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:20:51.894139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743963+0000","last_change":"2024-01-24T17:42:56.737351+0000","last_active":"2024-01-24T17:42:56.743963+0000","last_peered":"2024-01-24T17:42:56.743963+0000","last_clean":"2024-01-24T17:42:56.743963+0000","last_became_active":"2024-01-24T17:42:56.734410+0000","last_became_peered":"2024-01-24T17:42:56.734410+0000","last_unstale":"2024-01-24T17:42:56.743963+0000","last_undegraded":"2024-01-24T17:42:56.743963+0000","last_fullsized":"2024-01-24T17:42:56.743963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:51:25.559716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745609+0000","last_change":"2024-01-24T17:42:56.742905+0000","last_active":"2024-01-24T17:42:56.745609+0000","last_peered":"2024-01-24T17:42:56.745609+0000","last_clean":"2024-01-24T17:42:56.745609+0000","last_became_active":"2024-01-24T17:42:56.742634+0000","last_became_peered":"2024-01-24T17:42:56.742634+0000","last_unstale":"2024-01-24T17:42:56.745609+0000","last_undegraded":"2024-01-24T17:42:56.745609+0000","last_fullsized":"2024-01-24T17:42:56.745609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:14:23.221888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.755913+0000","last_change":"2024-01-24T17:42:56.755229+0000","last_active":"2024-01-24T17:42:56.755913+0000","last_peered":"2024-01-24T17:42:56.755913+0000","last_clean":"2024-01-24T17:42:56.755913+0000","last_became_active":"2024-01-24T17:42:56.754883+0000","last_became_peered":"2024-01-24T17:42:56.754883+0000","last_unstale":"2024-01-24T17:42:56.755913+0000","last_undegraded":"2024-01-24T17:42:56.755913+0000","last_fullsized":"2024-01-24T17:42:56.755913+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:54:18.631844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757615+0000","last_change":"2024-01-24T17:42:56.749550+0000","last_active":"2024-01-24T17:42:56.757615+0000","last_peered":"2024-01-24T17:42:56.757615+0000","last_clean":"2024-01-24T17:42:56.757615+0000","last_became_active":"2024-01-24T17:42:56.749171+0000","last_became_peered":"2024-01-24T17:42:56.749171+0000","last_unstale":"2024-01-24T17:42:56.757615+0000","last_undegraded":"2024-01-24T17:42:56.757615+0000","last_fullsized":"2024-01-24T17:42:56.757615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:02:23.409598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752712+0000","last_change":"2024-01-24T17:42:56.751514+0000","last_active":"2024-01-24T17:42:56.752712+0000","last_peered":"2024-01-24T17:42:56.752712+0000","last_clean":"2024-01-24T17:42:56.752712+0000","last_became_active":"2024-01-24T17:42:56.751328+0000","last_became_peered":"2024-01-24T17:42:56.751328+0000","last_unstale":"2024-01-24T17:42:56.752712+0000","last_undegraded":"2024-01-24T17:42:56.752712+0000","last_fullsized":"2024-01-24T17:42:56.752712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:59:43.195369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745239+0000","last_change":"2024-01-24T17:42:56.743213+0000","last_active":"2024-01-24T17:42:56.745239+0000","last_peered":"2024-01-24T17:42:56.745239+0000","last_clean":"2024-01-24T17:42:56.745239+0000","last_became_active":"2024-01-24T17:42:56.742931+0000","last_became_peered":"2024-01-24T17:42:56.742931+0000","last_unstale":"2024-01-24T17:42:56.745239+0000","last_undegraded":"2024-01-24T17:42:56.745239+0000","last_fullsized":"2024-01-24T17:42:56.745239+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:03:24.612012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760439+0000","last_change":"2024-01-24T17:42:56.758827+0000","last_active":"2024-01-24T17:42:56.760439+0000","last_peered":"2024-01-24T17:42:56.760439+0000","last_clean":"2024-01-24T17:42:56.760439+0000","last_became_active":"2024-01-24T17:42:56.758580+0000","last_became_peered":"2024-01-24T17:42:56.758580+0000","last_unstale":"2024-01-24T17:42:56.760439+0000","last_undegraded":"2024-01-24T17:42:56.760439+0000","last_fullsized":"2024-01-24T17:42:56.760439+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:14:23.221888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757223+0000","last_change":"2024-01-24T17:42:56.749418+0000","last_active":"2024-01-24T17:42:56.757223+0000","last_peered":"2024-01-24T17:42:56.757223+0000","last_clean":"2024-01-24T17:42:56.757223+0000","last_became_active":"2024-01-24T17:42:56.749066+0000","last_became_peered":"2024-01-24T17:42:56.749066+0000","last_unstale":"2024-01-24T17:42:56.757223+0000","last_undegraded":"2024-01-24T17:42:56.757223+0000","last_fullsized":"2024-01-24T17:42:56.757223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:12:29.477504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.737771+0000","last_change":"2024-01-24T17:42:56.736296+0000","last_active":"2024-01-24T17:42:56.737771+0000","last_peered":"2024-01-24T17:42:56.737771+0000","last_clean":"2024-01-24T17:42:56.737771+0000","last_became_active":"2024-01-24T17:42:56.735854+0000","last_became_peered":"2024-01-24T17:42:56.735854+0000","last_unstale":"2024-01-24T17:42:56.737771+0000","last_undegraded":"2024-01-24T17:42:56.737771+0000","last_fullsized":"2024-01-24T17:42:56.737771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:36.527103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.755662+0000","last_change":"2024-01-24T17:42:56.753426+0000","last_active":"2024-01-24T17:42:56.755662+0000","last_peered":"2024-01-24T17:42:56.755662+0000","last_clean":"2024-01-24T17:42:56.755662+0000","last_became_active":"2024-01-24T17:42:56.753112+0000","last_became_peered":"2024-01-24T17:42:56.753112+0000","last_unstale":"2024-01-24T17:42:56.755662+0000","last_undegraded":"2024-01-24T17:42:56.755662+0000","last_fullsized":"2024-01-24T17:42:56.755662+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:53:28.617889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743412+0000","last_change":"2024-01-24T17:42:56.740102+0000","last_active":"2024-01-24T17:42:56.743412+0000","last_peered":"2024-01-24T17:42:56.743412+0000","last_clean":"2024-01-24T17:42:56.743412+0000","last_became_active":"2024-01-24T17:42:56.739843+0000","last_became_peered":"2024-01-24T17:42:56.739843+0000","last_unstale":"2024-01-24T17:42:56.743412+0000","last_undegraded":"2024-01-24T17:42:56.743412+0000","last_fullsized":"2024-01-24T17:42:56.743412+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:33:44.616167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757751+0000","last_change":"2024-01-24T17:42:56.753010+0000","last_active":"2024-01-24T17:42:56.757751+0000","last_peered":"2024-01-24T17:42:56.757751+0000","last_clean":"2024-01-24T17:42:56.757751+0000","last_became_active":"2024-01-24T17:42:56.752673+0000","last_became_peered":"2024-01-24T17:42:56.752673+0000","last_unstale":"2024-01-24T17:42:56.757751+0000","last_undegraded":"2024-01-24T17:42:56.757751+0000","last_fullsized":"2024-01-24T17:42:56.757751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:43:09.446749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752509+0000","last_change":"2024-01-24T17:42:56.744035+0000","last_active":"2024-01-24T17:42:56.752509+0000","last_peered":"2024-01-24T17:42:56.752509+0000","last_clean":"2024-01-24T17:42:56.752509+0000","last_became_active":"2024-01-24T17:42:56.742467+0000","last_became_peered":"2024-01-24T17:42:56.742467+0000","last_unstale":"2024-01-24T17:42:56.752509+0000","last_undegraded":"2024-01-24T17:42:56.752509+0000","last_fullsized":"2024-01-24T17:42:56.752509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:21.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.741265+0000","last_change":"2024-01-24T17:42:56.737215+0000","last_active":"2024-01-24T17:42:56.741265+0000","last_peered":"2024-01-24T17:42:56.741265+0000","last_clean":"2024-01-24T17:42:56.741265+0000","last_became_active":"2024-01-24T17:42:56.736944+0000","last_became_peered":"2024-01-24T17:42:56.736944+0000","last_unstale":"2024-01-24T17:42:56.741265+0000","last_undegraded":"2024-01-24T17:42:56.741265+0000","last_fullsized":"2024-01-24T17:42:56.741265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:22:25.447077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.744334+0000","last_change":"2024-01-24T17:42:56.740365+0000","last_active":"2024-01-24T17:42:56.744334+0000","last_peered":"2024-01-24T17:42:56.744334+0000","last_clean":"2024-01-24T17:42:56.744334+0000","last_became_active":"2024-01-24T17:42:56.740091+0000","last_became_peered":"2024-01-24T17:42:56.740091+0000","last_unstale":"2024-01-24T17:42:56.744334+0000","last_undegraded":"2024-01-24T17:42:56.744334+0000","last_fullsized":"2024-01-24T17:42:56.744334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:38.514073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.756317+0000","last_change":"2024-01-24T17:42:56.755448+0000","last_active":"2024-01-24T17:42:56.756317+0000","last_peered":"2024-01-24T17:42:56.756317+0000","last_clean":"2024-01-24T17:42:56.756317+0000","last_became_active":"2024-01-24T17:42:56.755090+0000","last_became_peered":"2024-01-24T17:42:56.755090+0000","last_unstale":"2024-01-24T17:42:56.756317+0000","last_undegraded":"2024-01-24T17:42:56.756317+0000","last_fullsized":"2024-01-24T17:42:56.756317+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:31:42.965801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.747027+0000","last_change":"2024-01-24T17:42:56.740112+0000","last_active":"2024-01-24T17:42:56.747027+0000","last_peered":"2024-01-24T17:42:56.747027+0000","last_clean":"2024-01-24T17:42:56.747027+0000","last_became_active":"2024-01-24T17:42:56.738970+0000","last_became_peered":"2024-01-24T17:42:56.738970+0000","last_unstale":"2024-01-24T17:42:56.747027+0000","last_undegraded":"2024-01-24T17:42:56.747027+0000","last_fullsized":"2024-01-24T17:42:56.747027+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:51:25.559716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759298+0000","last_change":"2024-01-24T17:42:56.758311+0000","last_active":"2024-01-24T17:42:56.759298+0000","last_peered":"2024-01-24T17:42:56.759298+0000","last_clean":"2024-01-24T17:42:56.759298+0000","last_became_active":"2024-01-24T17:42:56.758101+0000","last_became_peered":"2024-01-24T17:42:56.758101+0000","last_unstale":"2024-01-24T17:42:56.759298+0000","last_undegraded":"2024-01-24T17:42:56.759298+0000","last_fullsized":"2024-01-24T17:42:56.759298+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:12:29.477504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.761072+0000","last_change":"2024-01-24T17:42:56.756728+0000","last_active":"2024-01-24T17:42:56.761072+0000","last_peered":"2024-01-24T17:42:56.761072+0000","last_clean":"2024-01-24T17:42:56.761072+0000","last_became_active":"2024-01-24T17:42:56.755944+0000","last_became_peered":"2024-01-24T17:42:56.755944+0000","last_unstale":"2024-01-24T17:42:56.761072+0000","last_undegraded":"2024-01-24T17:42:56.761072+0000","last_fullsized":"2024-01-24T17:42:56.761072+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:59:32.121764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757253+0000","last_change":"2024-01-24T17:42:56.748302+0000","last_active":"2024-01-24T17:42:56.757253+0000","last_peered":"2024-01-24T17:42:56.757253+0000","last_clean":"2024-01-24T17:42:56.757253+0000","last_became_active":"2024-01-24T17:42:56.748010+0000","last_became_peered":"2024-01-24T17:42:56.748010+0000","last_unstale":"2024-01-24T17:42:56.757253+0000","last_undegraded":"2024-01-24T17:42:56.757253+0000","last_fullsized":"2024-01-24T17:42:56.757253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:55.658075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.734484+0000","last_change":"2024-01-24T17:42:56.726873+0000","last_active":"2024-01-24T17:42:56.734484+0000","last_peered":"2024-01-24T17:42:56.734484+0000","last_clean":"2024-01-24T17:42:56.734484+0000","last_became_active":"2024-01-24T17:42:56.726133+0000","last_became_peered":"2024-01-24T17:42:56.726133+0000","last_unstale":"2024-01-24T17:42:56.734484+0000","last_undegraded":"2024-01-24T17:42:56.734484+0000","last_fullsized":"2024-01-24T17:42:56.734484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:55:06.270349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.755925+0000","last_change":"2024-01-24T17:42:56.754441+0000","last_active":"2024-01-24T17:42:56.755925+0000","last_peered":"2024-01-24T17:42:56.755925+0000","last_clean":"2024-01-24T17:42:56.755925+0000","last_became_active":"2024-01-24T17:42:56.754160+0000","last_became_peered":"2024-01-24T17:42:56.754160+0000","last_unstale":"2024-01-24T17:42:56.755925+0000","last_undegraded":"2024-01-24T17:42:56.755925+0000","last_fullsized":"2024-01-24T17:42:56.755925+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:36.375174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.755741+0000","last_change":"2024-01-24T17:42:56.753764+0000","last_active":"2024-01-24T17:42:56.755741+0000","last_peered":"2024-01-24T17:42:56.755741+0000","last_clean":"2024-01-24T17:42:56.755741+0000","last_became_active":"2024-01-24T17:42:56.753457+0000","last_became_peered":"2024-01-24T17:42:56.753457+0000","last_unstale":"2024-01-24T17:42:56.755741+0000","last_undegraded":"2024-01-24T17:42:56.755741+0000","last_fullsized":"2024-01-24T17:42:56.755741+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:06.149355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.731999+0000","last_change":"2024-01-24T17:42:56.728192+0000","last_active":"2024-01-24T17:42:56.731999+0000","last_peered":"2024-01-24T17:42:56.731999+0000","last_clean":"2024-01-24T17:42:56.731999+0000","last_became_active":"2024-01-24T17:42:56.727906+0000","last_became_peered":"2024-01-24T17:42:56.727906+0000","last_unstale":"2024-01-24T17:42:56.731999+0000","last_undegraded":"2024-01-24T17:42:56.731999+0000","last_fullsized":"2024-01-24T17:42:56.731999+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:14:23.221888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.744966+0000","last_change":"2024-01-24T17:42:56.742346+0000","last_active":"2024-01-24T17:42:56.744966+0000","last_peered":"2024-01-24T17:42:56.744966+0000","last_clean":"2024-01-24T17:42:56.744966+0000","last_became_active":"2024-01-24T17:42:56.742099+0000","last_became_peered":"2024-01-24T17:42:56.742099+0000","last_unstale":"2024-01-24T17:42:56.744966+0000","last_undegraded":"2024-01-24T17:42:56.744966+0000","last_fullsized":"2024-01-24T17:42:56.744966+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:36.527103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.737081+0000","last_change":"2024-01-24T17:42:56.734581+0000","last_active":"2024-01-24T17:42:56.737081+0000","last_peered":"2024-01-24T17:42:56.737081+0000","last_clean":"2024-01-24T17:42:56.737081+0000","last_became_active":"2024-01-24T17:42:56.734355+0000","last_became_peered":"2024-01-24T17:42:56.734355+0000","last_unstale":"2024-01-24T17:42:56.737081+0000","last_undegraded":"2024-01-24T17:42:56.737081+0000","last_fullsized":"2024-01-24T17:42:56.737081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:02:23.409598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.753220+0000","last_change":"2024-01-24T17:42:56.748649+0000","last_active":"2024-01-24T17:42:56.753220+0000","last_peered":"2024-01-24T17:42:56.753220+0000","last_clean":"2024-01-24T17:42:56.753220+0000","last_became_active":"2024-01-24T17:42:56.746870+0000","last_became_peered":"2024-01-24T17:42:56.746870+0000","last_unstale":"2024-01-24T17:42:56.753220+0000","last_undegraded":"2024-01-24T17:42:56.753220+0000","last_fullsized":"2024-01-24T17:42:56.753220+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:39.993826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.739499+0000","last_change":"2024-01-24T17:42:56.235391+0000","last_active":"2024-01-24T17:42:56.739499+0000","last_peered":"2024-01-24T17:42:56.739499+0000","last_clean":"2024-01-24T17:42:56.739499+0000","last_became_active":"2024-01-24T17:42:56.235025+0000","last_became_peered":"2024-01-24T17:42:56.235025+0000","last_unstale":"2024-01-24T17:42:56.739499+0000","last_undegraded":"2024-01-24T17:42:56.739499+0000","last_fullsized":"2024-01-24T17:42:56.739499+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:55:05.258720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.749616+0000","last_change":"2024-01-24T17:42:56.747675+0000","last_active":"2024-01-24T17:42:56.749616+0000","last_peered":"2024-01-24T17:42:56.749616+0000","last_clean":"2024-01-24T17:42:56.749616+0000","last_became_active":"2024-01-24T17:42:56.747420+0000","last_became_peered":"2024-01-24T17:42:56.747420+0000","last_unstale":"2024-01-24T17:42:56.749616+0000","last_undegraded":"2024-01-24T17:42:56.749616+0000","last_fullsized":"2024-01-24T17:42:56.749616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:53:28.617889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757175+0000","last_change":"2024-01-24T17:42:56.252947+0000","last_active":"2024-01-24T17:42:56.757175+0000","last_peered":"2024-01-24T17:42:56.757175+0000","last_clean":"2024-01-24T17:42:56.757175+0000","last_became_active":"2024-01-24T17:42:56.252591+0000","last_became_peered":"2024-01-24T17:42:56.252591+0000","last_unstale":"2024-01-24T17:42:56.757175+0000","last_undegraded":"2024-01-24T17:42:56.757175+0000","last_fullsized":"2024-01-24T17:42:56.757175+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:23:49.193498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759294+0000","last_change":"2024-01-24T17:42:56.758374+0000","last_active":"2024-01-24T17:42:56.759294+0000","last_peered":"2024-01-24T17:42:56.759294+0000","last_clean":"2024-01-24T17:42:56.759294+0000","last_became_active":"2024-01-24T17:42:56.758130+0000","last_became_peered":"2024-01-24T17:42:56.758130+0000","last_unstale":"2024-01-24T17:42:56.759294+0000","last_undegraded":"2024-01-24T17:42:56.759294+0000","last_fullsized":"2024-01-24T17:42:56.759294+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:55.658075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752440+0000","last_change":"2024-01-24T17:42:56.259106+0000","last_active":"2024-01-24T17:42:56.752440+0000","last_peered":"2024-01-24T17:42:56.752440+0000","last_clean":"2024-01-24T17:42:56.752440+0000","last_became_active":"2024-01-24T17:42:56.256653+0000","last_became_peered":"2024-01-24T17:42:56.256653+0000","last_unstale":"2024-01-24T17:42:56.752440+0000","last_undegraded":"2024-01-24T17:42:56.752440+0000","last_fullsized":"2024-01-24T17:42:56.752440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:43:56.262799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760147+0000","last_change":"2024-01-24T17:42:56.754905+0000","last_active":"2024-01-24T17:42:56.760147+0000","last_peered":"2024-01-24T17:42:56.760147+0000","last_clean":"2024-01-24T17:42:56.760147+0000","last_became_active":"2024-01-24T17:42:56.754600+0000","last_became_peered":"2024-01-24T17:42:56.754600+0000","last_unstale":"2024-01-24T17:42:56.760147+0000","last_undegraded":"2024-01-24T17:42:56.760147+0000","last_fullsized":"2024-01-24T17:42:56.760147+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:59:01.230583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759707+0000","last_change":"2024-01-24T17:42:56.256942+0000","last_active":"2024-01-24T17:42:56.759707+0000","last_peered":"2024-01-24T17:42:56.759707+0000","last_clean":"2024-01-24T17:42:56.759707+0000","last_became_active":"2024-01-24T17:42:56.255777+0000","last_became_peered":"2024-01-24T17:42:56.255777+0000","last_unstale":"2024-01-24T17:42:56.759707+0000","last_undegraded":"2024-01-24T17:42:56.759707+0000","last_fullsized":"2024-01-24T17:42:56.759707+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:20.984980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.750667+0000","last_change":"2024-01-24T17:42:56.749671+0000","last_active":"2024-01-24T17:42:56.750667+0000","last_peered":"2024-01-24T17:42:56.750667+0000","last_clean":"2024-01-24T17:42:56.750667+0000","last_became_active":"2024-01-24T17:42:56.749389+0000","last_became_peered":"2024-01-24T17:42:56.749389+0000","last_unstale":"2024-01-24T17:42:56.750667+0000","last_undegraded":"2024-01-24T17:42:56.750667+0000","last_fullsized":"2024-01-24T17:42:56.750667+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:20:51.894139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.739469+0000","last_change":"2024-01-24T17:42:56.252212+0000","last_active":"2024-01-24T17:42:56.739469+0000","last_peered":"2024-01-24T17:42:56.739469+0000","last_clean":"2024-01-24T17:42:56.739469+0000","last_became_active":"2024-01-24T17:42:56.251588+0000","last_became_peered":"2024-01-24T17:42:56.251588+0000","last_unstale":"2024-01-24T17:42:56.739469+0000","last_undegraded":"2024-01-24T17:42:56.739469+0000","last_fullsized":"2024-01-24T17:42:56.739469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:46:59.399468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.749681+0000","last_change":"2024-01-24T17:42:56.748278+0000","last_active":"2024-01-24T17:42:56.749681+0000","last_peered":"2024-01-24T17:42:56.749681+0000","last_clean":"2024-01-24T17:42:56.749681+0000","last_became_active":"2024-01-24T17:42:56.748053+0000","last_became_peered":"2024-01-24T17:42:56.748053+0000","last_unstale":"2024-01-24T17:42:56.749681+0000","last_undegraded":"2024-01-24T17:42:56.749681+0000","last_fullsized":"2024-01-24T17:42:56.749681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:06.149355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.748827+0000","last_change":"2024-01-24T17:42:56.247421+0000","last_active":"2024-01-24T17:42:56.748827+0000","last_peered":"2024-01-24T17:42:56.748827+0000","last_clean":"2024-01-24T17:42:56.748827+0000","last_became_active":"2024-01-24T17:42:56.247167+0000","last_became_peered":"2024-01-24T17:42:56.247167+0000","last_unstale":"2024-01-24T17:42:56.748827+0000","last_undegraded":"2024-01-24T17:42:56.748827+0000","last_fullsized":"2024-01-24T17:42:56.748827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:40:00.835921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.735330+0000","last_change":"2024-01-24T17:42:56.730877+0000","last_active":"2024-01-24T17:42:56.735330+0000","last_peered":"2024-01-24T17:42:56.735330+0000","last_clean":"2024-01-24T17:42:56.735330+0000","last_became_active":"2024-01-24T17:42:56.730600+0000","last_became_peered":"2024-01-24T17:42:56.730600+0000","last_unstale":"2024-01-24T17:42:56.735330+0000","last_undegraded":"2024-01-24T17:42:56.735330+0000","last_fullsized":"2024-01-24T17:42:56.735330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:03.834363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.731813+0000","last_change":"2024-01-24T17:42:56.251161+0000","last_active":"2024-01-24T17:42:56.731813+0000","last_peered":"2024-01-24T17:42:56.731813+0000","last_clean":"2024-01-24T17:42:56.731813+0000","last_became_active":"2024-01-24T17:42:56.250185+0000","last_became_peered":"2024-01-24T17:42:56.250185+0000","last_unstale":"2024-01-24T17:42:56.731813+0000","last_undegraded":"2024-01-24T17:42:56.731813+0000","last_fullsized":"2024-01-24T17:42:56.731813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:00.768831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757285+0000","last_change":"2024-01-24T17:42:56.745233+0000","last_active":"2024-01-24T17:42:56.757285+0000","last_peered":"2024-01-24T17:42:56.757285+0000","last_clean":"2024-01-24T17:42:56.757285+0000","last_became_active":"2024-01-24T17:42:56.744945+0000","last_became_peered":"2024-01-24T17:42:56.744945+0000","last_unstale":"2024-01-24T17:42:56.757285+0000","last_undegraded":"2024-01-24T17:42:56.757285+0000","last_fullsized":"2024-01-24T17:42:56.757285+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:39.993826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.722840+0000","last_change":"2024-01-24T17:42:56.258443+0000","last_active":"2024-01-24T17:42:56.722840+0000","last_peered":"2024-01-24T17:42:56.722840+0000","last_clean":"2024-01-24T17:42:56.722840+0000","last_became_active":"2024-01-24T17:42:56.257953+0000","last_became_peered":"2024-01-24T17:42:56.257953+0000","last_unstale":"2024-01-24T17:42:56.722840+0000","last_undegraded":"2024-01-24T17:42:56.722840+0000","last_fullsized":"2024-01-24T17:42:56.722840+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:43:56.262799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.750582+0000","last_change":"2024-01-24T17:42:56.748283+0000","last_active":"2024-01-24T17:42:56.750582+0000","last_peered":"2024-01-24T17:42:56.750582+0000","last_clean":"2024-01-24T17:42:56.750582+0000","last_became_active":"2024-01-24T17:42:56.747512+0000","last_became_peered":"2024-01-24T17:42:56.747512+0000","last_unstale":"2024-01-24T17:42:56.750582+0000","last_undegraded":"2024-01-24T17:42:56.750582+0000","last_fullsized":"2024-01-24T17:42:56.750582+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:43:17.406752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.715634+0000","last_change":"2024-01-24T17:42:56.238679+0000","last_active":"2024-01-24T17:42:56.715634+0000","last_peered":"2024-01-24T17:42:56.715634+0000","last_clean":"2024-01-24T17:42:56.715634+0000","last_became_active":"2024-01-24T17:42:56.237459+0000","last_became_peered":"2024-01-24T17:42:56.237459+0000","last_unstale":"2024-01-24T17:42:56.715634+0000","last_undegraded":"2024-01-24T17:42:56.715634+0000","last_fullsized":"2024-01-24T17:42:56.715634+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:59:20.984980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760091+0000","last_change":"2024-01-24T17:42:56.753985+0000","last_active":"2024-01-24T17:42:56.760091+0000","last_peered":"2024-01-24T17:42:56.760091+0000","last_clean":"2024-01-24T17:42:56.760091+0000","last_became_active":"2024-01-24T17:42:56.750382+0000","last_became_peered":"2024-01-24T17:42:56.750382+0000","last_unstale":"2024-01-24T17:42:56.760091+0000","last_undegraded":"2024-01-24T17:42:56.760091+0000","last_fullsized":"2024-01-24T17:42:56.760091+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:20:00.525823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.718485+0000","last_change":"2024-01-24T17:42:56.254962+0000","last_active":"2024-01-24T17:42:56.718485+0000","last_peered":"2024-01-24T17:42:56.718485+0000","last_clean":"2024-01-24T17:42:56.718485+0000","last_became_active":"2024-01-24T17:42:56.254502+0000","last_became_peered":"2024-01-24T17:42:56.254502+0000","last_unstale":"2024-01-24T17:42:56.718485+0000","last_undegraded":"2024-01-24T17:42:56.718485+0000","last_fullsized":"2024-01-24T17:42:56.718485+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:22:24.435448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759365+0000","last_change":"2024-01-24T17:42:56.749886+0000","last_active":"2024-01-24T17:42:56.759365+0000","last_peered":"2024-01-24T17:42:56.759365+0000","last_clean":"2024-01-24T17:42:56.759365+0000","last_became_active":"2024-01-24T17:42:56.749447+0000","last_became_peered":"2024-01-24T17:42:56.749447+0000","last_unstale":"2024-01-24T17:42:56.759365+0000","last_undegraded":"2024-01-24T17:42:56.759365+0000","last_fullsized":"2024-01-24T17:42:56.759365+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:43:38.170351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.729204+0000","last_change":"2024-01-24T17:42:56.249420+0000","last_active":"2024-01-24T17:42:56.729204+0000","last_peered":"2024-01-24T17:42:56.729204+0000","last_clean":"2024-01-24T17:42:56.729204+0000","last_became_active":"2024-01-24T17:42:56.248932+0000","last_became_peered":"2024-01-24T17:42:56.248932+0000","last_unstale":"2024-01-24T17:42:56.729204+0000","last_undegraded":"2024-01-24T17:42:56.729204+0000","last_fullsized":"2024-01-24T17:42:56.729204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:35.515474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752677+0000","last_change":"2024-01-24T17:42:56.751187+0000","last_active":"2024-01-24T17:42:56.752677+0000","last_peered":"2024-01-24T17:42:56.752677+0000","last_clean":"2024-01-24T17:42:56.752677+0000","last_became_active":"2024-01-24T17:42:56.750933+0000","last_became_peered":"2024-01-24T17:42:56.750933+0000","last_unstale":"2024-01-24T17:42:56.752677+0000","last_undegraded":"2024-01-24T17:42:56.752677+0000","last_fullsized":"2024-01-24T17:42:56.752677+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:12:25.015751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.766983+0000","last_change":"2024-01-24T17:42:56.258040+0000","last_active":"2024-01-24T17:42:56.766983+0000","last_peered":"2024-01-24T17:42:56.766983+0000","last_clean":"2024-01-24T17:42:56.766983+0000","last_became_active":"2024-01-24T17:42:56.256207+0000","last_became_peered":"2024-01-24T17:42:56.256207+0000","last_unstale":"2024-01-24T17:42:56.766983+0000","last_undegraded":"2024-01-24T17:42:56.766983+0000","last_fullsized":"2024-01-24T17:42:56.766983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:40:00.835921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.741117+0000","last_change":"2024-01-24T17:42:56.736381+0000","last_active":"2024-01-24T17:42:56.741117+0000","last_peered":"2024-01-24T17:42:56.741117+0000","last_clean":"2024-01-24T17:42:56.741117+0000","last_became_active":"2024-01-24T17:42:56.735687+0000","last_became_peered":"2024-01-24T17:42:56.735687+0000","last_unstale":"2024-01-24T17:42:56.741117+0000","last_undegraded":"2024-01-24T17:42:56.741117+0000","last_fullsized":"2024-01-24T17:42:56.741117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:15:06.409283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.740007+0000","last_change":"2024-01-24T17:42:56.249175+0000","last_active":"2024-01-24T17:42:56.740007+0000","last_peered":"2024-01-24T17:42:56.740007+0000","last_clean":"2024-01-24T17:42:56.740007+0000","last_became_active":"2024-01-24T17:42:56.248569+0000","last_became_peered":"2024-01-24T17:42:56.248569+0000","last_unstale":"2024-01-24T17:42:56.740007+0000","last_undegraded":"2024-01-24T17:42:56.740007+0000","last_fullsized":"2024-01-24T17:42:56.740007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:54:17.620215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.761127+0000","last_change":"2024-01-24T17:42:56.756710+0000","last_active":"2024-01-24T17:42:56.761127+0000","last_peered":"2024-01-24T17:42:56.761127+0000","last_clean":"2024-01-24T17:42:56.761127+0000","last_became_active":"2024-01-24T17:42:56.756115+0000","last_became_peered":"2024-01-24T17:42:56.756115+0000","last_unstale":"2024-01-24T17:42:56.761127+0000","last_undegraded":"2024-01-24T17:42:56.761127+0000","last_fullsized":"2024-01-24T17:42:56.761127+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:17:06.165401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.734588+0000","last_change":"2024-01-24T17:42:56.259428+0000","last_active":"2024-01-24T17:42:56.734588+0000","last_peered":"2024-01-24T17:42:56.734588+0000","last_clean":"2024-01-24T17:42:56.734588+0000","last_became_active":"2024-01-24T17:42:56.259025+0000","last_became_peered":"2024-01-24T17:42:56.259025+0000","last_unstale":"2024-01-24T17:42:56.734588+0000","last_undegraded":"2024-01-24T17:42:56.734588+0000","last_fullsized":"2024-01-24T17:42:56.734588+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:46:24.456601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.737431+0000","last_change":"2024-01-24T17:42:56.730653+0000","last_active":"2024-01-24T17:42:56.737431+0000","last_peered":"2024-01-24T17:42:56.737431+0000","last_clean":"2024-01-24T17:42:56.737431+0000","last_became_active":"2024-01-24T17:42:56.730364+0000","last_became_peered":"2024-01-24T17:42:56.730364+0000","last_unstale":"2024-01-24T17:42:56.737431+0000","last_undegraded":"2024-01-24T17:42:56.737431+0000","last_fullsized":"2024-01-24T17:42:56.737431+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:22:25.447077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.741047+0000","last_change":"2024-01-24T17:42:56.255206+0000","last_active":"2024-01-24T17:42:56.741047+0000","last_peered":"2024-01-24T17:42:56.741047+0000","last_clean":"2024-01-24T17:42:56.741047+0000","last_became_active":"2024-01-24T17:42:56.254920+0000","last_became_peered":"2024-01-24T17:42:56.254920+0000","last_unstale":"2024-01-24T17:42:56.741047+0000","last_undegraded":"2024-01-24T17:42:56.741047+0000","last_fullsized":"2024-01-24T17:42:56.741047+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:21:39.218288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.737110+0000","last_change":"2024-01-24T17:42:56.734296+0000","last_active":"2024-01-24T17:42:56.737110+0000","last_peered":"2024-01-24T17:42:56.737110+0000","last_clean":"2024-01-24T17:42:56.737110+0000","last_became_active":"2024-01-24T17:42:56.734049+0000","last_became_peered":"2024-01-24T17:42:56.734049+0000","last_unstale":"2024-01-24T17:42:56.737110+0000","last_undegraded":"2024-01-24T17:42:56.737110+0000","last_fullsized":"2024-01-24T17:42:56.737110+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:17:06.165401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.758398+0000","last_change":"2024-01-24T17:42:56.252287+0000","last_active":"2024-01-24T17:42:56.758398+0000","last_peered":"2024-01-24T17:42:56.758398+0000","last_clean":"2024-01-24T17:42:56.758398+0000","last_became_active":"2024-01-24T17:42:56.251984+0000","last_became_peered":"2024-01-24T17:42:56.251984+0000","last_unstale":"2024-01-24T17:42:56.758398+0000","last_undegraded":"2024-01-24T17:42:56.758398+0000","last_fullsized":"2024-01-24T17:42:56.758398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:18.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757428+0000","last_change":"2024-01-24T17:42:56.756812+0000","last_active":"2024-01-24T17:42:56.757428+0000","last_peered":"2024-01-24T17:42:56.757428+0000","last_clean":"2024-01-24T17:42:56.757428+0000","last_became_active":"2024-01-24T17:42:56.756607+0000","last_became_peered":"2024-01-24T17:42:56.756607+0000","last_unstale":"2024-01-24T17:42:56.757428+0000","last_undegraded":"2024-01-24T17:42:56.757428+0000","last_fullsized":"2024-01-24T17:42:56.757428+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:58:31.111806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757462+0000","last_change":"2024-01-24T17:42:56.261245+0000","last_active":"2024-01-24T17:42:56.757462+0000","last_peered":"2024-01-24T17:42:56.757462+0000","last_clean":"2024-01-24T17:42:56.757462+0000","last_became_active":"2024-01-24T17:42:56.260901+0000","last_became_peered":"2024-01-24T17:42:56.260901+0000","last_unstale":"2024-01-24T17:42:56.757462+0000","last_undegraded":"2024-01-24T17:42:56.757462+0000","last_fullsized":"2024-01-24T17:42:56.757462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:10:44.352326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.735591+0000","last_change":"2024-01-24T17:42:56.733725+0000","last_active":"2024-01-24T17:42:56.735591+0000","last_peered":"2024-01-24T17:42:56.735591+0000","last_clean":"2024-01-24T17:42:56.735591+0000","last_became_active":"2024-01-24T17:42:56.732597+0000","last_became_peered":"2024-01-24T17:42:56.732597+0000","last_unstale":"2024-01-24T17:42:56.735591+0000","last_undegraded":"2024-01-24T17:42:56.735591+0000","last_fullsized":"2024-01-24T17:42:56.735591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:40:01.847550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.739214+0000","last_change":"2024-01-24T17:42:56.251773+0000","last_active":"2024-01-24T17:42:56.739214+0000","last_peered":"2024-01-24T17:42:56.739214+0000","last_clean":"2024-01-24T17:42:56.739214+0000","last_became_active":"2024-01-24T17:42:56.250308+0000","last_became_peered":"2024-01-24T17:42:56.250308+0000","last_unstale":"2024-01-24T17:42:56.739214+0000","last_undegraded":"2024-01-24T17:42:56.739214+0000","last_fullsized":"2024-01-24T17:42:56.739214+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:02.822734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745594+0000","last_change":"2024-01-24T17:42:56.737465+0000","last_active":"2024-01-24T17:42:56.745594+0000","last_peered":"2024-01-24T17:42:56.745594+0000","last_clean":"2024-01-24T17:42:56.745594+0000","last_became_active":"2024-01-24T17:42:56.733921+0000","last_became_peered":"2024-01-24T17:42:56.733921+0000","last_unstale":"2024-01-24T17:42:56.745594+0000","last_undegraded":"2024-01-24T17:42:56.745594+0000","last_fullsized":"2024-01-24T17:42:56.745594+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:53:28.617889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.725636+0000","last_change":"2024-01-24T17:42:56.243862+0000","last_active":"2024-01-24T17:42:56.725636+0000","last_peered":"2024-01-24T17:42:56.725636+0000","last_clean":"2024-01-24T17:42:56.725636+0000","last_became_active":"2024-01-24T17:42:56.243553+0000","last_became_peered":"2024-01-24T17:42:56.243553+0000","last_unstale":"2024-01-24T17:42:56.725636+0000","last_undegraded":"2024-01-24T17:42:56.725636+0000","last_fullsized":"2024-01-24T17:42:56.725636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:53:27.606261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.752530+0000","last_change":"2024-01-24T17:42:56.744370+0000","last_active":"2024-01-24T17:42:56.752530+0000","last_peered":"2024-01-24T17:42:56.752530+0000","last_clean":"2024-01-24T17:42:56.752530+0000","last_became_active":"2024-01-24T17:42:56.743596+0000","last_became_peered":"2024-01-24T17:42:56.743596+0000","last_unstale":"2024-01-24T17:42:56.752530+0000","last_undegraded":"2024-01-24T17:42:56.752530+0000","last_fullsized":"2024-01-24T17:42:56.752530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:21:36.607556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.713804+0000","last_change":"2024-01-24T17:42:56.255970+0000","last_active":"2024-01-24T17:42:56.713804+0000","last_peered":"2024-01-24T17:42:56.713804+0000","last_clean":"2024-01-24T17:42:56.713804+0000","last_became_active":"2024-01-24T17:42:56.253488+0000","last_became_peered":"2024-01-24T17:42:56.253488+0000","last_unstale":"2024-01-24T17:42:56.713804+0000","last_undegraded":"2024-01-24T17:42:56.713804+0000","last_fullsized":"2024-01-24T17:42:56.713804+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:46:24.456601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.741170+0000","last_change":"2024-01-24T17:42:56.736521+0000","last_active":"2024-01-24T17:42:56.741170+0000","last_peered":"2024-01-24T17:42:56.741170+0000","last_clean":"2024-01-24T17:42:56.741170+0000","last_became_active":"2024-01-24T17:42:56.736055+0000","last_became_peered":"2024-01-24T17:42:56.736055+0000","last_unstale":"2024-01-24T17:42:56.741170+0000","last_undegraded":"2024-01-24T17:42:56.741170+0000","last_fullsized":"2024-01-24T17:42:56.741170+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:56:25.512110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.758196+0000","last_change":"2024-01-24T17:42:56.258651+0000","last_active":"2024-01-24T17:42:56.758196+0000","last_peered":"2024-01-24T17:42:56.758196+0000","last_clean":"2024-01-24T17:42:56.758196+0000","last_became_active":"2024-01-24T17:42:56.257290+0000","last_became_peered":"2024-01-24T17:42:56.257290+0000","last_unstale":"2024-01-24T17:42:56.758196+0000","last_undegraded":"2024-01-24T17:42:56.758196+0000","last_fullsized":"2024-01-24T17:42:56.758196+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:43:08.435120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745470+0000","last_change":"2024-01-24T17:42:56.737159+0000","last_active":"2024-01-24T17:42:56.745470+0000","last_peered":"2024-01-24T17:42:56.745470+0000","last_clean":"2024-01-24T17:42:56.745470+0000","last_became_active":"2024-01-24T17:42:56.736595+0000","last_became_peered":"2024-01-24T17:42:56.736595+0000","last_unstale":"2024-01-24T17:42:56.745470+0000","last_undegraded":"2024-01-24T17:42:56.745470+0000","last_fullsized":"2024-01-24T17:42:56.745470+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:06.149355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.724401+0000","last_change":"2024-01-24T17:42:56.250980+0000","last_active":"2024-01-24T17:42:56.724401+0000","last_peered":"2024-01-24T17:42:56.724401+0000","last_clean":"2024-01-24T17:42:56.724401+0000","last_became_active":"2024-01-24T17:42:56.250686+0000","last_became_peered":"2024-01-24T17:42:56.250686+0000","last_unstale":"2024-01-24T17:42:56.724401+0000","last_undegraded":"2024-01-24T17:42:56.724401+0000","last_fullsized":"2024-01-24T17:42:56.724401+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:03:23.600383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760432+0000","last_change":"2024-01-24T17:42:56.758705+0000","last_active":"2024-01-24T17:42:56.760432+0000","last_peered":"2024-01-24T17:42:56.760432+0000","last_clean":"2024-01-24T17:42:56.760432+0000","last_became_active":"2024-01-24T17:42:56.758285+0000","last_became_peered":"2024-01-24T17:42:56.758285+0000","last_unstale":"2024-01-24T17:42:56.760432+0000","last_undegraded":"2024-01-24T17:42:56.760432+0000","last_fullsized":"2024-01-24T17:42:56.760432+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:36.375174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.727977+0000","last_change":"2024-01-24T17:42:56.244187+0000","last_active":"2024-01-24T17:42:56.727977+0000","last_peered":"2024-01-24T17:42:56.727977+0000","last_clean":"2024-01-24T17:42:56.727977+0000","last_became_active":"2024-01-24T17:42:56.243927+0000","last_became_peered":"2024-01-24T17:42:56.243927+0000","last_unstale":"2024-01-24T17:42:56.727977+0000","last_undegraded":"2024-01-24T17:42:56.727977+0000","last_fullsized":"2024-01-24T17:42:56.727977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:22:00.592365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.736062+0000","last_change":"2024-01-24T17:42:56.732222+0000","last_active":"2024-01-24T17:42:56.736062+0000","last_peered":"2024-01-24T17:42:56.736062+0000","last_clean":"2024-01-24T17:42:56.736062+0000","last_became_active":"2024-01-24T17:42:56.731328+0000","last_became_peered":"2024-01-24T17:42:56.731328+0000","last_unstale":"2024-01-24T17:42:56.736062+0000","last_undegraded":"2024-01-24T17:42:56.736062+0000","last_fullsized":"2024-01-24T17:42:56.736062+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:42:14.183902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.726998+0000","last_change":"2024-01-24T17:42:56.239354+0000","last_active":"2024-01-24T17:42:56.726998+0000","last_peered":"2024-01-24T17:42:56.726998+0000","last_clean":"2024-01-24T17:42:56.726998+0000","last_became_active":"2024-01-24T17:42:56.238986+0000","last_became_peered":"2024-01-24T17:42:56.238986+0000","last_unstale":"2024-01-24T17:42:56.726998+0000","last_undegraded":"2024-01-24T17:42:56.726998+0000","last_fullsized":"2024-01-24T17:42:56.726998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:33:37.502444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.753123+0000","last_change":"2024-01-24T17:42:56.747157+0000","last_active":"2024-01-24T17:42:56.753123+0000","last_peered":"2024-01-24T17:42:56.753123+0000","last_clean":"2024-01-24T17:42:56.753123+0000","last_became_active":"2024-01-24T17:42:56.746856+0000","last_became_peered":"2024-01-24T17:42:56.746856+0000","last_unstale":"2024-01-24T17:42:56.753123+0000","last_undegraded":"2024-01-24T17:42:56.753123+0000","last_fullsized":"2024-01-24T17:42:56.753123+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:46:25.468230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.748731+0000","last_change":"2024-01-24T17:42:56.254369+0000","last_active":"2024-01-24T17:42:56.748731+0000","last_peered":"2024-01-24T17:42:56.748731+0000","last_clean":"2024-01-24T17:42:56.748731+0000","last_became_active":"2024-01-24T17:42:56.253957+0000","last_became_peered":"2024-01-24T17:42:56.253957+0000","last_unstale":"2024-01-24T17:42:56.748731+0000","last_undegraded":"2024-01-24T17:42:56.748731+0000","last_fullsized":"2024-01-24T17:42:56.748731+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:03:23.600383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.745740+0000","last_change":"2024-01-24T17:42:56.741877+0000","last_active":"2024-01-24T17:42:56.745740+0000","last_peered":"2024-01-24T17:42:56.745740+0000","last_clean":"2024-01-24T17:42:56.745740+0000","last_became_active":"2024-01-24T17:42:56.741614+0000","last_became_peered":"2024-01-24T17:42:56.741614+0000","last_unstale":"2024-01-24T17:42:56.745740+0000","last_undegraded":"2024-01-24T17:42:56.745740+0000","last_fullsized":"2024-01-24T17:42:56.745740+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:55:00.594223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.718739+0000","last_change":"2024-01-24T17:42:56.254749+0000","last_active":"2024-01-24T17:42:56.718739+0000","last_peered":"2024-01-24T17:42:56.718739+0000","last_clean":"2024-01-24T17:42:56.718739+0000","last_became_active":"2024-01-24T17:42:56.254146+0000","last_became_peered":"2024-01-24T17:42:56.254146+0000","last_unstale":"2024-01-24T17:42:56.718739+0000","last_undegraded":"2024-01-24T17:42:56.718739+0000","last_fullsized":"2024-01-24T17:42:56.718739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:56:24.500482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.757705+0000","last_change":"2024-01-24T17:42:56.755210+0000","last_active":"2024-01-24T17:42:56.757705+0000","last_peered":"2024-01-24T17:42:56.757705+0000","last_clean":"2024-01-24T17:42:56.757705+0000","last_became_active":"2024-01-24T17:42:56.754010+0000","last_became_peered":"2024-01-24T17:42:56.754010+0000","last_unstale":"2024-01-24T17:42:56.757705+0000","last_undegraded":"2024-01-24T17:42:56.757705+0000","last_fullsized":"2024-01-24T17:42:56.757705+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:56:25.512110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.736194+0000","last_change":"2024-01-24T17:42:56.252599+0000","last_active":"2024-01-24T17:42:56.736194+0000","last_peered":"2024-01-24T17:42:56.736194+0000","last_clean":"2024-01-24T17:42:56.736194+0000","last_became_active":"2024-01-24T17:42:56.252343+0000","last_became_peered":"2024-01-24T17:42:56.252343+0000","last_unstale":"2024-01-24T17:42:56.736194+0000","last_undegraded":"2024-01-24T17:42:56.736194+0000","last_fullsized":"2024-01-24T17:42:56.736194+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:42:13.172274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.736883+0000","last_change":"2024-01-24T17:42:56.733135+0000","last_active":"2024-01-24T17:42:56.736883+0000","last_peered":"2024-01-24T17:42:56.736883+0000","last_clean":"2024-01-24T17:42:56.736883+0000","last_became_active":"2024-01-24T17:42:56.732855+0000","last_became_peered":"2024-01-24T17:42:56.732855+0000","last_unstale":"2024-01-24T17:42:56.736883+0000","last_undegraded":"2024-01-24T17:42:56.736883+0000","last_fullsized":"2024-01-24T17:42:56.736883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:55:06.270349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.729280+0000","last_change":"2024-01-24T17:42:56.249135+0000","last_active":"2024-01-24T17:42:56.729280+0000","last_peered":"2024-01-24T17:42:56.729280+0000","last_clean":"2024-01-24T17:42:56.729280+0000","last_became_active":"2024-01-24T17:42:56.248826+0000","last_became_peered":"2024-01-24T17:42:56.248826+0000","last_unstale":"2024-01-24T17:42:56.729280+0000","last_undegraded":"2024-01-24T17:42:56.729280+0000","last_fullsized":"2024-01-24T17:42:56.729280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:46:59.399468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.760435+0000","last_change":"2024-01-24T17:42:56.755020+0000","last_active":"2024-01-24T17:42:56.760435+0000","last_peered":"2024-01-24T17:42:56.760435+0000","last_clean":"2024-01-24T17:42:56.760435+0000","last_became_active":"2024-01-24T17:42:56.754652+0000","last_became_peered":"2024-01-24T17:42:56.754652+0000","last_unstale":"2024-01-24T17:42:56.760435+0000","last_undegraded":"2024-01-24T17:42:56.760435+0000","last_fullsized":"2024-01-24T17:42:56.760435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:03:05.433687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.726275+0000","last_change":"2024-01-24T17:42:56.244865+0000","last_active":"2024-01-24T17:42:56.726275+0000","last_peered":"2024-01-24T17:42:56.726275+0000","last_clean":"2024-01-24T17:42:56.726275+0000","last_became_active":"2024-01-24T17:42:56.244578+0000","last_became_peered":"2024-01-24T17:42:56.244578+0000","last_unstale":"2024-01-24T17:42:56.726275+0000","last_undegraded":"2024-01-24T17:42:56.726275+0000","last_fullsized":"2024-01-24T17:42:56.726275+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:02.822734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.759402+0000","last_change":"2024-01-24T17:42:56.750138+0000","last_active":"2024-01-24T17:42:56.759402+0000","last_peered":"2024-01-24T17:42:56.759402+0000","last_clean":"2024-01-24T17:42:56.759402+0000","last_became_active":"2024-01-24T17:42:56.749804+0000","last_became_peered":"2024-01-24T17:42:56.749804+0000","last_unstale":"2024-01-24T17:42:56.759402+0000","last_undegraded":"2024-01-24T17:42:56.759402+0000","last_fullsized":"2024-01-24T17:42:56.759402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:06.149355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.724479+0000","last_change":"2024-01-24T17:42:56.250644+0000","last_active":"2024-01-24T17:42:56.724479+0000","last_peered":"2024-01-24T17:42:56.724479+0000","last_clean":"2024-01-24T17:42:56.724479+0000","last_became_active":"2024-01-24T17:42:56.250358+0000","last_became_peered":"2024-01-24T17:42:56.250358+0000","last_unstale":"2024-01-24T17:42:56.724479+0000","last_undegraded":"2024-01-24T17:42:56.724479+0000","last_fullsized":"2024-01-24T17:42:56.724479+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:42:13.172274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.743501+0000","last_change":"2024-01-24T17:42:56.740628+0000","last_active":"2024-01-24T17:42:56.743501+0000","last_peered":"2024-01-24T17:42:56.743501+0000","last_clean":"2024-01-24T17:42:56.743501+0000","last_became_active":"2024-01-24T17:42:56.739664+0000","last_became_peered":"2024-01-24T17:42:56.739664+0000","last_unstale":"2024-01-24T17:42:56.743501+0000","last_undegraded":"2024-01-24T17:42:56.743501+0000","last_fullsized":"2024-01-24T17:42:56.743501+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:55:01.851549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.736198+0000","last_change":"2024-01-24T17:42:56.252929+0000","last_active":"2024-01-24T17:42:56.736198+0000","last_peered":"2024-01-24T17:42:56.736198+0000","last_clean":"2024-01-24T17:42:56.736198+0000","last_became_active":"2024-01-24T17:42:56.252574+0000","last_became_peered":"2024-01-24T17:42:56.252574+0000","last_unstale":"2024-01-24T17:42:56.736198+0000","last_undegraded":"2024-01-24T17:42:56.736198+0000","last_fullsized":"2024-01-24T17:42:56.736198+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:20:50.882511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.746777+0000","last_change":"2024-01-24T17:42:56.740881+0000","last_active":"2024-01-24T17:42:56.746777+0000","last_peered":"2024-01-24T17:42:56.746777+0000","last_clean":"2024-01-24T17:42:56.746777+0000","last_became_active":"2024-01-24T17:42:56.738865+0000","last_became_peered":"2024-01-24T17:42:56.738865+0000","last_unstale":"2024-01-24T17:42:56.746777+0000","last_undegraded":"2024-01-24T17:42:56.746777+0000","last_fullsized":"2024-01-24T17:42:56.746777+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:56.207972+0000","last_clean_scrub_stamp":"2024-01-24T17:42:56.207972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:23:36.375174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.718176+0000","last_change":"2024-01-24T17:42:56.242233+0000","last_active":"2024-01-24T17:42:56.718176+0000","last_peered":"2024-01-24T17:42:56.718176+0000","last_clean":"2024-01-24T17:42:56.718176+0000","last_became_active":"2024-01-24T17:42:56.241454+0000","last_became_peered":"2024-01-24T17:42:56.241454+0000","last_unstale":"2024-01-24T17:42:56.718176+0000","last_undegraded":"2024-01-24T17:42:56.718176+0000","last_fullsized":"2024-01-24T17:42:56.718176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:42:55.196343+0000","last_clean_scrub_stamp":"2024-01-24T17:42:55.196343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:14:22.210259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-24T17:42:56.233645+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:42:56.233645+0000","last_peered":"2024-01-24T17:42:56.233645+0000","last_clean":"2024-01-24T17:42:56.233645+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:42:56.233645+0000","last_undegraded":"2024-01-24T17:42:56.233645+0000","last_fullsized":"2024-01-24T17:42:56.233645+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":360448,"data_stored":138326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.099}]},{"osd":1,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.349},{"interface":"front","average":{"1min":1.238,"5min":1.238,"15min":1.238},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.82199999999999995}]},{"osd":3,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.5760000000000001,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":1.0780000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.333,"15min":1.333},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.6459999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.585}]},{"osd":5,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.6739999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.175},{"interface":"front","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3719999999999999,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.0009999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.554,"15min":1.554},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.6960000000000002,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.157},{"interface":"front","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4830000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.675}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":385024,"data_stored":146665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.256},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.83499999999999996}]},{"osd":1,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.506}]},{"osd":2,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.0940000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.399}]},{"osd":4,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.446},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":1.4790000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.4849999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.3360000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:42:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.2050000000000001,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.419},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6920000000000002,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.3680000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":736051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":1.123},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.232}]},{"osd":1,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.286,"15min":1.286},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":0.94399999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.282},{"interface":"front","average":{"1min":1.329,"5min":1.329,"15min":1.329},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":3.1349999999999998,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.3979999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":3.036,"5min":3.036,"15min":3.036},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.345,"15min":1.345},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.361}]},{"osd":4,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.9890000000000001,"5min":1.9890000000000001,"15min":1.9890000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":1.073}]},{"osd":6,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.1150000000000002,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":1.302},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.4430000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:42:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":1.345},{"interface":"front","average":{"1min":1.657,"5min":1.657,"15min":1.657},"min":{"1min":1.262,"5min":1.262,"15min":1.262},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.4690000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":146687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.877}]},{"osd":1,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.75},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.1559999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.83599999999999997}]},{"osd":3,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.9409999999999998,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.365}]},{"osd":5,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.028,"5min":2.028,"15min":2.028},"last":1.248},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.117}]},{"osd":6,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":3.0840000000000001,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":1.498},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.331}]},{"osd":7,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.29},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":4.3250000000000002,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.448}]}]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":389120,"data_stored":145877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":5.1929999999999996},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":8.6989999999999998}]},{"osd":1,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":1.8720000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":3.7240000000000002}]},{"osd":2,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":3.4630000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":4.2919999999999998}]},{"osd":4,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":6.7469999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":3.4049999999999998}]},{"osd":5,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":4.3789999999999996},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":2.2149999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":2.0110000000000001,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":9.5440000000000005},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":4.8170000000000002}]},{"osd":7,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":6.6529999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31472,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":360448,"data_stored":138326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.94299999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.044},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.627,"5min":1.627,"15min":1.627},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":1.081}]},{"osd":4,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.7120000000000002,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.3839999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.155}]},{"osd":5,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.101},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.532}]},{"osd":6,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":3.7410000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.5900000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:42:23 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.617,"15min":1.617},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":3.343,"5min":3.343,"15min":3.343},"last":1.214},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.6970000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":970752,"data_stored":735539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.877}]},{"osd":2,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.895,"5min":1.895,"15min":1.895},"last":0.61899999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":3.149,"5min":3.149,"15min":3.149},"last":0.96899999999999997}]},{"osd":4,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.377},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.423}]},{"osd":5,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.375,"15min":1.375},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.9809999999999999,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":0.92300000000000004}]},{"osd":6,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":3.2850000000000001,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.153},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":0.94599999999999995}]},{"osd":7,"last update":"Wed Jan 24 17:42:26 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.484,"15min":1.484},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.8439999999999999,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.244},{"interface":"front","average":{"1min":1.581,"5min":1.581,"15min":1.581},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":3.4350000000000001,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.212}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31488,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340352,"statfs":{"total":96636764160,"available":96604520448,"internally_reserved":0,"allocated":376832,"data_stored":145191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.7290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.53900000000000003}]},{"osd":2,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":1.294},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.0529999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.2,"5min":1.2,"15min":1.2},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.1819999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":3.3740000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.0880000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.9300000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.401}]},{"osd":6,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.552},{"interface":"front","average":{"1min":1.484,"5min":1.484,"15min":1.484},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.4430000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:42:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":3.0230000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.159},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":3.1600000000000001,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.476}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:43:02.267 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-24T17:43:02.267 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-24T17:43:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.538+0000 7f53fca00640 1 Processor -- start 2024-01-24T17:43:02.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.539+0000 7f53fca00640 1 -- start start 2024-01-24T17:43:02.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.540+0000 7f53fca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f53f415c9b0 0x7f53f40d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.540+0000 7f53fca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.540+0000 7f53fca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f40d05c0 con 0x7f53f40cddf0 2024-01-24T17:43:02.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.540+0000 7f53fca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f40cfd30 con 0x7f53f415c9b0 2024-01-24T17:43:02.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.540+0000 7f53fca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f40d08a0 con 0x7f53f40cf360 2024-01-24T17:43:02.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53f9600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f53f40cddf0 0x7f53f40d1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40172/0 (socket says 172.21.15.39:40172) 2024-01-24T17:43:02.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53f9600640 1 -- 172.21.15.39:0/2902916714 learned_addr learned my addr 172.21.15.39:0/2902916714 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:02.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53fa000640 1 --2- 172.21.15.39:0/2902916714 >> v2:172.21.15.39:3300/0 conn(0x7f53f415c9b0 0x7f53f40d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53faa00640 1 --2- 172.21.15.39:0/2902916714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53fa000640 1 -- 172.21.15.39:0/2902916714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 msgr2=0x7f53f40cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:02.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53f8c00640 1 -- 172.21.15.39:0/2902916714 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670125894 0 0) 0x7f53f40d05c0 con 0x7f53f40cddf0 2024-01-24T17:43:02.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53fa000640 1 --2- 172.21.15.39:0/2902916714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53fa000640 1 -- 172.21.15.39:0/2902916714 >> v1:172.21.15.39:6789/0 conn(0x7f53f40cddf0 legacy=0x7f53f40d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53fa000640 1 -- 172.21.15.39:0/2902916714 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f40d0b80 con 0x7f53f415c9b0 2024-01-24T17:43:02.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.541+0000 7f53faa00640 1 --2- 172.21.15.39:0/2902916714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:02.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.542+0000 7f53fa000640 1 --2- 172.21.15.39:0/2902916714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f40d1660 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0097a0 tx=0x7f53ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770e33402e569483 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53f8c00640 1 -- 172.21.15.39:0/2902916714 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec02e070 con 0x7f53f415c9b0 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53f8c00640 1 -- 172.21.15.39:0/2902916714 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ec0315a0 con 0x7f53f415c9b0 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53f8c00640 1 -- 172.21.15.39:0/2902916714 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec031930 con 0x7f53f415c9b0 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53fca00640 1 -- 172.21.15.39:0/2902916714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 msgr2=0x7f53f40d1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53fca00640 1 --2- 172.21.15.39:0/2902916714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f40d1660 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f53ec0097a0 tx=0x7f53ec02b840 comp rx=0 tx=0).stop 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53fca00640 1 -- 172.21.15.39:0/2902916714 shutdown_connections 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53fca00640 1 --2- 172.21.15.39:0/2902916714 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53fca00640 1 --2- 172.21.15.39:0/2902916714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f40d1660 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53fca00640 1 -- 172.21.15.39:0/2902916714 >> 172.21.15.39:0/2902916714 conn(0x7f53f406ae70 msgr2=0x7f53f406b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.543+0000 7f53fca00640 1 -- 172.21.15.39:0/2902916714 shutdown_connections 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.544+0000 7f53fca00640 1 -- 172.21.15.39:0/2902916714 wait complete. 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.545+0000 7f53fca00640 1 Processor -- start 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.545+0000 7f53fca00640 1 -- start start 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.546+0000 7f53fca00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53f40cddf0 0x7f53f40c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.546+0000 7f53fca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.546+0000 7f53fca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f41f1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.546+0000 7f53fca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f41f5700 con 0x7f53f415c9b0 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.546+0000 7f53fca00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f41f6520 con 0x7f53f40cddf0 2024-01-24T17:43:02.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.546+0000 7f53fca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f41f5910 con 0x7f53f40cf360 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53faa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f41f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53faa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f41f1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35014/0 (socket says 172.21.15.39:35014) 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53faa00640 1 -- 172.21.15.39:0/1402816412 learned_addr learned my addr 172.21.15.39:0/1402816412 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53f9600640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53fa000640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53f40cddf0 0x7f53f40c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53faa00640 1 -- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 msgr2=0x7f53f40c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53faa00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53faa00640 1 -- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53f40cddf0 msgr2=0x7f53f40c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53faa00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53f40cddf0 0x7f53f40c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.547+0000 7f53faa00640 1 -- 172.21.15.39:0/1402816412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ec009400 con 0x7f53f415c9b0 2024-01-24T17:43:02.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.548+0000 7f53faa00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f41f1590 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f53e400c820 tx=0x7f53e400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:02.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.548+0000 7f53f2000640 1 -- 172.21.15.39:0/1402816412 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e4007a30 con 0x7f53f415c9b0 2024-01-24T17:43:02.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.548+0000 7f53f2000640 1 -- 172.21.15.39:0/1402816412 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e4007b90 con 0x7f53f415c9b0 2024-01-24T17:43:02.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.548+0000 7f53f2000640 1 -- 172.21.15.39:0/1402816412 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e400ce10 con 0x7f53f415c9b0 2024-01-24T17:43:02.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.548+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f41f5bf0 con 0x7f53f415c9b0 2024-01-24T17:43:02.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.548+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f41f61b0 con 0x7f53f415c9b0 2024-01-24T17:43:02.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.549+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f4066f10 con 0x7f53f415c9b0 2024-01-24T17:43:02.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.550+0000 7f53f2000640 1 -- 172.21.15.39:0/1402816412 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f53e40113f0 con 0x7f53f415c9b0 2024-01-24T17:43:02.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.551+0000 7f53f2000640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f53cc06a430 0x7f53cc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:02.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.551+0000 7f53f2000640 1 -- 172.21.15.39:0/1402816412 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53e4088a40 con 0x7f53f415c9b0 2024-01-24T17:43:02.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.552+0000 7f53fa000640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f53cc06a430 0x7f53cc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:02.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.553+0000 7f53fa000640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f53cc06a430 0x7f53cc06c8f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53ec036000 tx=0x7f53ec03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:02.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.556+0000 7f53f2000640 1 -- 172.21.15.39:0/1402816412 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f53e4054020 con 0x7f53f415c9b0 2024-01-24T17:43:02.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.819+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f53f40b80f0 con 0x7f53f415c9b0 2024-01-24T17:43:02.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.829+0000 7f53f2000640 1 -- 172.21.15.39:0/1402816412 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7f53e401f0e0 con 0x7f53f415c9b0 2024-01-24T17:43:02.831 INFO:teuthology.orchestra.run.smithi039.stderr:updated caps for client.0 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f53cc06a430 msgr2=0x7f53cc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f53cc06a430 0x7f53cc06c8f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53ec036000 tx=0x7f53ec03d040 comp rx=0 tx=0).stop 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 msgr2=0x7f53f41f1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f41f1590 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f53e400c820 tx=0x7f53e400cce0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 shutdown_connections 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f53cc06a430 0x7f53cc06c8f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f415c9b0 0x7f53f41f1590 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53f40cf360 0x7f53f40c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 --2- 172.21.15.39:0/1402816412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53f40cddf0 0x7f53f40c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 >> 172.21.15.39:0/1402816412 conn(0x7f53f406ae70 msgr2=0x7f53f415b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:02.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 shutdown_connections 2024-01-24T17:43:02.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:02.832+0000 7f53fca00640 1 -- 172.21.15.39:0/1402816412 wait complete. 2024-01-24T17:43:02.851 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:43:03.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.061+0000 7fa1e8e00640 1 Processor -- start 2024-01-24T17:43:03.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.061+0000 7fa1e8e00640 1 -- start start 2024-01-24T17:43:03.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.063+0000 7fa1e8e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415aa80 0x7fa1e40c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.063+0000 7fa1e8e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1e4065910 0x7fa1e406c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.063+0000 7fa1e8e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e406ae50 con 0x7fa1e415ae60 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.063+0000 7fa1e8e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e40ca270 con 0x7fa1e4065910 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.063+0000 7fa1e8e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e40c9f90 con 0x7fa1e415aa80 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.063+0000 7fa1e1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1e4065910 0x7fa1e406c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.063+0000 7fa1e1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1e4065910 0x7fa1e406c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50488/0 (socket says 172.21.15.39:50488) 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.063+0000 7fa1e1400640 1 -- 172.21.15.39:0/3519964767 learned_addr learned my addr 172.21.15.39:0/3519964767 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.064+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3519964767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415aa80 0x7fa1e40c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.064+0000 7fa1e1400640 1 -- 172.21.15.39:0/3519964767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415aa80 msgr2=0x7fa1e40c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.064+0000 7fa1e1400640 1 --2- 172.21.15.39:0/3519964767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415aa80 0x7fa1e40c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.064+0000 7fa1e1400640 1 -- 172.21.15.39:0/3519964767 >> v1:172.21.15.100:6789/0 conn(0x7fa1e415ae60 legacy=0x7fa1e40b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.064+0000 7fa1e1400640 1 -- 172.21.15.39:0/3519964767 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e406a800 con 0x7fa1e4065910 2024-01-24T17:43:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.065+0000 7fa1e1400640 1 --2- 172.21.15.39:0/3519964767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e4065910 0x7fa1e406c780 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009e50 tx=0x7fa1cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d08765199004444 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.065+0000 7fa1e0a00640 1 -- 172.21.15.39:0/3519964767 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc02ca70 con 0x7fa1e4065910 2024-01-24T17:43:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.065+0000 7fa1e0a00640 1 -- 172.21.15.39:0/3519964767 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc02cbd0 con 0x7fa1e4065910 2024-01-24T17:43:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.065+0000 7fa1e0a00640 1 -- 172.21.15.39:0/3519964767 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc0343f0 con 0x7fa1e4065910 2024-01-24T17:43:03.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.066+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3519964767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e4065910 msgr2=0x7fa1e406c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.066+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3519964767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e4065910 0x7fa1e406c780 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009e50 tx=0x7fa1cc02bed0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.066+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3519964767 shutdown_connections 2024-01-24T17:43:03.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.066+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3519964767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e4065910 0x7fa1e406c780 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.066+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3519964767 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415aa80 0x7fa1e40c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.066+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3519964767 >> 172.21.15.39:0/3519964767 conn(0x7fa1e40b0980 msgr2=0x7fa1e40b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:03.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.066+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3519964767 shutdown_connections 2024-01-24T17:43:03.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.066+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3519964767 wait complete. 2024-01-24T17:43:03.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.069+0000 7fa1e8e00640 1 Processor -- start 2024-01-24T17:43:03.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.069+0000 7fa1e8e00640 1 -- start start 2024-01-24T17:43:03.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.070+0000 7fa1e8e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1e4065910 0x7fa1e41e4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.070+0000 7fa1e8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e415aa80 0x7fa1e40c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.070+0000 7fa1e8e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415ae60 0x7fa1e41e1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.070+0000 7fa1e8e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e41e5240 con 0x7fa1e415aa80 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.070+0000 7fa1e8e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e40c2990 con 0x7fa1e4065910 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.070+0000 7fa1e8e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e40c18d0 con 0x7fa1e415ae60 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.070+0000 7fa1e1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e415aa80 0x7fa1e40c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e415aa80 0x7fa1e40c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50504/0 (socket says 172.21.15.39:50504) 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1400640 1 -- 172.21.15.39:0/3377007209 learned_addr learned my addr 172.21.15.39:0/3377007209 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1e4065910 0x7fa1e41e4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e2800640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415ae60 0x7fa1e41e1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1400640 1 -- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415ae60 msgr2=0x7fa1e41e1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1400640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415ae60 0x7fa1e41e1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1400640 1 -- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1e4065910 msgr2=0x7fa1e41e4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1400640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1e4065910 0x7fa1e41e4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1400640 1 -- 172.21.15.39:0/3377007209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc009b30 con 0x7fa1e415aa80 2024-01-24T17:43:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.071+0000 7fa1e1400640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e415aa80 0x7fa1e40c44f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009e50 tx=0x7fa1cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.072+0000 7fa1da000640 1 -- 172.21.15.39:0/3377007209 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc009510 con 0x7fa1e415aa80 2024-01-24T17:43:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.072+0000 7fa1da000640 1 -- 172.21.15.39:0/3377007209 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc03e450 con 0x7fa1e415aa80 2024-01-24T17:43:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.072+0000 7fa1da000640 1 -- 172.21.15.39:0/3377007209 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc03e800 con 0x7fa1e415aa80 2024-01-24T17:43:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.072+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e40c2c00 con 0x7fa1e415aa80 2024-01-24T17:43:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.072+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e41e3480 con 0x7fa1e415aa80 2024-01-24T17:43:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.073+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e4073ef0 con 0x7fa1e415aa80 2024-01-24T17:43:03.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.074+0000 7fa1da000640 1 -- 172.21.15.39:0/3377007209 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa1cc03e960 con 0x7fa1e415aa80 2024-01-24T17:43:03.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.075+0000 7fa1da000640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1bc06a430 0x7fa1bc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.075+0000 7fa1da000640 1 -- 172.21.15.39:0/3377007209 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1cc0ab780 con 0x7fa1e415aa80 2024-01-24T17:43:03.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.076+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1bc06a430 0x7fa1bc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.077+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1bc06a430 0x7fa1bc06c8f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4006fd0 tx=0x7fa1d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:03.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.080+0000 7fa1da000640 1 -- 172.21.15.39:0/3377007209 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa1cc076d70 con 0x7fa1e415aa80 2024-01-24T17:43:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.319+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1e40b1540 con 0x7fa1e415aa80 2024-01-24T17:43:03.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.320+0000 7fa1da000640 1 -- 172.21.15.39:0/3377007209 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fa1cc07a390 con 0x7fa1e415aa80 2024-01-24T17:43:03.322 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:03.322 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4688,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/648542221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":648542221},{"type":"v1","addr":"172.21.15.100:6837","nonce":648542221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/188830546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":188830546},{"type":"v1","addr":"172.21.15.100:6835","nonce":188830546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/4294097622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4294097622},{"type":"v1","addr":"172.21.15.39:6834","nonce":4294097622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:42:56.701344+0000","modified":"2024-01-24T17:43:00.247932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5175":{"gid":5175,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/3577904963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3577904963},{"type":"v1","addr":"172.21.15.39:6835","nonce":3577904963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-24T17:43:03.322 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1bc06a430 msgr2=0x7fa1bc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1bc06a430 0x7fa1bc06c8f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4006fd0 tx=0x7fa1d4008040 comp rx=0 tx=0).stop 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e415aa80 msgr2=0x7fa1e40c44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e415aa80 0x7fa1e40c44f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009e50 tx=0x7fa1cc004a40 comp rx=0 tx=0).stop 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 shutdown_connections 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1e415ae60 0x7fa1e41e1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1bc06a430 0x7fa1bc06c8f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e415aa80 0x7fa1e40c44f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 --2- 172.21.15.39:0/3377007209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1e4065910 0x7fa1e41e4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 >> 172.21.15.39:0/3377007209 conn(0x7fa1e40b0980 msgr2=0x7fa1e40ca400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:03.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 shutdown_connections 2024-01-24T17:43:03.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.323+0000 7fa1e8e00640 1 -- 172.21.15.39:0/3377007209 wait complete. 2024-01-24T17:43:03.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:42:56.701344+0000', 'modified': '2024-01-24T17:43:00.247932+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5175}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5175': {'gid': 5175, 'name': 'a', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.39:6835/3577904963', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6833', 'nonce': 3577904963}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 3577904963}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:43:03.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T17:43:03.747 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-24T17:43:03.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.972+0000 7f7507e00640 1 Processor -- start 2024-01-24T17:43:03.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.973+0000 7f7507e00640 1 -- start start 2024-01-24T17:43:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.974+0000 7f7507e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f75080cdc90 0x7f75080713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.974+0000 7f7507e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75080ce120 con 0x7f75080cd830 2024-01-24T17:43:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.974+0000 7f7507e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750815e010 con 0x7f750806bdb0 2024-01-24T17:43:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.974+0000 7f7507e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7508159d50 con 0x7f75080cdc90 2024-01-24T17:43:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.975+0000 7f7507400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f75080cdc90 0x7f75080713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.975+0000 7f74ffe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f75080cd830 0x7f7508070ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58324/0 (socket says 172.21.15.39:58324) 2024-01-24T17:43:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.975+0000 7f74ffe00640 1 -- 172.21.15.39:0/3719988665 learned_addr learned my addr 172.21.15.39:0/3719988665 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.975+0000 7f7506000640 1 -- 172.21.15.39:0/3719988665 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900635517 0 0) 0x7f75080ce120 con 0x7f75080cd830 2024-01-24T17:43:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.975+0000 7f7506000640 1 -- 172.21.15.39:0/3719988665 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e8005730 con 0x7f75080cd830 2024-01-24T17:43:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.975+0000 7f7507400640 1 -- 172.21.15.39:0/3719988665 >> v1:172.21.15.100:6789/0 conn(0x7f750806bdb0 legacy=0x7f750806c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.975+0000 7f7507400640 1 -- 172.21.15.39:0/3719988665 >> v1:172.21.15.39:6789/0 conn(0x7f75080cd830 legacy=0x7f7508070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.975+0000 7f7507400640 1 -- 172.21.15.39:0/3719988665 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750815a030 con 0x7f75080cdc90 2024-01-24T17:43:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.977+0000 7f7507400640 1 --2- 172.21.15.39:0/3719988665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75080cdc90 0x7f75080713f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7500009d90 tx=0x7f750002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0b2be372b1d155 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.977+0000 7f7506000640 1 -- 172.21.15.39:0/3719988665 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750002cd00 con 0x7f75080cdc90 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.977+0000 7f7506000640 1 -- 172.21.15.39:0/3719988665 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750002ce60 con 0x7f75080cdc90 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.977+0000 7f7506000640 1 -- 172.21.15.39:0/3719988665 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500034660 con 0x7f75080cdc90 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.977+0000 7f7507e00640 1 -- 172.21.15.39:0/3719988665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75080cdc90 msgr2=0x7f75080713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.977+0000 7f7507e00640 1 --2- 172.21.15.39:0/3719988665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75080cdc90 0x7f75080713f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7500009d90 tx=0x7f750002bf50 comp rx=0 tx=0).stop 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.978+0000 7f7507e00640 1 -- 172.21.15.39:0/3719988665 shutdown_connections 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.978+0000 7f7507e00640 1 --2- 172.21.15.39:0/3719988665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75080cdc90 0x7f75080713f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.978+0000 7f7507e00640 1 -- 172.21.15.39:0/3719988665 >> 172.21.15.39:0/3719988665 conn(0x7f750806a2a0 msgr2=0x7f75080afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.978+0000 7f7507e00640 1 -- 172.21.15.39:0/3719988665 shutdown_connections 2024-01-24T17:43:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.978+0000 7f7507e00640 1 -- 172.21.15.39:0/3719988665 wait complete. 2024-01-24T17:43:03.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.979+0000 7f7507e00640 1 Processor -- start 2024-01-24T17:43:03.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.980+0000 7f7507e00640 1 -- start start 2024-01-24T17:43:03.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.980+0000 7f7507e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f750806bdb0 0x7f7508154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.980+0000 7f7507e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75080cd830 0x7f75080b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7507e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75080cdc90 0x7f75080b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7507e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75080b10f0 con 0x7f750806bdb0 2024-01-24T17:43:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7507e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75080b1f90 con 0x7f75080cdc90 2024-01-24T17:43:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7507e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75080b2270 con 0x7f75080cd830 2024-01-24T17:43:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7506a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f750806bdb0 0x7f7508154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7506a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f750806bdb0 0x7f7508154b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50526/0 (socket says 172.21.15.39:50526) 2024-01-24T17:43:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7506a00640 1 -- 172.21.15.39:0/1413756611 learned_addr learned my addr 172.21.15.39:0/1413756611 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f74ffe00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75080cd830 0x7f75080b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7506a00640 1 -- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75080cd830 msgr2=0x7f75080b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7506a00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75080cd830 0x7f75080b1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7507400640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75080cdc90 0x7f75080b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7506a00640 1 -- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75080cdc90 msgr2=0x7f75080b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7506a00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75080cdc90 0x7f75080b1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.981+0000 7f7506a00640 1 -- 172.21.15.39:0/1413756611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500009a70 con 0x7f750806bdb0 2024-01-24T17:43:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.982+0000 7f7506a00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f750806bdb0 0x7f7508154b80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f74f800c400 tx=0x7f74f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.982+0000 7f74fea00640 1 -- 172.21.15.39:0/1413756611 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f80052d0 con 0x7f750806bdb0 2024-01-24T17:43:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.982+0000 7f74fea00640 1 -- 172.21.15.39:0/1413756611 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f8005430 con 0x7f750806bdb0 2024-01-24T17:43:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.982+0000 7f74fea00640 1 -- 172.21.15.39:0/1413756611 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8003830 con 0x7f750806bdb0 2024-01-24T17:43:03.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.982+0000 7f7507e00640 1 -- 172.21.15.39:0/1413756611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75080b2550 con 0x7f750806bdb0 2024-01-24T17:43:03.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.982+0000 7f7507e00640 1 -- 172.21.15.39:0/1413756611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7508156ad0 con 0x7f750806bdb0 2024-01-24T17:43:03.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.985+0000 7f74fea00640 1 -- 172.21.15.39:0/1413756611 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f74f8003990 con 0x7f750806bdb0 2024-01-24T17:43:03.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.986+0000 7f74fea00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74e006a430 0x7f74e006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:03.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.986+0000 7f74fea00640 1 -- 172.21.15.39:0/1413756611 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74f8087e30 con 0x7f750806bdb0 2024-01-24T17:43:03.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.987+0000 7f74ffe00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74e006a430 0x7f74e006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:03.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.987+0000 7f7510195740 1 Processor -- start 2024-01-24T17:43:03.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.987+0000 7f7510195740 1 -- start start 2024-01-24T17:43:04.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f74ffe00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74e006a430 0x7f74e006c8f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f75080b2f70 tx=0x7f74f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f7510195740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d225b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f7510195740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf2150 0x557579d1f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f7510195740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d1f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f7510195740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557579ccd6c0 con 0x557579cf2150 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f7510195740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x557579c62070 con 0x557579cf5d50 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f7510195740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x557579c5f800 con 0x557579d221d0 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f74ffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf2150 0x557579d1f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f74ffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf2150 0x557579d1f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50538/0 (socket says 172.21.15.39:50538) 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.988+0000 7f74ffe00640 1 -- 172.21.15.39:0/2088587672 learned_addr learned my addr 172.21.15.39:0/2088587672 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f7506a00640 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d225b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74ffe00640 1 -- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 msgr2=0x557579d225b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74ffe00640 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d225b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74ffe00640 1 -- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 msgr2=0x557579d1f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f7507400640 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d1f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74ffe00640 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d1f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74ffe00640 1 -- 172.21.15.39:0/2088587672 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557579c22780 con 0x557579cf2150 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f7506a00640 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d225b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f7507400640 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d1f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74ffe00640 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf2150 0x557579d1f440 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f74f0001e50 tx=0x7f74f0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d932fa5de2be54 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74f7400640 1 -- 172.21.15.39:0/2088587672 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f000d9a0 con 0x557579cf2150 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74f7400640 1 -- 172.21.15.39:0/2088587672 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f000db00 con 0x557579cf2150 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.989+0000 7f74f7400640 1 -- 172.21.15.39:0/2088587672 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f000de70 con 0x557579cf2150 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 -- 172.21.15.39:0/2088587672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf2150 msgr2=0x557579d1f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf2150 0x557579d1f440 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f74f0001e50 tx=0x7f74f0010000 comp rx=0 tx=0).stop 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 -- 172.21.15.39:0/2088587672 shutdown_connections 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d1f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf2150 0x557579d1f440 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 --2- 172.21.15.39:0/2088587672 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d225b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 -- 172.21.15.39:0/2088587672 >> 172.21.15.39:0/2088587672 conn(0x557579cf4460 msgr2=0x557579d3cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 -- 172.21.15.39:0/2088587672 shutdown_connections 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.990+0000 7f7510195740 1 -- 172.21.15.39:0/2088587672 wait complete. 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.991+0000 7f7510195740 1 -- start start 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.991+0000 7f7510195740 1 Processor -- start 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.992+0000 7f7510195740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d0e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.992+0000 7f7510195740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d27320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.992+0000 7f7510195740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579d0efd0 0x557579d282a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.992+0000 7f7510195740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557579c5b0f0 con 0x557579d0efd0 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.992+0000 7f7510195740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x557579bfc8c0 con 0x557579cf5d50 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.992+0000 7f7510195740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x557579bfc560 con 0x557579d221d0 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.992+0000 7f7510195740 10 client.0 _ll_register_callbacks cb 0x557579c54a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.992+0000 7f7507400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579d0efd0 0x557579d282a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7507400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579d0efd0 0x557579d282a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50540/0 (socket says 172.21.15.39:50540) 2024-01-24T17:43:04.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7507400640 1 -- 172.21.15.39:0/3867451419 learned_addr learned my addr 172.21.15.39:0/3867451419 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7506a00640 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d0e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f74ffe00640 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d27320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7507400640 1 -- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 msgr2=0x557579d27320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7507400640 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d27320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7507400640 1 -- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 msgr2=0x557579d0e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7507400640 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d0e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7507400640 1 -- 172.21.15.39:0/3867451419 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e8005730 con 0x557579d0efd0 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7506a00640 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d0e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f7507400640 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579d0efd0 0x557579d282a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7500009d90 tx=0x7f7500004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73286278f4122058 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.993+0000 7f74efe00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f74efe00640 1 -- 172.21.15.39:0/3867451419 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500034b80 con 0x557579d0efd0 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f74efe00640 1 -- 172.21.15.39:0/3867451419 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7500034ce0 con 0x557579d0efd0 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 1 -- 172.21.15.39:0/3867451419 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557579c85640 con 0x557579d0efd0 2024-01-24T17:43:04.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f74efe00640 1 -- 172.21.15.39:0/3867451419 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7500034ea0 con 0x557579d0efd0 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 10 client.5346 fetch_fsmap learned FSMap version 15 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 1 -- 172.21.15.39:0/3867451419 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x557579c59770 con 0x557579d0efd0 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f74efe00640 1 -- 172.21.15.39:0/3867451419 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557579c85640 con 0x557579d0efd0 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f74efe00640 1 -- 172.21.15.39:0/3867451419 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 15) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7500046030 con 0x557579d0efd0 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 10 client.5346 fetch_fsmap finished waiting for FSMap version 15 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 10 client.5346 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 20 client.5346 populate_metadata read hostname 'smithi039' 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] conn(0x557579d24010 0x557579d25690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 4 client.5346 mds_command: new command op to 5175 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.994+0000 7f7510195740 1 -- 172.21.15.39:0/3867451419 --> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557579c4d940 con 0x557579d24010 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.995+0000 7f7506a00640 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] conn(0x557579d24010 0x557579d25690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.995+0000 7f7506a00640 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] conn(0x557579d24010 0x557579d25690 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29748ea7a0b1a0bd server_cookie=e447082ed5bbe53b in_seq=0 out_seq=0 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.995+0000 7f74efe00640 10 client.5346 ms_handle_connect on v2:172.21.15.39:6833/3577904963 2024-01-24T17:43:04.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.996+0000 7f74efe00640 1 -- 172.21.15.39:0/3867451419 <== mds.0 v2:172.21.15.39:6833/3577904963 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x557579c4d940 con 0x557579d24010 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.996+0000 7f74efe00640 10 client.5346 handle_command_reply: tid=0 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.996+0000 7f7510195740 1 client.5346 shutdown 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.996+0000 7f7510195740 1 -- 172.21.15.39:0/3867451419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579d0efd0 msgr2=0x557579d282a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.996+0000 7f7510195740 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579d0efd0 0x557579d282a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7500009d90 tx=0x7f7500004ab0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 1 -- 172.21.15.39:0/3867451419 shutdown_connections 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] conn(0x557579d24010 0x557579d25690 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579d0efd0 0x557579d282a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d27320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 1 --2- 172.21.15.39:0/3867451419 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579cf5d50 0x557579d0e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f74efe00640 0 client.5346 ms_handle_reset on v2:172.21.15.39:6833/3577904963 2024-01-24T17:43:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 1 -- 172.21.15.39:0/3867451419 >> 172.21.15.39:0/3867451419 conn(0x557579cf2150 msgr2=0x557579d404f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 1 -- 172.21.15.39:0/3867451419 shutdown_connections 2024-01-24T17:43:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 1 -- 172.21.15.39:0/3867451419 wait complete. 2024-01-24T17:43:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:03.997+0000 7f7510195740 20 client.5346 trim_cache size 0 max 16384 2024-01-24T17:43:04.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.013+0000 7f7510195740 1 Processor -- start 2024-01-24T17:43:04.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.014+0000 7f7510195740 1 -- start start 2024-01-24T17:43:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.014+0000 7f7510195740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579d0e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.014+0000 7f7510195740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579d0efd0 0x557579d0e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.014+0000 7f7510195740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d404f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.014+0000 7f7510195740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575799c2f20 con 0x557579cf5d50 2024-01-24T17:43:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.014+0000 7f7510195740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x557579d3da50 con 0x557579d0efd0 2024-01-24T17:43:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.014+0000 7f7510195740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x557579d228c0 con 0x557579d221d0 2024-01-24T17:43:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579d0e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579d0e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50554/0 (socket says 172.21.15.39:50554) 2024-01-24T17:43:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 -- 172.21.15.39:0/1035711770 learned_addr learned my addr 172.21.15.39:0/1035711770 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 -- 172.21.15.39:0/1035711770 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 msgr2=0x557579d404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 --2- 172.21.15.39:0/1035711770 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 -- 172.21.15.39:0/1035711770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579d0efd0 msgr2=0x557579d0e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 --2- 172.21.15.39:0/1035711770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579d0efd0 0x557579d0e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 -- 172.21.15.39:0/1035711770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557579c8f2f0 con 0x557579cf5d50 2024-01-24T17:43:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.015+0000 7f7506a00640 1 --2- 172.21.15.39:0/1035711770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579d0e400 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f74f8007fd0 tx=0x7f74f8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8875478777daea server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f74f7e00640 1 -- 172.21.15.39:0/1035711770 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f805d0c0 con 0x557579cf5d50 2024-01-24T17:43:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f74f7e00640 1 -- 172.21.15.39:0/1035711770 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f80552b0 con 0x557579cf5d50 2024-01-24T17:43:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f74f7e00640 1 -- 172.21.15.39:0/1035711770 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8053880 con 0x557579cf5d50 2024-01-24T17:43:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 -- 172.21.15.39:0/1035711770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 msgr2=0x557579d0e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 --2- 172.21.15.39:0/1035711770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579d0e400 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f74f8007fd0 tx=0x7f74f8001080 comp rx=0 tx=0).stop 2024-01-24T17:43:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 -- 172.21.15.39:0/1035711770 shutdown_connections 2024-01-24T17:43:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 --2- 172.21.15.39:0/1035711770 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 --2- 172.21.15.39:0/1035711770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579d0efd0 0x557579d0e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 --2- 172.21.15.39:0/1035711770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579d0e400 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 -- 172.21.15.39:0/1035711770 >> 172.21.15.39:0/1035711770 conn(0x557579cf2150 msgr2=0x557579d27fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:04.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 -- 172.21.15.39:0/1035711770 shutdown_connections 2024-01-24T17:43:04.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.016+0000 7f7510195740 1 -- 172.21.15.39:0/1035711770 wait complete. 2024-01-24T17:43:04.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.017+0000 7f7510195740 1 -- start start 2024-01-24T17:43:04.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.017+0000 7f7510195740 1 Processor -- start 2024-01-24T17:43:04.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.018+0000 7f7510195740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579dab560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.018+0000 7f7510195740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579d0efd0 0x557579dac490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.018+0000 7f7510195740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d55cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.018+0000 7f7510195740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557579dabaa0 con 0x557579cf5d50 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.018+0000 7f7510195740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x557579d56890 con 0x557579d0efd0 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.018+0000 7f7510195740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x557579d1fa90 con 0x557579d221d0 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.018+0000 7f7510195740 10 client.0 _ll_register_callbacks cb 0x7f7500037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.018+0000 7f7506a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579dab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7506a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579dab560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50560/0 (socket says 172.21.15.39:50560) 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7506a00640 1 -- 172.21.15.39:0/280237243 learned_addr learned my addr 172.21.15.39:0/280237243 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7506a00640 1 -- 172.21.15.39:0/280237243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 msgr2=0x557579d55cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7506a00640 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d55cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7506a00640 1 -- 172.21.15.39:0/280237243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579d0efd0 msgr2=0x557579dac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7506a00640 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579d0efd0 0x557579dac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7506a00640 1 -- 172.21.15.39:0/280237243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7508156ad0 con 0x557579cf5d50 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7506a00640 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579dab560 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f74f8056e70 tx=0x7f74f80862c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6174fc4208e85d4a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f74efe00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f74efe00640 1 -- 172.21.15.39:0/280237243 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f805ecc0 con 0x557579cf5d50 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f74efe00640 1 -- 172.21.15.39:0/280237243 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74f8074020 con 0x557579cf5d50 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.019+0000 7f7510195740 1 -- 172.21.15.39:0/280237243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74f000db00 con 0x557579cf5d50 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f74efe00640 1 -- 172.21.15.39:0/280237243 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74f805ee20 con 0x557579cf5d50 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f7510195740 10 client.5352 fetch_fsmap learned FSMap version 15 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f7510195740 1 -- 172.21.15.39:0/280237243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x557579bfbad0 con 0x557579cf5d50 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f74efe00640 1 -- 172.21.15.39:0/280237243 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f000db00 con 0x557579cf5d50 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f74efe00640 1 -- 172.21.15.39:0/280237243 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 15) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f74f806aba0 con 0x557579cf5d50 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f7510195740 10 client.5352 fetch_fsmap finished waiting for FSMap version 15 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f7510195740 10 client.5352 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f7510195740 20 client.5352 populate_metadata read hostname 'smithi039' 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f7510195740 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] conn(0x557579d0e400 0x557579d0e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f7510195740 4 client.5352 mds_command: new command op to 5175 tid=0[{"prefix": "status"}] 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f7510195740 1 -- 172.21.15.39:0/280237243 --> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557579c54170 con 0x557579d0e400 2024-01-24T17:43:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.020+0000 7f74ffe00640 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] conn(0x557579d0e400 0x557579d0e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f74ffe00640 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] conn(0x557579d0e400 0x557579d0e7e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e929a724b695bf5a server_cookie=e81bd3a9555f2f1 in_seq=0 out_seq=0 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f74efe00640 10 client.5352 ms_handle_connect on v2:172.21.15.39:6833/3577904963 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f74efe00640 1 -- 172.21.15.39:0/280237243 <== mds.0 v2:172.21.15.39:6833/3577904963 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x557579c54170 con 0x557579d0e400 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f74efe00640 10 client.5352 handle_command_reply: tid=0 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 client.5352 shutdown 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 -- 172.21.15.39:0/280237243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 msgr2=0x557579dab560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579dab560 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f74f8056e70 tx=0x7f74f80862c0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 -- 172.21.15.39:0/280237243 shutdown_connections 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.39:6833/3577904963,v1:172.21.15.39:6835/3577904963] conn(0x557579d0e400 0x557579d0e7e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557579d221d0 0x557579d55cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557579d0efd0 0x557579dac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f74efe00640 0 client.5352 ms_handle_reset on v2:172.21.15.39:6833/3577904963 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 --2- 172.21.15.39:0/280237243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557579cf5d50 0x557579dab560 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.021+0000 7f7510195740 1 -- 172.21.15.39:0/280237243 >> 172.21.15.39:0/280237243 conn(0x557579cf2150 msgr2=0x557579d1cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7510195740 1 -- 172.21.15.39:0/280237243 shutdown_connections 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7510195740 1 -- 172.21.15.39:0/280237243 wait complete. 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7510195740 20 client.5352 trim_cache size 0 max 16384 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5175, 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:43:04.031 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 7.3159070450000003, 2024-01-24T17:43:04.032 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 15, 2024-01-24T17:43:04.032 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 34, 2024-01-24T17:43:04.032 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 34, 2024-01-24T17:43:04.032 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 15.76585195 2024-01-24T17:43:04.032 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 -- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74e006a430 msgr2=0x7f74e006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74e006a430 0x7f74e006c8f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f75080b2f70 tx=0x7f74f0009040 comp rx=0 tx=0).stop 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 -- 172.21.15.39:0/1413756611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f750806bdb0 msgr2=0x7f7508154b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f750806bdb0 0x7f7508154b80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f74f800c400 tx=0x7f74f800c8c0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 -- 172.21.15.39:0/1413756611 shutdown_connections 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75080cdc90 0x7f75080b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75080cd830 0x7f75080b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74e006a430 0x7f74e006c8f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 --2- 172.21.15.39:0/1413756611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f750806bdb0 0x7f7508154b80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.022+0000 7f7507e00640 1 -- 172.21.15.39:0/1413756611 >> 172.21.15.39:0/1413756611 conn(0x7f750806a2a0 msgr2=0x7f750815ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.023+0000 7f7507e00640 1 -- 172.21.15.39:0/1413756611 shutdown_connections 2024-01-24T17:43:04.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.023+0000 7f7507e00640 1 -- 172.21.15.39:0/1413756611 wait complete. 2024-01-24T17:43:04.041 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:43:04.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.255+0000 7f819da00640 1 Processor -- start 2024-01-24T17:43:04.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.256+0000 7f819da00640 1 -- start start 2024-01-24T17:43:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.257+0000 7f819da00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f819815cc90 0x7f81980ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.257+0000 7f819da00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f81980cde00 0x7f8198160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.257+0000 7f819da00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81980ce8b0 con 0x7f81980cb2b0 2024-01-24T17:43:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.258+0000 7f819da00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81980ceb80 con 0x7f81980cde00 2024-01-24T17:43:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.258+0000 7f819da00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81980cee60 con 0x7f819815cc90 2024-01-24T17:43:04.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.258+0000 7f8197e00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f81980cb2b0 0x7f81980cb690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:53850/0 (socket says 172.21.15.39:53850) 2024-01-24T17:43:04.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.258+0000 7f8197e00640 1 -- 172.21.15.39:0/4253076212 learned_addr learned my addr 172.21.15.39:0/4253076212 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:04.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.258+0000 7f818fe00640 1 --2- 172.21.15.39:0/4253076212 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f819815cc90 0x7f81980ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.258+0000 7f819d000640 1 --2- 172.21.15.39:0/4253076212 >> v2:172.21.15.100:3300/0 conn(0x7f81980cde00 0x7f8198160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.259+0000 7f818fe00640 1 -- 172.21.15.39:0/4253076212 >> v2:172.21.15.100:3300/0 conn(0x7f81980cde00 msgr2=0x7f8198160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.259+0000 7f818fe00640 1 --2- 172.21.15.39:0/4253076212 >> v2:172.21.15.100:3300/0 conn(0x7f81980cde00 0x7f8198160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.259+0000 7f818fe00640 1 -- 172.21.15.39:0/4253076212 >> v1:172.21.15.100:6789/0 conn(0x7f81980cb2b0 legacy=0x7f81980cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.259+0000 7f818fe00640 1 -- 172.21.15.39:0/4253076212 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81980cf140 con 0x7f819815cc90 2024-01-24T17:43:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.259+0000 7f8197400640 1 -- 172.21.15.39:0/4253076212 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521946645 0 0) 0x7f81980ce8b0 con 0x7f81980cb2b0 2024-01-24T17:43:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.260+0000 7f818fe00640 1 --2- 172.21.15.39:0/4253076212 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f819815cc90 0x7f81980ce370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f81800097a0 tx=0x7f818002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc36751605ab8cbb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.261+0000 7f8197400640 1 -- 172.21.15.39:0/4253076212 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180030610 con 0x7f819815cc90 2024-01-24T17:43:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.261+0000 7f8197400640 1 -- 172.21.15.39:0/4253076212 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8180030770 con 0x7f819815cc90 2024-01-24T17:43:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.261+0000 7f8197400640 1 -- 172.21.15.39:0/4253076212 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180030b00 con 0x7f819815cc90 2024-01-24T17:43:04.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.261+0000 7f819da00640 1 -- 172.21.15.39:0/4253076212 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f819815cc90 msgr2=0x7f81980ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.261+0000 7f819da00640 1 --2- 172.21.15.39:0/4253076212 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f819815cc90 0x7f81980ce370 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f81800097a0 tx=0x7f818002b960 comp rx=0 tx=0).stop 2024-01-24T17:43:04.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.262+0000 7f819da00640 1 -- 172.21.15.39:0/4253076212 shutdown_connections 2024-01-24T17:43:04.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.262+0000 7f819da00640 1 --2- 172.21.15.39:0/4253076212 >> v2:172.21.15.100:3300/0 conn(0x7f81980cde00 0x7f8198160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.262+0000 7f819da00640 1 --2- 172.21.15.39:0/4253076212 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f819815cc90 0x7f81980ce370 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.262+0000 7f819da00640 1 -- 172.21.15.39:0/4253076212 >> 172.21.15.39:0/4253076212 conn(0x7f819806a090 msgr2=0x7f819806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.262+0000 7f819da00640 1 -- 172.21.15.39:0/4253076212 shutdown_connections 2024-01-24T17:43:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.262+0000 7f819da00640 1 -- 172.21.15.39:0/4253076212 wait complete. 2024-01-24T17:43:04.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.263+0000 7f819da00640 1 Processor -- start 2024-01-24T17:43:04.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.264+0000 7f819da00640 1 -- start start 2024-01-24T17:43:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.264+0000 7f819da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81980cb2b0 0x7f819814fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.264+0000 7f819da00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81980cde00 0x7f8198070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f819da00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8198071360 0x7f819815df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f819da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81980717f0 con 0x7f81980cb2b0 2024-01-24T17:43:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f819da00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819814e1a0 con 0x7f81980cde00 2024-01-24T17:43:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f819da00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81980c3c00 con 0x7f8198071360 2024-01-24T17:43:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f8197e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81980cb2b0 0x7f819814fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f8197e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81980cb2b0 0x7f819814fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50564/0 (socket says 172.21.15.39:50564) 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f8197e00640 1 -- 172.21.15.39:0/2665634288 learned_addr learned my addr 172.21.15.39:0/2665634288 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f819d000640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8198071360 0x7f819815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f8197e00640 1 -- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8198071360 msgr2=0x7f819815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f818fe00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81980cde00 0x7f8198070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f8197e00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8198071360 0x7f819815df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f8197e00640 1 -- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81980cde00 msgr2=0x7f8198070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f8197e00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81980cde00 0x7f8198070e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.265+0000 7f8197e00640 1 -- 172.21.15.39:0/2665634288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8180009400 con 0x7f81980cb2b0 2024-01-24T17:43:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.266+0000 7f8197e00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81980cb2b0 0x7f819814fdd0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f818800c4a0 tx=0x7f818800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:04.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.266+0000 7f8194c00640 1 -- 172.21.15.39:0/2665634288 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81880074e0 con 0x7f81980cb2b0 2024-01-24T17:43:04.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.266+0000 7f8194c00640 1 -- 172.21.15.39:0/2665634288 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8188007640 con 0x7f81980cb2b0 2024-01-24T17:43:04.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.266+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81980c3ee0 con 0x7f81980cb2b0 2024-01-24T17:43:04.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.266+0000 7f8194c00640 1 -- 172.21.15.39:0/2665634288 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188004a00 con 0x7f81980cb2b0 2024-01-24T17:43:04.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.266+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81980c44a0 con 0x7f81980cb2b0 2024-01-24T17:43:04.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.267+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819804dff0 con 0x7f81980cb2b0 2024-01-24T17:43:04.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.269+0000 7f8194c00640 1 -- 172.21.15.39:0/2665634288 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8188004b60 con 0x7f81980cb2b0 2024-01-24T17:43:04.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.269+0000 7f8194c00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f817006a430 0x7f817006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:04.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.270+0000 7f8194c00640 1 -- 172.21.15.39:0/2665634288 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8188087d90 con 0x7f81980cb2b0 2024-01-24T17:43:04.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.270+0000 7f818fe00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f817006a430 0x7f817006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:04.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.272+0000 7f818fe00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f817006a430 0x7f817006c8f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8180036000 tx=0x7f818003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.275+0000 7f8194c00640 1 -- 172.21.15.39:0/2665634288 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8188053380 con 0x7f81980cb2b0 2024-01-24T17:43:04.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.507+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f81980cc5c0 con 0x7f81980cb2b0 2024-01-24T17:43:04.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.508+0000 7f8194c00640 1 -- 172.21.15.39:0/2665634288 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8188018070 con 0x7f81980cb2b0 2024-01-24T17:43:04.510 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:43:04.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f817006a430 msgr2=0x7f817006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f817006a430 0x7f817006c8f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8180036000 tx=0x7f818003b040 comp rx=0 tx=0).stop 2024-01-24T17:43:04.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81980cb2b0 msgr2=0x7f819814fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:04.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81980cb2b0 0x7f819814fdd0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f818800c4a0 tx=0x7f818800c960 comp rx=0 tx=0).stop 2024-01-24T17:43:04.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 shutdown_connections 2024-01-24T17:43:04.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8198071360 0x7f819815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f81980cde00 0x7f8198070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f817006a430 0x7f817006c8f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 --2- 172.21.15.39:0/2665634288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f81980cb2b0 0x7f819814fdd0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:04.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.510+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 >> 172.21.15.39:0/2665634288 conn(0x7f819806a090 msgr2=0x7f81980ae000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:04.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.511+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 shutdown_connections 2024-01-24T17:43:04.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:04.511+0000 7f819da00640 1 -- 172.21.15.39:0/2665634288 wait complete. 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:43:04.532 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:43:04.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:04.533 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:04.585 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42547sec preferred_lft 42547sec 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:04.586 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:43:04.587 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:04.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:04.588 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:43:04.681 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:43:04.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:04.686 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:43:04.747 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:43:04.747 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:43:04.748 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:43:04.748 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:43:04.748 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:43:04.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:04.749 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:43:04.749 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:43:04.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:43:04.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:43:04.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:43:04.749 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:43:04.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:04.942 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:43:04.997 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:43:04.997 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:43:04.998 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:43:04.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:04.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:43:05.101 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:43:05.101 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:43:05.101 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:05.159 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:43:05.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:43:05.160 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:05.215 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:43:05.313 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:05.313 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:05.412 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:05.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:05.417 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:43:05.474 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:43:05.474 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:43:05.481 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:43:05.560 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:05.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:05.630 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:05.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:05.634 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:43:05.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:43:05.674+0000 7f65f5237480 -1 init, newargv = 0x55afc351c400 newargc=15 2024-01-24T17:43:05.676 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[50687]: starting ceph client 2024-01-24T17:43:05.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[50687]: starting fuse 2024-01-24T17:43:06.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:43:06.888 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:06.888 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:06.985 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:06.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:06.991 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:43:07.049 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:43:07.049 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:43:07.049 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:43:07.049 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:43:07.049 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:43:07.049 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:43:07.049 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:43:07.050 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:43:07.051 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:43:07.051 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:43:07.051 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:43:07.051 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:43:07.051 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:43:07.051 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:43:07.051 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:43:07.051 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:43:07.238 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.50687.asok 2024-01-24T17:43:07.417 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.50687.asok 2024-01-24T17:43:07.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:07.418 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.50687.asok status 2024-01-24T17:43:07.664 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:43:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:43:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:43:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:43:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:43:07.664 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "50687", 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5361, 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5361 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:43:07.665 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 930633368 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 930633368 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.5361 192.168.0.1:0/930633368", 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/930633368", 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 15, 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 34, 2024-01-24T17:43:07.666 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:43:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:43:07.667 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:43:07.667 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:43:07.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:07.681 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:43:07.681 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:07.737 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:43:07.737 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:07.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:07.737 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:07.838 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:43:08.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.061+0000 7f7f18a00640 1 Processor -- start 2024-01-24T17:43:08.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.061+0000 7f7f18a00640 1 -- start start 2024-01-24T17:43:08.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.062+0000 7f7f18a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1415cbf0 0x7f7f140d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:08.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.062+0000 7f7f18a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7f1415db60 0x7f7f140ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:08.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.062+0000 7f7f18a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1415dfe0 con 0x7f7f140d0da0 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f18a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f140d0a90 con 0x7f7f1415db60 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f18a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140cd760 con 0x7f7f1415cbf0 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f12a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7f1415db60 0x7f7f140ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f12a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7f1415db60 0x7f7f140ce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50568/0 (socket says 172.21.15.39:50568) 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f12a00640 1 -- 172.21.15.39:0/3739915701 learned_addr learned my addr 172.21.15.39:0/3739915701 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f13400640 1 --2- 172.21.15.39:0/3739915701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1415cbf0 0x7f7f140d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f12a00640 1 -- 172.21.15.39:0/3739915701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1415cbf0 msgr2=0x7f7f140d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f12a00640 1 --2- 172.21.15.39:0/3739915701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1415cbf0 0x7f7f140d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f12a00640 1 -- 172.21.15.39:0/3739915701 >> v1:172.21.15.100:6789/0 conn(0x7f7f140d0da0 legacy=0x7f7f1415e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:08.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.063+0000 7f7f12a00640 1 -- 172.21.15.39:0/3739915701 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140cda40 con 0x7f7f1415db60 2024-01-24T17:43:08.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.064+0000 7f7f12a00640 1 --2- 172.21.15.39:0/3739915701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415db60 0x7f7f140ce240 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7efc009870 tx=0x7f7efc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a858f7073022b6e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:08.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.065+0000 7f7f12000640 1 -- 172.21.15.39:0/3739915701 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc031640 con 0x7f7f1415db60 2024-01-24T17:43:08.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.065+0000 7f7f12000640 1 -- 172.21.15.39:0/3739915701 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc0317a0 con 0x7f7f1415db60 2024-01-24T17:43:08.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.065+0000 7f7f12000640 1 -- 172.21.15.39:0/3739915701 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc031b30 con 0x7f7f1415db60 2024-01-24T17:43:08.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.066+0000 7f7f18a00640 1 -- 172.21.15.39:0/3739915701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415db60 msgr2=0x7f7f140ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:08.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.066+0000 7f7f18a00640 1 --2- 172.21.15.39:0/3739915701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415db60 0x7f7f140ce240 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7efc009870 tx=0x7f7efc02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.066+0000 7f7f18a00640 1 -- 172.21.15.39:0/3739915701 shutdown_connections 2024-01-24T17:43:08.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.066+0000 7f7f18a00640 1 --2- 172.21.15.39:0/3739915701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415db60 0x7f7f140ce240 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.066+0000 7f7f18a00640 1 --2- 172.21.15.39:0/3739915701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f1415cbf0 0x7f7f140d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.066+0000 7f7f18a00640 1 -- 172.21.15.39:0/3739915701 >> 172.21.15.39:0/3739915701 conn(0x7f7f1409bb90 msgr2=0x7f7f140d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:08.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.066+0000 7f7f18a00640 1 -- 172.21.15.39:0/3739915701 shutdown_connections 2024-01-24T17:43:08.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.066+0000 7f7f18a00640 1 -- 172.21.15.39:0/3739915701 wait complete. 2024-01-24T17:43:08.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.068+0000 7f7f18a00640 1 Processor -- start 2024-01-24T17:43:08.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.068+0000 7f7f18a00640 1 -- start start 2024-01-24T17:43:08.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f18a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f140d0da0 0x7f7f1409ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:08.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f18a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415cbf0 0x7f7f140c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:08.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f18a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f1415db60 0x7f7f140c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:08.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f18a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140c8bb0 con 0x7f7f1415cbf0 2024-01-24T17:43:08.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f18a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f141f6420 con 0x7f7f1415db60 2024-01-24T17:43:08.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f18a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1409e4b0 con 0x7f7f140d0da0 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f12a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415cbf0 0x7f7f140c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f12a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415cbf0 0x7f7f140c8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50580/0 (socket says 172.21.15.39:50580) 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.069+0000 7f7f12a00640 1 -- 172.21.15.39:0/4229581970 learned_addr learned my addr 172.21.15.39:0/4229581970 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.070+0000 7f7f13400640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f140d0da0 0x7f7f1409ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.070+0000 7f7f12a00640 1 -- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f140d0da0 msgr2=0x7f7f1409ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.070+0000 7f7f12a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f140d0da0 0x7f7f1409ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.070+0000 7f7f12a00640 1 -- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f1415db60 msgr2=0x7f7f140c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.070+0000 7f7f12a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f1415db60 0x7f7f140c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.070+0000 7f7f12a00640 1 -- 172.21.15.39:0/4229581970 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc0094d0 con 0x7f7f1415cbf0 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.070+0000 7f7f12a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415cbf0 0x7f7f140c8130 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7efc02b930 tx=0x7f7efc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.070+0000 7f7f0b400640 1 -- 172.21.15.39:0/4229581970 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc03ac60 con 0x7f7f1415cbf0 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.071+0000 7f7f0b400640 1 -- 172.21.15.39:0/4229581970 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc03ae00 con 0x7f7f1415cbf0 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.071+0000 7f7f0b400640 1 -- 172.21.15.39:0/4229581970 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc044630 con 0x7f7f1415cbf0 2024-01-24T17:43:08.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.071+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f141f5660 con 0x7f7f1415cbf0 2024-01-24T17:43:08.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.071+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f141f5c20 con 0x7f7f1415cbf0 2024-01-24T17:43:08.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.072+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1404dff0 con 0x7f7f1415cbf0 2024-01-24T17:43:08.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.073+0000 7f7f0b400640 1 -- 172.21.15.39:0/4229581970 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7efc02f050 con 0x7f7f1415cbf0 2024-01-24T17:43:08.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.074+0000 7f7f0b400640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7eec06a460 0x7f7eec06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:08.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.074+0000 7f7f0b400640 1 -- 172.21.15.39:0/4229581970 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7efc0aa990 con 0x7f7f1415cbf0 2024-01-24T17:43:08.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.075+0000 7f7f13400640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7eec06a460 0x7f7eec06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:08.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.076+0000 7f7f13400640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7eec06a460 0x7f7eec06c920 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7f141f6630 tx=0x7f7f04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:08.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.079+0000 7f7f0b400640 1 -- 172.21.15.39:0/4229581970 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7efc075f20 con 0x7f7f1415cbf0 2024-01-24T17:43:08.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.318+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7f140b4f60 con 0x7f7f1415cbf0 2024-01-24T17:43:08.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.319+0000 7f7f0b400640 1 -- 172.21.15.39:0/4229581970 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7efc04d450 con 0x7f7f1415cbf0 2024-01-24T17:43:08.321 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:43:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7eec06a460 msgr2=0x7f7eec06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7eec06a460 0x7f7eec06c920 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7f141f6630 tx=0x7f7f04008040 comp rx=0 tx=0).stop 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415cbf0 msgr2=0x7f7f140c8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415cbf0 0x7f7f140c8130 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7efc02b930 tx=0x7f7efc0046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 shutdown_connections 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f1415db60 0x7f7f140c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7eec06a460 0x7f7eec06c920 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f1415cbf0 0x7f7f140c8130 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 --2- 172.21.15.39:0/4229581970 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f140d0da0 0x7f7f1409ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 >> 172.21.15.39:0/4229581970 conn(0x7f7f1409bb90 msgr2=0x7f7f140ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 shutdown_connections 2024-01-24T17:43:08.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:08.322+0000 7f7f18a00640 1 -- 172.21.15.39:0/4229581970 wait complete. 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:43:08.342 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:43:08.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:08.343 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:08.399 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42541sec preferred_lft 42541sec 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:43:08.400 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:43:08.401 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:08.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:08.402 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:43:08.488 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:43:08.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:08.496 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:43:08.549 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:43:08.550 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:43:08.550 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:43:08.550 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:43:08.550 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:43:08.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:08.551 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:43:08.551 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:43:08.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:43:08.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:43:08.551 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:43:08.551 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:43:08.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:08.735 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:43:08.785 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:43:08.785 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:43:08.786 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:43:08.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:08.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:43:08.877 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-24T17:43:08.877 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:43:08.877 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:43:08.930 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:43:08.930 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:43:08.931 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:43:08.979 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:43:09.074 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:09.074 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:09.165 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:09.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:09.171 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:43:09.223 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:43:09.223 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-24T17:43:09.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:43:09.305 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:09.306 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:09.373 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:09.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:09.376 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:43:09.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:43:09.417+0000 7fd0607db480 -1 init, newargv = 0x55dfada9de60 newargc=15 2024-01-24T17:43:09.418 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[43805]: starting ceph client 2024-01-24T17:43:09.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[43805]: starting fuse 2024-01-24T17:43:10.518 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:43:10.620 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:10.620 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:10.711 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:10.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:10.716 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:43:10.771 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:43:10.772 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:43:10.772 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:43:10.772 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:43:10.773 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:43:10.773 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:43:10.773 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:43:10.773 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:43:10.773 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:43:10.773 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-24T17:43:10.774 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:43:10.774 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:43:10.774 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:43:10.774 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:43:10.774 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:43:10.775 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:43:10.775 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:43:10.775 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:43:10.775 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:43:10.775 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:43:10.775 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:43:10.776 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:43:10.776 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:43:10.776 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:43:10.776 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:43:10.776 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:43:10.776 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:43:10.777 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2024-01-24T17:43:10.777 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:43:10.951 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.43805.asok 2024-01-24T17:43:11.132 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43805.asok 2024-01-24T17:43:11.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:11.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43805.asok status 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "43805", 2024-01-24T17:43:11.375 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 5062, 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5062 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:43:11.376 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1894779038 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1894779038 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.5062 192.168.0.1:0/1894779038", 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/1894779038", 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 15, 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 34, 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:43:11.377 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:43:11.378 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:43:11.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:11.390 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:43:11.390 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:43:11.442 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:43:11.442 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:43:11.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:11.443 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:43:11.533 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds ms_async_reap_threshold 1 2024-01-24T17:43:11.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.770+0000 7f87d2200640 1 Processor -- start 2024-01-24T17:43:11.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.770+0000 7f87d2200640 1 -- start start 2024-01-24T17:43:11.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.771+0000 7f87d2200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f87cc1597f0 0x7f87cc06d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:11.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.771+0000 7f87d2200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc0b00e0 0x7f87cc0b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:11.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87d2200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc0b1bc0 con 0x7f87cc06bfa0 2024-01-24T17:43:11.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87d2200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87cc0ca0d0 con 0x7f87cc1597f0 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87d2200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0b0a90 con 0x7f87cc0b00e0 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87cb400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f87cc1597f0 0x7f87cc06d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87cb400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f87cc1597f0 0x7f87cc06d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50582/0 (socket says 172.21.15.39:50582) 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87cb400640 1 -- 172.21.15.39:0/3808946871 learned_addr learned my addr 172.21.15.39:0/3808946871 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87cb400640 1 -- 172.21.15.39:0/3808946871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc0b00e0 msgr2=0x7f87cc0b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87cb400640 1 --2- 172.21.15.39:0/3808946871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc0b00e0 0x7f87cc0b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87cb400640 1 -- 172.21.15.39:0/3808946871 >> v1:172.21.15.39:6789/0 conn(0x7f87cc06bfa0 legacy=0x7f87cc0b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.772+0000 7f87cb400640 1 -- 172.21.15.39:0/3808946871 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc06c4d0 con 0x7f87cc1597f0 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.773+0000 7f87cb400640 1 --2- 172.21.15.39:0/3808946871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc1597f0 0x7f87cc06d890 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0098a0 tx=0x7f87bc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de66e48cf0d057d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87caa00640 1 -- 172.21.15.39:0/3808946871 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc030840 con 0x7f87cc1597f0 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87caa00640 1 -- 172.21.15.39:0/3808946871 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc0309a0 con 0x7f87cc1597f0 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87caa00640 1 -- 172.21.15.39:0/3808946871 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc030d30 con 0x7f87cc1597f0 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87d2200640 1 -- 172.21.15.39:0/3808946871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc1597f0 msgr2=0x7f87cc06d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:11.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87d2200640 1 --2- 172.21.15.39:0/3808946871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc1597f0 0x7f87cc06d890 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0098a0 tx=0x7f87bc02b980 comp rx=0 tx=0).stop 2024-01-24T17:43:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87d2200640 1 -- 172.21.15.39:0/3808946871 shutdown_connections 2024-01-24T17:43:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87d2200640 1 --2- 172.21.15.39:0/3808946871 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc0b00e0 0x7f87cc0b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87d2200640 1 --2- 172.21.15.39:0/3808946871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc1597f0 0x7f87cc06d890 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.774+0000 7f87d2200640 1 -- 172.21.15.39:0/3808946871 >> 172.21.15.39:0/3808946871 conn(0x7f87cc0cb0e0 msgr2=0x7f87cc0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.775+0000 7f87d2200640 1 -- 172.21.15.39:0/3808946871 shutdown_connections 2024-01-24T17:43:11.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.775+0000 7f87d2200640 1 -- 172.21.15.39:0/3808946871 wait complete. 2024-01-24T17:43:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.776+0000 7f87d2200640 1 Processor -- start 2024-01-24T17:43:11.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.776+0000 7f87d2200640 1 -- start start 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.777+0000 7f87d2200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc06bfa0 0x7f87cc14eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.777+0000 7f87d2200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc0b00e0 0x7f87cc14f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.777+0000 7f87d2200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87cc1597f0 0x7f87cc14f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.777+0000 7f87d2200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc15afd0 con 0x7f87cc0b00e0 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.777+0000 7f87d2200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0c6150 con 0x7f87cc1597f0 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.777+0000 7f87d2200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0c6430 con 0x7f87cc06bfa0 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc0b00e0 0x7f87cc14f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc0b00e0 0x7f87cc14f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50588/0 (socket says 172.21.15.39:50588) 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 -- 172.21.15.39:0/2902622287 learned_addr learned my addr 172.21.15.39:0/2902622287 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87cb400640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc06bfa0 0x7f87cc14eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87cbe00640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87cc1597f0 0x7f87cc14f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 -- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc06bfa0 msgr2=0x7f87cc14eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:11.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc06bfa0 0x7f87cc14eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 -- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87cc1597f0 msgr2=0x7f87cc14f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:11.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87cc1597f0 0x7f87cc14f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 -- 172.21.15.39:0/2902622287 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc0094d0 con 0x7f87cc0b00e0 2024-01-24T17:43:11.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.778+0000 7f87c2800640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc0b00e0 0x7f87cc14f390 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c450 tx=0x7f87b400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:11.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.779+0000 7f87c3e00640 1 -- 172.21.15.39:0/2902622287 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b40052d0 con 0x7f87cc0b00e0 2024-01-24T17:43:11.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.779+0000 7f87c3e00640 1 -- 172.21.15.39:0/2902622287 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b4005430 con 0x7f87cc0b00e0 2024-01-24T17:43:11.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.779+0000 7f87c3e00640 1 -- 172.21.15.39:0/2902622287 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b4012870 con 0x7f87cc0b00e0 2024-01-24T17:43:11.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.779+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87cc0c6710 con 0x7f87cc0b00e0 2024-01-24T17:43:11.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.779+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc0c6cd0 con 0x7f87cc0b00e0 2024-01-24T17:43:11.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.780+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87cc073ef0 con 0x7f87cc0b00e0 2024-01-24T17:43:11.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.781+0000 7f87c3e00640 1 -- 172.21.15.39:0/2902622287 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f87b40129d0 con 0x7f87cc0b00e0 2024-01-24T17:43:11.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.782+0000 7f87c3e00640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f87a406a430 0x7f87a406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:11.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.782+0000 7f87c3e00640 1 -- 172.21.15.39:0/2902622287 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87b4088540 con 0x7f87cc0b00e0 2024-01-24T17:43:11.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.783+0000 7f87cb400640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f87a406a430 0x7f87a406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:11.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.784+0000 7f87cb400640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f87a406a430 0x7f87a406c8f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f87bc036000 tx=0x7f87bc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:11.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.787+0000 7f87c3e00640 1 -- 172.21.15.39:0/2902622287 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f87b4053a00 con 0x7f87cc0b00e0 2024-01-24T17:43:11.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.982+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=ms_async_reap_threshold}] v 0) v1 -- 0x7f87cc070850 con 0x7f87cc0b00e0 2024-01-24T17:43:11.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.990+0000 7f87c3e00640 1 -- 172.21.15.39:0/2902622287 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=ms_async_reap_threshold}]=0 v11)=0 v11) v1 ==== 121+0+0 (secure 0 0 0) 0x7f87b4057020 con 0x7f87cc0b00e0 2024-01-24T17:43:11.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f87a406a430 msgr2=0x7f87a406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:11.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f87a406a430 0x7f87a406c8f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f87bc036000 tx=0x7f87bc03b040 comp rx=0 tx=0).stop 2024-01-24T17:43:11.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc0b00e0 msgr2=0x7f87cc14f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:11.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc0b00e0 0x7f87cc14f390 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c450 tx=0x7f87b400c910 comp rx=0 tx=0).stop 2024-01-24T17:43:11.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 shutdown_connections 2024-01-24T17:43:11.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87cc1597f0 0x7f87cc14f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f87a406a430 0x7f87a406c8f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87cc0b00e0 0x7f87cc14f390 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 --2- 172.21.15.39:0/2902622287 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f87cc06bfa0 0x7f87cc14eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:11.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 >> 172.21.15.39:0/2902622287 conn(0x7f87cc0cb0e0 msgr2=0x7f87cc0ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:11.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 shutdown_connections 2024-01-24T17:43:11.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:11.992+0000 7f87d2200640 1 -- 172.21.15.39:0/2902622287 wait complete. 2024-01-24T17:43:12.012 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:43:12.013 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:12.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:12.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.292+0000 7f3710400640 1 Processor -- start 2024-01-24T17:43:12.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.292+0000 7f3710400640 1 -- start start 2024-01-24T17:43:12.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.293+0000 7f3710400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f37081654d0 0x7f37080c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.293+0000 7f3710400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f37080c90d0 0x7f3708070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.293+0000 7f3710400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081658b0 0x7f37080caf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.293+0000 7f3710400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37080cc9d0 con 0x7f37080c90d0 2024-01-24T17:43:12.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.293+0000 7f3710400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37080cc630 con 0x7f37081654d0 2024-01-24T17:43:12.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.293+0000 7f3710400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37080c9570 con 0x7f37081658b0 2024-01-24T17:43:12.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f37080c90d0 0x7f3708070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f37080c90d0 0x7f3708070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50592/0 (socket says 172.21.15.39:50592) 2024-01-24T17:43:12.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370d000640 1 -- 172.21.15.39:0/4255845278 learned_addr learned my addr 172.21.15.39:0/4255845278 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:12.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370da00640 1 --2- 172.21.15.39:0/4255845278 >> v2:172.21.15.100:3300/0 conn(0x7f37081654d0 0x7f37080c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370e400640 1 --2- 172.21.15.39:0/4255845278 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081658b0 0x7f37080caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370d000640 1 -- 172.21.15.39:0/4255845278 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081658b0 msgr2=0x7f37080caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:12.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370d000640 1 --2- 172.21.15.39:0/4255845278 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081658b0 0x7f37080caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370d000640 1 -- 172.21.15.39:0/4255845278 >> v2:172.21.15.100:3300/0 conn(0x7f37081654d0 msgr2=0x7f37080c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:12.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370d000640 1 --2- 172.21.15.39:0/4255845278 >> v2:172.21.15.100:3300/0 conn(0x7f37081654d0 0x7f37080c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370d000640 1 -- 172.21.15.39:0/4255845278 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37080c9850 con 0x7f37080c90d0 2024-01-24T17:43:12.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.294+0000 7f370da00640 1 --2- 172.21.15.39:0/4255845278 >> v2:172.21.15.100:3300/0 conn(0x7f37081654d0 0x7f37080c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:12.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.295+0000 7f370d000640 1 --2- 172.21.15.39:0/4255845278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f3708070030 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f37080cc2a0 tx=0x7f36f002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb8537f76031180 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:12.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.296+0000 7f3707e00640 1 -- 172.21.15.39:0/4255845278 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0031740 con 0x7f37080c90d0 2024-01-24T17:43:12.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.296+0000 7f3707e00640 1 -- 172.21.15.39:0/4255845278 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f00318a0 con 0x7f37080c90d0 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.296+0000 7f3707e00640 1 -- 172.21.15.39:0/4255845278 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0031c30 con 0x7f37080c90d0 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.296+0000 7f3710400640 1 -- 172.21.15.39:0/4255845278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 msgr2=0x7f3708070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.296+0000 7f3710400640 1 --2- 172.21.15.39:0/4255845278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f3708070030 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f37080cc2a0 tx=0x7f36f002b7e0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.297+0000 7f3710400640 1 -- 172.21.15.39:0/4255845278 shutdown_connections 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.297+0000 7f3710400640 1 --2- 172.21.15.39:0/4255845278 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081658b0 0x7f37080caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.297+0000 7f3710400640 1 --2- 172.21.15.39:0/4255845278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f3708070030 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.297+0000 7f3710400640 1 --2- 172.21.15.39:0/4255845278 >> v2:172.21.15.100:3300/0 conn(0x7f37081654d0 0x7f37080c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.297+0000 7f3710400640 1 -- 172.21.15.39:0/4255845278 >> 172.21.15.39:0/4255845278 conn(0x7f370806a3e0 msgr2=0x7f370806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.297+0000 7f3710400640 1 -- 172.21.15.39:0/4255845278 shutdown_connections 2024-01-24T17:43:12.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.297+0000 7f3710400640 1 -- 172.21.15.39:0/4255845278 wait complete. 2024-01-24T17:43:12.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.299+0000 7f3710400640 1 Processor -- start 2024-01-24T17:43:12.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.299+0000 7f3710400640 1 -- start start 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f3710400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f37080c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f3710400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081654d0 0x7f3708071080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f3710400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37081658b0 0x7f37080715c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f3710400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37080b1140 con 0x7f37080c90d0 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f3710400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37080c4c60 con 0x7f37081658b0 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f3710400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37080b13e0 con 0x7f37081654d0 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f37080c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f37080c4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50602/0 (socket says 172.21.15.39:50602) 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370da00640 1 -- 172.21.15.39:0/919416981 learned_addr learned my addr 172.21.15.39:0/919416981 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370d000640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081654d0 0x7f3708071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370da00640 1 -- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081654d0 msgr2=0x7f3708071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370da00640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081654d0 0x7f3708071080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370da00640 1 -- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37081658b0 msgr2=0x7f37080715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370da00640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37081658b0 0x7f37080715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.300+0000 7f370da00640 1 -- 172.21.15.39:0/919416981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f00094d0 con 0x7f37080c90d0 2024-01-24T17:43:12.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.301+0000 7f370da00640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f37080c4720 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f36f800e810 tx=0x7f36f800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:12.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.301+0000 7f3705600640 1 -- 172.21.15.39:0/919416981 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8004030 con 0x7f37080c90d0 2024-01-24T17:43:12.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.301+0000 7f3705600640 1 -- 172.21.15.39:0/919416981 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8004190 con 0x7f37080c90d0 2024-01-24T17:43:12.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.301+0000 7f3705600640 1 -- 172.21.15.39:0/919416981 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8004520 con 0x7f37080c90d0 2024-01-24T17:43:12.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.301+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37080b16c0 con 0x7f37080c90d0 2024-01-24T17:43:12.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.301+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370814e490 con 0x7f37080c90d0 2024-01-24T17:43:12.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.302+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37080c9b30 con 0x7f37080c90d0 2024-01-24T17:43:12.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.303+0000 7f3705600640 1 -- 172.21.15.39:0/919416981 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f36f800c6d0 con 0x7f37080c90d0 2024-01-24T17:43:12.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.305+0000 7f3705600640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36e006a430 0x7f36e006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.305+0000 7f3705600640 1 -- 172.21.15.39:0/919416981 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36f80885a0 con 0x7f37080c90d0 2024-01-24T17:43:12.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.305+0000 7f370d000640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36e006a430 0x7f36e006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.306+0000 7f370d000640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36e006a430 0x7f36e006c8f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f36f0035000 tx=0x7f36f0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:12.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.309+0000 7f3705600640 1 -- 172.21.15.39:0/919416981 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f36f8053af0 con 0x7f37080c90d0 2024-01-24T17:43:12.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.503+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3708073ef0 con 0x7f37080c90d0 2024-01-24T17:43:12.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.506+0000 7f3705600640 1 -- 172.21.15.39:0/919416981 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f36f8057110 con 0x7f37080c90d0 2024-01-24T17:43:12.507 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:12.507 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:57.705789+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:12.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36e006a430 msgr2=0x7f36e006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36e006a430 0x7f36e006c8f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f36f0035000 tx=0x7f36f0008030 comp rx=0 tx=0).stop 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 msgr2=0x7f37080c4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f37080c4720 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f36f800e810 tx=0x7f36f800ecd0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 shutdown_connections 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f37081658b0 0x7f37080715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f37081654d0 0x7f3708071080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36e006a430 0x7f36e006c8f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 --2- 172.21.15.39:0/919416981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37080c90d0 0x7f37080c4720 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 >> 172.21.15.39:0/919416981 conn(0x7f370806a3e0 msgr2=0x7f370806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 shutdown_connections 2024-01-24T17:43:12.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.509+0000 7f3710400640 1 -- 172.21.15.39:0/919416981 wait complete. 2024-01-24T17:43:12.528 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:42:54.883732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:42:55.727240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:43:12.529 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:43:12.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.755+0000 7fcca3e00640 1 Processor -- start 2024-01-24T17:43:12.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.755+0000 7fcca3e00640 1 -- start start 2024-01-24T17:43:12.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.757+0000 7fcca3e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcca415a820 0x7fcca40cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.757+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca40cf870 0x7fcca415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.757+0000 7fcca3e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca4157750 con 0x7fcca4157370 2024-01-24T17:43:12.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.757+0000 7fcca3e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca415acd0 con 0x7fcca415a820 2024-01-24T17:43:12.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.757+0000 7fcca3e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca415a390 con 0x7fcca40cf870 2024-01-24T17:43:12.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.758+0000 7fcca2a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fcca4157370 0x7fcca415d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58348/0 (socket says 172.21.15.39:58348) 2024-01-24T17:43:12.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.758+0000 7fcca2a00640 1 -- 172.21.15.39:0/3760270613 learned_addr learned my addr 172.21.15.39:0/3760270613 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:12.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.758+0000 7fcca3400640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca40cf870 0x7fcca415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.758+0000 7fcca2000640 1 --2- 172.21.15.39:0/3760270613 >> v2:172.21.15.100:3300/0 conn(0x7fcca415a820 0x7fcca40cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.758+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074321961 0 0) 0x7fcc94001110 con 0x7fcca4157370 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.758+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc84003520 con 0x7fcca4157370 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1292831602 0 0) 0x7fcc84003520 con 0x7fcca4157370 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca40cf870 msgr2=0x7fcca415ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca40cf870 0x7fcca415ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 >> v2:172.21.15.100:3300/0 conn(0x7fcca415a820 msgr2=0x7fcca40cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 --2- 172.21.15.39:0/3760270613 >> v2:172.21.15.100:3300/0 conn(0x7fcca415a820 0x7fcca40cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca4071f60 con 0x7fcca4157370 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fcc94001800 con 0x7fcca4157370 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 >> v1:172.21.15.39:6789/0 conn(0x7fcca4157370 legacy=0x7fcca415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc840039a0 0x7fcc84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840047c0 0x7fcc84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca40cf870 0x7fcc84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca415acd0 con 0x7fcc840039a0 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca415a390 con 0x7fcc840047c0 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8400b980 con 0x7fcca40cf870 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca2a00640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc840039a0 0x7fcc84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.759+0000 7fcca2000640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840047c0 0x7fcc84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.760+0000 7fcca3400640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca40cf870 0x7fcc84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.760+0000 7fcca2a00640 1 -- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca40cf870 msgr2=0x7fcc84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.760+0000 7fcca2a00640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca40cf870 0x7fcc84005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.760+0000 7fcca2a00640 1 -- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840047c0 msgr2=0x7fcc84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.760+0000 7fcca2a00640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840047c0 0x7fcc84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.760+0000 7fcca2a00640 1 -- 172.21.15.39:0/3760270613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc84003520 con 0x7fcc840039a0 2024-01-24T17:43:12.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.761+0000 7fcca2a00640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc840039a0 0x7fcc84003dc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcc94007fc0 tx=0x7fcc840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1ecc35ed687232 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:12.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.761+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94005ad0 con 0x7fcc840039a0 2024-01-24T17:43:12.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.761+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94005c30 con 0x7fcc840039a0 2024-01-24T17:43:12.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.761+0000 7fcca1600640 1 -- 172.21.15.39:0/3760270613 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9402c7e0 con 0x7fcc840039a0 2024-01-24T17:43:12.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.762+0000 7fcca3e00640 1 -- 172.21.15.39:0/3760270613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc840039a0 msgr2=0x7fcc84003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.762+0000 7fcca3e00640 1 --2- 172.21.15.39:0/3760270613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc840039a0 0x7fcc84003dc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcc94007fc0 tx=0x7fcc840057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.762+0000 7fcca3400640 1 -- 172.21.15.39:0/3760270613 reap_dead start 2024-01-24T17:43:12.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.762+0000 7fcca3e00640 1 -- 172.21.15.39:0/3760270613 shutdown_connections 2024-01-24T17:43:12.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.762+0000 7fcca3e00640 1 -- 172.21.15.39:0/3760270613 >> 172.21.15.39:0/3760270613 conn(0x7fcca40b10a0 msgr2=0x7fcca40b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:12.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.762+0000 7fcca3e00640 1 -- 172.21.15.39:0/3760270613 shutdown_connections 2024-01-24T17:43:12.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.762+0000 7fcca3e00640 1 -- 172.21.15.39:0/3760270613 wait complete. 2024-01-24T17:43:12.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.764+0000 7fcca3e00640 1 Processor -- start 2024-01-24T17:43:12.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.764+0000 7fcca3e00640 1 -- start start 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.765+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840039a0 0x7fcca40c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.765+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca4155da0 0x7fcca40cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.765+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcca4156220 0x7fcca40cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.765+0000 7fcca3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40b3930 con 0x7fcca4156220 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.765+0000 7fcca3e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40c4820 con 0x7fcc840039a0 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.765+0000 7fcca3e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca40c3f80 con 0x7fcca4155da0 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.765+0000 7fcca3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcca4156220 0x7fcca40cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.765+0000 7fcca3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcca4156220 0x7fcca40cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50618/0 (socket says 172.21.15.39:50618) 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca3400640 1 -- 172.21.15.39:0/1117393460 learned_addr learned my addr 172.21.15.39:0/1117393460 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca2a00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840039a0 0x7fcca40c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca2000640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca4155da0 0x7fcca40cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca3400640 1 -- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca4155da0 msgr2=0x7fcca40cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca3400640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca4155da0 0x7fcca40cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca3400640 1 -- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840039a0 msgr2=0x7fcca40c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca3400640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840039a0 0x7fcca40c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca3400640 1 -- 172.21.15.39:0/1117393460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca40c4260 con 0x7fcca4156220 2024-01-24T17:43:12.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca2a00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840039a0 0x7fcca40c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:12.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca2000640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca4155da0 0x7fcca40cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:12.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.766+0000 7fcca3400640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcca4156220 0x7fcca40cd600 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcca40b1c20 tx=0x7fcc9c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:12.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.767+0000 7fcc9aa00640 1 -- 172.21.15.39:0/1117393460 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c00e070 con 0x7fcca4156220 2024-01-24T17:43:12.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.767+0000 7fcc9aa00640 1 -- 172.21.15.39:0/1117393460 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c011e40 con 0x7fcca4156220 2024-01-24T17:43:12.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.767+0000 7fcc9aa00640 1 -- 172.21.15.39:0/1117393460 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c0039e0 con 0x7fcca4156220 2024-01-24T17:43:12.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.767+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca40c4540 con 0x7fcca4156220 2024-01-24T17:43:12.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.767+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca40c61b0 con 0x7fcca4156220 2024-01-24T17:43:12.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.768+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca40b3420 con 0x7fcca4156220 2024-01-24T17:43:12.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.769+0000 7fcc9aa00640 1 -- 172.21.15.39:0/1117393460 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcc9c003b60 con 0x7fcca4156220 2024-01-24T17:43:12.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.770+0000 7fcc9aa00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcc7c06a480 0x7fcc7c06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:12.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.771+0000 7fcc9aa00640 1 -- 172.21.15.39:0/1117393460 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc9c089100 con 0x7fcca4156220 2024-01-24T17:43:12.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.771+0000 7fcca2a00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcc7c06a480 0x7fcc7c06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:12.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.772+0000 7fcca2a00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcc7c06a480 0x7fcc7c06c940 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc94034000 tx=0x7fcc94037040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:12.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.775+0000 7fcc9aa00640 1 -- 172.21.15.39:0/1117393460 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcca40b3420 con 0x7fcca4156220 2024-01-24T17:43:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.970+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcca40b3ae0 con 0x7fcca4156220 2024-01-24T17:43:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.971+0000 7fcc9aa00640 1 -- 172.21.15.39:0/1117393460 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcca40b3ae0 con 0x7fcca4156220 2024-01-24T17:43:12.972 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:43:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcc7c06a480 msgr2=0x7fcc7c06c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcc7c06a480 0x7fcc7c06c940 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc94034000 tx=0x7fcc94037040 comp rx=0 tx=0).stop 2024-01-24T17:43:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcca4156220 msgr2=0x7fcca40cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcca4156220 0x7fcca40cd600 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fcca40b1c20 tx=0x7fcc9c0115a0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 shutdown_connections 2024-01-24T17:43:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcc7c06a480 0x7fcc7c06c940 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcca4156220 0x7fcca40cd600 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcca4155da0 0x7fcca40cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 --2- 172.21.15.39:0/1117393460 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcc840039a0 0x7fcca40c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 >> 172.21.15.39:0/1117393460 conn(0x7fcca40b10a0 msgr2=0x7fcca4072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 shutdown_connections 2024-01-24T17:43:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:12.974+0000 7fcca3e00640 1 -- 172.21.15.39:0/1117393460 wait complete. 2024-01-24T17:43:12.993 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:43:13.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.214+0000 7f9077000640 1 Processor -- start 2024-01-24T17:43:13.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.214+0000 7f9077000640 1 -- start start 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f9077000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90700b2230 0x7f90700b2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f9077000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f907006bdd0 0x7f9070164b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f9077000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90700c3b20 0x7f9070165090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f9077000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90700ce1b0 con 0x7f90700b2230 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f9077000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907006c7d0 con 0x7f907006bdd0 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f9077000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90700b2b50 con 0x7f90700c3b20 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f906fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90700b2230 0x7f90700b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f906fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f90700b2230 0x7f90700b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50010/0 (socket says 172.21.15.39:50010) 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f906fe00640 1 -- 172.21.15.39:0/3736795177 learned_addr learned my addr 172.21.15.39:0/3736795177 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:13.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f906fe00640 1 -- 172.21.15.39:0/3736795177 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90700c3b20 msgr2=0x7f9070165090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:13.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f906fe00640 1 --2- 172.21.15.39:0/3736795177 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90700c3b20 0x7f9070165090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f906fe00640 1 -- 172.21.15.39:0/3736795177 >> v2:172.21.15.100:3300/0 conn(0x7f907006bdd0 msgr2=0x7f9070164b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:13.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f906fe00640 1 --2- 172.21.15.39:0/3736795177 >> v2:172.21.15.100:3300/0 conn(0x7f907006bdd0 0x7f9070164b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.216+0000 7f906fe00640 1 -- 172.21.15.39:0/3736795177 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907006ca20 con 0x7f90700b2230 2024-01-24T17:43:13.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.218+0000 7f906fe00640 1 --2- 172.21.15.39:0/3736795177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700b2230 0x7f90700b2610 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f90700736b0 tx=0x7f906002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b052e7b5e48bdeaa server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:13.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.218+0000 7f906ea00640 1 -- 172.21.15.39:0/3736795177 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060031870 con 0x7f90700b2230 2024-01-24T17:43:13.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.218+0000 7f906ea00640 1 -- 172.21.15.39:0/3736795177 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90600319d0 con 0x7f90700b2230 2024-01-24T17:43:13.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.218+0000 7f906ea00640 1 -- 172.21.15.39:0/3736795177 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060031d60 con 0x7f90700b2230 2024-01-24T17:43:13.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 -- 172.21.15.39:0/3736795177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700b2230 msgr2=0x7f90700b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 --2- 172.21.15.39:0/3736795177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700b2230 0x7f90700b2610 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f90700736b0 tx=0x7f906002b910 comp rx=0 tx=0).stop 2024-01-24T17:43:13.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 -- 172.21.15.39:0/3736795177 shutdown_connections 2024-01-24T17:43:13.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 --2- 172.21.15.39:0/3736795177 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f90700c3b20 0x7f9070165090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 --2- 172.21.15.39:0/3736795177 >> v2:172.21.15.100:3300/0 conn(0x7f907006bdd0 0x7f9070164b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 --2- 172.21.15.39:0/3736795177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700b2230 0x7f90700b2610 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 -- 172.21.15.39:0/3736795177 >> 172.21.15.39:0/3736795177 conn(0x7f90700cd230 msgr2=0x7f90700cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:13.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 -- 172.21.15.39:0/3736795177 shutdown_connections 2024-01-24T17:43:13.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.219+0000 7f9077000640 1 -- 172.21.15.39:0/3736795177 wait complete. 2024-01-24T17:43:13.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.221+0000 7f9077000640 1 Processor -- start 2024-01-24T17:43:13.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.221+0000 7f9077000640 1 -- start start 2024-01-24T17:43:13.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9077000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f907006bdd0 0x7f907015a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9077000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90700b2230 0x7f9070163790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9077000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700c3b20 0x7f9070164890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9077000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90701eace0 con 0x7f90700c3b20 2024-01-24T17:43:13.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9077000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907015a7e0 con 0x7f90700b2230 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9077000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9070163cd0 con 0x7f907006bdd0 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9075000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700c3b20 0x7f9070164890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9075000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700c3b20 0x7f9070164890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50016/0 (socket says 172.21.15.39:50016) 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.222+0000 7f9075000640 1 -- 172.21.15.39:0/4216090234 learned_addr learned my addr 172.21.15.39:0/4216090234 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f906fe00640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f907006bdd0 0x7f907015a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f906f400640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90700b2230 0x7f9070163790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9075000640 1 -- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f907006bdd0 msgr2=0x7f907015a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9075000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f907006bdd0 0x7f907015a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9075000640 1 -- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90700b2230 msgr2=0x7f9070163790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9075000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90700b2230 0x7f9070163790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9075000640 1 -- 172.21.15.39:0/4216090234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600094d0 con 0x7f90700c3b20 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f906f400640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90700b2230 0x7f9070163790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9075000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700c3b20 0x7f9070164890 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f906800b970 tx=0x7f906800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:13.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9067e00640 1 -- 172.21.15.39:0/4216090234 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906800cb40 con 0x7f90700c3b20 2024-01-24T17:43:13.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9067e00640 1 -- 172.21.15.39:0/4216090234 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906800ccc0 con 0x7f90700c3b20 2024-01-24T17:43:13.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9067e00640 1 -- 172.21.15.39:0/4216090234 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068014680 con 0x7f90700c3b20 2024-01-24T17:43:13.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.223+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9070163f70 con 0x7f90700c3b20 2024-01-24T17:43:13.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.224+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90700c7d00 con 0x7f90700c3b20 2024-01-24T17:43:13.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.225+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90700c7880 con 0x7f90700c3b20 2024-01-24T17:43:13.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.226+0000 7f9067e00640 1 -- 172.21.15.39:0/4216090234 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f906801d3b0 con 0x7f90700c3b20 2024-01-24T17:43:13.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.227+0000 7f9067e00640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f904806a430 0x7f904806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.227+0000 7f9067e00640 1 -- 172.21.15.39:0/4216090234 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90680888d0 con 0x7f90700c3b20 2024-01-24T17:43:13.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.227+0000 7f906fe00640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f904806a430 0x7f904806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.229+0000 7f906fe00640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f904806a430 0x7f904806c8f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9060036000 tx=0x7f906003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:13.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.231+0000 7f9067e00640 1 -- 172.21.15.39:0/4216090234 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f90700c7880 con 0x7f90700c3b20 2024-01-24T17:43:13.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.422+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90700c7880 con 0x7f90700c3b20 2024-01-24T17:43:13.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.422+0000 7f9067e00640 1 -- 172.21.15.39:0/4216090234 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90700c7880 con 0x7f90700c3b20 2024-01-24T17:43:13.423 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:43:13.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f904806a430 msgr2=0x7f904806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f904806a430 0x7f904806c8f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9060036000 tx=0x7f906003c040 comp rx=0 tx=0).stop 2024-01-24T17:43:13.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700c3b20 msgr2=0x7f9070164890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700c3b20 0x7f9070164890 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f906800b970 tx=0x7f906800be30 comp rx=0 tx=0).stop 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 shutdown_connections 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f904806a430 0x7f904806c8f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90700c3b20 0x7f9070164890 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90700b2230 0x7f9070163790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 --2- 172.21.15.39:0/4216090234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f907006bdd0 0x7f907015a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 >> 172.21.15.39:0/4216090234 conn(0x7f90700cd230 msgr2=0x7f90700b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.425+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 shutdown_connections 2024-01-24T17:43:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.426+0000 7f9077000640 1 -- 172.21.15.39:0/4216090234 wait complete. 2024-01-24T17:43:13.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:43:13.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.669+0000 7f8dfd400640 1 Processor -- start 2024-01-24T17:43:13.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.670+0000 7f8dfd400640 1 -- start start 2024-01-24T17:43:13.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.671+0000 7f8dfd400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 0x7f8df80d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.671+0000 7f8dfd400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df80b38a0 con 0x7f8df815c8d0 2024-01-24T17:43:13.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.671+0000 7f8dfd400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df80d13e0 con 0x7f8df80b2710 2024-01-24T17:43:13.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.671+0000 7f8dfd400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80d0ab0 con 0x7f8df80d2e80 2024-01-24T17:43:13.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.671+0000 7f8df7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8df815c8d0 0x7f8df80d2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50008/0 (socket says 172.21.15.39:50008) 2024-01-24T17:43:13.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.671+0000 7f8df7e00640 1 -- 172.21.15.39:0/2012120005 learned_addr learned my addr 172.21.15.39:0/2012120005 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:13.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.671+0000 7f8df7400640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 0x7f8df80d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685176092 0 0) 0x7f8df80b38a0 con 0x7f8df815c8d0 2024-01-24T17:43:13.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd8003520 con 0x7f8df815c8d0 2024-01-24T17:43:13.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3631628877 0 0) 0x7f8dd8003520 con 0x7f8df815c8d0 2024-01-24T17:43:13.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 msgr2=0x7f8df80d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:13.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 0x7f8df80d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 >> v1:172.21.15.100:6789/0 conn(0x7f8df80b2710 legacy=0x7f8df80cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80d0d90 con 0x7f8df815c8d0 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8de80019f0 con 0x7f8df815c8d0 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 >> v1:172.21.15.39:6789/0 conn(0x7f8df815c8d0 legacy=0x7f8df80d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dd80039a0 0x7f8dd8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 0x7f8dd80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dd8004d00 0x7f8dd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80d0ab0 con 0x7f8dd80039a0 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd800ba60 con 0x7f8dd8004d00 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd800bc20 con 0x7f8df80d2e80 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.672+0000 7f8df7e00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dd80039a0 0x7f8dd8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.673+0000 7f8df7400640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 0x7f8dd80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.673+0000 7f8dfca00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dd8004d00 0x7f8dd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.673+0000 7f8df7e00640 1 -- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 msgr2=0x7f8dd80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.673+0000 7f8df7e00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 0x7f8dd80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.673+0000 7f8df7e00640 1 -- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dd8004d00 msgr2=0x7f8dd8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:13.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.673+0000 7f8df7e00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8dd8004d00 0x7f8dd8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.673+0000 7f8df7e00640 1 -- 172.21.15.39:0/2012120005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8003520 con 0x7f8dd80039a0 2024-01-24T17:43:13.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.673+0000 7f8df7400640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8df80d2e80 0x7f8dd80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.674+0000 7f8df7e00640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dd80039a0 0x7f8dd8003dc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8de8002230 tx=0x7f8dd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb472bb42df8c35 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.674+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8034630 con 0x7f8dd80039a0 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.674+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de802f070 con 0x7f8dd80039a0 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.674+0000 7f8df6a00640 1 -- 172.21.15.39:0/2012120005 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8034a10 con 0x7f8dd80039a0 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.675+0000 7f8dfd400640 1 -- 172.21.15.39:0/2012120005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dd80039a0 msgr2=0x7f8dd8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.675+0000 7f8dfd400640 1 --2- 172.21.15.39:0/2012120005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dd80039a0 0x7f8dd8003dc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8de8002230 tx=0x7f8dd80057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.675+0000 7f8dfca00640 1 -- 172.21.15.39:0/2012120005 reap_dead start 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.675+0000 7f8dfd400640 1 -- 172.21.15.39:0/2012120005 shutdown_connections 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.675+0000 7f8dfd400640 1 -- 172.21.15.39:0/2012120005 >> 172.21.15.39:0/2012120005 conn(0x7f8df8071dd0 msgr2=0x7f8df809bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:13.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.675+0000 7f8dfd400640 1 -- 172.21.15.39:0/2012120005 shutdown_connections 2024-01-24T17:43:13.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.675+0000 7f8dfd400640 1 -- 172.21.15.39:0/2012120005 wait complete. 2024-01-24T17:43:13.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.677+0000 7f8dfd400640 1 Processor -- start 2024-01-24T17:43:13.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.677+0000 7f8dfd400640 1 -- start start 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.678+0000 7f8dfd400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dd80039a0 0x7f8df8152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.678+0000 7f8dfd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8df81f2740 0x7f8df80b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.678+0000 7f8dfd400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8df81f2bc0 0x7f8df80b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.678+0000 7f8dfd400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df815f450 con 0x7f8df81f2740 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.678+0000 7f8dfd400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df815e840 con 0x7f8df81f2bc0 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.678+0000 7f8dfd400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df815eb20 con 0x7f8dd80039a0 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8df81f2740 0x7f8df80b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8df81f2740 0x7f8df80b75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50028/0 (socket says 172.21.15.39:50028) 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7400640 1 -- 172.21.15.39:0/2923666540 learned_addr learned my addr 172.21.15.39:0/2923666540 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8dfca00640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8df81f2bc0 0x7f8df80b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7e00640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dd80039a0 0x7f8df8152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7400640 1 -- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dd80039a0 msgr2=0x7f8df8152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dd80039a0 0x7f8df8152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7400640 1 -- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8df81f2bc0 msgr2=0x7f8df80b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8df81f2bc0 0x7f8df80b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8df7400640 1 -- 172.21.15.39:0/2923666540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df815ee00 con 0x7f8df81f2740 2024-01-24T17:43:13.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.679+0000 7f8dfca00640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8df81f2bc0 0x7f8df80b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:13.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.680+0000 7f8df7400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8df81f2740 0x7f8df80b75d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8de00116d0 tx=0x7f8de0011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.680+0000 7f8defe00640 1 -- 172.21.15.39:0/2923666540 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de000f9c0 con 0x7f8df81f2740 2024-01-24T17:43:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.680+0000 7f8defe00640 1 -- 172.21.15.39:0/2923666540 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de000fb20 con 0x7f8df81f2740 2024-01-24T17:43:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.680+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df815f0e0 con 0x7f8df81f2740 2024-01-24T17:43:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.680+0000 7f8defe00640 1 -- 172.21.15.39:0/2923666540 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de000fed0 con 0x7f8df81f2740 2024-01-24T17:43:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.680+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df81f2270 con 0x7f8df81f2740 2024-01-24T17:43:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.681+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df804dff0 con 0x7f8df81f2740 2024-01-24T17:43:13.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.682+0000 7f8defe00640 1 -- 172.21.15.39:0/2923666540 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8de0015460 con 0x7f8df81f2740 2024-01-24T17:43:13.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.684+0000 7f8defe00640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8dd006a430 0x7f8dd006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:13.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.684+0000 7f8defe00640 1 -- 172.21.15.39:0/2923666540 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8de0089110 con 0x7f8df81f2740 2024-01-24T17:43:13.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.684+0000 7f8df7e00640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8dd006a430 0x7f8dd006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:13.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.685+0000 7f8df7e00640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8dd006a430 0x7f8dd006c8f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8de8034000 tx=0x7f8de802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:13.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.688+0000 7f8defe00640 1 -- 172.21.15.39:0/2923666540 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8de00546f0 con 0x7f8df81f2740 2024-01-24T17:43:13.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.884+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8df80b8150 con 0x7f8df81f2740 2024-01-24T17:43:13.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.885+0000 7f8defe00640 1 -- 172.21.15.39:0/2923666540 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8de0057d10 con 0x7f8df81f2740 2024-01-24T17:43:13.886 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:43:13.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8dd006a430 msgr2=0x7f8dd006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8dd006a430 0x7f8dd006c8f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8de8034000 tx=0x7f8de802f000 comp rx=0 tx=0).stop 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8df81f2740 msgr2=0x7f8df80b75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8df81f2740 0x7f8df80b75d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8de00116d0 tx=0x7f8de0011b90 comp rx=0 tx=0).stop 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 shutdown_connections 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8df81f2bc0 0x7f8df80b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8dd006a430 0x7f8dd006c8f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8df81f2740 0x7f8df80b75d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 --2- 172.21.15.39:0/2923666540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8dd80039a0 0x7f8df8152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.887+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 >> 172.21.15.39:0/2923666540 conn(0x7f8df8071dd0 msgr2=0x7f8df815b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.888+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 shutdown_connections 2024-01-24T17:43:13.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:13.888+0000 7f8dfd400640 1 -- 172.21.15.39:0/2923666540 wait complete. 2024-01-24T17:43:13.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:43:14.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.133+0000 7fae77200640 1 Processor -- start 2024-01-24T17:43:14.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.133+0000 7fae77200640 1 -- start start 2024-01-24T17:43:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.135+0000 7fae77200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae70070300 0x7fae700706e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.135+0000 7fae77200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7015f260 con 0x7fae7015ad60 2024-01-24T17:43:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.135+0000 7fae77200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae70070c20 con 0x7fae700caea0 2024-01-24T17:43:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.135+0000 7fae77200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7015e730 con 0x7fae70070300 2024-01-24T17:43:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.135+0000 7fae67e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fae7015ad60 0x7fae7015eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50012/0 (socket says 172.21.15.39:50012) 2024-01-24T17:43:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.135+0000 7fae67e00640 1 -- 172.21.15.39:0/3018653837 learned_addr learned my addr 172.21.15.39:0/3018653837 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6fe00640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae70070300 0x7fae700706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382354950 0 0) 0x7fae7015f260 con 0x7fae7015ad60 2024-01-24T17:43:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae50003520 con 0x7fae7015ad60 2024-01-24T17:43:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1610146911 0 0) 0x7fae50003520 con 0x7fae7015ad60 2024-01-24T17:43:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae70070300 msgr2=0x7fae700706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae70070300 0x7fae700706e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 >> v1:172.21.15.100:6789/0 conn(0x7fae700caea0 legacy=0x7fae700c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7015b5c0 con 0x7fae7015ad60 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fae58001670 con 0x7fae7015ad60 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 >> v1:172.21.15.39:6789/0 conn(0x7fae7015ad60 legacy=0x7fae7015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae500039a0 0x7fae50003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae70070300 0x7fae500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae50004d00 0x7fae50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7015e730 con 0x7fae500039a0 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5000bb10 con 0x7fae50004d00 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.136+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5000bcd0 con 0x7fae70070300 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.137+0000 7fae6fe00640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae500039a0 0x7fae50003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.137+0000 7fae67e00640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae70070300 0x7fae500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.137+0000 7fae6fe00640 1 -- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae70070300 msgr2=0x7fae500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.137+0000 7fae6fe00640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae70070300 0x7fae500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.137+0000 7fae6fe00640 1 -- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae50004d00 msgr2=0x7fae50005180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-24T17:43:14.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.137+0000 7fae6fe00640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae50004d00 0x7fae50005180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.137+0000 7fae6fe00640 1 -- 172.21.15.39:0/3018653837 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae50003520 con 0x7fae500039a0 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.138+0000 7fae6fe00640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae500039a0 0x7fae50003dc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fae60009ac0 tx=0x7fae500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f0e1497ceb79e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.139+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6002c6b0 con 0x7fae500039a0 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.139+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6002c810 con 0x7fae500039a0 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.139+0000 7fae6f400640 1 -- 172.21.15.39:0/3018653837 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6002cba0 con 0x7fae500039a0 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.139+0000 7fae77200640 1 -- 172.21.15.39:0/3018653837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae500039a0 msgr2=0x7fae50003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.139+0000 7fae77200640 1 --2- 172.21.15.39:0/3018653837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae500039a0 0x7fae50003dc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fae60009ac0 tx=0x7fae500057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.139+0000 7fae75200640 1 -- 172.21.15.39:0/3018653837 reap_dead start 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.139+0000 7fae77200640 1 -- 172.21.15.39:0/3018653837 shutdown_connections 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.139+0000 7fae77200640 1 -- 172.21.15.39:0/3018653837 >> 172.21.15.39:0/3018653837 conn(0x7fae7006a2a0 msgr2=0x7fae7006bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.140+0000 7fae77200640 1 -- 172.21.15.39:0/3018653837 shutdown_connections 2024-01-24T17:43:14.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.140+0000 7fae77200640 1 -- 172.21.15.39:0/3018653837 wait complete. 2024-01-24T17:43:14.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.141+0000 7fae77200640 1 Processor -- start 2024-01-24T17:43:14.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.141+0000 7fae77200640 1 -- start start 2024-01-24T17:43:14.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.142+0000 7fae77200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae500039a0 0x7fae701ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.142+0000 7fae77200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae700addd0 0x7fae701f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.142+0000 7fae77200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae701ed200 0x7fae701ed680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.142+0000 7fae77200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae701f16a0 con 0x7fae700addd0 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.142+0000 7fae77200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae701eec40 con 0x7fae701ed200 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.142+0000 7fae77200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae701edbc0 con 0x7fae500039a0 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae67e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae700addd0 0x7fae701f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae67e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae700addd0 0x7fae701f1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50044/0 (socket says 172.21.15.39:50044) 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae67e00640 1 -- 172.21.15.39:0/3678132486 learned_addr learned my addr 172.21.15.39:0/3678132486 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae6fe00640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae500039a0 0x7fae701ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae75200640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae701ed200 0x7fae701ed680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae67e00640 1 -- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae500039a0 msgr2=0x7fae701ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae67e00640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae500039a0 0x7fae701ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae67e00640 1 -- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae701ed200 msgr2=0x7fae701ed680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae67e00640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae701ed200 0x7fae701ed680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.143+0000 7fae67e00640 1 -- 172.21.15.39:0/3678132486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae700c4e70 con 0x7fae700addd0 2024-01-24T17:43:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.144+0000 7fae67e00640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae700addd0 0x7fae701f1ac0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fae5800e790 tx=0x7fae5800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.144+0000 7fae6cc00640 1 -- 172.21.15.39:0/3678132486 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5800eea0 con 0x7fae700addd0 2024-01-24T17:43:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.144+0000 7fae6cc00640 1 -- 172.21.15.39:0/3678132486 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5800c870 con 0x7fae700addd0 2024-01-24T17:43:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.144+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae701f1330 con 0x7fae700addd0 2024-01-24T17:43:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.144+0000 7fae6cc00640 1 -- 172.21.15.39:0/3678132486 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5800cc20 con 0x7fae700addd0 2024-01-24T17:43:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.144+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae700c4570 con 0x7fae700addd0 2024-01-24T17:43:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.145+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae701f17e0 con 0x7fae700addd0 2024-01-24T17:43:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.146+0000 7fae6cc00640 1 -- 172.21.15.39:0/3678132486 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fae5801d460 con 0x7fae700addd0 2024-01-24T17:43:14.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.147+0000 7fae6cc00640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae4806a430 0x7fae4806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.147+0000 7fae6cc00640 1 -- 172.21.15.39:0/3678132486 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae58088770 con 0x7fae700addd0 2024-01-24T17:43:14.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.148+0000 7fae6fe00640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae4806a430 0x7fae4806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.149+0000 7fae6fe00640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae4806a430 0x7fae4806c8f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fae60036000 tx=0x7fae60030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:14.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.151+0000 7fae6cc00640 1 -- 172.21.15.39:0/3678132486 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fae701f17e0 con 0x7fae700addd0 2024-01-24T17:43:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.360+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fae700c4e70 con 0x7fae700addd0 2024-01-24T17:43:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.361+0000 7fae6cc00640 1 -- 172.21.15.39:0/3678132486 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7fae700c4e70 con 0x7fae700addd0 2024-01-24T17:43:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:43:14.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.363+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae4806a430 msgr2=0x7fae4806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.363+0000 7fae77200640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae4806a430 0x7fae4806c8f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fae60036000 tx=0x7fae60030000 comp rx=0 tx=0).stop 2024-01-24T17:43:14.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.363+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae700addd0 msgr2=0x7fae701f1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.363+0000 7fae77200640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae700addd0 0x7fae701f1ac0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fae5800e790 tx=0x7fae5800ec50 comp rx=0 tx=0).stop 2024-01-24T17:43:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.363+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 shutdown_connections 2024-01-24T17:43:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.364+0000 7fae77200640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae701ed200 0x7fae701ed680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.364+0000 7fae77200640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae4806a430 0x7fae4806c8f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.364+0000 7fae77200640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae700addd0 0x7fae701f1ac0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.364+0000 7fae77200640 1 --2- 172.21.15.39:0/3678132486 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae500039a0 0x7fae701ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.364+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 >> 172.21.15.39:0/3678132486 conn(0x7fae7006a2a0 msgr2=0x7fae7015d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.364+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 shutdown_connections 2024-01-24T17:43:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.364+0000 7fae77200640 1 -- 172.21.15.39:0/3678132486 wait complete. 2024-01-24T17:43:14.382 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:43:14.382 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:14.433 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.697+0000 7f24fde00640 1 Processor -- start 2024-01-24T17:43:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.698+0000 7f24fde00640 1 -- start start 2024-01-24T17:43:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.699+0000 7f24fde00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f24f815cdf0 0x7f24f80cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.699+0000 7f24fde00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f80cef60 0x7f24f80d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.699+0000 7f24fde00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f80cf5a0 con 0x7f24f80d0d30 2024-01-24T17:43:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.699+0000 7f24fde00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f80cf3c0 con 0x7f24f815cdf0 2024-01-24T17:43:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.699+0000 7f24fde00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80d11b0 con 0x7f24f80cef60 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.699+0000 7f24f7400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f24f80d0d30 0x7f24f80cd110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50026/0 (socket says 172.21.15.39:50026) 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.699+0000 7f24f7400640 1 -- 172.21.15.39:0/1467448106 learned_addr learned my addr 172.21.15.39:0/1467448106 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.699+0000 7f24f6a00640 1 --2- 172.21.15.39:0/1467448106 >> v2:172.21.15.100:3300/0 conn(0x7f24f815cdf0 0x7f24f80cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f6000640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f80cef60 0x7f24f80d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171989423 0 0) 0x7f24f80cf5a0 con 0x7f24f80d0d30 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d8003520 con 0x7f24f80d0d30 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4276894402 0 0) 0x7f24d8003520 con 0x7f24f80d0d30 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f80cef60 msgr2=0x7f24f80d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f80cef60 0x7f24f80d07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 >> v2:172.21.15.100:3300/0 conn(0x7f24f815cdf0 msgr2=0x7f24f80cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 --2- 172.21.15.39:0/1467448106 >> v2:172.21.15.100:3300/0 conn(0x7f24f815cdf0 0x7f24f80cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f80cf880 con 0x7f24f80d0d30 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f24f0001a40 con 0x7f24f80d0d30 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 >> v1:172.21.15.39:6789/0 conn(0x7f24f80d0d30 legacy=0x7f24f80cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f80cef60 0x7f24d8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d8003ef0 0x7f24d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d8004d00 0x7f24d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80cf3c0 con 0x7f24d8004d00 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80d11b0 con 0x7f24d8003ef0 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.700+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d800b980 con 0x7f24f80cef60 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f7400640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d8004d00 0x7f24d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f6a00640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f80cef60 0x7f24d8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f6000640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d8003ef0 0x7f24d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f7400640 1 -- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f80cef60 msgr2=0x7f24d8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:14.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f7400640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f80cef60 0x7f24d8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f7400640 1 -- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d8003ef0 msgr2=0x7f24d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f7400640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d8003ef0 0x7f24d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f7400640 1 -- 172.21.15.39:0/1467448106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d8003520 con 0x7f24d8004d00 2024-01-24T17:43:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.701+0000 7f24f6000640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d8003ef0 0x7f24d80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.702+0000 7f24f7400640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d8004d00 0x7f24d8005180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f24f000a7d0 tx=0x7f24d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da26c181643ca431 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0034630 con 0x7f24d8004d00 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f002f070 con 0x7f24d8004d00 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24f5600640 1 -- 172.21.15.39:0/1467448106 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0034a30 con 0x7f24d8004d00 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24fde00640 1 -- 172.21.15.39:0/1467448106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d8004d00 msgr2=0x7f24d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24fde00640 1 --2- 172.21.15.39:0/1467448106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24d8004d00 0x7f24d8005180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f24f000a7d0 tx=0x7f24d80057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24f7400640 1 -- 172.21.15.39:0/1467448106 reap_dead start 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24fde00640 1 -- 172.21.15.39:0/1467448106 shutdown_connections 2024-01-24T17:43:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24fde00640 1 -- 172.21.15.39:0/1467448106 >> 172.21.15.39:0/1467448106 conn(0x7f24f8099b60 msgr2=0x7f24f8089fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:14.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24fde00640 1 -- 172.21.15.39:0/1467448106 shutdown_connections 2024-01-24T17:43:14.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.703+0000 7f24fde00640 1 -- 172.21.15.39:0/1467448106 wait complete. 2024-01-24T17:43:14.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.705+0000 7f24fde00640 1 Processor -- start 2024-01-24T17:43:14.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.705+0000 7f24fde00640 1 -- start start 2024-01-24T17:43:14.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.706+0000 7f24fde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f81516c0 0x7f24f8158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.706+0000 7f24fde00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24f8158a80 0x7f24f8152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.706+0000 7f24fde00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f81532b0 0x7f24f81e5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.706+0000 7f24fde00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f8158f00 con 0x7f24f81516c0 2024-01-24T17:43:14.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.706+0000 7f24fde00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f8154e60 con 0x7f24f8158a80 2024-01-24T17:43:14.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.706+0000 7f24fde00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f8155140 con 0x7f24f81532b0 2024-01-24T17:43:14.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f81516c0 0x7f24f8158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f81516c0 0x7f24f8158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50060/0 (socket says 172.21.15.39:50060) 2024-01-24T17:43:14.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 -- 172.21.15.39:0/2540507413 learned_addr learned my addr 172.21.15.39:0/2540507413 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:14.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6000640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24f8158a80 0x7f24f8152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f7400640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f81532b0 0x7f24f81e5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 -- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f81532b0 msgr2=0x7f24f81e5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f81532b0 0x7f24f81e5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 -- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24f8158a80 msgr2=0x7f24f8152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24f8158a80 0x7f24f8152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 -- 172.21.15.39:0/2540507413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8155420 con 0x7f24f81516c0 2024-01-24T17:43:14.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6000640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24f8158a80 0x7f24f8152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:14.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f7400640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f81532b0 0x7f24f81e5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:14.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.707+0000 7f24f6a00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f81516c0 0x7f24f8158540 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f24e80116d0 tx=0x7f24e8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:14.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.708+0000 7f24ef400640 1 -- 172.21.15.39:0/2540507413 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e800f9c0 con 0x7f24f81516c0 2024-01-24T17:43:14.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.708+0000 7f24ef400640 1 -- 172.21.15.39:0/2540507413 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e800fb20 con 0x7f24f81516c0 2024-01-24T17:43:14.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.708+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f8155700 con 0x7f24f81516c0 2024-01-24T17:43:14.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.708+0000 7f24ef400640 1 -- 172.21.15.39:0/2540507413 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e800fe90 con 0x7f24f81516c0 2024-01-24T17:43:14.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.708+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f8151aa0 con 0x7f24f81516c0 2024-01-24T17:43:14.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.709+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f8073ef0 con 0x7f24f81516c0 2024-01-24T17:43:14.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.710+0000 7f24ef400640 1 -- 172.21.15.39:0/2540507413 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f24e8015460 con 0x7f24f81516c0 2024-01-24T17:43:14.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.711+0000 7f24ef400640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24d006a430 0x7f24d006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:14.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.711+0000 7f24ef400640 1 -- 172.21.15.39:0/2540507413 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24e8088e80 con 0x7f24f81516c0 2024-01-24T17:43:14.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.712+0000 7f24f6000640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24d006a430 0x7f24d006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:14.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.713+0000 7f24f6000640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24d006a430 0x7f24d006c8f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24f8158300 tx=0x7f24e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:14.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.716+0000 7f24ef400640 1 -- 172.21.15.39:0/2540507413 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f24e8054460 con 0x7f24f81516c0 2024-01-24T17:43:14.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.917+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24f80cdac0 con 0x7f24f81516c0 2024-01-24T17:43:14.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.921+0000 7f24ef400640 1 -- 172.21.15.39:0/2540507413 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f24e8057a80 con 0x7f24f81516c0 2024-01-24T17:43:14.921 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:14.921 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:42:57.705789+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24d006a430 msgr2=0x7f24d006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24d006a430 0x7f24d006c8f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24f8158300 tx=0x7f24e0008040 comp rx=0 tx=0).stop 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f81516c0 msgr2=0x7f24f8158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f81516c0 0x7f24f8158540 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f24e80116d0 tx=0x7f24e8011b90 comp rx=0 tx=0).stop 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 shutdown_connections 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24f81532b0 0x7f24f81e5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24f8158a80 0x7f24f8152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24d006a430 0x7f24d006c8f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 --2- 172.21.15.39:0/2540507413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f81516c0 0x7f24f8158540 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:14.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 >> 172.21.15.39:0/2540507413 conn(0x7f24f8099b60 msgr2=0x7f24f806f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:14.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 shutdown_connections 2024-01-24T17:43:14.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:14.924+0000 7f24fde00640 1 -- 172.21.15.39:0/2540507413 wait complete. 2024-01-24T17:43:14.944 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:42:54.883732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:42:55.727240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:43:14.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:43:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.175+0000 7f5532c00640 1 Processor -- start 2024-01-24T17:43:15.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.175+0000 7f5532c00640 1 -- start start 2024-01-24T17:43:15.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.176+0000 7f5532c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f552c15c8d0 0x7f552c0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.176+0000 7f5532c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 0x7f552c0cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.176+0000 7f5532c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c0b38a0 con 0x7f552c0b2710 2024-01-24T17:43:15.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5532c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f552c0d13e0 con 0x7f552c15c8d0 2024-01-24T17:43:15.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5532c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552c0d0ab0 con 0x7f552c0d2e80 2024-01-24T17:43:15.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5531800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f552c15c8d0 0x7f552c0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5531800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f552c15c8d0 0x7f552c0d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50066/0 (socket says 172.21.15.39:50066) 2024-01-24T17:43:15.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5531800640 1 -- 172.21.15.39:0/1256748885 learned_addr learned my addr 172.21.15.39:0/1256748885 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:15.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5530e00640 1 --1- 172.21.15.39:0/1256748885 >> v1:172.21.15.39:6789/0 conn(0x7f552c0b2710 0x7f552c0d2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50042/0 (socket says 172.21.15.39:50042) 2024-01-24T17:43:15.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5532200640 1 --2- 172.21.15.39:0/1256748885 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 0x7f552c0cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5531800640 1 -- 172.21.15.39:0/1256748885 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 msgr2=0x7f552c0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:15.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5531800640 1 --2- 172.21.15.39:0/1256748885 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 0x7f552c0cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.177+0000 7f5531800640 1 -- 172.21.15.39:0/1256748885 >> v1:172.21.15.39:6789/0 conn(0x7f552c0b2710 legacy=0x7f552c0d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.178+0000 7f5531800640 1 -- 172.21.15.39:0/1256748885 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c0d0d90 con 0x7f552c15c8d0 2024-01-24T17:43:15.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.178+0000 7f552be00640 1 -- 172.21.15.39:0/1256748885 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445626229 0 0) 0x7f552c0b38a0 con 0x7f552c0b2710 2024-01-24T17:43:15.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.179+0000 7f5531800640 1 --2- 172.21.15.39:0/1256748885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c15c8d0 0x7f552c0d03b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f551c009870 tx=0x7f551c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c855611999617bda server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:15.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.179+0000 7f552be00640 1 -- 172.21.15.39:0/1256748885 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c030840 con 0x7f552c15c8d0 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.179+0000 7f552be00640 1 -- 172.21.15.39:0/1256748885 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c0309a0 con 0x7f552c15c8d0 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.179+0000 7f552be00640 1 -- 172.21.15.39:0/1256748885 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c030d30 con 0x7f552c15c8d0 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.180+0000 7f5532c00640 1 -- 172.21.15.39:0/1256748885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c15c8d0 msgr2=0x7f552c0d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.180+0000 7f5532c00640 1 --2- 172.21.15.39:0/1256748885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c15c8d0 0x7f552c0d03b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f551c009870 tx=0x7f551c02b980 comp rx=0 tx=0).stop 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.180+0000 7f5532c00640 1 -- 172.21.15.39:0/1256748885 shutdown_connections 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.180+0000 7f5532c00640 1 --2- 172.21.15.39:0/1256748885 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 0x7f552c0cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.180+0000 7f5532c00640 1 --2- 172.21.15.39:0/1256748885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c15c8d0 0x7f552c0d03b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.180+0000 7f5532c00640 1 -- 172.21.15.39:0/1256748885 >> 172.21.15.39:0/1256748885 conn(0x7f552c071dd0 msgr2=0x7f552c09bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.180+0000 7f5532c00640 1 -- 172.21.15.39:0/1256748885 shutdown_connections 2024-01-24T17:43:15.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.180+0000 7f5532c00640 1 -- 172.21.15.39:0/1256748885 wait complete. 2024-01-24T17:43:15.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.182+0000 7f5532c00640 1 Processor -- start 2024-01-24T17:43:15.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.182+0000 7f5532c00640 1 -- start start 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5532c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c0b2710 0x7f552c09e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5532c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 0x7f552c0ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5532c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f552c15c8d0 0x7f552c09c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5532c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c0cbd80 con 0x7f552c0b2710 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5532c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c09d030 con 0x7f552c15c8d0 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5532c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552c09d310 con 0x7f552c0d2e80 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5531800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c0b2710 0x7f552c09e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5531800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c0b2710 0x7f552c09e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50068/0 (socket says 172.21.15.39:50068) 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5531800640 1 -- 172.21.15.39:0/3511225601 learned_addr learned my addr 172.21.15.39:0/3511225601 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5532200640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f552c15c8d0 0x7f552c09c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5531800640 1 -- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 msgr2=0x7f552c0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5531800640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 0x7f552c0ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5531800640 1 -- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f552c15c8d0 msgr2=0x7f552c09c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5531800640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f552c15c8d0 0x7f552c09c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.183+0000 7f5531800640 1 -- 172.21.15.39:0/3511225601 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c0094d0 con 0x7f552c0b2710 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.184+0000 7f5531800640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c0b2710 0x7f552c09e0d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f551c009870 tx=0x7f551c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.184+0000 7f5529600640 1 -- 172.21.15.39:0/3511225601 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c03e070 con 0x7f552c0b2710 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.184+0000 7f5529600640 1 -- 172.21.15.39:0/3511225601 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c004030 con 0x7f552c0b2710 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.184+0000 7f5529600640 1 -- 172.21.15.39:0/3511225601 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c0043c0 con 0x7f552c0b2710 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.184+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c09d5f0 con 0x7f552c0b2710 2024-01-24T17:43:15.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.184+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552c09dbb0 con 0x7f552c0b2710 2024-01-24T17:43:15.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.185+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552c04dff0 con 0x7f552c0b2710 2024-01-24T17:43:15.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.186+0000 7f5529600640 1 -- 172.21.15.39:0/3511225601 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f551c042070 con 0x7f552c0b2710 2024-01-24T17:43:15.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.188+0000 7f5529600640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f550406a430 0x7f550406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.188+0000 7f5529600640 1 -- 172.21.15.39:0/3511225601 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f551c0aa810 con 0x7f552c0b2710 2024-01-24T17:43:15.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.188+0000 7f5530e00640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f550406a430 0x7f550406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.189+0000 7f5530e00640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f550406a430 0x7f550406c8f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5514007fd0 tx=0x7f5514007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:15.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.192+0000 7f5529600640 1 -- 172.21.15.39:0/3511225601 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f551c075df0 con 0x7f552c0b2710 2024-01-24T17:43:15.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.394+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f552c0b86a0 con 0x7f552c0b2710 2024-01-24T17:43:15.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.395+0000 7f5529600640 1 -- 172.21.15.39:0/3511225601 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7f551c034020 con 0x7f552c0b2710 2024-01-24T17:43:15.395 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:43:15.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f550406a430 msgr2=0x7f550406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f550406a430 0x7f550406c8f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5514007fd0 tx=0x7f5514007920 comp rx=0 tx=0).stop 2024-01-24T17:43:15.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c0b2710 msgr2=0x7f552c09e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c0b2710 0x7f552c09e0d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f551c009870 tx=0x7f551c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 shutdown_connections 2024-01-24T17:43:15.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f552c15c8d0 0x7f552c09c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f552c0d2e80 0x7f552c0ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f550406a430 0x7f550406c8f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 --2- 172.21.15.39:0/3511225601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f552c0b2710 0x7f552c09e0d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 >> 172.21.15.39:0/3511225601 conn(0x7f552c071dd0 msgr2=0x7f552c0cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:15.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 shutdown_connections 2024-01-24T17:43:15.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.397+0000 7f5532c00640 1 -- 172.21.15.39:0/3511225601 wait complete. 2024-01-24T17:43:15.417 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:43:15.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.645+0000 7fe318e00640 1 Processor -- start 2024-01-24T17:43:15.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.646+0000 7fe318e00640 1 -- start start 2024-01-24T17:43:15.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.647+0000 7fe318e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe31415c9e0 0x7fe3140b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.647+0000 7fe318e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3140b7e30 con 0x7fe3140b8250 2024-01-24T17:43:15.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.647+0000 7fe318e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3140b6d90 con 0x7fe3140cdd60 2024-01-24T17:43:15.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.647+0000 7fe318e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140b7070 con 0x7fe31415c9e0 2024-01-24T17:43:15.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.647+0000 7fe312a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe3140b8250 0x7fe3140cd650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50052/0 (socket says 172.21.15.39:50052) 2024-01-24T17:43:15.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.647+0000 7fe312a00640 1 -- 172.21.15.39:0/2907842758 learned_addr learned my addr 172.21.15.39:0/2907842758 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:15.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe313400640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe31415c9e0 0x7fe3140b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921942389 0 0) 0x7fe2fc001110 con 0x7fe3140b8250 2024-01-24T17:43:15.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f4003520 con 0x7fe3140b8250 2024-01-24T17:43:15.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145942872 0 0) 0x7fe2f4003520 con 0x7fe3140b8250 2024-01-24T17:43:15.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe31415c9e0 msgr2=0x7fe3140b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe31415c9e0 0x7fe3140b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 >> v1:172.21.15.100:6789/0 conn(0x7fe3140cdd60 legacy=0x7fe3140d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140b7350 con 0x7fe3140b8250 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fe2fc001800 con 0x7fe3140b8250 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.648+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 >> v1:172.21.15.39:6789/0 conn(0x7fe3140b8250 legacy=0x7fe3140cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe312000640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f40039a0 0x7fe2f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe312000640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40047c0 0x7fe2f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe312000640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe31415c9e0 0x7fe2f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3140b7070 con 0x7fe2f40039a0 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f400bac0 con 0x7fe2f40047c0 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f400bc80 con 0x7fe31415c9e0 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe313400640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f40039a0 0x7fe2f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe312a00640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40047c0 0x7fe2f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe313400640 1 -- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe31415c9e0 msgr2=0x7fe2f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:15.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe313e00640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe31415c9e0 0x7fe2f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe313400640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe31415c9e0 0x7fe2f4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe313400640 1 -- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40047c0 msgr2=0x7fe2f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:15.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe313400640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40047c0 0x7fe2f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe313400640 1 -- 172.21.15.39:0/2907842758 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f4003520 con 0x7fe2f40039a0 2024-01-24T17:43:15.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.649+0000 7fe312a00640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40047c0 0x7fe2f4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:15.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.651+0000 7fe313400640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f40039a0 0x7fe2f4003dc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe304009a90 tx=0x7fe2f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a1ef521646c980 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:15.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.651+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30402c680 con 0x7fe2f40039a0 2024-01-24T17:43:15.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.651+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30402c7e0 con 0x7fe2f40039a0 2024-01-24T17:43:15.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.651+0000 7fe312000640 1 -- 172.21.15.39:0/2907842758 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30402cb70 con 0x7fe2f40039a0 2024-01-24T17:43:15.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.651+0000 7fe318e00640 1 -- 172.21.15.39:0/2907842758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f40039a0 msgr2=0x7fe2f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.651+0000 7fe318e00640 1 --2- 172.21.15.39:0/2907842758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2f40039a0 0x7fe2f4003dc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe304009a90 tx=0x7fe2f40057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.652+0000 7fe313e00640 1 -- 172.21.15.39:0/2907842758 reap_dead start 2024-01-24T17:43:15.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.652+0000 7fe318e00640 1 -- 172.21.15.39:0/2907842758 shutdown_connections 2024-01-24T17:43:15.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.652+0000 7fe318e00640 1 -- 172.21.15.39:0/2907842758 >> 172.21.15.39:0/2907842758 conn(0x7fe314071790 msgr2=0x7fe314071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:15.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.652+0000 7fe318e00640 1 -- 172.21.15.39:0/2907842758 shutdown_connections 2024-01-24T17:43:15.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.652+0000 7fe318e00640 1 -- 172.21.15.39:0/2907842758 wait complete. 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.654+0000 7fe318e00640 1 Processor -- start 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.654+0000 7fe318e00640 1 -- start start 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe318e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40039a0 0x7fe314156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe318e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3140c9a50 0x7fe314151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe318e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3140cca40 0x7fe31415e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe318e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe314153ab0 con 0x7fe3140c9a50 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe318e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3141567c0 con 0x7fe2f40039a0 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe318e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe314156a80 con 0x7fe3140cca40 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe312a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3140c9a50 0x7fe314151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe312a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3140c9a50 0x7fe314151120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50096/0 (socket says 172.21.15.39:50096) 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe312a00640 1 -- 172.21.15.39:0/2160887077 learned_addr learned my addr 172.21.15.39:0/2160887077 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe313e00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3140cca40 0x7fe31415e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe313400640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40039a0 0x7fe314156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe312a00640 1 -- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3140cca40 msgr2=0x7fe31415e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe312a00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3140cca40 0x7fe31415e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe312a00640 1 -- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40039a0 msgr2=0x7fe314156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe312a00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40039a0 0x7fe314156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe312a00640 1 -- 172.21.15.39:0/2160887077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140c76f0 con 0x7fe3140c9a50 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.655+0000 7fe313400640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40039a0 0x7fe314156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.656+0000 7fe312a00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3140c9a50 0x7fe314151120 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00d8c0 tx=0x7fe2fc00dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.656+0000 7fe30b400640 1 -- 172.21.15.39:0/2160887077 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc0074e0 con 0x7fe3140c9a50 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.656+0000 7fe30b400640 1 -- 172.21.15.39:0/2160887077 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc007640 con 0x7fe3140c9a50 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.656+0000 7fe30b400640 1 -- 172.21.15.39:0/2160887077 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc0049e0 con 0x7fe3140c9a50 2024-01-24T17:43:15.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.656+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3140c79d0 con 0x7fe3140c9a50 2024-01-24T17:43:15.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.656+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3140c7f90 con 0x7fe3140c9a50 2024-01-24T17:43:15.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.658+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe314067040 con 0x7fe3140c9a50 2024-01-24T17:43:15.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.658+0000 7fe30b400640 1 -- 172.21.15.39:0/2160887077 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe2fc004b40 con 0x7fe3140c9a50 2024-01-24T17:43:15.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.659+0000 7fe30b400640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe2ec06a430 0x7fe2ec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:15.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.659+0000 7fe30b400640 1 -- 172.21.15.39:0/2160887077 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe2fc088130 con 0x7fe3140c9a50 2024-01-24T17:43:15.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.660+0000 7fe313400640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe2ec06a430 0x7fe2ec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:15.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.661+0000 7fe313400640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe2ec06a430 0x7fe2ec06c8f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe304036000 tx=0x7fe304030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:15.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.664+0000 7fe30b400640 1 -- 172.21.15.39:0/2160887077 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe2fc053750 con 0x7fe3140c9a50 2024-01-24T17:43:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.857+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe3140b7e30 con 0x7fe3140c9a50 2024-01-24T17:43:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.858+0000 7fe30b400640 1 -- 172.21.15.39:0/2160887077 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v35) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe3140b7e30 con 0x7fe3140c9a50 2024-01-24T17:43:15.860 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe2ec06a430 msgr2=0x7fe2ec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe2ec06a430 0x7fe2ec06c8f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe304036000 tx=0x7fe304030000 comp rx=0 tx=0).stop 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3140c9a50 msgr2=0x7fe314151120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3140c9a50 0x7fe314151120 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00d8c0 tx=0x7fe2fc00dd80 comp rx=0 tx=0).stop 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 shutdown_connections 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3140cca40 0x7fe31415e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe2ec06a430 0x7fe2ec06c8f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3140c9a50 0x7fe314151120 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 --2- 172.21.15.39:0/2160887077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe2f40039a0 0x7fe314156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 >> 172.21.15.39:0/2160887077 conn(0x7fe314071790 msgr2=0x7fe3140d2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 shutdown_connections 2024-01-24T17:43:15.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:15.861+0000 7fe318e00640 1 -- 172.21.15.39:0/2160887077 wait complete. 2024-01-24T17:43:15.882 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:43:16.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.106+0000 7ff197e00640 1 Processor -- start 2024-01-24T17:43:16.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.107+0000 7ff197e00640 1 -- start start 2024-01-24T17:43:16.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.108+0000 7ff197e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff1980accd0 0x7ff19806d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.108+0000 7ff197e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1980ad0b0 con 0x7ff1980cb540 2024-01-24T17:43:16.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.108+0000 7ff197e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1980b2370 con 0x7ff19806c3a0 2024-01-24T17:43:16.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.108+0000 7ff197e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19806d840 con 0x7ff1980accd0 2024-01-24T17:43:16.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.109+0000 7ff197400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff1980cb540 0x7ff1980ab030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50064/0 (socket says 172.21.15.39:50064) 2024-01-24T17:43:16.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.109+0000 7ff197400640 1 -- 172.21.15.39:0/4184000785 learned_addr learned my addr 172.21.15.39:0/4184000785 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:16.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.109+0000 7ff196a00640 1 --2- 172.21.15.39:0/4184000785 >> v2:172.21.15.100:3300/0 conn(0x7ff1980accd0 0x7ff19806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.109+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011217612 0 0) 0x7ff1980ad0b0 con 0x7ff1980cb540 2024-01-24T17:43:16.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.109+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff178003520 con 0x7ff1980cb540 2024-01-24T17:43:16.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782815846 0 0) 0x7ff178003520 con 0x7ff1980cb540 2024-01-24T17:43:16.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 >> v2:172.21.15.100:3300/0 conn(0x7ff1980accd0 msgr2=0x7ff19806d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:16.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 --2- 172.21.15.39:0/4184000785 >> v2:172.21.15.100:3300/0 conn(0x7ff1980accd0 0x7ff19806d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 >> v1:172.21.15.100:6789/0 conn(0x7ff19806c3a0 legacy=0x7ff1980cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1980b2650 con 0x7ff1980cb540 2024-01-24T17:43:16.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff1900019f0 con 0x7ff1980cb540 2024-01-24T17:43:16.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 >> v1:172.21.15.39:6789/0 conn(0x7ff1980cb540 legacy=0x7ff1980ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff178003970 0x7ff178003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff178004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff178005180 0x7ff17800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19806d840 con 0x7ff1780047c0 2024-01-24T17:43:16.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17800c020 con 0x7ff178005180 2024-01-24T17:43:16.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17800c1e0 con 0x7ff178003970 2024-01-24T17:43:16.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff196000640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff178004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.110+0000 7ff197400640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff178005180 0x7ff17800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.111+0000 7ff196a00640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff178003970 0x7ff178003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.111+0000 7ff196000640 1 -- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff178003970 msgr2=0x7ff178003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:16.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.111+0000 7ff196000640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff178003970 0x7ff178003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.111+0000 7ff196000640 1 -- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff178005180 msgr2=0x7ff17800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.111+0000 7ff196000640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff178005180 0x7ff17800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.111+0000 7ff196000640 1 -- 172.21.15.39:0/4184000785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff178003520 con 0x7ff1780047c0 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.111+0000 7ff197400640 1 -- 172.21.15.39:0/4184000785 reap_dead start 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.111+0000 7ff196a00640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff178003970 0x7ff178003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.112+0000 7ff196000640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff178004c40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff180001750 tx=0x7ff1780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2851e8561191d784 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.112+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1800052d0 con 0x7ff1780047c0 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.112+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18002e050 con 0x7ff1780047c0 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.112+0000 7ff195600640 1 -- 172.21.15.39:0/4184000785 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180003830 con 0x7ff1780047c0 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.113+0000 7ff197e00640 1 -- 172.21.15.39:0/4184000785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 msgr2=0x7ff178004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.113+0000 7ff197e00640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff178004c40 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff180001750 tx=0x7ff1780057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.113+0000 7ff197e00640 1 -- 172.21.15.39:0/4184000785 shutdown_connections 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.113+0000 7ff197e00640 1 --2- 172.21.15.39:0/4184000785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff178004c40 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.113+0000 7ff197e00640 1 -- 172.21.15.39:0/4184000785 >> 172.21.15.39:0/4184000785 conn(0x7ff198069e50 msgr2=0x7ff19806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.113+0000 7ff197e00640 1 -- 172.21.15.39:0/4184000785 shutdown_connections 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.113+0000 7ff197e00640 1 -- 172.21.15.39:0/4184000785 wait complete. 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.115+0000 7ff197e00640 1 Processor -- start 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.115+0000 7ff197e00640 1 -- start start 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff197e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff1980ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff197e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1981f19d0 0x7ff1980b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff197e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1981582f0 0x7ff198158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff197e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19815c670 con 0x7ff1780047c0 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff197e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff198154be0 con 0x7ff1981f19d0 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff197e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff198158cb0 con 0x7ff1981582f0 2024-01-24T17:43:16.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff196a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff1980ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff196a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff1980ae050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50126/0 (socket says 172.21.15.39:50126) 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.116+0000 7ff196a00640 1 -- 172.21.15.39:0/2463020034 learned_addr learned my addr 172.21.15.39:0/2463020034 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff196000640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1981f19d0 0x7ff1980b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff197400640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1981582f0 0x7ff198158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff196a00640 1 -- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1981582f0 msgr2=0x7ff198158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff196a00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1981582f0 0x7ff198158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff196a00640 1 -- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1981f19d0 msgr2=0x7ff1980b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff196a00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1981f19d0 0x7ff1980b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff196a00640 1 -- 172.21.15.39:0/2463020034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1980c8820 con 0x7ff1780047c0 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff196000640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1981f19d0 0x7ff1980b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff197400640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1981582f0 0x7ff198158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff196a00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff1980ae050 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff1880107c0 tx=0x7ff188010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff18ea00640 1 -- 172.21.15.39:0/2463020034 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188004220 con 0x7ff1780047c0 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.117+0000 7ff18ea00640 1 -- 172.21.15.39:0/2463020034 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff188004380 con 0x7ff1780047c0 2024-01-24T17:43:16.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.118+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1981f1e30 con 0x7ff1780047c0 2024-01-24T17:43:16.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.118+0000 7ff18ea00640 1 -- 172.21.15.39:0/2463020034 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18800e970 con 0x7ff1780047c0 2024-01-24T17:43:16.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.118+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19815bfb0 con 0x7ff1780047c0 2024-01-24T17:43:16.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.119+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19815c8e0 con 0x7ff1780047c0 2024-01-24T17:43:16.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.120+0000 7ff18ea00640 1 -- 172.21.15.39:0/2463020034 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff18800ead0 con 0x7ff1780047c0 2024-01-24T17:43:16.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.121+0000 7ff18ea00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff17006a460 0x7ff17006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.121+0000 7ff18ea00640 1 -- 172.21.15.39:0/2463020034 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff188088a90 con 0x7ff1780047c0 2024-01-24T17:43:16.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.121+0000 7ff196000640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff17006a460 0x7ff17006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.122+0000 7ff196000640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff17006a460 0x7ff17006c920 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff180033000 tx=0x7ff180009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:16.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.125+0000 7ff18ea00640 1 -- 172.21.15.39:0/2463020034 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff188053ff0 con 0x7ff1780047c0 2024-01-24T17:43:16.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.330+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff19804dff0 con 0x7ff1780047c0 2024-01-24T17:43:16.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.331+0000 7ff18ea00640 1 -- 172.21.15.39:0/2463020034 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff188021020 con 0x7ff1780047c0 2024-01-24T17:43:16.332 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:43:16.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.334+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff17006a460 msgr2=0x7ff17006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.334+0000 7ff197e00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff17006a460 0x7ff17006c920 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff180033000 tx=0x7ff180009030 comp rx=0 tx=0).stop 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 msgr2=0x7ff1980ae050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff1980ae050 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff1880107c0 tx=0x7ff188010c80 comp rx=0 tx=0).stop 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 shutdown_connections 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1981582f0 0x7ff198158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1981f19d0 0x7ff1980b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff17006a460 0x7ff17006c920 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 --2- 172.21.15.39:0/2463020034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1780047c0 0x7ff1980ae050 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 >> 172.21.15.39:0/2463020034 conn(0x7ff198069e50 msgr2=0x7ff19815a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 shutdown_connections 2024-01-24T17:43:16.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.335+0000 7ff197e00640 1 -- 172.21.15.39:0/2463020034 wait complete. 2024-01-24T17:43:16.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:43:16.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.572+0000 7f3aad400640 1 Processor -- start 2024-01-24T17:43:16.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.573+0000 7f3aad400640 1 -- start start 2024-01-24T17:43:16.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.574+0000 7f3aad400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3aa80cf990 0x7f3aa80cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.574+0000 7f3aad400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3aa815d020 0x7f3aa815d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.574+0000 7f3aad400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa80d02b0 con 0x7f3aa8160610 2024-01-24T17:43:16.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.574+0000 7f3aad400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa815e080 con 0x7f3aa815d020 2024-01-24T17:43:16.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.574+0000 7f3aad400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa815dac0 con 0x7f3aa80cf990 2024-01-24T17:43:16.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.575+0000 7f3aa5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3aa815d020 0x7f3aa815d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.575+0000 7f3aa5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3aa815d020 0x7f3aa815d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50130/0 (socket says 172.21.15.39:50130) 2024-01-24T17:43:16.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.575+0000 7f3aa5e00640 1 -- 172.21.15.39:0/2663707119 learned_addr learned my addr 172.21.15.39:0/2663707119 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:16.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.575+0000 7f3aa6800640 1 --2- 172.21.15.39:0/2663707119 >> v2:172.21.15.100:3300/0 conn(0x7f3aa80cf990 0x7f3aa80cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.575+0000 7f3aa5e00640 1 -- 172.21.15.39:0/2663707119 >> v2:172.21.15.100:3300/0 conn(0x7f3aa80cf990 msgr2=0x7f3aa80cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:16.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.575+0000 7f3aa5e00640 1 --2- 172.21.15.39:0/2663707119 >> v2:172.21.15.100:3300/0 conn(0x7f3aa80cf990 0x7f3aa80cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.575+0000 7f3aa5e00640 1 -- 172.21.15.39:0/2663707119 >> v1:172.21.15.100:6789/0 conn(0x7f3aa8160610 legacy=0x7f3aa8160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.575+0000 7f3aa5e00640 1 -- 172.21.15.39:0/2663707119 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa815dda0 con 0x7f3aa815d020 2024-01-24T17:43:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.576+0000 7f3aa5e00640 1 --2- 172.21.15.39:0/2663707119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa815d020 0x7f3aa815d490 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80b4c00 tx=0x7f3a9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419bf9adc7db804 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.577+0000 7f3aa5400640 1 -- 172.21.15.39:0/2663707119 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90031870 con 0x7f3aa815d020 2024-01-24T17:43:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.577+0000 7f3aa5400640 1 -- 172.21.15.39:0/2663707119 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a900319d0 con 0x7f3aa815d020 2024-01-24T17:43:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.577+0000 7f3aa5400640 1 -- 172.21.15.39:0/2663707119 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90031d60 con 0x7f3aa815d020 2024-01-24T17:43:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.577+0000 7f3aad400640 1 -- 172.21.15.39:0/2663707119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa815d020 msgr2=0x7f3aa815d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.577+0000 7f3aad400640 1 --2- 172.21.15.39:0/2663707119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa815d020 0x7f3aa815d490 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80b4c00 tx=0x7f3a9002b910 comp rx=0 tx=0).stop 2024-01-24T17:43:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.577+0000 7f3aad400640 1 -- 172.21.15.39:0/2663707119 shutdown_connections 2024-01-24T17:43:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.577+0000 7f3aad400640 1 --2- 172.21.15.39:0/2663707119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa815d020 0x7f3aa815d490 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.578+0000 7f3aad400640 1 --2- 172.21.15.39:0/2663707119 >> v2:172.21.15.100:3300/0 conn(0x7f3aa80cf990 0x7f3aa80cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.578+0000 7f3aad400640 1 -- 172.21.15.39:0/2663707119 >> 172.21.15.39:0/2663707119 conn(0x7f3aa8071790 msgr2=0x7f3aa8071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:16.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.578+0000 7f3aad400640 1 -- 172.21.15.39:0/2663707119 shutdown_connections 2024-01-24T17:43:16.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.578+0000 7f3aad400640 1 -- 172.21.15.39:0/2663707119 wait complete. 2024-01-24T17:43:16.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.579+0000 7f3aad400640 1 Processor -- start 2024-01-24T17:43:16.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.580+0000 7f3aad400640 1 -- start start 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.580+0000 7f3aad400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3aa80cf990 0x7f3aa814ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aad400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aa815d020 0x7f3aa8156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aad400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa8160610 0x7f3aa815bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aad400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa814d3f0 con 0x7f3aa8160610 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aad400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80b3f80 con 0x7f3aa815d020 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aad400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa80cc8d0 con 0x7f3aa80cf990 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa8160610 0x7f3aa815bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa8160610 0x7f3aa815bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50134/0 (socket says 172.21.15.39:50134) 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa7200640 1 -- 172.21.15.39:0/803133277 learned_addr learned my addr 172.21.15.39:0/803133277 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa5e00640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aa815d020 0x7f3aa8156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa6800640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3aa80cf990 0x7f3aa814ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa7200640 1 -- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3aa80cf990 msgr2=0x7f3aa814ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa7200640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3aa80cf990 0x7f3aa814ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa7200640 1 -- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aa815d020 msgr2=0x7f3aa8156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa7200640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aa815d020 0x7f3aa8156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa7200640 1 -- 172.21.15.39:0/803133277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a900094d0 con 0x7f3aa8160610 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.581+0000 7f3aa5e00640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aa815d020 0x7f3aa8156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.582+0000 7f3aa7200640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa8160610 0x7f3aa815bcc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000c7b0 tx=0x7f3aa000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.582+0000 7f3a9ea00640 1 -- 172.21.15.39:0/803133277 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0004b60 con 0x7f3aa8160610 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.582+0000 7f3a9ea00640 1 -- 172.21.15.39:0/803133277 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa0004cc0 con 0x7f3aa8160610 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.582+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa80cbcc0 con 0x7f3aa8160610 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.582+0000 7f3a9ea00640 1 -- 172.21.15.39:0/803133277 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa000aa30 con 0x7f3aa8160610 2024-01-24T17:43:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.582+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa80cc280 con 0x7f3aa8160610 2024-01-24T17:43:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.583+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa815e330 con 0x7f3aa8160610 2024-01-24T17:43:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.585+0000 7f3a9ea00640 1 -- 172.21.15.39:0/803133277 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3aa000ab90 con 0x7f3aa8160610 2024-01-24T17:43:16.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.585+0000 7f3a9ea00640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a8006a430 0x7f3a8006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:16.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.586+0000 7f3a9ea00640 1 -- 172.21.15.39:0/803133277 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3aa0088b60 con 0x7f3aa8160610 2024-01-24T17:43:16.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.586+0000 7f3aa6800640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a8006a430 0x7f3a8006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:16.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.587+0000 7f3aa6800640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a8006a430 0x7f3a8006c8f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3a98006fd0 tx=0x7f3a98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:16.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.590+0000 7f3a9ea00640 1 -- 172.21.15.39:0/803133277 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3aa00540c0 con 0x7f3aa8160610 2024-01-24T17:43:16.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.800+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3aa80cba30 con 0x7f3aa8160610 2024-01-24T17:43:16.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.801+0000 7f3a9ea00640 1 -- 172.21.15.39:0/803133277 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3aa00576e0 con 0x7f3aa8160610 2024-01-24T17:43:16.802 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:43:16.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a8006a430 msgr2=0x7f3a8006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a8006a430 0x7f3a8006c8f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3a98006fd0 tx=0x7f3a98008040 comp rx=0 tx=0).stop 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa8160610 msgr2=0x7f3aa815bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa8160610 0x7f3aa815bcc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000c7b0 tx=0x7f3aa000cc70 comp rx=0 tx=0).stop 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 shutdown_connections 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3a8006a430 0x7f3a8006c8f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aa8160610 0x7f3aa815bcc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aa815d020 0x7f3aa8156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 --2- 172.21.15.39:0/803133277 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3aa80cf990 0x7f3aa814ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 >> 172.21.15.39:0/803133277 conn(0x7f3aa8071790 msgr2=0x7f3aa815aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 shutdown_connections 2024-01-24T17:43:16.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:16.804+0000 7f3aad400640 1 -- 172.21.15.39:0/803133277 wait complete. 2024-01-24T17:43:16.824 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:43:16.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:16.824 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:16.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[50687]: fuse finished with error 0 and tester_r 0 2024-01-24T17:43:16.926 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:43:20.084 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:43:20.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:43:23.028 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:43:23.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:23.028 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:43:23.089 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:43:23.089 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:23.145 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:23.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.374+0000 7ff528200640 1 Processor -- start 2024-01-24T17:43:23.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.374+0000 7ff528200640 1 -- start start 2024-01-24T17:43:23.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.376+0000 7ff528200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff52015d020 0x7ff52015d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.376+0000 7ff528200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff52015eb80 0x7ff52015f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.376+0000 7ff528200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52015dac0 con 0x7ff5200cf990 2024-01-24T17:43:23.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.376+0000 7ff528200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52015e080 con 0x7ff52015d020 2024-01-24T17:43:23.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.376+0000 7ff528200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52015dda0 con 0x7ff52015eb80 2024-01-24T17:43:23.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.376+0000 7ff525800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff5200cf990 0x7ff5200cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:51532/0 (socket says 172.21.15.39:51532) 2024-01-24T17:43:23.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.376+0000 7ff525800640 1 -- 172.21.15.39:0/791082970 learned_addr learned my addr 172.21.15.39:0/791082970 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:23.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.376+0000 7ff524e00640 1 --2- 172.21.15.39:0/791082970 >> v2:172.21.15.39:3300/0 conn(0x7ff52015d020 0x7ff52015d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.377+0000 7ff51fe00640 1 -- 172.21.15.39:0/791082970 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972330407 0 0) 0x7ff52015dac0 con 0x7ff5200cf990 2024-01-24T17:43:23.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.377+0000 7ff51fe00640 1 -- 172.21.15.39:0/791082970 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff500003520 con 0x7ff5200cf990 2024-01-24T17:43:23.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.377+0000 7ff526200640 1 --2- 172.21.15.39:0/791082970 >> v2:172.21.15.100:3300/0 conn(0x7ff52015eb80 0x7ff52015f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.377+0000 7ff524e00640 1 -- 172.21.15.39:0/791082970 >> v2:172.21.15.100:3300/0 conn(0x7ff52015eb80 msgr2=0x7ff52015f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:23.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.377+0000 7ff524e00640 1 --2- 172.21.15.39:0/791082970 >> v2:172.21.15.100:3300/0 conn(0x7ff52015eb80 0x7ff52015f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.377+0000 7ff524e00640 1 -- 172.21.15.39:0/791082970 >> v1:172.21.15.39:6789/0 conn(0x7ff5200cf990 legacy=0x7ff5200cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.377+0000 7ff524e00640 1 -- 172.21.15.39:0/791082970 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52015e380 con 0x7ff52015d020 2024-01-24T17:43:23.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.377+0000 7ff526200640 1 --2- 172.21.15.39:0/791082970 >> v2:172.21.15.100:3300/0 conn(0x7ff52015eb80 0x7ff52015f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:23.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.378+0000 7ff524e00640 1 --2- 172.21.15.39:0/791082970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015d020 0x7ff52015d490 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff5200b4900 tx=0x7ff50802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e9a5bfb95df357 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:23.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.378+0000 7ff51fe00640 1 -- 172.21.15.39:0/791082970 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508031830 con 0x7ff52015d020 2024-01-24T17:43:23.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff51fe00640 1 -- 172.21.15.39:0/791082970 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff508031990 con 0x7ff52015d020 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff51fe00640 1 -- 172.21.15.39:0/791082970 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508031d20 con 0x7ff52015d020 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff528200640 1 -- 172.21.15.39:0/791082970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015d020 msgr2=0x7ff52015d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff528200640 1 --2- 172.21.15.39:0/791082970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015d020 0x7ff52015d490 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff5200b4900 tx=0x7ff50802b8d0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff528200640 1 -- 172.21.15.39:0/791082970 shutdown_connections 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff528200640 1 --2- 172.21.15.39:0/791082970 >> v2:172.21.15.100:3300/0 conn(0x7ff52015eb80 0x7ff52015f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff528200640 1 --2- 172.21.15.39:0/791082970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015d020 0x7ff52015d490 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff528200640 1 -- 172.21.15.39:0/791082970 >> 172.21.15.39:0/791082970 conn(0x7ff520071790 msgr2=0x7ff520071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff528200640 1 -- 172.21.15.39:0/791082970 shutdown_connections 2024-01-24T17:43:23.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.379+0000 7ff528200640 1 -- 172.21.15.39:0/791082970 wait complete. 2024-01-24T17:43:23.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.381+0000 7ff528200640 1 Processor -- start 2024-01-24T17:43:23.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.381+0000 7ff528200640 1 -- start start 2024-01-24T17:43:23.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff528200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5200cf990 0x7ff520159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff528200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff52015d020 0x7ff5200d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff528200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015eb80 0x7ff5200c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff528200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200c7580 con 0x7ff52015eb80 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff528200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200d2660 con 0x7ff52015d020 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff528200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200d2940 con 0x7ff5200cf990 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff526200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015eb80 0x7ff5200c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff526200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015eb80 0x7ff5200c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55688/0 (socket says 172.21.15.39:55688) 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.382+0000 7ff526200640 1 -- 172.21.15.39:0/2353156692 learned_addr learned my addr 172.21.15.39:0/2353156692 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff525800640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5200cf990 0x7ff520159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff524e00640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff52015d020 0x7ff5200d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff526200640 1 -- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5200cf990 msgr2=0x7ff520159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff526200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5200cf990 0x7ff520159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff526200640 1 -- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff52015d020 msgr2=0x7ff5200d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff526200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff52015d020 0x7ff5200d1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff526200640 1 -- 172.21.15.39:0/2353156692 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508009400 con 0x7ff52015eb80 2024-01-24T17:43:23.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff526200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015eb80 0x7ff5200c9ad0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff51800cd50 tx=0x7ff51800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:23.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff51d600640 1 -- 172.21.15.39:0/2353156692 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51800f680 con 0x7ff52015eb80 2024-01-24T17:43:23.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff51d600640 1 -- 172.21.15.39:0/2353156692 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51800f800 con 0x7ff52015eb80 2024-01-24T17:43:23.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff51d600640 1 -- 172.21.15.39:0/2353156692 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51800fbb0 con 0x7ff52015eb80 2024-01-24T17:43:23.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.383+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5200c7840 con 0x7ff52015eb80 2024-01-24T17:43:23.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.384+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5200cd020 con 0x7ff52015eb80 2024-01-24T17:43:23.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.385+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5200c7d60 con 0x7ff52015eb80 2024-01-24T17:43:23.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.386+0000 7ff51d600640 1 -- 172.21.15.39:0/2353156692 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff51800b040 con 0x7ff52015eb80 2024-01-24T17:43:23.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.387+0000 7ff51d600640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff4f806a430 0x7ff4f806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.387+0000 7ff51d600640 1 -- 172.21.15.39:0/2353156692 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff518089310 con 0x7ff52015eb80 2024-01-24T17:43:23.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.388+0000 7ff525800640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff4f806a430 0x7ff4f806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.389+0000 7ff525800640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff4f806a430 0x7ff4f806c8f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff5100011e0 tx=0x7ff510009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:23.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.392+0000 7ff51d600640 1 -- 172.21.15.39:0/2353156692 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff5200c7d60 con 0x7ff52015eb80 2024-01-24T17:43:23.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.588+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5200c7d60 con 0x7ff52015eb80 2024-01-24T17:43:23.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.591+0000 7ff51d600640 1 -- 172.21.15.39:0/2353156692 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16290 (secure 0 0 0) 0x7ff5200c7d60 con 0x7ff52015eb80 2024-01-24T17:43:23.592 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:23.592 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:43:16.261174+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.1c","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.32","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff4f806a430 msgr2=0x7ff4f806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff4f806a430 0x7ff4f806c8f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff5100011e0 tx=0x7ff510009040 comp rx=0 tx=0).stop 2024-01-24T17:43:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015eb80 msgr2=0x7ff5200c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015eb80 0x7ff5200c9ad0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff51800cd50 tx=0x7ff51800a650 comp rx=0 tx=0).stop 2024-01-24T17:43:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 shutdown_connections 2024-01-24T17:43:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff4f806a430 0x7ff4f806c8f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff52015eb80 0x7ff5200c9ad0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff52015d020 0x7ff5200d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 --2- 172.21.15.39:0/2353156692 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5200cf990 0x7ff520159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 >> 172.21.15.39:0/2353156692 conn(0x7ff520071790 msgr2=0x7ff52015c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:23.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 shutdown_connections 2024-01-24T17:43:23.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.594+0000 7ff528200640 1 -- 172.21.15.39:0/2353156692 wait complete. 2024-01-24T17:43:23.615 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:42:54.883732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:42:55.727240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:43:23.616 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:43:23.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.840+0000 7f36c0200640 1 Processor -- start 2024-01-24T17:43:23.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.841+0000 7f36c0200640 1 -- start start 2024-01-24T17:43:23.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.842+0000 7f36c0200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36b81597f0 0x7f36b806d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.842+0000 7f36c0200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 0x7f36b80b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.842+0000 7f36c0200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f36b80b00e0 0x7f36b80b0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.842+0000 7f36c0200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b80ca0d0 con 0x7f36b81597f0 2024-01-24T17:43:23.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.842+0000 7f36c0200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b80b0ad0 con 0x7f36b80b00e0 2024-01-24T17:43:23.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.842+0000 7f36c0200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b806c4d0 con 0x7f36b806bfa0 2024-01-24T17:43:23.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bd800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36b81597f0 0x7f36b806d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bd800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36b81597f0 0x7f36b806d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55698/0 (socket says 172.21.15.39:55698) 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bd800640 1 -- 172.21.15.39:0/3043871431 learned_addr learned my addr 172.21.15.39:0/3043871431 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36be200640 1 --2- 172.21.15.39:0/3043871431 >> v2:172.21.15.100:3300/0 conn(0x7f36b80b00e0 0x7f36b80b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bd800640 1 -- 172.21.15.39:0/3043871431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 msgr2=0x7f36b80b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bce00640 1 --2- 172.21.15.39:0/3043871431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 0x7f36b80b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bd800640 1 --2- 172.21.15.39:0/3043871431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 0x7f36b80b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bd800640 1 -- 172.21.15.39:0/3043871431 >> v2:172.21.15.100:3300/0 conn(0x7f36b80b00e0 msgr2=0x7f36b80b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bd800640 1 --2- 172.21.15.39:0/3043871431 >> v2:172.21.15.100:3300/0 conn(0x7f36b80b00e0 0x7f36b80b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bd800640 1 -- 172.21.15.39:0/3043871431 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b806c7b0 con 0x7f36b81597f0 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36be200640 1 --2- 172.21.15.39:0/3043871431 >> v2:172.21.15.100:3300/0 conn(0x7f36b80b00e0 0x7f36b80b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:23.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.843+0000 7f36bce00640 1 --2- 172.21.15.39:0/3043871431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 0x7f36b80b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:23.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.844+0000 7f36bd800640 1 --2- 172.21.15.39:0/3043871431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b81597f0 0x7f36b806d890 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f36b80cd550 tx=0x7f36a802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b75809310fd7ad server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.845+0000 7f36b7e00640 1 -- 172.21.15.39:0/3043871431 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8031860 con 0x7f36b81597f0 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.845+0000 7f36b7e00640 1 -- 172.21.15.39:0/3043871431 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a80319c0 con 0x7f36b81597f0 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.845+0000 7f36b7e00640 1 -- 172.21.15.39:0/3043871431 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a8031d50 con 0x7f36b81597f0 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.845+0000 7f36c0200640 1 -- 172.21.15.39:0/3043871431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b81597f0 msgr2=0x7f36b806d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.845+0000 7f36c0200640 1 --2- 172.21.15.39:0/3043871431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b81597f0 0x7f36b806d890 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f36b80cd550 tx=0x7f36a802b900 comp rx=0 tx=0).stop 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.845+0000 7f36c0200640 1 -- 172.21.15.39:0/3043871431 shutdown_connections 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.846+0000 7f36c0200640 1 --2- 172.21.15.39:0/3043871431 >> v2:172.21.15.100:3300/0 conn(0x7f36b80b00e0 0x7f36b80b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.846+0000 7f36c0200640 1 --2- 172.21.15.39:0/3043871431 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 0x7f36b80b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.846+0000 7f36c0200640 1 --2- 172.21.15.39:0/3043871431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b81597f0 0x7f36b806d890 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.846+0000 7f36c0200640 1 -- 172.21.15.39:0/3043871431 >> 172.21.15.39:0/3043871431 conn(0x7f36b80cb0e0 msgr2=0x7f36b80cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.846+0000 7f36c0200640 1 -- 172.21.15.39:0/3043871431 shutdown_connections 2024-01-24T17:43:23.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.846+0000 7f36c0200640 1 -- 172.21.15.39:0/3043871431 wait complete. 2024-01-24T17:43:23.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.847+0000 7f36c0200640 1 Processor -- start 2024-01-24T17:43:23.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.848+0000 7f36c0200640 1 -- start start 2024-01-24T17:43:23.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.848+0000 7f36c0200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 0x7f36b814fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.848+0000 7f36c0200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b80b00e0 0x7f36b81f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36c0200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b81597f0 0x7f36b81f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36c0200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b815b990 con 0x7f36b80b00e0 2024-01-24T17:43:23.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36c0200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b80ab960 con 0x7f36b81597f0 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36c0200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b815b0f0 con 0x7f36b806bfa0 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36bce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b80b00e0 0x7f36b81f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36bce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b80b00e0 0x7f36b81f5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55710/0 (socket says 172.21.15.39:55710) 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36bce00640 1 -- 172.21.15.39:0/3833169468 learned_addr learned my addr 172.21.15.39:0/3833169468 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36bce00640 1 -- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 msgr2=0x7f36b814fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36bce00640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 0x7f36b814fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36bce00640 1 -- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b81597f0 msgr2=0x7f36b81f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36be200640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b81597f0 0x7f36b81f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36bce00640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b81597f0 0x7f36b81f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.849+0000 7f36bce00640 1 -- 172.21.15.39:0/3833169468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a80094d0 con 0x7f36b80b00e0 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.850+0000 7f36bce00640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b80b00e0 0x7f36b81f5a20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f36a000c7f0 tx=0x7f36a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.850+0000 7f36b5600640 1 -- 172.21.15.39:0/3833169468 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a0004230 con 0x7f36b80b00e0 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.850+0000 7f36b5600640 1 -- 172.21.15.39:0/3833169468 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a00043b0 con 0x7f36b80b00e0 2024-01-24T17:43:23.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.850+0000 7f36b5600640 1 -- 172.21.15.39:0/3833169468 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a0004520 con 0x7f36b80b00e0 2024-01-24T17:43:23.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.850+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b815b3d0 con 0x7f36b80b00e0 2024-01-24T17:43:23.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.850+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b81500a0 con 0x7f36b80b00e0 2024-01-24T17:43:23.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.851+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b8073ef0 con 0x7f36b80b00e0 2024-01-24T17:43:23.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.852+0000 7f36b5600640 1 -- 172.21.15.39:0/3833169468 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f36a0007a70 con 0x7f36b80b00e0 2024-01-24T17:43:23.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.853+0000 7f36b5600640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f369006a480 0x7f369006c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:23.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.853+0000 7f36b5600640 1 -- 172.21.15.39:0/3833169468 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f36a0088ac0 con 0x7f36b80b00e0 2024-01-24T17:43:23.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.854+0000 7f36bd800640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f369006a480 0x7f369006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:23.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.855+0000 7f36bd800640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f369006a480 0x7f369006c940 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36a8036000 tx=0x7f36a8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:23.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:23.858+0000 7f36b5600640 1 -- 172.21.15.39:0/3833169468 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f36a0054010 con 0x7f36b80b00e0 2024-01-24T17:43:24.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.051+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f36b80af0c0 con 0x7f36b80b00e0 2024-01-24T17:43:24.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.052+0000 7f36b5600640 1 -- 172.21.15.39:0/3833169468 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f36a001e0a0 con 0x7f36b80b00e0 2024-01-24T17:43:24.052 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:43:24.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f369006a480 msgr2=0x7f369006c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f369006a480 0x7f369006c940 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36a8036000 tx=0x7f36a8030000 comp rx=0 tx=0).stop 2024-01-24T17:43:24.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b80b00e0 msgr2=0x7f36b81f5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b80b00e0 0x7f36b81f5a20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f36a000c7f0 tx=0x7f36a000ccb0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 shutdown_connections 2024-01-24T17:43:24.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36b81597f0 0x7f36b81f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f369006a480 0x7f369006c940 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36b80b00e0 0x7f36b81f5a20 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 --2- 172.21.15.39:0/3833169468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36b806bfa0 0x7f36b814fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.054+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 >> 172.21.15.39:0/3833169468 conn(0x7f36b80cb0e0 msgr2=0x7f36b80ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:24.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.055+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 shutdown_connections 2024-01-24T17:43:24.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.055+0000 7f36c0200640 1 -- 172.21.15.39:0/3833169468 wait complete. 2024-01-24T17:43:24.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:43:24.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.302+0000 7ff217000640 1 Processor -- start 2024-01-24T17:43:24.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.303+0000 7ff217000640 1 -- start start 2024-01-24T17:43:24.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff217000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff21015fff0 0x7ff2100b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff217000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff2100b2a10 0x7ff2100d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff217000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100d12f0 0x7ff2100b4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff217000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2100d17a0 con 0x7ff2100b2a10 2024-01-24T17:43:24.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff217000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2100b2e90 con 0x7ff21015fff0 2024-01-24T17:43:24.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff217000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100cda90 con 0x7ff2100d12f0 2024-01-24T17:43:24.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff207e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff2100b2a10 0x7ff2100d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff207e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff2100b2a10 0x7ff2100d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55714/0 (socket says 172.21.15.39:55714) 2024-01-24T17:43:24.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff207e00640 1 -- 172.21.15.39:0/3820020475 learned_addr learned my addr 172.21.15.39:0/3820020475 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff215000640 1 --2- 172.21.15.39:0/3820020475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100d12f0 0x7ff2100b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.304+0000 7ff207e00640 1 -- 172.21.15.39:0/3820020475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100d12f0 msgr2=0x7ff2100b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.305+0000 7ff207e00640 1 --2- 172.21.15.39:0/3820020475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100d12f0 0x7ff2100b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.305+0000 7ff207e00640 1 -- 172.21.15.39:0/3820020475 >> v2:172.21.15.100:3300/0 conn(0x7ff21015fff0 msgr2=0x7ff2100b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.305+0000 7ff20fe00640 1 --2- 172.21.15.39:0/3820020475 >> v2:172.21.15.100:3300/0 conn(0x7ff21015fff0 0x7ff2100b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.305+0000 7ff207e00640 1 --2- 172.21.15.39:0/3820020475 >> v2:172.21.15.100:3300/0 conn(0x7ff21015fff0 0x7ff2100b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.305+0000 7ff207e00640 1 -- 172.21.15.39:0/3820020475 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2100d32c0 con 0x7ff2100b2a10 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.305+0000 7ff20fe00640 1 --2- 172.21.15.39:0/3820020475 >> v2:172.21.15.100:3300/0 conn(0x7ff21015fff0 0x7ff2100b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.306+0000 7ff207e00640 1 --2- 172.21.15.39:0/3820020475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100b2a10 0x7ff2100d0db0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2100d28e0 tx=0x7ff1f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734298b265653992 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.306+0000 7ff20f400640 1 -- 172.21.15.39:0/3820020475 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8031870 con 0x7ff2100b2a10 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.306+0000 7ff20f400640 1 -- 172.21.15.39:0/3820020475 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f80319d0 con 0x7ff2100b2a10 2024-01-24T17:43:24.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.306+0000 7ff20f400640 1 -- 172.21.15.39:0/3820020475 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8031d60 con 0x7ff2100b2a10 2024-01-24T17:43:24.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 -- 172.21.15.39:0/3820020475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100b2a10 msgr2=0x7ff2100d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 --2- 172.21.15.39:0/3820020475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100b2a10 0x7ff2100d0db0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2100d28e0 tx=0x7ff1f802b910 comp rx=0 tx=0).stop 2024-01-24T17:43:24.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 -- 172.21.15.39:0/3820020475 shutdown_connections 2024-01-24T17:43:24.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 --2- 172.21.15.39:0/3820020475 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100d12f0 0x7ff2100b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 --2- 172.21.15.39:0/3820020475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100b2a10 0x7ff2100d0db0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 --2- 172.21.15.39:0/3820020475 >> v2:172.21.15.100:3300/0 conn(0x7ff21015fff0 0x7ff2100b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 -- 172.21.15.39:0/3820020475 >> 172.21.15.39:0/3820020475 conn(0x7ff21009bcc0 msgr2=0x7ff2100d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:24.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 -- 172.21.15.39:0/3820020475 shutdown_connections 2024-01-24T17:43:24.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.307+0000 7ff217000640 1 -- 172.21.15.39:0/3820020475 wait complete. 2024-01-24T17:43:24.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.309+0000 7ff217000640 1 Processor -- start 2024-01-24T17:43:24.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.309+0000 7ff217000640 1 -- start start 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff217000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100b2a10 0x7ff2100caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff217000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100d12f0 0x7ff2101547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff217000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff21015fff0 0x7ff2101566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff217000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21015b220 con 0x7ff2100d12f0 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff217000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210157410 con 0x7ff21015fff0 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff217000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff210157130 con 0x7ff2100b2a10 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff207e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100d12f0 0x7ff2101547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff207e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100d12f0 0x7ff2101547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55724/0 (socket says 172.21.15.39:55724) 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff207e00640 1 -- 172.21.15.39:0/1630275732 learned_addr learned my addr 172.21.15.39:0/1630275732 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:24.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff207e00640 1 -- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100b2a10 msgr2=0x7ff2100caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff20fe00640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100b2a10 0x7ff2100caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff215000640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff21015fff0 0x7ff2101566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff207e00640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100b2a10 0x7ff2100caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.310+0000 7ff207e00640 1 -- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff21015fff0 msgr2=0x7ff2101566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff207e00640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff21015fff0 0x7ff2101566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff207e00640 1 -- 172.21.15.39:0/1630275732 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80094d0 con 0x7ff2100d12f0 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff215000640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff21015fff0 0x7ff2101566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff207e00640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100d12f0 0x7ff2101547c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff1f802b950 tx=0x7ff1f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff20cc00640 1 -- 172.21.15.39:0/1630275732 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8031820 con 0x7ff2100d12f0 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff20cc00640 1 -- 172.21.15.39:0/1630275732 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f80443f0 con 0x7ff2100d12f0 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff20cc00640 1 -- 172.21.15.39:0/1630275732 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80447a0 con 0x7ff2100d12f0 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21015abd0 con 0x7ff2100d12f0 2024-01-24T17:43:24.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.311+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff210154d00 con 0x7ff2100d12f0 2024-01-24T17:43:24.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.313+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21015aa30 con 0x7ff2100d12f0 2024-01-24T17:43:24.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.314+0000 7ff20cc00640 1 -- 172.21.15.39:0/1630275732 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff1f804d450 con 0x7ff2100d12f0 2024-01-24T17:43:24.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.314+0000 7ff20cc00640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff1e806a430 0x7ff1e806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.315+0000 7ff20cc00640 1 -- 172.21.15.39:0/1630275732 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff1f802d040 con 0x7ff2100d12f0 2024-01-24T17:43:24.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.315+0000 7ff20fe00640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff1e806a430 0x7ff1e806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.316+0000 7ff20fe00640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff1e806a430 0x7ff1e806c8f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff20000b7c0 tx=0x7ff200008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:24.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.319+0000 7ff20cc00640 1 -- 172.21.15.39:0/1630275732 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff1f80ddcb0 con 0x7ff2100d12f0 2024-01-24T17:43:24.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.515+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff210073ef0 con 0x7ff2100d12f0 2024-01-24T17:43:24.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.516+0000 7ff20cc00640 1 -- 172.21.15.39:0/1630275732 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff21015aa30 con 0x7ff2100d12f0 2024-01-24T17:43:24.517 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:43:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff1e806a430 msgr2=0x7ff1e806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff1e806a430 0x7ff1e806c8f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff20000b7c0 tx=0x7ff200008040 comp rx=0 tx=0).stop 2024-01-24T17:43:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100d12f0 msgr2=0x7ff2101547c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100d12f0 0x7ff2101547c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff1f802b950 tx=0x7ff1f80046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 shutdown_connections 2024-01-24T17:43:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff21015fff0 0x7ff2101566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff1e806a430 0x7ff1e806c8f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100d12f0 0x7ff2101547c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 --2- 172.21.15.39:0/1630275732 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2100b2a10 0x7ff2100caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 >> 172.21.15.39:0/1630275732 conn(0x7ff21009bcc0 msgr2=0x7ff21006af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:24.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 shutdown_connections 2024-01-24T17:43:24.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.519+0000 7ff217000640 1 -- 172.21.15.39:0/1630275732 wait complete. 2024-01-24T17:43:24.540 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:43:24.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.767+0000 7ff387c00640 1 Processor -- start 2024-01-24T17:43:24.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.767+0000 7ff387c00640 1 -- start start 2024-01-24T17:43:24.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.768+0000 7ff387c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff3800cc5c0 0x7ff3800ca130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.769+0000 7ff387c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800cdb80 con 0x7ff380158fc0 2024-01-24T17:43:24.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.769+0000 7ff387c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800c9690 con 0x7ff3800ca670 2024-01-24T17:43:24.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.769+0000 7ff387c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3800caae0 con 0x7ff3800cc5c0 2024-01-24T17:43:24.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.769+0000 7ff385200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff380158fc0 0x7ff38006bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:51538/0 (socket says 172.21.15.39:51538) 2024-01-24T17:43:24.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.769+0000 7ff385200640 1 -- 172.21.15.39:0/3884583413 learned_addr learned my addr 172.21.15.39:0/3884583413 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.769+0000 7ff37fe00640 1 --2- 172.21.15.39:0/3884583413 >> v2:172.21.15.100:3300/0 conn(0x7ff3800cc5c0 0x7ff3800ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.769+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673671656 0 0) 0x7ff370001490 con 0x7ff380158fc0 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.769+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff360003520 con 0x7ff380158fc0 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 485068405 0 0) 0x7ff360003520 con 0x7ff380158fc0 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 >> v2:172.21.15.100:3300/0 conn(0x7ff3800cc5c0 msgr2=0x7ff3800ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 --2- 172.21.15.39:0/3884583413 >> v2:172.21.15.100:3300/0 conn(0x7ff3800cc5c0 0x7ff3800ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 >> v1:172.21.15.100:6789/0 conn(0x7ff3800ca670 legacy=0x7ff380070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800cd5c0 con 0x7ff380158fc0 2024-01-24T17:43:24.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff370001b80 con 0x7ff380158fc0 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 >> v1:172.21.15.39:6789/0 conn(0x7ff380158fc0 legacy=0x7ff38006bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3600039a0 0x7ff360003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff360004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360005180 0x7ff36000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800caae0 con 0x7ff3600047c0 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36000c050 con 0x7ff3600039a0 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36000d320 con 0x7ff360005180 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.770+0000 7ff37fe00640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff360004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff385200640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3600039a0 0x7ff360003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff385c00640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360005180 0x7ff36000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff37fe00640 1 -- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360005180 msgr2=0x7ff36000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff37fe00640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360005180 0x7ff36000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff37fe00640 1 -- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3600039a0 msgr2=0x7ff360003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff37fe00640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3600039a0 0x7ff360003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff37fe00640 1 -- 172.21.15.39:0/3884583413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff360003520 con 0x7ff3600047c0 2024-01-24T17:43:24.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff385c00640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff360005180 0x7ff36000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:24.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.771+0000 7ff385c00640 1 -- 172.21.15.39:0/3884583413 reap_dead start 2024-01-24T17:43:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.772+0000 7ff37fe00640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff360004c40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff368009ac0 tx=0x7ff3600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea8cbaa5d05df22 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.772+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36802c6b0 con 0x7ff3600047c0 2024-01-24T17:43:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.772+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36802c810 con 0x7ff3600047c0 2024-01-24T17:43:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.772+0000 7ff37f400640 1 -- 172.21.15.39:0/3884583413 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36802cba0 con 0x7ff3600047c0 2024-01-24T17:43:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.773+0000 7ff387c00640 1 -- 172.21.15.39:0/3884583413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 msgr2=0x7ff360004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.773+0000 7ff387c00640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff360004c40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff368009ac0 tx=0x7ff3600057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.773+0000 7ff387c00640 1 -- 172.21.15.39:0/3884583413 shutdown_connections 2024-01-24T17:43:24.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.773+0000 7ff387c00640 1 --2- 172.21.15.39:0/3884583413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff360004c40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.773+0000 7ff387c00640 1 -- 172.21.15.39:0/3884583413 >> 172.21.15.39:0/3884583413 conn(0x7ff3800cb0e0 msgr2=0x7ff3800cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.773+0000 7ff387c00640 1 -- 172.21.15.39:0/3884583413 shutdown_connections 2024-01-24T17:43:24.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.773+0000 7ff387c00640 1 -- 172.21.15.39:0/3884583413 wait complete. 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.775+0000 7ff387c00640 1 Processor -- start 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.775+0000 7ff387c00640 1 -- start start 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff387c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff380154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff387c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff380159db0 0x7ff3800ae370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff387c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff38015a210 0x7ff3800ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff387c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38015b3a0 con 0x7ff3600047c0 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff387c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38015a790 con 0x7ff380159db0 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff387c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38015aa70 con 0x7ff38015a210 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff385200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff380154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff385200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff380154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55742/0 (socket says 172.21.15.39:55742) 2024-01-24T17:43:24.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.776+0000 7ff385200640 1 -- 172.21.15.39:0/753606853 learned_addr learned my addr 172.21.15.39:0/753606853 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff37fe00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff380159db0 0x7ff3800ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff385200640 1 -- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff38015a210 msgr2=0x7ff3800ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff385200640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff38015a210 0x7ff3800ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff385200640 1 -- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff380159db0 msgr2=0x7ff3800ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff385200640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff380159db0 0x7ff3800ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff385200640 1 -- 172.21.15.39:0/753606853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38015ad50 con 0x7ff3600047c0 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff385200640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff380154a40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff370007ae0 tx=0x7ff37000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff37cc00640 1 -- 172.21.15.39:0/753606853 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370013420 con 0x7ff3600047c0 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff37cc00640 1 -- 172.21.15.39:0/753606853 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3700135a0 con 0x7ff3600047c0 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38015b030 con 0x7ff3600047c0 2024-01-24T17:43:24.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.777+0000 7ff37cc00640 1 -- 172.21.15.39:0/753606853 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370013950 con 0x7ff3600047c0 2024-01-24T17:43:24.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.778+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff380154f80 con 0x7ff3600047c0 2024-01-24T17:43:24.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.779+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff380073ef0 con 0x7ff3600047c0 2024-01-24T17:43:24.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.780+0000 7ff37cc00640 1 -- 172.21.15.39:0/753606853 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff370013ab0 con 0x7ff3600047c0 2024-01-24T17:43:24.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.781+0000 7ff37cc00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff35806a430 0x7ff35806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:24.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.781+0000 7ff37cc00640 1 -- 172.21.15.39:0/753606853 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff370089180 con 0x7ff3600047c0 2024-01-24T17:43:24.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.781+0000 7ff37fe00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff35806a430 0x7ff35806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:24.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.782+0000 7ff37fe00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff35806a430 0x7ff35806c8f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff368036000 tx=0x7ff368030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:24.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.785+0000 7ff37cc00640 1 -- 172.21.15.39:0/753606853 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff370054650 con 0x7ff3600047c0 2024-01-24T17:43:24.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.982+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3800cdb80 con 0x7ff3600047c0 2024-01-24T17:43:24.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.982+0000 7ff37cc00640 1 -- 172.21.15.39:0/753606853 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff3800cdb80 con 0x7ff3600047c0 2024-01-24T17:43:24.983 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:43:24.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff35806a430 msgr2=0x7ff35806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff35806a430 0x7ff35806c8f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff368036000 tx=0x7ff368030000 comp rx=0 tx=0).stop 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 msgr2=0x7ff380154a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff380154a40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff370007ae0 tx=0x7ff37000e530 comp rx=0 tx=0).stop 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 shutdown_connections 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff38015a210 0x7ff3800ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff380159db0 0x7ff3800ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff35806a430 0x7ff35806c8f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 --2- 172.21.15.39:0/753606853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3600047c0 0x7ff380154a40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 >> 172.21.15.39:0/753606853 conn(0x7ff3800cb0e0 msgr2=0x7ff380157230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 shutdown_connections 2024-01-24T17:43:24.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:24.985+0000 7ff387c00640 1 -- 172.21.15.39:0/753606853 wait complete. 2024-01-24T17:43:25.005 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:43:25.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.231+0000 7f3c95000640 1 Processor -- start 2024-01-24T17:43:25.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.231+0000 7f3c95000640 1 -- start start 2024-01-24T17:43:25.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.232+0000 7f3c95000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 0x7f3c9015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.232+0000 7f3c95000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9015dac0 con 0x7f3c900cf990 2024-01-24T17:43:25.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.232+0000 7f3c95000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9015dda0 con 0x7f3c9015eb80 2024-01-24T17:43:25.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.232+0000 7f3c95000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9015e080 con 0x7f3c9015d020 2024-01-24T17:43:25.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8de00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3c900cf990 0x7f3c900cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:51540/0 (socket says 172.21.15.39:51540) 2024-01-24T17:43:25.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8de00640 1 -- 172.21.15.39:0/410029223 learned_addr learned my addr 172.21.15.39:0/410029223 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352340252 0 0) 0x7f3c9015dac0 con 0x7f3c900cf990 2024-01-24T17:43:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c70003520 con 0x7f3c900cf990 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8d400640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 0x7f3c9015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197154420 0 0) 0x7f3c70003520 con 0x7f3c900cf990 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 msgr2=0x7f3c9015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8ca00640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 0x7f3c9015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.233+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 >> v1:172.21.15.100:6789/0 conn(0x7f3c9015eb80 legacy=0x7f3c9015f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9015e380 con 0x7f3c900cf990 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f3c80001670 con 0x7f3c900cf990 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 >> v1:172.21.15.39:6789/0 conn(0x7f3c900cf990 legacy=0x7f3c900cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c700039a0 0x7f3c70003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 0x7f3c70005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9015e080 con 0x7f3c700047c0 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7000bb10 con 0x7f3c700039a0 2024-01-24T17:43:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7000bcd0 con 0x7f3c9015d020 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8d400640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8e800640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 0x7f3c70005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8de00640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c700039a0 0x7f3c70003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8d400640 1 -- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 msgr2=0x7f3c70005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8d400640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 0x7f3c70005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8d400640 1 -- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c700039a0 msgr2=0x7f3c70003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8d400640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c700039a0 0x7f3c70003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8d400640 1 -- 172.21.15.39:0/410029223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c70003520 con 0x7f3c700047c0 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8de00640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c700039a0 0x7f3c70003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:25.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.234+0000 7f3c8e800640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c9015d020 0x7f3c70005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:25.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c8d400640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c70004c40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800ba30 tx=0x7f3c700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db472d954eea1f21 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:25.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78032630 con 0x7f3c700047c0 2024-01-24T17:43:25.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c78032790 con 0x7f3c700047c0 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c8ca00640 1 -- 172.21.15.39:0/410029223 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78032b20 con 0x7f3c700047c0 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c95000640 1 -- 172.21.15.39:0/410029223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 msgr2=0x7f3c70004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c95000640 1 --2- 172.21.15.39:0/410029223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c70004c40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800ba30 tx=0x7f3c700057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c8e800640 1 -- 172.21.15.39:0/410029223 reap_dead start 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c95000640 1 -- 172.21.15.39:0/410029223 shutdown_connections 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.236+0000 7f3c95000640 1 -- 172.21.15.39:0/410029223 >> 172.21.15.39:0/410029223 conn(0x7f3c90071790 msgr2=0x7f3c90071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.237+0000 7f3c95000640 1 -- 172.21.15.39:0/410029223 shutdown_connections 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.237+0000 7f3c95000640 1 -- 172.21.15.39:0/410029223 wait complete. 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.238+0000 7f3c95000640 1 Processor -- start 2024-01-24T17:43:25.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.239+0000 7f3c95000640 1 -- start start 2024-01-24T17:43:25.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.239+0000 7f3c95000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c90157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c95000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c900c8c00 0x7f3c900c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c95000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c900ca9a0 0x7f3c900cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c95000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c901f6b50 con 0x7f3c700047c0 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c95000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900c66d0 con 0x7f3c900c8c00 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c95000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c900c69b0 con 0x7f3c900ca9a0 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c8de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c90157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c8de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c90157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55764/0 (socket says 172.21.15.39:55764) 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c8de00640 1 -- 172.21.15.39:0/711729649 learned_addr learned my addr 172.21.15.39:0/711729649 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c8de00640 1 -- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c900ca9a0 msgr2=0x7f3c900cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c8de00640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c900ca9a0 0x7f3c900cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c8de00640 1 -- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c900c8c00 msgr2=0x7f3c900c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c8de00640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c900c8c00 0x7f3c900c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.240+0000 7f3c8de00640 1 -- 172.21.15.39:0/711729649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900c6c90 con 0x7f3c700047c0 2024-01-24T17:43:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.241+0000 7f3c8de00640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c90157dd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000d800 tx=0x7f3c8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.241+0000 7f3c86000640 1 -- 172.21.15.39:0/711729649 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c800074e0 con 0x7f3c700047c0 2024-01-24T17:43:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.241+0000 7f3c86000640 1 -- 172.21.15.39:0/711729649 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c80007640 con 0x7f3c700047c0 2024-01-24T17:43:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.241+0000 7f3c86000640 1 -- 172.21.15.39:0/711729649 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80004a00 con 0x7f3c700047c0 2024-01-24T17:43:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.241+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c900c6f70 con 0x7f3c700047c0 2024-01-24T17:43:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.241+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c90158310 con 0x7f3c700047c0 2024-01-24T17:43:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.242+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9015caa0 con 0x7f3c700047c0 2024-01-24T17:43:25.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.243+0000 7f3c86000640 1 -- 172.21.15.39:0/711729649 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3c80004b60 con 0x7f3c700047c0 2024-01-24T17:43:25.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.244+0000 7f3c86000640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c6806a460 0x7f3c6806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.244+0000 7f3c86000640 1 -- 172.21.15.39:0/711729649 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3c800882c0 con 0x7f3c700047c0 2024-01-24T17:43:25.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.245+0000 7f3c8d400640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c6806a460 0x7f3c6806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.246+0000 7f3c8d400640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c6806a460 0x7f3c6806c920 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c78037000 tx=0x7f3c78032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:25.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.248+0000 7f3c86000640 1 -- 172.21.15.39:0/711729649 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3c800538d0 con 0x7f3c700047c0 2024-01-24T17:43:25.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.456+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3c9015e630 con 0x7f3c700047c0 2024-01-24T17:43:25.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.457+0000 7f3c86000640 1 -- 172.21.15.39:0/711729649 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3c80056ef0 con 0x7f3c700047c0 2024-01-24T17:43:25.458 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c6806a460 msgr2=0x7f3c6806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c6806a460 0x7f3c6806c920 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c78037000 tx=0x7f3c78032000 comp rx=0 tx=0).stop 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 msgr2=0x7f3c90157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c90157dd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000d800 tx=0x7f3c8000dcc0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 shutdown_connections 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c900ca9a0 0x7f3c900cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c900c8c00 0x7f3c900c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c6806a460 0x7f3c6806c920 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 --2- 172.21.15.39:0/711729649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c700047c0 0x7f3c90157dd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 >> 172.21.15.39:0/711729649 conn(0x7f3c90071790 msgr2=0x7f3c9015f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:25.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 shutdown_connections 2024-01-24T17:43:25.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.460+0000 7f3c95000640 1 -- 172.21.15.39:0/711729649 wait complete. 2024-01-24T17:43:25.478 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:43:25.478 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:25.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:25.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.761+0000 7fd132a00640 1 Processor -- start 2024-01-24T17:43:25.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.762+0000 7fd132a00640 1 -- start start 2024-01-24T17:43:25.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.763+0000 7fd132a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd12c15ca50 0x7fd12c0d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.763+0000 7fd132a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0cfd20 con 0x7fd12c0d1680 2024-01-24T17:43:25.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.763+0000 7fd132a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0d0240 con 0x7fd12c0cf940 2024-01-24T17:43:25.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.763+0000 7fd132a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12c0cff60 con 0x7fd12c15ca50 2024-01-24T17:43:25.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.764+0000 7fd131600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd12c15ca50 0x7fd12c0d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.764+0000 7fd130c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd12c0d1680 0x7fd12c0cf230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:51548/0 (socket says 172.21.15.39:51548) 2024-01-24T17:43:25.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.764+0000 7fd130c00640 1 -- 172.21.15.39:0/2388775163 learned_addr learned my addr 172.21.15.39:0/2388775163 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:25.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.764+0000 7fd131600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd12c15ca50 0x7fd12c0d1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55780/0 (socket says 172.21.15.39:55780) 2024-01-24T17:43:25.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.764+0000 7fd131600640 1 -- 172.21.15.39:0/2388775163 >> v1:172.21.15.100:6789/0 conn(0x7fd12c0cf940 legacy=0x7fd12c0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:25.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.764+0000 7fd131600640 1 -- 172.21.15.39:0/2388775163 >> v1:172.21.15.39:6789/0 conn(0x7fd12c0d1680 legacy=0x7fd12c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.764+0000 7fd131600640 1 -- 172.21.15.39:0/2388775163 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0d0660 con 0x7fd12c15ca50 2024-01-24T17:43:25.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.764+0000 7fd12be00640 1 -- 172.21.15.39:0/2388775163 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886970352 0 0) 0x7fd114001110 con 0x7fd12c0d1680 2024-01-24T17:43:25.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.765+0000 7fd131600640 1 --2- 172.21.15.39:0/2388775163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c15ca50 0x7fd12c0d1140 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd11c0097a0 tx=0x7fd11c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd166622bec09054 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:25.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd12be00640 1 -- 172.21.15.39:0/2388775163 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c02e070 con 0x7fd12c15ca50 2024-01-24T17:43:25.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd12be00640 1 -- 172.21.15.39:0/2388775163 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c0315a0 con 0x7fd12c15ca50 2024-01-24T17:43:25.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd12be00640 1 -- 172.21.15.39:0/2388775163 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c031930 con 0x7fd12c15ca50 2024-01-24T17:43:25.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd132a00640 1 -- 172.21.15.39:0/2388775163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c15ca50 msgr2=0x7fd12c0d1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd132a00640 1 --2- 172.21.15.39:0/2388775163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c15ca50 0x7fd12c0d1140 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd11c0097a0 tx=0x7fd11c02b840 comp rx=0 tx=0).stop 2024-01-24T17:43:25.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd132a00640 1 -- 172.21.15.39:0/2388775163 shutdown_connections 2024-01-24T17:43:25.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd132a00640 1 --2- 172.21.15.39:0/2388775163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c15ca50 0x7fd12c0d1140 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd132a00640 1 -- 172.21.15.39:0/2388775163 >> 172.21.15.39:0/2388775163 conn(0x7fd12c071dd0 msgr2=0x7fd12c09bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:25.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd132a00640 1 -- 172.21.15.39:0/2388775163 shutdown_connections 2024-01-24T17:43:25.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.766+0000 7fd132a00640 1 -- 172.21.15.39:0/2388775163 wait complete. 2024-01-24T17:43:25.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.768+0000 7fd132a00640 1 Processor -- start 2024-01-24T17:43:25.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.768+0000 7fd132a00640 1 -- start start 2024-01-24T17:43:25.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.769+0000 7fd132a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c0cf940 0x7fd12c158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.769+0000 7fd132a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd12c0d1680 0x7fd12c1f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.769+0000 7fd132a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd12c15ca50 0x7fd12c0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.769+0000 7fd132a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1f2e60 con 0x7fd12c0cf940 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.769+0000 7fd132a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c159a40 con 0x7fd12c0d1680 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.769+0000 7fd132a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c159d20 con 0x7fd12c15ca50 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c0cf940 0x7fd12c158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c0cf940 0x7fd12c158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55788/0 (socket says 172.21.15.39:55788) 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 -- 172.21.15.39:0/134522358 learned_addr learned my addr 172.21.15.39:0/134522358 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd132000640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd12c15ca50 0x7fd12c0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd130c00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd12c0d1680 0x7fd12c1f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 -- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd12c15ca50 msgr2=0x7fd12c0c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd12c15ca50 0x7fd12c0c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 -- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd12c0d1680 msgr2=0x7fd12c1f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd12c0d1680 0x7fd12c1f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 -- 172.21.15.39:0/134522358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c009400 con 0x7fd12c0cf940 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd130c00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd12c0d1680 0x7fd12c1f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.770+0000 7fd131600640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c0cf940 0x7fd12c158ca0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd11c0097a0 tx=0x7fd11c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.771+0000 7fd129600640 1 -- 172.21.15.39:0/134522358 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c004230 con 0x7fd12c0cf940 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.771+0000 7fd129600640 1 -- 172.21.15.39:0/134522358 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c0043b0 con 0x7fd12c0cf940 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.771+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c15a000 con 0x7fd12c0cf940 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.771+0000 7fd129600640 1 -- 172.21.15.39:0/134522358 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c004520 con 0x7fd12c0cf940 2024-01-24T17:43:25.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.771+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c15a5c0 con 0x7fd12c0cf940 2024-01-24T17:43:25.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.772+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12c0677b0 con 0x7fd12c0cf940 2024-01-24T17:43:25.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.773+0000 7fd129600640 1 -- 172.21.15.39:0/134522358 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd11c032880 con 0x7fd12c0cf940 2024-01-24T17:43:25.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.774+0000 7fd129600640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd10406a460 0x7fd10406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:25.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.774+0000 7fd129600640 1 -- 172.21.15.39:0/134522358 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd11c0aabc0 con 0x7fd12c0cf940 2024-01-24T17:43:25.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.775+0000 7fd130c00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd10406a460 0x7fd10406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:25.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.776+0000 7fd130c00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd10406a460 0x7fd10406c920 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd12c1efa50 tx=0x7fd114009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:25.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.781+0000 7fd129600640 1 -- 172.21.15.39:0/134522358 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd11c076090 con 0x7fd12c0cf940 2024-01-24T17:43:25.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.966+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd12c0677b0 con 0x7fd12c0cf940 2024-01-24T17:43:25.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.968+0000 7fd129600640 1 -- 172.21.15.39:0/134522358 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16290 (secure 0 0 0) 0x7fd11c0796b0 con 0x7fd12c0cf940 2024-01-24T17:43:25.969 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:25.969 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:43:16.261174+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.1c","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.32","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd10406a460 msgr2=0x7fd10406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd10406a460 0x7fd10406c920 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd12c1efa50 tx=0x7fd114009040 comp rx=0 tx=0).stop 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c0cf940 msgr2=0x7fd12c158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c0cf940 0x7fd12c158ca0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd11c0097a0 tx=0x7fd11c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 shutdown_connections 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd12c15ca50 0x7fd12c0c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd12c0d1680 0x7fd12c1f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd10406a460 0x7fd10406c920 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 --2- 172.21.15.39:0/134522358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c0cf940 0x7fd12c158ca0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:25.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.970+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 >> 172.21.15.39:0/134522358 conn(0x7fd12c071dd0 msgr2=0x7fd12c0cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:25.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.971+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 shutdown_connections 2024-01-24T17:43:25.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:25.971+0000 7fd132a00640 1 -- 172.21.15.39:0/134522358 wait complete. 2024-01-24T17:43:25.991 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:42:54.883732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:42:55.727240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:43:25.991 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:43:26.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.204+0000 7feb5f000640 1 Processor -- start 2024-01-24T17:43:26.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.204+0000 7feb5f000640 1 -- start start 2024-01-24T17:43:26.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.205+0000 7feb5f000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7feb580accd0 0x7feb5806d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.205+0000 7feb5f000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580ad0b0 con 0x7feb5806c3a0 2024-01-24T17:43:26.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.205+0000 7feb5f000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580b2370 con 0x7feb580cb540 2024-01-24T17:43:26.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.205+0000 7feb5f000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5806d840 con 0x7feb580accd0 2024-01-24T17:43:26.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.206+0000 7feb5d200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7feb5806c3a0 0x7feb580cae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:51554/0 (socket says 172.21.15.39:51554) 2024-01-24T17:43:26.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.206+0000 7feb5d200640 1 -- 172.21.15.39:0/2688354545 learned_addr learned my addr 172.21.15.39:0/2688354545 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:26.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.206+0000 7feb5dc00640 1 --2- 172.21.15.39:0/2688354545 >> v2:172.21.15.100:3300/0 conn(0x7feb580accd0 0x7feb5806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.206+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897551357 0 0) 0x7feb580ad0b0 con 0x7feb5806c3a0 2024-01-24T17:43:26.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.206+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb38003520 con 0x7feb5806c3a0 2024-01-24T17:43:26.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2621789949 0 0) 0x7feb38003520 con 0x7feb5806c3a0 2024-01-24T17:43:26.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 >> v2:172.21.15.100:3300/0 conn(0x7feb580accd0 msgr2=0x7feb5806d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:26.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 --2- 172.21.15.39:0/2688354545 >> v2:172.21.15.100:3300/0 conn(0x7feb580accd0 0x7feb5806d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 >> v1:172.21.15.100:6789/0 conn(0x7feb580cb540 legacy=0x7feb580ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580b2650 con 0x7feb5806c3a0 2024-01-24T17:43:26.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7feb40001670 con 0x7feb5806c3a0 2024-01-24T17:43:26.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 >> v1:172.21.15.39:6789/0 conn(0x7feb5806c3a0 legacy=0x7feb580cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb380039c0 0x7feb38003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb380047c0 0x7feb38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb38005180 0x7feb3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5806d840 con 0x7feb38005180 2024-01-24T17:43:26.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3800c080 con 0x7feb380047c0 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3800c240 con 0x7feb380039c0 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb5e600640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb38005180 0x7feb3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb5dc00640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb380039c0 0x7feb38003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.207+0000 7feb5d200640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb380047c0 0x7feb38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.208+0000 7feb5e600640 1 -- 172.21.15.39:0/2688354545 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb380039c0 msgr2=0x7feb38003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.208+0000 7feb5e600640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb380039c0 0x7feb38003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.208+0000 7feb5e600640 1 -- 172.21.15.39:0/2688354545 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb380047c0 msgr2=0x7feb38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.208+0000 7feb5e600640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb380047c0 0x7feb38004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.208+0000 7feb5e600640 1 -- 172.21.15.39:0/2688354545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38003520 con 0x7feb38005180 2024-01-24T17:43:26.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.209+0000 7feb5e600640 1 -- 172.21.15.39:0/2688354545 reap_dead start 2024-01-24T17:43:26.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.209+0000 7feb5e600640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb38005180 0x7feb3800b980 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7feb50001720 tx=0x7feb380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebeebe215e67ec2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:26.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.209+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb500052d0 con 0x7feb38005180 2024-01-24T17:43:26.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.209+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5002e050 con 0x7feb38005180 2024-01-24T17:43:26.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.209+0000 7feb57e00640 1 -- 172.21.15.39:0/2688354545 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb500035f0 con 0x7feb38005180 2024-01-24T17:43:26.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.209+0000 7feb5f000640 1 -- 172.21.15.39:0/2688354545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb38005180 msgr2=0x7feb3800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.209+0000 7feb5f000640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb38005180 0x7feb3800b980 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7feb50001720 tx=0x7feb380057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.210+0000 7feb5f000640 1 -- 172.21.15.39:0/2688354545 shutdown_connections 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.210+0000 7feb5f000640 1 --2- 172.21.15.39:0/2688354545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb38005180 0x7feb3800b980 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.210+0000 7feb5f000640 1 -- 172.21.15.39:0/2688354545 >> 172.21.15.39:0/2688354545 conn(0x7feb58069e50 msgr2=0x7feb5806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.210+0000 7feb5f000640 1 -- 172.21.15.39:0/2688354545 shutdown_connections 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.210+0000 7feb5f000640 1 -- 172.21.15.39:0/2688354545 wait complete. 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.211+0000 7feb5f000640 1 Processor -- start 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.212+0000 7feb5f000640 1 -- start start 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.212+0000 7feb5f000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb38005180 0x7feb581f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.212+0000 7feb5f000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb580ade80 0x7feb580ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.212+0000 7feb5f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb580af4b0 0x7feb580af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.212+0000 7feb5f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5815af30 con 0x7feb580af4b0 2024-01-24T17:43:26.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.212+0000 7feb5f000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb581f1ee0 con 0x7feb38005180 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.212+0000 7feb5f000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb580afe70 con 0x7feb580ade80 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5e600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb580af4b0 0x7feb580af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5e600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb580af4b0 0x7feb580af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55812/0 (socket says 172.21.15.39:55812) 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5e600640 1 -- 172.21.15.39:0/2995156650 learned_addr learned my addr 172.21.15.39:0/2995156650 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5dc00640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb38005180 0x7feb581f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5d200640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb580ade80 0x7feb580ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5e600640 1 -- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb580ade80 msgr2=0x7feb580ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5e600640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb580ade80 0x7feb580ae300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5e600640 1 -- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb38005180 msgr2=0x7feb581f19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5e600640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb38005180 0x7feb581f19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5e600640 1 -- 172.21.15.39:0/2995156650 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580ae900 con 0x7feb580af4b0 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.213+0000 7feb5d200640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb580ade80 0x7feb580ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:26.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.214+0000 7feb5e600640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb580af4b0 0x7feb580af930 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feb50001720 tx=0x7feb50001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:26.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.214+0000 7feb55600640 1 -- 172.21.15.39:0/2995156650 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50007a30 con 0x7feb580af4b0 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.214+0000 7feb55600640 1 -- 172.21.15.39:0/2995156650 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb50007020 con 0x7feb580af4b0 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.214+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb580aebe0 con 0x7feb580af4b0 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.214+0000 7feb55600640 1 -- 172.21.15.39:0/2995156650 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50007390 con 0x7feb580af4b0 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.214+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb580af1a0 con 0x7feb580af4b0 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.215+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5804dff0 con 0x7feb580af4b0 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.216+0000 7feb55600640 1 -- 172.21.15.39:0/2995156650 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7feb500074f0 con 0x7feb580af4b0 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.217+0000 7feb55600640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb3006a430 0x7feb3006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.217+0000 7feb55600640 1 -- 172.21.15.39:0/2995156650 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7feb50037040 con 0x7feb580af4b0 2024-01-24T17:43:26.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.218+0000 7feb5dc00640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb3006a430 0x7feb3006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.219+0000 7feb5dc00640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb3006a430 0x7feb3006c8f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7feb48007380 tx=0x7feb4800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:26.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.221+0000 7feb55600640 1 -- 172.21.15.39:0/2995156650 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7feb50074b30 con 0x7feb580af4b0 2024-01-24T17:43:26.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.415+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb580cca30 con 0x7feb580af4b0 2024-01-24T17:43:26.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.416+0000 7feb55600640 1 -- 172.21.15.39:0/2995156650 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb50078150 con 0x7feb580af4b0 2024-01-24T17:43:26.417 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.418+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb3006a430 msgr2=0x7feb3006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.418+0000 7feb5f000640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb3006a430 0x7feb3006c8f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7feb48007380 tx=0x7feb4800b040 comp rx=0 tx=0).stop 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb580af4b0 msgr2=0x7feb580af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb580af4b0 0x7feb580af930 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feb50001720 tx=0x7feb50001120 comp rx=0 tx=0).stop 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 shutdown_connections 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7feb3006a430 0x7feb3006c8f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb580af4b0 0x7feb580af930 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7feb580ade80 0x7feb580ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 --2- 172.21.15.39:0/2995156650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feb38005180 0x7feb581f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 >> 172.21.15.39:0/2995156650 conn(0x7feb58069e50 msgr2=0x7feb58159490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 shutdown_connections 2024-01-24T17:43:26.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.419+0000 7feb5f000640 1 -- 172.21.15.39:0/2995156650 wait complete. 2024-01-24T17:43:26.437 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:43:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.658+0000 7fb875000640 1 Processor -- start 2024-01-24T17:43:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.658+0000 7fb875000640 1 -- start start 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb875000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8701597f0 0x7fb87006d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb875000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8700b00e0 0x7fb8700b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb875000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8700b1bc0 con 0x7fb87006bfa0 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb875000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8700ca0d0 con 0x7fb8701597f0 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb875000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700b0a90 con 0x7fb8700b00e0 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb86de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8701597f0 0x7fb87006d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb86de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8701597f0 0x7fb87006d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55816/0 (socket says 172.21.15.39:55816) 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb86de00640 1 -- 172.21.15.39:0/1231648279 learned_addr learned my addr 172.21.15.39:0/1231648279 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.660+0000 7fb867e00640 1 --1- 172.21.15.39:0/1231648279 >> v1:172.21.15.39:6789/0 conn(0x7fb87006bfa0 0x7fb8700b14b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:51566/0 (socket says 172.21.15.39:51566) 2024-01-24T17:43:26.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.661+0000 7fb86e800640 1 --2- 172.21.15.39:0/1231648279 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8700b00e0 0x7fb8700b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.661+0000 7fb86de00640 1 -- 172.21.15.39:0/1231648279 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8700b00e0 msgr2=0x7fb8700b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:26.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.661+0000 7fb86de00640 1 --2- 172.21.15.39:0/1231648279 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8700b00e0 0x7fb8700b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.661+0000 7fb86de00640 1 -- 172.21.15.39:0/1231648279 >> v1:172.21.15.39:6789/0 conn(0x7fb87006bfa0 legacy=0x7fb8700b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.661+0000 7fb86de00640 1 -- 172.21.15.39:0/1231648279 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87006c4d0 con 0x7fb8701597f0 2024-01-24T17:43:26.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.662+0000 7fb86de00640 1 --2- 172.21.15.39:0/1231648279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8701597f0 0x7fb87006d890 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb8600098a0 tx=0x7fb86002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e601f63cfd9622b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:26.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.662+0000 7fb86d400640 1 -- 172.21.15.39:0/1231648279 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb860030840 con 0x7fb8701597f0 2024-01-24T17:43:26.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb86d400640 1 -- 172.21.15.39:0/1231648279 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8600309a0 con 0x7fb8701597f0 2024-01-24T17:43:26.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb86d400640 1 -- 172.21.15.39:0/1231648279 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb860030d30 con 0x7fb8701597f0 2024-01-24T17:43:26.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb875000640 1 -- 172.21.15.39:0/1231648279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8701597f0 msgr2=0x7fb87006d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb875000640 1 --2- 172.21.15.39:0/1231648279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8701597f0 0x7fb87006d890 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb8600098a0 tx=0x7fb86002b980 comp rx=0 tx=0).stop 2024-01-24T17:43:26.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb875000640 1 -- 172.21.15.39:0/1231648279 shutdown_connections 2024-01-24T17:43:26.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb875000640 1 --2- 172.21.15.39:0/1231648279 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8700b00e0 0x7fb8700b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb875000640 1 --2- 172.21.15.39:0/1231648279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8701597f0 0x7fb87006d890 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb875000640 1 -- 172.21.15.39:0/1231648279 >> 172.21.15.39:0/1231648279 conn(0x7fb8700cb0e0 msgr2=0x7fb8700cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:26.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.663+0000 7fb875000640 1 -- 172.21.15.39:0/1231648279 shutdown_connections 2024-01-24T17:43:26.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.664+0000 7fb875000640 1 -- 172.21.15.39:0/1231648279 wait complete. 2024-01-24T17:43:26.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.665+0000 7fb875000640 1 Processor -- start 2024-01-24T17:43:26.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.665+0000 7fb875000640 1 -- start start 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.666+0000 7fb875000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb87006bfa0 0x7fb87014d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.666+0000 7fb875000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8700b00e0 0x7fb87014e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.666+0000 7fb875000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8701597f0 0x7fb8700c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.666+0000 7fb875000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700c7b00 con 0x7fb8700b00e0 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.666+0000 7fb875000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700c84d0 con 0x7fb87006bfa0 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.666+0000 7fb875000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700c6c70 con 0x7fb8701597f0 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.666+0000 7fb867e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8700b00e0 0x7fb87014e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb867e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8700b00e0 0x7fb87014e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55818/0 (socket says 172.21.15.39:55818) 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb867e00640 1 -- 172.21.15.39:0/1995329825 learned_addr learned my addr 172.21.15.39:0/1995329825 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb86de00640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb87006bfa0 0x7fb87014d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb86e800640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8701597f0 0x7fb8700c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb867e00640 1 -- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8701597f0 msgr2=0x7fb8700c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb867e00640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8701597f0 0x7fb8700c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb867e00640 1 -- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb87006bfa0 msgr2=0x7fb87014d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb867e00640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb87006bfa0 0x7fb87014d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb867e00640 1 -- 172.21.15.39:0/1995329825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8600094d0 con 0x7fb8700b00e0 2024-01-24T17:43:26.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb86de00640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb87006bfa0 0x7fb87014d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:26.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.667+0000 7fb867e00640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8700b00e0 0x7fb87014e150 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c410 tx=0x7fb85800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:26.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.668+0000 7fb866000640 1 -- 172.21.15.39:0/1995329825 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8580074e0 con 0x7fb8700b00e0 2024-01-24T17:43:26.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.668+0000 7fb866000640 1 -- 172.21.15.39:0/1995329825 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb858007640 con 0x7fb8700b00e0 2024-01-24T17:43:26.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.668+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8700c6ef0 con 0x7fb8700b00e0 2024-01-24T17:43:26.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.668+0000 7fb866000640 1 -- 172.21.15.39:0/1995329825 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8580049e0 con 0x7fb8700b00e0 2024-01-24T17:43:26.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.668+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8700c74b0 con 0x7fb8700b00e0 2024-01-24T17:43:26.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.669+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8700c7cb0 con 0x7fb8700b00e0 2024-01-24T17:43:26.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.670+0000 7fb866000640 1 -- 172.21.15.39:0/1995329825 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb858004b40 con 0x7fb8700b00e0 2024-01-24T17:43:26.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.671+0000 7fb866000640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb84806a460 0x7fb84806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:26.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.671+0000 7fb866000640 1 -- 172.21.15.39:0/1995329825 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb858088400 con 0x7fb8700b00e0 2024-01-24T17:43:26.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.671+0000 7fb86de00640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb84806a460 0x7fb84806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:26.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.672+0000 7fb86de00640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb84806a460 0x7fb84806c920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb860036000 tx=0x7fb86003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:26.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.675+0000 7fb866000640 1 -- 172.21.15.39:0/1995329825 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb8700c7cb0 con 0x7fb8700b00e0 2024-01-24T17:43:26.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.868+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8700c7cb0 con 0x7fb8700b00e0 2024-01-24T17:43:26.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.869+0000 7fb866000640 1 -- 172.21.15.39:0/1995329825 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb8700c7cb0 con 0x7fb8700b00e0 2024-01-24T17:43:26.870 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:43:26.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.871+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb84806a460 msgr2=0x7fb84806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.871+0000 7fb875000640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb84806a460 0x7fb84806c920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb860036000 tx=0x7fb86003b040 comp rx=0 tx=0).stop 2024-01-24T17:43:26.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.871+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8700b00e0 msgr2=0x7fb87014e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.871+0000 7fb875000640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8700b00e0 0x7fb87014e150 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c410 tx=0x7fb85800c8d0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.871+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 shutdown_connections 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.872+0000 7fb875000640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8701597f0 0x7fb8700c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.872+0000 7fb875000640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb84806a460 0x7fb84806c920 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.872+0000 7fb875000640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8700b00e0 0x7fb87014e150 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.872+0000 7fb875000640 1 --2- 172.21.15.39:0/1995329825 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb87006bfa0 0x7fb87014d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.872+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 >> 172.21.15.39:0/1995329825 conn(0x7fb8700cb0e0 msgr2=0x7fb8700af9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.872+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 shutdown_connections 2024-01-24T17:43:26.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:26.872+0000 7fb875000640 1 -- 172.21.15.39:0/1995329825 wait complete. 2024-01-24T17:43:26.892 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:43:27.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.115+0000 7f7721e00640 1 Processor -- start 2024-01-24T17:43:27.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.115+0000 7f7721e00640 1 -- start start 2024-01-24T17:43:27.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.117+0000 7f7721e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c1597f0 0x7f771c06d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.117+0000 7f7721e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f771c06bfa0 0x7f771c0b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.117+0000 7f7721e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c06c4d0 con 0x7f771c0b00e0 2024-01-24T17:43:27.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.117+0000 7f7721e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771c0ca0d0 con 0x7f771c06bfa0 2024-01-24T17:43:27.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.117+0000 7f7721e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c06c7b0 con 0x7f771c1597f0 2024-01-24T17:43:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.117+0000 7f7713e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f771c06bfa0 0x7f771c0b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.117+0000 7f7713e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f771c06bfa0 0x7f771c0b14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55822/0 (socket says 172.21.15.39:55822) 2024-01-24T17:43:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.117+0000 7f7713e00640 1 -- 172.21.15.39:0/1815722492 learned_addr learned my addr 172.21.15.39:0/1815722492 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.118+0000 7f771aa00640 1 --2- 172.21.15.39:0/1815722492 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c1597f0 0x7f771c06d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.118+0000 7f7713e00640 1 -- 172.21.15.39:0/1815722492 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c1597f0 msgr2=0x7f771c06d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.118+0000 7f7713e00640 1 --2- 172.21.15.39:0/1815722492 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c1597f0 0x7f771c06d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.118+0000 7f7713e00640 1 -- 172.21.15.39:0/1815722492 >> v1:172.21.15.100:6789/0 conn(0x7f771c0b00e0 legacy=0x7f771c0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.118+0000 7f7713e00640 1 -- 172.21.15.39:0/1815722492 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c06ca90 con 0x7f771c06bfa0 2024-01-24T17:43:27.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.119+0000 7f7713e00640 1 --2- 172.21.15.39:0/1815722492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c06bfa0 0x7f771c0b14b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f770400bb10 tx=0x7f770402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e406534f27993a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:27.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.119+0000 7f771a000640 1 -- 172.21.15.39:0/1815722492 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770402ee70 con 0x7f771c06bfa0 2024-01-24T17:43:27.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.119+0000 7f771a000640 1 -- 172.21.15.39:0/1815722492 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7704036440 con 0x7f771c06bfa0 2024-01-24T17:43:27.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.119+0000 7f771a000640 1 -- 172.21.15.39:0/1815722492 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77040367d0 con 0x7f771c06bfa0 2024-01-24T17:43:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.120+0000 7f7721e00640 1 -- 172.21.15.39:0/1815722492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c06bfa0 msgr2=0x7f771c0b14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.120+0000 7f7721e00640 1 --2- 172.21.15.39:0/1815722492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c06bfa0 0x7f771c0b14b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f770400bb10 tx=0x7f770402e0c0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.120+0000 7f7721e00640 1 -- 172.21.15.39:0/1815722492 shutdown_connections 2024-01-24T17:43:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.120+0000 7f7721e00640 1 --2- 172.21.15.39:0/1815722492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c06bfa0 0x7f771c0b14b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.120+0000 7f7721e00640 1 --2- 172.21.15.39:0/1815722492 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c1597f0 0x7f771c06d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.120+0000 7f7721e00640 1 -- 172.21.15.39:0/1815722492 >> 172.21.15.39:0/1815722492 conn(0x7f771c0cb0e0 msgr2=0x7f771c0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.120+0000 7f7721e00640 1 -- 172.21.15.39:0/1815722492 shutdown_connections 2024-01-24T17:43:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.120+0000 7f7721e00640 1 -- 172.21.15.39:0/1815722492 wait complete. 2024-01-24T17:43:27.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.122+0000 7f7721e00640 1 Processor -- start 2024-01-24T17:43:27.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.122+0000 7f7721e00640 1 -- start start 2024-01-24T17:43:27.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.123+0000 7f7721e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f771c06bfa0 0x7f771c1f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.123+0000 7f7721e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c0b00e0 0x7f771c0c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.123+0000 7f7721e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c1597f0 0x7f771c1f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.123+0000 7f7721e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1f3530 con 0x7f771c1597f0 2024-01-24T17:43:27.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.123+0000 7f7721e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1f6a40 con 0x7f771c06bfa0 2024-01-24T17:43:27.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.123+0000 7f7721e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1f2f70 con 0x7f771c0b00e0 2024-01-24T17:43:27.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.123+0000 7f771b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c1597f0 0x7f771c1f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.123+0000 7f771b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c1597f0 0x7f771c1f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55834/0 (socket says 172.21.15.39:55834) 2024-01-24T17:43:27.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f771b400640 1 -- 172.21.15.39:0/2256568750 learned_addr learned my addr 172.21.15.39:0/2256568750 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:27.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f771aa00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f771c06bfa0 0x7f771c1f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f7713e00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c0b00e0 0x7f771c0c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f771b400640 1 -- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c0b00e0 msgr2=0x7f771c0c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f771b400640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c0b00e0 0x7f771c0c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f771b400640 1 -- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f771c06bfa0 msgr2=0x7f771c1f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f771b400640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f771c06bfa0 0x7f771c1f6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f771b400640 1 -- 172.21.15.39:0/2256568750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770400b770 con 0x7f771c1597f0 2024-01-24T17:43:27.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.124+0000 7f771b400640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c1597f0 0x7f771c1f2600 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f771400c780 tx=0x7f771400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.125+0000 7f7713400640 1 -- 172.21.15.39:0/2256568750 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714004b60 con 0x7f771c1597f0 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.125+0000 7f7713400640 1 -- 172.21.15.39:0/2256568750 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771400ce90 con 0x7f771c1597f0 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.125+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c1f3250 con 0x7f771c1597f0 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.125+0000 7f7713400640 1 -- 172.21.15.39:0/2256568750 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771400a850 con 0x7f771c1597f0 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.125+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1f5fe0 con 0x7f771c1597f0 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.126+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c1f61d0 con 0x7f771c1597f0 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.127+0000 7f7713400640 1 -- 172.21.15.39:0/2256568750 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f771400a9b0 con 0x7f771c1597f0 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.128+0000 7f7713400640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f76f406a430 0x7f76f406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.128+0000 7f7713400640 1 -- 172.21.15.39:0/2256568750 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7714088dd0 con 0x7f771c1597f0 2024-01-24T17:43:27.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.129+0000 7f771aa00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f76f406a430 0x7f76f406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.130+0000 7f771aa00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f76f406a430 0x7f76f406c8f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f770c006fd0 tx=0x7f770c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:27.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.133+0000 7f7713400640 1 -- 172.21.15.39:0/2256568750 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7714054330 con 0x7f771c1597f0 2024-01-24T17:43:27.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.327+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f771c073ef0 con 0x7f771c1597f0 2024-01-24T17:43:27.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.328+0000 7f7713400640 1 -- 172.21.15.39:0/2256568750 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7714057950 con 0x7f771c1597f0 2024-01-24T17:43:27.331 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.330+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f76f406a430 msgr2=0x7f76f406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.330+0000 7f7721e00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f76f406a430 0x7f76f406c8f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f770c006fd0 tx=0x7f770c008040 comp rx=0 tx=0).stop 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.330+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c1597f0 msgr2=0x7f771c1f2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.330+0000 7f7721e00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c1597f0 0x7f771c1f2600 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f771400c780 tx=0x7f771400cc40 comp rx=0 tx=0).stop 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.331+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 shutdown_connections 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.331+0000 7f7721e00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f76f406a430 0x7f76f406c8f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.331+0000 7f7721e00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771c1597f0 0x7f771c1f2600 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.331+0000 7f7721e00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f771c0b00e0 0x7f771c0c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.331+0000 7f7721e00640 1 --2- 172.21.15.39:0/2256568750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f771c06bfa0 0x7f771c1f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.331+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 >> 172.21.15.39:0/2256568750 conn(0x7f771c0cb0e0 msgr2=0x7f771c0cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.331+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 shutdown_connections 2024-01-24T17:43:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.331+0000 7f7721e00640 1 -- 172.21.15.39:0/2256568750 wait complete. 2024-01-24T17:43:27.350 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:43:27.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.568+0000 7fab4d000640 1 Processor -- start 2024-01-24T17:43:27.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.568+0000 7fab4d000640 1 -- start start 2024-01-24T17:43:27.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.569+0000 7fab4d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fab480cbfe0 0x7fab480cc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab4d000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4815e1b0 con 0x7fab481598e0 2024-01-24T17:43:27.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab4d000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab480cb3c0 con 0x7fab480b0530 2024-01-24T17:43:27.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab4d000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab480cc9b0 con 0x7fab480cbfe0 2024-01-24T17:43:27.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab46a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fab480cbfe0 0x7fab480cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab47400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fab481598e0 0x7fab480cfce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:51572/0 (socket says 172.21.15.39:51572) 2024-01-24T17:43:27.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab47400640 1 -- 172.21.15.39:0/2774200985 learned_addr learned my addr 172.21.15.39:0/2774200985 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:27.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab46a00640 1 -- 172.21.15.39:0/2774200985 >> v1:172.21.15.100:6789/0 conn(0x7fab480b0530 legacy=0x7fab480b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:27.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab46a00640 1 -- 172.21.15.39:0/2774200985 >> v1:172.21.15.39:6789/0 conn(0x7fab481598e0 legacy=0x7fab480cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.571+0000 7fab46a00640 1 -- 172.21.15.39:0/2774200985 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4815d2c0 con 0x7fab480cbfe0 2024-01-24T17:43:27.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.570+0000 7fab46000640 1 -- 172.21.15.39:0/2774200985 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548854822 0 0) 0x7fab4815e1b0 con 0x7fab481598e0 2024-01-24T17:43:27.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.572+0000 7fab46a00640 1 --2- 172.21.15.39:0/2774200985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480cbfe0 0x7fab480cc470 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fab300097a0 tx=0x7fab3002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f6380a03da097d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:27.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.572+0000 7fab46000640 1 -- 172.21.15.39:0/2774200985 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3002e070 con 0x7fab480cbfe0 2024-01-24T17:43:27.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.572+0000 7fab46000640 1 -- 172.21.15.39:0/2774200985 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab300315a0 con 0x7fab480cbfe0 2024-01-24T17:43:27.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.572+0000 7fab46000640 1 -- 172.21.15.39:0/2774200985 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30031930 con 0x7fab480cbfe0 2024-01-24T17:43:27.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.573+0000 7fab4d000640 1 -- 172.21.15.39:0/2774200985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480cbfe0 msgr2=0x7fab480cc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.573+0000 7fab4d000640 1 --2- 172.21.15.39:0/2774200985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480cbfe0 0x7fab480cc470 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fab300097a0 tx=0x7fab3002b840 comp rx=0 tx=0).stop 2024-01-24T17:43:27.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.573+0000 7fab4d000640 1 -- 172.21.15.39:0/2774200985 shutdown_connections 2024-01-24T17:43:27.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.573+0000 7fab4d000640 1 --2- 172.21.15.39:0/2774200985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480cbfe0 0x7fab480cc470 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.573+0000 7fab4d000640 1 -- 172.21.15.39:0/2774200985 >> 172.21.15.39:0/2774200985 conn(0x7fab48098a40 msgr2=0x7fab480cf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:27.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.573+0000 7fab4d000640 1 -- 172.21.15.39:0/2774200985 shutdown_connections 2024-01-24T17:43:27.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.573+0000 7fab4d000640 1 -- 172.21.15.39:0/2774200985 wait complete. 2024-01-24T17:43:27.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.575+0000 7fab4d000640 1 Processor -- start 2024-01-24T17:43:27.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.575+0000 7fab4d000640 1 -- start start 2024-01-24T17:43:27.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab4d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480b0530 0x7fab48153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab4d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab480cbfe0 0x7fab48155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab4d000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab481598e0 0x7fab4815c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab4d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48154970 con 0x7fab480b0530 2024-01-24T17:43:27.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab4d000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48154c50 con 0x7fab481598e0 2024-01-24T17:43:27.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab4d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab48154f30 con 0x7fab480cbfe0 2024-01-24T17:43:27.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab47400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480b0530 0x7fab48153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab47400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480b0530 0x7fab48153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55852/0 (socket says 172.21.15.39:55852) 2024-01-24T17:43:27.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab47400640 1 -- 172.21.15.39:0/3802385461 learned_addr learned my addr 172.21.15.39:0/3802385461 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab47400640 1 -- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab480cbfe0 msgr2=0x7fab48155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab47400640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab480cbfe0 0x7fab48155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab47400640 1 -- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab481598e0 msgr2=0x7fab4815c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab47400640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab481598e0 0x7fab4815c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.576+0000 7fab47400640 1 -- 172.21.15.39:0/3802385461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30009400 con 0x7fab480b0530 2024-01-24T17:43:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.577+0000 7fab47400640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480b0530 0x7fab48153ed0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fab3800c430 tx=0x7fab3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:27.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.577+0000 7fab3f400640 1 -- 172.21.15.39:0/3802385461 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab380074e0 con 0x7fab480b0530 2024-01-24T17:43:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.577+0000 7fab3f400640 1 -- 172.21.15.39:0/3802385461 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab38007660 con 0x7fab480b0530 2024-01-24T17:43:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.577+0000 7fab3f400640 1 -- 172.21.15.39:0/3802385461 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38004a20 con 0x7fab480b0530 2024-01-24T17:43:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.577+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab48155210 con 0x7fab480b0530 2024-01-24T17:43:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.577+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab48155f50 con 0x7fab480b0530 2024-01-24T17:43:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.578+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab48155730 con 0x7fab480b0530 2024-01-24T17:43:27.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.579+0000 7fab3f400640 1 -- 172.21.15.39:0/3802385461 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fab38004b80 con 0x7fab480b0530 2024-01-24T17:43:27.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.580+0000 7fab3f400640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab2006a430 0x7fab2006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:27.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.580+0000 7fab3f400640 1 -- 172.21.15.39:0/3802385461 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fab380887b0 con 0x7fab480b0530 2024-01-24T17:43:27.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.581+0000 7fab46a00640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab2006a430 0x7fab2006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:27.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.582+0000 7fab46a00640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab2006a430 0x7fab2006c8f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fab30036000 tx=0x7fab30030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:27.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.585+0000 7fab3f400640 1 -- 172.21.15.39:0/3802385461 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fab48155730 con 0x7fab480b0530 2024-01-24T17:43:27.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.799+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fab48155730 con 0x7fab480b0530 2024-01-24T17:43:27.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.800+0000 7fab3f400640 1 -- 172.21.15.39:0/3802385461 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7fab48155730 con 0x7fab480b0530 2024-01-24T17:43:27.801 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:43:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab2006a430 msgr2=0x7fab2006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab2006a430 0x7fab2006c8f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fab30036000 tx=0x7fab30030000 comp rx=0 tx=0).stop 2024-01-24T17:43:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480b0530 msgr2=0x7fab48153ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480b0530 0x7fab48153ed0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fab3800c430 tx=0x7fab3800c8f0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 shutdown_connections 2024-01-24T17:43:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fab481598e0 0x7fab4815c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fab480cbfe0 0x7fab48155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fab2006a430 0x7fab2006c8f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 --2- 172.21.15.39:0/3802385461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab480b0530 0x7fab48153ed0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 >> 172.21.15.39:0/3802385461 conn(0x7fab48098a40 msgr2=0x7fab480cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 shutdown_connections 2024-01-24T17:43:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:27.803+0000 7fab4d000640 1 -- 172.21.15.39:0/3802385461 wait complete. 2024-01-24T17:43:27.821 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:43:27.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:27.821 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:43:27.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[43805]: fuse finished with error 0 and tester_r 0 2024-01-24T17:43:27.918 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:43:34.020 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:43:34.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:34.021 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:43:34.080 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:43:34.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.317+0000 7f3700400640 1 Processor -- start 2024-01-24T17:43:34.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.317+0000 7f3700400640 1 -- start start 2024-01-24T17:43:34.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.319+0000 7f3700400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36f80710e0 0x7f36f80ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:34.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.319+0000 7f3700400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80cad00 con 0x7f36f80b06d0 2024-01-24T17:43:34.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.319+0000 7f3700400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f8071650 con 0x7f36f8159620 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.319+0000 7f3700400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f8071930 con 0x7f36f80710e0 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.319+0000 7f36fd000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f36f80b06d0 0x7f36f80cc380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59822/0 (socket says 172.21.15.39:59822) 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.319+0000 7f36fd000640 1 -- 172.21.15.39:0/1556285894 learned_addr learned my addr 172.21.15.39:0/1556285894 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.319+0000 7f36fe400640 1 --2- 172.21.15.39:0/1556285894 >> v2:172.21.15.39:3300/0 conn(0x7f36f80710e0 0x7f36f80ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.320+0000 7f36f7e00640 1 -- 172.21.15.39:0/1556285894 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761017106 0 0) 0x7f36f80cad00 con 0x7f36f80b06d0 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.320+0000 7f36f7e00640 1 -- 172.21.15.39:0/1556285894 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d8005730 con 0x7f36f80b06d0 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.320+0000 7f36fe400640 1 -- 172.21.15.39:0/1556285894 >> v1:172.21.15.100:6789/0 conn(0x7f36f8159620 legacy=0x7f36f80affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.320+0000 7f36fe400640 1 -- 172.21.15.39:0/1556285894 >> v1:172.21.15.39:6789/0 conn(0x7f36f80b06d0 legacy=0x7f36f80cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:34.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.320+0000 7f36fe400640 1 -- 172.21.15.39:0/1556285894 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f8071c10 con 0x7f36f80710e0 2024-01-24T17:43:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.321+0000 7f36fe400640 1 --2- 172.21.15.39:0/1556285894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f80710e0 0x7f36f80ca7c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009dc0 tx=0x7f36f002bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144dfe85c7ca7b37 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.321+0000 7f36f7e00640 1 -- 172.21.15.39:0/1556285894 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f002ccd0 con 0x7f36f80710e0 2024-01-24T17:43:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.321+0000 7f36f7e00640 1 -- 172.21.15.39:0/1556285894 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f002ce30 con 0x7f36f80710e0 2024-01-24T17:43:34.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.321+0000 7f36f7e00640 1 -- 172.21.15.39:0/1556285894 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0034660 con 0x7f36f80710e0 2024-01-24T17:43:34.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.322+0000 7f3700400640 1 -- 172.21.15.39:0/1556285894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f80710e0 msgr2=0x7f36f80ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:34.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.322+0000 7f3700400640 1 --2- 172.21.15.39:0/1556285894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f80710e0 0x7f36f80ca7c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009dc0 tx=0x7f36f002bf20 comp rx=0 tx=0).stop 2024-01-24T17:43:34.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.322+0000 7f3700400640 1 -- 172.21.15.39:0/1556285894 shutdown_connections 2024-01-24T17:43:34.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.322+0000 7f3700400640 1 --2- 172.21.15.39:0/1556285894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f80710e0 0x7f36f80ca7c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:34.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.322+0000 7f3700400640 1 -- 172.21.15.39:0/1556285894 >> 172.21.15.39:0/1556285894 conn(0x7f36f806a3e0 msgr2=0x7f36f806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:34.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.322+0000 7f3700400640 1 -- 172.21.15.39:0/1556285894 shutdown_connections 2024-01-24T17:43:34.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.322+0000 7f3700400640 1 -- 172.21.15.39:0/1556285894 wait complete. 2024-01-24T17:43:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.324+0000 7f3700400640 1 Processor -- start 2024-01-24T17:43:34.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.324+0000 7f3700400640 1 -- start start 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.325+0000 7f3700400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36f80710e0 0x7f36f80c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.325+0000 7f3700400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36f80b06d0 0x7f36f80cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.325+0000 7f3700400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f8159620 0x7f36f814f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.325+0000 7f3700400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f81521b0 con 0x7f36f8159620 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.325+0000 7f3700400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80c2530 con 0x7f36f80710e0 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.325+0000 7f3700400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80c2810 con 0x7f36f80b06d0 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.325+0000 7f36fe400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f8159620 0x7f36f814f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fe400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f8159620 0x7f36f814f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51784/0 (socket says 172.21.15.39:51784) 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fe400640 1 -- 172.21.15.39:0/1685733484 learned_addr learned my addr 172.21.15.39:0/1685733484 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fda00640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36f80710e0 0x7f36f80c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fd000640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36f80b06d0 0x7f36f80cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:34.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fe400640 1 -- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36f80b06d0 msgr2=0x7f36f80cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:34.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fe400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36f80b06d0 0x7f36f80cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:34.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fe400640 1 -- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36f80710e0 msgr2=0x7f36f80c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:34.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fe400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36f80710e0 0x7f36f80c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:34.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fe400640 1 -- 172.21.15.39:0/1685733484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f0009a70 con 0x7f36f8159620 2024-01-24T17:43:34.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.326+0000 7f36fe400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f8159620 0x7f36f814f200 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f36f002dfd0 tx=0x7f36f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:34.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.327+0000 7f36f5600640 1 -- 172.21.15.39:0/1685733484 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f003d600 con 0x7f36f8159620 2024-01-24T17:43:34.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.327+0000 7f36f5600640 1 -- 172.21.15.39:0/1685733484 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f003d760 con 0x7f36f8159620 2024-01-24T17:43:34.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.327+0000 7f36f5600640 1 -- 172.21.15.39:0/1685733484 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f003daf0 con 0x7f36f8159620 2024-01-24T17:43:34.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.327+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f80c2af0 con 0x7f36f8159620 2024-01-24T17:43:34.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.327+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f80c30b0 con 0x7f36f8159620 2024-01-24T17:43:34.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.328+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f8073ef0 con 0x7f36f8159620 2024-01-24T17:43:34.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.329+0000 7f36f5600640 1 -- 172.21.15.39:0/1685733484 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f36f003b040 con 0x7f36f8159620 2024-01-24T17:43:34.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.330+0000 7f36f5600640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36d006a430 0x7f36d006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:34.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.330+0000 7f36f5600640 1 -- 172.21.15.39:0/1685733484 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f36f00aaf00 con 0x7f36f8159620 2024-01-24T17:43:34.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.330+0000 7f36fda00640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36d006a430 0x7f36d006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:34.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.331+0000 7f36fda00640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36d006a430 0x7f36d006c8f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f36e8006fd0 tx=0x7f36e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:34.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.334+0000 7f36f5600640 1 -- 172.21.15.39:0/1685733484 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f36f0076460 con 0x7f36f8159620 2024-01-24T17:43:34.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.568+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36f80b3860 con 0x7f36f8159620 2024-01-24T17:43:34.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.569+0000 7f36f5600640 1 -- 172.21.15.39:0/1685733484 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f36f0039090 con 0x7f36f8159620 2024-01-24T17:43:34.570 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:34.570 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4688,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/648542221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":648542221},{"type":"v1","addr":"172.21.15.100:6837","nonce":648542221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/188830546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":188830546},{"type":"v1","addr":"172.21.15.100:6835","nonce":188830546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/4294097622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4294097622},{"type":"v1","addr":"172.21.15.39:6834","nonce":4294097622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:42:56.701344+0000","modified":"2024-01-24T17:43:00.247932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5175":{"gid":5175,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/3577904963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3577904963},{"type":"v1","addr":"172.21.15.39:6835","nonce":3577904963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-24T17:43:34.570 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36d006a430 msgr2=0x7f36d006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36d006a430 0x7f36d006c8f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f36e8006fd0 tx=0x7f36e8008040 comp rx=0 tx=0).stop 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f8159620 msgr2=0x7f36f814f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f8159620 0x7f36f814f200 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f36f002dfd0 tx=0x7f36f0004a40 comp rx=0 tx=0).stop 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 shutdown_connections 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36d006a430 0x7f36d006c8f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36f8159620 0x7f36f814f200 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36f80b06d0 0x7f36f80cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 --2- 172.21.15.39:0/1685733484 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36f80710e0 0x7f36f80c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:34.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.572+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 >> 172.21.15.39:0/1685733484 conn(0x7f36f806a3e0 msgr2=0x7f36f80c96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:34.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.573+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 shutdown_connections 2024-01-24T17:43:34.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:34.573+0000 7f3700400640 1 -- 172.21.15.39:0/1685733484 wait complete. 2024-01-24T17:43:34.594 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-01-24T17:43:34.835 INFO:teuthology.orchestra.run.smithi039.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":88,"msgr_send_messages":74,"msgr_recv_bytes":120829,"msgr_send_bytes":37311,"msgr_created_connections":9,"msgr_active_connections":1,"msgr_running_total_time":0.033340265,"msgr_running_send_time":0.005959458,"msgr_running_recv_time":0.090596842,"msgr_running_fast_dispatch_time":0.001506181,"msgr_send_messages_queue_lat":{"avgcount":72,"sum":0.055989981,"avgtime":0.000777638},"msgr_handle_ack_lat":{"avgcount":25,"sum":0.000005237,"avgtime":0.000000209},"msgr_recv_encrypted_bytes":113760,"msgr_send_encrypted_bytes":15920},"AsyncMessenger::Worker-1":{"msgr_recv_messages":45,"msgr_send_messages":38,"msgr_recv_bytes":15034,"msgr_send_bytes":86797,"msgr_created_connections":7,"msgr_active_connections":3,"msgr_running_total_time":0.027626371,"msgr_running_send_time":0.003656777,"msgr_running_recv_time":0.021545838,"msgr_running_fast_dispatch_time":0.001980055,"msgr_send_messages_queue_lat":{"avgcount":36,"sum":0.245679905,"avgtime":0.006824441},"msgr_handle_ack_lat":{"avgcount":27,"sum":0.000006643,"avgtime":0.000000246},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":72192},"AsyncMessenger::Worker-2":{"msgr_recv_messages":28,"msgr_send_messages":31,"msgr_recv_bytes":7424,"msgr_send_bytes":14553,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":0.007434687,"msgr_running_send_time":0.000784876,"msgr_running_recv_time":0.005386954,"msgr_running_fast_dispatch_time":0.000881847,"msgr_send_messages_queue_lat":{"avgcount":29,"sum":0.123374424,"avgtime":0.004254290},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":51,"sum":0.003039696,"avgtime":0.000059601}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2,"sum":0.000019244,"avgtime":0.000009622}},"mds":{"request":3,"reply":3,"reply_latency":{"avgcount":3,"sum":0.001225357,"avgtime":0.000408452},"slow_reply":0,"forward":0,"dir_fetch_complete":0,"dir_fetch_keys":0,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":0,"inodes_bottom":10,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":0,"caps":0,"subtrees":2,"traverse":5,"traverse_hit":5,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":0,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":0,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":6,"evex":0,"evtrm":0,"ev":6,"evexg":0,"evexd":0,"segadd":1,"segex":0,"segtrm":0,"seg":1,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4197177,"rdpos":0,"jlat":{"avgcount":6,"sum":0.025756309,"avgtime":0.004292718},"replayed":0},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":0,"cap+":2,"cap-":2,"rss":39424,"heap":207196},"mds_server":{"dispatch_client_request":3,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":8,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":2,"sum":0.000824563,"avgtime":0.000412281},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":1,"sum":0.000400794,"avgtime":0.000400794},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":0,"session_add":2,"session_remove":2,"sessions_open":0,"sessions_stale":0,"total_load":2,"average_load":1,"avg_session_uptime":13,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":213331,"buffer_anon_items":58,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5696,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51440,"mds_co_items":211,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":45,"op_send_bytes":14083,"op_resend":37,"op_reply":45,"op_latency":{"avgcount":45,"sum":1.292973935,"avgtime":0.028732754},"op_inflight":0,"oplen_avg":{"avgcount":45,"sum":69},"op":45,"op_r":0,"op_w":45,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":11,"osdop_read":0,"osdop_write":6,"osdop_writefull":8,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":22,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":36,"map_full":0,"map_inc":3,"osd_sessions":6,"osd_session_open":6,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":3},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":154,"get_sum":123846,"get_or_fail_fail":0,"get_or_fail_success":154,"take":0,"take_sum":0,"put":154,"put_sum":123846,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":45,"take_sum":14083,"put":45,"put_sum":14083,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":45,"take_sum":45,"put":45,"put_sum":45,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a2aa08a0a0":{"val":0,"max":3758096384,"get_started":0,"get":6,"get_sum":2873,"get_or_fail_fail":0,"get_or_fail_success":6,"take":0,"take_sum":0,"put":6,"put_sum":2873,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-01-24T17:43:34.836 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 25, "avgtime": 2.09e-07, "sum": 5.237e-06 }, "msgr_recv_bytes": 120829, "msgr_recv_encrypted_bytes": 113760, "msgr_recv_messages": 88, "msgr_running_fast_dispatch_time": 0.001506181, "msgr_running_recv_time": 0.090596842, "msgr_running_send_time": 0.005959458, "msgr_running_total_time": 0.033340265, "msgr_send_bytes": 37311, "msgr_send_encrypted_bytes": 15920, "msgr_send_messages": 74, "msgr_send_messages_queue_lat": { "avgcount": 72, "avgtime": 0.000777638, "sum": 0.055989981 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 27, "avgtime": 2.46e-07, "sum": 6.643e-06 }, "msgr_recv_bytes": 15034, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 45, "msgr_running_fast_dispatch_time": 0.001980055, "msgr_running_recv_time": 0.021545838, "msgr_running_send_time": 0.003656777, "msgr_running_total_time": 0.027626371, "msgr_send_bytes": 86797, "msgr_send_encrypted_bytes": 72192, "msgr_send_messages": 38, "msgr_send_messages_queue_lat": { "avgcount": 36, "avgtime": 0.006824441, "sum": 0.245679905 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 7424, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 28, "msgr_running_fast_dispatch_time": 0.000881847, "msgr_running_recv_time": 0.005386954, "msgr_running_send_time": 0.000784876, "msgr_running_total_time": 0.007434687, "msgr_send_bytes": 14553, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 31, "msgr_send_messages_queue_lat": { "avgcount": 29, "avgtime": 0.00425429, "sum": 0.123374424 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 51, "avgtime": 5.9601e-05, "sum": 0.003039696 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2, "avgtime": 9.622e-06, "sum": 1.9244e-05 }, "queue_len": 0 }, "mds": { "caps": 0, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 0, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 0, "dir_fetch_keys": 0, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 0, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 10, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 0, "inodes_with_caps": 0, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 3, "reply_latency": { "avgcount": 3, "avgtime": 0.000408452, "sum": 0.001225357 }, "request": 3, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6, "evadd": 6, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 6, "avgtime": 0.004292718, "sum": 0.025756309 }, "rdpos": 0, "replayed": 0, "seg": 1, "segadd": 1, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 4197177 }, "mds_mem": { "cap": 0, "cap+": 2, "cap-": 2, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 207196, "ino": 13, "ino+": 13, "ino-": 0, "rss": 39424 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 8, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000412281, "sum": 0.000824563 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 1, "avgtime": 0.000400794, "sum": 0.000400794 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 13, "mdthresh_evicted": 0, "session_add": 2, "session_count": 0, "session_remove": 2, "sessions_open": 0, "sessions_stale": 0, "total_load": 2 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 213331, "buffer_anon_items": 58, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51440, "mds_co_items": 211, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5696, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 36, "map_full": 0, "map_inc": 3, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 45, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 45, "avgtime": 0.028732754, "sum": 1.292973935 }, "op_pg": 0, "op_r": 0, "op_reply": 45, "op_resend": 37, "op_rmw": 0, "op_send": 45, "op_send_bytes": 14083, "op_w": 45, "oplen_avg": { "avgcount": 45, "sum": 69 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 6, "osd_sessions": 6, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 11, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 22, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 6, "osdop_writefull": 8, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 1, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 3 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 154, "get_or_fail_fail": 0, "get_or_fail_success": 154, "get_started": 0, "get_sum": 123846, "max": 104857600, "put": 154, "put_sum": 123846, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 45, "put_sum": 14083, "take": 45, "take_sum": 14083, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 45, "put_sum": 45, "take": 45, "take_sum": 45, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a2aa08a0a0": { "get": 6, "get_or_fail_fail": 0, "get_or_fail_success": 6, "get_started": 0, "get_sum": 2873, "max": 3758096384, "put": 6, "put_sum": 2873, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-01-24T17:43:34.837 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:43:35.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.062+0000 7ff321e00640 1 Processor -- start 2024-01-24T17:43:35.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.062+0000 7ff321e00640 1 -- start start 2024-01-24T17:43:35.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.063+0000 7ff321e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff31c15cbf0 0x7ff31c0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:35.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.063+0000 7ff321e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff31c15db60 0x7ff31c0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:35.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.063+0000 7ff321e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c15e010 con 0x7ff31c0d0da0 2024-01-24T17:43:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.063+0000 7ff321e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31c0d0a90 con 0x7ff31c15cbf0 2024-01-24T17:43:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.063+0000 7ff321e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31c0cd760 con 0x7ff31c15db60 2024-01-24T17:43:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff321400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff31c0d0da0 0x7ff31c15e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59836/0 (socket says 172.21.15.39:59836) 2024-01-24T17:43:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff321400640 1 -- 172.21.15.39:0/2695788021 learned_addr learned my addr 172.21.15.39:0/2695788021 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff320a00640 1 --2- 172.21.15.39:0/2695788021 >> v2:172.21.15.39:3300/0 conn(0x7ff31c15cbf0 0x7ff31c0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff31be00640 1 --2- 172.21.15.39:0/2695788021 >> v2:172.21.15.100:3300/0 conn(0x7ff31c15db60 0x7ff31c0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff320a00640 1 -- 172.21.15.39:0/2695788021 >> v2:172.21.15.100:3300/0 conn(0x7ff31c15db60 msgr2=0x7ff31c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:35.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff320a00640 1 --2- 172.21.15.39:0/2695788021 >> v2:172.21.15.100:3300/0 conn(0x7ff31c15db60 0x7ff31c0ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff320a00640 1 -- 172.21.15.39:0/2695788021 >> v1:172.21.15.39:6789/0 conn(0x7ff31c0d0da0 legacy=0x7ff31c15e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:35.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff320a00640 1 -- 172.21.15.39:0/2695788021 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c0cda40 con 0x7ff31c15cbf0 2024-01-24T17:43:35.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.064+0000 7ff31b400640 1 -- 172.21.15.39:0/2695788021 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088148096 0 0) 0x7ff31c15e010 con 0x7ff31c0d0da0 2024-01-24T17:43:35.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.066+0000 7ff320a00640 1 --2- 172.21.15.39:0/2695788021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c15cbf0 0x7ff31c0d03b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009790 tx=0x7ff30c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ceecfcf115f9fd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:35.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.066+0000 7ff31b400640 1 -- 172.21.15.39:0/2695788021 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c02d040 con 0x7ff31c15cbf0 2024-01-24T17:43:35.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.066+0000 7ff31b400640 1 -- 172.21.15.39:0/2695788021 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c031810 con 0x7ff31c15cbf0 2024-01-24T17:43:35.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.066+0000 7ff31b400640 1 -- 172.21.15.39:0/2695788021 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c031ba0 con 0x7ff31c15cbf0 2024-01-24T17:43:35.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.066+0000 7ff321e00640 1 -- 172.21.15.39:0/2695788021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c15cbf0 msgr2=0x7ff31c0d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:35.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.066+0000 7ff321e00640 1 --2- 172.21.15.39:0/2695788021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c15cbf0 0x7ff31c0d03b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff30c009790 tx=0x7ff30c02b8b0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.067+0000 7ff321e00640 1 -- 172.21.15.39:0/2695788021 shutdown_connections 2024-01-24T17:43:35.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.067+0000 7ff321e00640 1 --2- 172.21.15.39:0/2695788021 >> v2:172.21.15.100:3300/0 conn(0x7ff31c15db60 0x7ff31c0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.067+0000 7ff321e00640 1 --2- 172.21.15.39:0/2695788021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c15cbf0 0x7ff31c0d03b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.067+0000 7ff321e00640 1 -- 172.21.15.39:0/2695788021 >> 172.21.15.39:0/2695788021 conn(0x7ff31c09bb90 msgr2=0x7ff31c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:35.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.067+0000 7ff321e00640 1 -- 172.21.15.39:0/2695788021 shutdown_connections 2024-01-24T17:43:35.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.067+0000 7ff321e00640 1 -- 172.21.15.39:0/2695788021 wait complete. 2024-01-24T17:43:35.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.068+0000 7ff321e00640 1 Processor -- start 2024-01-24T17:43:35.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.069+0000 7ff321e00640 1 -- start start 2024-01-24T17:43:35.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.069+0000 7ff321e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c0d0da0 0x7ff31c0cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:35.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.069+0000 7ff321e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff31c15cbf0 0x7ff31c156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:35.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff321e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff31c15db60 0x7ff31c145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:35.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff321e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c157010 con 0x7ff31c0d0da0 2024-01-24T17:43:35.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff321e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c157220 con 0x7ff31c15cbf0 2024-01-24T17:43:35.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff321e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c157500 con 0x7ff31c15db60 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff320a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c0d0da0 0x7ff31c0cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff320a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c0d0da0 0x7ff31c0cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51812/0 (socket says 172.21.15.39:51812) 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff320a00640 1 -- 172.21.15.39:0/1300258979 learned_addr learned my addr 172.21.15.39:0/1300258979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff31be00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff31c15cbf0 0x7ff31c156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff321400640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff31c15db60 0x7ff31c145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff320a00640 1 -- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff31c15db60 msgr2=0x7ff31c145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff320a00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff31c15db60 0x7ff31c145910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff320a00640 1 -- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff31c15cbf0 msgr2=0x7ff31c156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff320a00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff31c15cbf0 0x7ff31c156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff320a00640 1 -- 172.21.15.39:0/1300258979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c0093a0 con 0x7ff31c0d0da0 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.070+0000 7ff321400640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff31c15db60 0x7ff31c145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:35.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.071+0000 7ff320a00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c0d0da0 0x7ff31c0cc490 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff30c000c00 tx=0x7ff30c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:35.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.071+0000 7ff318c00640 1 -- 172.21.15.39:0/1300258979 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c03ad60 con 0x7ff31c0d0da0 2024-01-24T17:43:35.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.071+0000 7ff318c00640 1 -- 172.21.15.39:0/1300258979 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c0443f0 con 0x7ff31c0d0da0 2024-01-24T17:43:35.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.071+0000 7ff318c00640 1 -- 172.21.15.39:0/1300258979 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30c0447a0 con 0x7ff31c0d0da0 2024-01-24T17:43:35.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.071+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c1577e0 con 0x7ff31c0d0da0 2024-01-24T17:43:35.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.071+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c157da0 con 0x7ff31c0d0da0 2024-01-24T17:43:35.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.072+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c157f90 con 0x7ff31c0d0da0 2024-01-24T17:43:35.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.073+0000 7ff318c00640 1 -- 172.21.15.39:0/1300258979 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff30c04d450 con 0x7ff31c0d0da0 2024-01-24T17:43:35.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.075+0000 7ff318c00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff2f406a430 0x7ff2f406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:35.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.075+0000 7ff318c00640 1 -- 172.21.15.39:0/1300258979 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff30c0aab80 con 0x7ff31c0d0da0 2024-01-24T17:43:35.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.075+0000 7ff31be00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff2f406a430 0x7ff2f406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:35.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.076+0000 7ff31be00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff2f406a430 0x7ff2f406c8f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff31c1f5e40 tx=0x7ff304008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:35.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.079+0000 7ff318c00640 1 -- 172.21.15.39:0/1300258979 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff30c076180 con 0x7ff31c0d0da0 2024-01-24T17:43:35.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.313+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff31c04dff0 con 0x7ff31c0d0da0 2024-01-24T17:43:35.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.314+0000 7ff318c00640 1 -- 172.21.15.39:0/1300258979 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff30c035070 con 0x7ff31c0d0da0 2024-01-24T17:43:35.315 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:43:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff2f406a430 msgr2=0x7ff2f406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff2f406a430 0x7ff2f406c8f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff31c1f5e40 tx=0x7ff304008040 comp rx=0 tx=0).stop 2024-01-24T17:43:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c0d0da0 msgr2=0x7ff31c0cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c0d0da0 0x7ff31c0cc490 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff30c000c00 tx=0x7ff30c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 shutdown_connections 2024-01-24T17:43:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff31c15db60 0x7ff31c145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff31c15cbf0 0x7ff31c156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff2f406a430 0x7ff2f406c8f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 --2- 172.21.15.39:0/1300258979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff31c0d0da0 0x7ff31c0cc490 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.316+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 >> 172.21.15.39:0/1300258979 conn(0x7ff31c09bb90 msgr2=0x7ff31c0d2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.317+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 shutdown_connections 2024-01-24T17:43:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:35.317+0000 7ff321e00640 1 -- 172.21.15.39:0/1300258979 wait complete. 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:43:35.338 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:43:35.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:35.339 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:43:35.393 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:43:35.393 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:35.394 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42516sec preferred_lft 42516sec 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:43:35.395 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:35.396 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:43:35.396 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:43:35.396 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:43:35.396 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:35.396 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:43:35.396 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:43:35.396 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:43:35.396 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:43:35.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:35.397 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:43:35.491 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:43:35.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:35.496 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:43:35.550 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:43:35.550 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:43:35.550 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:43:35.550 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:43:35.550 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:43:35.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:35.551 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:43:35.551 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:43:35.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:43:35.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:43:35.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:43:35.551 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:43:35.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:35.743 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:43:35.797 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:43:35.797 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:43:35.798 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:43:35.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:35.798 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:43:35.902 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:43:35.902 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:43:35.903 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:35.961 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:43:35.961 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:43:35.961 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:36.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:43:36.107 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:36.107 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:36.201 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:36.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:36.206 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:43:36.264 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:43:36.264 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:43:36.269 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:43:36.343 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:36.343 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:36.413 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:36.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:36.417 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:43:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:43:36.455+0000 7fde335ff480 -1 init, newargv = 0x55d1fedef650 newargc=15 2024-01-24T17:43:36.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[52228]: starting ceph client 2024-01-24T17:43:36.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[52228]: starting fuse 2024-01-24T17:43:37.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:43:37.668 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:43:37.668 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:43:37.769 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:43:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:43:37.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:37.775 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:43:37.831 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:43:37.831 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:43:37.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:43:37.831 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:43:37.831 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:43:37.831 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:43:37.831 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:43:37.831 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:43:37.832 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:43:37.833 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:43:37.833 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:43:37.833 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:43:37.833 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:43:37.833 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:43:38.020 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.52228.asok 2024-01-24T17:43:38.199 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52228.asok 2024-01-24T17:43:38.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:38.200 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52228.asok status 2024-01-24T17:43:38.443 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:43:38.443 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:43:38.443 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:43:38.443 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:43:38.443 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:43:38.443 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:43:38.443 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "52228", 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5538, 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5538 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:43:38.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3013974296 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3013974296 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.5538 192.168.0.1:0/3013974296", 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/3013974296", 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 15, 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 36, 2024-01-24T17:43:38.445 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:43:38.446 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:43:38.446 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:43:38.446 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:43:38.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:38.460 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:43:38.460 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:38.518 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:43:38.518 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:38.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:38.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:38.623 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-01-24T17:43:38.864 INFO:teuthology.orchestra.run.smithi039.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":95,"msgr_send_messages":79,"msgr_recv_bytes":122872,"msgr_send_bytes":38816,"msgr_created_connections":10,"msgr_active_connections":2,"msgr_running_total_time":0.035172146,"msgr_running_send_time":0.006496567,"msgr_running_recv_time":0.091595035,"msgr_running_fast_dispatch_time":0.001769620,"msgr_send_messages_queue_lat":{"avgcount":77,"sum":0.056228599,"avgtime":0.000730241},"msgr_handle_ack_lat":{"avgcount":35,"sum":0.000007681,"avgtime":0.000000219},"msgr_recv_encrypted_bytes":114032,"msgr_send_encrypted_bytes":16512},"AsyncMessenger::Worker-1":{"msgr_recv_messages":45,"msgr_send_messages":38,"msgr_recv_bytes":15034,"msgr_send_bytes":86797,"msgr_created_connections":7,"msgr_active_connections":3,"msgr_running_total_time":0.027626371,"msgr_running_send_time":0.003656777,"msgr_running_recv_time":0.021545838,"msgr_running_fast_dispatch_time":0.001980055,"msgr_send_messages_queue_lat":{"avgcount":36,"sum":0.245679905,"avgtime":0.006824441},"msgr_handle_ack_lat":{"avgcount":27,"sum":0.000006643,"avgtime":0.000000246},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":72192},"AsyncMessenger::Worker-2":{"msgr_recv_messages":30,"msgr_send_messages":33,"msgr_recv_bytes":7916,"msgr_send_bytes":15707,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":0.007603462,"msgr_running_send_time":0.000835283,"msgr_running_recv_time":0.005437944,"msgr_running_fast_dispatch_time":0.000941073,"msgr_send_messages_queue_lat":{"avgcount":31,"sum":0.123428787,"avgtime":0.003981573},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":54,"sum":0.003101454,"avgtime":0.000057434}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2,"sum":0.000019244,"avgtime":0.000009622}},"mds":{"request":4,"reply":4,"reply_latency":{"avgcount":4,"sum":0.001479525,"avgtime":0.000369881},"slow_reply":0,"forward":0,"dir_fetch_complete":0,"dir_fetch_keys":0,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":0,"inodes_bottom":10,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":7,"traverse_hit":7,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":0,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":0,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":7,"evex":0,"evtrm":0,"ev":7,"evexg":0,"evexd":0,"segadd":1,"segex":0,"segtrm":0,"seg":1,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4197619,"rdpos":0,"jlat":{"avgcount":7,"sum":0.030200254,"avgtime":0.004314322},"replayed":0},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":1,"cap+":3,"cap-":2,"rss":39808,"heap":207196},"mds_server":{"dispatch_client_request":4,"dispatch_server_request":0,"handle_client_request":4,"handle_client_session":11,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":3,"sum":0.001078731,"avgtime":0.000359577},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":1,"sum":0.000400794,"avgtime":0.000400794},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":3,"session_remove":2,"sessions_open":1,"sessions_stale":0,"total_load":3,"average_load":3,"avg_session_uptime":1,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":213659,"buffer_anon_items":62,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5696,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51800,"mds_co_items":212,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":47,"op_send_bytes":14615,"op_resend":37,"op_reply":47,"op_latency":{"avgcount":47,"sum":1.305973671,"avgtime":0.027786673},"op_inflight":0,"oplen_avg":{"avgcount":47,"sum":71},"op":47,"op_r":0,"op_w":47,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":11,"osdop_read":0,"osdop_write":7,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":22,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":36,"map_full":0,"map_inc":3,"osd_sessions":6,"osd_session_open":6,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":4},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":163,"get_sum":125519,"get_or_fail_fail":0,"get_or_fail_success":163,"take":0,"take_sum":0,"put":163,"put_sum":125519,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":47,"take_sum":14615,"put":47,"put_sum":14615,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":47,"take_sum":47,"put":47,"put_sum":47,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a2aa08a0a0":{"val":0,"max":3758096384,"get_started":0,"get":7,"get_sum":3315,"get_or_fail_fail":0,"get_or_fail_success":7,"take":0,"take_sum":0,"put":7,"put_sum":3315,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-01-24T17:43:38.866 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 2, "msgr_created_connections": 10, "msgr_handle_ack_lat": { "avgcount": 35, "avgtime": 2.19e-07, "sum": 7.681e-06 }, "msgr_recv_bytes": 122872, "msgr_recv_encrypted_bytes": 114032, "msgr_recv_messages": 95, "msgr_running_fast_dispatch_time": 0.00176962, "msgr_running_recv_time": 0.091595035, "msgr_running_send_time": 0.006496567, "msgr_running_total_time": 0.035172146, "msgr_send_bytes": 38816, "msgr_send_encrypted_bytes": 16512, "msgr_send_messages": 79, "msgr_send_messages_queue_lat": { "avgcount": 77, "avgtime": 0.000730241, "sum": 0.056228599 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 27, "avgtime": 2.46e-07, "sum": 6.643e-06 }, "msgr_recv_bytes": 15034, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 45, "msgr_running_fast_dispatch_time": 0.001980055, "msgr_running_recv_time": 0.021545838, "msgr_running_send_time": 0.003656777, "msgr_running_total_time": 0.027626371, "msgr_send_bytes": 86797, "msgr_send_encrypted_bytes": 72192, "msgr_send_messages": 38, "msgr_send_messages_queue_lat": { "avgcount": 36, "avgtime": 0.006824441, "sum": 0.245679905 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 7916, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 30, "msgr_running_fast_dispatch_time": 0.000941073, "msgr_running_recv_time": 0.005437944, "msgr_running_send_time": 0.000835283, "msgr_running_total_time": 0.007603462, "msgr_send_bytes": 15707, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 33, "msgr_send_messages_queue_lat": { "avgcount": 31, "avgtime": 0.003981573, "sum": 0.123428787 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 54, "avgtime": 5.7434e-05, "sum": 0.003101454 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2, "avgtime": 9.622e-06, "sum": 1.9244e-05 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 0, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 0, "dir_fetch_keys": 0, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 0, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 10, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 4, "reply_latency": { "avgcount": 4, "avgtime": 0.000369881, "sum": 0.001479525 }, "request": 4, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 7, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 7, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7, "evadd": 7, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 7, "avgtime": 0.004314322, "sum": 0.030200254 }, "rdpos": 0, "replayed": 0, "seg": 1, "segadd": 1, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 4197619 }, "mds_mem": { "cap": 1, "cap+": 3, "cap-": 2, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 207196, "ino": 13, "ino+": 13, "ino-": 0, "rss": 39808 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4, "dispatch_server_request": 0, "handle_client_request": 4, "handle_client_session": 11, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 3, "avgtime": 0.000359577, "sum": 0.001078731 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 1, "avgtime": 0.000400794, "sum": 0.000400794 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 3, "avg_session_uptime": 1, "mdthresh_evicted": 0, "session_add": 3, "session_count": 1, "session_remove": 2, "sessions_open": 1, "sessions_stale": 0, "total_load": 3 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 213659, "buffer_anon_items": 62, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51800, "mds_co_items": 212, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5696, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 36, "map_full": 0, "map_inc": 3, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 47, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 47, "avgtime": 0.027786673, "sum": 1.305973671 }, "op_pg": 0, "op_r": 0, "op_reply": 47, "op_resend": 37, "op_rmw": 0, "op_send": 47, "op_send_bytes": 14615, "op_w": 47, "oplen_avg": { "avgcount": 47, "sum": 71 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 6, "osd_sessions": 6, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 11, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 22, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 7, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 1, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 4 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 163, "get_or_fail_fail": 0, "get_or_fail_success": 163, "get_started": 0, "get_sum": 125519, "max": 104857600, "put": 163, "put_sum": 125519, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 47, "put_sum": 14615, "take": 47, "take_sum": 14615, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 47, "put_sum": 47, "take": 47, "take_sum": 47, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a2aa08a0a0": { "get": 7, "get_or_fail_fail": 0, "get_or_fail_success": 7, "get_started": 0, "get_sum": 3315, "max": 3758096384, "put": 7, "put_sum": 3315, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-01-24T17:43:38.866 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:43:38.866 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:38.919 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:39.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.137+0000 7f6f3d800640 1 Processor -- start 2024-01-24T17:43:39.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.138+0000 7f6f3d800640 1 -- start start 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.139+0000 7f6f3d800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6f380d03b0 0x7f6f3815db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.139+0000 7f6f3d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d0da0 0x7f6f380d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.139+0000 7f6f3d800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380d0860 con 0x7f6f3815cbf0 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.139+0000 7f6f3d800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f380d1770 con 0x7f6f380d03b0 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.139+0000 7f6f3d800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380d0a90 con 0x7f6f380d0da0 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f37400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6f380d03b0 0x7f6f3815db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f37400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6f380d03b0 0x7f6f3815db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51822/0 (socket says 172.21.15.39:51822) 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f37400640 1 -- 172.21.15.39:0/1847807053 learned_addr learned my addr 172.21.15.39:0/1847807053 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f3ce00640 1 --2- 172.21.15.39:0/1847807053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d0da0 0x7f6f380d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f37400640 1 -- 172.21.15.39:0/1847807053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d0da0 msgr2=0x7f6f380d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:39.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f37400640 1 --2- 172.21.15.39:0/1847807053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d0da0 0x7f6f380d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f37400640 1 -- 172.21.15.39:0/1847807053 >> v1:172.21.15.100:6789/0 conn(0x7f6f3815cbf0 legacy=0x7f6f380ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f37400640 1 -- 172.21.15.39:0/1847807053 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380cd760 con 0x7f6f380d03b0 2024-01-24T17:43:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.140+0000 7f6f3ce00640 1 --2- 172.21.15.39:0/1847807053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d0da0 0x7f6f380d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.141+0000 7f6f37400640 1 --2- 172.21.15.39:0/1847807053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d03b0 0x7f6f3815db60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6f200098a0 tx=0x7f6f2002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea00307b3ce0b6b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.142+0000 7f6f36a00640 1 -- 172.21.15.39:0/1847807053 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20031640 con 0x7f6f380d03b0 2024-01-24T17:43:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.142+0000 7f6f36a00640 1 -- 172.21.15.39:0/1847807053 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f200317a0 con 0x7f6f380d03b0 2024-01-24T17:43:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.142+0000 7f6f36a00640 1 -- 172.21.15.39:0/1847807053 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20031b30 con 0x7f6f380d03b0 2024-01-24T17:43:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.142+0000 7f6f3d800640 1 -- 172.21.15.39:0/1847807053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d03b0 msgr2=0x7f6f3815db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.142+0000 7f6f3d800640 1 --2- 172.21.15.39:0/1847807053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d03b0 0x7f6f3815db60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6f200098a0 tx=0x7f6f2002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.143+0000 7f6f3d800640 1 -- 172.21.15.39:0/1847807053 shutdown_connections 2024-01-24T17:43:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.143+0000 7f6f3d800640 1 --2- 172.21.15.39:0/1847807053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d03b0 0x7f6f3815db60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.143+0000 7f6f3d800640 1 --2- 172.21.15.39:0/1847807053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d0da0 0x7f6f380d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.143+0000 7f6f3d800640 1 -- 172.21.15.39:0/1847807053 >> 172.21.15.39:0/1847807053 conn(0x7f6f3809bb90 msgr2=0x7f6f380d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.143+0000 7f6f3d800640 1 -- 172.21.15.39:0/1847807053 shutdown_connections 2024-01-24T17:43:39.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.143+0000 7f6f3d800640 1 -- 172.21.15.39:0/1847807053 wait complete. 2024-01-24T17:43:39.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.144+0000 7f6f3d800640 1 Processor -- start 2024-01-24T17:43:39.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.145+0000 7f6f3d800640 1 -- start start 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.145+0000 7f6f3d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d03b0 0x7f6f380b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.145+0000 7f6f3d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d0da0 0x7f6f380c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.145+0000 7f6f3d800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f3815cbf0 0x7f6f381580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f3d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f381e5280 con 0x7f6f380d0da0 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f3d800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380ca460 con 0x7f6f3815cbf0 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f3d800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380ca740 con 0x7f6f380d03b0 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d0da0 0x7f6f380c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d0da0 0x7f6f380c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51824/0 (socket says 172.21.15.39:51824) 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37400640 1 -- 172.21.15.39:0/2416153612 learned_addr learned my addr 172.21.15.39:0/2416153612 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37e00640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d03b0 0x7f6f380b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f3ce00640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f3815cbf0 0x7f6f381580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37400640 1 -- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d03b0 msgr2=0x7f6f380b7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37400640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d03b0 0x7f6f380b7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37400640 1 -- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f3815cbf0 msgr2=0x7f6f381580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37400640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f3815cbf0 0x7f6f381580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f37400640 1 -- 172.21.15.39:0/2416153612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f200094d0 con 0x7f6f380d0da0 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.146+0000 7f6f3ce00640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f3815cbf0 0x7f6f381580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.147+0000 7f6f37400640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d0da0 0x7f6f380c7d10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6f20000c00 tx=0x7f6f200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.147+0000 7f6f2fe00640 1 -- 172.21.15.39:0/2416153612 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2003ac60 con 0x7f6f380d0da0 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.147+0000 7f6f2fe00640 1 -- 172.21.15.39:0/2416153612 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2003ade0 con 0x7f6f380d0da0 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.147+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f380caa20 con 0x7f6f380d0da0 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.147+0000 7f6f2fe00640 1 -- 172.21.15.39:0/2416153612 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20044630 con 0x7f6f380d0da0 2024-01-24T17:43:39.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.147+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f380cafe0 con 0x7f6f380d0da0 2024-01-24T17:43:39.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.148+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3804dff0 con 0x7f6f380d0da0 2024-01-24T17:43:39.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.149+0000 7f6f2fe00640 1 -- 172.21.15.39:0/2416153612 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6f2002f050 con 0x7f6f380d0da0 2024-01-24T17:43:39.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.150+0000 7f6f2fe00640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f1006a430 0x7f6f1006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.150+0000 7f6f2fe00640 1 -- 172.21.15.39:0/2416153612 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6f200aab00 con 0x7f6f380d0da0 2024-01-24T17:43:39.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.151+0000 7f6f37e00640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f1006a430 0x7f6f1006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.152+0000 7f6f37e00640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f1006a430 0x7f6f1006c8f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6f28004e50 tx=0x7f6f28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:39.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.154+0000 7f6f2fe00640 1 -- 172.21.15.39:0/2416153612 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6f200ace60 con 0x7f6f380d0da0 2024-01-24T17:43:39.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.350+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f380cdcf0 con 0x7f6f380d0da0 2024-01-24T17:43:39.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.353+0000 7f6f2fe00640 1 -- 172.21.15.39:0/2416153612 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16290 (secure 0 0 0) 0x7f6f20035090 con 0x7f6f380d0da0 2024-01-24T17:43:39.354 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:39.354 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:43:16.261174+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.1c","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.32","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:39.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.355+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f1006a430 msgr2=0x7f6f1006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.355+0000 7f6f3d800640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f1006a430 0x7f6f1006c8f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6f28004e50 tx=0x7f6f28009040 comp rx=0 tx=0).stop 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d0da0 msgr2=0x7f6f380c7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d0da0 0x7f6f380c7d10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6f20000c00 tx=0x7f6f200046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 shutdown_connections 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f3815cbf0 0x7f6f381580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f1006a430 0x7f6f1006c8f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f380d0da0 0x7f6f380c7d10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 --2- 172.21.15.39:0/2416153612 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6f380d03b0 0x7f6f380b7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 >> 172.21.15.39:0/2416153612 conn(0x7f6f3809bb90 msgr2=0x7f6f380d3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 shutdown_connections 2024-01-24T17:43:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.356+0000 7f6f3d800640 1 -- 172.21.15.39:0/2416153612 wait complete. 2024-01-24T17:43:39.375 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:42:54.883732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:42:55.727240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:43:39.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:43:39.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.586+0000 7f7e97000640 1 Processor -- start 2024-01-24T17:43:39.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.586+0000 7f7e97000640 1 -- start start 2024-01-24T17:43:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.588+0000 7f7e97000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7e9015ccb0 0x7f7e900cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.588+0000 7f7e97000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.588+0000 7f7e97000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9015d090 con 0x7f7e900ce390 2024-01-24T17:43:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.588+0000 7f7e97000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e900cac20 con 0x7f7e9015ccb0 2024-01-24T17:43:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.588+0000 7f7e97000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900ced60 con 0x7f7e900c9d60 2024-01-24T17:43:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.588+0000 7f7e95c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7e9015ccb0 0x7f7e900cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.588+0000 7f7e95c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7e9015ccb0 0x7f7e900cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51834/0 (socket says 172.21.15.39:51834) 2024-01-24T17:43:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.588+0000 7f7e95c00640 1 -- 172.21.15.39:0/2281114415 learned_addr learned my addr 172.21.15.39:0/2281114415 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:39.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.589+0000 7f7e95c00640 1 -- 172.21.15.39:0/2281114415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 msgr2=0x7f7e900ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:39.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.589+0000 7f7e96600640 1 --2- 172.21.15.39:0/2281114415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.589+0000 7f7e95c00640 1 --2- 172.21.15.39:0/2281114415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.589+0000 7f7e95c00640 1 -- 172.21.15.39:0/2281114415 >> v1:172.21.15.100:6789/0 conn(0x7f7e900ce390 legacy=0x7f7e900cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:39.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.589+0000 7f7e95c00640 1 -- 172.21.15.39:0/2281114415 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9006c540 con 0x7f7e9015ccb0 2024-01-24T17:43:39.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.589+0000 7f7e96600640 1 --2- 172.21.15.39:0/2281114415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:39.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.590+0000 7f7e95c00640 1 --2- 172.21.15.39:0/2281114415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e900cda90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009850 tx=0x7f7e8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db8effcfa4498be server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:39.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.590+0000 7f7e8fe00640 1 -- 172.21.15.39:0/2281114415 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e80004230 con 0x7f7e9015ccb0 2024-01-24T17:43:39.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.590+0000 7f7e8fe00640 1 -- 172.21.15.39:0/2281114415 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e80004390 con 0x7f7e9015ccb0 2024-01-24T17:43:39.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.590+0000 7f7e8fe00640 1 -- 172.21.15.39:0/2281114415 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e80031830 con 0x7f7e9015ccb0 2024-01-24T17:43:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.591+0000 7f7e97000640 1 -- 172.21.15.39:0/2281114415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 msgr2=0x7f7e900cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.591+0000 7f7e97000640 1 --2- 172.21.15.39:0/2281114415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e900cda90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009850 tx=0x7f7e8002b8d0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.591+0000 7f7e97000640 1 -- 172.21.15.39:0/2281114415 shutdown_connections 2024-01-24T17:43:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.591+0000 7f7e97000640 1 --2- 172.21.15.39:0/2281114415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.591+0000 7f7e97000640 1 --2- 172.21.15.39:0/2281114415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e900cda90 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.591+0000 7f7e97000640 1 -- 172.21.15.39:0/2281114415 >> 172.21.15.39:0/2281114415 conn(0x7f7e90069e50 msgr2=0x7f7e9006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.591+0000 7f7e97000640 1 -- 172.21.15.39:0/2281114415 shutdown_connections 2024-01-24T17:43:39.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.591+0000 7f7e97000640 1 -- 172.21.15.39:0/2281114415 wait complete. 2024-01-24T17:43:39.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.593+0000 7f7e97000640 1 Processor -- start 2024-01-24T17:43:39.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.593+0000 7f7e97000640 1 -- start start 2024-01-24T17:43:39.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e97000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e97000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e900ce390 0x7f7e900c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e97000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e901f1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e97000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900aba30 con 0x7f7e9015ccb0 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e97000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900c2590 con 0x7f7e900ce390 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e97000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900c2870 con 0x7f7e900c9d60 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e96600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e901f1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e96600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e901f1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51844/0 (socket says 172.21.15.39:51844) 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.594+0000 7f7e96600640 1 -- 172.21.15.39:0/2573010098 learned_addr learned my addr 172.21.15.39:0/2573010098 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.595+0000 7f7e95c00640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.595+0000 7f7e95200640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e900ce390 0x7f7e900c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.595+0000 7f7e96600640 1 -- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 msgr2=0x7f7e900c1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.595+0000 7f7e96600640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900c1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.595+0000 7f7e96600640 1 -- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e900ce390 msgr2=0x7f7e900c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.595+0000 7f7e96600640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e900ce390 0x7f7e900c30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.595+0000 7f7e96600640 1 -- 172.21.15.39:0/2573010098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80009460 con 0x7f7e9015ccb0 2024-01-24T17:43:39.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.595+0000 7f7e96600640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e901f1400 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800cea0 tx=0x7f7e8800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:39.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.596+0000 7f7e8d600640 1 -- 172.21.15.39:0/2573010098 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88003b90 con 0x7f7e9015ccb0 2024-01-24T17:43:39.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.596+0000 7f7e8d600640 1 -- 172.21.15.39:0/2573010098 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e88003d10 con 0x7f7e9015ccb0 2024-01-24T17:43:39.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.596+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e900c2b50 con 0x7f7e9015ccb0 2024-01-24T17:43:39.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.596+0000 7f7e8d600640 1 -- 172.21.15.39:0/2573010098 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88004260 con 0x7f7e9015ccb0 2024-01-24T17:43:39.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.596+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e900c2280 con 0x7f7e9015ccb0 2024-01-24T17:43:39.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.597+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e900c2d40 con 0x7f7e9015ccb0 2024-01-24T17:43:39.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.598+0000 7f7e8d600640 1 -- 172.21.15.39:0/2573010098 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7e880043c0 con 0x7f7e9015ccb0 2024-01-24T17:43:39.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.599+0000 7f7e8d600640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e6806a430 0x7f7e6806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:39.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.599+0000 7f7e8d600640 1 -- 172.21.15.39:0/2573010098 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7e88089400 con 0x7f7e9015ccb0 2024-01-24T17:43:39.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.599+0000 7f7e95c00640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e6806a430 0x7f7e6806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:39.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.601+0000 7f7e95c00640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e6806a430 0x7f7e6806c8f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7e80035000 tx=0x7f7e80008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:39.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.604+0000 7f7e8d600640 1 -- 172.21.15.39:0/2573010098 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7e88054850 con 0x7f7e9015ccb0 2024-01-24T17:43:39.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.799+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e900abec0 con 0x7f7e9015ccb0 2024-01-24T17:43:39.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.800+0000 7f7e8d600640 1 -- 172.21.15.39:0/2573010098 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e900abec0 con 0x7f7e9015ccb0 2024-01-24T17:43:39.800 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:43:39.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.802+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e6806a430 msgr2=0x7f7e6806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.802+0000 7f7e97000640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e6806a430 0x7f7e6806c8f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7e80035000 tx=0x7f7e80008030 comp rx=0 tx=0).stop 2024-01-24T17:43:39.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.802+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 msgr2=0x7f7e901f1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:39.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.802+0000 7f7e97000640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e901f1400 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800cea0 tx=0x7f7e8800cf90 comp rx=0 tx=0).stop 2024-01-24T17:43:39.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.803+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 shutdown_connections 2024-01-24T17:43:39.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.803+0000 7f7e97000640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e6806a430 0x7f7e6806c8f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.803+0000 7f7e97000640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e9015ccb0 0x7f7e901f1400 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.803+0000 7f7e97000640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e900ce390 0x7f7e900c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.803+0000 7f7e97000640 1 --2- 172.21.15.39:0/2573010098 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e900c9d60 0x7f7e900c1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:39.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.803+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 >> 172.21.15.39:0/2573010098 conn(0x7f7e90069e50 msgr2=0x7f7e900ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:39.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.803+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 shutdown_connections 2024-01-24T17:43:39.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:39.803+0000 7f7e97000640 1 -- 172.21.15.39:0/2573010098 wait complete. 2024-01-24T17:43:39.821 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:43:40.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.038+0000 7efcc3e00640 1 Processor -- start 2024-01-24T17:43:40.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.038+0000 7efcc3e00640 1 -- start start 2024-01-24T17:43:40.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.040+0000 7efcc3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efcc415c8d0 0x7efcc40d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.040+0000 7efcc3e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efcc40b2710 0x7efcc40d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.040+0000 7efcc3e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc40b38a0 con 0x7efcc40d2cb0 2024-01-24T17:43:40.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.040+0000 7efcc3e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc40d3100 con 0x7efcc415c8d0 2024-01-24T17:43:40.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.040+0000 7efcc3e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc40d13e0 con 0x7efcc40b2710 2024-01-24T17:43:40.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.040+0000 7efcc2a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efcc415c8d0 0x7efcc40d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.040+0000 7efcc3400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efcc40d2cb0 0x7efcc40cf160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59848/0 (socket says 172.21.15.39:59848) 2024-01-24T17:43:40.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.040+0000 7efcc3400640 1 -- 172.21.15.39:0/1740559775 learned_addr learned my addr 172.21.15.39:0/1740559775 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:40.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.041+0000 7efcc2000640 1 --2- 172.21.15.39:0/1740559775 >> v2:172.21.15.100:3300/0 conn(0x7efcc40b2710 0x7efcc40d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.041+0000 7efcc2a00640 1 -- 172.21.15.39:0/1740559775 >> v2:172.21.15.100:3300/0 conn(0x7efcc40b2710 msgr2=0x7efcc40d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:40.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.041+0000 7efcc2a00640 1 --2- 172.21.15.39:0/1740559775 >> v2:172.21.15.100:3300/0 conn(0x7efcc40b2710 0x7efcc40d2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.041+0000 7efcc2a00640 1 -- 172.21.15.39:0/1740559775 >> v1:172.21.15.39:6789/0 conn(0x7efcc40d2cb0 legacy=0x7efcc40cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.041+0000 7efcc2a00640 1 -- 172.21.15.39:0/1740559775 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc40d0ab0 con 0x7efcc415c8d0 2024-01-24T17:43:40.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.041+0000 7efcc1600640 1 -- 172.21.15.39:0/1740559775 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829836987 0 0) 0x7efcc40b38a0 con 0x7efcc40d2cb0 2024-01-24T17:43:40.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.041+0000 7efcc2000640 1 --2- 172.21.15.39:0/1740559775 >> v2:172.21.15.100:3300/0 conn(0x7efcc40b2710 0x7efcc40d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:40.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.043+0000 7efcc2a00640 1 --2- 172.21.15.39:0/1740559775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc415c8d0 0x7efcc40d03b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efcc40cd0f0 tx=0x7efcb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7c446850edf3b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.043+0000 7efcc1600640 1 -- 172.21.15.39:0/1740559775 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4031830 con 0x7efcc415c8d0 2024-01-24T17:43:40.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.043+0000 7efcc1600640 1 -- 172.21.15.39:0/1740559775 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb4031990 con 0x7efcc415c8d0 2024-01-24T17:43:40.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.043+0000 7efcc1600640 1 -- 172.21.15.39:0/1740559775 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4031d20 con 0x7efcc415c8d0 2024-01-24T17:43:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.044+0000 7efcc3e00640 1 -- 172.21.15.39:0/1740559775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc415c8d0 msgr2=0x7efcc40d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.044+0000 7efcc3e00640 1 --2- 172.21.15.39:0/1740559775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc415c8d0 0x7efcc40d03b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efcc40cd0f0 tx=0x7efcb402b8d0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.044+0000 7efcc3e00640 1 -- 172.21.15.39:0/1740559775 shutdown_connections 2024-01-24T17:43:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.044+0000 7efcc3e00640 1 --2- 172.21.15.39:0/1740559775 >> v2:172.21.15.100:3300/0 conn(0x7efcc40b2710 0x7efcc40d2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.044+0000 7efcc3e00640 1 --2- 172.21.15.39:0/1740559775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc415c8d0 0x7efcc40d03b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.044+0000 7efcc3e00640 1 -- 172.21.15.39:0/1740559775 >> 172.21.15.39:0/1740559775 conn(0x7efcc4071dd0 msgr2=0x7efcc409bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.044+0000 7efcc3e00640 1 -- 172.21.15.39:0/1740559775 shutdown_connections 2024-01-24T17:43:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.044+0000 7efcc3e00640 1 -- 172.21.15.39:0/1740559775 wait complete. 2024-01-24T17:43:40.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.046+0000 7efcc3e00640 1 Processor -- start 2024-01-24T17:43:40.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.046+0000 7efcc3e00640 1 -- start start 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc40b2710 0x7efcc4146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc40d2cb0 0x7efcc406bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc3e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efcc415c8d0 0x7efcc406c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40b0d30 con 0x7efcc40b2710 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc3e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40b83f0 con 0x7efcc415c8d0 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc3e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40b7e30 con 0x7efcc40d2cb0 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc40b2710 0x7efcc4146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc40b2710 0x7efcc4146500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51874/0 (socket says 172.21.15.39:51874) 2024-01-24T17:43:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.047+0000 7efcc2a00640 1 -- 172.21.15.39:0/691003492 learned_addr learned my addr 172.21.15.39:0/691003492 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcc2a00640 1 -- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc40d2cb0 msgr2=0x7efcc406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcc2a00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc40d2cb0 0x7efcc406bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcc2a00640 1 -- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efcc415c8d0 msgr2=0x7efcc406c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcc2a00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efcc415c8d0 0x7efcc406c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcc2a00640 1 -- 172.21.15.39:0/691003492 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb4009400 con 0x7efcc40b2710 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcc2a00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc40b2710 0x7efcc4146500 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009890 tx=0x7efcb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcbaa00640 1 -- 172.21.15.39:0/691003492 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb403ba70 con 0x7efcc40b2710 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcbaa00640 1 -- 172.21.15.39:0/691003492 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb403bbf0 con 0x7efcc40b2710 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcbaa00640 1 -- 172.21.15.39:0/691003492 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4045400 con 0x7efcc40b2710 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.048+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc40b8110 con 0x7efcc40b2710 2024-01-24T17:43:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.049+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc4147290 con 0x7efcc40b2710 2024-01-24T17:43:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.050+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc404dff0 con 0x7efcc40b2710 2024-01-24T17:43:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.050+0000 7efcbaa00640 1 -- 172.21.15.39:0/691003492 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efcb404e440 con 0x7efcc40b2710 2024-01-24T17:43:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.051+0000 7efcbaa00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9c06a430 0x7efc9c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.051+0000 7efcbaa00640 1 -- 172.21.15.39:0/691003492 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efcb40aadf0 con 0x7efcc40b2710 2024-01-24T17:43:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.052+0000 7efcc2000640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9c06a430 0x7efc9c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.053+0000 7efcc2000640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9c06a430 0x7efc9c06c8f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efcac006fd0 tx=0x7efcac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.056+0000 7efcbaa00640 1 -- 172.21.15.39:0/691003492 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efcb40763e0 con 0x7efcc40b2710 2024-01-24T17:43:40.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.248+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efcc40d1ba0 con 0x7efcc40b2710 2024-01-24T17:43:40.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.249+0000 7efcbaa00640 1 -- 172.21.15.39:0/691003492 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7efcb4079a00 con 0x7efcc40b2710 2024-01-24T17:43:40.249 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9c06a430 msgr2=0x7efc9c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9c06a430 0x7efc9c06c8f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efcac006fd0 tx=0x7efcac008040 comp rx=0 tx=0).stop 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc40b2710 msgr2=0x7efcc4146500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc40b2710 0x7efcc4146500 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009890 tx=0x7efcb40046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 shutdown_connections 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efcc415c8d0 0x7efcc406c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc40d2cb0 0x7efcc406bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9c06a430 0x7efc9c06c8f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 --2- 172.21.15.39:0/691003492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc40b2710 0x7efcc4146500 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 >> 172.21.15.39:0/691003492 conn(0x7efcc4071dd0 msgr2=0x7efcc40b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 shutdown_connections 2024-01-24T17:43:40.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.251+0000 7efcc3e00640 1 -- 172.21.15.39:0/691003492 wait complete. 2024-01-24T17:43:40.272 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:43:40.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.494+0000 7f170fe00640 1 Processor -- start 2024-01-24T17:43:40.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.494+0000 7f170fe00640 1 -- start start 2024-01-24T17:43:40.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.496+0000 7f170fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f171015ccb0 0x7f17100cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.496+0000 7f170fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f17100c9d60 0x7f17100ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.496+0000 7f170fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171015d090 con 0x7f17100ce390 2024-01-24T17:43:40.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.496+0000 7f170fe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17100cac20 con 0x7f171015ccb0 2024-01-24T17:43:40.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.496+0000 7f170fe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17100ced60 con 0x7f17100c9d60 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.496+0000 7f1707e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f17100ce390 0x7f17100cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59864/0 (socket says 172.21.15.39:59864) 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.496+0000 7f1707e00640 1 -- 172.21.15.39:0/3858281815 learned_addr learned my addr 172.21.15.39:0/3858281815 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.496+0000 7f170ea00640 1 --2- 172.21.15.39:0/3858281815 >> v2:172.21.15.39:3300/0 conn(0x7f171015ccb0 0x7f17100cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.497+0000 7f170f400640 1 --2- 172.21.15.39:0/3858281815 >> v2:172.21.15.100:3300/0 conn(0x7f17100c9d60 0x7f17100ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.497+0000 7f170e000640 1 -- 172.21.15.39:0/3858281815 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377967017 0 0) 0x7f171015d090 con 0x7f17100ce390 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.497+0000 7f170e000640 1 -- 172.21.15.39:0/3858281815 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f0003520 con 0x7f17100ce390 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.497+0000 7f170ea00640 1 -- 172.21.15.39:0/3858281815 >> v2:172.21.15.100:3300/0 conn(0x7f17100c9d60 msgr2=0x7f17100ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.497+0000 7f170ea00640 1 --2- 172.21.15.39:0/3858281815 >> v2:172.21.15.100:3300/0 conn(0x7f17100c9d60 0x7f17100ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.497+0000 7f170ea00640 1 -- 172.21.15.39:0/3858281815 >> v1:172.21.15.39:6789/0 conn(0x7f17100ce390 legacy=0x7f17100cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.497+0000 7f170ea00640 1 -- 172.21.15.39:0/3858281815 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171006c540 con 0x7f171015ccb0 2024-01-24T17:43:40.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.497+0000 7f170f400640 1 --2- 172.21.15.39:0/3858281815 >> v2:172.21.15.100:3300/0 conn(0x7f17100c9d60 0x7f17100ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:40.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.498+0000 7f170ea00640 1 --2- 172.21.15.39:0/3858281815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171015ccb0 0x7f17100cda90 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f171006dfc0 tx=0x7f170002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba65883bd619095 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170e000640 1 -- 172.21.15.39:0/3858281815 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700031830 con 0x7f171015ccb0 2024-01-24T17:43:40.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170e000640 1 -- 172.21.15.39:0/3858281815 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1700031990 con 0x7f171015ccb0 2024-01-24T17:43:40.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170e000640 1 -- 172.21.15.39:0/3858281815 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700031d20 con 0x7f171015ccb0 2024-01-24T17:43:40.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170fe00640 1 -- 172.21.15.39:0/3858281815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171015ccb0 msgr2=0x7f17100cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170fe00640 1 --2- 172.21.15.39:0/3858281815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171015ccb0 0x7f17100cda90 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f171006dfc0 tx=0x7f170002b8d0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170fe00640 1 -- 172.21.15.39:0/3858281815 shutdown_connections 2024-01-24T17:43:40.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170fe00640 1 --2- 172.21.15.39:0/3858281815 >> v2:172.21.15.100:3300/0 conn(0x7f17100c9d60 0x7f17100ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170fe00640 1 --2- 172.21.15.39:0/3858281815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171015ccb0 0x7f17100cda90 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170fe00640 1 -- 172.21.15.39:0/3858281815 >> 172.21.15.39:0/3858281815 conn(0x7f1710069e50 msgr2=0x7f171006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:40.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.499+0000 7f170fe00640 1 -- 172.21.15.39:0/3858281815 shutdown_connections 2024-01-24T17:43:40.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.500+0000 7f170fe00640 1 -- 172.21.15.39:0/3858281815 wait complete. 2024-01-24T17:43:40.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.501+0000 7f170fe00640 1 Processor -- start 2024-01-24T17:43:40.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.501+0000 7f170fe00640 1 -- start start 2024-01-24T17:43:40.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.502+0000 7f170fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17100c9d60 0x7f17101598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.502+0000 7f170fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17100ce390 0x7f17100c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.502+0000 7f170fe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171015ccb0 0x7f17100c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.502+0000 7f170fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17100c80f0 con 0x7f17100c9d60 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.502+0000 7f170fe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17100c7b30 con 0x7f171015ccb0 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.502+0000 7f170fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17100c5590 con 0x7f17100ce390 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17100c9d60 0x7f17101598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17100c9d60 0x7f17101598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51894/0 (socket says 172.21.15.39:51894) 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 -- 172.21.15.39:0/734339157 learned_addr learned my addr 172.21.15.39:0/734339157 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170f400640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171015ccb0 0x7f17100c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 -- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17100ce390 msgr2=0x7f17100c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17100ce390 0x7f17100c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 -- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171015ccb0 msgr2=0x7f17100c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171015ccb0 0x7f17100c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 -- 172.21.15.39:0/734339157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17000093a0 con 0x7f17100c9d60 2024-01-24T17:43:40.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.503+0000 7f170ea00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17100c9d60 0x7f17101598d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1700000c00 tx=0x7f17000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.504+0000 7f1706a00640 1 -- 172.21.15.39:0/734339157 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170003a8f0 con 0x7f17100c9d60 2024-01-24T17:43:40.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.504+0000 7f1706a00640 1 -- 172.21.15.39:0/734339157 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170003aa50 con 0x7f17100c9d60 2024-01-24T17:43:40.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.504+0000 7f1706a00640 1 -- 172.21.15.39:0/734339157 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170003ae00 con 0x7f17100c9d60 2024-01-24T17:43:40.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.504+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17100c5870 con 0x7f17100c9d60 2024-01-24T17:43:40.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.504+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1710159e10 con 0x7f17100c9d60 2024-01-24T17:43:40.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.505+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171004dff0 con 0x7f17100c9d60 2024-01-24T17:43:40.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.506+0000 7f1706a00640 1 -- 172.21.15.39:0/734339157 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f170004d450 con 0x7f17100c9d60 2024-01-24T17:43:40.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.507+0000 7f1706a00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16e806a430 0x7f16e806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.507+0000 7f1706a00640 1 -- 172.21.15.39:0/734339157 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f17000aab00 con 0x7f17100c9d60 2024-01-24T17:43:40.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.508+0000 7f1707e00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16e806a430 0x7f16e806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.509+0000 7f1707e00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16e806a430 0x7f16e806c8f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f171015a9d0 tx=0x7f16f8007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.511+0000 7f1706a00640 1 -- 172.21.15.39:0/734339157 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f17000ace60 con 0x7f17100c9d60 2024-01-24T17:43:40.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.705+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f17100cbae0 con 0x7f17100c9d60 2024-01-24T17:43:40.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.706+0000 7f1706a00640 1 -- 172.21.15.39:0/734339157 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1700035070 con 0x7f17100c9d60 2024-01-24T17:43:40.707 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16e806a430 msgr2=0x7f16e806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16e806a430 0x7f16e806c8f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f171015a9d0 tx=0x7f16f8007930 comp rx=0 tx=0).stop 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17100c9d60 msgr2=0x7f17101598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17100c9d60 0x7f17101598d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1700000c00 tx=0x7f17000046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 shutdown_connections 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171015ccb0 0x7f17100c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17100ce390 0x7f17100c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16e806a430 0x7f16e806c8f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 --2- 172.21.15.39:0/734339157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17100c9d60 0x7f17101598d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 >> 172.21.15.39:0/734339157 conn(0x7f1710069e50 msgr2=0x7f17100cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 shutdown_connections 2024-01-24T17:43:40.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.709+0000 7f170fe00640 1 -- 172.21.15.39:0/734339157 wait complete. 2024-01-24T17:43:40.730 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:43:40.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.955+0000 7fc2c4800640 1 Processor -- start 2024-01-24T17:43:40.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.955+0000 7fc2c4800640 1 -- start start 2024-01-24T17:43:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.957+0000 7fc2c4800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc2bc0cb3e0 0x7fc2bc0b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.957+0000 7fc2c4800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0c9180 con 0x7fc2bc15a950 2024-01-24T17:43:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.957+0000 7fc2c4800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0c9460 con 0x7fc2bc0caf50 2024-01-24T17:43:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.957+0000 7fc2c4800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc0b43d0 con 0x7fc2bc0cb3e0 2024-01-24T17:43:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.957+0000 7fc2c2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc2bc0cb3e0 0x7fc2bc0b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.957+0000 7fc2c2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc2bc0cb3e0 0x7fc2bc0b1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51908/0 (socket says 172.21.15.39:51908) 2024-01-24T17:43:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.957+0000 7fc2c2800640 1 -- 172.21.15.39:0/3817142038 learned_addr learned my addr 172.21.15.39:0/3817142038 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.957+0000 7fc2c1e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc2bc15a950 0x7fc2bc0ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59878/0 (socket says 172.21.15.39:59878) 2024-01-24T17:43:40.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.958+0000 7fc2c2800640 1 -- 172.21.15.39:0/3817142038 >> v1:172.21.15.100:6789/0 conn(0x7fc2bc0caf50 legacy=0x7fc2bc0b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.958+0000 7fc2c2800640 1 -- 172.21.15.39:0/3817142038 >> v1:172.21.15.39:6789/0 conn(0x7fc2bc15a950 legacy=0x7fc2bc0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.958+0000 7fc2c2800640 1 -- 172.21.15.39:0/3817142038 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0c9740 con 0x7fc2bc0cb3e0 2024-01-24T17:43:40.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.959+0000 7fc2c2800640 1 --2- 172.21.15.39:0/3817142038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0cb3e0 0x7fc2bc0b1aa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009870 tx=0x7fc2b402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f89d1b06438b3a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.959+0000 7fc2c0a00640 1 -- 172.21.15.39:0/3817142038 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b402f840 con 0x7fc2bc0cb3e0 2024-01-24T17:43:40.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.960+0000 7fc2c0a00640 1 -- 172.21.15.39:0/3817142038 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b402f9a0 con 0x7fc2bc0cb3e0 2024-01-24T17:43:40.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.960+0000 7fc2c0a00640 1 -- 172.21.15.39:0/3817142038 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b402fd30 con 0x7fc2bc0cb3e0 2024-01-24T17:43:40.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.960+0000 7fc2c4800640 1 -- 172.21.15.39:0/3817142038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0cb3e0 msgr2=0x7fc2bc0b1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.960+0000 7fc2c4800640 1 --2- 172.21.15.39:0/3817142038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0cb3e0 0x7fc2bc0b1aa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009870 tx=0x7fc2b402ba20 comp rx=0 tx=0).stop 2024-01-24T17:43:40.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.960+0000 7fc2c4800640 1 -- 172.21.15.39:0/3817142038 shutdown_connections 2024-01-24T17:43:40.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.960+0000 7fc2c4800640 1 --2- 172.21.15.39:0/3817142038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0cb3e0 0x7fc2bc0b1aa0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.960+0000 7fc2c4800640 1 -- 172.21.15.39:0/3817142038 >> 172.21.15.39:0/3817142038 conn(0x7fc2bc06a3e0 msgr2=0x7fc2bc06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:40.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.961+0000 7fc2c4800640 1 -- 172.21.15.39:0/3817142038 shutdown_connections 2024-01-24T17:43:40.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.961+0000 7fc2c4800640 1 -- 172.21.15.39:0/3817142038 wait complete. 2024-01-24T17:43:40.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.962+0000 7fc2c4800640 1 Processor -- start 2024-01-24T17:43:40.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.962+0000 7fc2c4800640 1 -- start start 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.963+0000 7fc2c4800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0caf50 0x7fc2bc0b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.963+0000 7fc2c4800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0cb3e0 0x7fc2bc0c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.963+0000 7fc2c4800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc15a950 0x7fc2bc0adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.963+0000 7fc2c4800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1f24d0 con 0x7fc2bc0caf50 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.963+0000 7fc2c4800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0b0c20 con 0x7fc2bc15a950 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.963+0000 7fc2c4800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0aeab0 con 0x7fc2bc0cb3e0 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0caf50 0x7fc2bc0b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0caf50 0x7fc2bc0b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51910/0 (socket says 172.21.15.39:51910) 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1e00640 1 -- 172.21.15.39:0/853647552 learned_addr learned my addr 172.21.15.39:0/853647552 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c2800640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc15a950 0x7fc2bc0adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1e00640 1 -- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0cb3e0 msgr2=0x7fc2bc0c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1400640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0cb3e0 0x7fc2bc0c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1e00640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0cb3e0 0x7fc2bc0c7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1e00640 1 -- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc15a950 msgr2=0x7fc2bc0adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:40.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1e00640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc15a950 0x7fc2bc0adea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:40.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c1e00640 1 -- 172.21.15.39:0/853647552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b40094d0 con 0x7fc2bc0caf50 2024-01-24T17:43:40.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.964+0000 7fc2c2800640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc15a950 0x7fc2bc0adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:40.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.965+0000 7fc2c1e00640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0caf50 0x7fc2bc0b06e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00c7a0 tx=0x7fc2ac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.965+0000 7fc2ba000640 1 -- 172.21.15.39:0/853647552 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac00a640 con 0x7fc2bc0caf50 2024-01-24T17:43:40.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.965+0000 7fc2ba000640 1 -- 172.21.15.39:0/853647552 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac00a7a0 con 0x7fc2bc0caf50 2024-01-24T17:43:40.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.965+0000 7fc2ba000640 1 -- 172.21.15.39:0/853647552 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac00ab10 con 0x7fc2bc0caf50 2024-01-24T17:43:40.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.965+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc0aed90 con 0x7fc2bc0caf50 2024-01-24T17:43:40.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.965+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc0b01c0 con 0x7fc2bc0caf50 2024-01-24T17:43:40.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.966+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc073ef0 con 0x7fc2bc0caf50 2024-01-24T17:43:40.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.967+0000 7fc2ba000640 1 -- 172.21.15.39:0/853647552 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc2ac00ac70 con 0x7fc2bc0caf50 2024-01-24T17:43:40.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.968+0000 7fc2ba000640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc29406a460 0x7fc29406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:40.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.968+0000 7fc2ba000640 1 -- 172.21.15.39:0/853647552 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc2ac087fa0 con 0x7fc2bc0caf50 2024-01-24T17:43:40.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.968+0000 7fc2c1400640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc29406a460 0x7fc29406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:40.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.970+0000 7fc2c1400640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc29406a460 0x7fc29406c920 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4000f60 tx=0x7fc2a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:40.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:40.972+0000 7fc2ba000640 1 -- 172.21.15.39:0/853647552 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc2ac0bacb0 con 0x7fc2bc0caf50 2024-01-24T17:43:41.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.182+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc2bc0ac970 con 0x7fc2bc0caf50 2024-01-24T17:43:41.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.183+0000 7fc2ba000640 1 -- 172.21.15.39:0/853647552 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc2ac017080 con 0x7fc2bc0caf50 2024-01-24T17:43:41.184 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc29406a460 msgr2=0x7fc29406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc29406a460 0x7fc29406c920 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4000f60 tx=0x7fc2a4009040 comp rx=0 tx=0).stop 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0caf50 msgr2=0x7fc2bc0b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0caf50 0x7fc2bc0b06e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00c7a0 tx=0x7fc2ac00cc60 comp rx=0 tx=0).stop 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 shutdown_connections 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2bc15a950 0x7fc2bc0adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc2bc0cb3e0 0x7fc2bc0c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc29406a460 0x7fc29406c920 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 --2- 172.21.15.39:0/853647552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc2bc0caf50 0x7fc2bc0b06e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 >> 172.21.15.39:0/853647552 conn(0x7fc2bc06a3e0 msgr2=0x7fc2bc0b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 shutdown_connections 2024-01-24T17:43:41.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.186+0000 7fc2c4800640 1 -- 172.21.15.39:0/853647552 wait complete. 2024-01-24T17:43:41.207 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:43:41.207 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:41.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:41.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.478+0000 7f4337200640 1 Processor -- start 2024-01-24T17:43:41.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.478+0000 7f4337200640 1 -- start start 2024-01-24T17:43:41.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.480+0000 7f4337200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f433815a720 0x7f43380cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.480+0000 7f4337200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380ce400 con 0x7f43380ce8e0 2024-01-24T17:43:41.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.480+0000 7f4337200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380ccef0 con 0x7f433815ab00 2024-01-24T17:43:41.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.480+0000 7f4337200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43380ce110 con 0x7f433815a720 2024-01-24T17:43:41.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4336800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f43380ce8e0 0x7f43380cb2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59884/0 (socket says 172.21.15.39:59884) 2024-01-24T17:43:41.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4336800640 1 -- 172.21.15.39:0/2741878023 learned_addr learned my addr 172.21.15.39:0/2741878023 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:41.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4335e00640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f433815a720 0x7f43380cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778665617 0 0) 0x7f43380ce400 con 0x7f43380ce8e0 2024-01-24T17:43:41.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4318003520 con 0x7f43380ce8e0 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1055288265 0 0) 0x7f4318003520 con 0x7f43380ce8e0 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f433815a720 msgr2=0x7f43380cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4335400640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f433815a720 0x7f43380cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.481+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 >> v1:172.21.15.100:6789/0 conn(0x7f433815ab00 legacy=0x7f43380cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433815b3b0 con 0x7f43380ce8e0 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f43300019f0 con 0x7f43380ce8e0 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 >> v1:172.21.15.39:6789/0 conn(0x7f43380ce8e0 legacy=0x7f43380cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180039a0 0x7f4318003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180047c0 0x7f4318004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f433815a720 0x7f43180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380ce110 con 0x7f43180047c0 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431800bab0 con 0x7f43180039a0 2024-01-24T17:43:41.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431800bc70 con 0x7f433815a720 2024-01-24T17:43:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f432fe00640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180047c0 0x7f4318004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4336800640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f433815a720 0x7f43180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.482+0000 7f4335e00640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180039a0 0x7f4318003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.483+0000 7f432fe00640 1 -- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f433815a720 msgr2=0x7f43180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.483+0000 7f432fe00640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f433815a720 0x7f43180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.483+0000 7f432fe00640 1 -- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180039a0 msgr2=0x7f4318003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.483+0000 7f432fe00640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180039a0 0x7f4318003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.483+0000 7f432fe00640 1 -- 172.21.15.39:0/2741878023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318003520 con 0x7f43180047c0 2024-01-24T17:43:41.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.484+0000 7f432fe00640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180047c0 0x7f4318004c40 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4320001750 tx=0x7f43180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed3e60577f80e9c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:41.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.484+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43200052d0 con 0x7f43180047c0 2024-01-24T17:43:41.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.484+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432002e050 con 0x7f43180047c0 2024-01-24T17:43:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.484+0000 7f4335400640 1 -- 172.21.15.39:0/2741878023 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43200035f0 con 0x7f43180047c0 2024-01-24T17:43:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.485+0000 7f4337200640 1 -- 172.21.15.39:0/2741878023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180047c0 msgr2=0x7f4318004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.485+0000 7f4337200640 1 --2- 172.21.15.39:0/2741878023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180047c0 0x7f4318004c40 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4320001750 tx=0x7f43180057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.485+0000 7f4336800640 1 -- 172.21.15.39:0/2741878023 reap_dead start 2024-01-24T17:43:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.485+0000 7f4337200640 1 -- 172.21.15.39:0/2741878023 shutdown_connections 2024-01-24T17:43:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.485+0000 7f4337200640 1 -- 172.21.15.39:0/2741878023 >> 172.21.15.39:0/2741878023 conn(0x7f433806a090 msgr2=0x7f433806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.485+0000 7f4337200640 1 -- 172.21.15.39:0/2741878023 shutdown_connections 2024-01-24T17:43:41.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.485+0000 7f4337200640 1 -- 172.21.15.39:0/2741878023 wait complete. 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.487+0000 7f4337200640 1 Processor -- start 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.487+0000 7f4337200640 1 -- start start 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4337200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180047c0 0x7f4338154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4337200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43380c3a70 0x7f43380c4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4337200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43380c4850 0x7f43381f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4337200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380af5d0 con 0x7f43380c4850 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4337200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380c4cd0 con 0x7f43380c3a70 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4337200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4338156a40 con 0x7f43180047c0 2024-01-24T17:43:41.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4336800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43380c4850 0x7f43381f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4336800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43380c4850 0x7f43381f13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51928/0 (socket says 172.21.15.39:51928) 2024-01-24T17:43:41.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4336800640 1 -- 172.21.15.39:0/1663034038 learned_addr learned my addr 172.21.15.39:0/1663034038 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:41.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f4335e00640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180047c0 0x7f4338154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.488+0000 7f432fe00640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43380c3a70 0x7f43380c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f4336800640 1 -- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180047c0 msgr2=0x7f4338154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f4336800640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180047c0 0x7f4338154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f4336800640 1 -- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43380c3a70 msgr2=0x7f43380c4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f4336800640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43380c3a70 0x7f43380c4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f4336800640 1 -- 172.21.15.39:0/1663034038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43380c2ef0 con 0x7f43380c4850 2024-01-24T17:43:41.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f4336800640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43380c4850 0x7f43381f13e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f43380b4520 tx=0x7f433000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:41.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f432e000640 1 -- 172.21.15.39:0/1663034038 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000ebb0 con 0x7f43380c4850 2024-01-24T17:43:41.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f432e000640 1 -- 172.21.15.39:0/1663034038 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433000ed10 con 0x7f43380c4850 2024-01-24T17:43:41.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.489+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43380c22e0 con 0x7f43380c4850 2024-01-24T17:43:41.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.490+0000 7f432e000640 1 -- 172.21.15.39:0/1663034038 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330013660 con 0x7f43380c4850 2024-01-24T17:43:41.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.490+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43380c28a0 con 0x7f43380c4850 2024-01-24T17:43:41.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.491+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433804dff0 con 0x7f43380c4850 2024-01-24T17:43:41.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.492+0000 7f432e000640 1 -- 172.21.15.39:0/1663034038 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f43300137c0 con 0x7f43380c4850 2024-01-24T17:43:41.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.493+0000 7f432e000640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431006a430 0x7f431006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.493+0000 7f432e000640 1 -- 172.21.15.39:0/1663034038 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4330088dd0 con 0x7f43380c4850 2024-01-24T17:43:41.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.493+0000 7f4335e00640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431006a430 0x7f431006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.495+0000 7f4335e00640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431006a430 0x7f431006c8f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4328009a10 tx=0x7f4328008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.497+0000 7f432e000640 1 -- 172.21.15.39:0/1663034038 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4330057dd0 con 0x7f43380c4850 2024-01-24T17:43:41.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.689+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f433806c8a0 con 0x7f43380c4850 2024-01-24T17:43:41.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.693+0000 7f432e000640 1 -- 172.21.15.39:0/1663034038 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16290 (secure 0 0 0) 0x7f433008a020 con 0x7f43380c4850 2024-01-24T17:43:41.693 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:41.694 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:43:16.261174+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.1c","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.32","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:41.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.695+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431006a430 msgr2=0x7f431006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.695+0000 7f4337200640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431006a430 0x7f431006c8f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4328009a10 tx=0x7f4328008040 comp rx=0 tx=0).stop 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.695+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43380c4850 msgr2=0x7f43381f13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.695+0000 7f4337200640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43380c4850 0x7f43381f13e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f43380b4520 tx=0x7f433000e530 comp rx=0 tx=0).stop 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.696+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 shutdown_connections 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.696+0000 7f4337200640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f431006a430 0x7f431006c8f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.696+0000 7f4337200640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43380c4850 0x7f43381f13e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.696+0000 7f4337200640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43380c3a70 0x7f43380c4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.696+0000 7f4337200640 1 --2- 172.21.15.39:0/1663034038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180047c0 0x7f4338154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.696+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 >> 172.21.15.39:0/1663034038 conn(0x7f433806a090 msgr2=0x7f43380ade90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.696+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 shutdown_connections 2024-01-24T17:43:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.696+0000 7f4337200640 1 -- 172.21.15.39:0/1663034038 wait complete. 2024-01-24T17:43:41.716 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:42:54.883732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:42:55.727240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:43:41.717 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:43:41.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.928+0000 7f27d9800640 1 Processor -- start 2024-01-24T17:43:41.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.928+0000 7f27d9800640 1 -- start start 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27d9800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f27d415ca40 0x7f27d4161170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27d9800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f27d40cce00 0x7f27d40cd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27d9800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d406d930 con 0x7f27d40cdb70 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27d9800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d406d460 con 0x7f27d40cce00 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27d9800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d40cd7d0 con 0x7f27d415ca40 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27d8e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f27d40cce00 0x7f27d40cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27d8e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f27d40cce00 0x7f27d40cd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51942/0 (socket says 172.21.15.39:51942) 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27d8e00640 1 -- 172.21.15.39:0/4010118699 learned_addr learned my addr 172.21.15.39:0/4010118699 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:41.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.930+0000 7f27cbe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f27d40cdb70 0x7f27d406cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59890/0 (socket says 172.21.15.39:59890) 2024-01-24T17:43:41.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.931+0000 7f27d3e00640 1 --2- 172.21.15.39:0/4010118699 >> v2:172.21.15.100:3300/0 conn(0x7f27d415ca40 0x7f27d4161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.931+0000 7f27d8e00640 1 -- 172.21.15.39:0/4010118699 >> v2:172.21.15.100:3300/0 conn(0x7f27d415ca40 msgr2=0x7f27d4161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:41.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.931+0000 7f27d8e00640 1 --2- 172.21.15.39:0/4010118699 >> v2:172.21.15.100:3300/0 conn(0x7f27d415ca40 0x7f27d4161170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.931+0000 7f27d8e00640 1 -- 172.21.15.39:0/4010118699 >> v1:172.21.15.39:6789/0 conn(0x7f27d40cdb70 legacy=0x7f27d406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.931+0000 7f27d8e00640 1 -- 172.21.15.39:0/4010118699 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d406d650 con 0x7f27d40cce00 2024-01-24T17:43:41.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.932+0000 7f27d8e00640 1 --2- 172.21.15.39:0/4010118699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cce00 0x7f27d40cd290 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f27d40cbef0 tx=0x7f27cc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4320e4753c6bb7a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:41.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.933+0000 7f27d3400640 1 -- 172.21.15.39:0/4010118699 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc02cb20 con 0x7f27d40cce00 2024-01-24T17:43:41.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.933+0000 7f27d3400640 1 -- 172.21.15.39:0/4010118699 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27cc02cc80 con 0x7f27d40cce00 2024-01-24T17:43:41.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.933+0000 7f27d3400640 1 -- 172.21.15.39:0/4010118699 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0343f0 con 0x7f27d40cce00 2024-01-24T17:43:41.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.933+0000 7f27d9800640 1 -- 172.21.15.39:0/4010118699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cce00 msgr2=0x7f27d40cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.933+0000 7f27d9800640 1 --2- 172.21.15.39:0/4010118699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cce00 0x7f27d40cd290 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f27d40cbef0 tx=0x7f27cc02bf80 comp rx=0 tx=0).stop 2024-01-24T17:43:41.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.934+0000 7f27d9800640 1 -- 172.21.15.39:0/4010118699 shutdown_connections 2024-01-24T17:43:41.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.934+0000 7f27d9800640 1 --2- 172.21.15.39:0/4010118699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cce00 0x7f27d40cd290 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.934+0000 7f27d9800640 1 --2- 172.21.15.39:0/4010118699 >> v2:172.21.15.100:3300/0 conn(0x7f27d415ca40 0x7f27d4161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.934+0000 7f27d9800640 1 -- 172.21.15.39:0/4010118699 >> 172.21.15.39:0/4010118699 conn(0x7f27d4069e50 msgr2=0x7f27d406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:41.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.934+0000 7f27d9800640 1 -- 172.21.15.39:0/4010118699 shutdown_connections 2024-01-24T17:43:41.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.934+0000 7f27d9800640 1 -- 172.21.15.39:0/4010118699 wait complete. 2024-01-24T17:43:41.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.936+0000 7f27d9800640 1 Processor -- start 2024-01-24T17:43:41.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.936+0000 7f27d9800640 1 -- start start 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27d9800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27d40cce00 0x7f27d40c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27d9800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cdb70 0x7f27d41e8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27d9800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27d415ca40 0x7f27d415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27d9800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d41e8740 con 0x7f27d40cdb70 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27d9800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40c6800 con 0x7f27d415ca40 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27d9800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40c5bf0 con 0x7f27d40cce00 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cdb70 0x7f27d41e8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cdb70 0x7f27d41e8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51956/0 (socket says 172.21.15.39:51956) 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27cbe00640 1 -- 172.21.15.39:0/3929987882 learned_addr learned my addr 172.21.15.39:0/3929987882 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27d3e00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27d40cce00 0x7f27d40c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.937+0000 7f27d8e00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27d415ca40 0x7f27d415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27cbe00640 1 -- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27d40cce00 msgr2=0x7f27d40c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27cbe00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27d40cce00 0x7f27d40c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27cbe00640 1 -- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27d415ca40 msgr2=0x7f27d415b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27cbe00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27d415ca40 0x7f27d415b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27cbe00640 1 -- 172.21.15.39:0/3929987882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc009b30 con 0x7f27d40cdb70 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27d8e00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27d415ca40 0x7f27d415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27cbe00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cdb70 0x7f27d41e8200 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00c440 tx=0x7f27bc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27d0c00640 1 -- 172.21.15.39:0/3929987882 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0074e0 con 0x7f27d40cdb70 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27d0c00640 1 -- 172.21.15.39:0/3929987882 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc007640 con 0x7f27d40cdb70 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27d0c00640 1 -- 172.21.15.39:0/3929987882 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc004a20 con 0x7f27d40cdb70 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d40c5ed0 con 0x7f27d40cdb70 2024-01-24T17:43:41.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.938+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d40c6490 con 0x7f27d40cdb70 2024-01-24T17:43:41.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.940+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d404dff0 con 0x7f27d40cdb70 2024-01-24T17:43:41.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.940+0000 7f27d0c00640 1 -- 172.21.15.39:0/3929987882 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f27bc004b80 con 0x7f27d40cdb70 2024-01-24T17:43:41.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.941+0000 7f27d0c00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27ac06a480 0x7f27ac06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:41.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.941+0000 7f27d0c00640 1 -- 172.21.15.39:0/3929987882 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f27bc088290 con 0x7f27d40cdb70 2024-01-24T17:43:41.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.942+0000 7f27d3e00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27ac06a480 0x7f27ac06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:41.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.943+0000 7f27d3e00640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27ac06a480 0x7f27ac06c940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f27d409c300 tx=0x7f27c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:41.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:41.946+0000 7f27d0c00640 1 -- 172.21.15.39:0/3929987882 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f27bc053870 con 0x7f27d40cdb70 2024-01-24T17:43:42.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.137+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27d40c6940 con 0x7f27d40cdb70 2024-01-24T17:43:42.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.138+0000 7f27d0c00640 1 -- 172.21.15.39:0/3929987882 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27bc056e90 con 0x7f27d40cdb70 2024-01-24T17:43:42.139 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:43:42.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.140+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27ac06a480 msgr2=0x7f27ac06c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.140+0000 7f27d9800640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27ac06a480 0x7f27ac06c940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f27d409c300 tx=0x7f27c4008040 comp rx=0 tx=0).stop 2024-01-24T17:43:42.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.140+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cdb70 msgr2=0x7f27d41e8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.140+0000 7f27d9800640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cdb70 0x7f27d41e8200 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00c440 tx=0x7f27bc00c900 comp rx=0 tx=0).stop 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.141+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 shutdown_connections 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.141+0000 7f27d9800640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27d415ca40 0x7f27d415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.141+0000 7f27d9800640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27ac06a480 0x7f27ac06c940 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.141+0000 7f27d9800640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27d40cdb70 0x7f27d41e8200 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.141+0000 7f27d9800640 1 --2- 172.21.15.39:0/3929987882 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27d40cce00 0x7f27d40c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.141+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 >> 172.21.15.39:0/3929987882 conn(0x7f27d4069e50 msgr2=0x7f27d40ab940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.141+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 shutdown_connections 2024-01-24T17:43:42.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.141+0000 7f27d9800640 1 -- 172.21.15.39:0/3929987882 wait complete. 2024-01-24T17:43:42.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:43:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.381+0000 7fcac6e00640 1 Processor -- start 2024-01-24T17:43:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.381+0000 7fcac6e00640 1 -- start start 2024-01-24T17:43:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.383+0000 7fcac6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac00accd0 0x7fcac006d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.383+0000 7fcac6e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcac006c3a0 0x7fcac00cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.383+0000 7fcac6e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac00ad0b0 con 0x7fcac00cb370 2024-01-24T17:43:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.383+0000 7fcac6e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac00cb800 con 0x7fcac006c3a0 2024-01-24T17:43:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.383+0000 7fcac6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac006d840 con 0x7fcac00accd0 2024-01-24T17:43:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.383+0000 7fcac6400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fcac00cb370 0x7fcac015e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59898/0 (socket says 172.21.15.39:59898) 2024-01-24T17:43:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.383+0000 7fcac6400640 1 -- 172.21.15.39:0/1752836129 learned_addr learned my addr 172.21.15.39:0/1752836129 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.383+0000 7fcac5000640 1 --2- 172.21.15.39:0/1752836129 >> v2:172.21.15.100:3300/0 conn(0x7fcac006c3a0 0x7fcac00cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcac5a00640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac00accd0 0x7fcac006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198737918 0 0) 0x7fcac00ad0b0 con 0x7fcac00cb370 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa0003520 con 0x7fcac00cb370 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063666143 0 0) 0x7fcaa0003520 con 0x7fcac00cb370 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac00accd0 msgr2=0x7fcac006d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac00accd0 0x7fcac006d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 >> v2:172.21.15.100:3300/0 conn(0x7fcac006c3a0 msgr2=0x7fcac00cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 --2- 172.21.15.39:0/1752836129 >> v2:172.21.15.100:3300/0 conn(0x7fcac006c3a0 0x7fcac00cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac00b2370 con 0x7fcac00cb370 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fcab80019f0 con 0x7fcac00cb370 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.384+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 >> v1:172.21.15.39:6789/0 conn(0x7fcac00cb370 legacy=0x7fcac015e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcabfe00640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac00accd0 0x7fcaa0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcabfe00640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0003f40 0x7fcaa00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcabfe00640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcaa0004d00 0x7fcaa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00cb800 con 0x7fcaa0003f40 2024-01-24T17:43:42.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac006d840 con 0x7fcaa0004d00 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa000b980 con 0x7fcac00accd0 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcac5000640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0003f40 0x7fcaa00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcac5a00640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac00accd0 0x7fcaa0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcac6400640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcaa0004d00 0x7fcaa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcac5000640 1 -- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac00accd0 msgr2=0x7fcaa0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcac5000640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac00accd0 0x7fcaa0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcac5000640 1 -- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcaa0004d00 msgr2=0x7fcaa0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.385+0000 7fcac5000640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcaa0004d00 0x7fcaa0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.386+0000 7fcac5000640 1 -- 172.21.15.39:0/1752836129 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa0003520 con 0x7fcaa0003f40 2024-01-24T17:43:42.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.387+0000 7fcac5000640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0003f40 0x7fcaa00047c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009af0 tx=0x7fcaa00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba73d3132aa806d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.387+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa802c710 con 0x7fcaa0003f40 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.387+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa802c870 con 0x7fcaa0003f40 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.387+0000 7fcabfe00640 1 -- 172.21.15.39:0/1752836129 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa802cc00 con 0x7fcaa0003f40 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.388+0000 7fcac6e00640 1 -- 172.21.15.39:0/1752836129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0003f40 msgr2=0x7fcaa00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.388+0000 7fcac6e00640 1 --2- 172.21.15.39:0/1752836129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0003f40 0x7fcaa00047c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009af0 tx=0x7fcaa00057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.388+0000 7fcac6400640 1 -- 172.21.15.39:0/1752836129 reap_dead start 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.388+0000 7fcac6e00640 1 -- 172.21.15.39:0/1752836129 shutdown_connections 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.388+0000 7fcac6e00640 1 -- 172.21.15.39:0/1752836129 >> 172.21.15.39:0/1752836129 conn(0x7fcac0069e50 msgr2=0x7fcac006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.388+0000 7fcac6e00640 1 -- 172.21.15.39:0/1752836129 shutdown_connections 2024-01-24T17:43:42.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.388+0000 7fcac6e00640 1 -- 172.21.15.39:0/1752836129 wait complete. 2024-01-24T17:43:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.390+0000 7fcac6e00640 1 Processor -- start 2024-01-24T17:43:42.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.390+0000 7fcac6e00640 1 -- start start 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac6e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac006ff40 0x7fcac0070320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcac00c6170 0x7fcac01f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac01f3a20 0x7fcac01ef410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac01f4410 con 0x7fcac00c6170 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac6e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00c65f0 con 0x7fcac006ff40 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00c7390 con 0x7fcac01f3a20 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcac00c6170 0x7fcac01f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcac00c6170 0x7fcac01f34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51974/0 (socket says 172.21.15.39:51974) 2024-01-24T17:43:42.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.391+0000 7fcac5000640 1 -- 172.21.15.39:0/922699241 learned_addr learned my addr 172.21.15.39:0/922699241 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:42.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac6400640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac01f3a20 0x7fcac01ef410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac5a00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac006ff40 0x7fcac0070320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac5000640 1 -- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac01f3a20 msgr2=0x7fcac01ef410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac5000640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac01f3a20 0x7fcac01ef410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac5000640 1 -- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac006ff40 msgr2=0x7fcac0070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac5000640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac006ff40 0x7fcac0070320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac5000640 1 -- 172.21.15.39:0/922699241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac01f3ea0 con 0x7fcac00c6170 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac5000640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcac00c6170 0x7fcac01f34e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8037040 tx=0x7fcaa8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcabd600640 1 -- 172.21.15.39:0/922699241 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8003810 con 0x7fcac00c6170 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcabd600640 1 -- 172.21.15.39:0/922699241 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa802de90 con 0x7fcac00c6170 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcabd600640 1 -- 172.21.15.39:0/922699241 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8003bb0 con 0x7fcac00c6170 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.392+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac01f4130 con 0x7fcac00c6170 2024-01-24T17:43:42.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.393+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac00c5970 con 0x7fcac00c6170 2024-01-24T17:43:42.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.394+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac014d920 con 0x7fcac00c6170 2024-01-24T17:43:42.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.395+0000 7fcabd600640 1 -- 172.21.15.39:0/922699241 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcaa8003d10 con 0x7fcac00c6170 2024-01-24T17:43:42.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.395+0000 7fcabd600640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca9806a430 0x7fca9806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.396+0000 7fcabd600640 1 -- 172.21.15.39:0/922699241 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcaa80aa800 con 0x7fcac00c6170 2024-01-24T17:43:42.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.396+0000 7fcac5a00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca9806a430 0x7fca9806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.397+0000 7fcac5a00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca9806a430 0x7fca9806c8f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009880 tx=0x7fcab0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:42.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.400+0000 7fcabd600640 1 -- 172.21.15.39:0/922699241 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcaa8075d60 con 0x7fcac00c6170 2024-01-24T17:43:42.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.591+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcac00c54d0 con 0x7fcac00c6170 2024-01-24T17:43:42.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.592+0000 7fcabd600640 1 -- 172.21.15.39:0/922699241 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcaa8079380 con 0x7fcac00c6170 2024-01-24T17:43:42.593 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.594+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca9806a430 msgr2=0x7fca9806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.594+0000 7fcac6e00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca9806a430 0x7fca9806c8f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009880 tx=0x7fcab0008040 comp rx=0 tx=0).stop 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.594+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcac00c6170 msgr2=0x7fcac01f34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.594+0000 7fcac6e00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcac00c6170 0x7fcac01f34e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8037040 tx=0x7fcaa8004730 comp rx=0 tx=0).stop 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.595+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 shutdown_connections 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.595+0000 7fcac6e00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcac01f3a20 0x7fcac01ef410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.595+0000 7fcac6e00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca9806a430 0x7fca9806c8f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.595+0000 7fcac6e00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcac00c6170 0x7fcac01f34e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.595+0000 7fcac6e00640 1 --2- 172.21.15.39:0/922699241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac006ff40 0x7fcac0070320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.595+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 >> 172.21.15.39:0/922699241 conn(0x7fcac0069e50 msgr2=0x7fcac00cce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:42.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.595+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 shutdown_connections 2024-01-24T17:43:42.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.595+0000 7fcac6e00640 1 -- 172.21.15.39:0/922699241 wait complete. 2024-01-24T17:43:42.615 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:43:42.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.839+0000 7fcae7c00640 1 Processor -- start 2024-01-24T17:43:42.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.839+0000 7fcae7c00640 1 -- start start 2024-01-24T17:43:42.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.841+0000 7fcae7c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0350 0x7fcae00ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.841+0000 7fcae7c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcae00cab20 0x7fcae00c1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.841+0000 7fcae7c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae00ccee0 con 0x7fcae00b0730 2024-01-24T17:43:42.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.841+0000 7fcae7c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae00c8d60 con 0x7fcae00cab20 2024-01-24T17:43:42.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.841+0000 7fcae7c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00cc2d0 con 0x7fcae00b0350 2024-01-24T17:43:42.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.842+0000 7fcad7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcae00cab20 0x7fcae00c1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.842+0000 7fcae5200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0350 0x7fcae00ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.842+0000 7fcad7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcae00cab20 0x7fcae00c1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:60786/0 (socket says 172.21.15.39:60786) 2024-01-24T17:43:42.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.842+0000 7fcae5200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0350 0x7fcae00ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:36554/0 (socket says 172.21.15.39:36554) 2024-01-24T17:43:42.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.842+0000 7fcad7e00640 1 -- 172.21.15.39:0/1360985899 learned_addr learned my addr 172.21.15.39:0/1360985899 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:42.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.843+0000 7fcad7e00640 1 -- 172.21.15.39:0/1360985899 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0350 msgr2=0x7fcae00ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:42.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.843+0000 7fcad7e00640 1 --2- 172.21.15.39:0/1360985899 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0350 0x7fcae00ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.843+0000 7fcad7e00640 1 -- 172.21.15.39:0/1360985899 >> v1:172.21.15.100:6789/0 conn(0x7fcae00b0730 legacy=0x7fcae006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.843+0000 7fcad7e00640 1 -- 172.21.15.39:0/1360985899 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae00cc5b0 con 0x7fcae00cab20 2024-01-24T17:43:42.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.844+0000 7fcad7e00640 1 --2- 172.21.15.39:0/1360985899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 0x7fcae00c1620 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009870 tx=0x7fcac802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df219c7eb2d1bc5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.845+0000 7fcadfe00640 1 -- 172.21.15.39:0/1360985899 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8030840 con 0x7fcae00cab20 2024-01-24T17:43:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.845+0000 7fcadfe00640 1 -- 172.21.15.39:0/1360985899 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac80309a0 con 0x7fcae00cab20 2024-01-24T17:43:42.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.845+0000 7fcadfe00640 1 -- 172.21.15.39:0/1360985899 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8030d30 con 0x7fcae00cab20 2024-01-24T17:43:42.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.845+0000 7fcae7c00640 1 -- 172.21.15.39:0/1360985899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 msgr2=0x7fcae00c1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.845+0000 7fcae7c00640 1 --2- 172.21.15.39:0/1360985899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 0x7fcae00c1620 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009870 tx=0x7fcac802b980 comp rx=0 tx=0).stop 2024-01-24T17:43:42.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.846+0000 7fcae7c00640 1 -- 172.21.15.39:0/1360985899 shutdown_connections 2024-01-24T17:43:42.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.846+0000 7fcae7c00640 1 --2- 172.21.15.39:0/1360985899 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 0x7fcae00c1620 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.846+0000 7fcae7c00640 1 --2- 172.21.15.39:0/1360985899 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0350 0x7fcae00ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.846+0000 7fcae7c00640 1 -- 172.21.15.39:0/1360985899 >> 172.21.15.39:0/1360985899 conn(0x7fcae00cb0e0 msgr2=0x7fcae00cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:42.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.846+0000 7fcae7c00640 1 -- 172.21.15.39:0/1360985899 shutdown_connections 2024-01-24T17:43:42.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.846+0000 7fcae7c00640 1 -- 172.21.15.39:0/1360985899 wait complete. 2024-01-24T17:43:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.847+0000 7fcae7c00640 1 Processor -- start 2024-01-24T17:43:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.848+0000 7fcae7c00640 1 -- start start 2024-01-24T17:43:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.848+0000 7fcae7c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae00b0350 0x7fcae0160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.848+0000 7fcae7c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0730 0x7fcae00b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.848+0000 7fcae7c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 0x7fcae00b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae7c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00c6650 con 0x7fcae00b0350 2024-01-24T17:43:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae7c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00a74b0 con 0x7fcae00cab20 2024-01-24T17:43:42.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae7c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00b33f0 con 0x7fcae00b0730 2024-01-24T17:43:42.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae00b0350 0x7fcae0160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae00b0350 0x7fcae0160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51988/0 (socket says 172.21.15.39:51988) 2024-01-24T17:43:42.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5200640 1 -- 172.21.15.39:0/4115317522 learned_addr learned my addr 172.21.15.39:0/4115317522 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5c00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 0x7fcae00b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcad7e00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0730 0x7fcae00b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5200640 1 -- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0730 msgr2=0x7fcae00b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5200640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0730 0x7fcae00b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5200640 1 -- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 msgr2=0x7fcae00b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5200640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 0x7fcae00b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5200640 1 -- 172.21.15.39:0/4115317522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac80094d0 con 0x7fcae00b0350 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.849+0000 7fcae5c00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 0x7fcae00b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.850+0000 7fcae5200640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae00b0350 0x7fcae0160af0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcad000e870 tx=0x7fcad000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.850+0000 7fcadd600640 1 -- 172.21.15.39:0/4115317522 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad00094f0 con 0x7fcae00b0350 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.850+0000 7fcadd600640 1 -- 172.21.15.39:0/4115317522 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad000c940 con 0x7fcae00b0350 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.850+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae00b36d0 con 0x7fcae00b0350 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.850+0000 7fcadd600640 1 -- 172.21.15.39:0/4115317522 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad000ccf0 con 0x7fcae00b0350 2024-01-24T17:43:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.850+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae00b3c90 con 0x7fcae00b0350 2024-01-24T17:43:42.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.851+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae0073ef0 con 0x7fcae00b0350 2024-01-24T17:43:42.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.852+0000 7fcadd600640 1 -- 172.21.15.39:0/4115317522 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcad0013470 con 0x7fcae00b0350 2024-01-24T17:43:42.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.853+0000 7fcadd600640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab806a430 0x7fcab806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.853+0000 7fcadd600640 1 -- 172.21.15.39:0/4115317522 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcad0088ab0 con 0x7fcae00b0350 2024-01-24T17:43:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.854+0000 7fcad7e00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab806a430 0x7fcab806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.855+0000 7fcad7e00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab806a430 0x7fcab806c8f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcac8036000 tx=0x7fcac803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:42.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:42.857+0000 7fcadd600640 1 -- 172.21.15.39:0/4115317522 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcad0054010 con 0x7fcae00b0350 2024-01-24T17:43:43.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.051+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcae00cbe80 con 0x7fcae00b0350 2024-01-24T17:43:43.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.052+0000 7fcadd600640 1 -- 172.21.15.39:0/4115317522 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcad00170a0 con 0x7fcae00b0350 2024-01-24T17:43:43.052 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:43:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.054+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab806a430 msgr2=0x7fcab806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.054+0000 7fcae7c00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab806a430 0x7fcab806c8f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcac8036000 tx=0x7fcac803b040 comp rx=0 tx=0).stop 2024-01-24T17:43:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.054+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae00b0350 msgr2=0x7fcae0160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.054+0000 7fcae7c00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae00b0350 0x7fcae0160af0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcad000e870 tx=0x7fcad000ed30 comp rx=0 tx=0).stop 2024-01-24T17:43:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.055+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 shutdown_connections 2024-01-24T17:43:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.055+0000 7fcae7c00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcae00cab20 0x7fcae00b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.055+0000 7fcae7c00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcae00b0730 0x7fcae00b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.055+0000 7fcae7c00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab806a430 0x7fcab806c8f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.055+0000 7fcae7c00640 1 --2- 172.21.15.39:0/4115317522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae00b0350 0x7fcae0160af0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.055+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 >> 172.21.15.39:0/4115317522 conn(0x7fcae00cb0e0 msgr2=0x7fcae00aed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.055+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 shutdown_connections 2024-01-24T17:43:43.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.055+0000 7fcae7c00640 1 -- 172.21.15.39:0/4115317522 wait complete. 2024-01-24T17:43:43.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:43:43.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.290+0000 7f9a42600640 1 Processor -- start 2024-01-24T17:43:43.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.290+0000 7f9a42600640 1 -- start start 2024-01-24T17:43:43.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.291+0000 7f9a42600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c159620 0x7f9a3c0ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:43.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.291+0000 7f9a42600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9a3c0b06d0 0x7f9a3c0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:43.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.291+0000 7f9a42600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0cad00 con 0x7f9a3c0710e0 2024-01-24T17:43:43.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.291+0000 7f9a42600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a3c071650 con 0x7f9a3c0b06d0 2024-01-24T17:43:43.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.291+0000 7f9a42600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c071930 con 0x7f9a3c159620 2024-01-24T17:43:43.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.292+0000 7f9a3b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c159620 0x7f9a3c0ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:43.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.292+0000 7f9a3b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c159620 0x7f9a3c0ca7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:46090/0 (socket says 172.21.15.39:46090) 2024-01-24T17:43:43.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.292+0000 7f9a3b400640 1 -- 172.21.15.39:0/479039198 learned_addr learned my addr 172.21.15.39:0/479039198 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:43.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.292+0000 7f9a3be00640 1 --2- 172.21.15.39:0/479039198 >> v2:172.21.15.100:3300/0 conn(0x7f9a3c0b06d0 0x7f9a3c0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:43.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.293+0000 7f9a3be00640 1 -- 172.21.15.39:0/479039198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c159620 msgr2=0x7f9a3c0ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:43.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.293+0000 7f9a3be00640 1 --2- 172.21.15.39:0/479039198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c159620 0x7f9a3c0ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.293+0000 7f9a3be00640 1 -- 172.21.15.39:0/479039198 >> v1:172.21.15.100:6789/0 conn(0x7f9a3c0710e0 legacy=0x7f9a3c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:43.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.293+0000 7f9a3be00640 1 -- 172.21.15.39:0/479039198 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c071c10 con 0x7f9a3c0b06d0 2024-01-24T17:43:43.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.294+0000 7f9a3be00640 1 --2- 172.21.15.39:0/479039198 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0b06d0 0x7f9a3c0cc380 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009d90 tx=0x7f9a3402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ef3b32f4723cdb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:43.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.295+0000 7f9a3a000640 1 -- 172.21.15.39:0/479039198 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3402cc50 con 0x7f9a3c0b06d0 2024-01-24T17:43:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.295+0000 7f9a3a000640 1 -- 172.21.15.39:0/479039198 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3402cdb0 con 0x7f9a3c0b06d0 2024-01-24T17:43:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.295+0000 7f9a3a000640 1 -- 172.21.15.39:0/479039198 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a34034660 con 0x7f9a3c0b06d0 2024-01-24T17:43:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.295+0000 7f9a42600640 1 -- 172.21.15.39:0/479039198 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0b06d0 msgr2=0x7f9a3c0cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.295+0000 7f9a42600640 1 --2- 172.21.15.39:0/479039198 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0b06d0 0x7f9a3c0cc380 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009d90 tx=0x7f9a3402bea0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.296+0000 7f9a42600640 1 -- 172.21.15.39:0/479039198 shutdown_connections 2024-01-24T17:43:43.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.296+0000 7f9a42600640 1 --2- 172.21.15.39:0/479039198 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0b06d0 0x7f9a3c0cc380 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.296+0000 7f9a42600640 1 --2- 172.21.15.39:0/479039198 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c159620 0x7f9a3c0ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.296+0000 7f9a42600640 1 -- 172.21.15.39:0/479039198 >> 172.21.15.39:0/479039198 conn(0x7f9a3c06a3e0 msgr2=0x7f9a3c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:43.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.296+0000 7f9a42600640 1 -- 172.21.15.39:0/479039198 shutdown_connections 2024-01-24T17:43:43.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.296+0000 7f9a42600640 1 -- 172.21.15.39:0/479039198 wait complete. 2024-01-24T17:43:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.298+0000 7f9a42600640 1 Processor -- start 2024-01-24T17:43:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.298+0000 7f9a42600640 1 -- start start 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a42600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0710e0 0x7f9a3c145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a42600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c0b06d0 0x7f9a3c151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a42600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a3c159620 0x7f9a3c151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a42600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c0c8460 con 0x7f9a3c159620 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a42600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c0c7ea0 con 0x7f9a3c0710e0 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a42600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c0c8180 con 0x7f9a3c0b06d0 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a3c159620 0x7f9a3c151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a3c159620 0x7f9a3c151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51192/0 (socket says 172.21.15.39:51192) 2024-01-24T17:43:43.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3be00640 1 -- 172.21.15.39:0/2131994061 learned_addr learned my addr 172.21.15.39:0/2131994061 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3b400640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0710e0 0x7f9a3c145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3aa00640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c0b06d0 0x7f9a3c151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3be00640 1 -- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c0b06d0 msgr2=0x7f9a3c151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3be00640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c0b06d0 0x7f9a3c151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3be00640 1 -- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0710e0 msgr2=0x7f9a3c145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.299+0000 7f9a3be00640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0710e0 0x7f9a3c145ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.300+0000 7f9a3be00640 1 -- 172.21.15.39:0/2131994061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a34009a70 con 0x7f9a3c159620 2024-01-24T17:43:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.300+0000 7f9a3be00640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a3c159620 0x7f9a3c151f50 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009eb0 tx=0x7f9a34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:43.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.300+0000 7f9a33400640 1 -- 172.21.15.39:0/2131994061 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3403e610 con 0x7f9a3c159620 2024-01-24T17:43:43.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.300+0000 7f9a33400640 1 -- 172.21.15.39:0/2131994061 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3403e770 con 0x7f9a3c159620 2024-01-24T17:43:43.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.300+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c0b1db0 con 0x7f9a3c159620 2024-01-24T17:43:43.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.300+0000 7f9a33400640 1 -- 172.21.15.39:0/2131994061 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3403eb20 con 0x7f9a3c159620 2024-01-24T17:43:43.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.300+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3c146020 con 0x7f9a3c159620 2024-01-24T17:43:43.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.302+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3c06c840 con 0x7f9a3c159620 2024-01-24T17:43:43.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.303+0000 7f9a33400640 1 -- 172.21.15.39:0/2131994061 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9a3403b050 con 0x7f9a3c159620 2024-01-24T17:43:43.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.304+0000 7f9a33400640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a1406a430 0x7f9a1406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:43.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.304+0000 7f9a33400640 1 -- 172.21.15.39:0/2131994061 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9a340ab250 con 0x7f9a3c159620 2024-01-24T17:43:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.305+0000 7f9a3b400640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a1406a430 0x7f9a1406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:43.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.306+0000 7f9a3b400640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a1406a430 0x7f9a1406c8f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009ac0 tx=0x7f9a2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:43.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.308+0000 7f9a33400640 1 -- 172.21.15.39:0/2131994061 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9a340767b0 con 0x7f9a3c159620 2024-01-24T17:43:43.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.517+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9a3c0b22d0 con 0x7f9a3c159620 2024-01-24T17:43:43.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.518+0000 7f9a33400640 1 -- 172.21.15.39:0/2131994061 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9a3c0b22d0 con 0x7f9a3c159620 2024-01-24T17:43:43.518 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:43:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a1406a430 msgr2=0x7f9a1406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a1406a430 0x7f9a1406c8f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009ac0 tx=0x7f9a2c008040 comp rx=0 tx=0).stop 2024-01-24T17:43:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a3c159620 msgr2=0x7f9a3c151f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a3c159620 0x7f9a3c151f50 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009eb0 tx=0x7f9a34004a40 comp rx=0 tx=0).stop 2024-01-24T17:43:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 shutdown_connections 2024-01-24T17:43:43.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a1406a430 0x7f9a1406c8f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a3c159620 0x7f9a3c151f50 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a3c0b06d0 0x7f9a3c151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 --2- 172.21.15.39:0/2131994061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a3c0710e0 0x7f9a3c145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:43.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.520+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 >> 172.21.15.39:0/2131994061 conn(0x7f9a3c06a3e0 msgr2=0x7f9a3c14fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:43.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.521+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 shutdown_connections 2024-01-24T17:43:43.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:43.521+0000 7f9a42600640 1 -- 172.21.15.39:0/2131994061 wait complete. 2024-01-24T17:43:43.541 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:43:43.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:43.541 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:43:43.634 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[52228]: fuse finished with error 0 and tester_r 0 2024-01-24T17:43:43.640 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:43:49.742 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:43:49.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:49.742 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:43:49.805 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-01-24T17:43:50.043 INFO:teuthology.orchestra.run.smithi039.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":104,"msgr_send_messages":83,"msgr_recv_bytes":125896,"msgr_send_bytes":40710,"msgr_created_connections":10,"msgr_active_connections":1,"msgr_running_total_time":0.037801245,"msgr_running_send_time":0.007644971,"msgr_running_recv_time":0.092266611,"msgr_running_fast_dispatch_time":0.002274665,"msgr_send_messages_queue_lat":{"avgcount":81,"sum":0.056505412,"avgtime":0.000697597},"msgr_handle_ack_lat":{"avgcount":41,"sum":0.000008903,"avgtime":0.000000217},"msgr_recv_encrypted_bytes":114848,"msgr_send_encrypted_bytes":18288},"AsyncMessenger::Worker-1":{"msgr_recv_messages":45,"msgr_send_messages":41,"msgr_recv_bytes":15034,"msgr_send_bytes":92125,"msgr_created_connections":7,"msgr_active_connections":3,"msgr_running_total_time":0.027950212,"msgr_running_send_time":0.003912108,"msgr_running_recv_time":0.021545838,"msgr_running_fast_dispatch_time":0.001980055,"msgr_send_messages_queue_lat":{"avgcount":39,"sum":0.245977883,"avgtime":0.006307125},"msgr_handle_ack_lat":{"avgcount":27,"sum":0.000006643,"avgtime":0.000000246},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":77520},"AsyncMessenger::Worker-2":{"msgr_recv_messages":31,"msgr_send_messages":34,"msgr_recv_bytes":8162,"msgr_send_bytes":16159,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":0.007845889,"msgr_running_send_time":0.000915447,"msgr_running_recv_time":0.005496944,"msgr_running_fast_dispatch_time":0.001021517,"msgr_send_messages_queue_lat":{"avgcount":32,"sum":0.123499397,"avgtime":0.003859356},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":56,"sum":0.003333971,"avgtime":0.000059535}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2,"sum":0.000019244,"avgtime":0.000009622}},"mds":{"request":4,"reply":4,"reply_latency":{"avgcount":4,"sum":0.001479525,"avgtime":0.000369881},"slow_reply":0,"forward":0,"dir_fetch_complete":0,"dir_fetch_keys":0,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":0,"inodes_bottom":10,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":0,"caps":0,"subtrees":2,"traverse":7,"traverse_hit":7,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":0,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":0,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":8,"evex":0,"evtrm":0,"ev":8,"evexg":0,"evexd":0,"segadd":1,"segex":0,"segtrm":0,"seg":1,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4197760,"rdpos":0,"jlat":{"avgcount":8,"sum":0.034526051,"avgtime":0.004315756},"replayed":0},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":0,"cap+":3,"cap-":3,"rss":40192,"heap":207196},"mds_server":{"dispatch_client_request":4,"dispatch_server_request":0,"handle_client_request":4,"handle_client_session":12,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":3,"sum":0.001078731,"avgtime":0.000359577},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":1,"sum":0.000400794,"avgtime":0.000400794},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":0,"session_add":3,"session_remove":3,"sessions_open":0,"sessions_stale":0,"total_load":3,"average_load":3,"avg_session_uptime":6,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":213643,"buffer_anon_items":61,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5696,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51440,"mds_co_items":211,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":48,"op_send_bytes":14756,"op_resend":37,"op_reply":48,"op_latency":{"avgcount":48,"sum":1.309973590,"avgtime":0.027291116},"op_inflight":0,"oplen_avg":{"avgcount":48,"sum":72},"op":48,"op_r":0,"op_w":48,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":11,"osdop_read":0,"osdop_write":8,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":22,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":36,"map_full":0,"map_inc":3,"osd_sessions":6,"osd_session_open":6,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":4},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":173,"get_sum":127733,"get_or_fail_fail":0,"get_or_fail_success":173,"take":0,"take_sum":0,"put":173,"put_sum":127733,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":48,"take_sum":14756,"put":48,"put_sum":14756,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":48,"take_sum":48,"put":48,"put_sum":48,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a2aa08a0a0":{"val":0,"max":3758096384,"get_started":0,"get":8,"get_sum":3456,"get_or_fail_fail":0,"get_or_fail_success":8,"take":0,"take_sum":0,"put":8,"put_sum":3456,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-01-24T17:43:50.044 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 10, "msgr_handle_ack_lat": { "avgcount": 41, "avgtime": 2.17e-07, "sum": 8.903e-06 }, "msgr_recv_bytes": 125896, "msgr_recv_encrypted_bytes": 114848, "msgr_recv_messages": 104, "msgr_running_fast_dispatch_time": 0.002274665, "msgr_running_recv_time": 0.092266611, "msgr_running_send_time": 0.007644971, "msgr_running_total_time": 0.037801245, "msgr_send_bytes": 40710, "msgr_send_encrypted_bytes": 18288, "msgr_send_messages": 83, "msgr_send_messages_queue_lat": { "avgcount": 81, "avgtime": 0.000697597, "sum": 0.056505412 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 27, "avgtime": 2.46e-07, "sum": 6.643e-06 }, "msgr_recv_bytes": 15034, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 45, "msgr_running_fast_dispatch_time": 0.001980055, "msgr_running_recv_time": 0.021545838, "msgr_running_send_time": 0.003912108, "msgr_running_total_time": 0.027950212, "msgr_send_bytes": 92125, "msgr_send_encrypted_bytes": 77520, "msgr_send_messages": 41, "msgr_send_messages_queue_lat": { "avgcount": 39, "avgtime": 0.006307125, "sum": 0.245977883 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 8162, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 31, "msgr_running_fast_dispatch_time": 0.001021517, "msgr_running_recv_time": 0.005496944, "msgr_running_send_time": 0.000915447, "msgr_running_total_time": 0.007845889, "msgr_send_bytes": 16159, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 34, "msgr_send_messages_queue_lat": { "avgcount": 32, "avgtime": 0.003859356, "sum": 0.123499397 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 56, "avgtime": 5.9535e-05, "sum": 0.003333971 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2, "avgtime": 9.622e-06, "sum": 1.9244e-05 }, "queue_len": 0 }, "mds": { "caps": 0, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 0, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 0, "dir_fetch_keys": 0, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 0, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 10, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 0, "inodes_with_caps": 0, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 4, "reply_latency": { "avgcount": 4, "avgtime": 0.000369881, "sum": 0.001479525 }, "request": 4, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 7, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 7, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8, "evadd": 8, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 8, "avgtime": 0.004315756, "sum": 0.034526051 }, "rdpos": 0, "replayed": 0, "seg": 1, "segadd": 1, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 4197760 }, "mds_mem": { "cap": 0, "cap+": 3, "cap-": 3, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 207196, "ino": 13, "ino+": 13, "ino-": 0, "rss": 40192 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4, "dispatch_server_request": 0, "handle_client_request": 4, "handle_client_session": 12, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 3, "avgtime": 0.000359577, "sum": 0.001078731 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 1, "avgtime": 0.000400794, "sum": 0.000400794 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 3, "avg_session_uptime": 6, "mdthresh_evicted": 0, "session_add": 3, "session_count": 0, "session_remove": 3, "sessions_open": 0, "sessions_stale": 0, "total_load": 3 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 213643, "buffer_anon_items": 61, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51440, "mds_co_items": 211, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5696, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 36, "map_full": 0, "map_inc": 3, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 48, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 48, "avgtime": 0.027291116, "sum": 1.30997359 }, "op_pg": 0, "op_r": 0, "op_reply": 48, "op_resend": 37, "op_rmw": 0, "op_send": 48, "op_send_bytes": 14756, "op_w": 48, "oplen_avg": { "avgcount": 48, "sum": 72 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 6, "osd_sessions": 6, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 11, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 22, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 8, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 1, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 4 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 173, "get_or_fail_fail": 0, "get_or_fail_success": 173, "get_started": 0, "get_sum": 127733, "max": 104857600, "put": 173, "put_sum": 127733, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 48, "put_sum": 14756, "take": 48, "take_sum": 14756, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 48, "put_sum": 48, "take": 48, "take_sum": 48, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a2aa08a0a0": { "get": 8, "get_or_fail_fail": 0, "get_or_fail_success": 8, "get_started": 0, "get_sum": 3456, "max": 3758096384, "put": 8, "put_sum": 3456, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-01-24T17:43:50.044 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-01-24T17:43:50.045 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:43:50.045 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:43:50.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:43:50.232 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:43:50.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:43:50.286 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:43:50.286 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:43:50.286 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:50.335 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:43:50.335 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:43:50.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:50.335 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:43:50.450 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-24T17:43:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:43:50.452 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:43:50.453 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:50.508 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:43:50.509 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:43:50.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:50.509 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:43:50.560 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-24T17:43:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:43:50.561 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:43:50.561 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:50.623 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:43:50.623 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:43:50.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:50.623 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-24T17:43:50.682 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-24T17:43:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:43:50.683 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:43:50.684 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:43:50.740 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:43:50.740 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:43:50.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:43:50.741 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-24T17:43:50.794 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-24T17:43:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:43:50.795 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:43:51.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.026+0000 7f6912a00640 1 Processor -- start 2024-01-24T17:43:51.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.026+0000 7f6912a00640 1 -- start start 2024-01-24T17:43:51.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.028+0000 7f6912a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f690c15a950 0x7f690c0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.028+0000 7f6912a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0caaa0 0x7f690c0caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.028+0000 7f6912a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690c0c9180 con 0x7f690c0cb490 2024-01-24T17:43:51.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.028+0000 7f6912a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f690c0b43d0 con 0x7f690c15a950 2024-01-24T17:43:51.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.028+0000 7f6912a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0c8f70 con 0x7f690c0caaa0 2024-01-24T17:43:51.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.028+0000 7f690be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f690c15a950 0x7f690c0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.028+0000 7f6910a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f690c0cb490 0x7f690c0affc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41876/0 (socket says 172.21.15.39:41876) 2024-01-24T17:43:51.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.028+0000 7f6910a00640 1 -- 172.21.15.39:0/1115276441 learned_addr learned my addr 172.21.15.39:0/1115276441 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:51.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.029+0000 7f690be00640 1 -- 172.21.15.39:0/1115276441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0caaa0 msgr2=0x7f690c0caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:51.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.029+0000 7f690be00640 1 --2- 172.21.15.39:0/1115276441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0caaa0 0x7f690c0caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.029+0000 7f690be00640 1 -- 172.21.15.39:0/1115276441 >> v1:172.21.15.39:6789/0 conn(0x7f690c0cb490 legacy=0x7f690c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.029+0000 7f690be00640 1 -- 172.21.15.39:0/1115276441 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c0c9460 con 0x7f690c15a950 2024-01-24T17:43:51.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.030+0000 7f690be00640 1 --2- 172.21.15.39:0/1115276441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0ca560 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f68fc0097a0 tx=0x7f68fc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251f4e013ff7b8aa server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.030+0000 7f690b400640 1 -- 172.21.15.39:0/1115276441 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc02d040 con 0x7f690c15a950 2024-01-24T17:43:51.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.030+0000 7f690b400640 1 -- 172.21.15.39:0/1115276441 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc031810 con 0x7f690c15a950 2024-01-24T17:43:51.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.030+0000 7f690b400640 1 -- 172.21.15.39:0/1115276441 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc031ba0 con 0x7f690c15a950 2024-01-24T17:43:51.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.031+0000 7f6912a00640 1 -- 172.21.15.39:0/1115276441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 msgr2=0x7f690c0ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.031+0000 7f6912a00640 1 --2- 172.21.15.39:0/1115276441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0ca560 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f68fc0097a0 tx=0x7f68fc02b8b0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.031+0000 7f6912a00640 1 -- 172.21.15.39:0/1115276441 shutdown_connections 2024-01-24T17:43:51.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.031+0000 7f6912a00640 1 --2- 172.21.15.39:0/1115276441 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0caaa0 0x7f690c0caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.031+0000 7f6912a00640 1 --2- 172.21.15.39:0/1115276441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0ca560 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.031+0000 7f6912a00640 1 -- 172.21.15.39:0/1115276441 >> 172.21.15.39:0/1115276441 conn(0x7f690c06a3e0 msgr2=0x7f690c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:51.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.031+0000 7f6912a00640 1 -- 172.21.15.39:0/1115276441 shutdown_connections 2024-01-24T17:43:51.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.031+0000 7f6912a00640 1 -- 172.21.15.39:0/1115276441 wait complete. 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.033+0000 7f6912a00640 1 Processor -- start 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.033+0000 7f6912a00640 1 -- start start 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.034+0000 7f6912a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0caaa0 0x7f690c157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.034+0000 7f6912a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0cb490 0x7f690c0b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.034+0000 7f6912a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.034+0000 7f6912a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0b3d20 con 0x7f690c15a950 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.034+0000 7f6912a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c157b60 con 0x7f690c0caaa0 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.034+0000 7f6912a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f690c0b3110 con 0x7f690c0cb490 2024-01-24T17:43:51.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0c5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51220/0 (socket says 172.21.15.39:51220) 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 -- 172.21.15.39:0/2634267566 learned_addr learned my addr 172.21.15.39:0/2634267566 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6903200640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0cb490 0x7f690c0b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 -- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0cb490 msgr2=0x7f690c0b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f690be00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0caaa0 0x7f690c157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0cb490 0x7f690c0b26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 -- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0caaa0 msgr2=0x7f690c157620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0caaa0 0x7f690c157620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 -- 172.21.15.39:0/2634267566 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc009400 con 0x7f690c15a950 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f690be00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0caaa0 0x7f690c157620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.035+0000 7f6910a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0c5fa0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f690400cbf0 tx=0x7f690400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.036+0000 7f6908c00640 1 -- 172.21.15.39:0/2634267566 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690400ad40 con 0x7f690c15a950 2024-01-24T17:43:51.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.036+0000 7f6908c00640 1 -- 172.21.15.39:0/2634267566 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6904012400 con 0x7f690c15a950 2024-01-24T17:43:51.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.036+0000 7f6908c00640 1 -- 172.21.15.39:0/2634267566 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690400aea0 con 0x7f690c15a950 2024-01-24T17:43:51.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.036+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c0b33f0 con 0x7f690c15a950 2024-01-24T17:43:51.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.036+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c0b39b0 con 0x7f690c15a950 2024-01-24T17:43:51.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.037+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690c0b3ed0 con 0x7f690c15a950 2024-01-24T17:43:51.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.038+0000 7f6908c00640 1 -- 172.21.15.39:0/2634267566 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6904007070 con 0x7f690c15a950 2024-01-24T17:43:51.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.039+0000 7f6908c00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68e406a430 0x7f68e406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.039+0000 7f6908c00640 1 -- 172.21.15.39:0/2634267566 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6904088840 con 0x7f690c15a950 2024-01-24T17:43:51.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.040+0000 7f690be00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68e406a430 0x7f68e406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.041+0000 7f690be00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68e406a430 0x7f68e406c8f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f68fc036000 tx=0x7f68fc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.044+0000 7f6908c00640 1 -- 172.21.15.39:0/2634267566 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f690c0b3ed0 con 0x7f690c15a950 2024-01-24T17:43:51.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.275+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f690c073ef0 con 0x7f690c15a950 2024-01-24T17:43:51.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.277+0000 7f6908c00640 1 -- 172.21.15.39:0/2634267566 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f6904017090 con 0x7f690c15a950 2024-01-24T17:43:51.278 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:51.279 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4688,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/648542221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":648542221},{"type":"v1","addr":"172.21.15.100:6837","nonce":648542221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/188830546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":188830546},{"type":"v1","addr":"172.21.15.100:6835","nonce":188830546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/4294097622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4294097622},{"type":"v1","addr":"172.21.15.39:6834","nonce":4294097622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:42:56.701344+0000","modified":"2024-01-24T17:43:00.247932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5175":{"gid":5175,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/3577904963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3577904963},{"type":"v1","addr":"172.21.15.39:6835","nonce":3577904963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-24T17:43:51.279 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-01-24T17:43:51.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68e406a430 msgr2=0x7f68e406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68e406a430 0x7f68e406c8f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f68fc036000 tx=0x7f68fc03c040 comp rx=0 tx=0).stop 2024-01-24T17:43:51.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 msgr2=0x7f690c0c5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0c5fa0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f690400cbf0 tx=0x7f690400a430 comp rx=0 tx=0).stop 2024-01-24T17:43:51.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 shutdown_connections 2024-01-24T17:43:51.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68e406a430 0x7f68e406c8f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f690c15a950 0x7f690c0c5fa0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f690c0cb490 0x7f690c0b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 --2- 172.21.15.39:0/2634267566 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c0caaa0 0x7f690c157620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.279+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 >> 172.21.15.39:0/2634267566 conn(0x7f690c06a3e0 msgr2=0x7f690c0adf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:51.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.280+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 shutdown_connections 2024-01-24T17:43:51.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.280+0000 7f6912a00640 1 -- 172.21.15.39:0/2634267566 wait complete. 2024-01-24T17:43:51.301 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:51.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.517+0000 7f30a1600640 1 Processor -- start 2024-01-24T17:43:51.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.517+0000 7f30a1600640 1 -- start start 2024-01-24T17:43:51.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f30a1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f309c15ca30 0x7f309c0cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f30a1600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f309c0cd3c0 0x7f309c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f30a1600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c90d0 0x7f309c0caf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f30a1600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c0c9c10 con 0x7f309c15ca30 2024-01-24T17:43:51.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f30a1600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c0c9650 con 0x7f309c0cd3c0 2024-01-24T17:43:51.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f30a1600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0c9930 con 0x7f309c0c90d0 2024-01-24T17:43:51.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f309aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f309c15ca30 0x7f309c0cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f309aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f309c15ca30 0x7f309c0cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51228/0 (socket says 172.21.15.39:51228) 2024-01-24T17:43:51.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f309aa00640 1 -- 172.21.15.39:0/3525842888 learned_addr learned my addr 172.21.15.39:0/3525842888 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f309a000640 1 --2- 172.21.15.39:0/3525842888 >> v2:172.21.15.100:3300/0 conn(0x7f309c0cd3c0 0x7f309c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.519+0000 7f309b400640 1 --2- 172.21.15.39:0/3525842888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c90d0 0x7f309c0caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.520+0000 7f309aa00640 1 -- 172.21.15.39:0/3525842888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c90d0 msgr2=0x7f309c0caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.520+0000 7f309aa00640 1 --2- 172.21.15.39:0/3525842888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c90d0 0x7f309c0caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.520+0000 7f309aa00640 1 -- 172.21.15.39:0/3525842888 >> v2:172.21.15.100:3300/0 conn(0x7f309c0cd3c0 msgr2=0x7f309c0c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.520+0000 7f309aa00640 1 --2- 172.21.15.39:0/3525842888 >> v2:172.21.15.100:3300/0 conn(0x7f309c0cd3c0 0x7f309c0c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.520+0000 7f309aa00640 1 -- 172.21.15.39:0/3525842888 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0cc680 con 0x7f309c15ca30 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.520+0000 7f309a000640 1 --2- 172.21.15.39:0/3525842888 >> v2:172.21.15.100:3300/0 conn(0x7f309c0cd3c0 0x7f309c0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.520+0000 7f309b400640 1 --2- 172.21.15.39:0/3525842888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c90d0 0x7f309c0caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.521+0000 7f309aa00640 1 --2- 172.21.15.39:0/3525842888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0cce80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f309c06e230 tx=0x7f308c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbdeb8e647023407 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.521+0000 7f3099600640 1 -- 172.21.15.39:0/3525842888 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c031870 con 0x7f309c15ca30 2024-01-24T17:43:51.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.521+0000 7f3099600640 1 -- 172.21.15.39:0/3525842888 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c0319d0 con 0x7f309c15ca30 2024-01-24T17:43:51.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.521+0000 7f3099600640 1 -- 172.21.15.39:0/3525842888 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c031d60 con 0x7f309c15ca30 2024-01-24T17:43:51.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 -- 172.21.15.39:0/3525842888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 msgr2=0x7f309c0cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 --2- 172.21.15.39:0/3525842888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0cce80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f309c06e230 tx=0x7f308c02b910 comp rx=0 tx=0).stop 2024-01-24T17:43:51.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 -- 172.21.15.39:0/3525842888 shutdown_connections 2024-01-24T17:43:51.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 --2- 172.21.15.39:0/3525842888 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c90d0 0x7f309c0caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 --2- 172.21.15.39:0/3525842888 >> v2:172.21.15.100:3300/0 conn(0x7f309c0cd3c0 0x7f309c0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 --2- 172.21.15.39:0/3525842888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0cce80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 -- 172.21.15.39:0/3525842888 >> 172.21.15.39:0/3525842888 conn(0x7f309c06a3e0 msgr2=0x7f309c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:51.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 -- 172.21.15.39:0/3525842888 shutdown_connections 2024-01-24T17:43:51.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.522+0000 7f30a1600640 1 -- 172.21.15.39:0/3525842888 wait complete. 2024-01-24T17:43:51.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.524+0000 7f30a1600640 1 Processor -- start 2024-01-24T17:43:51.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.524+0000 7f30a1600640 1 -- start start 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f30a1600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f309c0c90d0 0x7f309c1f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f30a1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f30a1600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c6ba0 0x7f309c0c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f30a1600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c146270 con 0x7f309c15ca30 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f30a1600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c1466f0 con 0x7f309c0c90d0 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f30a1600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c146900 con 0x7f309c0c6ba0 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f309a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f309a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0c6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51234/0 (socket says 172.21.15.39:51234) 2024-01-24T17:43:51.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.525+0000 7f309a000640 1 -- 172.21.15.39:0/2329263662 learned_addr learned my addr 172.21.15.39:0/2329263662 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:51.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309b400640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c6ba0 0x7f309c0c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309aa00640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f309c0c90d0 0x7f309c1f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309a000640 1 -- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c6ba0 msgr2=0x7f309c0c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309a000640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c6ba0 0x7f309c0c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309a000640 1 -- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f309c0c90d0 msgr2=0x7f309c1f1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309a000640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f309c0c90d0 0x7f309c1f1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309a000640 1 -- 172.21.15.39:0/2329263662 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c0094d0 con 0x7f309c15ca30 2024-01-24T17:43:51.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309aa00640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f309c0c90d0 0x7f309c1f1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:51.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f309a000640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0c6660 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f308400c7f0 tx=0x7f308400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.526+0000 7f3092a00640 1 -- 172.21.15.39:0/2329263662 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084007a30 con 0x7f309c15ca30 2024-01-24T17:43:51.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.527+0000 7f3092a00640 1 -- 172.21.15.39:0/2329263662 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3084007b90 con 0x7f309c15ca30 2024-01-24T17:43:51.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.527+0000 7f3092a00640 1 -- 172.21.15.39:0/2329263662 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308400ce40 con 0x7f309c15ca30 2024-01-24T17:43:51.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.527+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c146be0 con 0x7f309c15ca30 2024-01-24T17:43:51.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.527+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c1471a0 con 0x7f309c15ca30 2024-01-24T17:43:51.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.528+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c1473c0 con 0x7f309c15ca30 2024-01-24T17:43:51.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.529+0000 7f3092a00640 1 -- 172.21.15.39:0/2329263662 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f30840113f0 con 0x7f309c15ca30 2024-01-24T17:43:51.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.530+0000 7f3092a00640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307406a430 0x7f307406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.530+0000 7f3092a00640 1 -- 172.21.15.39:0/2329263662 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3084088bd0 con 0x7f309c15ca30 2024-01-24T17:43:51.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.530+0000 7f309aa00640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307406a430 0x7f307406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.531+0000 7f309aa00640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307406a430 0x7f307406c8f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f308c036000 tx=0x7f308c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.534+0000 7f3092a00640 1 -- 172.21.15.39:0/2329263662 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f30840540a0 con 0x7f309c15ca30 2024-01-24T17:43:51.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.728+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f309c073ef0 con 0x7f309c15ca30 2024-01-24T17:43:51.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.730+0000 7f3092a00640 1 -- 172.21.15.39:0/2329263662 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16290 (secure 0 0 0) 0x7f308401f090 con 0x7f309c15ca30 2024-01-24T17:43:51.732 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:51.732 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:43:16.261174+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.1c","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.32","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307406a430 msgr2=0x7f307406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307406a430 0x7f307406c8f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f308c036000 tx=0x7f308c03c040 comp rx=0 tx=0).stop 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 msgr2=0x7f309c0c6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0c6660 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f308400c7f0 tx=0x7f308400cbf0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 shutdown_connections 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f309c0c6ba0 0x7f309c0c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307406a430 0x7f307406c8f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f309c15ca30 0x7f309c0c6660 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 --2- 172.21.15.39:0/2329263662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f309c0c90d0 0x7f309c1f1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 >> 172.21.15.39:0/2329263662 conn(0x7f309c06a3e0 msgr2=0x7f309c0ab8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 shutdown_connections 2024-01-24T17:43:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.733+0000 7f30a1600640 1 -- 172.21.15.39:0/2329263662 wait complete. 2024-01-24T17:43:51.755 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:42:54.883732+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:42:55.727240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:43:51.755 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:43:51.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.977+0000 7f311f600640 1 Processor -- start 2024-01-24T17:43:51.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.978+0000 7f311f600640 1 -- start start 2024-01-24T17:43:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.979+0000 7f311f600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f31180b00e0 0x7f31180b0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.979+0000 7f311f600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31180b0a50 con 0x7f311806bfa0 2024-01-24T17:43:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.979+0000 7f311f600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31180b1bc0 con 0x7f31181597f0 2024-01-24T17:43:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.979+0000 7f311f600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31180ca0d0 con 0x7f31180b00e0 2024-01-24T17:43:51.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.980+0000 7f311d600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f31180b00e0 0x7f31180b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.980+0000 7f3117e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f311806bfa0 0x7f31180b14b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41886/0 (socket says 172.21.15.39:41886) 2024-01-24T17:43:51.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.980+0000 7f3117e00640 1 -- 172.21.15.39:0/567372757 learned_addr learned my addr 172.21.15.39:0/567372757 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:51.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.980+0000 7f311d600640 1 -- 172.21.15.39:0/567372757 >> v1:172.21.15.100:6789/0 conn(0x7f31181597f0 legacy=0x7f311806d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:51.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.980+0000 7f311d600640 1 -- 172.21.15.39:0/567372757 >> v1:172.21.15.39:6789/0 conn(0x7f311806bfa0 legacy=0x7f31180b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.980+0000 7f311d600640 1 -- 172.21.15.39:0/567372757 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311806c4d0 con 0x7f31180b00e0 2024-01-24T17:43:51.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.981+0000 7f311d600640 1 --2- 172.21.15.39:0/567372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31180b0510 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3110009d90 tx=0x7f311002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2648020dece876 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f3117400640 1 -- 172.21.15.39:0/567372757 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311002cc60 con 0x7f31180b00e0 2024-01-24T17:43:51.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f3117400640 1 -- 172.21.15.39:0/567372757 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311002cdc0 con 0x7f31180b00e0 2024-01-24T17:43:51.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f3117400640 1 -- 172.21.15.39:0/567372757 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110034660 con 0x7f31180b00e0 2024-01-24T17:43:51.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f311f600640 1 -- 172.21.15.39:0/567372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 msgr2=0x7f31180b0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f311f600640 1 --2- 172.21.15.39:0/567372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31180b0510 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3110009d90 tx=0x7f311002beb0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f311f600640 1 -- 172.21.15.39:0/567372757 shutdown_connections 2024-01-24T17:43:51.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f311f600640 1 --2- 172.21.15.39:0/567372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31180b0510 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f311f600640 1 -- 172.21.15.39:0/567372757 >> 172.21.15.39:0/567372757 conn(0x7f31180cb0e0 msgr2=0x7f31180cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f311f600640 1 -- 172.21.15.39:0/567372757 shutdown_connections 2024-01-24T17:43:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.982+0000 7f311f600640 1 -- 172.21.15.39:0/567372757 wait complete. 2024-01-24T17:43:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.984+0000 7f311f600640 1 Processor -- start 2024-01-24T17:43:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.984+0000 7f311f600640 1 -- start start 2024-01-24T17:43:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.985+0000 7f311f600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f311806bfa0 0x7f311815c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.985+0000 7f311f600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31181f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.985+0000 7f311f600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31181597f0 0x7f31180ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.985+0000 7f311f600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180c5730 con 0x7f31180b00e0 2024-01-24T17:43:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.985+0000 7f311f600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31181f38e0 con 0x7f31181597f0 2024-01-24T17:43:51.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.985+0000 7f311f600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31181f2a10 con 0x7f311806bfa0 2024-01-24T17:43:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.986+0000 7f3117e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31181f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.986+0000 7f311cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f311806bfa0 0x7f311815c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.986+0000 7f3117e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31181f33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51248/0 (socket says 172.21.15.39:51248) 2024-01-24T17:43:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.986+0000 7f3117e00640 1 -- 172.21.15.39:0/2222216306 learned_addr learned my addr 172.21.15.39:0/2222216306 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.986+0000 7f311d600640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31181597f0 0x7f31180ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.986+0000 7f3117e00640 1 -- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f311806bfa0 msgr2=0x7f311815c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.986+0000 7f3117e00640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f311806bfa0 0x7f311815c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.986+0000 7f3117e00640 1 -- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31181597f0 msgr2=0x7f31180ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f3117e00640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31181597f0 0x7f31180ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f3117e00640 1 -- 172.21.15.39:0/2222216306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110009a70 con 0x7f31180b00e0 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f311d600640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31181597f0 0x7f31180ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f3117e00640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31181f33a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f310000c480 tx=0x7f310000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f3114c00640 1 -- 172.21.15.39:0/2222216306 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100005ce0 con 0x7f31180b00e0 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f3114c00640 1 -- 172.21.15.39:0/2222216306 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3100005e40 con 0x7f31180b00e0 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31181f2cf0 con 0x7f31180b00e0 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f3114c00640 1 -- 172.21.15.39:0/2222216306 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31000072b0 con 0x7f31180b00e0 2024-01-24T17:43:51.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.987+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31180c5940 con 0x7f31180b00e0 2024-01-24T17:43:51.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.989+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3118073ef0 con 0x7f31180b00e0 2024-01-24T17:43:51.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.989+0000 7f3114c00640 1 -- 172.21.15.39:0/2222216306 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3100007410 con 0x7f31180b00e0 2024-01-24T17:43:51.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.990+0000 7f3114c00640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f30f006a430 0x7f30f006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:51.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.990+0000 7f3114c00640 1 -- 172.21.15.39:0/2222216306 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3100087ed0 con 0x7f31180b00e0 2024-01-24T17:43:51.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.991+0000 7f311cc00640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f30f006a430 0x7f30f006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:51.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.992+0000 7f311cc00640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f30f006a430 0x7f30f006c8f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3118159080 tx=0x7f3108009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:51.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:51.995+0000 7f3114c00640 1 -- 172.21.15.39:0/2222216306 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f31000534e0 con 0x7f31180b00e0 2024-01-24T17:43:52.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.188+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f311815c7c0 con 0x7f31180b00e0 2024-01-24T17:43:52.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.189+0000 7f3114c00640 1 -- 172.21.15.39:0/2222216306 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31000170a0 con 0x7f31180b00e0 2024-01-24T17:43:52.191 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f30f006a430 msgr2=0x7f30f006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f30f006a430 0x7f30f006c8f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3118159080 tx=0x7f3108009060 comp rx=0 tx=0).stop 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 msgr2=0x7f31181f33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31181f33a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f310000c480 tx=0x7f310000c940 comp rx=0 tx=0).stop 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 shutdown_connections 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31181597f0 0x7f31180ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f30f006a430 0x7f30f006c8f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31180b00e0 0x7f31181f33a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 --2- 172.21.15.39:0/2222216306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f311806bfa0 0x7f311815c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 >> 172.21.15.39:0/2222216306 conn(0x7f31180cb0e0 msgr2=0x7f31180709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:52.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 shutdown_connections 2024-01-24T17:43:52.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.192+0000 7f311f600640 1 -- 172.21.15.39:0/2222216306 wait complete. 2024-01-24T17:43:52.214 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:43:52.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.428+0000 7f3573e00640 1 Processor -- start 2024-01-24T17:43:52.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.428+0000 7f3573e00640 1 -- start start 2024-01-24T17:43:52.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.429+0000 7f3573e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f357415fed0 0x7f35740cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.429+0000 7f3573e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f357406a380 0x7f35740c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.430+0000 7f3573e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3574071320 con 0x7f35740cb830 2024-01-24T17:43:52.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.430+0000 7f3573e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3574071600 con 0x7f357415fed0 2024-01-24T17:43:52.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.430+0000 7f3573e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35740718e0 con 0x7f357406a380 2024-01-24T17:43:52.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.430+0000 7f3572a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f357415fed0 0x7f35740cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.430+0000 7f3573400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f357406a380 0x7f35740c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.430+0000 7f3572a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f357415fed0 0x7f35740cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:41360/0 (socket says 172.21.15.39:41360) 2024-01-24T17:43:52.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.430+0000 7f3572a00640 1 -- 172.21.15.39:0/1521961124 learned_addr learned my addr 172.21.15.39:0/1521961124 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:52.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.431+0000 7f3573400640 1 -- 172.21.15.39:0/1521961124 >> v2:172.21.15.100:3300/0 conn(0x7f357415fed0 msgr2=0x7f35740cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:52.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.431+0000 7f3573400640 1 --2- 172.21.15.39:0/1521961124 >> v2:172.21.15.100:3300/0 conn(0x7f357415fed0 0x7f35740cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.431+0000 7f3573400640 1 -- 172.21.15.39:0/1521961124 >> v1:172.21.15.100:6789/0 conn(0x7f35740cb830 legacy=0x7f3574069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.431+0000 7f3573400640 1 -- 172.21.15.39:0/1521961124 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3574071bc0 con 0x7f357406a380 2024-01-24T17:43:52.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.431+0000 7f3572a00640 1 --2- 172.21.15.39:0/1521961124 >> v2:172.21.15.100:3300/0 conn(0x7f357415fed0 0x7f35740cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:43:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.433+0000 7f3573400640 1 --2- 172.21.15.39:0/1521961124 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f357406a380 0x7f35740c9b60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f356c009d90 tx=0x7f356c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24e20b0854fcf171 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.433+0000 7f3572000640 1 -- 172.21.15.39:0/1521961124 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c02cc50 con 0x7f357406a380 2024-01-24T17:43:52.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.433+0000 7f3572000640 1 -- 172.21.15.39:0/1521961124 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c02cdb0 con 0x7f357406a380 2024-01-24T17:43:52.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.433+0000 7f3572000640 1 -- 172.21.15.39:0/1521961124 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c034660 con 0x7f357406a380 2024-01-24T17:43:52.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.434+0000 7f3573e00640 1 -- 172.21.15.39:0/1521961124 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f357406a380 msgr2=0x7f35740c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.434+0000 7f3573e00640 1 --2- 172.21.15.39:0/1521961124 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f357406a380 0x7f35740c9b60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f356c009d90 tx=0x7f356c02bea0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.434+0000 7f3573e00640 1 -- 172.21.15.39:0/1521961124 shutdown_connections 2024-01-24T17:43:52.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.434+0000 7f3573e00640 1 --2- 172.21.15.39:0/1521961124 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f357406a380 0x7f35740c9b60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.434+0000 7f3573e00640 1 --2- 172.21.15.39:0/1521961124 >> v2:172.21.15.100:3300/0 conn(0x7f357415fed0 0x7f35740cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.434+0000 7f3573e00640 1 -- 172.21.15.39:0/1521961124 >> 172.21.15.39:0/1521961124 conn(0x7f35740b10a0 msgr2=0x7f35740b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:52.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.434+0000 7f3573e00640 1 -- 172.21.15.39:0/1521961124 shutdown_connections 2024-01-24T17:43:52.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.434+0000 7f3573e00640 1 -- 172.21.15.39:0/1521961124 wait complete. 2024-01-24T17:43:52.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.436+0000 7f3573e00640 1 Processor -- start 2024-01-24T17:43:52.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.436+0000 7f3573e00640 1 -- start start 2024-01-24T17:43:52.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f357406a380 0x7f357414fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35740cb830 0x7f35740c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f357415fed0 0x7f35740c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35740c4fd0 con 0x7f357415fed0 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35740c4a10 con 0x7f357406a380 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35740c4cf0 con 0x7f35740cb830 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f357415fed0 0x7f35740c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f357415fed0 0x7f35740c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51254/0 (socket says 172.21.15.39:51254) 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.437+0000 7f3573400640 1 -- 172.21.15.39:0/1434276455 learned_addr learned my addr 172.21.15.39:0/1434276455 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f3572a00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f357406a380 0x7f357414fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f356be00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35740cb830 0x7f35740c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f3573400640 1 -- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35740cb830 msgr2=0x7f35740c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f3573400640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35740cb830 0x7f35740c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f3573400640 1 -- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f357406a380 msgr2=0x7f357414fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f3573400640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f357406a380 0x7f357414fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f3573400640 1 -- 172.21.15.39:0/1434276455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35640094d0 con 0x7f357415fed0 2024-01-24T17:43:52.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f356be00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35740cb830 0x7f35740c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:52.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.438+0000 7f3573400640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f357415fed0 0x7f35740c4030 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f356c037040 tx=0x7f356c004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:52.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.439+0000 7f356aa00640 1 -- 172.21.15.39:0/1434276455 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c040650 con 0x7f357415fed0 2024-01-24T17:43:52.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.439+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c009a70 con 0x7f357415fed0 2024-01-24T17:43:52.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.439+0000 7f356aa00640 1 -- 172.21.15.39:0/1434276455 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c0407b0 con 0x7f357415fed0 2024-01-24T17:43:52.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.439+0000 7f356aa00640 1 -- 172.21.15.39:0/1434276455 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c040b60 con 0x7f357415fed0 2024-01-24T17:43:52.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.439+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35740c55e0 con 0x7f357415fed0 2024-01-24T17:43:52.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.440+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35740b1d10 con 0x7f357415fed0 2024-01-24T17:43:52.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.441+0000 7f356aa00640 1 -- 172.21.15.39:0/1434276455 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f356c040cc0 con 0x7f357415fed0 2024-01-24T17:43:52.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.442+0000 7f356aa00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f354c06a430 0x7f354c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.442+0000 7f356aa00640 1 -- 172.21.15.39:0/1434276455 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f356c0ab790 con 0x7f357415fed0 2024-01-24T17:43:52.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.442+0000 7f3572a00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f354c06a430 0x7f354c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.444+0000 7f3572a00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f354c06a430 0x7f354c06c8f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3564009c50 tx=0x7f3564008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:52.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.446+0000 7f356aa00640 1 -- 172.21.15.39:0/1434276455 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f356c076d30 con 0x7f357415fed0 2024-01-24T17:43:52.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.643+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3574067040 con 0x7f357415fed0 2024-01-24T17:43:52.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.644+0000 7f356aa00640 1 -- 172.21.15.39:0/1434276455 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7f356c009450 con 0x7f357415fed0 2024-01-24T17:43:52.646 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f354c06a430 msgr2=0x7f354c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f354c06a430 0x7f354c06c8f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3564009c50 tx=0x7f3564008040 comp rx=0 tx=0).stop 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f357415fed0 msgr2=0x7f35740c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f357415fed0 0x7f35740c4030 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f356c037040 tx=0x7f356c004a90 comp rx=0 tx=0).stop 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 shutdown_connections 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f354c06a430 0x7f354c06c8f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f357415fed0 0x7f35740c4030 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f35740cb830 0x7f35740c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 --2- 172.21.15.39:0/1434276455 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f357406a380 0x7f357414fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 >> 172.21.15.39:0/1434276455 conn(0x7f35740b10a0 msgr2=0x7f35740c85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:52.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 shutdown_connections 2024-01-24T17:43:52.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.647+0000 7f3573e00640 1 -- 172.21.15.39:0/1434276455 wait complete. 2024-01-24T17:43:52.668 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:43:52.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.892+0000 7fdba8a00640 1 Processor -- start 2024-01-24T17:43:52.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.892+0000 7fdba8a00640 1 -- start start 2024-01-24T17:43:52.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.893+0000 7fdba8a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdba415c8d0 0x7fdba40d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdba8a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40b38a0 con 0x7fdba40b2710 2024-01-24T17:43:52.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdba8a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40d13e0 con 0x7fdba40d2e80 2024-01-24T17:43:52.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdba8a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba40d0ab0 con 0x7fdba415c8d0 2024-01-24T17:43:52.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdba2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdba415c8d0 0x7fdba40d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdb99c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fdba40b2710 0x7fdba40d2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41894/0 (socket says 172.21.15.39:41894) 2024-01-24T17:43:52.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdb99c00640 1 -- 172.21.15.39:0/2524904684 learned_addr learned my addr 172.21.15.39:0/2524904684 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:52.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdba2800640 1 -- 172.21.15.39:0/2524904684 >> v1:172.21.15.100:6789/0 conn(0x7fdba40d2e80 legacy=0x7fdba40cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:52.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdba2800640 1 -- 172.21.15.39:0/2524904684 >> v1:172.21.15.39:6789/0 conn(0x7fdba40b2710 legacy=0x7fdba40d2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.895+0000 7fdba2800640 1 -- 172.21.15.39:0/2524904684 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40d0d90 con 0x7fdba415c8d0 2024-01-24T17:43:52.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.894+0000 7fdba1e00640 1 -- 172.21.15.39:0/2524904684 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213056365 0 0) 0x7fdba40b38a0 con 0x7fdba40b2710 2024-01-24T17:43:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.896+0000 7fdba2800640 1 --2- 172.21.15.39:0/2524904684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba415c8d0 0x7fdba40d03b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdb940097a0 tx=0x7fdb9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4285a4099e9237 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.896+0000 7fdba1e00640 1 -- 172.21.15.39:0/2524904684 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94031830 con 0x7fdba415c8d0 2024-01-24T17:43:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.896+0000 7fdba1e00640 1 -- 172.21.15.39:0/2524904684 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb94031990 con 0x7fdba415c8d0 2024-01-24T17:43:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.896+0000 7fdba1e00640 1 -- 172.21.15.39:0/2524904684 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94031d20 con 0x7fdba415c8d0 2024-01-24T17:43:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.896+0000 7fdba8a00640 1 -- 172.21.15.39:0/2524904684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba415c8d0 msgr2=0x7fdba40d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.896+0000 7fdba8a00640 1 --2- 172.21.15.39:0/2524904684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba415c8d0 0x7fdba40d03b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdb940097a0 tx=0x7fdb9402b8d0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.897+0000 7fdba8a00640 1 -- 172.21.15.39:0/2524904684 shutdown_connections 2024-01-24T17:43:52.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.897+0000 7fdba8a00640 1 --2- 172.21.15.39:0/2524904684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba415c8d0 0x7fdba40d03b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.897+0000 7fdba8a00640 1 -- 172.21.15.39:0/2524904684 >> 172.21.15.39:0/2524904684 conn(0x7fdba4071dd0 msgr2=0x7fdba409bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:52.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.897+0000 7fdba8a00640 1 -- 172.21.15.39:0/2524904684 shutdown_connections 2024-01-24T17:43:52.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.897+0000 7fdba8a00640 1 -- 172.21.15.39:0/2524904684 wait complete. 2024-01-24T17:43:52.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.899+0000 7fdba8a00640 1 Processor -- start 2024-01-24T17:43:52.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.899+0000 7fdba8a00640 1 -- start start 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba40b2710 0x7fdba409ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba8a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdba40d2e80 0x7fdba409d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba8a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdba415c8d0 0x7fdba4158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba8a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40b8780 con 0x7fdba40b2710 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba8a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40a0510 con 0x7fdba415c8d0 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba8a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba409c5f0 con 0x7fdba40d2e80 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba40b2710 0x7fdba409ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba40b2710 0x7fdba409ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51262/0 (socket says 172.21.15.39:51262) 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba2800640 1 -- 172.21.15.39:0/1475523234 learned_addr learned my addr 172.21.15.39:0/1475523234 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba3200640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdba415c8d0 0x7fdba4158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba2800640 1 -- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdba40d2e80 msgr2=0x7fdba409d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdb99c00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdba40d2e80 0x7fdba409d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba2800640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdba40d2e80 0x7fdba409d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba2800640 1 -- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdba415c8d0 msgr2=0x7fdba4158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba2800640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdba415c8d0 0x7fdba4158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba2800640 1 -- 172.21.15.39:0/1475523234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb94009400 con 0x7fdba40b2710 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdba3200640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdba415c8d0 0x7fdba4158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.900+0000 7fdb99c00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdba40d2e80 0x7fdba409d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.901+0000 7fdba2800640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba40b2710 0x7fdba409ffd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdb940098c0 tx=0x7fdb940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.901+0000 7fdb9b400640 1 -- 172.21.15.39:0/1475523234 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9403ba70 con 0x7fdba40b2710 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.901+0000 7fdb9b400640 1 -- 172.21.15.39:0/1475523234 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9403bbd0 con 0x7fdba40b2710 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.901+0000 7fdb9b400640 1 -- 172.21.15.39:0/1475523234 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb940453f0 con 0x7fdba40b2710 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.901+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba409c8d0 con 0x7fdba40b2710 2024-01-24T17:43:52.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.901+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba409ce90 con 0x7fdba40b2710 2024-01-24T17:43:52.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.903+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba404dff0 con 0x7fdba40b2710 2024-01-24T17:43:52.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.903+0000 7fdb9b400640 1 -- 172.21.15.39:0/1475523234 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdb9404e450 con 0x7fdba40b2710 2024-01-24T17:43:52.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.904+0000 7fdb9b400640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb7c06a430 0x7fdb7c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:52.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.904+0000 7fdb9b400640 1 -- 172.21.15.39:0/1475523234 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdb940aac60 con 0x7fdba40b2710 2024-01-24T17:43:52.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.905+0000 7fdb99c00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb7c06a430 0x7fdb7c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:52.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.906+0000 7fdb99c00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb7c06a430 0x7fdb7c06c8f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c007fd0 tx=0x7fdb8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:52.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:52.908+0000 7fdb9b400640 1 -- 172.21.15.39:0/1475523234 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdb940762e0 con 0x7fdba40b2710 2024-01-24T17:43:53.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.104+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdba415e370 con 0x7fdba40b2710 2024-01-24T17:43:53.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.105+0000 7fdb9b400640 1 -- 172.21.15.39:0/1475523234 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdb94079900 con 0x7fdba40b2710 2024-01-24T17:43:53.107 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.107+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb7c06a430 msgr2=0x7fdb7c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.107+0000 7fdba8a00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb7c06a430 0x7fdb7c06c8f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c007fd0 tx=0x7fdb8c009040 comp rx=0 tx=0).stop 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.107+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba40b2710 msgr2=0x7fdba409ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.107+0000 7fdba8a00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba40b2710 0x7fdba409ffd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdb940098c0 tx=0x7fdb940046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.108+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 shutdown_connections 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.108+0000 7fdba8a00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdba415c8d0 0x7fdba4158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.108+0000 7fdba8a00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdba40d2e80 0x7fdba409d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.108+0000 7fdba8a00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb7c06a430 0x7fdb7c06c8f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.108+0000 7fdba8a00640 1 --2- 172.21.15.39:0/1475523234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdba40b2710 0x7fdba409ffd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.108+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 >> 172.21.15.39:0/1475523234 conn(0x7fdba4071dd0 msgr2=0x7fdba40cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.108+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 shutdown_connections 2024-01-24T17:43:53.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.108+0000 7fdba8a00640 1 -- 172.21.15.39:0/1475523234 wait complete. 2024-01-24T17:43:53.129 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:43:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.352+0000 7f0ceca00640 1 Processor -- start 2024-01-24T17:43:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.352+0000 7f0ceca00640 1 -- start start 2024-01-24T17:43:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.354+0000 7f0ceca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ce80cb2b0 0x7f0ce80cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.354+0000 7f0ceca00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0ce815cc90 0x7f0ce80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.354+0000 7f0ceca00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce8161490 con 0x7f0ce80cde00 2024-01-24T17:43:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.354+0000 7f0ceca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce80cbbd0 con 0x7f0ce80cb2b0 2024-01-24T17:43:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.354+0000 7f0ceca00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce80ce8b0 con 0x7f0ce815cc90 2024-01-24T17:43:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.354+0000 7f0ce7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ce80cb2b0 0x7f0ce80cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.354+0000 7f0ce7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ce80cb2b0 0x7f0ce80cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35140/0 (socket says 172.21.15.39:35140) 2024-01-24T17:43:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.354+0000 7f0ce7400640 1 -- 172.21.15.39:0/2828862962 learned_addr learned my addr 172.21.15.39:0/2828862962 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.355+0000 7f0cdfe00640 1 --2- 172.21.15.39:0/2828862962 >> v2:172.21.15.100:3300/0 conn(0x7f0ce815cc90 0x7f0ce80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.355+0000 7f0ce7400640 1 -- 172.21.15.39:0/2828862962 >> v2:172.21.15.100:3300/0 conn(0x7f0ce815cc90 msgr2=0x7f0ce80ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.355+0000 7f0ce7400640 1 --2- 172.21.15.39:0/2828862962 >> v2:172.21.15.100:3300/0 conn(0x7f0ce815cc90 0x7f0ce80ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.355+0000 7f0ce7400640 1 -- 172.21.15.39:0/2828862962 >> v1:172.21.15.100:6789/0 conn(0x7f0ce80cde00 legacy=0x7f0ce8160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.355+0000 7f0ce7400640 1 -- 172.21.15.39:0/2828862962 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80ceb80 con 0x7f0ce80cb2b0 2024-01-24T17:43:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.355+0000 7f0cdfe00640 1 --2- 172.21.15.39:0/2828862962 >> v2:172.21.15.100:3300/0 conn(0x7f0ce815cc90 0x7f0ce80ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:53.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.356+0000 7f0ce7400640 1 --2- 172.21.15.39:0/2828862962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce80cb2b0 0x7f0ce80cb690 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80b4340 tx=0x7f0cd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea200d79142c9244 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:53.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.356+0000 7f0ce6a00640 1 -- 172.21.15.39:0/2828862962 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8031870 con 0x7f0ce80cb2b0 2024-01-24T17:43:53.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.356+0000 7f0ce6a00640 1 -- 172.21.15.39:0/2828862962 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd80319d0 con 0x7f0ce80cb2b0 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.356+0000 7f0ce6a00640 1 -- 172.21.15.39:0/2828862962 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8031d60 con 0x7f0ce80cb2b0 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.357+0000 7f0ceca00640 1 -- 172.21.15.39:0/2828862962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce80cb2b0 msgr2=0x7f0ce80cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.357+0000 7f0ceca00640 1 --2- 172.21.15.39:0/2828862962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce80cb2b0 0x7f0ce80cb690 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80b4340 tx=0x7f0cd802b910 comp rx=0 tx=0).stop 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.357+0000 7f0ceca00640 1 -- 172.21.15.39:0/2828862962 shutdown_connections 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.357+0000 7f0ceca00640 1 --2- 172.21.15.39:0/2828862962 >> v2:172.21.15.100:3300/0 conn(0x7f0ce815cc90 0x7f0ce80ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.357+0000 7f0ceca00640 1 --2- 172.21.15.39:0/2828862962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce80cb2b0 0x7f0ce80cb690 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.357+0000 7f0ceca00640 1 -- 172.21.15.39:0/2828862962 >> 172.21.15.39:0/2828862962 conn(0x7f0ce806a090 msgr2=0x7f0ce806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.357+0000 7f0ceca00640 1 -- 172.21.15.39:0/2828862962 shutdown_connections 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.357+0000 7f0ceca00640 1 -- 172.21.15.39:0/2828862962 wait complete. 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.359+0000 7f0ceca00640 1 Processor -- start 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.359+0000 7f0ceca00640 1 -- start start 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ceca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ce80cb2b0 0x7f0ce8144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ceca00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ce80cde00 0x7f0ce80c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ceca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce815fb40 0x7f0ce815d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ceca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80c2f90 con 0x7f0ce815fb40 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ceca00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80c3df0 con 0x7f0ce80cde00 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ceca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80c31e0 con 0x7f0ce80cb2b0 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ce7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce815fb40 0x7f0ce815d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ce7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce815fb40 0x7f0ce815d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35148/0 (socket says 172.21.15.39:35148) 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.360+0000 7f0ce7e00640 1 -- 172.21.15.39:0/3394720841 learned_addr learned my addr 172.21.15.39:0/3394720841 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0cdfe00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ce80cde00 0x7f0ce80c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0ce7400640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ce80cb2b0 0x7f0ce8144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0ce7e00640 1 -- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ce80cb2b0 msgr2=0x7f0ce8144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0ce7e00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ce80cb2b0 0x7f0ce8144f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0ce7e00640 1 -- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ce80cde00 msgr2=0x7f0ce80c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0ce7e00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ce80cde00 0x7f0ce80c4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0ce7e00640 1 -- 172.21.15.39:0/3394720841 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd80094d0 con 0x7f0ce815fb40 2024-01-24T17:43:53.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0ce7400640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ce80cb2b0 0x7f0ce8144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:53.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0ce7e00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce815fb40 0x7f0ce815d080 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000c780 tx=0x7f0ce000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:53.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0cdf400640 1 -- 172.21.15.39:0/3394720841 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0003d80 con 0x7f0ce815fb40 2024-01-24T17:43:53.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.361+0000 7f0cdf400640 1 -- 172.21.15.39:0/3394720841 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce000ce90 con 0x7f0ce815fb40 2024-01-24T17:43:53.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.362+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce80c34c0 con 0x7f0ce815fb40 2024-01-24T17:43:53.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.362+0000 7f0cdf400640 1 -- 172.21.15.39:0/3394720841 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce00052b0 con 0x7f0ce815fb40 2024-01-24T17:43:53.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.362+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce80c3a80 con 0x7f0ce815fb40 2024-01-24T17:43:53.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.363+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce80c3f60 con 0x7f0ce815fb40 2024-01-24T17:43:53.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.364+0000 7f0cdf400640 1 -- 172.21.15.39:0/3394720841 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0ce0005410 con 0x7f0ce815fb40 2024-01-24T17:43:53.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.364+0000 7f0cdf400640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cc006a460 0x7f0cc006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.365+0000 7f0cdf400640 1 -- 172.21.15.39:0/3394720841 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0ce0088200 con 0x7f0ce815fb40 2024-01-24T17:43:53.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.365+0000 7f0ce7400640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cc006a460 0x7f0cc006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.366+0000 7f0ce7400640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cc006a460 0x7f0cc006c920 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8036000 tx=0x7f0cd803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:53.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.369+0000 7f0cdf400640 1 -- 172.21.15.39:0/3394720841 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0ce0053770 con 0x7f0ce815fb40 2024-01-24T17:43:53.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.599+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ce804dff0 con 0x7f0ce815fb40 2024-01-24T17:43:53.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.600+0000 7f0cdf400640 1 -- 172.21.15.39:0/3394720841 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f0ce0056d90 con 0x7f0ce815fb40 2024-01-24T17:43:53.602 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:53.602 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4688,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/648542221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":648542221},{"type":"v1","addr":"172.21.15.100:6837","nonce":648542221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/188830546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":188830546},{"type":"v1","addr":"172.21.15.100:6835","nonce":188830546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/4294097622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4294097622},{"type":"v1","addr":"172.21.15.39:6834","nonce":4294097622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:42:56.701344+0000","modified":"2024-01-24T17:43:00.247932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5175":{"gid":5175,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/3577904963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3577904963},{"type":"v1","addr":"172.21.15.39:6835","nonce":3577904963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-24T17:43:53.602 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-01-24T17:43:53.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cc006a460 msgr2=0x7f0cc006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cc006a460 0x7f0cc006c920 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8036000 tx=0x7f0cd803c040 comp rx=0 tx=0).stop 2024-01-24T17:43:53.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce815fb40 msgr2=0x7f0ce815d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce815fb40 0x7f0ce815d080 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000c780 tx=0x7f0ce000cc40 comp rx=0 tx=0).stop 2024-01-24T17:43:53.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 shutdown_connections 2024-01-24T17:43:53.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cc006a460 0x7f0cc006c920 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ce815fb40 0x7f0ce815d080 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ce80cde00 0x7f0ce80c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 --2- 172.21.15.39:0/3394720841 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ce80cb2b0 0x7f0ce8144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 >> 172.21.15.39:0/3394720841 conn(0x7f0ce806a090 msgr2=0x7f0ce806a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:53.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 shutdown_connections 2024-01-24T17:43:53.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.603+0000 7f0ceca00640 1 -- 172.21.15.39:0/3394720841 wait complete. 2024-01-24T17:43:53.626 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:53.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.834+0000 7f992be00640 1 Processor -- start 2024-01-24T17:43:53.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.834+0000 7f992be00640 1 -- start start 2024-01-24T17:43:53.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f992be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f99240caf50 0x7f99240cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f992be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f99240b1390 0x7f99240b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f992be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99240c9180 con 0x7f992415a950 2024-01-24T17:43:53.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f992be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99240cb920 con 0x7f99240b1390 2024-01-24T17:43:53.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f992be00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99240b43d0 con 0x7f99240caf50 2024-01-24T17:43:53.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f9929400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f992415a950 0x7f99240ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50778/0 (socket says 172.21.15.39:50778) 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f9929400640 1 -- 172.21.15.39:0/4068580323 learned_addr learned my addr 172.21.15.39:0/4068580323 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f9929e00640 1 --2- 172.21.15.39:0/4068580323 >> v2:172.21.15.39:3300/0 conn(0x7f99240b1390 0x7f99240b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f9928a00640 1 --2- 172.21.15.39:0/4068580323 >> v2:172.21.15.100:3300/0 conn(0x7f99240caf50 0x7f99240cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f9923e00640 1 -- 172.21.15.39:0/4068580323 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951755346 0 0) 0x7f99240c9180 con 0x7f992415a950 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.836+0000 7f9923e00640 1 -- 172.21.15.39:0/4068580323 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9904003520 con 0x7f992415a950 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.837+0000 7f9929e00640 1 -- 172.21.15.39:0/4068580323 >> v2:172.21.15.100:3300/0 conn(0x7f99240caf50 msgr2=0x7f99240cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.837+0000 7f9929e00640 1 --2- 172.21.15.39:0/4068580323 >> v2:172.21.15.100:3300/0 conn(0x7f99240caf50 0x7f99240cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.837+0000 7f9929e00640 1 -- 172.21.15.39:0/4068580323 >> v1:172.21.15.39:6789/0 conn(0x7f992415a950 legacy=0x7f99240ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.837+0000 7f9929e00640 1 -- 172.21.15.39:0/4068580323 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240c9460 con 0x7f99240b1390 2024-01-24T17:43:53.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.838+0000 7f9929e00640 1 --2- 172.21.15.39:0/4068580323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99240b1390 0x7f99240b17c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f992406fd40 tx=0x7f991c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a4cb3a9fa476f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:53.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.838+0000 7f9923e00640 1 -- 172.21.15.39:0/4068580323 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c02cc80 con 0x7f99240b1390 2024-01-24T17:43:53.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.838+0000 7f9923e00640 1 -- 172.21.15.39:0/4068580323 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991c02cde0 con 0x7f99240b1390 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.838+0000 7f9923e00640 1 -- 172.21.15.39:0/4068580323 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c034660 con 0x7f99240b1390 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.839+0000 7f992be00640 1 -- 172.21.15.39:0/4068580323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99240b1390 msgr2=0x7f99240b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.839+0000 7f992be00640 1 --2- 172.21.15.39:0/4068580323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99240b1390 0x7f99240b17c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f992406fd40 tx=0x7f991c02bed0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.839+0000 7f992be00640 1 -- 172.21.15.39:0/4068580323 shutdown_connections 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.839+0000 7f992be00640 1 --2- 172.21.15.39:0/4068580323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99240b1390 0x7f99240b17c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.839+0000 7f992be00640 1 --2- 172.21.15.39:0/4068580323 >> v2:172.21.15.100:3300/0 conn(0x7f99240caf50 0x7f99240cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.839+0000 7f992be00640 1 -- 172.21.15.39:0/4068580323 >> 172.21.15.39:0/4068580323 conn(0x7f992406a3e0 msgr2=0x7f992406bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.839+0000 7f992be00640 1 -- 172.21.15.39:0/4068580323 shutdown_connections 2024-01-24T17:43:53.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.839+0000 7f992be00640 1 -- 172.21.15.39:0/4068580323 wait complete. 2024-01-24T17:43:53.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.841+0000 7f992be00640 1 Processor -- start 2024-01-24T17:43:53.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.841+0000 7f992be00640 1 -- start start 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f992be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99240b1390 0x7f99240cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f992be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240caf50 0x7f99240ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f992be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f992415a950 0x7f9924149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f992be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9924149bc0 con 0x7f992415a950 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f992be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992415cb70 con 0x7f99240b1390 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f992be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992415aeb0 con 0x7f99240caf50 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f992415a950 0x7f9924149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f992415a950 0x7f9924149680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35162/0 (socket says 172.21.15.39:35162) 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929e00640 1 -- 172.21.15.39:0/3180453964 learned_addr learned my addr 172.21.15.39:0/3180453964 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929400640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99240b1390 0x7f99240cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929e00640 1 -- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240caf50 msgr2=0x7f99240ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929e00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240caf50 0x7f99240ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929e00640 1 -- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99240b1390 msgr2=0x7f99240cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:53.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929e00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99240b1390 0x7f99240cc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:53.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.842+0000 7f9929e00640 1 -- 172.21.15.39:0/3180453964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c009a70 con 0x7f992415a950 2024-01-24T17:43:53.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.843+0000 7f9929e00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f992415a950 0x7f9924149680 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f991c009e80 tx=0x7f991c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:53.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.843+0000 7f9921600640 1 -- 172.21.15.39:0/3180453964 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c040650 con 0x7f992415a950 2024-01-24T17:43:53.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.843+0000 7f9921600640 1 -- 172.21.15.39:0/3180453964 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991c0407b0 con 0x7f992415a950 2024-01-24T17:43:53.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.843+0000 7f9921600640 1 -- 172.21.15.39:0/3180453964 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c040b60 con 0x7f992415a950 2024-01-24T17:43:53.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.843+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992415b190 con 0x7f992415a950 2024-01-24T17:43:53.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.843+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99241462e0 con 0x7f992415a950 2024-01-24T17:43:53.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.844+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9924073ef0 con 0x7f992415a950 2024-01-24T17:43:53.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.845+0000 7f9921600640 1 -- 172.21.15.39:0/3180453964 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f991c040cc0 con 0x7f992415a950 2024-01-24T17:43:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.846+0000 7f9921600640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f98fc06a460 0x7f98fc06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.846+0000 7f9921600640 1 -- 172.21.15.39:0/3180453964 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f991c0ab420 con 0x7f992415a950 2024-01-24T17:43:53.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.847+0000 7f9929400640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f98fc06a460 0x7f98fc06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:53.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.848+0000 7f9929400640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f98fc06a460 0x7f98fc06c920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f99140011e0 tx=0x7f9914009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:53.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:53.851+0000 7f9921600640 1 -- 172.21.15.39:0/3180453964 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f991c076990 con 0x7f992415a950 2024-01-24T17:43:54.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.042+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f992415cce0 con 0x7f992415a950 2024-01-24T17:43:54.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.045+0000 7f9921600640 1 -- 172.21.15.39:0/3180453964 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16290 (secure 0 0 0) 0x7f991c009450 con 0x7f992415a950 2024-01-24T17:43:54.047 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:54.047 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:43:16.261174+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.1c","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.32","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:54.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f98fc06a460 msgr2=0x7f98fc06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f98fc06a460 0x7f98fc06c920 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f99140011e0 tx=0x7f9914009040 comp rx=0 tx=0).stop 2024-01-24T17:43:54.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f992415a950 msgr2=0x7f9924149680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f992415a950 0x7f9924149680 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f991c009e80 tx=0x7f991c004a40 comp rx=0 tx=0).stop 2024-01-24T17:43:54.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 shutdown_connections 2024-01-24T17:43:54.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f98fc06a460 0x7f98fc06c920 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f992415a950 0x7f9924149680 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99240caf50 0x7f99240ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 --2- 172.21.15.39:0/3180453964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99240b1390 0x7f99240cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 >> 172.21.15.39:0/3180453964 conn(0x7f992406a3e0 msgr2=0x7f9924158990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:54.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 shutdown_connections 2024-01-24T17:43:54.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.048+0000 7f992be00640 1 -- 172.21.15.39:0/3180453964 wait complete. 2024-01-24T17:43:54.070 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-24T17:43:54.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:43:54.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.285+0000 7f9296c00640 1 Processor -- start 2024-01-24T17:43:54.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.285+0000 7f9296c00640 1 -- start start 2024-01-24T17:43:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.287+0000 7f9296c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f92900d03b0 0x7f929015db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.287+0000 7f9296c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d0da0 0x7f92900d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.287+0000 7f9296c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92900d0860 con 0x7f929015cbf0 2024-01-24T17:43:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.287+0000 7f9296c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92900d1770 con 0x7f92900d03b0 2024-01-24T17:43:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.287+0000 7f9296c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92900d0a90 con 0x7f92900d0da0 2024-01-24T17:43:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.287+0000 7f9294e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f92900d03b0 0x7f929015db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.287+0000 7f9294e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f92900d03b0 0x7f929015db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35178/0 (socket says 172.21.15.39:35178) 2024-01-24T17:43:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.287+0000 7f9294e00640 1 -- 172.21.15.39:0/2047659055 learned_addr learned my addr 172.21.15.39:0/2047659055 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:54.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.288+0000 7f9296200640 1 --2- 172.21.15.39:0/2047659055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d0da0 0x7f92900d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.288+0000 7f9294e00640 1 -- 172.21.15.39:0/2047659055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d0da0 msgr2=0x7f92900d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:54.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.288+0000 7f9294e00640 1 --2- 172.21.15.39:0/2047659055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d0da0 0x7f92900d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.288+0000 7f9294e00640 1 -- 172.21.15.39:0/2047659055 >> v1:172.21.15.100:6789/0 conn(0x7f929015cbf0 legacy=0x7f92900ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:54.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.288+0000 7f9294e00640 1 -- 172.21.15.39:0/2047659055 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92900cd760 con 0x7f92900d03b0 2024-01-24T17:43:54.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.288+0000 7f9296200640 1 --2- 172.21.15.39:0/2047659055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d0da0 0x7f92900d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:54.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.289+0000 7f9294e00640 1 --2- 172.21.15.39:0/2047659055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92900d03b0 0x7f929015db60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f92780098a0 tx=0x7f927802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f03cd55692d8346 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:54.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.289+0000 7f928fe00640 1 -- 172.21.15.39:0/2047659055 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9278031640 con 0x7f92900d03b0 2024-01-24T17:43:54.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.289+0000 7f928fe00640 1 -- 172.21.15.39:0/2047659055 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92780317a0 con 0x7f92900d03b0 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.289+0000 7f928fe00640 1 -- 172.21.15.39:0/2047659055 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9278031b30 con 0x7f92900d03b0 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.290+0000 7f9296c00640 1 -- 172.21.15.39:0/2047659055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92900d03b0 msgr2=0x7f929015db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.290+0000 7f9296c00640 1 --2- 172.21.15.39:0/2047659055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92900d03b0 0x7f929015db60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f92780098a0 tx=0x7f927802b8f0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.290+0000 7f9296c00640 1 -- 172.21.15.39:0/2047659055 shutdown_connections 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.290+0000 7f9296c00640 1 --2- 172.21.15.39:0/2047659055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92900d03b0 0x7f929015db60 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.290+0000 7f9296c00640 1 --2- 172.21.15.39:0/2047659055 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d0da0 0x7f92900d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.290+0000 7f9296c00640 1 -- 172.21.15.39:0/2047659055 >> 172.21.15.39:0/2047659055 conn(0x7f929009bb90 msgr2=0x7f92900d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.290+0000 7f9296c00640 1 -- 172.21.15.39:0/2047659055 shutdown_connections 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.290+0000 7f9296c00640 1 -- 172.21.15.39:0/2047659055 wait complete. 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.292+0000 7f9296c00640 1 Processor -- start 2024-01-24T17:43:54.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.292+0000 7f9296c00640 1 -- start start 2024-01-24T17:43:54.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d03b0 0x7f92900b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92900d0da0 0x7f92900c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f929015cbf0 0x7f92901580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92901e5280 con 0x7f929015cbf0 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92900ca460 con 0x7f92900d0da0 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92900ca740 con 0x7f92900d03b0 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f929015cbf0 0x7f92901580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f929015cbf0 0x7f92901580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35192/0 (socket says 172.21.15.39:35192) 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296200640 1 -- 172.21.15.39:0/1498427446 learned_addr learned my addr 172.21.15.39:0/1498427446 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296200640 1 -- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d03b0 msgr2=0x7f92900b7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.293+0000 7f9296200640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d03b0 0x7f92900b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f9296200640 1 -- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92900d0da0 msgr2=0x7f92900c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f9296200640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92900d0da0 0x7f92900c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f9296200640 1 -- 172.21.15.39:0/1498427446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92780094d0 con 0x7f929015cbf0 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f9296200640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f929015cbf0 0x7f92901580d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f928800cd50 tx=0x7f928800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f928d600640 1 -- 172.21.15.39:0/1498427446 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928800f680 con 0x7f929015cbf0 2024-01-24T17:43:54.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f928d600640 1 -- 172.21.15.39:0/1498427446 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928800f800 con 0x7f929015cbf0 2024-01-24T17:43:54.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f928d600640 1 -- 172.21.15.39:0/1498427446 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928800fbb0 con 0x7f929015cbf0 2024-01-24T17:43:54.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92900caa20 con 0x7f929015cbf0 2024-01-24T17:43:54.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.294+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92900cafe0 con 0x7f929015cbf0 2024-01-24T17:43:54.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.295+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92900cb1d0 con 0x7f929015cbf0 2024-01-24T17:43:54.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.296+0000 7f928d600640 1 -- 172.21.15.39:0/1498427446 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f928800b040 con 0x7f929015cbf0 2024-01-24T17:43:54.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.297+0000 7f928d600640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f926806a430 0x7f926806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.297+0000 7f928d600640 1 -- 172.21.15.39:0/1498427446 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9288089490 con 0x7f929015cbf0 2024-01-24T17:43:54.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.298+0000 7f9295800640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f926806a430 0x7f926806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.299+0000 7f9295800640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f926806a430 0x7f926806c8f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9280003d10 tx=0x7f9280007a00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:54.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.302+0000 7f928d600640 1 -- 172.21.15.39:0/1498427446 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9288054960 con 0x7f929015cbf0 2024-01-24T17:43:54.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.539+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92900cdcf0 con 0x7f929015cbf0 2024-01-24T17:43:54.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.539+0000 7f928d600640 1 -- 172.21.15.39:0/1498427446 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f9288057f80 con 0x7f929015cbf0 2024-01-24T17:43:54.541 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:54.541 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4688,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/648542221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":648542221},{"type":"v1","addr":"172.21.15.100:6837","nonce":648542221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/188830546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":188830546},{"type":"v1","addr":"172.21.15.100:6835","nonce":188830546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/4294097622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4294097622},{"type":"v1","addr":"172.21.15.39:6834","nonce":4294097622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:42:56.701344+0000","modified":"2024-01-24T17:43:00.247932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5175":{"gid":5175,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/3577904963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3577904963},{"type":"v1","addr":"172.21.15.39:6835","nonce":3577904963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-24T17:43:54.542 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-01-24T17:43:54.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f926806a430 msgr2=0x7f926806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f926806a430 0x7f926806c8f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9280003d10 tx=0x7f9280007a00 comp rx=0 tx=0).stop 2024-01-24T17:43:54.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f929015cbf0 msgr2=0x7f92901580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f929015cbf0 0x7f92901580d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f928800cd50 tx=0x7f928800a650 comp rx=0 tx=0).stop 2024-01-24T17:43:54.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 shutdown_connections 2024-01-24T17:43:54.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f926806a430 0x7f926806c8f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f929015cbf0 0x7f92901580d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92900d0da0 0x7f92900c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 --2- 172.21.15.39:0/1498427446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f92900d03b0 0x7f92900b7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 >> 172.21.15.39:0/1498427446 conn(0x7f929009bb90 msgr2=0x7f92900d3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:54.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.542+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 shutdown_connections 2024-01-24T17:43:54.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.543+0000 7f9296c00640 1 -- 172.21.15.39:0/1498427446 wait complete. 2024-01-24T17:43:54.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:43:54.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.779+0000 7fbf47200640 1 Processor -- start 2024-01-24T17:43:54.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.779+0000 7fbf47200640 1 -- start start 2024-01-24T17:43:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.781+0000 7fbf47200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbf48065910 0x7fbf481610c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.781+0000 7fbf47200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf48161600 con 0x7fbf4815ca20 2024-01-24T17:43:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.781+0000 7fbf47200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf48065df0 con 0x7fbf4806e110 2024-01-24T17:43:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.781+0000 7fbf47200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf480c25c0 con 0x7fbf48065910 2024-01-24T17:43:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.781+0000 7fbf45e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fbf4815ca20 0x7fbf480cde00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50784/0 (socket says 172.21.15.39:50784) 2024-01-24T17:43:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.781+0000 7fbf45e00640 1 -- 172.21.15.39:0/1054460031 learned_addr learned my addr 172.21.15.39:0/1054460031 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf3fe00640 1 --2- 172.21.15.39:0/1054460031 >> v2:172.21.15.100:3300/0 conn(0x7fbf48065910 0x7fbf481610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438032968 0 0) 0x7fbf38001110 con 0x7fbf4815ca20 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf28003520 con 0x7fbf4815ca20 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 958731354 0 0) 0x7fbf28003520 con 0x7fbf4815ca20 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 >> v2:172.21.15.100:3300/0 conn(0x7fbf48065910 msgr2=0x7fbf481610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 --2- 172.21.15.39:0/1054460031 >> v2:172.21.15.100:3300/0 conn(0x7fbf48065910 0x7fbf481610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 >> v1:172.21.15.100:6789/0 conn(0x7fbf4806e110 legacy=0x7fbf48069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4806d630 con 0x7fbf4815ca20 2024-01-24T17:43:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fbf38001800 con 0x7fbf4815ca20 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 >> v1:172.21.15.39:6789/0 conn(0x7fbf4815ca20 legacy=0x7fbf480cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28003970 0x7fbf28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf280047c0 0x7fbf28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf28005180 0x7fbf2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2800c0a0 con 0x7fbf28005180 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2800c210 con 0x7fbf280047c0 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2800c3d0 con 0x7fbf28003970 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.782+0000 7fbf3fe00640 1 --2- 172.21.15.39:0/1054460031 >> v2:172.21.15.100:3300/0 conn(0x7fbf48065910 0x7fbf481610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.783+0000 7fbf46800640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf28005180 0x7fbf2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.783+0000 7fbf45e00640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28003970 0x7fbf28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.783+0000 7fbf46800640 1 -- 172.21.15.39:0/1054460031 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28003970 msgr2=0x7fbf28003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.783+0000 7fbf46800640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28003970 0x7fbf28003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.783+0000 7fbf46800640 1 -- 172.21.15.39:0/1054460031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf280047c0 msgr2=0x7fbf28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.783+0000 7fbf46800640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf280047c0 0x7fbf28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.783+0000 7fbf46800640 1 -- 172.21.15.39:0/1054460031 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf28003520 con 0x7fbf28005180 2024-01-24T17:43:54.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.784+0000 7fbf46800640 1 -- 172.21.15.39:0/1054460031 reap_dead start 2024-01-24T17:43:54.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.784+0000 7fbf46800640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf28005180 0x7fbf2800b980 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbf40001e50 tx=0x7fbf280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2df4a2d1783d063 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:54.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.784+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40003b70 con 0x7fbf28005180 2024-01-24T17:43:54.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.784+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf40003d10 con 0x7fbf28005180 2024-01-24T17:43:54.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.784+0000 7fbf45400640 1 -- 172.21.15.39:0/1054460031 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40005310 con 0x7fbf28005180 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.785+0000 7fbf47200640 1 -- 172.21.15.39:0/1054460031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf28005180 msgr2=0x7fbf2800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.785+0000 7fbf47200640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf28005180 0x7fbf2800b980 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbf40001e50 tx=0x7fbf280057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.785+0000 7fbf47200640 1 -- 172.21.15.39:0/1054460031 shutdown_connections 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.785+0000 7fbf47200640 1 --2- 172.21.15.39:0/1054460031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf28005180 0x7fbf2800b980 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.785+0000 7fbf47200640 1 -- 172.21.15.39:0/1054460031 >> 172.21.15.39:0/1054460031 conn(0x7fbf480b0740 msgr2=0x7fbf480b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.785+0000 7fbf47200640 1 -- 172.21.15.39:0/1054460031 shutdown_connections 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.785+0000 7fbf47200640 1 -- 172.21.15.39:0/1054460031 wait complete. 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.787+0000 7fbf47200640 1 Processor -- start 2024-01-24T17:43:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.787+0000 7fbf47200640 1 -- start start 2024-01-24T17:43:54.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf47200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28005180 0x7fbf480b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf47200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf480b3a50 0x7fbf480b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf47200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf4815d1e0 0x7fbf480b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf47200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480c52a0 con 0x7fbf480b3a50 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf47200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480b1870 con 0x7fbf4815d1e0 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf47200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf480b1b50 con 0x7fbf28005180 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf480b3a50 0x7fbf480b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf480b3a50 0x7fbf480b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35212/0 (socket says 172.21.15.39:35212) 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf3fe00640 1 -- 172.21.15.39:0/2574239670 learned_addr learned my addr 172.21.15.39:0/2574239670 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf45e00640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28005180 0x7fbf480b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.788+0000 7fbf46800640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf4815d1e0 0x7fbf480b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3fe00640 1 -- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28005180 msgr2=0x7fbf480b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3fe00640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28005180 0x7fbf480b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3fe00640 1 -- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf4815d1e0 msgr2=0x7fbf480b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3fe00640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf4815d1e0 0x7fbf480b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3fe00640 1 -- 172.21.15.39:0/2574239670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480b1e30 con 0x7fbf480b3a50 2024-01-24T17:43:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3fe00640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf480b3a50 0x7fbf480b2910 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000e600 tx=0x7fbf3000eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3e000640 1 -- 172.21.15.39:0/2574239670 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30004230 con 0x7fbf480b3a50 2024-01-24T17:43:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3e000640 1 -- 172.21.15.39:0/2574239670 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf30004390 con 0x7fbf480b3a50 2024-01-24T17:43:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf3e000640 1 -- 172.21.15.39:0/2574239670 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30003850 con 0x7fbf480b3a50 2024-01-24T17:43:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.789+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf480b2110 con 0x7fbf480b3a50 2024-01-24T17:43:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.790+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf480b4600 con 0x7fbf480b3a50 2024-01-24T17:43:54.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.791+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4804dff0 con 0x7fbf480b3a50 2024-01-24T17:43:54.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.791+0000 7fbf3e000640 1 -- 172.21.15.39:0/2574239670 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbf300039b0 con 0x7fbf480b3a50 2024-01-24T17:43:54.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.792+0000 7fbf3e000640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf2006a460 0x7fbf2006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:54.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.792+0000 7fbf3e000640 1 -- 172.21.15.39:0/2574239670 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbf30088590 con 0x7fbf480b3a50 2024-01-24T17:43:54.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.793+0000 7fbf45e00640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf2006a460 0x7fbf2006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:54.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.794+0000 7fbf45e00640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf2006a460 0x7fbf2006c920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbf380012b0 tx=0x7fbf38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:54.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:54.796+0000 7fbf3e000640 1 -- 172.21.15.39:0/2574239670 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbf30053af0 con 0x7fbf480b3a50 2024-01-24T17:43:55.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.036+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf4806dbc0 con 0x7fbf480b3a50 2024-01-24T17:43:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.037+0000 7fbf3e000640 1 -- 172.21.15.39:0/2574239670 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fbf30057110 con 0x7fbf480b3a50 2024-01-24T17:43:55.039 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:55.039 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4688,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/648542221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":648542221},{"type":"v1","addr":"172.21.15.100:6837","nonce":648542221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/188830546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":188830546},{"type":"v1","addr":"172.21.15.100:6835","nonce":188830546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/4294097622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4294097622},{"type":"v1","addr":"172.21.15.39:6834","nonce":4294097622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:42:56.701344+0000","modified":"2024-01-24T17:43:00.247932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5175":{"gid":5175,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/3577904963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3577904963},{"type":"v1","addr":"172.21.15.39:6835","nonce":3577904963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-24T17:43:55.039 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 15 2024-01-24T17:43:55.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf2006a460 msgr2=0x7fbf2006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf2006a460 0x7fbf2006c920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbf380012b0 tx=0x7fbf38009040 comp rx=0 tx=0).stop 2024-01-24T17:43:55.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf480b3a50 msgr2=0x7fbf480b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf480b3a50 0x7fbf480b2910 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000e600 tx=0x7fbf3000eac0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 shutdown_connections 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf4815d1e0 0x7fbf480b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbf2006a460 0x7fbf2006c920 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf480b3a50 0x7fbf480b2910 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 --2- 172.21.15.39:0/2574239670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf28005180 0x7fbf480b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 >> 172.21.15.39:0/2574239670 conn(0x7fbf480b0740 msgr2=0x7fbf4815bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 shutdown_connections 2024-01-24T17:43:55.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.041+0000 7fbf47200640 1 -- 172.21.15.39:0/2574239670 wait complete. 2024-01-24T17:43:55.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:43:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.271+0000 7f0967200640 1 Processor -- start 2024-01-24T17:43:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.272+0000 7f0967200640 1 -- start start 2024-01-24T17:43:55.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.273+0000 7f0967200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 0x7f09680af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.273+0000 7f0967200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680b1fd0 con 0x7f09680cb010 2024-01-24T17:43:55.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.273+0000 7f0967200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680af180 con 0x7f09680b0550 2024-01-24T17:43:55.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.273+0000 7f0967200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09680ae8e0 con 0x7f096815dda0 2024-01-24T17:43:55.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.273+0000 7f0965400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f09680cb010 0x7f09680afe40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50800/0 (socket says 172.21.15.39:50800) 2024-01-24T17:43:55.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.274+0000 7f0965400640 1 -- 172.21.15.39:0/2312351857 learned_addr learned my addr 172.21.15.39:0/2312351857 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:55.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.274+0000 7f0965e00640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 0x7f09680af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.274+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057777124 0 0) 0x7f09680b1fd0 con 0x7f09680cb010 2024-01-24T17:43:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.274+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0948003520 con 0x7f09680cb010 2024-01-24T17:43:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.274+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 999429713 0 0) 0x7f0948003520 con 0x7f09680cb010 2024-01-24T17:43:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.274+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 msgr2=0x7f09680af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.274+0000 7f0964a00640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 0x7f09680af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 >> v1:172.21.15.100:6789/0 conn(0x7f09680b0550 legacy=0x7f09680ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680aebc0 con 0x7f09680cb010 2024-01-24T17:43:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f09500019f0 con 0x7f09680cb010 2024-01-24T17:43:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 >> v1:172.21.15.39:6789/0 conn(0x7f09680cb010 legacy=0x7f09680afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480039a0 0x7f0948003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09480047c0 0x7f0948004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 0x7f0948005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680ae8e0 con 0x7f09480047c0 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094800ba60 con 0x7f09480039a0 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094800bc20 con 0x7f096815dda0 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0965400640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09480047c0 0x7f0948004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0965e00640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480039a0 0x7f0948003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0966800640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 0x7f0948005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0965400640 1 -- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 msgr2=0x7f0948005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0965400640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 0x7f0948005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0965400640 1 -- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480039a0 msgr2=0x7f0948003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0965400640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480039a0 0x7f0948003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0965400640 1 -- 172.21.15.39:0/2312351857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0948003520 con 0x7f09480047c0 2024-01-24T17:43:55.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0966800640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f096815dda0 0x7f0948005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:55.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.275+0000 7f0965e00640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480039a0 0x7f0948003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:55.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.276+0000 7f0965400640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09480047c0 0x7f0948004c40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f09680cb410 tx=0x7f09480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e0f42145f73172 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:55.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.277+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950034630 con 0x7f09480047c0 2024-01-24T17:43:55.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.277+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095002f070 con 0x7f09480047c0 2024-01-24T17:43:55.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.277+0000 7f0964a00640 1 -- 172.21.15.39:0/2312351857 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09500349d0 con 0x7f09480047c0 2024-01-24T17:43:55.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.277+0000 7f0967200640 1 -- 172.21.15.39:0/2312351857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09480047c0 msgr2=0x7f0948004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.277+0000 7f0967200640 1 --2- 172.21.15.39:0/2312351857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09480047c0 0x7f0948004c40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f09680cb410 tx=0x7f09480057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.277+0000 7f0966800640 1 -- 172.21.15.39:0/2312351857 reap_dead start 2024-01-24T17:43:55.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.277+0000 7f0967200640 1 -- 172.21.15.39:0/2312351857 shutdown_connections 2024-01-24T17:43:55.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.277+0000 7f0967200640 1 -- 172.21.15.39:0/2312351857 >> 172.21.15.39:0/2312351857 conn(0x7f0968099a00 msgr2=0x7f09680d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:55.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.278+0000 7f0967200640 1 -- 172.21.15.39:0/2312351857 shutdown_connections 2024-01-24T17:43:55.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.278+0000 7f0967200640 1 -- 172.21.15.39:0/2312351857 wait complete. 2024-01-24T17:43:55.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.279+0000 7f0967200640 1 Processor -- start 2024-01-24T17:43:55.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.279+0000 7f0967200640 1 -- start start 2024-01-24T17:43:55.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.280+0000 7f0967200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480047c0 0x7f0968152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.280+0000 7f0967200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09680cc480 0x7f09680cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.280+0000 7f0967200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f09680b32c0 0x7f09680b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.280+0000 7f0967200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680cce40 con 0x7f09680cc480 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.280+0000 7f0967200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680b3fb0 con 0x7f09480047c0 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.280+0000 7f0967200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09680b4290 con 0x7f09680b32c0 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09680cc480 0x7f09680cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09680cc480 0x7f09680cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35230/0 (socket says 172.21.15.39:35230) 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965400640 1 -- 172.21.15.39:0/3441747494 learned_addr learned my addr 172.21.15.39:0/3441747494 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0966800640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f09680b32c0 0x7f09680b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965e00640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480047c0 0x7f0968152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965400640 1 -- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f09680b32c0 msgr2=0x7f09680b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965400640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f09680b32c0 0x7f09680b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965400640 1 -- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480047c0 msgr2=0x7f0968152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965400640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480047c0 0x7f0968152c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.281+0000 7f0965400640 1 -- 172.21.15.39:0/3441747494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680b4570 con 0x7f09680cc480 2024-01-24T17:43:55.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.282+0000 7f0965400640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09680cc480 0x7f09680cc900 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f095000a810 tx=0x7f09680afcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:55.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.282+0000 7f095e000640 1 -- 172.21.15.39:0/3441747494 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950004b60 con 0x7f09680cc480 2024-01-24T17:43:55.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.282+0000 7f095e000640 1 -- 172.21.15.39:0/3441747494 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0950004ce0 con 0x7f09680cc480 2024-01-24T17:43:55.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.282+0000 7f095e000640 1 -- 172.21.15.39:0/3441747494 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09500359d0 con 0x7f09680cc480 2024-01-24T17:43:55.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.282+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09680b4850 con 0x7f09680cc480 2024-01-24T17:43:55.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.282+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09680c7770 con 0x7f09680cc480 2024-01-24T17:43:55.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.283+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09680b4a10 con 0x7f09680cc480 2024-01-24T17:43:55.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.284+0000 7f095e000640 1 -- 172.21.15.39:0/3441747494 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0950031030 con 0x7f09680cc480 2024-01-24T17:43:55.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.286+0000 7f095e000640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f094006a460 0x7f094006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.286+0000 7f095e000640 1 -- 172.21.15.39:0/3441747494 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f09500aae10 con 0x7f09680cc480 2024-01-24T17:43:55.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.286+0000 7f0965e00640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f094006a460 0x7f094006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.287+0000 7f0965e00640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f094006a460 0x7f094006c920 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0968156a60 tx=0x7f0958008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:55.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.290+0000 7f095e000640 1 -- 172.21.15.39:0/3441747494 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0950076380 con 0x7f09680cc480 2024-01-24T17:43:55.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.487+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09680cce40 con 0x7f09680cc480 2024-01-24T17:43:55.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.491+0000 7f095e000640 1 -- 172.21.15.39:0/3441747494 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16290 (secure 0 0 0) 0x7f095004c090 con 0x7f09680cc480 2024-01-24T17:43:55.493 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:43:55.493 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:43:16.261174+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:42:54.883732+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-24T17:42:55.727240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.1c","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.32","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:43:55.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f094006a460 msgr2=0x7f094006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f094006a460 0x7f094006c920 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0968156a60 tx=0x7f0958008040 comp rx=0 tx=0).stop 2024-01-24T17:43:55.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09680cc480 msgr2=0x7f09680cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09680cc480 0x7f09680cc900 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f095000a810 tx=0x7f09680afcb0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 shutdown_connections 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f09680b32c0 0x7f09680b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f094006a460 0x7f094006c920 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09680cc480 0x7f09680cc900 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 --2- 172.21.15.39:0/3441747494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09480047c0 0x7f0968152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.493+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 >> 172.21.15.39:0/3441747494 conn(0x7f0968099a00 msgr2=0x7f09680b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.494+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 shutdown_connections 2024-01-24T17:43:55.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.494+0000 7f0967200640 1 -- 172.21.15.39:0/3441747494 wait complete. 2024-01-24T17:43:55.513 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-24T17:43:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.727+0000 7ff145200640 1 Processor -- start 2024-01-24T17:43:55.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.727+0000 7ff145200640 1 -- start start 2024-01-24T17:43:55.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.728+0000 7ff145200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff14015a950 0x7ff1400ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.728+0000 7ff145200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff1400caaa0 0x7ff1400caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.728+0000 7ff145200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.728+0000 7ff145200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1400b43d0 con 0x7ff14015a950 2024-01-24T17:43:55.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.728+0000 7ff145200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1400c8f70 con 0x7ff1400caaa0 2024-01-24T17:43:55.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.728+0000 7ff145200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400cb8a0 con 0x7ff1400cb490 2024-01-24T17:43:55.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff14015a950 0x7ff1400ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff14015a950 0x7ff1400ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35244/0 (socket says 172.21.15.39:35244) 2024-01-24T17:43:55.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13de00640 1 -- 172.21.15.39:0/3917784724 learned_addr learned my addr 172.21.15.39:0/3917784724 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff137e00640 1 --2- 172.21.15.39:0/3917784724 >> v2:172.21.15.100:3300/0 conn(0x7ff1400caaa0 0x7ff1400caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13e800640 1 --2- 172.21.15.39:0/3917784724 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13de00640 1 -- 172.21.15.39:0/3917784724 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 msgr2=0x7ff1400affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13de00640 1 --2- 172.21.15.39:0/3917784724 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13de00640 1 -- 172.21.15.39:0/3917784724 >> v2:172.21.15.100:3300/0 conn(0x7ff1400caaa0 msgr2=0x7ff1400caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13de00640 1 --2- 172.21.15.39:0/3917784724 >> v2:172.21.15.100:3300/0 conn(0x7ff1400caaa0 0x7ff1400caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13de00640 1 -- 172.21.15.39:0/3917784724 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1400c9180 con 0x7ff14015a950 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.729+0000 7ff13e800640 1 --2- 172.21.15.39:0/3917784724 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:55.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.730+0000 7ff13de00640 1 --2- 172.21.15.39:0/3917784724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff14015a950 0x7ff1400ca560 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff1400c90b0 tx=0x7ff13002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cbf4c4633656be server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.731+0000 7ff13d400640 1 -- 172.21.15.39:0/3917784724 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff130031640 con 0x7ff14015a950 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.731+0000 7ff13d400640 1 -- 172.21.15.39:0/3917784724 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1300317a0 con 0x7ff14015a950 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.731+0000 7ff13d400640 1 -- 172.21.15.39:0/3917784724 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff130031b30 con 0x7ff14015a950 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.731+0000 7ff145200640 1 -- 172.21.15.39:0/3917784724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff14015a950 msgr2=0x7ff1400ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.731+0000 7ff145200640 1 --2- 172.21.15.39:0/3917784724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff14015a950 0x7ff1400ca560 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff1400c90b0 tx=0x7ff13002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.732+0000 7ff145200640 1 -- 172.21.15.39:0/3917784724 shutdown_connections 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.732+0000 7ff145200640 1 --2- 172.21.15.39:0/3917784724 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.732+0000 7ff145200640 1 --2- 172.21.15.39:0/3917784724 >> v2:172.21.15.100:3300/0 conn(0x7ff1400caaa0 0x7ff1400caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.732+0000 7ff145200640 1 --2- 172.21.15.39:0/3917784724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff14015a950 0x7ff1400ca560 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.732+0000 7ff145200640 1 -- 172.21.15.39:0/3917784724 >> 172.21.15.39:0/3917784724 conn(0x7ff14006a3e0 msgr2=0x7ff14006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.732+0000 7ff145200640 1 -- 172.21.15.39:0/3917784724 shutdown_connections 2024-01-24T17:43:55.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.732+0000 7ff145200640 1 -- 172.21.15.39:0/3917784724 wait complete. 2024-01-24T17:43:55.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.734+0000 7ff145200640 1 Processor -- start 2024-01-24T17:43:55.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.734+0000 7ff145200640 1 -- start start 2024-01-24T17:43:55.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.735+0000 7ff145200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1400caaa0 0x7ff140156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.735+0000 7ff145200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.735+0000 7ff145200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff14015a950 0x7ff1400c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.735+0000 7ff145200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400cc320 con 0x7ff1400caaa0 2024-01-24T17:43:55.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.735+0000 7ff145200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1400b2390 con 0x7ff14015a950 2024-01-24T17:43:55.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.735+0000 7ff145200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14015d5d0 con 0x7ff1400cb490 2024-01-24T17:43:55.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1400caaa0 0x7ff140156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1400caaa0 0x7ff140156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35256/0 (socket says 172.21.15.39:35256) 2024-01-24T17:43:55.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13de00640 1 -- 172.21.15.39:0/3398985913 learned_addr learned my addr 172.21.15.39:0/3398985913 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff137e00640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13e800640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff14015a950 0x7ff1400c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13de00640 1 -- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 msgr2=0x7ff1400c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13de00640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13de00640 1 -- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff14015a950 msgr2=0x7ff1400c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13de00640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff14015a950 0x7ff1400c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13de00640 1 -- 172.21.15.39:0/3398985913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300094d0 con 0x7ff1400caaa0 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff13e800640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff14015a950 0x7ff1400c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.736+0000 7ff137e00640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.737+0000 7ff13de00640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1400caaa0 0x7ff140156d10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff130000c00 tx=0x7ff1300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.737+0000 7ff136000640 1 -- 172.21.15.39:0/3398985913 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13003ac60 con 0x7ff1400caaa0 2024-01-24T17:43:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.737+0000 7ff136000640 1 -- 172.21.15.39:0/3398985913 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13003ade0 con 0x7ff1400caaa0 2024-01-24T17:43:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.737+0000 7ff136000640 1 -- 172.21.15.39:0/3398985913 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff130042630 con 0x7ff1400caaa0 2024-01-24T17:43:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.737+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14015d8b0 con 0x7ff1400caaa0 2024-01-24T17:43:55.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.737+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff140157250 con 0x7ff1400caaa0 2024-01-24T17:43:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.739+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14015ddd0 con 0x7ff1400caaa0 2024-01-24T17:43:55.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.739+0000 7ff136000640 1 -- 172.21.15.39:0/3398985913 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff13002f050 con 0x7ff1400caaa0 2024-01-24T17:43:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.740+0000 7ff136000640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff11806a460 0x7ff11806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.740+0000 7ff136000640 1 -- 172.21.15.39:0/3398985913 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff1300aa640 con 0x7ff1400caaa0 2024-01-24T17:43:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.741+0000 7ff137e00640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff11806a460 0x7ff11806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.742+0000 7ff137e00640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff11806a460 0x7ff11806c920 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff1400c52c0 tx=0x7ff128008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:55.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.744+0000 7ff136000640 1 -- 172.21.15.39:0/3398985913 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff130075b20 con 0x7ff1400caaa0 2024-01-24T17:43:55.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.977+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff1401573e0 con 0x7ff1400caaa0 2024-01-24T17:43:55.981 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:43:55.979+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:43:55.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.993+0000 7ff136000640 1 -- 172.21.15.39:0/3398985913 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff130031e60 con 0x7ff1400caaa0 2024-01-24T17:43:55.994 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-24T17:43:55.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff11806a460 msgr2=0x7ff11806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff11806a460 0x7ff11806c920 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff1400c52c0 tx=0x7ff128008040 comp rx=0 tx=0).stop 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1400caaa0 msgr2=0x7ff140156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1400caaa0 0x7ff140156d10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff130000c00 tx=0x7ff1300046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 shutdown_connections 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff14015a950 0x7ff1400c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff1400cb490 0x7ff1400c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff11806a460 0x7ff11806c920 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 --2- 172.21.15.39:0/3398985913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1400caaa0 0x7ff140156d10 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 >> 172.21.15.39:0/3398985913 conn(0x7ff14006a3e0 msgr2=0x7ff140159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 shutdown_connections 2024-01-24T17:43:55.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:55.996+0000 7ff145200640 1 -- 172.21.15.39:0/3398985913 wait complete. 2024-01-24T17:43:56.011 INFO:tasks.ceph.mds.a.smithi039.stderr: -2517> 2024-01-24T17:42:58.984+0000 7f6ff1200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:43:56.020 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-24T17:43:56.096 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:43:56.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.218+0000 7f4314000640 1 Processor -- start 2024-01-24T17:43:56.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.218+0000 7f4314000640 1 -- start start 2024-01-24T17:43:56.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.219+0000 7f4314000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f430c15cd20 0x7f430c0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.219+0000 7f4314000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f430c0d21f0 0x7f430c0b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.219+0000 7f4314000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430c0b6080 con 0x7f430c0cf420 2024-01-24T17:43:56.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.219+0000 7f4314000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f430c0b8a50 con 0x7f430c0d21f0 2024-01-24T17:43:56.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.219+0000 7f4314000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c15d100 con 0x7f430c15cd20 2024-01-24T17:43:56.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.220+0000 7f4312000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f430c0cf420 0x7f430c0d0c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50822/0 (socket says 172.21.15.39:50822) 2024-01-24T17:43:56.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.220+0000 7f4312000640 1 -- 172.21.15.39:0/4293069895 learned_addr learned my addr 172.21.15.39:0/4293069895 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.220+0000 7f4311600640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f430c15cd20 0x7f430c0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.220+0000 7f4310c00640 1 --2- 172.21.15.39:0/4293069895 >> v2:172.21.15.100:3300/0 conn(0x7f430c0d21f0 0x7f430c0b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.220+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219496381 0 0) 0x7f430c0b6080 con 0x7f430c0cf420 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.220+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42ec003520 con 0x7f430c0cf420 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749800188 0 0) 0x7f42ec003520 con 0x7f430c0cf420 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f430c15cd20 msgr2=0x7f430c0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f430c15cd20 0x7f430c0cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 >> v2:172.21.15.100:3300/0 conn(0x7f430c0d21f0 msgr2=0x7f430c0b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 --2- 172.21.15.39:0/4293069895 >> v2:172.21.15.100:3300/0 conn(0x7f430c0d21f0 0x7f430c0b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c0b84d0 con 0x7f430c0cf420 2024-01-24T17:43:56.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f43040019f0 con 0x7f430c0cf420 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 >> v1:172.21.15.39:6789/0 conn(0x7f430c0cf420 legacy=0x7f430c0d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42ec003970 0x7f42ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f430c15cd20 0x7f42ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42ec004d00 0x7f42ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0b8a50 con 0x7f42ec004d00 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c15d100 con 0x7f42ec003970 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec00b9b0 con 0x7f430c15cd20 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f4312000640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42ec004d00 0x7f42ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f4311600640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42ec003970 0x7f42ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.221+0000 7f4310c00640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f430c15cd20 0x7f42ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.222+0000 7f4312000640 1 -- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f430c15cd20 msgr2=0x7f42ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.222+0000 7f4312000640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f430c15cd20 0x7f42ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:56.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.222+0000 7f4312000640 1 -- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42ec003970 msgr2=0x7f42ec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:56.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.222+0000 7f4312000640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42ec003970 0x7f42ec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:56.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.222+0000 7f4312000640 1 -- 172.21.15.39:0/4293069895 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec003520 con 0x7f42ec004d00 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.223+0000 7f4312000640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42ec004d00 0x7f42ec005180 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f430400a810 tx=0x7f42ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957fb2e38f768527 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.223+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304034630 con 0x7f42ec004d00 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.223+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430402f070 con 0x7f42ec004d00 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.223+0000 7f430be00640 1 -- 172.21.15.39:0/4293069895 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43040349f0 con 0x7f42ec004d00 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.224+0000 7f4314000640 1 -- 172.21.15.39:0/4293069895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42ec004d00 msgr2=0x7f42ec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.224+0000 7f4314000640 1 --2- 172.21.15.39:0/4293069895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42ec004d00 0x7f42ec005180 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f430400a810 tx=0x7f42ec0057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.224+0000 7f4312000640 1 -- 172.21.15.39:0/4293069895 reap_dead start 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.224+0000 7f4314000640 1 -- 172.21.15.39:0/4293069895 shutdown_connections 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.224+0000 7f4314000640 1 -- 172.21.15.39:0/4293069895 >> 172.21.15.39:0/4293069895 conn(0x7f430c06ae50 msgr2=0x7f430c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.224+0000 7f4314000640 1 -- 172.21.15.39:0/4293069895 shutdown_connections 2024-01-24T17:43:56.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.224+0000 7f4314000640 1 -- 172.21.15.39:0/4293069895 wait complete. 2024-01-24T17:43:56.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.226+0000 7f4314000640 1 Processor -- start 2024-01-24T17:43:56.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.226+0000 7f4314000640 1 -- start start 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.227+0000 7f4314000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42ec004d00 0x7f430c14c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.227+0000 7f4314000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f430c0b2540 0x7f430c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.227+0000 7f4314000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f430c0d0b80 0x7f430c0d1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.227+0000 7f4314000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0b5260 con 0x7f430c0d0b80 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.227+0000 7f4314000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0d1540 con 0x7f430c0b2540 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.227+0000 7f4314000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c0c6540 con 0x7f42ec004d00 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.227+0000 7f4312000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f430c0d0b80 0x7f430c0d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4312000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f430c0d0b80 0x7f430c0d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35302/0 (socket says 172.21.15.39:35302) 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4312000640 1 -- 172.21.15.39:0/322141018 learned_addr learned my addr 172.21.15.39:0/322141018 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4310c00640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f430c0b2540 0x7f430c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4311600640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42ec004d00 0x7f430c14c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4312000640 1 -- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42ec004d00 msgr2=0x7f430c14c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:56.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4312000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42ec004d00 0x7f430c14c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:56.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4312000640 1 -- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f430c0b2540 msgr2=0x7f430c0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:56.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4312000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f430c0b2540 0x7f430c0b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:56.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4312000640 1 -- 172.21.15.39:0/322141018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c0b29c0 con 0x7f430c0d0b80 2024-01-24T17:43:56.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4310c00640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f430c0b2540 0x7f430c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:56.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.228+0000 7f4312000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f430c0d0b80 0x7f430c0d1000 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f430402cba0 tx=0x7f430c0b8660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:56.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.229+0000 7f4309600640 1 -- 172.21.15.39:0/322141018 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304001ef0 con 0x7f430c0d0b80 2024-01-24T17:43:56.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.229+0000 7f4309600640 1 -- 172.21.15.39:0/322141018 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4304034640 con 0x7f430c0d0b80 2024-01-24T17:43:56.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.229+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c0b46e0 con 0x7f430c0d0b80 2024-01-24T17:43:56.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.229+0000 7f4309600640 1 -- 172.21.15.39:0/322141018 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43040349b0 con 0x7f430c0d0b80 2024-01-24T17:43:56.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.229+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c0b4ca0 con 0x7f430c0d0b80 2024-01-24T17:43:56.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.230+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430c067040 con 0x7f430c0d0b80 2024-01-24T17:43:56.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.231+0000 7f4309600640 1 -- 172.21.15.39:0/322141018 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4304031030 con 0x7f430c0d0b80 2024-01-24T17:43:56.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.232+0000 7f4309600640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42e406a430 0x7f42e406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:56.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.232+0000 7f4309600640 1 -- 172.21.15.39:0/322141018 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f43040aada0 con 0x7f430c0d0b80 2024-01-24T17:43:56.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.232+0000 7f4311600640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42e406a430 0x7f42e406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:56.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.234+0000 7f4311600640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42e406a430 0x7f42e406c8f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009990 tx=0x7f42fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:56.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.236+0000 7f4309600640 1 -- 172.21.15.39:0/322141018 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4304079cb0 con 0x7f430c0d0b80 2024-01-24T17:43:56.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.471+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f430c067040 con 0x7f430c0d0b80 2024-01-24T17:43:57.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:56.998+0000 7f4309600640 1 -- 172.21.15.39:0/322141018 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4304079840 con 0x7f430c0d0b80 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42e406a430 msgr2=0x7f42e406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42e406a430 0x7f42e406c8f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009990 tx=0x7f42fc008040 comp rx=0 tx=0).stop 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f430c0d0b80 msgr2=0x7f430c0d1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f430c0d0b80 0x7f430c0d1000 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f430402cba0 tx=0x7f430c0b8660 comp rx=0 tx=0).stop 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 shutdown_connections 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42e406a430 0x7f42e406c8f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f430c0d0b80 0x7f430c0d1000 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f430c0b2540 0x7f430c0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 --2- 172.21.15.39:0/322141018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42ec004d00 0x7f430c14c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 >> 172.21.15.39:0/322141018 conn(0x7f430c06ae50 msgr2=0x7f430c0ca850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:57.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 shutdown_connections 2024-01-24T17:43:57.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.003+0000 7f4314000640 1 -- 172.21.15.39:0/322141018 wait complete. 2024-01-24T17:43:57.027 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-24T17:43:57.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.244+0000 7f2df2000640 1 Processor -- start 2024-01-24T17:43:57.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.244+0000 7f2df2000640 1 -- start start 2024-01-24T17:43:57.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.246+0000 7f2df2000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 0x7f2dec0cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:57.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.246+0000 7f2df2000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec06cc10 con 0x7f2dec0cb1c0 2024-01-24T17:43:57.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.246+0000 7f2df2000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec0ce730 con 0x7f2dec15c860 2024-01-24T17:43:57.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.246+0000 7f2df2000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0cea10 con 0x7f2dec0ce200 2024-01-24T17:43:57.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.246+0000 7f2df1600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2dec0cb1c0 0x7f2dec0d0be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50832/0 (socket says 172.21.15.39:50832) 2024-01-24T17:43:57.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.246+0000 7f2df1600640 1 -- 172.21.15.39:0/2720970649 learned_addr learned my addr 172.21.15.39:0/2720970649 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:43:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620406056 0 0) 0x7f2dec06cc10 con 0x7f2dec0cb1c0 2024-01-24T17:43:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dcc003520 con 0x7f2dec0cb1c0 2024-01-24T17:43:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2debe00640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 0x7f2dec0cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1126893806 0 0) 0x7f2dcc003520 con 0x7f2dec0cb1c0 2024-01-24T17:43:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 msgr2=0x7f2dec0cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 0x7f2dec0cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 >> v1:172.21.15.100:6789/0 conn(0x7f2dec15c860 legacy=0x7f2dec160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0cecf0 con 0x7f2dec0cb1c0 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2de4001670 con 0x7f2dec0cb1c0 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 >> v1:172.21.15.39:6789/0 conn(0x7f2dec0cb1c0 legacy=0x7f2dec0d0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dcc0039a0 0x7f2dcc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.247+0000 7f2deb400640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dcc0047c0 0x7f2dcc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2deb400640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 0x7f2dcc0051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0cea10 con 0x7f2dcc0047c0 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc00bac0 con 0x7f2dcc0039a0 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc00bc80 con 0x7f2dec0ce200 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2debe00640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dcc0047c0 0x7f2dcc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2df0c00640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dcc0039a0 0x7f2dcc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2df1600640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 0x7f2dcc0051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2debe00640 1 -- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 msgr2=0x7f2dcc0051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:57.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2debe00640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 0x7f2dcc0051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2debe00640 1 -- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dcc0039a0 msgr2=0x7f2dcc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:57.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2debe00640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dcc0039a0 0x7f2dcc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2debe00640 1 -- 172.21.15.39:0/2720970649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc003520 con 0x7f2dcc0047c0 2024-01-24T17:43:57.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.248+0000 7f2df1600640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dec0ce200 0x7f2dcc0051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:57.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.249+0000 7f2debe00640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dcc0047c0 0x7f2dcc004c40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400ba30 tx=0x7f2dcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5254f774c731c1df server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:57.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.250+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4032660 con 0x7f2dcc0047c0 2024-01-24T17:43:57.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.250+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd40327c0 con 0x7f2dcc0047c0 2024-01-24T17:43:57.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.250+0000 7f2deb400640 1 -- 172.21.15.39:0/2720970649 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4032b50 con 0x7f2dcc0047c0 2024-01-24T17:43:57.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.250+0000 7f2df2000640 1 -- 172.21.15.39:0/2720970649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dcc0047c0 msgr2=0x7f2dcc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:57.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.250+0000 7f2df2000640 1 --2- 172.21.15.39:0/2720970649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dcc0047c0 0x7f2dcc004c40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400ba30 tx=0x7f2dcc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.250+0000 7f2df1600640 1 -- 172.21.15.39:0/2720970649 reap_dead start 2024-01-24T17:43:57.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.251+0000 7f2df2000640 1 -- 172.21.15.39:0/2720970649 shutdown_connections 2024-01-24T17:43:57.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.251+0000 7f2df2000640 1 -- 172.21.15.39:0/2720970649 >> 172.21.15.39:0/2720970649 conn(0x7f2dec069c70 msgr2=0x7f2dec06a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:57.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.251+0000 7f2df2000640 1 -- 172.21.15.39:0/2720970649 shutdown_connections 2024-01-24T17:43:57.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.251+0000 7f2df2000640 1 -- 172.21.15.39:0/2720970649 wait complete. 2024-01-24T17:43:57.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.252+0000 7f2df2000640 1 Processor -- start 2024-01-24T17:43:57.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.253+0000 7f2df2000640 1 -- start start 2024-01-24T17:43:57.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.253+0000 7f2df2000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dcc0047c0 0x7f2dec0ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:57.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.253+0000 7f2df2000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dec0b40b0 0x7f2dec15bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:57.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dec0c98d0 0x7f2dec0c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:57.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df2000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0b5430 con 0x7f2dec0c98d0 2024-01-24T17:43:57.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df2000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0ca290 con 0x7f2dec0b40b0 2024-01-24T17:43:57.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df2000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0acce0 con 0x7f2dcc0047c0 2024-01-24T17:43:57.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dec0c98d0 0x7f2dec0c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:57.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dec0c98d0 0x7f2dec0c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35312/0 (socket says 172.21.15.39:35312) 2024-01-24T17:43:57.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df1600640 1 -- 172.21.15.39:0/3932447391 learned_addr learned my addr 172.21.15.39:0/3932447391 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:57.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df0c00640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dcc0047c0 0x7f2dec0ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:57.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df1600640 1 -- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dcc0047c0 msgr2=0x7f2dec0ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:57.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2debe00640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dec0b40b0 0x7f2dec15bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:57.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df1600640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dcc0047c0 0x7f2dec0ab8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df1600640 1 -- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dec0b40b0 msgr2=0x7f2dec15bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:57.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df1600640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dec0b40b0 0x7f2dec15bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:57.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.254+0000 7f2df1600640 1 -- 172.21.15.39:0/3932447391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0abe00 con 0x7f2dec0c98d0 2024-01-24T17:43:57.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.255+0000 7f2df1600640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dec0c98d0 0x7f2dec0c9d50 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2dec06f150 tx=0x7f2de400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:57.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.255+0000 7f2de8c00640 1 -- 172.21.15.39:0/3932447391 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4007440 con 0x7f2dec0c98d0 2024-01-24T17:43:57.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.255+0000 7f2de8c00640 1 -- 172.21.15.39:0/3932447391 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de40075a0 con 0x7f2dec0c98d0 2024-01-24T17:43:57.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.255+0000 7f2de8c00640 1 -- 172.21.15.39:0/3932447391 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4004a20 con 0x7f2dec0c98d0 2024-01-24T17:43:57.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.255+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dec0ac0d0 con 0x7f2dec0c98d0 2024-01-24T17:43:57.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.255+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dec0ac690 con 0x7f2dec0c98d0 2024-01-24T17:43:57.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.256+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dec04dff0 con 0x7f2dec0c98d0 2024-01-24T17:43:57.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.257+0000 7f2de8c00640 1 -- 172.21.15.39:0/3932447391 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2de4004b80 con 0x7f2dec0c98d0 2024-01-24T17:43:57.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.258+0000 7f2de8c00640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2dc406a430 0x7f2dc406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:57.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.258+0000 7f2de8c00640 1 -- 172.21.15.39:0/3932447391 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f2de4088710 con 0x7f2dec0c98d0 2024-01-24T17:43:57.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.259+0000 7f2df0c00640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2dc406a430 0x7f2dc406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:57.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.260+0000 7f2df0c00640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2dc406a430 0x7f2dc406c8f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc007fd0 tx=0x7f2ddc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:57.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.263+0000 7f2de8c00640 1 -- 172.21.15.39:0/3932447391 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2de4053b90 con 0x7f2dec0c98d0 2024-01-24T17:43:57.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:57.462+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2dec0ace20 con 0x7f2dec0c98d0 2024-01-24T17:43:58.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.382+0000 7f2de8c00640 1 -- 172.21.15.39:0/3932447391 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v38) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2de40571b0 con 0x7f2dec0c98d0 2024-01-24T17:43:58.385 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-01-24T17:43:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.387+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2dc406a430 msgr2=0x7f2dc406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.387+0000 7f2df2000640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2dc406a430 0x7f2dc406c8f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc007fd0 tx=0x7f2ddc009040 comp rx=0 tx=0).stop 2024-01-24T17:43:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.387+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dec0c98d0 msgr2=0x7f2dec0c9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.387+0000 7f2df2000640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dec0c98d0 0x7f2dec0c9d50 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2dec06f150 tx=0x7f2de400d9d0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.388+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 shutdown_connections 2024-01-24T17:43:58.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.388+0000 7f2df2000640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2dc406a430 0x7f2dc406c8f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.388+0000 7f2df2000640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2dec0c98d0 0x7f2dec0c9d50 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.388+0000 7f2df2000640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2dec0b40b0 0x7f2dec15bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.388+0000 7f2df2000640 1 --2- 172.21.15.39:0/3932447391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2dcc0047c0 0x7f2dec0ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.388+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 >> 172.21.15.39:0/3932447391 conn(0x7f2dec069c70 msgr2=0x7f2dec0b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.388+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 shutdown_connections 2024-01-24T17:43:58.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.388+0000 7f2df2000640 1 -- 172.21.15.39:0/3932447391 wait complete. 2024-01-24T17:43:58.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-24T17:43:58.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.632+0000 7efc89200640 1 Processor -- start 2024-01-24T17:43:58.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.632+0000 7efc89200640 1 -- start start 2024-01-24T17:43:58.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.633+0000 7efc89200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efc840b0350 0x7efc8406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:58.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.633+0000 7efc89200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc8406c9f0 0x7efc8415da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:58.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.633+0000 7efc89200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8406d4c0 con 0x7efc840b0730 2024-01-24T17:43:58.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.633+0000 7efc89200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc8406cf00 con 0x7efc840b0350 2024-01-24T17:43:58.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.633+0000 7efc89200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8406d1e0 con 0x7efc8406c9f0 2024-01-24T17:43:58.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.634+0000 7efc81e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efc840b0350 0x7efc8406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:58.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.634+0000 7efc81400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc8406c9f0 0x7efc8415da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:58.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.634+0000 7efc81400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc8406c9f0 0x7efc8415da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:44590/0 (socket says 172.21.15.39:44590) 2024-01-24T17:43:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.634+0000 7efc81400640 1 -- 172.21.15.39:0/1178230499 learned_addr learned my addr 172.21.15.39:0/1178230499 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.634+0000 7efc82800640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7efc840b0730 0x7efc84070010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:60712/0 (socket says 172.21.15.39:60712) 2024-01-24T17:43:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.635+0000 7efc81e00640 1 -- 172.21.15.39:0/1178230499 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc8406c9f0 msgr2=0x7efc8415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.635+0000 7efc81e00640 1 --2- 172.21.15.39:0/1178230499 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc8406c9f0 0x7efc8415da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.635+0000 7efc81e00640 1 -- 172.21.15.39:0/1178230499 >> v1:172.21.15.100:6789/0 conn(0x7efc840b0730 legacy=0x7efc84070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.635+0000 7efc81e00640 1 -- 172.21.15.39:0/1178230499 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc840b15e0 con 0x7efc840b0350 2024-01-24T17:43:58.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.636+0000 7efc81e00640 1 --2- 172.21.15.39:0/1178230499 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 0x7efc8406c4b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc740097a0 tx=0x7efc7402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17a05a5f629957fe server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:58.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.637+0000 7efc80a00640 1 -- 172.21.15.39:0/1178230499 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74031780 con 0x7efc840b0350 2024-01-24T17:43:58.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.637+0000 7efc80a00640 1 -- 172.21.15.39:0/1178230499 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc740318e0 con 0x7efc840b0350 2024-01-24T17:43:58.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.637+0000 7efc80a00640 1 -- 172.21.15.39:0/1178230499 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74031c70 con 0x7efc840b0350 2024-01-24T17:43:58.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.638+0000 7efc89200640 1 -- 172.21.15.39:0/1178230499 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 msgr2=0x7efc8406c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:58.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.638+0000 7efc89200640 1 --2- 172.21.15.39:0/1178230499 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 0x7efc8406c4b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc740097a0 tx=0x7efc7402b820 comp rx=0 tx=0).stop 2024-01-24T17:43:58.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.638+0000 7efc89200640 1 -- 172.21.15.39:0/1178230499 shutdown_connections 2024-01-24T17:43:58.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.638+0000 7efc89200640 1 --2- 172.21.15.39:0/1178230499 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc8406c9f0 0x7efc8415da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.638+0000 7efc89200640 1 --2- 172.21.15.39:0/1178230499 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 0x7efc8406c4b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.638+0000 7efc89200640 1 -- 172.21.15.39:0/1178230499 >> 172.21.15.39:0/1178230499 conn(0x7efc840cb0e0 msgr2=0x7efc840cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:58.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.638+0000 7efc89200640 1 -- 172.21.15.39:0/1178230499 shutdown_connections 2024-01-24T17:43:58.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.638+0000 7efc89200640 1 -- 172.21.15.39:0/1178230499 wait complete. 2024-01-24T17:43:58.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.640+0000 7efc89200640 1 Processor -- start 2024-01-24T17:43:58.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.640+0000 7efc89200640 1 -- start start 2024-01-24T17:43:58.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc89200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc8406c9f0 0x7efc84152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:58.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc89200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 0x7efc84156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:58.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc89200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc840b0730 0x7efc84156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:58.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc89200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84158300 con 0x7efc8406c9f0 2024-01-24T17:43:58.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc89200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84158550 con 0x7efc840b0350 2024-01-24T17:43:58.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc89200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc84158830 con 0x7efc840b0730 2024-01-24T17:43:58.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc81e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc8406c9f0 0x7efc84152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:58.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc81e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc8406c9f0 0x7efc84152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35322/0 (socket says 172.21.15.39:35322) 2024-01-24T17:43:58.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.641+0000 7efc81e00640 1 -- 172.21.15.39:0/35351609 learned_addr learned my addr 172.21.15.39:0/35351609 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:58.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc81400640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 0x7efc84156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:58.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc82800640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc840b0730 0x7efc84156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:58.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc81e00640 1 -- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc840b0730 msgr2=0x7efc84156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:58.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc81e00640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc840b0730 0x7efc84156b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc81e00640 1 -- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 msgr2=0x7efc84156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:58.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc81e00640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 0x7efc84156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:58.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc81e00640 1 -- 172.21.15.39:0/35351609 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74009400 con 0x7efc8406c9f0 2024-01-24T17:43:58.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc81400640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 0x7efc84156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:43:58.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc81e00640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc8406c9f0 0x7efc84152a10 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efc74034040 tx=0x7efc740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:58.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc7a000640 1 -- 172.21.15.39:0/35351609 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74032820 con 0x7efc8406c9f0 2024-01-24T17:43:58.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc7a000640 1 -- 172.21.15.39:0/35351609 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc740329a0 con 0x7efc8406c9f0 2024-01-24T17:43:58.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc841e2700 con 0x7efc8406c9f0 2024-01-24T17:43:58.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.642+0000 7efc7a000640 1 -- 172.21.15.39:0/35351609 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74032d50 con 0x7efc8406c9f0 2024-01-24T17:43:58.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.643+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc84152f50 con 0x7efc8406c9f0 2024-01-24T17:43:58.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.644+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc841e2560 con 0x7efc8406c9f0 2024-01-24T17:43:58.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.645+0000 7efc7a000640 1 -- 172.21.15.39:0/35351609 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efc7404e440 con 0x7efc8406c9f0 2024-01-24T17:43:58.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.646+0000 7efc7a000640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5c06a430 0x7efc5c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:58.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.646+0000 7efc7a000640 1 -- 172.21.15.39:0/35351609 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5473+0+0 (secure 0 0 0) 0x7efc740aaa30 con 0x7efc8406c9f0 2024-01-24T17:43:58.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.646+0000 7efc81400640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5c06a430 0x7efc5c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:58.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.647+0000 7efc81400640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5c06a430 0x7efc5c06c8f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efc84157370 tx=0x7efc6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:58.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.650+0000 7efc7a000640 1 -- 172.21.15.39:0/35351609 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efc841e2560 con 0x7efc8406c9f0 2024-01-24T17:43:58.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:58.847+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efc841e2560 con 0x7efc8406c9f0 2024-01-24T17:43:59.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.382+0000 7efc7a000640 1 -- 172.21.15.39:0/35351609 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v39) v1 ==== 169+0+0 (secure 0 0 0) 0x7efc841e2560 con 0x7efc8406c9f0 2024-01-24T17:43:59.385 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-01-24T17:43:59.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5c06a430 msgr2=0x7efc5c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:59.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5c06a430 0x7efc5c06c8f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efc84157370 tx=0x7efc6c008040 comp rx=0 tx=0).stop 2024-01-24T17:43:59.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc8406c9f0 msgr2=0x7efc84152a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:59.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc8406c9f0 0x7efc84152a10 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efc74034040 tx=0x7efc740046c0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 shutdown_connections 2024-01-24T17:43:59.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc840b0730 0x7efc84156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc840b0350 0x7efc84156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5c06a430 0x7efc5c06c8f0 secure :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efc84157370 tx=0x7efc6c008040 comp rx=0 tx=0).stop 2024-01-24T17:43:59.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 --2- 172.21.15.39:0/35351609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc8406c9f0 0x7efc84152a10 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.388+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 >> 172.21.15.39:0/35351609 conn(0x7efc840cb0e0 msgr2=0x7efc840c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:59.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.389+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 shutdown_connections 2024-01-24T17:43:59.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.389+0000 7efc89200640 1 -- 172.21.15.39:0/35351609 wait complete. 2024-01-24T17:43:59.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-24T17:43:59.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.620+0000 7f661b400640 1 Processor -- start 2024-01-24T17:43:59.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.621+0000 7f661b400640 1 -- start start 2024-01-24T17:43:59.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.622+0000 7f661b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f661415cad0 0x7f66140d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:59.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.622+0000 7f661b400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f66140b16e0 0x7f66140d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:59.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.622+0000 7f661b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f66140d1350 0x7f6614160710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:59.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.622+0000 7f661b400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66140cfe10 con 0x7f66140d1350 2024-01-24T17:43:59.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.622+0000 7f661b400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66140d37b0 con 0x7f66140b16e0 2024-01-24T17:43:59.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.622+0000 7f661b400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66140d1780 con 0x7f661415cad0 2024-01-24T17:43:59.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.622+0000 7f661aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f66140d1350 0x7f6614160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:59.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f66140d1350 0x7f6614160710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35334/0 (socket says 172.21.15.39:35334) 2024-01-24T17:43:59.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661aa00640 1 -- 172.21.15.39:0/4065564119 learned_addr learned my addr 172.21.15.39:0/4065564119 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:59.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661a000640 1 --2- 172.21.15.39:0/4065564119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f661415cad0 0x7f66140d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:59.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f6619600640 1 --2- 172.21.15.39:0/4065564119 >> v2:172.21.15.100:3300/0 conn(0x7f66140b16e0 0x7f66140d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:59.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661aa00640 1 -- 172.21.15.39:0/4065564119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f661415cad0 msgr2=0x7f66140d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:59.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661aa00640 1 --2- 172.21.15.39:0/4065564119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f661415cad0 0x7f66140d0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661aa00640 1 -- 172.21.15.39:0/4065564119 >> v2:172.21.15.100:3300/0 conn(0x7f66140b16e0 msgr2=0x7f66140d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:43:59.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661aa00640 1 --2- 172.21.15.39:0/4065564119 >> v2:172.21.15.100:3300/0 conn(0x7f66140b16e0 0x7f66140d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661aa00640 1 -- 172.21.15.39:0/4065564119 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66140b2040 con 0x7f66140d1350 2024-01-24T17:43:59.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f661a000640 1 --2- 172.21.15.39:0/4065564119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f661415cad0 0x7f66140d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:59.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.623+0000 7f6619600640 1 --2- 172.21.15.39:0/4065564119 >> v2:172.21.15.100:3300/0 conn(0x7f66140b16e0 0x7f66140d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:43:59.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.625+0000 7f661aa00640 1 --2- 172.21.15.39:0/4065564119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140d1350 0x7f6614160710 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f661408c1b0 tx=0x7f660c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bff2d72ec2fcb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:59.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.625+0000 7f6618c00640 1 -- 172.21.15.39:0/4065564119 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c031750 con 0x7f66140d1350 2024-01-24T17:43:59.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.625+0000 7f6618c00640 1 -- 172.21.15.39:0/4065564119 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c0318b0 con 0x7f66140d1350 2024-01-24T17:43:59.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.625+0000 7f6618c00640 1 -- 172.21.15.39:0/4065564119 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c031c40 con 0x7f66140d1350 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 -- 172.21.15.39:0/4065564119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140d1350 msgr2=0x7f6614160710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 --2- 172.21.15.39:0/4065564119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140d1350 0x7f6614160710 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f661408c1b0 tx=0x7f660c02b7f0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 -- 172.21.15.39:0/4065564119 shutdown_connections 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 --2- 172.21.15.39:0/4065564119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140d1350 0x7f6614160710 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 --2- 172.21.15.39:0/4065564119 >> v2:172.21.15.100:3300/0 conn(0x7f66140b16e0 0x7f66140d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 --2- 172.21.15.39:0/4065564119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f661415cad0 0x7f66140d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 -- 172.21.15.39:0/4065564119 >> 172.21.15.39:0/4065564119 conn(0x7f66140d2830 msgr2=0x7f66140d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 -- 172.21.15.39:0/4065564119 shutdown_connections 2024-01-24T17:43:59.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.626+0000 7f661b400640 1 -- 172.21.15.39:0/4065564119 wait complete. 2024-01-24T17:43:59.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.628+0000 7f661b400640 1 Processor -- start 2024-01-24T17:43:59.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.628+0000 7f661b400640 1 -- start start 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.629+0000 7f661b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140b16e0 0x7f66140b7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66140d1350 0x7f66140c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661b400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f661415cad0 0x7f6614158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661b400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66140b6230 con 0x7f66140b16e0 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661b400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66140b6770 con 0x7f661415cad0 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661b400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66140b6a50 con 0x7f66140d1350 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140b16e0 0x7f66140b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140b16e0 0x7f66140b7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35344/0 (socket says 172.21.15.39:35344) 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661a000640 1 -- 172.21.15.39:0/3883644180 learned_addr learned my addr 172.21.15.39:0/3883644180 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661a000640 1 -- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66140d1350 msgr2=0x7f66140c8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661a000640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66140d1350 0x7f66140c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661a000640 1 -- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f661415cad0 msgr2=0x7f6614158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:59.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661a000640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f661415cad0 0x7f6614158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.630+0000 7f661a000640 1 -- 172.21.15.39:0/3883644180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c0094d0 con 0x7f66140b16e0 2024-01-24T17:43:59.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.631+0000 7f661a000640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140b16e0 0x7f66140b7530 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6604007fa0 tx=0x7f660400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:59.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.631+0000 7f6612000640 1 -- 172.21.15.39:0/3883644180 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604003b90 con 0x7f66140b16e0 2024-01-24T17:43:59.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.631+0000 7f6612000640 1 -- 172.21.15.39:0/3883644180 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660400f040 con 0x7f66140b16e0 2024-01-24T17:43:59.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.631+0000 7f6612000640 1 -- 172.21.15.39:0/3883644180 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604004030 con 0x7f66140b16e0 2024-01-24T17:43:59.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.631+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66140b6d30 con 0x7f66140b16e0 2024-01-24T17:43:59.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.631+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66140c8c60 con 0x7f66140b16e0 2024-01-24T17:43:59.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.632+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661404dff0 con 0x7f66140b16e0 2024-01-24T17:43:59.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.633+0000 7f6612000640 1 -- 172.21.15.39:0/3883644180 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6604004190 con 0x7f66140b16e0 2024-01-24T17:43:59.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.634+0000 7f6612000640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f65ec06a430 0x7f65ec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:43:59.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.634+0000 7f6612000640 1 -- 172.21.15.39:0/3883644180 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f66040888b0 con 0x7f66140b16e0 2024-01-24T17:43:59.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.635+0000 7f6619600640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f65ec06a430 0x7f65ec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:43:59.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.636+0000 7f6619600640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f65ec06a430 0x7f65ec06c8f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f65fc006fd0 tx=0x7f65fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:43:59.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.639+0000 7f6612000640 1 -- 172.21.15.39:0/3883644180 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6604054250 con 0x7f66140b16e0 2024-01-24T17:43:59.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.833+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f66140d0bf0 con 0x7f66140b16e0 2024-01-24T17:43:59.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.840+0000 7f6612000640 1 -- 172.21.15.39:0/3883644180 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v12) v1 ==== 104+0+0 (secure 0 0 0) 0x7f660401a0a0 con 0x7f66140b16e0 2024-01-24T17:43:59.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.842+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f65ec06a430 msgr2=0x7f65ec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:59.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.842+0000 7f661b400640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f65ec06a430 0x7f65ec06c8f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f65fc006fd0 tx=0x7f65fc008040 comp rx=0 tx=0).stop 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.842+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140b16e0 msgr2=0x7f66140b7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.842+0000 7f661b400640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140b16e0 0x7f66140b7530 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6604007fa0 tx=0x7f660400d560 comp rx=0 tx=0).stop 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.843+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 shutdown_connections 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.843+0000 7f661b400640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f661415cad0 0x7f6614158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.843+0000 7f661b400640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66140d1350 0x7f66140c8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.843+0000 7f661b400640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f65ec06a430 0x7f65ec06c8f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.843+0000 7f661b400640 1 --2- 172.21.15.39:0/3883644180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66140b16e0 0x7f66140b7530 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.843+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 >> 172.21.15.39:0/3883644180 conn(0x7f66140d2830 msgr2=0x7f66140ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.843+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 shutdown_connections 2024-01-24T17:43:59.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:43:59.843+0000 7f661b400640 1 -- 172.21.15.39:0/3883644180 wait complete. 2024-01-24T17:43:59.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds ms_async_reap_threshold 2024-01-24T17:44:00.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.088+0000 7f9ddbe00640 1 Processor -- start 2024-01-24T17:44:00.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.088+0000 7f9ddbe00640 1 -- start start 2024-01-24T17:44:00.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.090+0000 7f9ddbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9ddc15aab0 0x7f9ddc0c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.090+0000 7f9ddbe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc0caaa0 con 0x7f9ddc15e480 2024-01-24T17:44:00.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.090+0000 7f9ddbe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc0c9750 con 0x7f9ddc15a6d0 2024-01-24T17:44:00.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.090+0000 7f9ddbe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ddc0ca800 con 0x7f9ddc15aab0 2024-01-24T17:44:00.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.090+0000 7f9dda000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9ddc15aab0 0x7f9ddc0c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.090+0000 7f9ddb400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9ddc15e480 0x7f9ddc06fe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50846/0 (socket says 172.21.15.39:50846) 2024-01-24T17:44:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.090+0000 7f9ddb400640 1 -- 172.21.15.39:0/3837970651 learned_addr learned my addr 172.21.15.39:0/3837970651 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.091+0000 7f9dda000640 1 -- 172.21.15.39:0/3837970651 >> v1:172.21.15.100:6789/0 conn(0x7f9ddc15a6d0 legacy=0x7f9ddc0c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.091+0000 7f9dda000640 1 -- 172.21.15.39:0/3837970651 >> v1:172.21.15.39:6789/0 conn(0x7f9ddc15e480 legacy=0x7f9ddc06fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.091+0000 7f9dda000640 1 -- 172.21.15.39:0/3837970651 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc0c9a30 con 0x7f9ddc15aab0 2024-01-24T17:44:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.091+0000 7f9dd9600640 1 -- 172.21.15.39:0/3837970651 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742367397 0 0) 0x7f9ddc0caaa0 con 0x7f9ddc15e480 2024-01-24T17:44:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.092+0000 7f9dda000640 1 --2- 172.21.15.39:0/3837970651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15aab0 0x7f9ddc0c1f80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40097a0 tx=0x7f9dc402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d22636504ced053 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.092+0000 7f9dd9600640 1 -- 172.21.15.39:0/3837970651 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4031830 con 0x7f9ddc15aab0 2024-01-24T17:44:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.092+0000 7f9dd9600640 1 -- 172.21.15.39:0/3837970651 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc4031990 con 0x7f9ddc15aab0 2024-01-24T17:44:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.092+0000 7f9dd9600640 1 -- 172.21.15.39:0/3837970651 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4031d20 con 0x7f9ddc15aab0 2024-01-24T17:44:00.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.093+0000 7f9ddbe00640 1 -- 172.21.15.39:0/3837970651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15aab0 msgr2=0x7f9ddc0c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.093+0000 7f9ddbe00640 1 --2- 172.21.15.39:0/3837970651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15aab0 0x7f9ddc0c1f80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40097a0 tx=0x7f9dc402b8d0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.093+0000 7f9ddbe00640 1 -- 172.21.15.39:0/3837970651 shutdown_connections 2024-01-24T17:44:00.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.093+0000 7f9ddbe00640 1 --2- 172.21.15.39:0/3837970651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15aab0 0x7f9ddc0c1f80 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.093+0000 7f9ddbe00640 1 -- 172.21.15.39:0/3837970651 >> 172.21.15.39:0/3837970651 conn(0x7f9ddc06a2a0 msgr2=0x7f9ddc0aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:00.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.093+0000 7f9ddbe00640 1 -- 172.21.15.39:0/3837970651 shutdown_connections 2024-01-24T17:44:00.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.093+0000 7f9ddbe00640 1 -- 172.21.15.39:0/3837970651 wait complete. 2024-01-24T17:44:00.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.095+0000 7f9ddbe00640 1 Processor -- start 2024-01-24T17:44:00.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.095+0000 7f9ddbe00640 1 -- start start 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ddc15a6d0 0x7f9ddc14e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9ddc15aab0 0x7f9ddc15af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15e480 0x7f9ddc0af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc15da30 con 0x7f9ddc15e480 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddbe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc15dd10 con 0x7f9ddc15a6d0 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddbe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc0ab8c0 con 0x7f9ddc15aab0 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15e480 0x7f9ddc0af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15e480 0x7f9ddc0af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35358/0 (socket says 172.21.15.39:35358) 2024-01-24T17:44:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.096+0000 7f9ddb400640 1 -- 172.21.15.39:0/26366547 learned_addr learned my addr 172.21.15.39:0/26366547 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9dda000640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9ddc15aab0 0x7f9ddc15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddaa00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ddc15a6d0 0x7f9ddc14e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddb400640 1 -- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9ddc15aab0 msgr2=0x7f9ddc15af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddb400640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9ddc15aab0 0x7f9ddc15af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddb400640 1 -- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ddc15a6d0 msgr2=0x7f9ddc14e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddb400640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ddc15a6d0 0x7f9ddc14e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddb400640 1 -- 172.21.15.39:0/26366547 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc4009400 con 0x7f9ddc15e480 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddaa00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ddc15a6d0 0x7f9ddc14e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddb400640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15e480 0x7f9ddc0af270 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400c450 tx=0x7f9dd400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9dd2a00640 1 -- 172.21.15.39:0/26366547 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40074e0 con 0x7f9ddc15e480 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9dd2a00640 1 -- 172.21.15.39:0/26366547 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd4007660 con 0x7f9ddc15e480 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9dd2a00640 1 -- 172.21.15.39:0/26366547 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4004a20 con 0x7f9ddc15e480 2024-01-24T17:44:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.097+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ddc0abba0 con 0x7f9ddc15e480 2024-01-24T17:44:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.098+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc14eb70 con 0x7f9ddc15e480 2024-01-24T17:44:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.099+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ddc15db70 con 0x7f9ddc15e480 2024-01-24T17:44:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.100+0000 7f9dd2a00640 1 -- 172.21.15.39:0/26366547 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9dd4004b80 con 0x7f9ddc15e480 2024-01-24T17:44:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.101+0000 7f9dd2a00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9db406a430 0x7f9db406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.101+0000 7f9dd2a00640 1 -- 172.21.15.39:0/26366547 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9dd4087880 con 0x7f9ddc15e480 2024-01-24T17:44:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.102+0000 7f9ddaa00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9db406a430 0x7f9db406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.103+0000 7f9ddaa00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9db406a430 0x7f9db406c8f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc0012e0 tx=0x7f9dcc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:00.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.106+0000 7f9dd2a00640 1 -- 172.21.15.39:0/26366547 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9ddc15db70 con 0x7f9ddc15e480 2024-01-24T17:44:00.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.308+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "ms_async_reap_threshold"} v 0) v1 -- 0x7f9ddc0ac0c0 con 0x7f9ddc15e480 2024-01-24T17:44:00.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.314+0000 7f9dd2a00640 1 -- 172.21.15.39:0/26366547 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "ms_async_reap_threshold"}]=0 v13) v1 ==== 106+0+0 (secure 0 0 0) 0x7f9ddc0ac0c0 con 0x7f9ddc15e480 2024-01-24T17:44:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.316+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9db406a430 msgr2=0x7f9db406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.316+0000 7f9ddbe00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9db406a430 0x7f9db406c8f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc0012e0 tx=0x7f9dcc009040 comp rx=0 tx=0).stop 2024-01-24T17:44:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15e480 msgr2=0x7f9ddc0af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15e480 0x7f9ddc0af270 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400c450 tx=0x7f9dd400c910 comp rx=0 tx=0).stop 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 shutdown_connections 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9db406a430 0x7f9db406c8f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ddc15e480 0x7f9ddc0af270 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9ddc15aab0 0x7f9ddc15af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 --2- 172.21.15.39:0/26366547 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9ddc15a6d0 0x7f9ddc14e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 >> 172.21.15.39:0/26366547 conn(0x7f9ddc06a2a0 msgr2=0x7f9ddc06bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 shutdown_connections 2024-01-24T17:44:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.317+0000 7f9ddbe00640 1 -- 172.21.15.39:0/26366547 wait complete. 2024-01-24T17:44:00.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close' 2024-01-24T17:44:00.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.545+0000 7f0448400640 1 Processor -- start 2024-01-24T17:44:00.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.545+0000 7f0448400640 1 -- start start 2024-01-24T17:44:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0448400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f044015c4e0 0x7f0440070970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0448400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0448400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f04400c9080 0x7f04400c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0448400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044015c8c0 con 0x7f04400c9080 2024-01-24T17:44:00.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0448400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044015da80 con 0x7f044015c4e0 2024-01-24T17:44:00.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0448400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044015cf00 con 0x7f04400c8c00 2024-01-24T17:44:00.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0446400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f04400c9080 0x7f04400c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0446400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f04400c9080 0x7f04400c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35366/0 (socket says 172.21.15.39:35366) 2024-01-24T17:44:00.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0446400640 1 -- 172.21.15.39:0/3668838046 learned_addr learned my addr 172.21.15.39:0/3668838046 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:00.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.547+0000 7f0445000640 1 --2- 172.21.15.39:0/3668838046 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.548+0000 7f0446400640 1 -- 172.21.15.39:0/3668838046 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 msgr2=0x7f04400c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:00.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.548+0000 7f0445a00640 1 --2- 172.21.15.39:0/3668838046 >> v2:172.21.15.100:3300/0 conn(0x7f044015c4e0 0x7f0440070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.548+0000 7f0446400640 1 --2- 172.21.15.39:0/3668838046 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.548+0000 7f0446400640 1 -- 172.21.15.39:0/3668838046 >> v2:172.21.15.100:3300/0 conn(0x7f044015c4e0 msgr2=0x7f0440070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:00.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.548+0000 7f0446400640 1 --2- 172.21.15.39:0/3668838046 >> v2:172.21.15.100:3300/0 conn(0x7f044015c4e0 0x7f0440070970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.548+0000 7f0446400640 1 -- 172.21.15.39:0/3668838046 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044015d1e0 con 0x7f04400c9080 2024-01-24T17:44:00.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.548+0000 7f0445000640 1 --2- 172.21.15.39:0/3668838046 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:00.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.548+0000 7f0445a00640 1 --2- 172.21.15.39:0/3668838046 >> v2:172.21.15.100:3300/0 conn(0x7f044015c4e0 0x7f0440070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:00.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.549+0000 7f0446400640 1 --2- 172.21.15.39:0/3668838046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f04400c9d80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f04400b10a0 tx=0x7f043802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57052df4ce0507ae server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:00.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.549+0000 7f043fe00640 1 -- 172.21.15.39:0/3668838046 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043802c9f0 con 0x7f04400c9080 2024-01-24T17:44:00.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.549+0000 7f043fe00640 1 -- 172.21.15.39:0/3668838046 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043802cb50 con 0x7f04400c9080 2024-01-24T17:44:00.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.549+0000 7f043fe00640 1 -- 172.21.15.39:0/3668838046 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04380343f0 con 0x7f04400c9080 2024-01-24T17:44:00.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 -- 172.21.15.39:0/3668838046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 msgr2=0x7f04400c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 --2- 172.21.15.39:0/3668838046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f04400c9d80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f04400b10a0 tx=0x7f043802be50 comp rx=0 tx=0).stop 2024-01-24T17:44:00.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 -- 172.21.15.39:0/3668838046 shutdown_connections 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 --2- 172.21.15.39:0/3668838046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f04400c9d80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 --2- 172.21.15.39:0/3668838046 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 --2- 172.21.15.39:0/3668838046 >> v2:172.21.15.100:3300/0 conn(0x7f044015c4e0 0x7f0440070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 -- 172.21.15.39:0/3668838046 >> 172.21.15.39:0/3668838046 conn(0x7f044006a3e0 msgr2=0x7f044006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 -- 172.21.15.39:0/3668838046 shutdown_connections 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.550+0000 7f0448400640 1 -- 172.21.15.39:0/3668838046 wait complete. 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.552+0000 7f0448400640 1 Processor -- start 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.552+0000 7f0448400640 1 -- start start 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0448400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0448400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f0440135f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0448400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f044015c4e0 0x7f0440146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0448400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04401364b0 con 0x7f04400c9080 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0448400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04401366e0 con 0x7f044015c4e0 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0448400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04401369c0 con 0x7f04400c8c00 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0445000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f0440135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0445000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f0440135f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35376/0 (socket says 172.21.15.39:35376) 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.553+0000 7f0445000640 1 -- 172.21.15.39:0/1207766651 learned_addr learned my addr 172.21.15.39:0/1207766651 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f0445a00640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f0446400640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f044015c4e0 0x7f0440146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f0445000640 1 -- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 msgr2=0x7f04400b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f0445000640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f0445000640 1 -- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f044015c4e0 msgr2=0x7f0440146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f0445000640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f044015c4e0 0x7f0440146720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f0445000640 1 -- 172.21.15.39:0/1207766651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438009b30 con 0x7f04400c9080 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f0445000640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f0440135f70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f042800c7f0 tx=0x7f042800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f043d600640 1 -- 172.21.15.39:0/1207766651 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428007820 con 0x7f04400c9080 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.554+0000 7f043d600640 1 -- 172.21.15.39:0/1207766651 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0428007980 con 0x7f04400c9080 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.555+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0440136ca0 con 0x7f04400c9080 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.555+0000 7f043d600640 1 -- 172.21.15.39:0/1207766651 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428007d30 con 0x7f04400c9080 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.555+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0440137260 con 0x7f04400c9080 2024-01-24T17:44:00.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.556+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0440073ef0 con 0x7f04400c9080 2024-01-24T17:44:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.557+0000 7f043d600640 1 -- 172.21.15.39:0/1207766651 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f04280113f0 con 0x7f04400c9080 2024-01-24T17:44:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.558+0000 7f043d600640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041806a460 0x7f041806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.558+0000 7f043d600640 1 -- 172.21.15.39:0/1207766651 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0428088740 con 0x7f04400c9080 2024-01-24T17:44:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.558+0000 7f0445a00640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041806a460 0x7f041806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.560+0000 7f0445a00640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041806a460 0x7f041806c920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f043000a990 tx=0x7f0430008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.562+0000 7f043d600640 1 -- 172.21.15.39:0/1207766651 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0428053fb0 con 0x7f04400c9080 2024-01-24T17:44:00.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:00.836+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close"]} v 0) v1 -- 0x7f04400cc9d0 con 0x7f04400c9080 2024-01-24T17:44:01.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.402+0000 7f043d600640 1 -- 172.21.15.39:0/1207766651 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close"]}]=0 v146) v1 ==== 144+0+0 (secure 0 0 0) 0x7f04280575d0 con 0x7f04400c9080 2024-01-24T17:44:01.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041806a460 msgr2=0x7f041806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:01.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041806a460 0x7f041806c920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f043000a990 tx=0x7f0430008040 comp rx=0 tx=0).stop 2024-01-24T17:44:01.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 msgr2=0x7f0440135f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f0440135f70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f042800c7f0 tx=0x7f042800ccb0 comp rx=0 tx=0).stop 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 shutdown_connections 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f044015c4e0 0x7f0440146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041806a460 0x7f041806c920 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04400c9080 0x7f0440135f70 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 --2- 172.21.15.39:0/1207766651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04400c8c00 0x7f04400b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.405+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 >> 172.21.15.39:0/1207766651 conn(0x7f044006a3e0 msgr2=0x7f0440159370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.406+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 shutdown_connections 2024-01-24T17:44:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.406+0000 7f0448400640 1 -- 172.21.15.39:0/1207766651 wait complete. 2024-01-24T17:44:01.430 INFO:tasks.cephfs_test_runner:That when a client unmounts, the thread count on the MDS goes back ... ok 2024-01-24T17:44:01.430 INFO:tasks.cephfs_test_runner:Starting test: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) Check that mds evicts blocklisted client 2024-01-24T17:44:01.430 INFO:tasks.cephfs_test_runner:test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2024-01-24T17:44:01.431 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted' 2024-01-24T17:44:01.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.656+0000 7fdc33000640 1 Processor -- start 2024-01-24T17:44:01.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.656+0000 7fdc33000640 1 -- start start 2024-01-24T17:44:01.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.657+0000 7fdc33000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c15db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:01.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.658+0000 7fdc33000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c0d0830 con 0x7fdc2c0d0da0 2024-01-24T17:44:01.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.658+0000 7fdc33000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c0d0a90 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.658+0000 7fdc33000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0cd760 con 0x7fdc2c0d03b0 2024-01-24T17:44:01.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.658+0000 7fdc32600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fdc2c0d0da0 0x7fdc2c15e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50848/0 (socket says 172.21.15.39:50848) 2024-01-24T17:44:01.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.658+0000 7fdc32600640 1 -- 172.21.15.39:0/1039995592 learned_addr learned my addr 172.21.15.39:0/1039995592 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:01.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.658+0000 7fdc31200640 1 --2- 172.21.15.39:0/1039995592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c15db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:01.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.659+0000 7fdc2be00640 1 -- 172.21.15.39:0/1039995592 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953979402 0 0) 0x7fdc2c0d0830 con 0x7fdc2c0d0da0 2024-01-24T17:44:01.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.659+0000 7fdc2be00640 1 -- 172.21.15.39:0/1039995592 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc0c003520 con 0x7fdc2c0d0da0 2024-01-24T17:44:01.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.659+0000 7fdc2be00640 1 -- 172.21.15.39:0/1039995592 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880756372 0 0) 0x7fdc2c0d0a90 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.659+0000 7fdc2be00640 1 -- 172.21.15.39:0/1039995592 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc2c0d0830 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.659+0000 7fdc31200640 1 -- 172.21.15.39:0/1039995592 >> v1:172.21.15.100:6789/0 conn(0x7fdc2c15cbf0 legacy=0x7fdc2c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:01.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.659+0000 7fdc31200640 1 -- 172.21.15.39:0/1039995592 >> v1:172.21.15.39:6789/0 conn(0x7fdc2c0d0da0 legacy=0x7fdc2c15e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:01.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.659+0000 7fdc31200640 1 -- 172.21.15.39:0/1039995592 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c0cda40 con 0x7fdc2c0d03b0 2024-01-24T17:44:01.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.660+0000 7fdc31200640 1 --2- 172.21.15.39:0/1039995592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c15db60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097d0 tx=0x7fdc1402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52d69f5ec5990a08 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:01.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.661+0000 7fdc2be00640 1 -- 172.21.15.39:0/1039995592 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14030800 con 0x7fdc2c0d03b0 2024-01-24T17:44:01.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.661+0000 7fdc2be00640 1 -- 172.21.15.39:0/1039995592 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc14030960 con 0x7fdc2c0d03b0 2024-01-24T17:44:01.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.661+0000 7fdc2be00640 1 -- 172.21.15.39:0/1039995592 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14030cf0 con 0x7fdc2c0d03b0 2024-01-24T17:44:01.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.662+0000 7fdc33000640 1 -- 172.21.15.39:0/1039995592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 msgr2=0x7fdc2c15db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:01.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.662+0000 7fdc33000640 1 --2- 172.21.15.39:0/1039995592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c15db60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097d0 tx=0x7fdc1402b940 comp rx=0 tx=0).stop 2024-01-24T17:44:01.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.662+0000 7fdc33000640 1 -- 172.21.15.39:0/1039995592 shutdown_connections 2024-01-24T17:44:01.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.662+0000 7fdc33000640 1 --2- 172.21.15.39:0/1039995592 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c15db60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:01.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.662+0000 7fdc33000640 1 -- 172.21.15.39:0/1039995592 >> 172.21.15.39:0/1039995592 conn(0x7fdc2c09bb90 msgr2=0x7fdc2c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:01.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.662+0000 7fdc33000640 1 -- 172.21.15.39:0/1039995592 shutdown_connections 2024-01-24T17:44:01.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.662+0000 7fdc33000640 1 -- 172.21.15.39:0/1039995592 wait complete. 2024-01-24T17:44:01.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.663+0000 7fdc33000640 1 Processor -- start 2024-01-24T17:44:01.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.664+0000 7fdc33000640 1 -- start start 2024-01-24T17:44:01.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc33000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c0c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:01.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc33000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc2c0d0da0 0x7fdc2c1561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:01.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc33000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc2c15cbf0 0x7fdc2c1486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:01.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc33000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c157790 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc33000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c156b10 con 0x7fdc2c0d0da0 2024-01-24T17:44:01.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc33000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c156df0 con 0x7fdc2c0d03b0 2024-01-24T17:44:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc32600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc2c15cbf0 0x7fdc2c1486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc32600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc2c15cbf0 0x7fdc2c1486d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35378/0 (socket says 172.21.15.39:35378) 2024-01-24T17:44:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc32600640 1 -- 172.21.15.39:0/3953393675 learned_addr learned my addr 172.21.15.39:0/3953393675 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc31200640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc2c0d0da0 0x7fdc2c1561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.665+0000 7fdc31c00640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c0c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:01.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc32600640 1 -- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 msgr2=0x7fdc2c0c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc32600640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c0c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc32600640 1 -- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc2c0d0da0 msgr2=0x7fdc2c1561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc32600640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc2c0d0da0 0x7fdc2c1561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc32600640 1 -- 172.21.15.39:0/3953393675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc14009400 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc31c00640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c0c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc31200640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc2c0d0da0 0x7fdc2c1561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc32600640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc2c15cbf0 0x7fdc2c1486d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400cba0 tx=0x7fdc2400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc29600640 1 -- 172.21.15.39:0/3953393675 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2400ace0 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc29600640 1 -- 172.21.15.39:0/3953393675 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2400ae40 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc29600640 1 -- 172.21.15.39:0/3953393675 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc24012620 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.666+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c1570d0 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.667+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c0c8720 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.668+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c0c8910 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.669+0000 7fdc29600640 1 -- 172.21.15.39:0/3953393675 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdc24007070 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.670+0000 7fdc29600640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc0406a430 0x7fdc0406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:01.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.670+0000 7fdc29600640 1 -- 172.21.15.39:0/3953393675 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdc24088770 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.671+0000 7fdc31c00640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc0406a430 0x7fdc0406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:01.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.672+0000 7fdc31c00640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc0406a430 0x7fdc0406c8f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c007fd0 tx=0x7fdc1c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:01.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.675+0000 7fdc29600640 1 -- 172.21.15.39:0/3953393675 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdc240579c0 con 0x7fdc2c15cbf0 2024-01-24T17:44:01.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:01.938+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted"]} v 0) v1 -- 0x7fdc2c1578d0 con 0x7fdc2c15cbf0 2024-01-24T17:44:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.412+0000 7fdc29600640 1 -- 172.21.15.39:0/3953393675 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted"]}]=0 v147) v1 ==== 156+0+0 (secure 0 0 0) 0x7fdc2c1578d0 con 0x7fdc2c15cbf0 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.415+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc0406a430 msgr2=0x7fdc0406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.415+0000 7fdc33000640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc0406a430 0x7fdc0406c8f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c007fd0 tx=0x7fdc1c009040 comp rx=0 tx=0).stop 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc2c15cbf0 msgr2=0x7fdc2c1486d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc2c15cbf0 0x7fdc2c1486d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400cba0 tx=0x7fdc2400a430 comp rx=0 tx=0).stop 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 shutdown_connections 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc0406a430 0x7fdc0406c8f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc2c15cbf0 0x7fdc2c1486d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc2c0d0da0 0x7fdc2c1561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 --2- 172.21.15.39:0/3953393675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdc2c0d03b0 0x7fdc2c0c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 >> 172.21.15.39:0/3953393675 conn(0x7fdc2c09bb90 msgr2=0x7fdc2c15db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 shutdown_connections 2024-01-24T17:44:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.416+0000 7fdc33000640 1 -- 172.21.15.39:0/3953393675 wait complete. 2024-01-24T17:44:02.437 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-24T17:44:02.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.657+0000 7f537fe00640 1 Processor -- start 2024-01-24T17:44:02.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.657+0000 7f537fe00640 1 -- start start 2024-01-24T17:44:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.659+0000 7f537fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f53780b0350 0x7f537806c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.659+0000 7f537fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f537806c9f0 0x7f537815da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:02.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.659+0000 7f537fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537806d4c0 con 0x7f53780b0730 2024-01-24T17:44:02.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.659+0000 7f537fe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f537806cf00 con 0x7f537806c9f0 2024-01-24T17:44:02.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.659+0000 7f537fe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f537806d1e0 con 0x7f53780b0350 2024-01-24T17:44:02.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.659+0000 7f537de00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f53780b0730 0x7f5378070010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50850/0 (socket says 172.21.15.39:50850) 2024-01-24T17:44:02.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.659+0000 7f537de00640 1 -- 172.21.15.39:0/2810934557 learned_addr learned my addr 172.21.15.39:0/2810934557 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.659+0000 7f537ca00640 1 --2- 172.21.15.39:0/2810934557 >> v2:172.21.15.39:3300/0 conn(0x7f537806c9f0 0x7f537815da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.660+0000 7f537d400640 1 --2- 172.21.15.39:0/2810934557 >> v2:172.21.15.100:3300/0 conn(0x7f53780b0350 0x7f537806c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.660+0000 7f5377e00640 1 -- 172.21.15.39:0/2810934557 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485863397 0 0) 0x7f537806d4c0 con 0x7f53780b0730 2024-01-24T17:44:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.660+0000 7f5377e00640 1 -- 172.21.15.39:0/2810934557 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5358003520 con 0x7f53780b0730 2024-01-24T17:44:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.660+0000 7f5377e00640 1 -- 172.21.15.39:0/2810934557 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 349281456 0 0) 0x7f5358003520 con 0x7f53780b0730 2024-01-24T17:44:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.660+0000 7f537ca00640 1 -- 172.21.15.39:0/2810934557 >> v2:172.21.15.100:3300/0 conn(0x7f53780b0350 msgr2=0x7f537806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.660+0000 7f537ca00640 1 --2- 172.21.15.39:0/2810934557 >> v2:172.21.15.100:3300/0 conn(0x7f53780b0350 0x7f537806c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.660+0000 7f537ca00640 1 -- 172.21.15.39:0/2810934557 >> v1:172.21.15.39:6789/0 conn(0x7f53780b0730 legacy=0x7f5378070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.660+0000 7f537ca00640 1 -- 172.21.15.39:0/2810934557 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780b15e0 con 0x7f537806c9f0 2024-01-24T17:44:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.662+0000 7f537ca00640 1 --2- 172.21.15.39:0/2810934557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537815da80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f537806e170 tx=0x7f536002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f65550c4e08b74 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.662+0000 7f5377e00640 1 -- 172.21.15.39:0/2810934557 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53700019f0 con 0x7f537806c9f0 2024-01-24T17:44:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.662+0000 7f5377e00640 1 -- 172.21.15.39:0/2810934557 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536002e050 con 0x7f537806c9f0 2024-01-24T17:44:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.662+0000 7f5377e00640 1 -- 172.21.15.39:0/2810934557 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360030710 con 0x7f537806c9f0 2024-01-24T17:44:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.663+0000 7f537fe00640 1 -- 172.21.15.39:0/2810934557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 msgr2=0x7f537815da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.663+0000 7f537fe00640 1 --2- 172.21.15.39:0/2810934557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537815da80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f537806e170 tx=0x7f536002b710 comp rx=0 tx=0).stop 2024-01-24T17:44:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.663+0000 7f537fe00640 1 -- 172.21.15.39:0/2810934557 shutdown_connections 2024-01-24T17:44:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.663+0000 7f537fe00640 1 --2- 172.21.15.39:0/2810934557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537815da80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.663+0000 7f537fe00640 1 --2- 172.21.15.39:0/2810934557 >> v2:172.21.15.100:3300/0 conn(0x7f53780b0350 0x7f537806c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.663+0000 7f537fe00640 1 -- 172.21.15.39:0/2810934557 >> 172.21.15.39:0/2810934557 conn(0x7f53780cb0e0 msgr2=0x7f53780cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.663+0000 7f537fe00640 1 -- 172.21.15.39:0/2810934557 shutdown_connections 2024-01-24T17:44:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.663+0000 7f537fe00640 1 -- 172.21.15.39:0/2810934557 wait complete. 2024-01-24T17:44:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.665+0000 7f537fe00640 1 Processor -- start 2024-01-24T17:44:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.665+0000 7f537fe00640 1 -- start start 2024-01-24T17:44:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53780b0350 0x7f53781f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537fe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53780b0730 0x7f53780c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53780cc230 con 0x7f537806c9f0 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537fe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53781e4360 con 0x7f53780b0730 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53781f2eb0 con 0x7f53780b0350 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537814bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35404/0 (socket says 172.21.15.39:35404) 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537d400640 1 -- 172.21.15.39:0/739425925 learned_addr learned my addr 172.21.15.39:0/739425925 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537d400640 1 -- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53780b0350 msgr2=0x7f53781f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.666+0000 7f537d400640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53780b0350 0x7f53781f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f537d400640 1 -- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53780b0730 msgr2=0x7f53780c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f537d400640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53780b0730 0x7f53780c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f537d400640 1 -- 172.21.15.39:0/739425925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5360009400 con 0x7f537806c9f0 2024-01-24T17:44:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f537d400640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537814bc60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f536800e9b0 tx=0x7f536800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f5375600640 1 -- 172.21.15.39:0/739425925 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5368009670 con 0x7f537806c9f0 2024-01-24T17:44:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f5375600640 1 -- 172.21.15.39:0/739425925 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536800c9a0 con 0x7f537806c9f0 2024-01-24T17:44:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f5375600640 1 -- 172.21.15.39:0/739425925 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536800cd50 con 0x7f537806c9f0 2024-01-24T17:44:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53781f28f0 con 0x7f537806c9f0 2024-01-24T17:44:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.667+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53781e4590 con 0x7f537806c9f0 2024-01-24T17:44:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.669+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5378073ef0 con 0x7f537806c9f0 2024-01-24T17:44:02.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.669+0000 7f5375600640 1 -- 172.21.15.39:0/739425925 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5368013470 con 0x7f537806c9f0 2024-01-24T17:44:02.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.670+0000 7f5375600640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f535006a430 0x7f535006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:02.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.670+0000 7f5375600640 1 -- 172.21.15.39:0/739425925 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5368088980 con 0x7f537806c9f0 2024-01-24T17:44:02.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.671+0000 7f537ca00640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f535006a430 0x7f535006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:02.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.672+0000 7f537ca00640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f535006a430 0x7f535006c8f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5360034000 tx=0x7f536002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:02.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.675+0000 7f5375600640 1 -- 172.21.15.39:0/739425925 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f53680541d0 con 0x7f537806c9f0 2024-01-24T17:44:02.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.873+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f53780c7040 con 0x7f537806c9f0 2024-01-24T17:44:02.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.882+0000 7f5375600640 1 -- 172.21.15.39:0/739425925 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f53680170c0 con 0x7f537806c9f0 2024-01-24T17:44:02.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.884+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f535006a430 msgr2=0x7f535006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.884+0000 7f537fe00640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f535006a430 0x7f535006c8f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5360034000 tx=0x7f536002f000 comp rx=0 tx=0).stop 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.884+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 msgr2=0x7f537814bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.884+0000 7f537fe00640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537814bc60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f536800e9b0 tx=0x7f536800ee70 comp rx=0 tx=0).stop 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.885+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 shutdown_connections 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.885+0000 7f537fe00640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53780b0730 0x7f53780c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.885+0000 7f537fe00640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53780b0350 0x7f53781f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.885+0000 7f537fe00640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f535006a430 0x7f535006c8f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.885+0000 7f537fe00640 1 --2- 172.21.15.39:0/739425925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f537806c9f0 0x7f537814bc60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.885+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 >> 172.21.15.39:0/739425925 conn(0x7f53780cb0e0 msgr2=0x7f53780ae0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.885+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 shutdown_connections 2024-01-24T17:44:02.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:02.885+0000 7f537fe00640 1 -- 172.21.15.39:0/739425925 wait complete. 2024-01-24T17:44:02.906 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:44:02.906 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:44:03.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:44:03.144 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:44:03.144 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:44:03.144 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:03.204 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:44:03.204 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:44:03.204 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:03.261 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:44:03.262 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:44:03.262 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:03.318 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:44:03.319 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:44:03.319 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:03.375 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:44:03.375 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-24T17:44:03.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:44:03.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.603+0000 7fb97cc00640 1 Processor -- start 2024-01-24T17:44:03.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.604+0000 7fb97cc00640 1 -- start start 2024-01-24T17:44:03.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.605+0000 7fb97cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780d03b0 0x7fb97815db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:03.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.605+0000 7fb97cc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9780d0830 con 0x7fb9780d0da0 2024-01-24T17:44:03.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.605+0000 7fb97cc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9780d0a90 con 0x7fb97815cbf0 2024-01-24T17:44:03.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.605+0000 7fb97cc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9780cd760 con 0x7fb9780d03b0 2024-01-24T17:44:03.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.605+0000 7fb977e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb9780d0da0 0x7fb97815e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48150/0 (socket says 172.21.15.39:48150) 2024-01-24T17:44:03.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb977e00640 1 -- 172.21.15.39:0/2161661911 learned_addr learned my addr 172.21.15.39:0/2161661911 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:03.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507444645 0 0) 0x7fb9780d0830 con 0x7fb9780d0da0 2024-01-24T17:44:03.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb958003520 con 0x7fb9780d0da0 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976a00640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780d03b0 0x7fb97815db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 707686243 0 0) 0x7fb958003520 con 0x7fb9780d0da0 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780d03b0 msgr2=0x7fb97815db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976000640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780d03b0 0x7fb97815db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 >> v1:172.21.15.100:6789/0 conn(0x7fb97815cbf0 legacy=0x7fb9780ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9780cda40 con 0x7fb9780d0da0 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.606+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fb970001670 con 0x7fb9780d0da0 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 >> v1:172.21.15.39:6789/0 conn(0x7fb9780d0da0 legacy=0x7fb97815e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb976000640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9580039a0 0x7fb958003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb976000640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780d03b0 0x7fb9580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb976000640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb958005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9780cd760 con 0x7fb958004d00 2024-01-24T17:44:03.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95800bb10 con 0x7fb9580039a0 2024-01-24T17:44:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95800bcd0 con 0x7fb9780d03b0 2024-01-24T17:44:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb977e00640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb958005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb977400640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9580039a0 0x7fb958003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.607+0000 7fb976a00640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780d03b0 0x7fb9580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.608+0000 7fb977e00640 1 -- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780d03b0 msgr2=0x7fb9580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.608+0000 7fb977e00640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780d03b0 0x7fb9580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.608+0000 7fb977e00640 1 -- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9580039a0 msgr2=0x7fb958003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.608+0000 7fb977e00640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9580039a0 0x7fb958003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.608+0000 7fb977e00640 1 -- 172.21.15.39:0/2161661911 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958003520 con 0x7fb958004d00 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.609+0000 7fb977e00640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb958005180 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb97000c240 tx=0x7fb9580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63fedcbc52d9da3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.609+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970005ad0 con 0x7fb958004d00 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.609+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb970005c30 con 0x7fb958004d00 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.609+0000 7fb976000640 1 -- 172.21.15.39:0/2161661911 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97002c8b0 con 0x7fb958004d00 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.610+0000 7fb97cc00640 1 -- 172.21.15.39:0/2161661911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 msgr2=0x7fb958005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.610+0000 7fb97cc00640 1 --2- 172.21.15.39:0/2161661911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb958005180 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb97000c240 tx=0x7fb9580057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.610+0000 7fb977e00640 1 -- 172.21.15.39:0/2161661911 reap_dead start 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.610+0000 7fb97cc00640 1 -- 172.21.15.39:0/2161661911 shutdown_connections 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.610+0000 7fb97cc00640 1 -- 172.21.15.39:0/2161661911 >> 172.21.15.39:0/2161661911 conn(0x7fb97809bb90 msgr2=0x7fb9780d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.610+0000 7fb97cc00640 1 -- 172.21.15.39:0/2161661911 shutdown_connections 2024-01-24T17:44:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.610+0000 7fb97cc00640 1 -- 172.21.15.39:0/2161661911 wait complete. 2024-01-24T17:44:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.612+0000 7fb97cc00640 1 Processor -- start 2024-01-24T17:44:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.612+0000 7fb97cc00640 1 -- start start 2024-01-24T17:44:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb97cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb9780c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb97cc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb978148840 0x7fb978148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb97cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780b53f0 0x7fb9780b5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb97cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9780b7200 con 0x7fb958004d00 2024-01-24T17:44:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb97cc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb978149200 con 0x7fb978148840 2024-01-24T17:44:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb97cc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9780b6160 con 0x7fb9780b53f0 2024-01-24T17:44:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb977400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb9780c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb977400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb9780c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38978/0 (socket says 172.21.15.39:38978) 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.613+0000 7fb977400640 1 -- 172.21.15.39:0/390483440 learned_addr learned my addr 172.21.15.39:0/390483440 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb977e00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780b53f0 0x7fb9780b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb976a00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb978148840 0x7fb978148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb977400640 1 -- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780b53f0 msgr2=0x7fb9780b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb977400640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780b53f0 0x7fb9780b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb977400640 1 -- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb978148840 msgr2=0x7fb978148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb977400640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb978148840 0x7fb978148cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb977400640 1 -- 172.21.15.39:0/390483440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9780b6440 con 0x7fb958004d00 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb976a00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb978148840 0x7fb978148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb977400640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb9780c8a60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb96800fb90 tx=0x7fb96800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb96f400640 1 -- 172.21.15.39:0/390483440 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96800fe90 con 0x7fb958004d00 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb96f400640 1 -- 172.21.15.39:0/390483440 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96800dc30 con 0x7fb958004d00 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb96f400640 1 -- 172.21.15.39:0/390483440 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9680039c0 con 0x7fb958004d00 2024-01-24T17:44:03.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.614+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9780b6720 con 0x7fb958004d00 2024-01-24T17:44:03.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.615+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9780b6ce0 con 0x7fb958004d00 2024-01-24T17:44:03.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.616+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9780b8650 con 0x7fb958004d00 2024-01-24T17:44:03.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.617+0000 7fb96f400640 1 -- 172.21.15.39:0/390483440 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb968003b20 con 0x7fb958004d00 2024-01-24T17:44:03.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.618+0000 7fb96f400640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb95006a460 0x7fb95006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:03.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.619+0000 7fb96f400640 1 -- 172.21.15.39:0/390483440 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb968088940 con 0x7fb958004d00 2024-01-24T17:44:03.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.619+0000 7fb976a00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb95006a460 0x7fb95006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:03.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.620+0000 7fb976a00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb95006a460 0x7fb95006c920 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb9780cc6c0 tx=0x7fb960008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:03.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.623+0000 7fb96f400640 1 -- 172.21.15.39:0/390483440 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb968054230 con 0x7fb958004d00 2024-01-24T17:44:03.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.861+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9780b6e70 con 0x7fb958004d00 2024-01-24T17:44:03.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.862+0000 7fb96f400640 1 -- 172.21.15.39:0/390483440 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3529 (secure 0 0 0) 0x7fb968057850 con 0x7fb958004d00 2024-01-24T17:44:03.863 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:03.863 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4688,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/648542221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":648542221},{"type":"v1","addr":"172.21.15.100:6837","nonce":648542221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4924,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/188830546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":188830546},{"type":"v1","addr":"172.21.15.100:6835","nonce":188830546}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":4939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/4294097622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":4294097622},{"type":"v1","addr":"172.21.15.39:6834","nonce":4294097622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5688,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3065004391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3065004391},{"type":"v1","addr":"172.21.15.39:6835","nonce":3065004391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[]} 2024-01-24T17:44:03.863 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 17 2024-01-24T17:44:03.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb95006a460 msgr2=0x7fb95006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb95006a460 0x7fb95006c920 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb9780cc6c0 tx=0x7fb960008040 comp rx=0 tx=0).stop 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 msgr2=0x7fb9780c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb9780c8a60 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb96800fb90 tx=0x7fb96800d520 comp rx=0 tx=0).stop 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 shutdown_connections 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9780b53f0 0x7fb9780b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb978148840 0x7fb978148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb95006a460 0x7fb95006c920 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 --2- 172.21.15.39:0/390483440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb958004d00 0x7fb9780c8a60 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.865+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 >> 172.21.15.39:0/390483440 conn(0x7fb97809bb90 msgr2=0x7fb97815edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.866+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 shutdown_connections 2024-01-24T17:44:03.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:03.866+0000 7fb97cc00640 1 -- 172.21.15.39:0/390483440 wait complete. 2024-01-24T17:44:03.886 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-24T17:44:03.887 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-24T17:44:03.887 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-24T17:44:03.887 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:44:03.887 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-24T17:44:03.887 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-24T17:44:03.887 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-24T17:44:03.887 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:44:03.888 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-24T17:44:03.888 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-24T17:44:03.888 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-24T17:44:03.888 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:44:03.888 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-24T17:44:03.888 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-24T17:44:03.888 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-24T17:44:03.889 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:44:03.908 INFO:tasks.ceph.mds.a:Stopped 2024-01-24T17:44:03.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-24T17:44:03.909 INFO:tasks.ceph.mds.c:Stopped 2024-01-24T17:44:03.909 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-24T17:44:03.910 INFO:tasks.ceph.mds.b:Stopped 2024-01-24T17:44:03.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-24T17:44:03.911 INFO:tasks.ceph.mds.d:Stopped 2024-01-24T17:44:03.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-24T17:44:03.961 INFO:tasks.ceph.mds.b:Started 2024-01-24T17:44:03.961 DEBUG:teuthology.parallel:result is None 2024-01-24T17:44:03.962 INFO:tasks.ceph.mds.a:Started 2024-01-24T17:44:03.962 DEBUG:teuthology.parallel:result is None 2024-01-24T17:44:03.967 INFO:tasks.ceph.mds.d:Started 2024-01-24T17:44:03.967 DEBUG:teuthology.parallel:result is None 2024-01-24T17:44:03.968 INFO:tasks.ceph.mds.c:Started 2024-01-24T17:44:03.968 DEBUG:teuthology.parallel:result is None 2024-01-24T17:44:03.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-24T17:44:04.118 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:44:04.119 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:44:04.119 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:44:04.120 INFO:tasks.ceph.mds.b.smithi100.stderr:2024-01-24T17:44:04.118+0000 7f523e600640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-01-24T17:44:04.120 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:44:04.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.159+0000 7f6fb8400640 1 Processor -- start 2024-01-24T17:44:04.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.159+0000 7f6fb8400640 1 -- start start 2024-01-24T17:44:04.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb8400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6fb00cf990 0x7f6fb00cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:04.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb8400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6fb015fb90 0x7f6fb015d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:04.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb8400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb00d02b0 con 0x7f6fb015d020 2024-01-24T17:44:04.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb8400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb015e080 con 0x7f6fb015fb90 2024-01-24T17:44:04.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb8400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb015dac0 con 0x7f6fb00cf990 2024-01-24T17:44:04.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb6400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6fb015fb90 0x7f6fb015d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:04.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb6400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6fb015fb90 0x7f6fb015d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39008/0 (socket says 172.21.15.39:39008) 2024-01-24T17:44:04.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb6400640 1 -- 172.21.15.39:0/1967581382 learned_addr learned my addr 172.21.15.39:0/1967581382 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:04.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.160+0000 7f6fb5a00640 1 --2- 172.21.15.39:0/1967581382 >> v2:172.21.15.100:3300/0 conn(0x7f6fb00cf990 0x7f6fb00cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:04.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.161+0000 7f6fb6400640 1 -- 172.21.15.39:0/1967581382 >> v2:172.21.15.100:3300/0 conn(0x7f6fb00cf990 msgr2=0x7f6fb00cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:04.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.161+0000 7f6fb6400640 1 --2- 172.21.15.39:0/1967581382 >> v2:172.21.15.100:3300/0 conn(0x7f6fb00cf990 0x7f6fb00cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:04.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.161+0000 7f6fb6400640 1 -- 172.21.15.39:0/1967581382 >> v1:172.21.15.100:6789/0 conn(0x7f6fb015d020 legacy=0x7f6fb0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:04.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.161+0000 7f6fb6400640 1 -- 172.21.15.39:0/1967581382 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb015dda0 con 0x7f6fb015fb90 2024-01-24T17:44:04.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.161+0000 7f6fb5a00640 1 --2- 172.21.15.39:0/1967581382 >> v2:172.21.15.100:3300/0 conn(0x7f6fb00cf990 0x7f6fb00cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.162+0000 7f6fb6400640 1 --2- 172.21.15.39:0/1967581382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb015d490 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00b4c00 tx=0x7f6fa802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574b77bfda193da4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.162+0000 7f6fafe00640 1 -- 172.21.15.39:0/1967581382 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa802cb20 con 0x7f6fb015fb90 2024-01-24T17:44:04.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.162+0000 7f6fafe00640 1 -- 172.21.15.39:0/1967581382 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802cc80 con 0x7f6fb015fb90 2024-01-24T17:44:04.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.162+0000 7f6fafe00640 1 -- 172.21.15.39:0/1967581382 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa80343f0 con 0x7f6fb015fb90 2024-01-24T17:44:04.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.163+0000 7f6fb8400640 1 -- 172.21.15.39:0/1967581382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 msgr2=0x7f6fb015d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:04.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.163+0000 7f6fb8400640 1 --2- 172.21.15.39:0/1967581382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb015d490 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00b4c00 tx=0x7f6fa802bf80 comp rx=0 tx=0).stop 2024-01-24T17:44:04.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.163+0000 7f6fb8400640 1 -- 172.21.15.39:0/1967581382 shutdown_connections 2024-01-24T17:44:04.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.163+0000 7f6fb8400640 1 --2- 172.21.15.39:0/1967581382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb015d490 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:04.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.163+0000 7f6fb8400640 1 --2- 172.21.15.39:0/1967581382 >> v2:172.21.15.100:3300/0 conn(0x7f6fb00cf990 0x7f6fb00cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:04.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.163+0000 7f6fb8400640 1 -- 172.21.15.39:0/1967581382 >> 172.21.15.39:0/1967581382 conn(0x7f6fb0071790 msgr2=0x7f6fb0071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:04.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.163+0000 7f6fb8400640 1 -- 172.21.15.39:0/1967581382 shutdown_connections 2024-01-24T17:44:04.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.163+0000 7f6fb8400640 1 -- 172.21.15.39:0/1967581382 wait complete. 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.165+0000 7f6fb8400640 1 Processor -- start 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.165+0000 7f6fb8400640 1 -- start start 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb8400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fb00cf990 0x7f6fb0159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb8400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fb015d020 0x7f6fb00c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb8400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb00c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb8400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00c92f0 con 0x7f6fb015fb90 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb8400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00d18a0 con 0x7f6fb015d020 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb8400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb00d1b80 con 0x7f6fb00cf990 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb00c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb00c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39018/0 (socket says 172.21.15.39:39018) 2024-01-24T17:44:04.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb6400640 1 -- 172.21.15.39:0/3821232852 learned_addr learned my addr 172.21.15.39:0/3821232852 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb5000640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fb015d020 0x7f6fb00c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.166+0000 7f6fb5a00640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fb00cf990 0x7f6fb0159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fb6400640 1 -- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fb00cf990 msgr2=0x7f6fb0159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fb6400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fb00cf990 0x7f6fb0159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fb6400640 1 -- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fb015d020 msgr2=0x7f6fb00c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fb6400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fb015d020 0x7f6fb00c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fb6400640 1 -- 172.21.15.39:0/3821232852 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa8009b30 con 0x7f6fb015fb90 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fb5a00640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fb00cf990 0x7f6fb0159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fb6400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb00c5270 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8036040 tx=0x7f6fa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fad600640 1 -- 172.21.15.39:0/3821232852 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8007e70 con 0x7f6fb015fb90 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.167+0000 7f6fad600640 1 -- 172.21.15.39:0/3821232852 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa8009510 con 0x7f6fb015fb90 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.168+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb00d1e60 con 0x7f6fb015fb90 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.168+0000 7f6fad600640 1 -- 172.21.15.39:0/3821232852 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa803e690 con 0x7f6fb015fb90 2024-01-24T17:44:04.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.168+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb01fef00 con 0x7f6fb015fb90 2024-01-24T17:44:04.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.169+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb0073ef0 con 0x7f6fb015fb90 2024-01-24T17:44:04.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.170+0000 7f6fad600640 1 -- 172.21.15.39:0/3821232852 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6fa803e7f0 con 0x7f6fb015fb90 2024-01-24T17:44:04.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.171+0000 7f6fad600640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f8806a430 0x7f6f8806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:04.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.171+0000 7f6fad600640 1 -- 172.21.15.39:0/3821232852 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6fa80ab120 con 0x7f6fb015fb90 2024-01-24T17:44:04.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.171+0000 7f6fb5a00640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f8806a430 0x7f6f8806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:04.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.172+0000 7f6fb5a00640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f8806a430 0x7f6f8806c8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00b4020 tx=0x7f6fa0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:04.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.175+0000 7f6fad600640 1 -- 172.21.15.39:0/3821232852 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6fa8076a00 con 0x7f6fb015fb90 2024-01-24T17:44:04.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:04.385+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f6fb00b8440 con 0x7f6fb015fb90 2024-01-24T17:44:05.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.236+0000 7f6fad600640 1 -- 172.21.15.39:0/3821232852 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v44) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6fa807a020 con 0x7f6fb015fb90 2024-01-24T17:44:05.237 INFO:teuthology.orchestra.run.smithi039.stderr: removed all blocklist entries 2024-01-24T17:44:05.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.240+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f8806a430 msgr2=0x7f6f8806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.240+0000 7f6fb8400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f8806a430 0x7f6f8806c8f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6fb00b4020 tx=0x7f6fa0008040 comp rx=0 tx=0).stop 2024-01-24T17:44:05.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.240+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 msgr2=0x7f6fb00c5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.240+0000 7f6fb8400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb00c5270 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8036040 tx=0x7f6fa8004a40 comp rx=0 tx=0).stop 2024-01-24T17:44:05.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.240+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 shutdown_connections 2024-01-24T17:44:05.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.241+0000 7f6fb8400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6f8806a430 0x7f6f8806c8f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.241+0000 7f6fb8400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb015fb90 0x7f6fb00c5270 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.241+0000 7f6fb8400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fb015d020 0x7f6fb00c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.241+0000 7f6fb8400640 1 --2- 172.21.15.39:0/3821232852 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fb00cf990 0x7f6fb0159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.241+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 >> 172.21.15.39:0/3821232852 conn(0x7f6fb0071790 msgr2=0x7f6fb00b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:05.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.241+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 shutdown_connections 2024-01-24T17:44:05.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.241+0000 7f6fb8400640 1 -- 172.21.15.39:0/3821232852 wait complete. 2024-01-24T17:44:05.261 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-24T17:44:05.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.476+0000 7fad9f000640 1 Processor -- start 2024-01-24T17:44:05.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.477+0000 7fad9f000640 1 -- start start 2024-01-24T17:44:05.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.478+0000 7fad9f000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9815a950 0x7fad980ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:05.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.478+0000 7fad9f000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad980c9180 con 0x7fad980cb660 2024-01-24T17:44:05.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.478+0000 7fad9f000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad980c9460 con 0x7fad980caaa0 2024-01-24T17:44:05.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.478+0000 7fad9f000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad980b43d0 con 0x7fad9815a950 2024-01-24T17:44:05.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.479+0000 7fad9d000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fad980cb660 0x7fad980b1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48162/0 (socket says 172.21.15.39:48162) 2024-01-24T17:44:05.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.479+0000 7fad9d000640 1 -- 172.21.15.39:0/4104454821 learned_addr learned my addr 172.21.15.39:0/4104454821 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:05.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.479+0000 7fad97e00640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9815a950 0x7fad980ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:05.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.479+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142715607 0 0) 0x7fad980c9180 con 0x7fad980cb660 2024-01-24T17:44:05.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.479+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad78003520 con 0x7fad980cb660 2024-01-24T17:44:05.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.479+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 846815673 0 0) 0x7fad78003520 con 0x7fad980cb660 2024-01-24T17:44:05.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.479+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9815a950 msgr2=0x7fad980ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:05.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.479+0000 7fad97400640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9815a950 0x7fad980ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 >> v1:172.21.15.100:6789/0 conn(0x7fad980caaa0 legacy=0x7fad980caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980c9740 con 0x7fad980cb660 2024-01-24T17:44:05.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fad900019f0 con 0x7fad980cb660 2024-01-24T17:44:05.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 >> v1:172.21.15.39:6789/0 conn(0x7fad980cb660 legacy=0x7fad980b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9815a950 0x7fad780039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:05.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:05.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad78004d00 0x7fad78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:05.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980b43d0 con 0x7fad78003f20 2024-01-24T17:44:05.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7800ba60 con 0x7fad78004d00 2024-01-24T17:44:05.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7800bc20 con 0x7fad9815a950 2024-01-24T17:44:05.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad8fe00640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:05.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad97e00640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9815a950 0x7fad780039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:05.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.480+0000 7fad9d000640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad78004d00 0x7fad78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:05.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.481+0000 7fad8fe00640 1 -- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9815a950 msgr2=0x7fad780039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:05.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.481+0000 7fad8fe00640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9815a950 0x7fad780039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.481+0000 7fad8fe00640 1 -- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad78004d00 msgr2=0x7fad78005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:05.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.481+0000 7fad8fe00640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad78004d00 0x7fad78005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.481+0000 7fad8fe00640 1 -- 172.21.15.39:0/4104454821 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78003520 con 0x7fad78003f20 2024-01-24T17:44:05.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.482+0000 7fad8fe00640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad780047c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fad80001720 tx=0x7fad780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ee655a1cc70853 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:05.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.482+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad800052d0 con 0x7fad78003f20 2024-01-24T17:44:05.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.482+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8002e050 con 0x7fad78003f20 2024-01-24T17:44:05.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.482+0000 7fad97400640 1 -- 172.21.15.39:0/4104454821 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad80003830 con 0x7fad78003f20 2024-01-24T17:44:05.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.483+0000 7fad9f000640 1 -- 172.21.15.39:0/4104454821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 msgr2=0x7fad780047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.483+0000 7fad9f000640 1 --2- 172.21.15.39:0/4104454821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad780047c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fad80001720 tx=0x7fad780057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.483+0000 7fad9d000640 1 -- 172.21.15.39:0/4104454821 reap_dead start 2024-01-24T17:44:05.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.483+0000 7fad9f000640 1 -- 172.21.15.39:0/4104454821 shutdown_connections 2024-01-24T17:44:05.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.483+0000 7fad9f000640 1 -- 172.21.15.39:0/4104454821 >> 172.21.15.39:0/4104454821 conn(0x7fad9806a3e0 msgr2=0x7fad9806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:05.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.483+0000 7fad9f000640 1 -- 172.21.15.39:0/4104454821 shutdown_connections 2024-01-24T17:44:05.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.483+0000 7fad9f000640 1 -- 172.21.15.39:0/4104454821 wait complete. 2024-01-24T17:44:05.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.485+0000 7fad9f000640 1 Processor -- start 2024-01-24T17:44:05.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.485+0000 7fad9f000640 1 -- start start 2024-01-24T17:44:05.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad9f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad981495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:05.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad9f000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad98155dd0 0x7fad98149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad9f000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9815b000 0x7fad9815b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad9f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980c7da0 con 0x7fad78003f20 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad9f000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980c8080 con 0x7fad9815b000 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad9f000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad980c8360 con 0x7fad98155dd0 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad97e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad981495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad97e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad981495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39040/0 (socket says 172.21.15.39:39040) 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.486+0000 7fad97e00640 1 -- 172.21.15.39:0/3031860053 learned_addr learned my addr 172.21.15.39:0/3031860053 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad8fe00640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad98155dd0 0x7fad98149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad9d000640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9815b000 0x7fad9815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad97e00640 1 -- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad98155dd0 msgr2=0x7fad98149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad97e00640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad98155dd0 0x7fad98149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad97e00640 1 -- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9815b000 msgr2=0x7fad9815b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad97e00640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9815b000 0x7fad9815b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad97e00640 1 -- 172.21.15.39:0/3031860053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980c5610 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad97e00640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad981495f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fad880119a0 tx=0x7fad88011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad94c00640 1 -- 172.21.15.39:0/3031860053 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8800faf0 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad94c00640 1 -- 172.21.15.39:0/3031860053 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8800fc50 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad980c58f0 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.487+0000 7fad94c00640 1 -- 172.21.15.39:0/3031860053 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad88014440 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.488+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9815bbb0 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.489+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad98073ef0 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.490+0000 7fad94c00640 1 -- 172.21.15.39:0/3031860053 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fad880145a0 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.491+0000 7fad94c00640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7006a480 0x7fad7006c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.491+0000 7fad94c00640 1 -- 172.21.15.39:0/3031860053 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad88088d40 con 0x7fad78003f20 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.492+0000 7fad8fe00640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7006a480 0x7fad7006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:05.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.493+0000 7fad8fe00640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7006a480 0x7fad7006c940 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fad80033000 tx=0x7fad80009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:05.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.496+0000 7fad94c00640 1 -- 172.21.15.39:0/3031860053 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fad880547f0 con 0x7fad78003f20 2024-01-24T17:44:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.757+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fad9815dc50 con 0x7fad78003f20 2024-01-24T17:44:05.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.758+0000 7fad94c00640 1 -- 172.21.15.39:0/3031860053 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fad88057e10 con 0x7fad78003f20 2024-01-24T17:44:05.759 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:05.759 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:44:05.759 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_dump": [ 2024-01-24T17:44:05.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.759 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.a", 2024-01-24T17:44:05.759 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlZZEtLBAAUxf4chL3tgy3zHlZoraXlg==", 2024-01-24T17:44:05.759 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.b", 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Fln4utEhAA52ngJFqrdfa9bx56mVhRVA==", 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:44:05.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.c", 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flo8iGARAAf+29RXWr4RHkILIz7em3mA==", 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.761 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.d", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flh+08IRAARhZxy//j8c9b3Hg9uV9ajg==", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.0", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCQS7FlOj/pLBAAzi89tAacOfDKdlU0BUJvvA==", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:44:05.762 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.1", 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCUS7FlbRAKDRAAAuAy47MDTtvmh8qNA3ZLTg==", 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.2", 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCXS7Fl7QmRGRAAVECH7oYTdvDPSHBy9WmHwA==", 2024-01-24T17:44:05.763 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.3", 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCaS7FlCQHIMxAA9OchsMzsKFxs110SeJ3Atw==", 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.4", 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCiS7FlDsBKBRAAXF2HafraT5nRbB5tBRUgWQ==", 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.5", 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQClS7FlfrXBHhAAy8GP3TLX/iok3OYPGTP5Mw==", 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:44:05.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.6", 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCoS7Fla11hNhAA/3oMP7cFtPSTdJl7eSnPTA==", 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.766 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.7", 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCsS7FlKFMtEhAAPmaFzr5XHVQaKZYkJiIarA==", 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.0", 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlTi/3LhAAc8FwXtLH2XjlFMO7wDMcSQ==", 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow r", 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.1", 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlH5s7ChAAlFfFDziTgfv4ibWO08N+4Q==", 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.2", 2024-01-24T17:44:05.768 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlIISlOhAAoVxxez7UjXDxyHQNNPnn2g==", 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.3", 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlO/U/FBAA2pI9aGSS2cZ3AMnn+W1fQA==", 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.admin", 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCGS7FlxKo7HhAAlu037DIc1WmWkz81BG3j7Q==", 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:44:05.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.x", 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlRbF6HxAAoFjlyWb5oqIkC3L1iPRmHw==", 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:05.771 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.y", 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7Fl512kFRAAdCpU+QjRmGJmMD5uTdFmHQ==", 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:44:05.772 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:44:05.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7006a480 msgr2=0x7fad7006c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7006a480 0x7fad7006c940 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fad80033000 tx=0x7fad80009030 comp rx=0 tx=0).stop 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 msgr2=0x7fad981495f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad981495f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fad880119a0 tx=0x7fad88011e60 comp rx=0 tx=0).stop 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 shutdown_connections 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9815b000 0x7fad9815b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad98155dd0 0x7fad98149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7006a480 0x7fad7006c940 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 --2- 172.21.15.39:0/3031860053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad78003f20 0x7fad981495f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.761+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 >> 172.21.15.39:0/3031860053 conn(0x7fad9806a3e0 msgr2=0x7fad9806fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.762+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 shutdown_connections 2024-01-24T17:44:05.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:05.762+0000 7fad9f000640 1 -- 172.21.15.39:0/3031860053 wait complete. 2024-01-24T17:44:05.783 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:44:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.006+0000 7f0831800640 1 Processor -- start 2024-01-24T17:44:06.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.006+0000 7f0831800640 1 -- start start 2024-01-24T17:44:06.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.007+0000 7f0831800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f082c0d03b0 0x7f082c15db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f0831800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c0d0da0 0x7f082c0d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f0831800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c0d0860 con 0x7f082c15cbf0 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f0831800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082c0d1770 con 0x7f082c0d03b0 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f0831800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0d0a90 con 0x7f082c0d0da0 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f082be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f082c15cbf0 0x7f082c0ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48178/0 (socket says 172.21.15.39:48178) 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f082be00640 1 -- 172.21.15.39:0/1677804351 learned_addr learned my addr 172.21.15.39:0/1677804351 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123952560 0 0) 0x7f082c0d0860 con 0x7f082c15cbf0 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f080c003520 con 0x7f082c15cbf0 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f0830e00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c0d0da0 0x7f082c0d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.008+0000 7f082b400640 1 --2- 172.21.15.39:0/1677804351 >> v2:172.21.15.100:3300/0 conn(0x7f082c0d03b0 0x7f082c15db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245076932 0 0) 0x7f080c003520 con 0x7f082c15cbf0 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c0d0da0 msgr2=0x7f082c0d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c0d0da0 0x7f082c0d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 >> v2:172.21.15.100:3300/0 conn(0x7f082c0d03b0 msgr2=0x7f082c15db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:06.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 --2- 172.21.15.39:0/1677804351 >> v2:172.21.15.100:3300/0 conn(0x7f082c0d03b0 0x7f082c15db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c0cd760 con 0x7f082c15cbf0 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f081c0017f0 con 0x7f082c15cbf0 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 >> v1:172.21.15.39:6789/0 conn(0x7f082c15cbf0 legacy=0x7f082c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c0d0da0 0x7f080c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f080c003fa0 0x7f080c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f080c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0d1770 con 0x7f080c004d00 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0d0a90 con 0x7f080c003fa0 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c00ba80 con 0x7f082c0d0da0 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f082b400640 1 --2- 172.21.15.39:0/1677804351 >> v2:172.21.15.100:3300/0 conn(0x7f082c0d03b0 0x7f082c15db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.009+0000 7f0830e00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f080c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.010+0000 7f082be00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c0d0da0 0x7f080c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.010+0000 7f082b400640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f080c003fa0 0x7f080c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.010+0000 7f0830e00640 1 -- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c0d0da0 msgr2=0x7f080c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:06.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.010+0000 7f0830e00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c0d0da0 0x7f080c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.010+0000 7f0830e00640 1 -- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f080c003fa0 msgr2=0x7f080c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:06.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.010+0000 7f0830e00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f080c003fa0 0x7f080c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.010+0000 7f0830e00640 1 -- 172.21.15.39:0/1677804351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c003520 con 0x7f080c004d00 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.011+0000 7f0830e00640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f080c005180 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0824007360 tx=0x7f080c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b603fa982664ffd4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.011+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082402ccb0 con 0x7f080c004d00 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.011+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082402ce30 con 0x7f080c004d00 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.011+0000 7f082aa00640 1 -- 172.21.15.39:0/1677804351 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0824034680 con 0x7f080c004d00 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.012+0000 7f0831800640 1 -- 172.21.15.39:0/1677804351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 msgr2=0x7f080c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.012+0000 7f0831800640 1 --2- 172.21.15.39:0/1677804351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f080c005180 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f0824007360 tx=0x7f080c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.012+0000 7f0830e00640 1 -- 172.21.15.39:0/1677804351 reap_dead start 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.012+0000 7f0831800640 1 -- 172.21.15.39:0/1677804351 shutdown_connections 2024-01-24T17:44:06.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.012+0000 7f0831800640 1 -- 172.21.15.39:0/1677804351 >> 172.21.15.39:0/1677804351 conn(0x7f082c09bb90 msgr2=0x7f082c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:06.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.012+0000 7f0831800640 1 -- 172.21.15.39:0/1677804351 shutdown_connections 2024-01-24T17:44:06.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.012+0000 7f0831800640 1 -- 172.21.15.39:0/1677804351 wait complete. 2024-01-24T17:44:06.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.014+0000 7f0831800640 1 Processor -- start 2024-01-24T17:44:06.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.014+0000 7f0831800640 1 -- start start 2024-01-24T17:44:06.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f0831800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f082c0b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f0831800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f082c1523e0 0x7f082c0cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f0831800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c15e960 0x7f082c15ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f0831800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c15f320 con 0x7f080c004d00 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f0831800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0cc080 con 0x7f082c1523e0 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f0831800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0cb470 con 0x7f082c15e960 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f082be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f082c0b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f082be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f082c0b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39056/0 (socket says 172.21.15.39:39056) 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f082be00640 1 -- 172.21.15.39:0/3616258711 learned_addr learned my addr 172.21.15.39:0/3616258711 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.015+0000 7f082b400640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f082c1523e0 0x7f082c0cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f0830e00640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c15e960 0x7f082c15ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f082be00640 1 -- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c15e960 msgr2=0x7f082c15ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f082be00640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c15e960 0x7f082c15ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f082be00640 1 -- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f082c1523e0 msgr2=0x7f082c0cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f082be00640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f082c1523e0 0x7f082c0cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f082be00640 1 -- 172.21.15.39:0/3616258711 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c0cb750 con 0x7f080c004d00 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f0830e00640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c15e960 0x7f082c15ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:06.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f082be00640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f082c0b6730 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f081c0011d0 tx=0x7f081c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f0823e00640 1 -- 172.21.15.39:0/3616258711 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c00ee60 con 0x7f080c004d00 2024-01-24T17:44:06.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f0823e00640 1 -- 172.21.15.39:0/3616258711 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081c0072e0 con 0x7f080c004d00 2024-01-24T17:44:06.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f0823e00640 1 -- 172.21.15.39:0/3616258711 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c0057a0 con 0x7f080c004d00 2024-01-24T17:44:06.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.016+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082c0cba30 con 0x7f080c004d00 2024-01-24T17:44:06.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.017+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082c0b6c70 con 0x7f080c004d00 2024-01-24T17:44:06.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.018+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082c04dff0 con 0x7f080c004d00 2024-01-24T17:44:06.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.019+0000 7f0823e00640 1 -- 172.21.15.39:0/3616258711 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f081c005900 con 0x7f080c004d00 2024-01-24T17:44:06.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.020+0000 7f0823e00640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f080406a430 0x7f080406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.020+0000 7f0823e00640 1 -- 172.21.15.39:0/3616258711 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f081c0886c0 con 0x7f080c004d00 2024-01-24T17:44:06.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.021+0000 7f082b400640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f080406a430 0x7f080406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.022+0000 7f082b400640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f080406a430 0x7f080406c8f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f082c0b86d0 tx=0x7f0814008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.025+0000 7f0823e00640 1 -- 172.21.15.39:0/3616258711 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f081c053ff0 con 0x7f080c004d00 2024-01-24T17:44:06.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.219+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f082c0b4f60 con 0x7f080c004d00 2024-01-24T17:44:06.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.221+0000 7f0823e00640 1 -- 172.21.15.39:0/3616258711 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f081c057610 con 0x7f080c004d00 2024-01-24T17:44:06.222 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:06.222 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:05.222296+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:44:06.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f080406a430 msgr2=0x7f080406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f080406a430 0x7f080406c8f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f082c0b86d0 tx=0x7f0814008040 comp rx=0 tx=0).stop 2024-01-24T17:44:06.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 msgr2=0x7f082c0b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f082c0b6730 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f081c0011d0 tx=0x7f081c00ec10 comp rx=0 tx=0).stop 2024-01-24T17:44:06.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 shutdown_connections 2024-01-24T17:44:06.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f082c15e960 0x7f082c15ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f082c1523e0 0x7f082c0cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f080406a430 0x7f080406c8f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 --2- 172.21.15.39:0/3616258711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f080c004d00 0x7f082c0b6730 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 >> 172.21.15.39:0/3616258711 conn(0x7f082c09bb90 msgr2=0x7f082c06ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:06.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 shutdown_connections 2024-01-24T17:44:06.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.224+0000 7f0831800640 1 -- 172.21.15.39:0/3616258711 wait complete. 2024-01-24T17:44:06.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:44:06.245 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:44:06.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.468+0000 7f49ff600640 1 Processor -- start 2024-01-24T17:44:06.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.468+0000 7f49ff600640 1 -- start start 2024-01-24T17:44:06.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.470+0000 7f49ff600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49f80b0350 0x7f49f806c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.470+0000 7f49ff600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f49f815e190 0x7f49f80b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.470+0000 7f49ff600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f806d4c0 con 0x7f49f806c9f0 2024-01-24T17:44:06.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.470+0000 7f49ff600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f806cf00 con 0x7f49f80b0350 2024-01-24T17:44:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.470+0000 7f49ff600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f806d1e0 con 0x7f49f815e190 2024-01-24T17:44:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.470+0000 7f49fcc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49f80b0350 0x7f49f806c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.470+0000 7f49fcc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49f80b0350 0x7f49f806c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39060/0 (socket says 172.21.15.39:39060) 2024-01-24T17:44:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.470+0000 7f49fcc00640 1 -- 172.21.15.39:0/2100012194 learned_addr learned my addr 172.21.15.39:0/2100012194 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.471+0000 7f49fcc00640 1 -- 172.21.15.39:0/2100012194 >> v2:172.21.15.100:3300/0 conn(0x7f49f815e190 msgr2=0x7f49f80b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.471+0000 7f49fcc00640 1 --2- 172.21.15.39:0/2100012194 >> v2:172.21.15.100:3300/0 conn(0x7f49f815e190 0x7f49f80b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.471+0000 7f49fcc00640 1 -- 172.21.15.39:0/2100012194 >> v1:172.21.15.100:6789/0 conn(0x7f49f806c9f0 legacy=0x7f49f815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.471+0000 7f49fcc00640 1 -- 172.21.15.39:0/2100012194 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f80b15e0 con 0x7f49f80b0350 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.472+0000 7f49fcc00640 1 --2- 172.21.15.39:0/2100012194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f80b0350 0x7f49f806c4b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f49f80ada80 tx=0x7f49e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd66bd8c52ea9b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.472+0000 7f49f7e00640 1 -- 172.21.15.39:0/2100012194 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e80317e0 con 0x7f49f80b0350 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.472+0000 7f49f7e00640 1 -- 172.21.15.39:0/2100012194 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e8031940 con 0x7f49f80b0350 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.472+0000 7f49f7e00640 1 -- 172.21.15.39:0/2100012194 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8031cd0 con 0x7f49f80b0350 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.473+0000 7f49ff600640 1 -- 172.21.15.39:0/2100012194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f80b0350 msgr2=0x7f49f806c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.473+0000 7f49ff600640 1 --2- 172.21.15.39:0/2100012194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f80b0350 0x7f49f806c4b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f49f80ada80 tx=0x7f49e802b880 comp rx=0 tx=0).stop 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.473+0000 7f49ff600640 1 -- 172.21.15.39:0/2100012194 shutdown_connections 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.473+0000 7f49ff600640 1 --2- 172.21.15.39:0/2100012194 >> v2:172.21.15.100:3300/0 conn(0x7f49f815e190 0x7f49f80b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.473+0000 7f49ff600640 1 --2- 172.21.15.39:0/2100012194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f80b0350 0x7f49f806c4b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.473+0000 7f49ff600640 1 -- 172.21.15.39:0/2100012194 >> 172.21.15.39:0/2100012194 conn(0x7f49f80cb0e0 msgr2=0x7f49f80cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:06.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.473+0000 7f49ff600640 1 -- 172.21.15.39:0/2100012194 shutdown_connections 2024-01-24T17:44:06.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.473+0000 7f49ff600640 1 -- 172.21.15.39:0/2100012194 wait complete. 2024-01-24T17:44:06.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.475+0000 7f49ff600640 1 Processor -- start 2024-01-24T17:44:06.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.475+0000 7f49ff600640 1 -- start start 2024-01-24T17:44:06.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49ff600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f806c9f0 0x7f49f815a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49ff600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80b0350 0x7f49f81478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49ff600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f815e190 0x7f49f81f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49ff600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f8147e00 con 0x7f49f806c9f0 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49ff600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f81f2690 con 0x7f49f80b0350 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49ff600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f80ab960 con 0x7f49f815e190 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49fcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f806c9f0 0x7f49f815a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49fcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f806c9f0 0x7f49f815a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39070/0 (socket says 172.21.15.39:39070) 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.476+0000 7f49fcc00640 1 -- 172.21.15.39:0/2516059210 learned_addr learned my addr 172.21.15.39:0/2516059210 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49fcc00640 1 -- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f815e190 msgr2=0x7f49f81f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49fcc00640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f815e190 0x7f49f81f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49fcc00640 1 -- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80b0350 msgr2=0x7f49f81478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49fcc00640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80b0350 0x7f49f81478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49fcc00640 1 -- 172.21.15.39:0/2516059210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e80094d0 con 0x7f49f806c9f0 2024-01-24T17:44:06.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49fcc00640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f806c9f0 0x7f49f815a120 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f49e802b8c0 tx=0x7f49e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49f5600640 1 -- 172.21.15.39:0/2516059210 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8032940 con 0x7f49f806c9f0 2024-01-24T17:44:06.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49f5600640 1 -- 172.21.15.39:0/2516059210 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e8032aa0 con 0x7f49f806c9f0 2024-01-24T17:44:06.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f80abc40 con 0x7f49f806c9f0 2024-01-24T17:44:06.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.477+0000 7f49f5600640 1 -- 172.21.15.39:0/2516059210 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8032e50 con 0x7f49f806c9f0 2024-01-24T17:44:06.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.478+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f80ac200 con 0x7f49f806c9f0 2024-01-24T17:44:06.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.479+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f80ac3f0 con 0x7f49f806c9f0 2024-01-24T17:44:06.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.480+0000 7f49f5600640 1 -- 172.21.15.39:0/2516059210 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f49e804e450 con 0x7f49f806c9f0 2024-01-24T17:44:06.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.481+0000 7f49f5600640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49d006a430 0x7f49d006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.481+0000 7f49f5600640 1 -- 172.21.15.39:0/2516059210 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f49e80aa890 con 0x7f49f806c9f0 2024-01-24T17:44:06.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.482+0000 7f49efe00640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49d006a430 0x7f49d006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.483+0000 7f49efe00640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49d006a430 0x7f49d006c8f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f49e0007fd0 tx=0x7f49e00079a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.485+0000 7f49f5600640 1 -- 172.21.15.39:0/2516059210 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f49e8076280 con 0x7f49f806c9f0 2024-01-24T17:44:06.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.679+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49f8073ef0 con 0x7f49f806c9f0 2024-01-24T17:44:06.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.680+0000 7f49f5600640 1 -- 172.21.15.39:0/2516059210 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49e80798a0 con 0x7f49f806c9f0 2024-01-24T17:44:06.681 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49d006a430 msgr2=0x7f49d006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49d006a430 0x7f49d006c8f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f49e0007fd0 tx=0x7f49e00079a0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f806c9f0 msgr2=0x7f49f815a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f806c9f0 0x7f49f815a120 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f49e802b8c0 tx=0x7f49e80046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 shutdown_connections 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49f815e190 0x7f49f81f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49f80b0350 0x7f49f81478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49d006a430 0x7f49d006c8f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 --2- 172.21.15.39:0/2516059210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f806c9f0 0x7f49f815a120 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.682+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 >> 172.21.15.39:0/2516059210 conn(0x7f49f80cb0e0 msgr2=0x7f49f80af970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:06.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.683+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 shutdown_connections 2024-01-24T17:44:06.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.683+0000 7f49ff600640 1 -- 172.21.15.39:0/2516059210 wait complete. 2024-01-24T17:44:06.703 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T17:44:06.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.923+0000 7f39c9800640 1 Processor -- start 2024-01-24T17:44:06.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.924+0000 7f39c9800640 1 -- start start 2024-01-24T17:44:06.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.925+0000 7f39c9800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c415c900 0x7f39c40d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.925+0000 7f39c9800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f39c406c2e0 0x7f39c40cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.925+0000 7f39c9800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c40d03e0 con 0x7f39c40d2640 2024-01-24T17:44:06.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.925+0000 7f39c9800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c40d0630 con 0x7f39c406c2e0 2024-01-24T17:44:06.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.925+0000 7f39c9800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c40d0910 con 0x7f39c415c900 2024-01-24T17:44:06.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.926+0000 7f39bbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f39c406c2e0 0x7f39c40cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.926+0000 7f39bbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f39c406c2e0 0x7f39c40cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39074/0 (socket says 172.21.15.39:39074) 2024-01-24T17:44:06.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.926+0000 7f39bbe00640 1 -- 172.21.15.39:0/183280065 learned_addr learned my addr 172.21.15.39:0/183280065 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:06.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.926+0000 7f39c3e00640 1 --2- 172.21.15.39:0/183280065 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c415c900 0x7f39c40d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.926+0000 7f39bbe00640 1 -- 172.21.15.39:0/183280065 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c415c900 msgr2=0x7f39c40d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:06.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.926+0000 7f39bbe00640 1 --2- 172.21.15.39:0/183280065 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c415c900 0x7f39c40d13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.926+0000 7f39bbe00640 1 -- 172.21.15.39:0/183280065 >> v1:172.21.15.100:6789/0 conn(0x7f39c40d2640 legacy=0x7f39c40d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:06.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.926+0000 7f39bbe00640 1 -- 172.21.15.39:0/183280065 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c40d0bf0 con 0x7f39c406c2e0 2024-01-24T17:44:06.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.928+0000 7f39bbe00640 1 --2- 172.21.15.39:0/183280065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c406c2e0 0x7f39c40cf950 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0098a0 tx=0x7f39ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7fcd28c0ef13e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.928+0000 7f39c3400640 1 -- 172.21.15.39:0/183280065 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac031640 con 0x7f39c406c2e0 2024-01-24T17:44:06.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.928+0000 7f39c3400640 1 -- 172.21.15.39:0/183280065 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac0317a0 con 0x7f39c406c2e0 2024-01-24T17:44:06.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.928+0000 7f39c3400640 1 -- 172.21.15.39:0/183280065 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac031b30 con 0x7f39c406c2e0 2024-01-24T17:44:06.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.929+0000 7f39c9800640 1 -- 172.21.15.39:0/183280065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c406c2e0 msgr2=0x7f39c40cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.929+0000 7f39c9800640 1 --2- 172.21.15.39:0/183280065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c406c2e0 0x7f39c40cf950 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0098a0 tx=0x7f39ac02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.929+0000 7f39c9800640 1 -- 172.21.15.39:0/183280065 shutdown_connections 2024-01-24T17:44:06.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.929+0000 7f39c9800640 1 --2- 172.21.15.39:0/183280065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c406c2e0 0x7f39c40cf950 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.929+0000 7f39c9800640 1 --2- 172.21.15.39:0/183280065 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c415c900 0x7f39c40d13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.929+0000 7f39c9800640 1 -- 172.21.15.39:0/183280065 >> 172.21.15.39:0/183280065 conn(0x7f39c409bb30 msgr2=0x7f39c4089fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:06.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.929+0000 7f39c9800640 1 -- 172.21.15.39:0/183280065 shutdown_connections 2024-01-24T17:44:06.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.929+0000 7f39c9800640 1 -- 172.21.15.39:0/183280065 wait complete. 2024-01-24T17:44:06.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.931+0000 7f39c9800640 1 Processor -- start 2024-01-24T17:44:06.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.932+0000 7f39c9800640 1 -- start start 2024-01-24T17:44:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c9800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c406c2e0 0x7f39c409c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c9800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39c40d2640 0x7f39c409f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c9800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c415c900 0x7f39c41f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c9800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c41f4a60 con 0x7f39c415c900 2024-01-24T17:44:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c9800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c41f3e50 con 0x7f39c40d2640 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c9800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c41f4130 con 0x7f39c406c2e0 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c415c900 0x7f39c41f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c415c900 0x7f39c41f4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39084/0 (socket says 172.21.15.39:39084) 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c8e00640 1 -- 172.21.15.39:0/2694218151 learned_addr learned my addr 172.21.15.39:0/2694218151 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39c3e00640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c406c2e0 0x7f39c409c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.933+0000 7f39bbe00640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39c40d2640 0x7f39c409f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c8e00640 1 -- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c406c2e0 msgr2=0x7f39c409c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c8e00640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c406c2e0 0x7f39c409c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c8e00640 1 -- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39c40d2640 msgr2=0x7f39c409f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c8e00640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39c40d2640 0x7f39c409f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c8e00640 1 -- 172.21.15.39:0/2694218151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0094d0 con 0x7f39c415c900 2024-01-24T17:44:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c8e00640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c415c900 0x7f39c41f4ef0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00c700 tx=0x7f39bc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c0c00640 1 -- 172.21.15.39:0/2694218151 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00aa50 con 0x7f39c415c900 2024-01-24T17:44:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c0c00640 1 -- 172.21.15.39:0/2694218151 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc00abb0 con 0x7f39c415c900 2024-01-24T17:44:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.934+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c41f4410 con 0x7f39c415c900 2024-01-24T17:44:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.935+0000 7f39c0c00640 1 -- 172.21.15.39:0/2694218151 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00ce10 con 0x7f39c415c900 2024-01-24T17:44:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.935+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c409cdf0 con 0x7f39c415c900 2024-01-24T17:44:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.936+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c41f4bd0 con 0x7f39c415c900 2024-01-24T17:44:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.937+0000 7f39c0c00640 1 -- 172.21.15.39:0/2694218151 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f39bc012420 con 0x7f39c415c900 2024-01-24T17:44:06.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.937+0000 7f39c0c00640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f399c06a400 0x7f399c06c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:06.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.938+0000 7f39c0c00640 1 -- 172.21.15.39:0/2694218151 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f39bc0887c0 con 0x7f39c415c900 2024-01-24T17:44:06.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.938+0000 7f39c3e00640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f399c06a400 0x7f399c06c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:06.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.939+0000 7f39c3e00640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f399c06a400 0x7f399c06c8c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f39b400a8b0 tx=0x7f39b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:06.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:06.942+0000 7f39c0c00640 1 -- 172.21.15.39:0/2694218151 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f39bc0540f0 con 0x7f39c415c900 2024-01-24T17:44:07.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.140+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f39c404dff0 con 0x7f39c415c900 2024-01-24T17:44:07.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.141+0000 7f39c0c00640 1 -- 172.21.15.39:0/2694218151 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v44) v1 ==== 84+0+65 (secure 0 0 0) 0x7f39bc016090 con 0x7f39c415c900 2024-01-24T17:44:07.142 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:07.142 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-01-24T17:44:07.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:07.142 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-01-24T17:44:07.142 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-01-24T17:44:07.142 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:07.142 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f399c06a400 msgr2=0x7f399c06c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f399c06a400 0x7f399c06c8c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f39b400a8b0 tx=0x7f39b4008040 comp rx=0 tx=0).stop 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c415c900 msgr2=0x7f39c41f4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c415c900 0x7f39c41f4ef0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00c700 tx=0x7f39bc00cbc0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 shutdown_connections 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f399c06a400 0x7f399c06c8c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39c415c900 0x7f39c41f4ef0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f39c40d2640 0x7f39c409f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 --2- 172.21.15.39:0/2694218151 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f39c406c2e0 0x7f39c409c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 >> 172.21.15.39:0/2694218151 conn(0x7f39c409bb30 msgr2=0x7f39c40b5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:07.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 shutdown_connections 2024-01-24T17:44:07.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.144+0000 7f39c9800640 1 -- 172.21.15.39:0/2694218151 wait complete. 2024-01-24T17:44:07.163 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T17:44:07.164 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T17:44:07.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.387+0000 7f752c600640 1 Processor -- start 2024-01-24T17:44:07.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.387+0000 7f752c600640 1 -- start start 2024-01-24T17:44:07.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.389+0000 7f752c600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75241597f0 0x7f752406d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:07.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.389+0000 7f752c600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f75240b00e0 0x7f75240b0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:07.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.389+0000 7f752c600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75240b1bc0 con 0x7f752406bfa0 2024-01-24T17:44:07.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.389+0000 7f752c600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75240ca0d0 con 0x7f75240b00e0 2024-01-24T17:44:07.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.389+0000 7f752c600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75240b0ad0 con 0x7f75241597f0 2024-01-24T17:44:07.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.389+0000 7f7529c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75241597f0 0x7f752406d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.390+0000 7f7529c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75241597f0 0x7f752406d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:45068/0 (socket says 172.21.15.39:45068) 2024-01-24T17:44:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.390+0000 7f7529c00640 1 -- 172.21.15.39:0/3631040780 learned_addr learned my addr 172.21.15.39:0/3631040780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.390+0000 7f752a600640 1 --2- 172.21.15.39:0/3631040780 >> v2:172.21.15.100:3300/0 conn(0x7f75240b00e0 0x7f75240b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.391+0000 7f7529c00640 1 -- 172.21.15.39:0/3631040780 >> v2:172.21.15.100:3300/0 conn(0x7f75240b00e0 msgr2=0x7f75240b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.391+0000 7f7529c00640 1 --2- 172.21.15.39:0/3631040780 >> v2:172.21.15.100:3300/0 conn(0x7f75240b00e0 0x7f75240b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.391+0000 7f7529c00640 1 -- 172.21.15.39:0/3631040780 >> v1:172.21.15.100:6789/0 conn(0x7f752406bfa0 legacy=0x7f75240b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:07.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.391+0000 7f7529c00640 1 -- 172.21.15.39:0/3631040780 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752406c4d0 con 0x7f75241597f0 2024-01-24T17:44:07.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.392+0000 7f7529c00640 1 --2- 172.21.15.39:0/3631040780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75241597f0 0x7f752406d890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7514009870 tx=0x7f751402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65876a3d24839d05 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:07.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.392+0000 7f7523e00640 1 -- 172.21.15.39:0/3631040780 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514030840 con 0x7f75241597f0 2024-01-24T17:44:07.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.392+0000 7f7523e00640 1 -- 172.21.15.39:0/3631040780 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75140309a0 con 0x7f75241597f0 2024-01-24T17:44:07.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.392+0000 7f7523e00640 1 -- 172.21.15.39:0/3631040780 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514030d30 con 0x7f75241597f0 2024-01-24T17:44:07.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.393+0000 7f752c600640 1 -- 172.21.15.39:0/3631040780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75241597f0 msgr2=0x7f752406d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:07.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.393+0000 7f752c600640 1 --2- 172.21.15.39:0/3631040780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75241597f0 0x7f752406d890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7514009870 tx=0x7f751402b980 comp rx=0 tx=0).stop 2024-01-24T17:44:07.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.393+0000 7f752c600640 1 -- 172.21.15.39:0/3631040780 shutdown_connections 2024-01-24T17:44:07.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.393+0000 7f752c600640 1 --2- 172.21.15.39:0/3631040780 >> v2:172.21.15.100:3300/0 conn(0x7f75240b00e0 0x7f75240b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.393+0000 7f752c600640 1 --2- 172.21.15.39:0/3631040780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75241597f0 0x7f752406d890 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.393+0000 7f752c600640 1 -- 172.21.15.39:0/3631040780 >> 172.21.15.39:0/3631040780 conn(0x7f75240cb0e0 msgr2=0x7f75240cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:07.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.393+0000 7f752c600640 1 -- 172.21.15.39:0/3631040780 shutdown_connections 2024-01-24T17:44:07.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.394+0000 7f752c600640 1 -- 172.21.15.39:0/3631040780 wait complete. 2024-01-24T17:44:07.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.396+0000 7f752c600640 1 Processor -- start 2024-01-24T17:44:07.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.396+0000 7f752c600640 1 -- start start 2024-01-24T17:44:07.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.397+0000 7f752c600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752406bfa0 0x7f75240abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:07.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.397+0000 7f752c600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75240b00e0 0x7f7524153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.397+0000 7f752c600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75241597f0 0x7f7524154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.397+0000 7f752c600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7524152260 con 0x7f752406bfa0 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.397+0000 7f752c600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75241548b0 con 0x7f75241597f0 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.397+0000 7f752c600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7524154b90 con 0x7f75240b00e0 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752406bfa0 0x7f75240abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752406bfa0 0x7f75240abb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39092/0 (socket says 172.21.15.39:39092) 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529c00640 1 -- 172.21.15.39:0/2498696489 learned_addr learned my addr 172.21.15.39:0/2498696489 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529200640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75240b00e0 0x7f7524153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:07.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f752a600640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75241597f0 0x7f7524154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:07.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529c00640 1 -- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75240b00e0 msgr2=0x7f7524153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:07.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529c00640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75240b00e0 0x7f7524153d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529c00640 1 -- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75241597f0 msgr2=0x7f7524154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529c00640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75241597f0 0x7f7524154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529c00640 1 -- 172.21.15.39:0/2498696489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140094d0 con 0x7f752406bfa0 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.398+0000 7f7529200640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75240b00e0 0x7f7524153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.399+0000 7f7529c00640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752406bfa0 0x7f75240abb30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f751402b9c0 tx=0x7f7514004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.399+0000 7f7521600640 1 -- 172.21.15.39:0/2498696489 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751403e040 con 0x7f752406bfa0 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.399+0000 7f7521600640 1 -- 172.21.15.39:0/2498696489 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7514039c30 con 0x7f752406bfa0 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.399+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7524154e70 con 0x7f752406bfa0 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.399+0000 7f7521600640 1 -- 172.21.15.39:0/2498696489 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75140443f0 con 0x7f752406bfa0 2024-01-24T17:44:07.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.399+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7524155430 con 0x7f752406bfa0 2024-01-24T17:44:07.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.401+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7524073ef0 con 0x7f752406bfa0 2024-01-24T17:44:07.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.402+0000 7f7521600640 1 -- 172.21.15.39:0/2498696489 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f751404d3b0 con 0x7f752406bfa0 2024-01-24T17:44:07.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.404+0000 7f7521600640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74fc06a430 0x7f74fc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:07.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.404+0000 7f7521600640 1 -- 172.21.15.39:0/2498696489 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f75140aaca0 con 0x7f752406bfa0 2024-01-24T17:44:07.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.404+0000 7f7529200640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74fc06a430 0x7f74fc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:07.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.405+0000 7f7529200640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74fc06a430 0x7f74fc06c8f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f750c007fd0 tx=0x7f750c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:07.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.408+0000 7f7521600640 1 -- 172.21.15.39:0/2498696489 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7514076540 con 0x7f752406bfa0 2024-01-24T17:44:07.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:07.604+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f75240c5260 con 0x7f752406bfa0 2024-01-24T17:44:08.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.464+0000 7f7521600640 1 -- 172.21.15.39:0/2498696489 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v45) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7514079b60 con 0x7f752406bfa0 2024-01-24T17:44:08.465 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-01-24T17:44:08.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.469+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74fc06a430 msgr2=0x7f74fc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74fc06a430 0x7f74fc06c8f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f750c007fd0 tx=0x7f750c009040 comp rx=0 tx=0).stop 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752406bfa0 msgr2=0x7f75240abb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752406bfa0 0x7f75240abb30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f751402b9c0 tx=0x7f7514004730 comp rx=0 tx=0).stop 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 shutdown_connections 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f75241597f0 0x7f7524154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f75240b00e0 0x7f7524153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f74fc06a430 0x7f74fc06c8f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 --2- 172.21.15.39:0/2498696489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f752406bfa0 0x7f75240abb30 secure :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f751402b9c0 tx=0x7f7514004730 comp rx=0 tx=0).stop 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.470+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 >> 172.21.15.39:0/2498696489 conn(0x7f75240cb0e0 msgr2=0x7f75240ca4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.471+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 shutdown_connections 2024-01-24T17:44:08.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.472+0000 7f752c600640 1 -- 172.21.15.39:0/2498696489 wait complete. 2024-01-24T17:44:08.495 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T17:44:08.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.717+0000 7fec43600640 1 Processor -- start 2024-01-24T17:44:08.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.717+0000 7fec43600640 1 -- start start 2024-01-24T17:44:08.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.718+0000 7fec43600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec3c158600 0x7fec3c0cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:08.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.718+0000 7fec43600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c0cb900 0x7fec3c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:08.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.718+0000 7fec43600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3c15b030 con 0x7fec3c0b1370 2024-01-24T17:44:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.718+0000 7fec43600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3c15bcb0 con 0x7fec3c158600 2024-01-24T17:44:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.718+0000 7fec43600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c0cbd90 con 0x7fec3c0cb900 2024-01-24T17:44:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.719+0000 7fec40c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec3c158600 0x7fec3c0cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.719+0000 7fec40c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec3c158600 0x7fec3c0cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39106/0 (socket says 172.21.15.39:39106) 2024-01-24T17:44:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.719+0000 7fec40c00640 1 -- 172.21.15.39:0/831109759 learned_addr learned my addr 172.21.15.39:0/831109759 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.719+0000 7fec41600640 1 --2- 172.21.15.39:0/831109759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c0cb900 0x7fec3c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.719+0000 7fec40c00640 1 -- 172.21.15.39:0/831109759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c0cb900 msgr2=0x7fec3c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:08.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.719+0000 7fec40c00640 1 --2- 172.21.15.39:0/831109759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c0cb900 0x7fec3c0b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:08.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.719+0000 7fec40c00640 1 -- 172.21.15.39:0/831109759 >> v1:172.21.15.39:6789/0 conn(0x7fec3c0b1370 legacy=0x7fec3c0b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:08.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.719+0000 7fec40c00640 1 -- 172.21.15.39:0/831109759 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c15b310 con 0x7fec3c158600 2024-01-24T17:44:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.720+0000 7fec40c00640 1 --2- 172.21.15.39:0/831109759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c158600 0x7fec3c0cb3c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fec24009880 tx=0x7fec2402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c429b2178a32daff server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.721+0000 7fec3b400640 1 -- 172.21.15.39:0/831109759 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2402e070 con 0x7fec3c158600 2024-01-24T17:44:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.721+0000 7fec3b400640 1 -- 172.21.15.39:0/831109759 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec24004030 con 0x7fec3c158600 2024-01-24T17:44:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.721+0000 7fec3b400640 1 -- 172.21.15.39:0/831109759 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24004420 con 0x7fec3c158600 2024-01-24T17:44:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.721+0000 7fec43600640 1 -- 172.21.15.39:0/831109759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c158600 msgr2=0x7fec3c0cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.721+0000 7fec43600640 1 --2- 172.21.15.39:0/831109759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c158600 0x7fec3c0cb3c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fec24009880 tx=0x7fec2402b840 comp rx=0 tx=0).stop 2024-01-24T17:44:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.722+0000 7fec43600640 1 -- 172.21.15.39:0/831109759 shutdown_connections 2024-01-24T17:44:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.722+0000 7fec43600640 1 --2- 172.21.15.39:0/831109759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c0cb900 0x7fec3c0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.722+0000 7fec43600640 1 --2- 172.21.15.39:0/831109759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c158600 0x7fec3c0cb3c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.722+0000 7fec43600640 1 -- 172.21.15.39:0/831109759 >> 172.21.15.39:0/831109759 conn(0x7fec3c06a090 msgr2=0x7fec3c06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.722+0000 7fec43600640 1 -- 172.21.15.39:0/831109759 shutdown_connections 2024-01-24T17:44:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.722+0000 7fec43600640 1 -- 172.21.15.39:0/831109759 wait complete. 2024-01-24T17:44:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.723+0000 7fec43600640 1 Processor -- start 2024-01-24T17:44:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.724+0000 7fec43600640 1 -- start start 2024-01-24T17:44:08.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.724+0000 7fec43600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c0b1370 0x7fec3c0adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:08.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.724+0000 7fec43600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec3c0cb900 0x7fec3c157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:08.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.724+0000 7fec43600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c158600 0x7fec3c0c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:08.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec43600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c1578a0 con 0x7fec3c0b1370 2024-01-24T17:44:08.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec43600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c0c30c0 con 0x7fec3c0cb900 2024-01-24T17:44:08.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec43600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c0c33a0 con 0x7fec3c158600 2024-01-24T17:44:08.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec40c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c0b1370 0x7fec3c0adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:08.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec40c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c0b1370 0x7fec3c0adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39118/0 (socket says 172.21.15.39:39118) 2024-01-24T17:44:08.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec40c00640 1 -- 172.21.15.39:0/1015815812 learned_addr learned my addr 172.21.15.39:0/1015815812 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:08.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec41600640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c158600 0x7fec3c0c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:08.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec3be00640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec3c0cb900 0x7fec3c157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:08.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec40c00640 1 -- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c158600 msgr2=0x7fec3c0c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:08.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec40c00640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c158600 0x7fec3c0c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:08.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec40c00640 1 -- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec3c0cb900 msgr2=0x7fec3c157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:08.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec40c00640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec3c0cb900 0x7fec3c157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:08.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.725+0000 7fec40c00640 1 -- 172.21.15.39:0/1015815812 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec24009460 con 0x7fec3c0b1370 2024-01-24T17:44:08.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.726+0000 7fec40c00640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c0b1370 0x7fec3c0adb90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fec24035040 tx=0x7fec240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:08.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.726+0000 7fec38c00640 1 -- 172.21.15.39:0/1015815812 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24003c20 con 0x7fec3c0b1370 2024-01-24T17:44:08.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.726+0000 7fec38c00640 1 -- 172.21.15.39:0/1015815812 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec24003da0 con 0x7fec3c0b1370 2024-01-24T17:44:08.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.726+0000 7fec38c00640 1 -- 172.21.15.39:0/1015815812 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24006280 con 0x7fec3c0b1370 2024-01-24T17:44:08.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.726+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c0c3680 con 0x7fec3c0b1370 2024-01-24T17:44:08.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.727+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c0c3c40 con 0x7fec3c0b1370 2024-01-24T17:44:08.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.728+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3c04e060 con 0x7fec3c0b1370 2024-01-24T17:44:08.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.729+0000 7fec38c00640 1 -- 172.21.15.39:0/1015815812 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fec240063e0 con 0x7fec3c0b1370 2024-01-24T17:44:08.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.730+0000 7fec38c00640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec1406a430 0x7fec1406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:08.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.730+0000 7fec38c00640 1 -- 172.21.15.39:0/1015815812 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fec240aa7e0 con 0x7fec3c0b1370 2024-01-24T17:44:08.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.731+0000 7fec3be00640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec1406a430 0x7fec1406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:08.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.732+0000 7fec3be00640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec1406a430 0x7fec1406c8f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0c1ee0 tx=0x7fec2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:08.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.735+0000 7fec38c00640 1 -- 172.21.15.39:0/1015815812 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fec24075fa0 con 0x7fec3c0b1370 2024-01-24T17:44:08.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:08.930+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fec3c06c4f0 con 0x7fec3c0b1370 2024-01-24T17:44:09.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.484+0000 7fec38c00640 1 -- 172.21.15.39:0/1015815812 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v46) v1 ==== 170+0+0 (secure 0 0 0) 0x7fec24046e90 con 0x7fec3c0b1370 2024-01-24T17:44:09.485 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-01-24T17:44:09.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.490+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec1406a430 msgr2=0x7fec1406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:09.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.490+0000 7fec43600640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec1406a430 0x7fec1406c8f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0c1ee0 tx=0x7fec2c009040 comp rx=0 tx=0).stop 2024-01-24T17:44:09.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.490+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c0b1370 msgr2=0x7fec3c0adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:09.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.490+0000 7fec43600640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c0b1370 0x7fec3c0adb90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fec24035040 tx=0x7fec240046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.491+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 shutdown_connections 2024-01-24T17:44:09.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.491+0000 7fec43600640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec3c158600 0x7fec3c0c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.491+0000 7fec43600640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec3c0cb900 0x7fec3c157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.491+0000 7fec43600640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec1406a430 0x7fec1406c8f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.491+0000 7fec43600640 1 --2- 172.21.15.39:0/1015815812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec3c0b1370 0x7fec3c0adb90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.491+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 >> 172.21.15.39:0/1015815812 conn(0x7fec3c06a090 msgr2=0x7fec3c0c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:09.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.491+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 shutdown_connections 2024-01-24T17:44:09.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.492+0000 7fec43600640 1 -- 172.21.15.39:0/1015815812 wait complete. 2024-01-24T17:44:09.526 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T17:44:09.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.707+0000 7efe7c000640 1 Processor -- start 2024-01-24T17:44:09.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.707+0000 7efe7c000640 1 -- start start 2024-01-24T17:44:09.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.709+0000 7efe7c000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efe7406bfa0 0x7efe740b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.709+0000 7efe7c000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 0x7efe740b0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.709+0000 7efe7c000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7406c4d0 con 0x7efe741597f0 2024-01-24T17:44:09.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.709+0000 7efe7c000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe740ca0d0 con 0x7efe7406bfa0 2024-01-24T17:44:09.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.709+0000 7efe7c000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe740b0a90 con 0x7efe740b00e0 2024-01-24T17:44:09.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.709+0000 7efe79600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efe741597f0 0x7efe7406d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48190/0 (socket says 172.21.15.39:48190) 2024-01-24T17:44:09.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.709+0000 7efe79600640 1 -- 172.21.15.39:0/3075297771 learned_addr learned my addr 172.21.15.39:0/3075297771 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:09.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe78c00640 1 --2- 172.21.15.39:0/3075297771 >> v2:172.21.15.100:3300/0 conn(0x7efe7406bfa0 0x7efe740b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe7a000640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 0x7efe740b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537270371 0 0) 0x7efe7406c4d0 con 0x7efe741597f0 2024-01-24T17:44:09.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe54003520 con 0x7efe741597f0 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3924890428 0 0) 0x7efe54003520 con 0x7efe741597f0 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 msgr2=0x7efe740b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 0x7efe740b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 >> v2:172.21.15.100:3300/0 conn(0x7efe7406bfa0 msgr2=0x7efe740b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 --2- 172.21.15.39:0/3075297771 >> v2:172.21.15.100:3300/0 conn(0x7efe7406bfa0 0x7efe740b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7406c7b0 con 0x7efe741597f0 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7efe640019f0 con 0x7efe741597f0 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 >> v1:172.21.15.39:6789/0 conn(0x7efe741597f0 legacy=0x7efe7406d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 0x7efe540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe54003f20 0x7efe540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe54004d00 0x7efe54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740ca0d0 con 0x7efe54004d00 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740b0a90 con 0x7efe54003f20 2024-01-24T17:44:09.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.710+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5400ba00 con 0x7efe740b00e0 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe7a000640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe54004d00 0x7efe54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe79600640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 0x7efe540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe78c00640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe54003f20 0x7efe540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe7a000640 1 -- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 msgr2=0x7efe540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe7a000640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 0x7efe540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe7a000640 1 -- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe54003f20 msgr2=0x7efe540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe7a000640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe54003f20 0x7efe540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe7a000640 1 -- 172.21.15.39:0/3075297771 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe54003520 con 0x7efe54004d00 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe78c00640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe54003f20 0x7efe540047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:09.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.711+0000 7efe79600640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe740b00e0 0x7efe540039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.712+0000 7efe7a000640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe54004d00 0x7efe54005180 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00ba30 tx=0x7efe540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ad75b31f605b97 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c032630 con 0x7efe54004d00 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c032790 con 0x7efe54004d00 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe73e00640 1 -- 172.21.15.39:0/3075297771 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c032b20 con 0x7efe54004d00 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe7c000640 1 -- 172.21.15.39:0/3075297771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe54004d00 msgr2=0x7efe54005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe7c000640 1 --2- 172.21.15.39:0/3075297771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe54004d00 0x7efe54005180 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00ba30 tx=0x7efe540057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe7a000640 1 -- 172.21.15.39:0/3075297771 reap_dead start 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe7c000640 1 -- 172.21.15.39:0/3075297771 shutdown_connections 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe7c000640 1 -- 172.21.15.39:0/3075297771 >> 172.21.15.39:0/3075297771 conn(0x7efe740cb0e0 msgr2=0x7efe740cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe7c000640 1 -- 172.21.15.39:0/3075297771 shutdown_connections 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.713+0000 7efe7c000640 1 -- 172.21.15.39:0/3075297771 wait complete. 2024-01-24T17:44:09.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.715+0000 7efe7c000640 1 Processor -- start 2024-01-24T17:44:09.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.715+0000 7efe7c000640 1 -- start start 2024-01-24T17:44:09.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.716+0000 7efe7c000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe54004d00 0x7efe74146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.716+0000 7efe7c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe740c5730 0x7efe740c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.716+0000 7efe7c000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe740c5bb0 0x7efe740c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.716+0000 7efe7c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe74146940 con 0x7efe740c5730 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.716+0000 7efe7c000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740b1520 con 0x7efe740c5bb0 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.716+0000 7efe7c000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe740c83b0 con 0x7efe54004d00 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.716+0000 7efe78c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe740c5730 0x7efe740c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe78c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe740c5730 0x7efe740c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39130/0 (socket says 172.21.15.39:39130) 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe78c00640 1 -- 172.21.15.39:0/1268973576 learned_addr learned my addr 172.21.15.39:0/1268973576 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe79600640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe54004d00 0x7efe74146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe7a000640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe740c5bb0 0x7efe740c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe78c00640 1 -- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe54004d00 msgr2=0x7efe74146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe78c00640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe54004d00 0x7efe74146400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe78c00640 1 -- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe740c5bb0 msgr2=0x7efe740c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe78c00640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe740c5bb0 0x7efe740c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe78c00640 1 -- 172.21.15.39:0/1268973576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe740c8690 con 0x7efe740c5730 2024-01-24T17:44:09.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe78c00640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe740c5730 0x7efe740c7330 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00bde0 tx=0x7efe5c013520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:09.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.717+0000 7efe71600640 1 -- 172.21.15.39:0/1268973576 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00de30 con 0x7efe740c5730 2024-01-24T17:44:09.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.718+0000 7efe71600640 1 -- 172.21.15.39:0/1268973576 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5c01a420 con 0x7efe740c5730 2024-01-24T17:44:09.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.718+0000 7efe71600640 1 -- 172.21.15.39:0/1268973576 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c01a7b0 con 0x7efe740c5730 2024-01-24T17:44:09.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.718+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe740b17c0 con 0x7efe740c5730 2024-01-24T17:44:09.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.718+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe740c7db0 con 0x7efe740c5730 2024-01-24T17:44:09.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.719+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe740cae20 con 0x7efe740c5730 2024-01-24T17:44:09.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.720+0000 7efe71600640 1 -- 172.21.15.39:0/1268973576 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efe5c01a910 con 0x7efe740c5730 2024-01-24T17:44:09.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.721+0000 7efe71600640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe4c06a460 0x7efe4c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:09.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.721+0000 7efe71600640 1 -- 172.21.15.39:0/1268973576 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5609+0+0 (secure 0 0 0) 0x7efe5c08d110 con 0x7efe740c5730 2024-01-24T17:44:09.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.722+0000 7efe79600640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe4c06a460 0x7efe4c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:09.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.723+0000 7efe79600640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe4c06a460 0x7efe4c06c920 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efe64002230 tx=0x7efe64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:09.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.726+0000 7efe71600640 1 -- 172.21.15.39:0/1268973576 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efe5c0586c0 con 0x7efe740c5730 2024-01-24T17:44:09.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.958+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7efe74073ef0 con 0x7efe740c5730 2024-01-24T17:44:09.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.973+0000 7efe71600640 1 -- 172.21.15.39:0/1268973576 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v19) v1 ==== 172+0+0 (secure 0 0 0) 0x7efe5c05bce0 con 0x7efe740c5730 2024-01-24T17:44:09.975 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 6 and data pool 7 2024-01-24T17:44:09.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.979+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe4c06a460 msgr2=0x7efe4c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:09.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.979+0000 7efe7c000640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe4c06a460 0x7efe4c06c920 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efe64002230 tx=0x7efe64009040 comp rx=0 tx=0).stop 2024-01-24T17:44:09.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.979+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe740c5730 msgr2=0x7efe740c7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:09.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.979+0000 7efe7c000640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe740c5730 0x7efe740c7330 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00bde0 tx=0x7efe5c013520 comp rx=0 tx=0).stop 2024-01-24T17:44:09.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.980+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 shutdown_connections 2024-01-24T17:44:09.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.980+0000 7efe7c000640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe740c5bb0 0x7efe740c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.980+0000 7efe7c000640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe4c06a460 0x7efe4c06c920 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.980+0000 7efe7c000640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe740c5730 0x7efe740c7330 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.980+0000 7efe7c000640 1 --2- 172.21.15.39:0/1268973576 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe54004d00 0x7efe74146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:09.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.980+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 >> 172.21.15.39:0/1268973576 conn(0x7efe740cb0e0 msgr2=0x7efe74157a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:09.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.980+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 shutdown_connections 2024-01-24T17:44:09.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:09.981+0000 7efe7c000640 1 -- 172.21.15.39:0/1268973576 wait complete. 2024-01-24T17:44:10.009 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:44:10.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.189+0000 7fce3bc00640 1 Processor -- start 2024-01-24T17:44:10.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.189+0000 7fce3bc00640 1 -- start start 2024-01-24T17:44:10.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.190+0000 7fce3bc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fce34070860 0x7fce340cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.190+0000 7fce3bc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fce340ccd00 0x7fce34070c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.190+0000 7fce3bc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340ab8c0 con 0x7fce340c9620 2024-01-24T17:44:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.190+0000 7fce3bc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce34071180 con 0x7fce34070860 2024-01-24T17:44:10.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.190+0000 7fce3bc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3415d150 con 0x7fce340ccd00 2024-01-24T17:44:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.190+0000 7fce39200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fce34070860 0x7fce340cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce39200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fce34070860 0x7fce340cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39136/0 (socket says 172.21.15.39:39136) 2024-01-24T17:44:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce39200640 1 -- 172.21.15.39:0/3173935777 learned_addr learned my addr 172.21.15.39:0/3173935777 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce39c00640 1 --1- 172.21.15.39:0/3173935777 >> v1:172.21.15.39:6789/0 conn(0x7fce340c9620 0x7fce340c9a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48192/0 (socket says 172.21.15.39:48192) 2024-01-24T17:44:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce33e00640 1 --2- 172.21.15.39:0/3173935777 >> v2:172.21.15.100:3300/0 conn(0x7fce340ccd00 0x7fce34070c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce39200640 1 -- 172.21.15.39:0/3173935777 >> v2:172.21.15.100:3300/0 conn(0x7fce340ccd00 msgr2=0x7fce34070c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:10.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce39200640 1 --2- 172.21.15.39:0/3173935777 >> v2:172.21.15.100:3300/0 conn(0x7fce340ccd00 0x7fce34070c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce39200640 1 -- 172.21.15.39:0/3173935777 >> v1:172.21.15.39:6789/0 conn(0x7fce340c9620 legacy=0x7fce340c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce39200640 1 -- 172.21.15.39:0/3173935777 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340cd470 con 0x7fce34070860 2024-01-24T17:44:10.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.191+0000 7fce33400640 1 -- 172.21.15.39:0/3173935777 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276812256 0 0) 0x7fce340ab8c0 con 0x7fce340c9620 2024-01-24T17:44:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.192+0000 7fce39200640 1 --2- 172.21.15.39:0/3173935777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce340cb1b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fce24009870 tx=0x7fce2402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cceb8019beca2f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce33400640 1 -- 172.21.15.39:0/3173935777 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce240317d0 con 0x7fce34070860 2024-01-24T17:44:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce33400640 1 -- 172.21.15.39:0/3173935777 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce24031930 con 0x7fce34070860 2024-01-24T17:44:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce33400640 1 -- 172.21.15.39:0/3173935777 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24031cc0 con 0x7fce34070860 2024-01-24T17:44:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce3bc00640 1 -- 172.21.15.39:0/3173935777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 msgr2=0x7fce340cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce3bc00640 1 --2- 172.21.15.39:0/3173935777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce340cb1b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fce24009870 tx=0x7fce2402b870 comp rx=0 tx=0).stop 2024-01-24T17:44:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce3bc00640 1 -- 172.21.15.39:0/3173935777 shutdown_connections 2024-01-24T17:44:10.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce3bc00640 1 --2- 172.21.15.39:0/3173935777 >> v2:172.21.15.100:3300/0 conn(0x7fce340ccd00 0x7fce34070c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce3bc00640 1 --2- 172.21.15.39:0/3173935777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce340cb1b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce3bc00640 1 -- 172.21.15.39:0/3173935777 >> 172.21.15.39:0/3173935777 conn(0x7fce340cb910 msgr2=0x7fce340cbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce3bc00640 1 -- 172.21.15.39:0/3173935777 shutdown_connections 2024-01-24T17:44:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.193+0000 7fce3bc00640 1 -- 172.21.15.39:0/3173935777 wait complete. 2024-01-24T17:44:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.195+0000 7fce3bc00640 1 Processor -- start 2024-01-24T17:44:10.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.195+0000 7fce3bc00640 1 -- start start 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce3bc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce3414b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce3bc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce340c9620 0x7fce3415dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce3bc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce340ccd00 0x7fce341f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce3bc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce341f3370 con 0x7fce34070860 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce3bc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3414b830 con 0x7fce340ccd00 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce3bc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce341f2db0 con 0x7fce340c9620 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce39200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce3414b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce39200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce3414b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39152/0 (socket says 172.21.15.39:39152) 2024-01-24T17:44:10.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.196+0000 7fce39200640 1 -- 172.21.15.39:0/2447298494 learned_addr learned my addr 172.21.15.39:0/2447298494 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:10.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.197+0000 7fce39c00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce340ccd00 0x7fce341f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.197+0000 7fce33e00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce340c9620 0x7fce3415dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.197+0000 7fce39200640 1 -- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce340c9620 msgr2=0x7fce3415dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.197+0000 7fce39200640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce340c9620 0x7fce3415dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.197+0000 7fce39200640 1 -- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce340ccd00 msgr2=0x7fce341f2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.197+0000 7fce39200640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce340ccd00 0x7fce341f2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.197+0000 7fce39200640 1 -- 172.21.15.39:0/2447298494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce240094d0 con 0x7fce34070860 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.197+0000 7fce39200640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce3414b2f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fce2402b8b0 tx=0x7fce240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.198+0000 7fce30c00640 1 -- 172.21.15.39:0/2447298494 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24032870 con 0x7fce34070860 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.198+0000 7fce30c00640 1 -- 172.21.15.39:0/2447298494 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce240329d0 con 0x7fce34070860 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.198+0000 7fce30c00640 1 -- 172.21.15.39:0/2447298494 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24032d80 con 0x7fce34070860 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.198+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce341f3090 con 0x7fce34070860 2024-01-24T17:44:10.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.198+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3415d770 con 0x7fce34070860 2024-01-24T17:44:10.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.199+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3415d2f0 con 0x7fce34070860 2024-01-24T17:44:10.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.200+0000 7fce30c00640 1 -- 172.21.15.39:0/2447298494 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fce24030070 con 0x7fce34070860 2024-01-24T17:44:10.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.201+0000 7fce30c00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce0c06a430 0x7fce0c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.201+0000 7fce30c00640 1 -- 172.21.15.39:0/2447298494 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce240aa840 con 0x7fce34070860 2024-01-24T17:44:10.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.201+0000 7fce33e00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce0c06a430 0x7fce0c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.203+0000 7fce33e00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce0c06a430 0x7fce0c06c8f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce1c006fd0 tx=0x7fce1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.205+0000 7fce30c00640 1 -- 172.21.15.39:0/2447298494 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fce3415d2f0 con 0x7fce34070860 2024-01-24T17:44:10.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.403+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce3415d2c0 con 0x7fce34070860 2024-01-24T17:44:10.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.406+0000 7fce30c00640 1 -- 172.21.15.39:0/2447298494 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16054 (secure 0 0 0) 0x7fce24075f00 con 0x7fce34070860 2024-01-24T17:44:10.407 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:10.407 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":47,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:09.960397+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:44:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce0c06a430 msgr2=0x7fce0c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce0c06a430 0x7fce0c06c8f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fce1c006fd0 tx=0x7fce1c008040 comp rx=0 tx=0).stop 2024-01-24T17:44:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 msgr2=0x7fce3414b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce3414b2f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fce2402b8b0 tx=0x7fce240046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 shutdown_connections 2024-01-24T17:44:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce340ccd00 0x7fce341f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce340c9620 0x7fce3415dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce0c06a430 0x7fce0c06c8f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 --2- 172.21.15.39:0/2447298494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce34070860 0x7fce3414b2f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 >> 172.21.15.39:0/2447298494 conn(0x7fce340cb910 msgr2=0x7fce340aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 shutdown_connections 2024-01-24T17:44:10.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.409+0000 7fce3bc00640 1 -- 172.21.15.39:0/2447298494 wait complete. 2024-01-24T17:44:10.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:44:10.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.648+0000 7f01b6e00640 1 Processor -- start 2024-01-24T17:44:10.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.648+0000 7f01b6e00640 1 -- start start 2024-01-24T17:44:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.650+0000 7f01b6e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f01b015cc90 0x7f01b00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.650+0000 7f01b6e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0161490 con 0x7f01b00cde00 2024-01-24T17:44:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.650+0000 7f01b6e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b00ce8b0 con 0x7f01b00cb2b0 2024-01-24T17:44:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.650+0000 7f01b6e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b00ceb80 con 0x7f01b015cc90 2024-01-24T17:44:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01b6400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f01b00cde00 0x7f01b0160d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48208/0 (socket says 172.21.15.39:48208) 2024-01-24T17:44:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01b6400640 1 -- 172.21.15.39:0/899640830 learned_addr learned my addr 172.21.15.39:0/899640830 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:10.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01b5000640 1 --2- 172.21.15.39:0/899640830 >> v2:172.21.15.100:3300/0 conn(0x7f01b015cc90 0x7f01b00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841606056 0 0) 0x7f01b0161490 con 0x7f01b00cde00 2024-01-24T17:44:10.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0190003520 con 0x7f01b00cde00 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3305489509 0 0) 0x7f0190003520 con 0x7f01b00cde00 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 >> v2:172.21.15.100:3300/0 conn(0x7f01b015cc90 msgr2=0x7f01b00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01afe00640 1 --2- 172.21.15.39:0/899640830 >> v2:172.21.15.100:3300/0 conn(0x7f01b015cc90 0x7f01b00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 >> v1:172.21.15.100:6789/0 conn(0x7f01b00cb2b0 legacy=0x7f01b00cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b00cee60 con 0x7f01b00cde00 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.651+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f01a80019f0 con 0x7f01b00cde00 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 >> v1:172.21.15.39:6789/0 conn(0x7f01b00cde00 legacy=0x7f01b0160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01afe00640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0190003970 0x7f0190003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01afe00640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01900047c0 0x7f0190004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01afe00640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0190005180 0x7f019000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b00ceb80 con 0x7f01900047c0 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019000c020 con 0x7f0190003970 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019000c1e0 con 0x7f0190005180 2024-01-24T17:44:10.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01b5000640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01900047c0 0x7f0190004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01b6400640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0190005180 0x7f019000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01b5a00640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0190003970 0x7f0190003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01b5000640 1 -- 172.21.15.39:0/899640830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0190005180 msgr2=0x7f019000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:10.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.652+0000 7f01b5000640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0190005180 0x7f019000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.653+0000 7f01b5000640 1 -- 172.21.15.39:0/899640830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0190003970 msgr2=0x7f0190003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:10.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.653+0000 7f01b5000640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0190003970 0x7f0190003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.653+0000 7f01b5000640 1 -- 172.21.15.39:0/899640830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190003520 con 0x7f01900047c0 2024-01-24T17:44:10.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.653+0000 7f01b6400640 1 -- 172.21.15.39:0/899640830 reap_dead start 2024-01-24T17:44:10.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.654+0000 7f01b5000640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01900047c0 0x7f0190004c40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0198009ac0 tx=0x7f01900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aeb0a918e5ae66a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:10.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.654+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019802c6d0 con 0x7f01900047c0 2024-01-24T17:44:10.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.654+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019802c830 con 0x7f01900047c0 2024-01-24T17:44:10.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.654+0000 7f01afe00640 1 -- 172.21.15.39:0/899640830 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019802cbc0 con 0x7f01900047c0 2024-01-24T17:44:10.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.655+0000 7f01b6e00640 1 -- 172.21.15.39:0/899640830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01900047c0 msgr2=0x7f0190004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.655+0000 7f01b6e00640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01900047c0 0x7f0190004c40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0198009ac0 tx=0x7f01900057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.655+0000 7f01b6e00640 1 -- 172.21.15.39:0/899640830 shutdown_connections 2024-01-24T17:44:10.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.655+0000 7f01b6e00640 1 --2- 172.21.15.39:0/899640830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01900047c0 0x7f0190004c40 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.655+0000 7f01b6e00640 1 -- 172.21.15.39:0/899640830 >> 172.21.15.39:0/899640830 conn(0x7f01b006a090 msgr2=0x7f01b006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:10.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.655+0000 7f01b6e00640 1 -- 172.21.15.39:0/899640830 shutdown_connections 2024-01-24T17:44:10.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.655+0000 7f01b6e00640 1 -- 172.21.15.39:0/899640830 wait complete. 2024-01-24T17:44:10.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.656+0000 7f01b6e00640 1 Processor -- start 2024-01-24T17:44:10.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.657+0000 7f01b6e00640 1 -- start start 2024-01-24T17:44:10.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.657+0000 7f01b6e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01900047c0 0x7f01b015da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.657+0000 7f01b6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b00c2400 0x7f01b0151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b01562c0 0x7f01b0156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b015e480 con 0x7f01b01562c0 2024-01-24T17:44:10.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0156c80 con 0x7f01900047c0 2024-01-24T17:44:10.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b00c7f10 con 0x7f01b00c2400 2024-01-24T17:44:10.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b01562c0 0x7f01b0156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b01562c0 0x7f01b0156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39174/0 (socket says 172.21.15.39:39174) 2024-01-24T17:44:10.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6400640 1 -- 172.21.15.39:0/938660623 learned_addr learned my addr 172.21.15.39:0/938660623 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b5000640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b00c2400 0x7f01b0151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b5a00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01900047c0 0x7f01b015da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6400640 1 -- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b00c2400 msgr2=0x7f01b0151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6400640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b00c2400 0x7f01b0151950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6400640 1 -- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01900047c0 msgr2=0x7f01b015da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6400640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01900047c0 0x7f01b015da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.658+0000 7f01b6400640 1 -- 172.21.15.39:0/938660623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b00c81f0 con 0x7f01b01562c0 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.659+0000 7f01b6400640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b01562c0 0x7f01b0156740 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f01b00c9df0 tx=0x7f01a8007af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.659+0000 7f01ad600640 1 -- 172.21.15.39:0/938660623 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a800ea40 con 0x7f01b01562c0 2024-01-24T17:44:10.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.659+0000 7f01ad600640 1 -- 172.21.15.39:0/938660623 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a800eba0 con 0x7f01b01562c0 2024-01-24T17:44:10.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.659+0000 7f01ad600640 1 -- 172.21.15.39:0/938660623 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a8013420 con 0x7f01b01562c0 2024-01-24T17:44:10.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.659+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b015e690 con 0x7f01b01562c0 2024-01-24T17:44:10.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.659+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b015df80 con 0x7f01b01562c0 2024-01-24T17:44:10.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.660+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b00c83e0 con 0x7f01b01562c0 2024-01-24T17:44:10.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.661+0000 7f01ad600640 1 -- 172.21.15.39:0/938660623 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f01a8013580 con 0x7f01b01562c0 2024-01-24T17:44:10.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.662+0000 7f01ad600640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018806a430 0x7f018806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:10.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.662+0000 7f01ad600640 1 -- 172.21.15.39:0/938660623 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01a8088ce0 con 0x7f01b01562c0 2024-01-24T17:44:10.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.663+0000 7f01b5a00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018806a430 0x7f018806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:10.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.664+0000 7f01b5a00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018806a430 0x7f018806c8f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f01a0007fd0 tx=0x7f01a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:10.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.666+0000 7f01ad600640 1 -- 172.21.15.39:0/938660623 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f01a8054230 con 0x7f01b01562c0 2024-01-24T17:44:10.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.864+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01b004dff0 con 0x7f01b01562c0 2024-01-24T17:44:10.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.865+0000 7f01ad600640 1 -- 172.21.15.39:0/938660623 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16054 (secure 0 0 0) 0x7f01a8057850 con 0x7f01b01562c0 2024-01-24T17:44:10.866 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:10.867 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":47,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:09.960397+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:44:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018806a430 msgr2=0x7f018806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018806a430 0x7f018806c8f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f01a0007fd0 tx=0x7f01a0009040 comp rx=0 tx=0).stop 2024-01-24T17:44:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b01562c0 msgr2=0x7f01b0156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b01562c0 0x7f01b0156740 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f01b00c9df0 tx=0x7f01a8007af0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 shutdown_connections 2024-01-24T17:44:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f018806a430 0x7f018806c8f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b01562c0 0x7f01b0156740 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f01b00c2400 0x7f01b0151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 --2- 172.21.15.39:0/938660623 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01900047c0 0x7f01b015da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:10.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.868+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 >> 172.21.15.39:0/938660623 conn(0x7f01b006a090 msgr2=0x7f01b00ad340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:10.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.869+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 shutdown_connections 2024-01-24T17:44:10.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:10.869+0000 7f01b6e00640 1 -- 172.21.15.39:0/938660623 wait complete. 2024-01-24T17:44:10.888 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T17:44:11.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.109+0000 7ff885000640 1 Processor -- start 2024-01-24T17:44:11.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.109+0000 7ff885000640 1 -- start start 2024-01-24T17:44:11.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.111+0000 7ff885000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff8800b16e0 0x7ff8800d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:11.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.111+0000 7ff885000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8800d37b0 con 0x7ff8800d1520 2024-01-24T17:44:11.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.111+0000 7ff885000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8800b2040 con 0x7ff88015cad0 2024-01-24T17:44:11.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.111+0000 7ff885000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8800cfe10 con 0x7ff8800b16e0 2024-01-24T17:44:11.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.111+0000 7ff877e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff8800d1520 0x7ff8800cdce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:48222/0 (socket says 172.21.15.39:48222) 2024-01-24T17:44:11.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.111+0000 7ff877e00640 1 -- 172.21.15.39:0/112160396 learned_addr learned my addr 172.21.15.39:0/112160396 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:11.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.111+0000 7ff87fe00640 1 --2- 172.21.15.39:0/112160396 >> v2:172.21.15.39:3300/0 conn(0x7ff8800b16e0 0x7ff8800d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:11.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.112+0000 7ff87ea00640 1 -- 172.21.15.39:0/112160396 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015169975 0 0) 0x7ff8800d37b0 con 0x7ff8800d1520 2024-01-24T17:44:11.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.112+0000 7ff87ea00640 1 -- 172.21.15.39:0/112160396 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff860003520 con 0x7ff8800d1520 2024-01-24T17:44:11.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.112+0000 7ff87fe00640 1 -- 172.21.15.39:0/112160396 >> v1:172.21.15.100:6789/0 conn(0x7ff88015cad0 legacy=0x7ff8800d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:11.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.112+0000 7ff87fe00640 1 -- 172.21.15.39:0/112160396 >> v1:172.21.15.39:6789/0 conn(0x7ff8800d1520 legacy=0x7ff8800cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:11.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.112+0000 7ff87fe00640 1 -- 172.21.15.39:0/112160396 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8800d34d0 con 0x7ff8800b16e0 2024-01-24T17:44:11.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.113+0000 7ff87fe00640 1 --2- 172.21.15.39:0/112160396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8800b16e0 0x7ff8800d0610 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff878009d90 tx=0x7ff87802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27c18d2d8968e63 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:11.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.114+0000 7ff87ea00640 1 -- 172.21.15.39:0/112160396 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87802cbd0 con 0x7ff8800b16e0 2024-01-24T17:44:11.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.114+0000 7ff87ea00640 1 -- 172.21.15.39:0/112160396 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87802cd30 con 0x7ff8800b16e0 2024-01-24T17:44:11.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.114+0000 7ff87ea00640 1 -- 172.21.15.39:0/112160396 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87802cea0 con 0x7ff8800b16e0 2024-01-24T17:44:11.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.114+0000 7ff885000640 1 -- 172.21.15.39:0/112160396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8800b16e0 msgr2=0x7ff8800d0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:11.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.114+0000 7ff885000640 1 --2- 172.21.15.39:0/112160396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8800b16e0 0x7ff8800d0610 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff878009d90 tx=0x7ff87802be20 comp rx=0 tx=0).stop 2024-01-24T17:44:11.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.114+0000 7ff885000640 1 -- 172.21.15.39:0/112160396 shutdown_connections 2024-01-24T17:44:11.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.114+0000 7ff885000640 1 --2- 172.21.15.39:0/112160396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8800b16e0 0x7ff8800d0610 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:11.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.114+0000 7ff885000640 1 -- 172.21.15.39:0/112160396 >> 172.21.15.39:0/112160396 conn(0x7ff8800d2830 msgr2=0x7ff8800d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:11.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.115+0000 7ff885000640 1 -- 172.21.15.39:0/112160396 shutdown_connections 2024-01-24T17:44:11.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.115+0000 7ff885000640 1 -- 172.21.15.39:0/112160396 wait complete. 2024-01-24T17:44:11.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.116+0000 7ff885000640 1 Processor -- start 2024-01-24T17:44:11.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.116+0000 7ff885000640 1 -- start start 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.117+0000 7ff885000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8800b16e0 0x7ff8800ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.117+0000 7ff885000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8800d1520 0x7ff8800cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.117+0000 7ff885000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88015cad0 0x7ff8800c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.117+0000 7ff885000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800c6f90 con 0x7ff88015cad0 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.117+0000 7ff885000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800cac70 con 0x7ff8800b16e0 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.117+0000 7ff885000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800ccbd0 con 0x7ff8800d1520 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88015cad0 0x7ff8800c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88015cad0 0x7ff8800c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39198/0 (socket says 172.21.15.39:39198) 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 -- 172.21.15.39:0/1338892891 learned_addr learned my addr 172.21.15.39:0/1338892891 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87f400640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8800b16e0 0x7ff8800ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:11.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff877e00640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8800d1520 0x7ff8800cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:11.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 -- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8800d1520 msgr2=0x7ff8800cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:11.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8800d1520 0x7ff8800cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:11.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 -- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8800b16e0 msgr2=0x7ff8800ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:11.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8800b16e0 0x7ff8800ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:11.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 -- 172.21.15.39:0/1338892891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff878009a70 con 0x7ff88015cad0 2024-01-24T17:44:11.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87f400640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8800b16e0 0x7ff8800ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:11.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff877e00640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8800d1520 0x7ff8800cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:11.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.118+0000 7ff87fe00640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88015cad0 0x7ff8800c8a70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff878009d90 tx=0x7ff878004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:11.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.119+0000 7ff877400640 1 -- 172.21.15.39:0/1338892891 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87802cea0 con 0x7ff88015cad0 2024-01-24T17:44:11.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.119+0000 7ff877400640 1 -- 172.21.15.39:0/1338892891 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff878040650 con 0x7ff88015cad0 2024-01-24T17:44:11.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.119+0000 7ff877400640 1 -- 172.21.15.39:0/1338892891 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878040a00 con 0x7ff88015cad0 2024-01-24T17:44:11.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.119+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8800c6310 con 0x7ff88015cad0 2024-01-24T17:44:11.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.119+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8800c68d0 con 0x7ff88015cad0 2024-01-24T17:44:11.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.120+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8800c70d0 con 0x7ff88015cad0 2024-01-24T17:44:11.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.121+0000 7ff877400640 1 -- 172.21.15.39:0/1338892891 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff878040b60 con 0x7ff88015cad0 2024-01-24T17:44:11.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.122+0000 7ff877400640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff85806a430 0x7ff85806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:11.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.122+0000 7ff877400640 1 -- 172.21.15.39:0/1338892891 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8780ab550 con 0x7ff88015cad0 2024-01-24T17:44:11.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.123+0000 7ff87f400640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff85806a430 0x7ff85806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:11.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.124+0000 7ff87f400640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff85806a430 0x7ff85806c8f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff870000fb0 tx=0x7ff870009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:11.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.127+0000 7ff877400640 1 -- 172.21.15.39:0/1338892891 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff8800c70d0 con 0x7ff88015cad0 2024-01-24T17:44:11.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.366+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff8800c70d0 con 0x7ff88015cad0 2024-01-24T17:44:11.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.987+0000 7ff877400640 1 -- 172.21.15.39:0/1338892891 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v22) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff8800c70d0 con 0x7ff88015cad0 2024-01-24T17:44:11.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff85806a430 msgr2=0x7ff85806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:11.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff85806a430 0x7ff85806c8f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff870000fb0 tx=0x7ff870009040 comp rx=0 tx=0).stop 2024-01-24T17:44:11.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88015cad0 msgr2=0x7ff8800c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:11.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88015cad0 0x7ff8800c8a70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff878009d90 tx=0x7ff878004a40 comp rx=0 tx=0).stop 2024-01-24T17:44:11.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 shutdown_connections 2024-01-24T17:44:11.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff85806a430 0x7ff85806c8f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:11.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff88015cad0 0x7ff8800c8a70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:11.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8800d1520 0x7ff8800cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:11.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 --2- 172.21.15.39:0/1338892891 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8800b16e0 0x7ff8800ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:11.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.991+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 >> 172.21.15.39:0/1338892891 conn(0x7ff8800d2830 msgr2=0x7ff88015bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:11.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.992+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 shutdown_connections 2024-01-24T17:44:11.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:11.992+0000 7ff885000640 1 -- 172.21.15.39:0/1338892891 wait complete. 2024-01-24T17:44:12.015 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-24T17:44:12.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-24T17:44:12.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.247+0000 7f175e000640 1 Processor -- start 2024-01-24T17:44:12.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.248+0000 7f175e000640 1 -- start start 2024-01-24T17:44:12.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.249+0000 7f175e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 0x7f175806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:12.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.249+0000 7f175e000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f17580d10b0 0x7f17580d2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:12.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.249+0000 7f175e000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175806c3f0 con 0x7f17580cf220 2024-01-24T17:44:12.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.249+0000 7f175e000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175806c6d0 con 0x7f17580d10b0 2024-01-24T17:44:12.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.249+0000 7f175e000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175806c9b0 con 0x7f17580d2630 2024-01-24T17:44:12.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.250+0000 7f1757e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f17580d10b0 0x7f17580d2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:12.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.250+0000 7f175cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 0x7f175806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:12.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.250+0000 7f1757e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f17580d10b0 0x7f17580d2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:39850/0 (socket says 172.21.15.39:39850) 2024-01-24T17:44:12.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.250+0000 7f1757e00640 1 -- 172.21.15.39:0/3788859812 learned_addr learned my addr 172.21.15.39:0/3788859812 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:12.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.251+0000 7f1757e00640 1 -- 172.21.15.39:0/3788859812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 msgr2=0x7f175806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:12.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.251+0000 7f1757e00640 1 --2- 172.21.15.39:0/3788859812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 0x7f175806bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:12.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.251+0000 7f1757e00640 1 -- 172.21.15.39:0/3788859812 >> v1:172.21.15.100:6789/0 conn(0x7f17580cf220 legacy=0x7f17580cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:12.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.251+0000 7f1757e00640 1 -- 172.21.15.39:0/3788859812 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175806cc90 con 0x7f17580d10b0 2024-01-24T17:44:12.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.253+0000 7f1757e00640 1 --2- 172.21.15.39:0/3788859812 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580d10b0 0x7f17580d2a10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1740009e50 tx=0x7f174002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb4a749f9d371817 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:12.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.253+0000 7f1757400640 1 -- 172.21.15.39:0/3788859812 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174002ca70 con 0x7f17580d10b0 2024-01-24T17:44:12.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.253+0000 7f1757400640 1 -- 172.21.15.39:0/3788859812 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174002cbd0 con 0x7f17580d10b0 2024-01-24T17:44:12.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.253+0000 7f1757400640 1 -- 172.21.15.39:0/3788859812 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17400343f0 con 0x7f17580d10b0 2024-01-24T17:44:12.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.254+0000 7f175e000640 1 -- 172.21.15.39:0/3788859812 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580d10b0 msgr2=0x7f17580d2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:12.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.254+0000 7f175e000640 1 --2- 172.21.15.39:0/3788859812 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580d10b0 0x7f17580d2a10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1740009e50 tx=0x7f174002bed0 comp rx=0 tx=0).stop 2024-01-24T17:44:12.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.254+0000 7f175e000640 1 -- 172.21.15.39:0/3788859812 shutdown_connections 2024-01-24T17:44:12.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.254+0000 7f175e000640 1 --2- 172.21.15.39:0/3788859812 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580d10b0 0x7f17580d2a10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:12.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.254+0000 7f175e000640 1 --2- 172.21.15.39:0/3788859812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 0x7f175806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:12.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.254+0000 7f175e000640 1 -- 172.21.15.39:0/3788859812 >> 172.21.15.39:0/3788859812 conn(0x7f175809bb90 msgr2=0x7f1758089fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:12.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.254+0000 7f175e000640 1 -- 172.21.15.39:0/3788859812 shutdown_connections 2024-01-24T17:44:12.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.254+0000 7f175e000640 1 -- 172.21.15.39:0/3788859812 wait complete. 2024-01-24T17:44:12.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.256+0000 7f175e000640 1 Processor -- start 2024-01-24T17:44:12.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.257+0000 7f175e000640 1 -- start start 2024-01-24T17:44:12.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f175e000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580cf220 0x7f1758153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:12.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f175e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17580d10b0 0x7f17580b7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:12.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f175e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 0x7f17580b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:12.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f175e000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17581f1e30 con 0x7f17580d10b0 2024-01-24T17:44:12.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f175e000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17580b81c0 con 0x7f17580cf220 2024-01-24T17:44:12.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f175e000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17581f2950 con 0x7f17580d2630 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f1757e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17580d10b0 0x7f17580b7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f1757e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17580d10b0 0x7f17580b7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39208/0 (socket says 172.21.15.39:39208) 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.258+0000 7f1757e00640 1 -- 172.21.15.39:0/3345460828 learned_addr learned my addr 172.21.15.39:0/3345460828 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.259+0000 7f175cc00640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580cf220 0x7f1758153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.259+0000 7f175d600640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 0x7f17580b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.259+0000 7f1757e00640 1 -- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 msgr2=0x7f17580b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.259+0000 7f1757e00640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 0x7f17580b5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.259+0000 7f1757e00640 1 -- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580cf220 msgr2=0x7f1758153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.259+0000 7f1757e00640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580cf220 0x7f1758153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.259+0000 7f1757e00640 1 -- 172.21.15.39:0/3345460828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1740009b30 con 0x7f17580d10b0 2024-01-24T17:44:12.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.259+0000 7f175cc00640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580cf220 0x7f1758153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:12.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.260+0000 7f1757e00640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17580d10b0 0x7f17580b7900 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1740036040 tx=0x7f1740004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:12.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.260+0000 7f1754c00640 1 -- 172.21.15.39:0/3345460828 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1740009510 con 0x7f17580d10b0 2024-01-24T17:44:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.260+0000 7f1754c00640 1 -- 172.21.15.39:0/3345460828 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174003e450 con 0x7f17580d10b0 2024-01-24T17:44:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.260+0000 7f1754c00640 1 -- 172.21.15.39:0/3345460828 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174003e800 con 0x7f17580d10b0 2024-01-24T17:44:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.260+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17581f2020 con 0x7f17580d10b0 2024-01-24T17:44:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.260+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17580b7e40 con 0x7f17580d10b0 2024-01-24T17:44:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.261+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1758067040 con 0x7f17580d10b0 2024-01-24T17:44:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.262+0000 7f1754c00640 1 -- 172.21.15.39:0/3345460828 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f174003e960 con 0x7f17580d10b0 2024-01-24T17:44:12.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.263+0000 7f1754c00640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f173006a430 0x7f173006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:12.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.263+0000 7f1754c00640 1 -- 172.21.15.39:0/3345460828 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17400ab6e0 con 0x7f17580d10b0 2024-01-24T17:44:12.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.264+0000 7f175cc00640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f173006a430 0x7f173006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.265+0000 7f175cc00640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f173006a430 0x7f173006c8f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1748009af0 tx=0x7f1748008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.268+0000 7f1754c00640 1 -- 172.21.15.39:0/3345460828 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1740076c40 con 0x7f17580d10b0 2024-01-24T17:44:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:12.498+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f17581f2a90 con 0x7f17580d10b0 2024-01-24T17:44:13.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.001+0000 7f1754c00640 1 -- 172.21.15.39:0/3345460828 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7f174007a260 con 0x7f17580d10b0 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f173006a430 msgr2=0x7f173006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f173006a430 0x7f173006c8f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1748009af0 tx=0x7f1748008040 comp rx=0 tx=0).stop 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17580d10b0 msgr2=0x7f17580b7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17580d10b0 0x7f17580b7900 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1740036040 tx=0x7f1740004a40 comp rx=0 tx=0).stop 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 shutdown_connections 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f17580d2630 0x7f17580b5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f173006a430 0x7f173006c8f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17580d10b0 0x7f17580b7900 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 --2- 172.21.15.39:0/3345460828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f17580cf220 0x7f1758153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 >> 172.21.15.39:0/3345460828 conn(0x7f175809bb90 msgr2=0x7f17580b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:13.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 shutdown_connections 2024-01-24T17:44:13.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.006+0000 7f175e000640 1 -- 172.21.15.39:0/3345460828 wait complete. 2024-01-24T17:44:13.034 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-24T17:44:13.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:44:13.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.262+0000 7fd23f000640 1 Processor -- start 2024-01-24T17:44:13.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.263+0000 7fd23f000640 1 -- start start 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.264+0000 7fd23f000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd23815a950 0x7fd2380ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.264+0000 7fd23f000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2380cb660 0x7fd2380b1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.264+0000 7fd23f000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2380c9180 con 0x7fd2380caaa0 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.264+0000 7fd23f000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2380b43d0 con 0x7fd2380cb660 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.264+0000 7fd23f000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380c8f70 con 0x7fd23815a950 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd23d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2380cb660 0x7fd2380b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd23d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2380cb660 0x7fd2380b1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38886/0 (socket says 172.21.15.39:38886) 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd23d000640 1 -- 172.21.15.39:0/4170610193 learned_addr learned my addr 172.21.15.39:0/4170610193 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd22f200640 1 --1- 172.21.15.39:0/4170610193 >> v1:172.21.15.39:6789/0 conn(0x7fd2380caaa0 0x7fd2380caf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:57362/0 (socket says 172.21.15.39:57362) 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd237e00640 1 --2- 172.21.15.39:0/4170610193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd23815a950 0x7fd2380ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd23d000640 1 -- 172.21.15.39:0/4170610193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd23815a950 msgr2=0x7fd2380ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:13.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd23d000640 1 --2- 172.21.15.39:0/4170610193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd23815a950 0x7fd2380ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd23d000640 1 -- 172.21.15.39:0/4170610193 >> v1:172.21.15.39:6789/0 conn(0x7fd2380caaa0 legacy=0x7fd2380caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.265+0000 7fd23d000640 1 -- 172.21.15.39:0/4170610193 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380c9460 con 0x7fd2380cb660 2024-01-24T17:44:13.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.266+0000 7fd23d000640 1 --2- 172.21.15.39:0/4170610193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2380cb660 0x7fd2380b1390 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd230009870 tx=0x7fd23002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ab0e006d9da90a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:13.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd237400640 1 -- 172.21.15.39:0/4170610193 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230030840 con 0x7fd2380cb660 2024-01-24T17:44:13.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd237400640 1 -- 172.21.15.39:0/4170610193 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2300309a0 con 0x7fd2380cb660 2024-01-24T17:44:13.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd237400640 1 -- 172.21.15.39:0/4170610193 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230030d30 con 0x7fd2380cb660 2024-01-24T17:44:13.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd23f000640 1 -- 172.21.15.39:0/4170610193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2380cb660 msgr2=0x7fd2380b1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd23f000640 1 --2- 172.21.15.39:0/4170610193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2380cb660 0x7fd2380b1390 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd230009870 tx=0x7fd23002b980 comp rx=0 tx=0).stop 2024-01-24T17:44:13.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd23f000640 1 -- 172.21.15.39:0/4170610193 shutdown_connections 2024-01-24T17:44:13.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd23f000640 1 --2- 172.21.15.39:0/4170610193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2380cb660 0x7fd2380b1390 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd23f000640 1 --2- 172.21.15.39:0/4170610193 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd23815a950 0x7fd2380ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd23f000640 1 -- 172.21.15.39:0/4170610193 >> 172.21.15.39:0/4170610193 conn(0x7fd23806a3e0 msgr2=0x7fd23806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:13.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd23f000640 1 -- 172.21.15.39:0/4170610193 shutdown_connections 2024-01-24T17:44:13.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.267+0000 7fd23f000640 1 -- 172.21.15.39:0/4170610193 wait complete. 2024-01-24T17:44:13.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.269+0000 7fd23f000640 1 Processor -- start 2024-01-24T17:44:13.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.269+0000 7fd23f000640 1 -- start start 2024-01-24T17:44:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.270+0000 7fd23f000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2380caaa0 0x7fd238158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.270+0000 7fd23f000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2380cb660 0x7fd2380abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.270+0000 7fd23f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23815a950 0x7fd23815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.270+0000 7fd23f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23815ad30 con 0x7fd23815a950 2024-01-24T17:44:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.270+0000 7fd23f000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380a74b0 con 0x7fd2380cb660 2024-01-24T17:44:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.270+0000 7fd23f000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380a7790 con 0x7fd2380caaa0 2024-01-24T17:44:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23815a950 0x7fd23815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23815a950 0x7fd23815d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38894/0 (socket says 172.21.15.39:38894) 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 -- 172.21.15.39:0/2882819541 learned_addr learned my addr 172.21.15.39:0/2882819541 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd237e00640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2380caaa0 0x7fd238158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd22f200640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2380cb660 0x7fd2380abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 -- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2380caaa0 msgr2=0x7fd238158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2380caaa0 0x7fd238158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 -- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2380cb660 msgr2=0x7fd2380abf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2380cb660 0x7fd2380abf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 -- 172.21.15.39:0/2882819541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300094d0 con 0x7fd23815a950 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd22f200640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2380cb660 0x7fd2380abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.271+0000 7fd23d000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23815a950 0x7fd23815d3a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd230009870 tx=0x7fd230004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.272+0000 7fd234c00640 1 -- 172.21.15.39:0/2882819541 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230039c90 con 0x7fd23815a950 2024-01-24T17:44:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.272+0000 7fd234c00640 1 -- 172.21.15.39:0/2882819541 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd230039e10 con 0x7fd23815a950 2024-01-24T17:44:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.272+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2380a9480 con 0x7fd23815a950 2024-01-24T17:44:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.272+0000 7fd234c00640 1 -- 172.21.15.39:0/2882819541 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2300446c0 con 0x7fd23815a950 2024-01-24T17:44:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.272+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2380a9a40 con 0x7fd23815a950 2024-01-24T17:44:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.273+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd238073ef0 con 0x7fd23815a950 2024-01-24T17:44:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.274+0000 7fd234c00640 1 -- 172.21.15.39:0/2882819541 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd23002f390 con 0x7fd23815a950 2024-01-24T17:44:13.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.275+0000 7fd234c00640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21006a430 0x7fd21006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.276+0000 7fd234c00640 1 -- 172.21.15.39:0/2882819541 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd23003e070 con 0x7fd23815a950 2024-01-24T17:44:13.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.276+0000 7fd237e00640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21006a430 0x7fd21006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.277+0000 7fd237e00640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21006a430 0x7fd21006c8f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd2380b3150 tx=0x7fd2280039b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:13.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.280+0000 7fd234c00640 1 -- 172.21.15.39:0/2882819541 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd230076710 con 0x7fd23815a950 2024-01-24T17:44:13.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.513+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2380a9c30 con 0x7fd23815a950 2024-01-24T17:44:13.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.515+0000 7fd234c00640 1 -- 172.21.15.39:0/2882819541 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fd230079d30 con 0x7fd23815a950 2024-01-24T17:44:13.516 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:13.516 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5763,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1418568795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1418568795},{"type":"v1","addr":"172.21.15.100:6837","nonce":1418568795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:44:09.961260+0000","modified":"2024-01-24T17:44:12.985097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/1882733155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1882733155},{"type":"v1","addr":"172.21.15.39:6835","nonce":1882733155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-24T17:44:13.516 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 23 2024-01-24T17:44:13.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.517+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21006a430 msgr2=0x7fd21006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.517+0000 7fd23f000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21006a430 0x7fd21006c8f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd2380b3150 tx=0x7fd2280039b0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.517+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23815a950 msgr2=0x7fd23815d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.517+0000 7fd23f000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23815a950 0x7fd23815d3a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd230009870 tx=0x7fd230004730 comp rx=0 tx=0).stop 2024-01-24T17:44:13.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.518+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 shutdown_connections 2024-01-24T17:44:13.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.518+0000 7fd23f000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21006a430 0x7fd21006c8f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.518+0000 7fd23f000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd23815a950 0x7fd23815d3a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.518+0000 7fd23f000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2380cb660 0x7fd2380abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.518+0000 7fd23f000640 1 --2- 172.21.15.39:0/2882819541 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2380caaa0 0x7fd238158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.518+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 >> 172.21.15.39:0/2882819541 conn(0x7fd23806a3e0 msgr2=0x7fd2380af260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:13.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.518+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 shutdown_connections 2024-01-24T17:44:13.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.518+0000 7fd23f000640 1 -- 172.21.15.39:0/2882819541 wait complete. 2024-01-24T17:44:13.539 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:44:13.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.756+0000 7f63e8e00640 1 Processor -- start 2024-01-24T17:44:13.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.756+0000 7f63e8e00640 1 -- start start 2024-01-24T17:44:13.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.758+0000 7f63e8e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f63e415a950 0x7f63e40ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.758+0000 7f63e8e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e40c9180 con 0x7f63e40cb660 2024-01-24T17:44:13.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.758+0000 7f63e8e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e40c9460 con 0x7f63e40caaa0 2024-01-24T17:44:13.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.758+0000 7f63e8e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e40b43d0 con 0x7f63e415a950 2024-01-24T17:44:13.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.758+0000 7f63e2800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f63e40cb660 0x7f63e40b1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:57376/0 (socket says 172.21.15.39:57376) 2024-01-24T17:44:13.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.758+0000 7f63e2800640 1 -- 172.21.15.39:0/965651246 learned_addr learned my addr 172.21.15.39:0/965651246 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:13.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.759+0000 7f63e1e00640 1 --2- 172.21.15.39:0/965651246 >> v2:172.21.15.100:3300/0 conn(0x7f63e415a950 0x7f63e40ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.759+0000 7f63e1400640 1 -- 172.21.15.39:0/965651246 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853234081 0 0) 0x7f63e40c9180 con 0x7f63e40cb660 2024-01-24T17:44:13.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.759+0000 7f63e1400640 1 -- 172.21.15.39:0/965651246 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c4003520 con 0x7f63e40cb660 2024-01-24T17:44:13.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.759+0000 7f63e1e00640 1 -- 172.21.15.39:0/965651246 >> v1:172.21.15.100:6789/0 conn(0x7f63e40caaa0 legacy=0x7f63e40caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.759+0000 7f63e1e00640 1 -- 172.21.15.39:0/965651246 >> v1:172.21.15.39:6789/0 conn(0x7f63e40cb660 legacy=0x7f63e40b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.759+0000 7f63e1e00640 1 -- 172.21.15.39:0/965651246 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e40c9740 con 0x7f63e415a950 2024-01-24T17:44:13.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.761+0000 7f63e1e00640 1 --2- 172.21.15.39:0/965651246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 0x7f63e40ca560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f63d40097a0 tx=0x7f63d402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49d9a1db36bd2ca7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:13.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.761+0000 7f63e1400640 1 -- 172.21.15.39:0/965651246 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4030610 con 0x7f63e415a950 2024-01-24T17:44:13.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e1400640 1 -- 172.21.15.39:0/965651246 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d4030770 con 0x7f63e415a950 2024-01-24T17:44:13.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e1400640 1 -- 172.21.15.39:0/965651246 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4030b00 con 0x7f63e415a950 2024-01-24T17:44:13.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e8e00640 1 -- 172.21.15.39:0/965651246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 msgr2=0x7f63e40ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e8e00640 1 --2- 172.21.15.39:0/965651246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 0x7f63e40ca560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f63d40097a0 tx=0x7f63d402b960 comp rx=0 tx=0).stop 2024-01-24T17:44:13.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e8e00640 1 -- 172.21.15.39:0/965651246 shutdown_connections 2024-01-24T17:44:13.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e8e00640 1 --2- 172.21.15.39:0/965651246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 0x7f63e40ca560 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e8e00640 1 -- 172.21.15.39:0/965651246 >> 172.21.15.39:0/965651246 conn(0x7f63e406a3e0 msgr2=0x7f63e406bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:13.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e8e00640 1 -- 172.21.15.39:0/965651246 shutdown_connections 2024-01-24T17:44:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.762+0000 7f63e8e00640 1 -- 172.21.15.39:0/965651246 wait complete. 2024-01-24T17:44:13.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.764+0000 7f63e8e00640 1 Processor -- start 2024-01-24T17:44:13.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.764+0000 7f63e8e00640 1 -- start start 2024-01-24T17:44:13.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.765+0000 7f63e8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e40caaa0 0x7f63e41f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.765+0000 7f63e8e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e40cb660 0x7f63e40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.765+0000 7f63e8e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 0x7f63e40b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.765+0000 7f63e8e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e41593c0 con 0x7f63e40caaa0 2024-01-24T17:44:13.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.765+0000 7f63e8e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40adea0 con 0x7f63e415a950 2024-01-24T17:44:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.765+0000 7f63e8e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40ae0f0 con 0x7f63e40cb660 2024-01-24T17:44:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.765+0000 7f63e1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e40caaa0 0x7f63e41f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.765+0000 7f63e1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e40caaa0 0x7f63e41f2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38902/0 (socket says 172.21.15.39:38902) 2024-01-24T17:44:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63e1e00640 1 -- 172.21.15.39:0/608577080 learned_addr learned my addr 172.21.15.39:0/608577080 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63e2800640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 0x7f63e40b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63d9200640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e40cb660 0x7f63e40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63e1e00640 1 -- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e40cb660 msgr2=0x7f63e40b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63e1e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e40cb660 0x7f63e40b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63e1e00640 1 -- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 msgr2=0x7f63e40b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63e1e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 0x7f63e40b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63e1e00640 1 -- 172.21.15.39:0/608577080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d4009400 con 0x7f63e40caaa0 2024-01-24T17:44:13.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.766+0000 7f63e2800640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 0x7f63e40b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.767+0000 7f63e1e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e40caaa0 0x7f63e41f2fc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f63d402bf70 tx=0x7f63d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.767+0000 7f63daa00640 1 -- 172.21.15.39:0/608577080 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d403e070 con 0x7f63e40caaa0 2024-01-24T17:44:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.767+0000 7f63daa00640 1 -- 172.21.15.39:0/608577080 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d4030e10 con 0x7f63e40caaa0 2024-01-24T17:44:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.767+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e40ae3d0 con 0x7f63e40caaa0 2024-01-24T17:44:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.767+0000 7f63daa00640 1 -- 172.21.15.39:0/608577080 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4039c30 con 0x7f63e40caaa0 2024-01-24T17:44:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.767+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e40ae990 con 0x7f63e40caaa0 2024-01-24T17:44:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.768+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e4073ef0 con 0x7f63e40caaa0 2024-01-24T17:44:13.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.769+0000 7f63daa00640 1 -- 172.21.15.39:0/608577080 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f63d404b400 con 0x7f63e40caaa0 2024-01-24T17:44:13.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.770+0000 7f63daa00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63bc06a430 0x7f63bc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:13.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.771+0000 7f63daa00640 1 -- 172.21.15.39:0/608577080 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63d40aad30 con 0x7f63e40caaa0 2024-01-24T17:44:13.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.771+0000 7f63d9200640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63bc06a430 0x7f63bc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:13.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.772+0000 7f63d9200640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63bc06a430 0x7f63bc06c8f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f63cc000f60 tx=0x7f63cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:13.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.775+0000 7f63daa00640 1 -- 172.21.15.39:0/608577080 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f63d4076290 con 0x7f63e40caaa0 2024-01-24T17:44:13.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.968+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63e40c9a20 con 0x7f63e40caaa0 2024-01-24T17:44:13.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.971+0000 7f63daa00640 1 -- 172.21.15.39:0/608577080 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16041 (secure 0 0 0) 0x7f63d40798b0 con 0x7f63e40caaa0 2024-01-24T17:44:13.973 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:13.973 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":48,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:10.966611+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:44:13.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63bc06a430 msgr2=0x7f63bc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63bc06a430 0x7f63bc06c8f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f63cc000f60 tx=0x7f63cc009040 comp rx=0 tx=0).stop 2024-01-24T17:44:13.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e40caaa0 msgr2=0x7f63e41f2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:13.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e40caaa0 0x7f63e41f2fc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f63d402bf70 tx=0x7f63d40046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 shutdown_connections 2024-01-24T17:44:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e415a950 0x7f63e40b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e40cb660 0x7f63e40b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63bc06a430 0x7f63bc06c8f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 --2- 172.21.15.39:0/608577080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e40caaa0 0x7f63e41f2fc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 >> 172.21.15.39:0/608577080 conn(0x7f63e406a3e0 msgr2=0x7f63e4070550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.974+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 shutdown_connections 2024-01-24T17:44:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:13.975+0000 7f63e8e00640 1 -- 172.21.15.39:0/608577080 wait complete. 2024-01-24T17:44:13.996 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-24T17:44:13.996 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:44:14.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.219+0000 7ff53aa00640 1 Processor -- start 2024-01-24T17:44:14.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.219+0000 7ff53aa00640 1 -- start start 2024-01-24T17:44:14.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff53aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff53415ab00 0x7ff5340cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff53aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5340ccd20 0x7ff5340ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff53aa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5340ce400 con 0x7ff53415a720 2024-01-24T17:44:14.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff53aa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53415b3b0 con 0x7ff53415ab00 2024-01-24T17:44:14.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff53aa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53415b690 con 0x7ff5340ccd20 2024-01-24T17:44:14.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff538c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff53415ab00 0x7ff5340cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff538c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff53415ab00 0x7ff5340cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38912/0 (socket says 172.21.15.39:38912) 2024-01-24T17:44:14.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff538c00640 1 -- 172.21.15.39:0/1175807798 learned_addr learned my addr 172.21.15.39:0/1175807798 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:14.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff539600640 1 --1- 172.21.15.39:0/1175807798 >> v1:172.21.15.39:6789/0 conn(0x7ff53415a720 0x7ff5340cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:57382/0 (socket says 172.21.15.39:57382) 2024-01-24T17:44:14.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff538c00640 1 -- 172.21.15.39:0/1175807798 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5340ccd20 msgr2=0x7ff5340ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:14.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff538c00640 1 --2- 172.21.15.39:0/1175807798 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5340ccd20 0x7ff5340ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff538c00640 1 -- 172.21.15.39:0/1175807798 >> v1:172.21.15.39:6789/0 conn(0x7ff53415a720 legacy=0x7ff5340cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.221+0000 7ff538c00640 1 -- 172.21.15.39:0/1175807798 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53415b970 con 0x7ff53415ab00 2024-01-24T17:44:14.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.223+0000 7ff538c00640 1 --2- 172.21.15.39:0/1175807798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340cc7e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009870 tx=0x7ff51c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d87c5bce375da1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:14.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.225+0000 7ff533e00640 1 -- 172.21.15.39:0/1175807798 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c031640 con 0x7ff53415ab00 2024-01-24T17:44:14.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.225+0000 7ff533e00640 1 -- 172.21.15.39:0/1175807798 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51c0317a0 con 0x7ff53415ab00 2024-01-24T17:44:14.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.225+0000 7ff533e00640 1 -- 172.21.15.39:0/1175807798 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c031b30 con 0x7ff53415ab00 2024-01-24T17:44:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.226+0000 7ff53aa00640 1 -- 172.21.15.39:0/1175807798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 msgr2=0x7ff5340cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.226+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1175807798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340cc7e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009870 tx=0x7ff51c02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.226+0000 7ff53aa00640 1 -- 172.21.15.39:0/1175807798 shutdown_connections 2024-01-24T17:44:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.226+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1175807798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340cc7e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.226+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1175807798 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff5340ccd20 0x7ff5340ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.226+0000 7ff53aa00640 1 -- 172.21.15.39:0/1175807798 >> 172.21.15.39:0/1175807798 conn(0x7ff53406a090 msgr2=0x7ff53406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.226+0000 7ff53aa00640 1 -- 172.21.15.39:0/1175807798 shutdown_connections 2024-01-24T17:44:14.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.226+0000 7ff53aa00640 1 -- 172.21.15.39:0/1175807798 wait complete. 2024-01-24T17:44:14.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.229+0000 7ff53aa00640 1 Processor -- start 2024-01-24T17:44:14.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.229+0000 7ff53aa00640 1 -- start start 2024-01-24T17:44:14.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff53aa00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff53415a720 0x7ff5340c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff53aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff53aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff534151650 0x7ff534151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff53aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5340c6ef0 con 0x7ff53415ab00 2024-01-24T17:44:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff53aa00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5340c71d0 con 0x7ff53415a720 2024-01-24T17:44:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff53aa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5340c74b0 con 0x7ff534151650 2024-01-24T17:44:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff538c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff538c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340c3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38920/0 (socket says 172.21.15.39:38920) 2024-01-24T17:44:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff538c00640 1 -- 172.21.15.39:0/1054778103 learned_addr learned my addr 172.21.15.39:0/1054778103 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff539600640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff53415a720 0x7ff5340c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff53a000640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff534151650 0x7ff534151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff538c00640 1 -- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff534151650 msgr2=0x7ff534151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff538c00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff534151650 0x7ff534151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff538c00640 1 -- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff53415a720 msgr2=0x7ff5340c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff538c00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff53415a720 0x7ff5340c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff538c00640 1 -- 172.21.15.39:0/1054778103 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c0094d0 con 0x7ff53415ab00 2024-01-24T17:44:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff539600640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff53415a720 0x7ff5340c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.230+0000 7ff53a000640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff534151650 0x7ff534151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.231+0000 7ff538c00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340c3c20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009870 tx=0x7ff51c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.231+0000 7ff531600640 1 -- 172.21.15.39:0/1054778103 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c03ac60 con 0x7ff53415ab00 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.231+0000 7ff531600640 1 -- 172.21.15.39:0/1054778103 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51c03ade0 con 0x7ff53415ab00 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.231+0000 7ff531600640 1 -- 172.21.15.39:0/1054778103 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c044630 con 0x7ff53415ab00 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.231+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5340c7790 con 0x7ff53415ab00 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.231+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff534152020 con 0x7ff53415ab00 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.233+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5340c7cb0 con 0x7ff53415ab00 2024-01-24T17:44:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.234+0000 7ff531600640 1 -- 172.21.15.39:0/1054778103 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff51c02f050 con 0x7ff53415ab00 2024-01-24T17:44:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.234+0000 7ff531600640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff50c06a460 0x7ff50c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.235+0000 7ff531600640 1 -- 172.21.15.39:0/1054778103 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff51c0aaa90 con 0x7ff53415ab00 2024-01-24T17:44:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.235+0000 7ff539600640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff50c06a460 0x7ff50c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.236+0000 7ff539600640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff50c06a460 0x7ff50c06c920 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff524000f60 tx=0x7ff524009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.239+0000 7ff531600640 1 -- 172.21.15.39:0/1054778103 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff5340c7cb0 con 0x7ff53415ab00 2024-01-24T17:44:14.429 INFO:tasks.ceph.mds.c.smithi039.stderr:2024-01-24T17:44:14.429+0000 7f1141400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:44:14.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.430+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5340c7340 con 0x7ff50c06a460 2024-01-24T17:44:14.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.455+0000 7ff531600640 1 -- 172.21.15.39:0/1054778103 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287275 (secure 0 0 0) 0x7ff5340c7340 con 0x7ff50c06a460 2024-01-24T17:44:14.461 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:14.463 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff50c06a460 msgr2=0x7ff50c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff50c06a460 0x7ff50c06c920 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff524000f60 tx=0x7ff524009040 comp rx=0 tx=0).stop 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 msgr2=0x7ff5340c3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340c3c20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009870 tx=0x7ff51c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 shutdown_connections 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff534151650 0x7ff534151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff50c06a460 0x7ff50c06c920 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff53415ab00 0x7ff5340c3c20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 --2- 172.21.15.39:0/1054778103 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff53415a720 0x7ff5340c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.462+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 >> 172.21.15.39:0/1054778103 conn(0x7ff53406a090 msgr2=0x7ff53415bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.463+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 shutdown_connections 2024-01-24T17:44:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.463+0000 7ff53aa00640 1 -- 172.21.15.39:0/1054778103 wait complete. 2024-01-24T17:44:14.486 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-01-24T17:44:14.267278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238188,"kb_used_data":5640,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754736532,"statfs":{"total":773094113280,"available":772850208768,"internally_reserved":0,"allocated":5775360,"data_stored":3044772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.803410"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147607+0000","last_change":"2024-01-24T17:44:09.988313+0000","last_active":"2024-01-24T17:44:11.147607+0000","last_peered":"2024-01-24T17:44:11.147607+0000","last_clean":"2024-01-24T17:44:11.147607+0000","last_became_active":"2024-01-24T17:44:09.987962+0000","last_became_peered":"2024-01-24T17:44:09.987962+0000","last_unstale":"2024-01-24T17:44:11.147607+0000","last_undegraded":"2024-01-24T17:44:11.147607+0000","last_fullsized":"2024-01-24T17:44:11.147607+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:34:38.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149565+0000","last_change":"2024-01-24T17:44:10.012031+0000","last_active":"2024-01-24T17:44:11.149565+0000","last_peered":"2024-01-24T17:44:11.149565+0000","last_clean":"2024-01-24T17:44:11.149565+0000","last_became_active":"2024-01-24T17:44:10.011640+0000","last_became_peered":"2024-01-24T17:44:10.011640+0000","last_unstale":"2024-01-24T17:44:11.149565+0000","last_undegraded":"2024-01-24T17:44:11.149565+0000","last_fullsized":"2024-01-24T17:44:11.149565+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:48:22.378247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149164+0000","last_change":"2024-01-24T17:44:10.007028+0000","last_active":"2024-01-24T17:44:11.149164+0000","last_peered":"2024-01-24T17:44:11.149164+0000","last_clean":"2024-01-24T17:44:11.149164+0000","last_became_active":"2024-01-24T17:44:10.006334+0000","last_became_peered":"2024-01-24T17:44:10.006334+0000","last_unstale":"2024-01-24T17:44:11.149164+0000","last_undegraded":"2024-01-24T17:44:11.149164+0000","last_fullsized":"2024-01-24T17:44:11.149164+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:36.242202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989015+0000","last_change":"2024-01-24T17:44:10.015889+0000","last_active":"2024-01-24T17:44:10.989015+0000","last_peered":"2024-01-24T17:44:10.989015+0000","last_clean":"2024-01-24T17:44:10.989015+0000","last_became_active":"2024-01-24T17:44:10.015499+0000","last_became_peered":"2024-01-24T17:44:10.015499+0000","last_unstale":"2024-01-24T17:44:10.989015+0000","last_undegraded":"2024-01-24T17:44:10.989015+0000","last_fullsized":"2024-01-24T17:44:10.989015+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:38:11.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147670+0000","last_change":"2024-01-24T17:44:10.001242+0000","last_active":"2024-01-24T17:44:11.147670+0000","last_peered":"2024-01-24T17:44:11.147670+0000","last_clean":"2024-01-24T17:44:11.147670+0000","last_became_active":"2024-01-24T17:44:10.000982+0000","last_became_peered":"2024-01-24T17:44:10.000982+0000","last_unstale":"2024-01-24T17:44:11.147670+0000","last_undegraded":"2024-01-24T17:44:11.147670+0000","last_fullsized":"2024-01-24T17:44:11.147670+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:57:21.200159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985302+0000","last_change":"2024-01-24T17:44:10.015448+0000","last_active":"2024-01-24T17:44:10.985302+0000","last_peered":"2024-01-24T17:44:10.985302+0000","last_clean":"2024-01-24T17:44:10.985302+0000","last_became_active":"2024-01-24T17:44:10.015153+0000","last_became_peered":"2024-01-24T17:44:10.015153+0000","last_unstale":"2024-01-24T17:44:10.985302+0000","last_undegraded":"2024-01-24T17:44:10.985302+0000","last_fullsized":"2024-01-24T17:44:10.985302+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:53.577142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984226+0000","last_change":"2024-01-24T17:44:10.015149+0000","last_active":"2024-01-24T17:44:10.984226+0000","last_peered":"2024-01-24T17:44:10.984226+0000","last_clean":"2024-01-24T17:44:10.984226+0000","last_became_active":"2024-01-24T17:44:10.014886+0000","last_became_peered":"2024-01-24T17:44:10.014886+0000","last_unstale":"2024-01-24T17:44:10.984226+0000","last_undegraded":"2024-01-24T17:44:10.984226+0000","last_fullsized":"2024-01-24T17:44:10.984226+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:36:07.346981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988400+0000","last_change":"2024-01-24T17:44:10.012722+0000","last_active":"2024-01-24T17:44:10.988400+0000","last_peered":"2024-01-24T17:44:10.988400+0000","last_clean":"2024-01-24T17:44:10.988400+0000","last_became_active":"2024-01-24T17:44:10.012384+0000","last_became_peered":"2024-01-24T17:44:10.012384+0000","last_unstale":"2024-01-24T17:44:10.988400+0000","last_undegraded":"2024-01-24T17:44:10.988400+0000","last_fullsized":"2024-01-24T17:44:10.988400+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:36:46.964115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.986259+0000","last_change":"2024-01-24T17:44:10.010919+0000","last_active":"2024-01-24T17:44:10.986259+0000","last_peered":"2024-01-24T17:44:10.986259+0000","last_clean":"2024-01-24T17:44:10.986259+0000","last_became_active":"2024-01-24T17:44:10.010469+0000","last_became_peered":"2024-01-24T17:44:10.010469+0000","last_unstale":"2024-01-24T17:44:10.986259+0000","last_undegraded":"2024-01-24T17:44:10.986259+0000","last_fullsized":"2024-01-24T17:44:10.986259+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:39:07.025772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150876+0000","last_change":"2024-01-24T17:44:10.013124+0000","last_active":"2024-01-24T17:44:11.150876+0000","last_peered":"2024-01-24T17:44:11.150876+0000","last_clean":"2024-01-24T17:44:11.150876+0000","last_became_active":"2024-01-24T17:44:10.011619+0000","last_became_peered":"2024-01-24T17:44:10.011619+0000","last_unstale":"2024-01-24T17:44:11.150876+0000","last_undegraded":"2024-01-24T17:44:11.150876+0000","last_fullsized":"2024-01-24T17:44:11.150876+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:03:55.983870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989013+0000","last_change":"2024-01-24T17:44:10.017844+0000","last_active":"2024-01-24T17:44:10.989013+0000","last_peered":"2024-01-24T17:44:10.989013+0000","last_clean":"2024-01-24T17:44:10.989013+0000","last_became_active":"2024-01-24T17:44:10.017209+0000","last_became_peered":"2024-01-24T17:44:10.017209+0000","last_unstale":"2024-01-24T17:44:10.989013+0000","last_undegraded":"2024-01-24T17:44:10.989013+0000","last_fullsized":"2024-01-24T17:44:10.989013+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:34:38.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149099+0000","last_change":"2024-01-24T17:44:09.989598+0000","last_active":"2024-01-24T17:44:11.149099+0000","last_peered":"2024-01-24T17:44:11.149099+0000","last_clean":"2024-01-24T17:44:11.149099+0000","last_became_active":"2024-01-24T17:44:09.989348+0000","last_became_peered":"2024-01-24T17:44:09.989348+0000","last_unstale":"2024-01-24T17:44:11.149099+0000","last_undegraded":"2024-01-24T17:44:11.149099+0000","last_fullsized":"2024-01-24T17:44:11.149099+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:24:30.205503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985371+0000","last_change":"2024-01-24T17:44:10.004575+0000","last_active":"2024-01-24T17:44:10.985371+0000","last_peered":"2024-01-24T17:44:10.985371+0000","last_clean":"2024-01-24T17:44:10.985371+0000","last_became_active":"2024-01-24T17:44:10.003711+0000","last_became_peered":"2024-01-24T17:44:10.003711+0000","last_unstale":"2024-01-24T17:44:10.985371+0000","last_undegraded":"2024-01-24T17:44:10.985371+0000","last_fullsized":"2024-01-24T17:44:10.985371+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:42.923012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988695+0000","last_change":"2024-01-24T17:44:10.010920+0000","last_active":"2024-01-24T17:44:10.988695+0000","last_peered":"2024-01-24T17:44:10.988695+0000","last_clean":"2024-01-24T17:44:10.988695+0000","last_became_active":"2024-01-24T17:44:10.009961+0000","last_became_peered":"2024-01-24T17:44:10.009961+0000","last_unstale":"2024-01-24T17:44:10.988695+0000","last_undegraded":"2024-01-24T17:44:10.988695+0000","last_fullsized":"2024-01-24T17:44:10.988695+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:26:07.520763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988648+0000","last_change":"2024-01-24T17:44:10.014459+0000","last_active":"2024-01-24T17:44:10.988648+0000","last_peered":"2024-01-24T17:44:10.988648+0000","last_clean":"2024-01-24T17:44:10.988648+0000","last_became_active":"2024-01-24T17:44:10.014045+0000","last_became_peered":"2024-01-24T17:44:10.014045+0000","last_unstale":"2024-01-24T17:44:10.988648+0000","last_undegraded":"2024-01-24T17:44:10.988648+0000","last_fullsized":"2024-01-24T17:44:10.988648+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:48:22.378247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.146976+0000","last_change":"2024-01-24T17:44:10.003350+0000","last_active":"2024-01-24T17:44:11.146976+0000","last_peered":"2024-01-24T17:44:11.146976+0000","last_clean":"2024-01-24T17:44:11.146976+0000","last_became_active":"2024-01-24T17:44:10.003102+0000","last_became_peered":"2024-01-24T17:44:10.003102+0000","last_unstale":"2024-01-24T17:44:11.146976+0000","last_undegraded":"2024-01-24T17:44:11.146976+0000","last_fullsized":"2024-01-24T17:44:11.146976+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:44.246072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149536+0000","last_change":"2024-01-24T17:44:09.992171+0000","last_active":"2024-01-24T17:44:11.149536+0000","last_peered":"2024-01-24T17:44:11.149536+0000","last_clean":"2024-01-24T17:44:11.149536+0000","last_became_active":"2024-01-24T17:44:09.991977+0000","last_became_peered":"2024-01-24T17:44:09.991977+0000","last_unstale":"2024-01-24T17:44:11.149536+0000","last_undegraded":"2024-01-24T17:44:11.149536+0000","last_fullsized":"2024-01-24T17:44:11.149536+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:49.636503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149147+0000","last_change":"2024-01-24T17:44:10.003551+0000","last_active":"2024-01-24T17:44:11.149147+0000","last_peered":"2024-01-24T17:44:11.149147+0000","last_clean":"2024-01-24T17:44:11.149147+0000","last_became_active":"2024-01-24T17:44:10.003330+0000","last_became_peered":"2024-01-24T17:44:10.003330+0000","last_unstale":"2024-01-24T17:44:11.149147+0000","last_undegraded":"2024-01-24T17:44:11.149147+0000","last_fullsized":"2024-01-24T17:44:11.149147+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:24:39.238240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147887+0000","last_change":"2024-01-24T17:44:09.995777+0000","last_active":"2024-01-24T17:44:11.147887+0000","last_peered":"2024-01-24T17:44:11.147887+0000","last_clean":"2024-01-24T17:44:11.147887+0000","last_became_active":"2024-01-24T17:44:09.995322+0000","last_became_peered":"2024-01-24T17:44:09.995322+0000","last_unstale":"2024-01-24T17:44:11.147887+0000","last_undegraded":"2024-01-24T17:44:11.147887+0000","last_fullsized":"2024-01-24T17:44:11.147887+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:38:11.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150363+0000","last_change":"2024-01-24T17:44:10.016759+0000","last_active":"2024-01-24T17:44:11.150363+0000","last_peered":"2024-01-24T17:44:11.150363+0000","last_clean":"2024-01-24T17:44:11.150363+0000","last_became_active":"2024-01-24T17:44:10.014917+0000","last_became_peered":"2024-01-24T17:44:10.014917+0000","last_unstale":"2024-01-24T17:44:11.150363+0000","last_undegraded":"2024-01-24T17:44:11.150363+0000","last_fullsized":"2024-01-24T17:44:11.150363+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:46.625289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989196+0000","last_change":"2024-01-24T17:44:10.002380+0000","last_active":"2024-01-24T17:44:10.989196+0000","last_peered":"2024-01-24T17:44:10.989196+0000","last_clean":"2024-01-24T17:44:10.989196+0000","last_became_active":"2024-01-24T17:44:10.002038+0000","last_became_peered":"2024-01-24T17:44:10.002038+0000","last_unstale":"2024-01-24T17:44:10.989196+0000","last_undegraded":"2024-01-24T17:44:10.989196+0000","last_fullsized":"2024-01-24T17:44:10.989196+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:20:03.837528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150101+0000","last_change":"2024-01-24T17:44:10.005209+0000","last_active":"2024-01-24T17:44:11.150101+0000","last_peered":"2024-01-24T17:44:11.150101+0000","last_clean":"2024-01-24T17:44:11.150101+0000","last_became_active":"2024-01-24T17:44:10.004937+0000","last_became_peered":"2024-01-24T17:44:10.004937+0000","last_unstale":"2024-01-24T17:44:11.150101+0000","last_undegraded":"2024-01-24T17:44:11.150101+0000","last_fullsized":"2024-01-24T17:44:11.150101+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:00.666938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984375+0000","last_change":"2024-01-24T17:44:09.992134+0000","last_active":"2024-01-24T17:44:10.984375+0000","last_peered":"2024-01-24T17:44:10.984375+0000","last_clean":"2024-01-24T17:44:10.984375+0000","last_became_active":"2024-01-24T17:44:09.991830+0000","last_became_peered":"2024-01-24T17:44:09.991830+0000","last_unstale":"2024-01-24T17:44:10.984375+0000","last_undegraded":"2024-01-24T17:44:10.984375+0000","last_fullsized":"2024-01-24T17:44:10.984375+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:03:06.693443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147523+0000","last_change":"2024-01-24T17:44:10.005994+0000","last_active":"2024-01-24T17:44:11.147523+0000","last_peered":"2024-01-24T17:44:11.147523+0000","last_clean":"2024-01-24T17:44:11.147523+0000","last_became_active":"2024-01-24T17:44:10.004637+0000","last_became_peered":"2024-01-24T17:44:10.004637+0000","last_unstale":"2024-01-24T17:44:11.147523+0000","last_undegraded":"2024-01-24T17:44:11.147523+0000","last_fullsized":"2024-01-24T17:44:11.147523+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:34:38.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147757+0000","last_change":"2024-01-24T17:44:09.988996+0000","last_active":"2024-01-24T17:44:11.147757+0000","last_peered":"2024-01-24T17:44:11.147757+0000","last_clean":"2024-01-24T17:44:11.147757+0000","last_became_active":"2024-01-24T17:44:09.988293+0000","last_became_peered":"2024-01-24T17:44:09.988293+0000","last_unstale":"2024-01-24T17:44:11.147757+0000","last_undegraded":"2024-01-24T17:44:11.147757+0000","last_fullsized":"2024-01-24T17:44:11.147757+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:49.636503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989293+0000","last_change":"2024-01-24T17:44:10.016776+0000","last_active":"2024-01-24T17:44:10.989293+0000","last_peered":"2024-01-24T17:44:10.989293+0000","last_clean":"2024-01-24T17:44:10.989293+0000","last_became_active":"2024-01-24T17:44:10.014859+0000","last_became_peered":"2024-01-24T17:44:10.014859+0000","last_unstale":"2024-01-24T17:44:10.989293+0000","last_undegraded":"2024-01-24T17:44:10.989293+0000","last_fullsized":"2024-01-24T17:44:10.989293+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:49.636503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984553+0000","last_change":"2024-01-24T17:44:10.000542+0000","last_active":"2024-01-24T17:44:10.984553+0000","last_peered":"2024-01-24T17:44:10.984553+0000","last_clean":"2024-01-24T17:44:10.984553+0000","last_became_active":"2024-01-24T17:44:10.000291+0000","last_became_peered":"2024-01-24T17:44:10.000291+0000","last_unstale":"2024-01-24T17:44:10.984553+0000","last_undegraded":"2024-01-24T17:44:10.984553+0000","last_fullsized":"2024-01-24T17:44:10.984553+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:42:25.924608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984749+0000","last_change":"2024-01-24T17:44:09.996598+0000","last_active":"2024-01-24T17:44:10.984749+0000","last_peered":"2024-01-24T17:44:10.984749+0000","last_clean":"2024-01-24T17:44:10.984749+0000","last_became_active":"2024-01-24T17:44:09.996219+0000","last_became_peered":"2024-01-24T17:44:09.996219+0000","last_unstale":"2024-01-24T17:44:10.984749+0000","last_undegraded":"2024-01-24T17:44:10.984749+0000","last_fullsized":"2024-01-24T17:44:10.984749+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:55:44.196397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147738+0000","last_change":"2024-01-24T17:44:10.011794+0000","last_active":"2024-01-24T17:44:11.147738+0000","last_peered":"2024-01-24T17:44:11.147738+0000","last_clean":"2024-01-24T17:44:11.147738+0000","last_became_active":"2024-01-24T17:44:10.011485+0000","last_became_peered":"2024-01-24T17:44:10.011485+0000","last_unstale":"2024-01-24T17:44:11.147738+0000","last_undegraded":"2024-01-24T17:44:11.147738+0000","last_fullsized":"2024-01-24T17:44:11.147738+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:03:16.674585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148319+0000","last_change":"2024-01-24T17:44:09.994315+0000","last_active":"2024-01-24T17:44:11.148319+0000","last_peered":"2024-01-24T17:44:11.148319+0000","last_clean":"2024-01-24T17:44:11.148319+0000","last_became_active":"2024-01-24T17:44:09.994062+0000","last_became_peered":"2024-01-24T17:44:09.994062+0000","last_unstale":"2024-01-24T17:44:11.148319+0000","last_undegraded":"2024-01-24T17:44:11.148319+0000","last_fullsized":"2024-01-24T17:44:11.148319+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:36:46.964115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984854+0000","last_change":"2024-01-24T17:44:09.997144+0000","last_active":"2024-01-24T17:44:10.984854+0000","last_peered":"2024-01-24T17:44:10.984854+0000","last_clean":"2024-01-24T17:44:10.984854+0000","last_became_active":"2024-01-24T17:44:09.996832+0000","last_became_peered":"2024-01-24T17:44:09.996832+0000","last_unstale":"2024-01-24T17:44:10.984854+0000","last_undegraded":"2024-01-24T17:44:10.984854+0000","last_fullsized":"2024-01-24T17:44:10.984854+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:59:47.528588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988461+0000","last_change":"2024-01-24T17:44:10.009997+0000","last_active":"2024-01-24T17:44:10.988461+0000","last_peered":"2024-01-24T17:44:10.988461+0000","last_clean":"2024-01-24T17:44:10.988461+0000","last_became_active":"2024-01-24T17:44:10.006753+0000","last_became_peered":"2024-01-24T17:44:10.006753+0000","last_unstale":"2024-01-24T17:44:10.988461+0000","last_undegraded":"2024-01-24T17:44:10.988461+0000","last_fullsized":"2024-01-24T17:44:10.988461+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:03:16.417542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150439+0000","last_change":"2024-01-24T17:44:10.012288+0000","last_active":"2024-01-24T17:44:11.150439+0000","last_peered":"2024-01-24T17:44:11.150439+0000","last_clean":"2024-01-24T17:44:11.150439+0000","last_became_active":"2024-01-24T17:44:10.012019+0000","last_became_peered":"2024-01-24T17:44:10.012019+0000","last_unstale":"2024-01-24T17:44:11.150439+0000","last_undegraded":"2024-01-24T17:44:11.150439+0000","last_fullsized":"2024-01-24T17:44:11.150439+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:42.923012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.986401+0000","last_change":"2024-01-24T17:44:09.506842+0000","last_active":"2024-01-24T17:44:10.986401+0000","last_peered":"2024-01-24T17:44:10.986401+0000","last_clean":"2024-01-24T17:44:10.986401+0000","last_became_active":"2024-01-24T17:44:09.505346+0000","last_became_peered":"2024-01-24T17:44:09.505346+0000","last_unstale":"2024-01-24T17:44:10.986401+0000","last_undegraded":"2024-01-24T17:44:10.986401+0000","last_fullsized":"2024-01-24T17:44:10.986401+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:35.222103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147873+0000","last_change":"2024-01-24T17:44:09.995977+0000","last_active":"2024-01-24T17:44:11.147873+0000","last_peered":"2024-01-24T17:44:11.147873+0000","last_clean":"2024-01-24T17:44:11.147873+0000","last_became_active":"2024-01-24T17:44:09.995445+0000","last_became_peered":"2024-01-24T17:44:09.995445+0000","last_unstale":"2024-01-24T17:44:11.147873+0000","last_undegraded":"2024-01-24T17:44:11.147873+0000","last_fullsized":"2024-01-24T17:44:11.147873+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:20:03.837528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984539+0000","last_change":"2024-01-24T17:44:09.516513+0000","last_active":"2024-01-24T17:44:10.984539+0000","last_peered":"2024-01-24T17:44:10.984539+0000","last_clean":"2024-01-24T17:44:10.984539+0000","last_became_active":"2024-01-24T17:44:09.516194+0000","last_became_peered":"2024-01-24T17:44:09.516194+0000","last_unstale":"2024-01-24T17:44:10.984539+0000","last_undegraded":"2024-01-24T17:44:10.984539+0000","last_fullsized":"2024-01-24T17:44:10.984539+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:43:45.174522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149189+0000","last_change":"2024-01-24T17:44:10.008307+0000","last_active":"2024-01-24T17:44:11.149189+0000","last_peered":"2024-01-24T17:44:11.149189+0000","last_clean":"2024-01-24T17:44:11.149189+0000","last_became_active":"2024-01-24T17:44:10.007996+0000","last_became_peered":"2024-01-24T17:44:10.007996+0000","last_unstale":"2024-01-24T17:44:11.149189+0000","last_undegraded":"2024-01-24T17:44:11.149189+0000","last_fullsized":"2024-01-24T17:44:11.149189+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:03:16.417542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.986012+0000","last_change":"2024-01-24T17:44:09.505972+0000","last_active":"2024-01-24T17:44:10.986012+0000","last_peered":"2024-01-24T17:44:10.986012+0000","last_clean":"2024-01-24T17:44:10.986012+0000","last_became_active":"2024-01-24T17:44:09.505370+0000","last_became_peered":"2024-01-24T17:44:09.505370+0000","last_unstale":"2024-01-24T17:44:10.986012+0000","last_undegraded":"2024-01-24T17:44:10.986012+0000","last_fullsized":"2024-01-24T17:44:10.986012+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:34:00.438070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989492+0000","last_change":"2024-01-24T17:44:10.016561+0000","last_active":"2024-01-24T17:44:10.989492+0000","last_peered":"2024-01-24T17:44:10.989492+0000","last_clean":"2024-01-24T17:44:10.989492+0000","last_became_active":"2024-01-24T17:44:10.016280+0000","last_became_peered":"2024-01-24T17:44:10.016280+0000","last_unstale":"2024-01-24T17:44:10.989492+0000","last_undegraded":"2024-01-24T17:44:10.989492+0000","last_fullsized":"2024-01-24T17:44:10.989492+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:44.246072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984823+0000","last_change":"2024-01-24T17:44:09.516601+0000","last_active":"2024-01-24T17:44:10.984823+0000","last_peered":"2024-01-24T17:44:10.984823+0000","last_clean":"2024-01-24T17:44:10.984823+0000","last_became_active":"2024-01-24T17:44:09.516310+0000","last_became_peered":"2024-01-24T17:44:09.516310+0000","last_unstale":"2024-01-24T17:44:10.984823+0000","last_undegraded":"2024-01-24T17:44:10.984823+0000","last_fullsized":"2024-01-24T17:44:10.984823+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:26:06.500664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984379+0000","last_change":"2024-01-24T17:44:10.000678+0000","last_active":"2024-01-24T17:44:10.984379+0000","last_peered":"2024-01-24T17:44:10.984379+0000","last_clean":"2024-01-24T17:44:10.984379+0000","last_became_active":"2024-01-24T17:44:10.000440+0000","last_became_peered":"2024-01-24T17:44:10.000440+0000","last_unstale":"2024-01-24T17:44:10.984379+0000","last_undegraded":"2024-01-24T17:44:10.984379+0000","last_fullsized":"2024-01-24T17:44:10.984379+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:15:24.877075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985884+0000","last_change":"2024-01-24T17:44:09.515026+0000","last_active":"2024-01-24T17:44:10.985884+0000","last_peered":"2024-01-24T17:44:10.985884+0000","last_clean":"2024-01-24T17:44:10.985884+0000","last_became_active":"2024-01-24T17:44:09.514761+0000","last_became_peered":"2024-01-24T17:44:09.514761+0000","last_unstale":"2024-01-24T17:44:10.985884+0000","last_undegraded":"2024-01-24T17:44:10.985884+0000","last_fullsized":"2024-01-24T17:44:10.985884+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:21:53.583579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150132+0000","last_change":"2024-01-24T17:44:10.015237+0000","last_active":"2024-01-24T17:44:11.150132+0000","last_peered":"2024-01-24T17:44:11.150132+0000","last_clean":"2024-01-24T17:44:11.150132+0000","last_became_active":"2024-01-24T17:44:10.014945+0000","last_became_peered":"2024-01-24T17:44:10.014945+0000","last_unstale":"2024-01-24T17:44:11.150132+0000","last_undegraded":"2024-01-24T17:44:11.150132+0000","last_fullsized":"2024-01-24T17:44:11.150132+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:26:07.520763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984412+0000","last_change":"2024-01-24T17:44:09.514280+0000","last_active":"2024-01-24T17:44:10.984412+0000","last_peered":"2024-01-24T17:44:10.984412+0000","last_clean":"2024-01-24T17:44:10.984412+0000","last_became_active":"2024-01-24T17:44:09.514009+0000","last_became_peered":"2024-01-24T17:44:09.514009+0000","last_unstale":"2024-01-24T17:44:10.984412+0000","last_undegraded":"2024-01-24T17:44:10.984412+0000","last_fullsized":"2024-01-24T17:44:10.984412+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:04:33.162077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149111+0000","last_change":"2024-01-24T17:44:10.009719+0000","last_active":"2024-01-24T17:44:11.149111+0000","last_peered":"2024-01-24T17:44:11.149111+0000","last_clean":"2024-01-24T17:44:11.149111+0000","last_became_active":"2024-01-24T17:44:10.006522+0000","last_became_peered":"2024-01-24T17:44:10.006522+0000","last_unstale":"2024-01-24T17:44:11.149111+0000","last_undegraded":"2024-01-24T17:44:11.149111+0000","last_fullsized":"2024-01-24T17:44:11.149111+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:03:16.417542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147187+0000","last_change":"2024-01-24T17:44:09.508753+0000","last_active":"2024-01-24T17:44:11.147187+0000","last_peered":"2024-01-24T17:44:11.147187+0000","last_clean":"2024-01-24T17:44:11.147187+0000","last_became_active":"2024-01-24T17:44:09.508490+0000","last_became_peered":"2024-01-24T17:44:09.508490+0000","last_unstale":"2024-01-24T17:44:11.147187+0000","last_undegraded":"2024-01-24T17:44:11.147187+0000","last_fullsized":"2024-01-24T17:44:11.147187+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:57:31.510691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984359+0000","last_change":"2024-01-24T17:44:10.004739+0000","last_active":"2024-01-24T17:44:10.984359+0000","last_peered":"2024-01-24T17:44:10.984359+0000","last_clean":"2024-01-24T17:44:10.984359+0000","last_became_active":"2024-01-24T17:44:10.004456+0000","last_became_peered":"2024-01-24T17:44:10.004456+0000","last_unstale":"2024-01-24T17:44:10.984359+0000","last_undegraded":"2024-01-24T17:44:10.984359+0000","last_fullsized":"2024-01-24T17:44:10.984359+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:25:33.425011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149358+0000","last_change":"2024-01-24T17:44:09.514515+0000","last_active":"2024-01-24T17:44:11.149358+0000","last_peered":"2024-01-24T17:44:11.149358+0000","last_clean":"2024-01-24T17:44:11.149358+0000","last_became_active":"2024-01-24T17:44:09.514229+0000","last_became_peered":"2024-01-24T17:44:09.514229+0000","last_unstale":"2024-01-24T17:44:11.149358+0000","last_undegraded":"2024-01-24T17:44:11.149358+0000","last_fullsized":"2024-01-24T17:44:11.149358+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:31:50.257431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149248+0000","last_change":"2024-01-24T17:44:10.007311+0000","last_active":"2024-01-24T17:44:11.149248+0000","last_peered":"2024-01-24T17:44:11.149248+0000","last_clean":"2024-01-24T17:44:11.149248+0000","last_became_active":"2024-01-24T17:44:10.007081+0000","last_became_peered":"2024-01-24T17:44:10.007081+0000","last_unstale":"2024-01-24T17:44:11.149248+0000","last_undegraded":"2024-01-24T17:44:11.149248+0000","last_fullsized":"2024-01-24T17:44:11.149248+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:13.437856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149711+0000","last_change":"2024-01-24T17:44:09.519719+0000","last_active":"2024-01-24T17:44:11.149711+0000","last_peered":"2024-01-24T17:44:11.149711+0000","last_clean":"2024-01-24T17:44:11.149711+0000","last_became_active":"2024-01-24T17:44:09.519225+0000","last_became_peered":"2024-01-24T17:44:09.519225+0000","last_unstale":"2024-01-24T17:44:11.149711+0000","last_undegraded":"2024-01-24T17:44:11.149711+0000","last_fullsized":"2024-01-24T17:44:11.149711+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:16:51.776116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149539+0000","last_change":"2024-01-24T17:44:09.993333+0000","last_active":"2024-01-24T17:44:11.149539+0000","last_peered":"2024-01-24T17:44:11.149539+0000","last_clean":"2024-01-24T17:44:11.149539+0000","last_became_active":"2024-01-24T17:44:09.993051+0000","last_became_peered":"2024-01-24T17:44:09.993051+0000","last_unstale":"2024-01-24T17:44:11.149539+0000","last_undegraded":"2024-01-24T17:44:11.149539+0000","last_fullsized":"2024-01-24T17:44:11.149539+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:15:30.995111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150900+0000","last_change":"2024-01-24T17:44:09.514538+0000","last_active":"2024-01-24T17:44:11.150900+0000","last_peered":"2024-01-24T17:44:11.150900+0000","last_clean":"2024-01-24T17:44:11.150900+0000","last_became_active":"2024-01-24T17:44:09.513829+0000","last_became_peered":"2024-01-24T17:44:09.513829+0000","last_unstale":"2024-01-24T17:44:11.150900+0000","last_undegraded":"2024-01-24T17:44:11.150900+0000","last_fullsized":"2024-01-24T17:44:11.150900+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:45:02.258807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988976+0000","last_change":"2024-01-24T17:44:10.012379+0000","last_active":"2024-01-24T17:44:10.988976+0000","last_peered":"2024-01-24T17:44:10.988976+0000","last_clean":"2024-01-24T17:44:10.988976+0000","last_became_active":"2024-01-24T17:44:10.012070+0000","last_became_peered":"2024-01-24T17:44:10.012070+0000","last_unstale":"2024-01-24T17:44:10.988976+0000","last_undegraded":"2024-01-24T17:44:10.988976+0000","last_fullsized":"2024-01-24T17:44:10.988976+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:24:30.205503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148940+0000","last_change":"2024-01-24T17:44:09.517515+0000","last_active":"2024-01-24T17:44:11.148940+0000","last_peered":"2024-01-24T17:44:11.148940+0000","last_clean":"2024-01-24T17:44:11.148940+0000","last_became_active":"2024-01-24T17:44:09.517247+0000","last_became_peered":"2024-01-24T17:44:09.517247+0000","last_unstale":"2024-01-24T17:44:11.148940+0000","last_undegraded":"2024-01-24T17:44:11.148940+0000","last_fullsized":"2024-01-24T17:44:11.148940+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:15:59.646839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984632+0000","last_change":"2024-01-24T17:44:10.015368+0000","last_active":"2024-01-24T17:44:10.984632+0000","last_peered":"2024-01-24T17:44:10.984632+0000","last_clean":"2024-01-24T17:44:10.984632+0000","last_became_active":"2024-01-24T17:44:10.014733+0000","last_became_peered":"2024-01-24T17:44:10.014733+0000","last_unstale":"2024-01-24T17:44:10.984632+0000","last_undegraded":"2024-01-24T17:44:10.984632+0000","last_fullsized":"2024-01-24T17:44:10.984632+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:50:12.903702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150613+0000","last_change":"2024-01-24T17:44:09.514688+0000","last_active":"2024-01-24T17:44:11.150613+0000","last_peered":"2024-01-24T17:44:11.150613+0000","last_clean":"2024-01-24T17:44:11.150613+0000","last_became_active":"2024-01-24T17:44:09.514165+0000","last_became_peered":"2024-01-24T17:44:09.514165+0000","last_unstale":"2024-01-24T17:44:11.150613+0000","last_undegraded":"2024-01-24T17:44:11.150613+0000","last_fullsized":"2024-01-24T17:44:11.150613+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:43.932754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149035+0000","last_change":"2024-01-24T17:44:10.003601+0000","last_active":"2024-01-24T17:44:11.149035+0000","last_peered":"2024-01-24T17:44:11.149035+0000","last_clean":"2024-01-24T17:44:11.149035+0000","last_became_active":"2024-01-24T17:44:10.003327+0000","last_became_peered":"2024-01-24T17:44:10.003327+0000","last_unstale":"2024-01-24T17:44:11.149035+0000","last_undegraded":"2024-01-24T17:44:11.149035+0000","last_fullsized":"2024-01-24T17:44:11.149035+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:21:54.603678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149180+0000","last_change":"2024-01-24T17:44:09.521600+0000","last_active":"2024-01-24T17:44:11.149180+0000","last_peered":"2024-01-24T17:44:11.149180+0000","last_clean":"2024-01-24T17:44:11.149180+0000","last_became_active":"2024-01-24T17:44:09.521098+0000","last_became_peered":"2024-01-24T17:44:09.521098+0000","last_unstale":"2024-01-24T17:44:11.149180+0000","last_undegraded":"2024-01-24T17:44:11.149180+0000","last_fullsized":"2024-01-24T17:44:11.149180+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:34:00.438070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149249+0000","last_change":"2024-01-24T17:44:09.993859+0000","last_active":"2024-01-24T17:44:11.149249+0000","last_peered":"2024-01-24T17:44:11.149249+0000","last_clean":"2024-01-24T17:44:11.149249+0000","last_became_active":"2024-01-24T17:44:09.993583+0000","last_became_peered":"2024-01-24T17:44:09.993583+0000","last_unstale":"2024-01-24T17:44:11.149249+0000","last_undegraded":"2024-01-24T17:44:11.149249+0000","last_fullsized":"2024-01-24T17:44:11.149249+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:59:47.528588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147366+0000","last_change":"2024-01-24T17:44:09.512507+0000","last_active":"2024-01-24T17:44:11.147366+0000","last_peered":"2024-01-24T17:44:11.147366+0000","last_clean":"2024-01-24T17:44:11.147366+0000","last_became_active":"2024-01-24T17:44:09.512192+0000","last_became_peered":"2024-01-24T17:44:09.512192+0000","last_unstale":"2024-01-24T17:44:11.147366+0000","last_undegraded":"2024-01-24T17:44:11.147366+0000","last_fullsized":"2024-01-24T17:44:11.147366+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:41.902913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149040+0000","last_change":"2024-01-24T17:44:10.009242+0000","last_active":"2024-01-24T17:44:11.149040+0000","last_peered":"2024-01-24T17:44:11.149040+0000","last_clean":"2024-01-24T17:44:11.149040+0000","last_became_active":"2024-01-24T17:44:10.008963+0000","last_became_peered":"2024-01-24T17:44:10.008963+0000","last_unstale":"2024-01-24T17:44:11.149040+0000","last_undegraded":"2024-01-24T17:44:11.149040+0000","last_fullsized":"2024-01-24T17:44:11.149040+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:43.106803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984928+0000","last_change":"2024-01-24T17:44:09.511291+0000","last_active":"2024-01-24T17:44:10.984928+0000","last_peered":"2024-01-24T17:44:10.984928+0000","last_clean":"2024-01-24T17:44:10.984928+0000","last_became_active":"2024-01-24T17:44:09.510821+0000","last_became_peered":"2024-01-24T17:44:09.510821+0000","last_unstale":"2024-01-24T17:44:10.984928+0000","last_undegraded":"2024-01-24T17:44:10.984928+0000","last_fullsized":"2024-01-24T17:44:10.984928+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:12.417757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148091+0000","last_change":"2024-01-24T17:44:09.983510+0000","last_active":"2024-01-24T17:44:11.148091+0000","last_peered":"2024-01-24T17:44:11.148091+0000","last_clean":"2024-01-24T17:44:11.148091+0000","last_became_active":"2024-01-24T17:44:09.982911+0000","last_became_peered":"2024-01-24T17:44:09.982911+0000","last_unstale":"2024-01-24T17:44:11.148091+0000","last_undegraded":"2024-01-24T17:44:11.148091+0000","last_fullsized":"2024-01-24T17:44:11.148091+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:44.246072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":19,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149615+0000","last_change":"2024-01-24T17:44:09.513971+0000","last_active":"2024-01-24T17:44:11.149615+0000","last_peered":"2024-01-24T17:44:11.149615+0000","last_clean":"2024-01-24T17:44:11.149615+0000","last_became_active":"2024-01-24T17:44:09.513487+0000","last_became_peered":"2024-01-24T17:44:09.513487+0000","last_unstale":"2024-01-24T17:44:11.149615+0000","last_undegraded":"2024-01-24T17:44:11.149615+0000","last_fullsized":"2024-01-24T17:44:11.149615+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:52.557043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150683+0000","last_change":"2024-01-24T17:44:10.001468+0000","last_active":"2024-01-24T17:44:11.150683+0000","last_peered":"2024-01-24T17:44:11.150683+0000","last_clean":"2024-01-24T17:44:11.150683+0000","last_became_active":"2024-01-24T17:44:10.001189+0000","last_became_peered":"2024-01-24T17:44:10.001189+0000","last_unstale":"2024-01-24T17:44:11.150683+0000","last_undegraded":"2024-01-24T17:44:11.150683+0000","last_fullsized":"2024-01-24T17:44:11.150683+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:39:07.025772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989401+0000","last_change":"2024-01-24T17:44:09.516892+0000","last_active":"2024-01-24T17:44:10.989401+0000","last_peered":"2024-01-24T17:44:10.989401+0000","last_clean":"2024-01-24T17:44:10.989401+0000","last_became_active":"2024-01-24T17:44:09.513860+0000","last_became_peered":"2024-01-24T17:44:09.513860+0000","last_unstale":"2024-01-24T17:44:10.989401+0000","last_undegraded":"2024-01-24T17:44:10.989401+0000","last_fullsized":"2024-01-24T17:44:10.989401+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:12.417757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149564+0000","last_change":"2024-01-24T17:44:10.009941+0000","last_active":"2024-01-24T17:44:11.149564+0000","last_peered":"2024-01-24T17:44:11.149564+0000","last_clean":"2024-01-24T17:44:11.149564+0000","last_became_active":"2024-01-24T17:44:10.009473+0000","last_became_peered":"2024-01-24T17:44:10.009473+0000","last_unstale":"2024-01-24T17:44:11.149564+0000","last_undegraded":"2024-01-24T17:44:11.149564+0000","last_fullsized":"2024-01-24T17:44:11.149564+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:53.577142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149346+0000","last_change":"2024-01-24T17:44:09.509070+0000","last_active":"2024-01-24T17:44:11.149346+0000","last_peered":"2024-01-24T17:44:11.149346+0000","last_clean":"2024-01-24T17:44:11.149346+0000","last_became_active":"2024-01-24T17:44:09.508731+0000","last_became_peered":"2024-01-24T17:44:09.508731+0000","last_unstale":"2024-01-24T17:44:11.149346+0000","last_undegraded":"2024-01-24T17:44:11.149346+0000","last_fullsized":"2024-01-24T17:44:11.149346+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:20:14.765740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149288+0000","last_change":"2024-01-24T17:44:10.013975+0000","last_active":"2024-01-24T17:44:11.149288+0000","last_peered":"2024-01-24T17:44:11.149288+0000","last_clean":"2024-01-24T17:44:11.149288+0000","last_became_active":"2024-01-24T17:44:10.013749+0000","last_became_peered":"2024-01-24T17:44:10.013749+0000","last_unstale":"2024-01-24T17:44:11.149288+0000","last_undegraded":"2024-01-24T17:44:11.149288+0000","last_fullsized":"2024-01-24T17:44:11.149288+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:26:07.520763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149439+0000","last_change":"2024-01-24T17:44:09.511864+0000","last_active":"2024-01-24T17:44:11.149439+0000","last_peered":"2024-01-24T17:44:11.149439+0000","last_clean":"2024-01-24T17:44:11.149439+0000","last_became_active":"2024-01-24T17:44:09.511481+0000","last_became_peered":"2024-01-24T17:44:09.511481+0000","last_unstale":"2024-01-24T17:44:11.149439+0000","last_undegraded":"2024-01-24T17:44:11.149439+0000","last_fullsized":"2024-01-24T17:44:11.149439+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:14:29.720521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149305+0000","last_change":"2024-01-24T17:44:10.007141+0000","last_active":"2024-01-24T17:44:11.149305+0000","last_peered":"2024-01-24T17:44:11.149305+0000","last_clean":"2024-01-24T17:44:11.149305+0000","last_became_active":"2024-01-24T17:44:10.006297+0000","last_became_peered":"2024-01-24T17:44:10.006297+0000","last_unstale":"2024-01-24T17:44:11.149305+0000","last_undegraded":"2024-01-24T17:44:11.149305+0000","last_fullsized":"2024-01-24T17:44:11.149305+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:33:47.593357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985606+0000","last_change":"2024-01-24T17:44:09.506825+0000","last_active":"2024-01-24T17:44:10.985606+0000","last_peered":"2024-01-24T17:44:10.985606+0000","last_clean":"2024-01-24T17:44:10.985606+0000","last_became_active":"2024-01-24T17:44:09.505893+0000","last_became_peered":"2024-01-24T17:44:09.505893+0000","last_unstale":"2024-01-24T17:44:10.985606+0000","last_undegraded":"2024-01-24T17:44:10.985606+0000","last_fullsized":"2024-01-24T17:44:10.985606+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:24:38.218141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149824+0000","last_change":"2024-01-24T17:44:10.006503+0000","last_active":"2024-01-24T17:44:11.149824+0000","last_peered":"2024-01-24T17:44:11.149824+0000","last_clean":"2024-01-24T17:44:11.149824+0000","last_became_active":"2024-01-24T17:44:10.006033+0000","last_became_peered":"2024-01-24T17:44:10.006033+0000","last_unstale":"2024-01-24T17:44:11.149824+0000","last_undegraded":"2024-01-24T17:44:11.149824+0000","last_fullsized":"2024-01-24T17:44:11.149824+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:04:34.182176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150142+0000","last_change":"2024-01-24T17:44:09.513431+0000","last_active":"2024-01-24T17:44:11.150142+0000","last_peered":"2024-01-24T17:44:11.150142+0000","last_clean":"2024-01-24T17:44:11.150142+0000","last_became_active":"2024-01-24T17:44:09.513127+0000","last_became_peered":"2024-01-24T17:44:09.513127+0000","last_unstale":"2024-01-24T17:44:11.150142+0000","last_undegraded":"2024-01-24T17:44:11.150142+0000","last_fullsized":"2024-01-24T17:44:11.150142+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:36:45.944016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985622+0000","last_change":"2024-01-24T17:44:09.999680+0000","last_active":"2024-01-24T17:44:10.985622+0000","last_peered":"2024-01-24T17:44:10.985622+0000","last_clean":"2024-01-24T17:44:10.985622+0000","last_became_active":"2024-01-24T17:44:09.999261+0000","last_became_peered":"2024-01-24T17:44:09.999261+0000","last_unstale":"2024-01-24T17:44:10.985622+0000","last_undegraded":"2024-01-24T17:44:10.985622+0000","last_fullsized":"2024-01-24T17:44:10.985622+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:00.666938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147422+0000","last_change":"2024-01-24T17:44:09.512707+0000","last_active":"2024-01-24T17:44:11.147422+0000","last_peered":"2024-01-24T17:44:11.147422+0000","last_clean":"2024-01-24T17:44:11.147422+0000","last_became_active":"2024-01-24T17:44:09.511352+0000","last_became_peered":"2024-01-24T17:44:09.511352+0000","last_unstale":"2024-01-24T17:44:11.147422+0000","last_undegraded":"2024-01-24T17:44:11.147422+0000","last_fullsized":"2024-01-24T17:44:11.147422+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:59:46.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988665+0000","last_change":"2024-01-24T17:44:10.005348+0000","last_active":"2024-01-24T17:44:10.988665+0000","last_peered":"2024-01-24T17:44:10.988665+0000","last_clean":"2024-01-24T17:44:10.988665+0000","last_became_active":"2024-01-24T17:44:10.005038+0000","last_became_peered":"2024-01-24T17:44:10.005038+0000","last_unstale":"2024-01-24T17:44:10.988665+0000","last_undegraded":"2024-01-24T17:44:10.988665+0000","last_fullsized":"2024-01-24T17:44:10.988665+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:57:21.200159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149488+0000","last_change":"2024-01-24T17:44:09.513261+0000","last_active":"2024-01-24T17:44:11.149488+0000","last_peered":"2024-01-24T17:44:11.149488+0000","last_clean":"2024-01-24T17:44:11.149488+0000","last_became_active":"2024-01-24T17:44:09.511938+0000","last_became_peered":"2024-01-24T17:44:09.511938+0000","last_unstale":"2024-01-24T17:44:11.149488+0000","last_undegraded":"2024-01-24T17:44:11.149488+0000","last_fullsized":"2024-01-24T17:44:11.149488+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:27:15.941486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988457+0000","last_change":"2024-01-24T17:44:10.009146+0000","last_active":"2024-01-24T17:44:10.988457+0000","last_peered":"2024-01-24T17:44:10.988457+0000","last_clean":"2024-01-24T17:44:10.988457+0000","last_became_active":"2024-01-24T17:44:10.008107+0000","last_became_peered":"2024-01-24T17:44:10.008107+0000","last_unstale":"2024-01-24T17:44:10.988457+0000","last_undegraded":"2024-01-24T17:44:10.988457+0000","last_fullsized":"2024-01-24T17:44:10.988457+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:43:46.194621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984203+0000","last_change":"2024-01-24T17:44:09.511280+0000","last_active":"2024-01-24T17:44:10.984203+0000","last_peered":"2024-01-24T17:44:10.984203+0000","last_clean":"2024-01-24T17:44:10.984203+0000","last_became_active":"2024-01-24T17:44:09.511034+0000","last_became_peered":"2024-01-24T17:44:09.511034+0000","last_unstale":"2024-01-24T17:44:10.984203+0000","last_undegraded":"2024-01-24T17:44:10.984203+0000","last_fullsized":"2024-01-24T17:44:10.984203+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:24:29.185403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984731+0000","last_change":"2024-01-24T17:44:10.005054+0000","last_active":"2024-01-24T17:44:10.984731+0000","last_peered":"2024-01-24T17:44:10.984731+0000","last_clean":"2024-01-24T17:44:10.984731+0000","last_became_active":"2024-01-24T17:44:10.004484+0000","last_became_peered":"2024-01-24T17:44:10.004484+0000","last_unstale":"2024-01-24T17:44:10.984731+0000","last_undegraded":"2024-01-24T17:44:10.984731+0000","last_fullsized":"2024-01-24T17:44:10.984731+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:37:01.164705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989459+0000","last_change":"2024-01-24T17:44:09.514456+0000","last_active":"2024-01-24T17:44:10.989459+0000","last_peered":"2024-01-24T17:44:10.989459+0000","last_clean":"2024-01-24T17:44:10.989459+0000","last_became_active":"2024-01-24T17:44:09.514131+0000","last_became_peered":"2024-01-24T17:44:09.514131+0000","last_unstale":"2024-01-24T17:44:10.989459+0000","last_undegraded":"2024-01-24T17:44:10.989459+0000","last_fullsized":"2024-01-24T17:44:10.989459+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:24:38.218141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150467+0000","last_change":"2024-01-24T17:44:10.006999+0000","last_active":"2024-01-24T17:44:11.150467+0000","last_peered":"2024-01-24T17:44:11.150467+0000","last_clean":"2024-01-24T17:44:11.150467+0000","last_became_active":"2024-01-24T17:44:10.006619+0000","last_became_peered":"2024-01-24T17:44:10.006619+0000","last_unstale":"2024-01-24T17:44:11.150467+0000","last_undegraded":"2024-01-24T17:44:11.150467+0000","last_fullsized":"2024-01-24T17:44:11.150467+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:24:30.205503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147752+0000","last_change":"2024-01-24T17:44:09.510989+0000","last_active":"2024-01-24T17:44:11.147752+0000","last_peered":"2024-01-24T17:44:11.147752+0000","last_clean":"2024-01-24T17:44:11.147752+0000","last_became_active":"2024-01-24T17:44:09.510665+0000","last_became_peered":"2024-01-24T17:44:09.510665+0000","last_unstale":"2024-01-24T17:44:11.147752+0000","last_undegraded":"2024-01-24T17:44:11.147752+0000","last_fullsized":"2024-01-24T17:44:11.147752+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:39:06.005673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984662+0000","last_change":"2024-01-24T17:44:10.015028+0000","last_active":"2024-01-24T17:44:10.984662+0000","last_peered":"2024-01-24T17:44:10.984662+0000","last_clean":"2024-01-24T17:44:10.984662+0000","last_became_active":"2024-01-24T17:44:10.014716+0000","last_became_peered":"2024-01-24T17:44:10.014716+0000","last_unstale":"2024-01-24T17:44:10.984662+0000","last_undegraded":"2024-01-24T17:44:10.984662+0000","last_fullsized":"2024-01-24T17:44:10.984662+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:19:35.252541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149741+0000","last_change":"2024-01-24T17:44:09.519257+0000","last_active":"2024-01-24T17:44:11.149741+0000","last_peered":"2024-01-24T17:44:11.149741+0000","last_clean":"2024-01-24T17:44:11.149741+0000","last_became_active":"2024-01-24T17:44:09.518939+0000","last_became_peered":"2024-01-24T17:44:09.518939+0000","last_unstale":"2024-01-24T17:44:11.149741+0000","last_undegraded":"2024-01-24T17:44:11.149741+0000","last_fullsized":"2024-01-24T17:44:11.149741+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:35.222103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149745+0000","last_change":"2024-01-24T17:44:10.006287+0000","last_active":"2024-01-24T17:44:11.149745+0000","last_peered":"2024-01-24T17:44:11.149745+0000","last_clean":"2024-01-24T17:44:11.149745+0000","last_became_active":"2024-01-24T17:44:10.005637+0000","last_became_peered":"2024-01-24T17:44:10.005637+0000","last_unstale":"2024-01-24T17:44:11.149745+0000","last_undegraded":"2024-01-24T17:44:11.149745+0000","last_fullsized":"2024-01-24T17:44:11.149745+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:43:46.194621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147780+0000","last_change":"2024-01-24T17:44:09.511553+0000","last_active":"2024-01-24T17:44:11.147780+0000","last_peered":"2024-01-24T17:44:11.147780+0000","last_clean":"2024-01-24T17:44:11.147780+0000","last_became_active":"2024-01-24T17:44:09.511237+0000","last_became_peered":"2024-01-24T17:44:09.511237+0000","last_unstale":"2024-01-24T17:44:11.147780+0000","last_undegraded":"2024-01-24T17:44:11.147780+0000","last_fullsized":"2024-01-24T17:44:11.147780+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:42.086704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988995+0000","last_change":"2024-01-24T17:44:10.002700+0000","last_active":"2024-01-24T17:44:10.988995+0000","last_peered":"2024-01-24T17:44:10.988995+0000","last_clean":"2024-01-24T17:44:10.988995+0000","last_became_active":"2024-01-24T17:44:10.002169+0000","last_became_peered":"2024-01-24T17:44:10.002169+0000","last_unstale":"2024-01-24T17:44:10.988995+0000","last_undegraded":"2024-01-24T17:44:10.988995+0000","last_fullsized":"2024-01-24T17:44:10.988995+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:03:16.674585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984479+0000","last_change":"2024-01-24T17:44:09.514378+0000","last_active":"2024-01-24T17:44:10.984479+0000","last_peered":"2024-01-24T17:44:10.984479+0000","last_clean":"2024-01-24T17:44:10.984479+0000","last_became_active":"2024-01-24T17:44:09.514073+0000","last_became_peered":"2024-01-24T17:44:09.514073+0000","last_unstale":"2024-01-24T17:44:10.984479+0000","last_undegraded":"2024-01-24T17:44:10.984479+0000","last_fullsized":"2024-01-24T17:44:10.984479+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:03:54.963771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148435+0000","last_change":"2024-01-24T17:44:10.015660+0000","last_active":"2024-01-24T17:44:11.148435+0000","last_peered":"2024-01-24T17:44:11.148435+0000","last_clean":"2024-01-24T17:44:11.148435+0000","last_became_active":"2024-01-24T17:44:10.015193+0000","last_became_peered":"2024-01-24T17:44:10.015193+0000","last_unstale":"2024-01-24T17:44:11.148435+0000","last_undegraded":"2024-01-24T17:44:11.148435+0000","last_fullsized":"2024-01-24T17:44:11.148435+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:34:01.458169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985124+0000","last_change":"2024-01-24T17:44:09.511479+0000","last_active":"2024-01-24T17:44:10.985124+0000","last_peered":"2024-01-24T17:44:10.985124+0000","last_clean":"2024-01-24T17:44:10.985124+0000","last_became_active":"2024-01-24T17:44:09.511203+0000","last_became_peered":"2024-01-24T17:44:09.511203+0000","last_unstale":"2024-01-24T17:44:10.985124+0000","last_undegraded":"2024-01-24T17:44:10.985124+0000","last_fullsized":"2024-01-24T17:44:10.985124+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:33:46.573258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148292+0000","last_change":"2024-01-24T17:44:10.000607+0000","last_active":"2024-01-24T17:44:11.148292+0000","last_peered":"2024-01-24T17:44:11.148292+0000","last_clean":"2024-01-24T17:44:11.148292+0000","last_became_active":"2024-01-24T17:44:10.000359+0000","last_became_peered":"2024-01-24T17:44:10.000359+0000","last_unstale":"2024-01-24T17:44:11.148292+0000","last_undegraded":"2024-01-24T17:44:11.148292+0000","last_fullsized":"2024-01-24T17:44:11.148292+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:03:16.674585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149265+0000","last_change":"2024-01-24T17:44:09.522112+0000","last_active":"2024-01-24T17:44:11.149265+0000","last_peered":"2024-01-24T17:44:11.149265+0000","last_clean":"2024-01-24T17:44:11.149265+0000","last_became_active":"2024-01-24T17:44:09.521810+0000","last_became_peered":"2024-01-24T17:44:09.521810+0000","last_unstale":"2024-01-24T17:44:11.149265+0000","last_undegraded":"2024-01-24T17:44:11.149265+0000","last_fullsized":"2024-01-24T17:44:11.149265+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:21:53.583579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989385+0000","last_change":"2024-01-24T17:44:10.007584+0000","last_active":"2024-01-24T17:44:10.989385+0000","last_peered":"2024-01-24T17:44:10.989385+0000","last_clean":"2024-01-24T17:44:10.989385+0000","last_became_active":"2024-01-24T17:44:10.005905+0000","last_became_peered":"2024-01-24T17:44:10.005905+0000","last_unstale":"2024-01-24T17:44:10.989385+0000","last_undegraded":"2024-01-24T17:44:10.989385+0000","last_fullsized":"2024-01-24T17:44:10.989385+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:45:03.278906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984785+0000","last_change":"2024-01-24T17:44:09.517800+0000","last_active":"2024-01-24T17:44:10.984785+0000","last_peered":"2024-01-24T17:44:10.984785+0000","last_clean":"2024-01-24T17:44:10.984785+0000","last_became_active":"2024-01-24T17:44:09.517546+0000","last_became_peered":"2024-01-24T17:44:09.517546+0000","last_unstale":"2024-01-24T17:44:10.984785+0000","last_undegraded":"2024-01-24T17:44:10.984785+0000","last_fullsized":"2024-01-24T17:44:10.984785+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:03:15.397443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-24T17:44:09.988868+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:44:09.988868+0000","last_peered":"2024-01-24T17:44:09.988868+0000","last_clean":"2024-01-24T17:44:09.988868+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:44:09.988868+0000","last_undegraded":"2024-01-24T17:44:09.988868+0000","last_fullsized":"2024-01-24T17:44:09.988868+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":232448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.0329999999999999,"15min":1.0189999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.028,"15min":1.026},"min":{"1min":0.75800000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.94499999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.087,"15min":1.1000000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.446,"5min":1.655,"15min":1.655},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1839999999999999,"15min":1.1919999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.8109999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.649}]},{"osd":2,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.165,"15min":1.1559999999999999},"min":{"1min":0.879,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.5660000000000001,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.008},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.204,"15min":1.226},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.321,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.141}]},{"osd":3,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.244,"15min":1.252},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.5529999999999999,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.173},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3009999999999999,"15min":1.3160000000000001},"min":{"1min":0.93400000000000005,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.464,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":1.242}]},{"osd":4,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.302,"15min":1.3220000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6299999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.319},{"interface":"front","average":{"1min":1.431,"5min":1.4379999999999999,"15min":1.4390000000000001},"min":{"1min":0.92500000000000004,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":1.774,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.288}]},{"osd":5,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.347,"15min":1.361},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7270000000000001,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.4319999999999999,"15min":1.446},"min":{"1min":0.94599999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.9510000000000001,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.528}]},{"osd":6,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.5269999999999999,"15min":1.5449999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.0310000000000001,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4910000000000001,"15min":1.494},"min":{"1min":1.0569999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.9330000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.3600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":573440,"data_stored":233030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0229999999999999,"15min":1.0209999999999999},"min":{"1min":0.623,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.77,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.018,"15min":1.01},"min":{"1min":0.69499999999999995,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.272}]},{"osd":1,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.997,"15min":0.98299999999999998},"min":{"1min":0.66700000000000004,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.1220000000000001,"15min":1.1140000000000001},"min":{"1min":0.78600000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.944,"5min":1.972,"15min":1.972},"last":1.234}]},{"osd":2,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1439999999999999,"15min":1.137},"min":{"1min":0.70399999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6850000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1579999999999999,"15min":1.163},"min":{"1min":0.78200000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.534,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.446}]},{"osd":3,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.2370000000000001,"15min":1.2529999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.0009999999999999,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.155},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2430000000000001,"15min":1.2569999999999999},"min":{"1min":0.81899999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5249999999999999,"5min":2.46,"15min":2.46},"last":1.357}]},{"osd":4,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.236,"15min":1.23},"min":{"1min":0.76000000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.643,"5min":1.988,"15min":1.988},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.3,"15min":1.3029999999999999},"min":{"1min":0.80500000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5349999999999999,"5min":1.849,"15min":1.849},"last":1.1850000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.288,"15min":1.29},"min":{"1min":0.78100000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.538,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.323},{"interface":"front","average":{"1min":1.419,"5min":1.4350000000000001,"15min":1.4379999999999999},"min":{"1min":1.091,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8680000000000001,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.4970000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3600000000000001,"15min":1.369},"min":{"1min":1.0549999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.8,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.381},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4570000000000001,"15min":1.464},"min":{"1min":0.88900000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.9019999999999999,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.556}]}]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":822804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.097,"15min":1.109},"min":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":1.508,"15min":1.508},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0349999999999999,"15min":1.0329999999999999},"min":{"1min":0.74399999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.0469999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0760000000000001,"15min":1.0740000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.254,"15min":1.2749999999999999},"min":{"1min":0.77000000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.5549999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.306}]},{"osd":2,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.246,"15min":1.2370000000000001},"min":{"1min":0.79400000000000004,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.655,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3,"15min":1.319},"min":{"1min":0.67600000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.8380000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.093}]},{"osd":3,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.3360000000000001,"15min":1.3560000000000001},"min":{"1min":0.81499999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.0150000000000001,"5min":3.036,"15min":3.036},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3320000000000001,"15min":1.3400000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.7290000000000001,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.5580000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.454,"15min":1.46},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":1.72},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.363,"15min":1.371},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":3.2810000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.2210000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3560000000000001,"15min":1.3620000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":1.425},{"interface":"front","average":{"1min":1.516,"5min":1.5289999999999999,"15min":1.5309999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1579999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.8089999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.492,"15min":1.504},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":3.161,"5min":3.161,"15min":3.161},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.6240000000000001,"15min":1.6459999999999999},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.0139999999999998,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.5129999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":585728,"data_stored":234070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.024,"15min":1.03},"min":{"1min":0.69999999999999996,"5min":0.621,"15min":0.621},"max":{"1min":1.5449999999999999,"5min":1.698,"15min":1.698},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.097,"15min":1.097},"min":{"1min":0.75,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.82799999999999996}]},{"osd":1,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.048,"15min":1.0409999999999999},"min":{"1min":0.75,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.2190000000000001,"15min":1.2270000000000001},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.73599999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.21,"15min":1.214},"min":{"1min":0.84399999999999997,"5min":0.75,"15min":0.75},"max":{"1min":1.5509999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":2.105},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.3340000000000001,"15min":1.3440000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.986,"5min":1.986,"15min":1.986},"last":1.1659999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.1990000000000001,"15min":1.196},"min":{"1min":0.88100000000000001,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3500000000000001,"15min":1.3540000000000001},"min":{"1min":0.90300000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.7150000000000001,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.8029999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.371,"15min":1.381},"min":{"1min":0.71199999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.321,"15min":1.3129999999999999},"min":{"1min":0.98299999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.732,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.2310000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4690000000000001,"15min":1.4770000000000001},"min":{"1min":0.83399999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9490000000000001,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":2.3159999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.482,"15min":1.486},"min":{"1min":0.91900000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":2.2610000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4350000000000001,"15min":1.427},"min":{"1min":0.879,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":2.0209999999999999},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6240000000000001,"15min":1.629},"min":{"1min":0.94399999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.1520000000000001,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":2.2309999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":577536,"data_stored":233978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":0.90100000000000002,"15min":0.86699999999999999},"min":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":5.1929999999999996,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.236,"5min":0.93500000000000005,"15min":0.88500000000000001},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":8.6989999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":1.1060000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.94599999999999995,"15min":0.95299999999999996},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.143,"5min":1.079,"15min":1.069},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":3.7240000000000002,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.4630000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.254,"5min":1.175,"15min":1.1619999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":4.2919999999999998,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":1.4350000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.323,"15min":1.2929999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":6.7469999999999999,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":1.349},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3600000000000001,"15min":1.357},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.4049999999999998,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.159}]},{"osd":5,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.448,"15min":1.4419999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.431,"15min":1.429},"min":{"1min":1.1259999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.2949999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.5529999999999999,"15min":1.51},"min":{"1min":0.89800000000000002,"5min":0.871,"15min":0.871},"max":{"1min":9.5440000000000005,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.4830000000000001,"15min":1.4670000000000001},"min":{"1min":0.93500000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.5009999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.546,"15min":1.5189999999999999},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":6.6970000000000001,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.603},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.55,"15min":1.528},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":6.6529999999999996,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.401}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":569344,"data_stored":232504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.36399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3069999999999999,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.93200000000000005,"15min":0.93899999999999995},"min":{"1min":0.44500000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.502,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.46500000000000002}]},{"osd":1,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.048,"15min":1.0589999999999999},"min":{"1min":0.55300000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.202,"15min":1.2250000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.706,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.51100000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.99199999999999999,"15min":0.999},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.2110000000000001,"15min":1.22},"min":{"1min":0.60699999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":0.54000000000000004}]},{"osd":4,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.754,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.3939999999999999,"15min":1.423},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8069999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":0.84599999999999997}]},{"osd":5,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.397,"15min":1.421},"min":{"1min":0.83099999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.6279999999999999,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.3280000000000001,"15min":1.3420000000000001},"min":{"1min":0.80600000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6279999999999999,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.86299999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.393,"15min":1.4099999999999999},"min":{"1min":0.86499999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.6830000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4690000000000001,"15min":1.4930000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.1629999999999998,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5660000000000001,"15min":1.6000000000000001},"min":{"1min":0.82399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.819,"5min":3.343,"15min":3.343},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.353,"5min":1.5,"15min":1.524},"min":{"1min":0.96299999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.7849999999999999,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":0.78600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1163264,"data_stored":822852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.879,"15min":0.86199999999999999},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.56100000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.538,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.014}]},{"osd":2,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.98699999999999999,"15min":0.995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.876},{"interface":"front","average":{"1min":1.028,"5min":1.105,"15min":1.1180000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.778,"5min":1.895,"15min":1.895},"last":1.0549999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1639999999999999,"15min":1.171},"min":{"1min":0.57499999999999996,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.863,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.236,"15min":1.258},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.591,"5min":3.149,"15min":3.149},"last":1.2549999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3680000000000001,"15min":1.3700000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.9370000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.097},{"interface":"front","average":{"1min":1.286,"5min":1.327,"15min":1.3340000000000001},"min":{"1min":0.88700000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.627,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.1200000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3660000000000001,"15min":1.3720000000000001},"min":{"1min":0.84399999999999997,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.7490000000000001,"5min":2.577,"15min":2.577},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.3740000000000001,"15min":1.381},"min":{"1min":0.83199999999999996,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.998,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.196}]},{"osd":6,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.4379999999999999,"15min":1.4570000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.863,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.159},{"interface":"front","average":{"1min":1.359,"5min":1.5249999999999999,"15min":1.5529999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.077,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.226}]},{"osd":7,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.478,"15min":1.482},"min":{"1min":0.83299999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0710000000000002,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.472},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.556,"15min":1.5720000000000001},"min":{"1min":0.93700000000000006,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":1.8129999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.427}]}]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":573440,"data_stored":233086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.81299999999999994,"15min":0.82199999999999995},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.95299999999999996,"15min":0.97699999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4399999999999999,"5min":1.494,"15min":1.494},"last":0.73099999999999998}]},{"osd":2,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82799999999999996,"15min":0.82899999999999996},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.84899999999999998,"5min":1.0720000000000001,"15min":1.109},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.216,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.81999999999999995}]},{"osd":3,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1890000000000001,"15min":1.2070000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.609,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.123,"5min":1.1839999999999999,"15min":1.194},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.605,"5min":2.407,"15min":2.407},"last":1.028}]},{"osd":4,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.397,"15min":1.4330000000000001},"min":{"1min":0.73299999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.544,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.119},{"interface":"front","average":{"1min":1.262,"5min":1.472,"15min":1.5069999999999999},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.675,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.1379999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.375,"15min":1.411},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.5249999999999999,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.01},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2609999999999999,"15min":1.266},"min":{"1min":0.87,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.7569999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.151}]},{"osd":6,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.3460000000000001,"15min":1.363},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.871,"5min":2.335,"15min":2.335},"last":1.25},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.4430000000000001,"15min":1.47},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.6990000000000001,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.1919999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.5169999999999999,"15min":1.552},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8260000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.591,"15min":1.623},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":1.9379999999999999,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.218}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:44:14.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:44:14.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.755+0000 7f34de200640 1 Processor -- start 2024-01-24T17:44:14.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.755+0000 7f34de200640 1 -- start start 2024-01-24T17:44:14.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.757+0000 7f34de200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f34d815a720 0x7f34d80cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.757+0000 7f34de200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815ab00 0x7f34d80cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.757+0000 7f34de200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d80ce400 con 0x7f34d80ccd20 2024-01-24T17:44:14.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.757+0000 7f34de200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d80ce110 con 0x7f34d815a720 2024-01-24T17:44:14.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.757+0000 7f34de200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d815b3b0 con 0x7f34d815ab00 2024-01-24T17:44:14.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.757+0000 7f34dce00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f34d815a720 0x7f34d80cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.757+0000 7f34dd800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f34d80ccd20 0x7f34d80ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:57386/0 (socket says 172.21.15.39:57386) 2024-01-24T17:44:14.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.757+0000 7f34dd800640 1 -- 172.21.15.39:0/2051176104 learned_addr learned my addr 172.21.15.39:0/2051176104 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:14.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.758+0000 7f34cfe00640 1 --2- 172.21.15.39:0/2051176104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815ab00 0x7f34d80cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.758+0000 7f34dce00640 1 -- 172.21.15.39:0/2051176104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815ab00 msgr2=0x7f34d80cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:14.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.758+0000 7f34dce00640 1 --2- 172.21.15.39:0/2051176104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815ab00 0x7f34d80cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.758+0000 7f34dce00640 1 -- 172.21.15.39:0/2051176104 >> v1:172.21.15.39:6789/0 conn(0x7f34d80ccd20 legacy=0x7f34d80ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.758+0000 7f34dce00640 1 -- 172.21.15.39:0/2051176104 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d815b690 con 0x7f34d815a720 2024-01-24T17:44:14.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.758+0000 7f34d7e00640 1 -- 172.21.15.39:0/2051176104 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183477291 0 0) 0x7f34d80ce400 con 0x7f34d80ccd20 2024-01-24T17:44:14.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.760+0000 7f34dce00640 1 --2- 172.21.15.39:0/2051176104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815a720 0x7f34d80cdbd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f34c80097a0 tx=0x7f34c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4c468b822926d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:14.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.760+0000 7f34d7e00640 1 -- 172.21.15.39:0/2051176104 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c802e070 con 0x7f34d815a720 2024-01-24T17:44:14.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.760+0000 7f34d7e00640 1 -- 172.21.15.39:0/2051176104 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c80315a0 con 0x7f34d815a720 2024-01-24T17:44:14.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.760+0000 7f34d7e00640 1 -- 172.21.15.39:0/2051176104 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8031930 con 0x7f34d815a720 2024-01-24T17:44:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.761+0000 7f34de200640 1 -- 172.21.15.39:0/2051176104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815a720 msgr2=0x7f34d80cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.761+0000 7f34de200640 1 --2- 172.21.15.39:0/2051176104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815a720 0x7f34d80cdbd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f34c80097a0 tx=0x7f34c802b840 comp rx=0 tx=0).stop 2024-01-24T17:44:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.761+0000 7f34de200640 1 -- 172.21.15.39:0/2051176104 shutdown_connections 2024-01-24T17:44:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.761+0000 7f34de200640 1 --2- 172.21.15.39:0/2051176104 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815ab00 0x7f34d80cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.761+0000 7f34de200640 1 --2- 172.21.15.39:0/2051176104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815a720 0x7f34d80cdbd0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.761+0000 7f34de200640 1 -- 172.21.15.39:0/2051176104 >> 172.21.15.39:0/2051176104 conn(0x7f34d806a090 msgr2=0x7f34d806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.761+0000 7f34de200640 1 -- 172.21.15.39:0/2051176104 shutdown_connections 2024-01-24T17:44:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.761+0000 7f34de200640 1 -- 172.21.15.39:0/2051176104 wait complete. 2024-01-24T17:44:14.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.763+0000 7f34de200640 1 Processor -- start 2024-01-24T17:44:14.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.764+0000 7f34de200640 1 -- start start 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34de200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815a720 0x7f34d8157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34de200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815ab00 0x7f34d81f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34de200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34d81f1950 0x7f34d8154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34de200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d81f66f0 con 0x7f34d815ab00 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34de200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8157b40 con 0x7f34d81f1950 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34de200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d8157e20 con 0x7f34d815a720 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34cfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815ab00 0x7f34d81f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34cfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815ab00 0x7f34d81f61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38928/0 (socket says 172.21.15.39:38928) 2024-01-24T17:44:14.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.765+0000 7f34cfe00640 1 -- 172.21.15.39:0/2809771538 learned_addr learned my addr 172.21.15.39:0/2809771538 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34dd800640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34d81f1950 0x7f34d8154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34cfe00640 1 -- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815a720 msgr2=0x7f34d8157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34cfe00640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815a720 0x7f34d8157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34cfe00640 1 -- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34d81f1950 msgr2=0x7f34d8154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34cfe00640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34d81f1950 0x7f34d8154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34cfe00640 1 -- 172.21.15.39:0/2809771538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009400 con 0x7f34d815ab00 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34dd800640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34d81f1950 0x7f34d8154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34cfe00640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815ab00 0x7f34d81f61b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f34c000d5f0 tx=0x7f34c000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34d5600640 1 -- 172.21.15.39:0/2809771538 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c00125f0 con 0x7f34d815ab00 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34d5600640 1 -- 172.21.15.39:0/2809771538 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c0012750 con 0x7f34d815ab00 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34d5600640 1 -- 172.21.15.39:0/2809771538 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c0012b00 con 0x7f34d815ab00 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.766+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d8153820 con 0x7f34d815ab00 2024-01-24T17:44:14.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.767+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d8153de0 con 0x7f34d815ab00 2024-01-24T17:44:14.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.768+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d804dff0 con 0x7f34d815ab00 2024-01-24T17:44:14.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.769+0000 7f34d5600640 1 -- 172.21.15.39:0/2809771538 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f34c0012c60 con 0x7f34d815ab00 2024-01-24T17:44:14.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.770+0000 7f34d5600640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f34b006a430 0x7f34b006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:14.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.770+0000 7f34d5600640 1 -- 172.21.15.39:0/2809771538 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34c0088e40 con 0x7f34d815ab00 2024-01-24T17:44:14.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.770+0000 7f34dce00640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f34b006a430 0x7f34b006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:14.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.771+0000 7f34dce00640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f34b006a430 0x7f34b006c8f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f34c8036000 tx=0x7f34c8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:14.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.774+0000 7f34d5600640 1 -- 172.21.15.39:0/2809771538 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f34c0054390 con 0x7f34d815ab00 2024-01-24T17:44:14.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.965+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34d815bc50 con 0x7f34b006a430 2024-01-24T17:44:14.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.987+0000 7f34d5600640 1 -- 172.21.15.39:0/2809771538 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287275 (secure 0 0 0) 0x7f34c8031990 con 0x7f34b006a430 2024-01-24T17:44:14.992 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:14.994 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:44:14.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f34b006a430 msgr2=0x7f34b006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f34b006a430 0x7f34b006c8f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f34c8036000 tx=0x7f34c8030000 comp rx=0 tx=0).stop 2024-01-24T17:44:14.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815ab00 msgr2=0x7f34d81f61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:14.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815ab00 0x7f34d81f61b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f34c000d5f0 tx=0x7f34c000dab0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 shutdown_connections 2024-01-24T17:44:14.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f34d81f1950 0x7f34d8154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f34b006a430 0x7f34b006c8f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34d815ab00 0x7f34d81f61b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 --2- 172.21.15.39:0/2809771538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f34d815a720 0x7f34d8157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:14.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.993+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 >> 172.21.15.39:0/2809771538 conn(0x7f34d806a090 msgr2=0x7f34d806a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:14.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.994+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 shutdown_connections 2024-01-24T17:44:14.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:14.994+0000 7f34de200640 1 -- 172.21.15.39:0/2809771538 wait complete. 2024-01-24T17:44:15.014 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-01-24T17:44:14.267278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238188,"kb_used_data":5640,"kb_used_omap":13,"kb_used_meta":232434,"kb_avail":754736532,"statfs":{"total":773094113280,"available":772850208768,"internally_reserved":0,"allocated":5775360,"data_stored":3044772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":238012504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.803410"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147607+0000","last_change":"2024-01-24T17:44:09.988313+0000","last_active":"2024-01-24T17:44:11.147607+0000","last_peered":"2024-01-24T17:44:11.147607+0000","last_clean":"2024-01-24T17:44:11.147607+0000","last_became_active":"2024-01-24T17:44:09.987962+0000","last_became_peered":"2024-01-24T17:44:09.987962+0000","last_unstale":"2024-01-24T17:44:11.147607+0000","last_undegraded":"2024-01-24T17:44:11.147607+0000","last_fullsized":"2024-01-24T17:44:11.147607+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:34:38.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149565+0000","last_change":"2024-01-24T17:44:10.012031+0000","last_active":"2024-01-24T17:44:11.149565+0000","last_peered":"2024-01-24T17:44:11.149565+0000","last_clean":"2024-01-24T17:44:11.149565+0000","last_became_active":"2024-01-24T17:44:10.011640+0000","last_became_peered":"2024-01-24T17:44:10.011640+0000","last_unstale":"2024-01-24T17:44:11.149565+0000","last_undegraded":"2024-01-24T17:44:11.149565+0000","last_fullsized":"2024-01-24T17:44:11.149565+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:48:22.378247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149164+0000","last_change":"2024-01-24T17:44:10.007028+0000","last_active":"2024-01-24T17:44:11.149164+0000","last_peered":"2024-01-24T17:44:11.149164+0000","last_clean":"2024-01-24T17:44:11.149164+0000","last_became_active":"2024-01-24T17:44:10.006334+0000","last_became_peered":"2024-01-24T17:44:10.006334+0000","last_unstale":"2024-01-24T17:44:11.149164+0000","last_undegraded":"2024-01-24T17:44:11.149164+0000","last_fullsized":"2024-01-24T17:44:11.149164+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:36.242202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989015+0000","last_change":"2024-01-24T17:44:10.015889+0000","last_active":"2024-01-24T17:44:10.989015+0000","last_peered":"2024-01-24T17:44:10.989015+0000","last_clean":"2024-01-24T17:44:10.989015+0000","last_became_active":"2024-01-24T17:44:10.015499+0000","last_became_peered":"2024-01-24T17:44:10.015499+0000","last_unstale":"2024-01-24T17:44:10.989015+0000","last_undegraded":"2024-01-24T17:44:10.989015+0000","last_fullsized":"2024-01-24T17:44:10.989015+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:38:11.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147670+0000","last_change":"2024-01-24T17:44:10.001242+0000","last_active":"2024-01-24T17:44:11.147670+0000","last_peered":"2024-01-24T17:44:11.147670+0000","last_clean":"2024-01-24T17:44:11.147670+0000","last_became_active":"2024-01-24T17:44:10.000982+0000","last_became_peered":"2024-01-24T17:44:10.000982+0000","last_unstale":"2024-01-24T17:44:11.147670+0000","last_undegraded":"2024-01-24T17:44:11.147670+0000","last_fullsized":"2024-01-24T17:44:11.147670+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:57:21.200159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985302+0000","last_change":"2024-01-24T17:44:10.015448+0000","last_active":"2024-01-24T17:44:10.985302+0000","last_peered":"2024-01-24T17:44:10.985302+0000","last_clean":"2024-01-24T17:44:10.985302+0000","last_became_active":"2024-01-24T17:44:10.015153+0000","last_became_peered":"2024-01-24T17:44:10.015153+0000","last_unstale":"2024-01-24T17:44:10.985302+0000","last_undegraded":"2024-01-24T17:44:10.985302+0000","last_fullsized":"2024-01-24T17:44:10.985302+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:53.577142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984226+0000","last_change":"2024-01-24T17:44:10.015149+0000","last_active":"2024-01-24T17:44:10.984226+0000","last_peered":"2024-01-24T17:44:10.984226+0000","last_clean":"2024-01-24T17:44:10.984226+0000","last_became_active":"2024-01-24T17:44:10.014886+0000","last_became_peered":"2024-01-24T17:44:10.014886+0000","last_unstale":"2024-01-24T17:44:10.984226+0000","last_undegraded":"2024-01-24T17:44:10.984226+0000","last_fullsized":"2024-01-24T17:44:10.984226+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:36:07.346981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988400+0000","last_change":"2024-01-24T17:44:10.012722+0000","last_active":"2024-01-24T17:44:10.988400+0000","last_peered":"2024-01-24T17:44:10.988400+0000","last_clean":"2024-01-24T17:44:10.988400+0000","last_became_active":"2024-01-24T17:44:10.012384+0000","last_became_peered":"2024-01-24T17:44:10.012384+0000","last_unstale":"2024-01-24T17:44:10.988400+0000","last_undegraded":"2024-01-24T17:44:10.988400+0000","last_fullsized":"2024-01-24T17:44:10.988400+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:36:46.964115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.986259+0000","last_change":"2024-01-24T17:44:10.010919+0000","last_active":"2024-01-24T17:44:10.986259+0000","last_peered":"2024-01-24T17:44:10.986259+0000","last_clean":"2024-01-24T17:44:10.986259+0000","last_became_active":"2024-01-24T17:44:10.010469+0000","last_became_peered":"2024-01-24T17:44:10.010469+0000","last_unstale":"2024-01-24T17:44:10.986259+0000","last_undegraded":"2024-01-24T17:44:10.986259+0000","last_fullsized":"2024-01-24T17:44:10.986259+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:39:07.025772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150876+0000","last_change":"2024-01-24T17:44:10.013124+0000","last_active":"2024-01-24T17:44:11.150876+0000","last_peered":"2024-01-24T17:44:11.150876+0000","last_clean":"2024-01-24T17:44:11.150876+0000","last_became_active":"2024-01-24T17:44:10.011619+0000","last_became_peered":"2024-01-24T17:44:10.011619+0000","last_unstale":"2024-01-24T17:44:11.150876+0000","last_undegraded":"2024-01-24T17:44:11.150876+0000","last_fullsized":"2024-01-24T17:44:11.150876+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:03:55.983870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989013+0000","last_change":"2024-01-24T17:44:10.017844+0000","last_active":"2024-01-24T17:44:10.989013+0000","last_peered":"2024-01-24T17:44:10.989013+0000","last_clean":"2024-01-24T17:44:10.989013+0000","last_became_active":"2024-01-24T17:44:10.017209+0000","last_became_peered":"2024-01-24T17:44:10.017209+0000","last_unstale":"2024-01-24T17:44:10.989013+0000","last_undegraded":"2024-01-24T17:44:10.989013+0000","last_fullsized":"2024-01-24T17:44:10.989013+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:34:38.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149099+0000","last_change":"2024-01-24T17:44:09.989598+0000","last_active":"2024-01-24T17:44:11.149099+0000","last_peered":"2024-01-24T17:44:11.149099+0000","last_clean":"2024-01-24T17:44:11.149099+0000","last_became_active":"2024-01-24T17:44:09.989348+0000","last_became_peered":"2024-01-24T17:44:09.989348+0000","last_unstale":"2024-01-24T17:44:11.149099+0000","last_undegraded":"2024-01-24T17:44:11.149099+0000","last_fullsized":"2024-01-24T17:44:11.149099+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:24:30.205503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985371+0000","last_change":"2024-01-24T17:44:10.004575+0000","last_active":"2024-01-24T17:44:10.985371+0000","last_peered":"2024-01-24T17:44:10.985371+0000","last_clean":"2024-01-24T17:44:10.985371+0000","last_became_active":"2024-01-24T17:44:10.003711+0000","last_became_peered":"2024-01-24T17:44:10.003711+0000","last_unstale":"2024-01-24T17:44:10.985371+0000","last_undegraded":"2024-01-24T17:44:10.985371+0000","last_fullsized":"2024-01-24T17:44:10.985371+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:42.923012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988695+0000","last_change":"2024-01-24T17:44:10.010920+0000","last_active":"2024-01-24T17:44:10.988695+0000","last_peered":"2024-01-24T17:44:10.988695+0000","last_clean":"2024-01-24T17:44:10.988695+0000","last_became_active":"2024-01-24T17:44:10.009961+0000","last_became_peered":"2024-01-24T17:44:10.009961+0000","last_unstale":"2024-01-24T17:44:10.988695+0000","last_undegraded":"2024-01-24T17:44:10.988695+0000","last_fullsized":"2024-01-24T17:44:10.988695+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:26:07.520763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988648+0000","last_change":"2024-01-24T17:44:10.014459+0000","last_active":"2024-01-24T17:44:10.988648+0000","last_peered":"2024-01-24T17:44:10.988648+0000","last_clean":"2024-01-24T17:44:10.988648+0000","last_became_active":"2024-01-24T17:44:10.014045+0000","last_became_peered":"2024-01-24T17:44:10.014045+0000","last_unstale":"2024-01-24T17:44:10.988648+0000","last_undegraded":"2024-01-24T17:44:10.988648+0000","last_fullsized":"2024-01-24T17:44:10.988648+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:48:22.378247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.146976+0000","last_change":"2024-01-24T17:44:10.003350+0000","last_active":"2024-01-24T17:44:11.146976+0000","last_peered":"2024-01-24T17:44:11.146976+0000","last_clean":"2024-01-24T17:44:11.146976+0000","last_became_active":"2024-01-24T17:44:10.003102+0000","last_became_peered":"2024-01-24T17:44:10.003102+0000","last_unstale":"2024-01-24T17:44:11.146976+0000","last_undegraded":"2024-01-24T17:44:11.146976+0000","last_fullsized":"2024-01-24T17:44:11.146976+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:44.246072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149536+0000","last_change":"2024-01-24T17:44:09.992171+0000","last_active":"2024-01-24T17:44:11.149536+0000","last_peered":"2024-01-24T17:44:11.149536+0000","last_clean":"2024-01-24T17:44:11.149536+0000","last_became_active":"2024-01-24T17:44:09.991977+0000","last_became_peered":"2024-01-24T17:44:09.991977+0000","last_unstale":"2024-01-24T17:44:11.149536+0000","last_undegraded":"2024-01-24T17:44:11.149536+0000","last_fullsized":"2024-01-24T17:44:11.149536+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:49.636503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149147+0000","last_change":"2024-01-24T17:44:10.003551+0000","last_active":"2024-01-24T17:44:11.149147+0000","last_peered":"2024-01-24T17:44:11.149147+0000","last_clean":"2024-01-24T17:44:11.149147+0000","last_became_active":"2024-01-24T17:44:10.003330+0000","last_became_peered":"2024-01-24T17:44:10.003330+0000","last_unstale":"2024-01-24T17:44:11.149147+0000","last_undegraded":"2024-01-24T17:44:11.149147+0000","last_fullsized":"2024-01-24T17:44:11.149147+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:24:39.238240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147887+0000","last_change":"2024-01-24T17:44:09.995777+0000","last_active":"2024-01-24T17:44:11.147887+0000","last_peered":"2024-01-24T17:44:11.147887+0000","last_clean":"2024-01-24T17:44:11.147887+0000","last_became_active":"2024-01-24T17:44:09.995322+0000","last_became_peered":"2024-01-24T17:44:09.995322+0000","last_unstale":"2024-01-24T17:44:11.147887+0000","last_undegraded":"2024-01-24T17:44:11.147887+0000","last_fullsized":"2024-01-24T17:44:11.147887+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:38:11.288691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150363+0000","last_change":"2024-01-24T17:44:10.016759+0000","last_active":"2024-01-24T17:44:11.150363+0000","last_peered":"2024-01-24T17:44:11.150363+0000","last_clean":"2024-01-24T17:44:11.150363+0000","last_became_active":"2024-01-24T17:44:10.014917+0000","last_became_peered":"2024-01-24T17:44:10.014917+0000","last_unstale":"2024-01-24T17:44:11.150363+0000","last_undegraded":"2024-01-24T17:44:11.150363+0000","last_fullsized":"2024-01-24T17:44:11.150363+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:46.625289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989196+0000","last_change":"2024-01-24T17:44:10.002380+0000","last_active":"2024-01-24T17:44:10.989196+0000","last_peered":"2024-01-24T17:44:10.989196+0000","last_clean":"2024-01-24T17:44:10.989196+0000","last_became_active":"2024-01-24T17:44:10.002038+0000","last_became_peered":"2024-01-24T17:44:10.002038+0000","last_unstale":"2024-01-24T17:44:10.989196+0000","last_undegraded":"2024-01-24T17:44:10.989196+0000","last_fullsized":"2024-01-24T17:44:10.989196+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:20:03.837528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150101+0000","last_change":"2024-01-24T17:44:10.005209+0000","last_active":"2024-01-24T17:44:11.150101+0000","last_peered":"2024-01-24T17:44:11.150101+0000","last_clean":"2024-01-24T17:44:11.150101+0000","last_became_active":"2024-01-24T17:44:10.004937+0000","last_became_peered":"2024-01-24T17:44:10.004937+0000","last_unstale":"2024-01-24T17:44:11.150101+0000","last_undegraded":"2024-01-24T17:44:11.150101+0000","last_fullsized":"2024-01-24T17:44:11.150101+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:00.666938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984375+0000","last_change":"2024-01-24T17:44:09.992134+0000","last_active":"2024-01-24T17:44:10.984375+0000","last_peered":"2024-01-24T17:44:10.984375+0000","last_clean":"2024-01-24T17:44:10.984375+0000","last_became_active":"2024-01-24T17:44:09.991830+0000","last_became_peered":"2024-01-24T17:44:09.991830+0000","last_unstale":"2024-01-24T17:44:10.984375+0000","last_undegraded":"2024-01-24T17:44:10.984375+0000","last_fullsized":"2024-01-24T17:44:10.984375+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:03:06.693443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147523+0000","last_change":"2024-01-24T17:44:10.005994+0000","last_active":"2024-01-24T17:44:11.147523+0000","last_peered":"2024-01-24T17:44:11.147523+0000","last_clean":"2024-01-24T17:44:11.147523+0000","last_became_active":"2024-01-24T17:44:10.004637+0000","last_became_peered":"2024-01-24T17:44:10.004637+0000","last_unstale":"2024-01-24T17:44:11.147523+0000","last_undegraded":"2024-01-24T17:44:11.147523+0000","last_fullsized":"2024-01-24T17:44:11.147523+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:34:38.208515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147757+0000","last_change":"2024-01-24T17:44:09.988996+0000","last_active":"2024-01-24T17:44:11.147757+0000","last_peered":"2024-01-24T17:44:11.147757+0000","last_clean":"2024-01-24T17:44:11.147757+0000","last_became_active":"2024-01-24T17:44:09.988293+0000","last_became_peered":"2024-01-24T17:44:09.988293+0000","last_unstale":"2024-01-24T17:44:11.147757+0000","last_undegraded":"2024-01-24T17:44:11.147757+0000","last_fullsized":"2024-01-24T17:44:11.147757+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:49.636503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989293+0000","last_change":"2024-01-24T17:44:10.016776+0000","last_active":"2024-01-24T17:44:10.989293+0000","last_peered":"2024-01-24T17:44:10.989293+0000","last_clean":"2024-01-24T17:44:10.989293+0000","last_became_active":"2024-01-24T17:44:10.014859+0000","last_became_peered":"2024-01-24T17:44:10.014859+0000","last_unstale":"2024-01-24T17:44:10.989293+0000","last_undegraded":"2024-01-24T17:44:10.989293+0000","last_fullsized":"2024-01-24T17:44:10.989293+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:49.636503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984553+0000","last_change":"2024-01-24T17:44:10.000542+0000","last_active":"2024-01-24T17:44:10.984553+0000","last_peered":"2024-01-24T17:44:10.984553+0000","last_clean":"2024-01-24T17:44:10.984553+0000","last_became_active":"2024-01-24T17:44:10.000291+0000","last_became_peered":"2024-01-24T17:44:10.000291+0000","last_unstale":"2024-01-24T17:44:10.984553+0000","last_undegraded":"2024-01-24T17:44:10.984553+0000","last_fullsized":"2024-01-24T17:44:10.984553+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:42:25.924608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984749+0000","last_change":"2024-01-24T17:44:09.996598+0000","last_active":"2024-01-24T17:44:10.984749+0000","last_peered":"2024-01-24T17:44:10.984749+0000","last_clean":"2024-01-24T17:44:10.984749+0000","last_became_active":"2024-01-24T17:44:09.996219+0000","last_became_peered":"2024-01-24T17:44:09.996219+0000","last_unstale":"2024-01-24T17:44:10.984749+0000","last_undegraded":"2024-01-24T17:44:10.984749+0000","last_fullsized":"2024-01-24T17:44:10.984749+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:55:44.196397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147738+0000","last_change":"2024-01-24T17:44:10.011794+0000","last_active":"2024-01-24T17:44:11.147738+0000","last_peered":"2024-01-24T17:44:11.147738+0000","last_clean":"2024-01-24T17:44:11.147738+0000","last_became_active":"2024-01-24T17:44:10.011485+0000","last_became_peered":"2024-01-24T17:44:10.011485+0000","last_unstale":"2024-01-24T17:44:11.147738+0000","last_undegraded":"2024-01-24T17:44:11.147738+0000","last_fullsized":"2024-01-24T17:44:11.147738+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:03:16.674585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148319+0000","last_change":"2024-01-24T17:44:09.994315+0000","last_active":"2024-01-24T17:44:11.148319+0000","last_peered":"2024-01-24T17:44:11.148319+0000","last_clean":"2024-01-24T17:44:11.148319+0000","last_became_active":"2024-01-24T17:44:09.994062+0000","last_became_peered":"2024-01-24T17:44:09.994062+0000","last_unstale":"2024-01-24T17:44:11.148319+0000","last_undegraded":"2024-01-24T17:44:11.148319+0000","last_fullsized":"2024-01-24T17:44:11.148319+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:36:46.964115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984854+0000","last_change":"2024-01-24T17:44:09.997144+0000","last_active":"2024-01-24T17:44:10.984854+0000","last_peered":"2024-01-24T17:44:10.984854+0000","last_clean":"2024-01-24T17:44:10.984854+0000","last_became_active":"2024-01-24T17:44:09.996832+0000","last_became_peered":"2024-01-24T17:44:09.996832+0000","last_unstale":"2024-01-24T17:44:10.984854+0000","last_undegraded":"2024-01-24T17:44:10.984854+0000","last_fullsized":"2024-01-24T17:44:10.984854+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:59:47.528588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988461+0000","last_change":"2024-01-24T17:44:10.009997+0000","last_active":"2024-01-24T17:44:10.988461+0000","last_peered":"2024-01-24T17:44:10.988461+0000","last_clean":"2024-01-24T17:44:10.988461+0000","last_became_active":"2024-01-24T17:44:10.006753+0000","last_became_peered":"2024-01-24T17:44:10.006753+0000","last_unstale":"2024-01-24T17:44:10.988461+0000","last_undegraded":"2024-01-24T17:44:10.988461+0000","last_fullsized":"2024-01-24T17:44:10.988461+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:03:16.417542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150439+0000","last_change":"2024-01-24T17:44:10.012288+0000","last_active":"2024-01-24T17:44:11.150439+0000","last_peered":"2024-01-24T17:44:11.150439+0000","last_clean":"2024-01-24T17:44:11.150439+0000","last_became_active":"2024-01-24T17:44:10.012019+0000","last_became_peered":"2024-01-24T17:44:10.012019+0000","last_unstale":"2024-01-24T17:44:11.150439+0000","last_undegraded":"2024-01-24T17:44:11.150439+0000","last_fullsized":"2024-01-24T17:44:11.150439+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:42.923012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.986401+0000","last_change":"2024-01-24T17:44:09.506842+0000","last_active":"2024-01-24T17:44:10.986401+0000","last_peered":"2024-01-24T17:44:10.986401+0000","last_clean":"2024-01-24T17:44:10.986401+0000","last_became_active":"2024-01-24T17:44:09.505346+0000","last_became_peered":"2024-01-24T17:44:09.505346+0000","last_unstale":"2024-01-24T17:44:10.986401+0000","last_undegraded":"2024-01-24T17:44:10.986401+0000","last_fullsized":"2024-01-24T17:44:10.986401+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:35.222103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147873+0000","last_change":"2024-01-24T17:44:09.995977+0000","last_active":"2024-01-24T17:44:11.147873+0000","last_peered":"2024-01-24T17:44:11.147873+0000","last_clean":"2024-01-24T17:44:11.147873+0000","last_became_active":"2024-01-24T17:44:09.995445+0000","last_became_peered":"2024-01-24T17:44:09.995445+0000","last_unstale":"2024-01-24T17:44:11.147873+0000","last_undegraded":"2024-01-24T17:44:11.147873+0000","last_fullsized":"2024-01-24T17:44:11.147873+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:20:03.837528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984539+0000","last_change":"2024-01-24T17:44:09.516513+0000","last_active":"2024-01-24T17:44:10.984539+0000","last_peered":"2024-01-24T17:44:10.984539+0000","last_clean":"2024-01-24T17:44:10.984539+0000","last_became_active":"2024-01-24T17:44:09.516194+0000","last_became_peered":"2024-01-24T17:44:09.516194+0000","last_unstale":"2024-01-24T17:44:10.984539+0000","last_undegraded":"2024-01-24T17:44:10.984539+0000","last_fullsized":"2024-01-24T17:44:10.984539+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:43:45.174522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149189+0000","last_change":"2024-01-24T17:44:10.008307+0000","last_active":"2024-01-24T17:44:11.149189+0000","last_peered":"2024-01-24T17:44:11.149189+0000","last_clean":"2024-01-24T17:44:11.149189+0000","last_became_active":"2024-01-24T17:44:10.007996+0000","last_became_peered":"2024-01-24T17:44:10.007996+0000","last_unstale":"2024-01-24T17:44:11.149189+0000","last_undegraded":"2024-01-24T17:44:11.149189+0000","last_fullsized":"2024-01-24T17:44:11.149189+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:03:16.417542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.986012+0000","last_change":"2024-01-24T17:44:09.505972+0000","last_active":"2024-01-24T17:44:10.986012+0000","last_peered":"2024-01-24T17:44:10.986012+0000","last_clean":"2024-01-24T17:44:10.986012+0000","last_became_active":"2024-01-24T17:44:09.505370+0000","last_became_peered":"2024-01-24T17:44:09.505370+0000","last_unstale":"2024-01-24T17:44:10.986012+0000","last_undegraded":"2024-01-24T17:44:10.986012+0000","last_fullsized":"2024-01-24T17:44:10.986012+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:34:00.438070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989492+0000","last_change":"2024-01-24T17:44:10.016561+0000","last_active":"2024-01-24T17:44:10.989492+0000","last_peered":"2024-01-24T17:44:10.989492+0000","last_clean":"2024-01-24T17:44:10.989492+0000","last_became_active":"2024-01-24T17:44:10.016280+0000","last_became_peered":"2024-01-24T17:44:10.016280+0000","last_unstale":"2024-01-24T17:44:10.989492+0000","last_undegraded":"2024-01-24T17:44:10.989492+0000","last_fullsized":"2024-01-24T17:44:10.989492+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:44.246072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984823+0000","last_change":"2024-01-24T17:44:09.516601+0000","last_active":"2024-01-24T17:44:10.984823+0000","last_peered":"2024-01-24T17:44:10.984823+0000","last_clean":"2024-01-24T17:44:10.984823+0000","last_became_active":"2024-01-24T17:44:09.516310+0000","last_became_peered":"2024-01-24T17:44:09.516310+0000","last_unstale":"2024-01-24T17:44:10.984823+0000","last_undegraded":"2024-01-24T17:44:10.984823+0000","last_fullsized":"2024-01-24T17:44:10.984823+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:26:06.500664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984379+0000","last_change":"2024-01-24T17:44:10.000678+0000","last_active":"2024-01-24T17:44:10.984379+0000","last_peered":"2024-01-24T17:44:10.984379+0000","last_clean":"2024-01-24T17:44:10.984379+0000","last_became_active":"2024-01-24T17:44:10.000440+0000","last_became_peered":"2024-01-24T17:44:10.000440+0000","last_unstale":"2024-01-24T17:44:10.984379+0000","last_undegraded":"2024-01-24T17:44:10.984379+0000","last_fullsized":"2024-01-24T17:44:10.984379+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:15:24.877075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985884+0000","last_change":"2024-01-24T17:44:09.515026+0000","last_active":"2024-01-24T17:44:10.985884+0000","last_peered":"2024-01-24T17:44:10.985884+0000","last_clean":"2024-01-24T17:44:10.985884+0000","last_became_active":"2024-01-24T17:44:09.514761+0000","last_became_peered":"2024-01-24T17:44:09.514761+0000","last_unstale":"2024-01-24T17:44:10.985884+0000","last_undegraded":"2024-01-24T17:44:10.985884+0000","last_fullsized":"2024-01-24T17:44:10.985884+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:21:53.583579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150132+0000","last_change":"2024-01-24T17:44:10.015237+0000","last_active":"2024-01-24T17:44:11.150132+0000","last_peered":"2024-01-24T17:44:11.150132+0000","last_clean":"2024-01-24T17:44:11.150132+0000","last_became_active":"2024-01-24T17:44:10.014945+0000","last_became_peered":"2024-01-24T17:44:10.014945+0000","last_unstale":"2024-01-24T17:44:11.150132+0000","last_undegraded":"2024-01-24T17:44:11.150132+0000","last_fullsized":"2024-01-24T17:44:11.150132+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:26:07.520763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984412+0000","last_change":"2024-01-24T17:44:09.514280+0000","last_active":"2024-01-24T17:44:10.984412+0000","last_peered":"2024-01-24T17:44:10.984412+0000","last_clean":"2024-01-24T17:44:10.984412+0000","last_became_active":"2024-01-24T17:44:09.514009+0000","last_became_peered":"2024-01-24T17:44:09.514009+0000","last_unstale":"2024-01-24T17:44:10.984412+0000","last_undegraded":"2024-01-24T17:44:10.984412+0000","last_fullsized":"2024-01-24T17:44:10.984412+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:04:33.162077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149111+0000","last_change":"2024-01-24T17:44:10.009719+0000","last_active":"2024-01-24T17:44:11.149111+0000","last_peered":"2024-01-24T17:44:11.149111+0000","last_clean":"2024-01-24T17:44:11.149111+0000","last_became_active":"2024-01-24T17:44:10.006522+0000","last_became_peered":"2024-01-24T17:44:10.006522+0000","last_unstale":"2024-01-24T17:44:11.149111+0000","last_undegraded":"2024-01-24T17:44:11.149111+0000","last_fullsized":"2024-01-24T17:44:11.149111+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:03:16.417542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147187+0000","last_change":"2024-01-24T17:44:09.508753+0000","last_active":"2024-01-24T17:44:11.147187+0000","last_peered":"2024-01-24T17:44:11.147187+0000","last_clean":"2024-01-24T17:44:11.147187+0000","last_became_active":"2024-01-24T17:44:09.508490+0000","last_became_peered":"2024-01-24T17:44:09.508490+0000","last_unstale":"2024-01-24T17:44:11.147187+0000","last_undegraded":"2024-01-24T17:44:11.147187+0000","last_fullsized":"2024-01-24T17:44:11.147187+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:57:31.510691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984359+0000","last_change":"2024-01-24T17:44:10.004739+0000","last_active":"2024-01-24T17:44:10.984359+0000","last_peered":"2024-01-24T17:44:10.984359+0000","last_clean":"2024-01-24T17:44:10.984359+0000","last_became_active":"2024-01-24T17:44:10.004456+0000","last_became_peered":"2024-01-24T17:44:10.004456+0000","last_unstale":"2024-01-24T17:44:10.984359+0000","last_undegraded":"2024-01-24T17:44:10.984359+0000","last_fullsized":"2024-01-24T17:44:10.984359+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:25:33.425011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149358+0000","last_change":"2024-01-24T17:44:09.514515+0000","last_active":"2024-01-24T17:44:11.149358+0000","last_peered":"2024-01-24T17:44:11.149358+0000","last_clean":"2024-01-24T17:44:11.149358+0000","last_became_active":"2024-01-24T17:44:09.514229+0000","last_became_peered":"2024-01-24T17:44:09.514229+0000","last_unstale":"2024-01-24T17:44:11.149358+0000","last_undegraded":"2024-01-24T17:44:11.149358+0000","last_fullsized":"2024-01-24T17:44:11.149358+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:31:50.257431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149248+0000","last_change":"2024-01-24T17:44:10.007311+0000","last_active":"2024-01-24T17:44:11.149248+0000","last_peered":"2024-01-24T17:44:11.149248+0000","last_clean":"2024-01-24T17:44:11.149248+0000","last_became_active":"2024-01-24T17:44:10.007081+0000","last_became_peered":"2024-01-24T17:44:10.007081+0000","last_unstale":"2024-01-24T17:44:11.149248+0000","last_undegraded":"2024-01-24T17:44:11.149248+0000","last_fullsized":"2024-01-24T17:44:11.149248+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:13.437856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149711+0000","last_change":"2024-01-24T17:44:09.519719+0000","last_active":"2024-01-24T17:44:11.149711+0000","last_peered":"2024-01-24T17:44:11.149711+0000","last_clean":"2024-01-24T17:44:11.149711+0000","last_became_active":"2024-01-24T17:44:09.519225+0000","last_became_peered":"2024-01-24T17:44:09.519225+0000","last_unstale":"2024-01-24T17:44:11.149711+0000","last_undegraded":"2024-01-24T17:44:11.149711+0000","last_fullsized":"2024-01-24T17:44:11.149711+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:16:51.776116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149539+0000","last_change":"2024-01-24T17:44:09.993333+0000","last_active":"2024-01-24T17:44:11.149539+0000","last_peered":"2024-01-24T17:44:11.149539+0000","last_clean":"2024-01-24T17:44:11.149539+0000","last_became_active":"2024-01-24T17:44:09.993051+0000","last_became_peered":"2024-01-24T17:44:09.993051+0000","last_unstale":"2024-01-24T17:44:11.149539+0000","last_undegraded":"2024-01-24T17:44:11.149539+0000","last_fullsized":"2024-01-24T17:44:11.149539+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:15:30.995111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150900+0000","last_change":"2024-01-24T17:44:09.514538+0000","last_active":"2024-01-24T17:44:11.150900+0000","last_peered":"2024-01-24T17:44:11.150900+0000","last_clean":"2024-01-24T17:44:11.150900+0000","last_became_active":"2024-01-24T17:44:09.513829+0000","last_became_peered":"2024-01-24T17:44:09.513829+0000","last_unstale":"2024-01-24T17:44:11.150900+0000","last_undegraded":"2024-01-24T17:44:11.150900+0000","last_fullsized":"2024-01-24T17:44:11.150900+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:45:02.258807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988976+0000","last_change":"2024-01-24T17:44:10.012379+0000","last_active":"2024-01-24T17:44:10.988976+0000","last_peered":"2024-01-24T17:44:10.988976+0000","last_clean":"2024-01-24T17:44:10.988976+0000","last_became_active":"2024-01-24T17:44:10.012070+0000","last_became_peered":"2024-01-24T17:44:10.012070+0000","last_unstale":"2024-01-24T17:44:10.988976+0000","last_undegraded":"2024-01-24T17:44:10.988976+0000","last_fullsized":"2024-01-24T17:44:10.988976+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:24:30.205503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148940+0000","last_change":"2024-01-24T17:44:09.517515+0000","last_active":"2024-01-24T17:44:11.148940+0000","last_peered":"2024-01-24T17:44:11.148940+0000","last_clean":"2024-01-24T17:44:11.148940+0000","last_became_active":"2024-01-24T17:44:09.517247+0000","last_became_peered":"2024-01-24T17:44:09.517247+0000","last_unstale":"2024-01-24T17:44:11.148940+0000","last_undegraded":"2024-01-24T17:44:11.148940+0000","last_fullsized":"2024-01-24T17:44:11.148940+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:15:59.646839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984632+0000","last_change":"2024-01-24T17:44:10.015368+0000","last_active":"2024-01-24T17:44:10.984632+0000","last_peered":"2024-01-24T17:44:10.984632+0000","last_clean":"2024-01-24T17:44:10.984632+0000","last_became_active":"2024-01-24T17:44:10.014733+0000","last_became_peered":"2024-01-24T17:44:10.014733+0000","last_unstale":"2024-01-24T17:44:10.984632+0000","last_undegraded":"2024-01-24T17:44:10.984632+0000","last_fullsized":"2024-01-24T17:44:10.984632+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:50:12.903702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150613+0000","last_change":"2024-01-24T17:44:09.514688+0000","last_active":"2024-01-24T17:44:11.150613+0000","last_peered":"2024-01-24T17:44:11.150613+0000","last_clean":"2024-01-24T17:44:11.150613+0000","last_became_active":"2024-01-24T17:44:09.514165+0000","last_became_peered":"2024-01-24T17:44:09.514165+0000","last_unstale":"2024-01-24T17:44:11.150613+0000","last_undegraded":"2024-01-24T17:44:11.150613+0000","last_fullsized":"2024-01-24T17:44:11.150613+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:43.932754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149035+0000","last_change":"2024-01-24T17:44:10.003601+0000","last_active":"2024-01-24T17:44:11.149035+0000","last_peered":"2024-01-24T17:44:11.149035+0000","last_clean":"2024-01-24T17:44:11.149035+0000","last_became_active":"2024-01-24T17:44:10.003327+0000","last_became_peered":"2024-01-24T17:44:10.003327+0000","last_unstale":"2024-01-24T17:44:11.149035+0000","last_undegraded":"2024-01-24T17:44:11.149035+0000","last_fullsized":"2024-01-24T17:44:11.149035+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:21:54.603678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149180+0000","last_change":"2024-01-24T17:44:09.521600+0000","last_active":"2024-01-24T17:44:11.149180+0000","last_peered":"2024-01-24T17:44:11.149180+0000","last_clean":"2024-01-24T17:44:11.149180+0000","last_became_active":"2024-01-24T17:44:09.521098+0000","last_became_peered":"2024-01-24T17:44:09.521098+0000","last_unstale":"2024-01-24T17:44:11.149180+0000","last_undegraded":"2024-01-24T17:44:11.149180+0000","last_fullsized":"2024-01-24T17:44:11.149180+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:34:00.438070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149249+0000","last_change":"2024-01-24T17:44:09.993859+0000","last_active":"2024-01-24T17:44:11.149249+0000","last_peered":"2024-01-24T17:44:11.149249+0000","last_clean":"2024-01-24T17:44:11.149249+0000","last_became_active":"2024-01-24T17:44:09.993583+0000","last_became_peered":"2024-01-24T17:44:09.993583+0000","last_unstale":"2024-01-24T17:44:11.149249+0000","last_undegraded":"2024-01-24T17:44:11.149249+0000","last_fullsized":"2024-01-24T17:44:11.149249+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:59:47.528588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147366+0000","last_change":"2024-01-24T17:44:09.512507+0000","last_active":"2024-01-24T17:44:11.147366+0000","last_peered":"2024-01-24T17:44:11.147366+0000","last_clean":"2024-01-24T17:44:11.147366+0000","last_became_active":"2024-01-24T17:44:09.512192+0000","last_became_peered":"2024-01-24T17:44:09.512192+0000","last_unstale":"2024-01-24T17:44:11.147366+0000","last_undegraded":"2024-01-24T17:44:11.147366+0000","last_fullsized":"2024-01-24T17:44:11.147366+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:41.902913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149040+0000","last_change":"2024-01-24T17:44:10.009242+0000","last_active":"2024-01-24T17:44:11.149040+0000","last_peered":"2024-01-24T17:44:11.149040+0000","last_clean":"2024-01-24T17:44:11.149040+0000","last_became_active":"2024-01-24T17:44:10.008963+0000","last_became_peered":"2024-01-24T17:44:10.008963+0000","last_unstale":"2024-01-24T17:44:11.149040+0000","last_undegraded":"2024-01-24T17:44:11.149040+0000","last_fullsized":"2024-01-24T17:44:11.149040+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:43.106803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984928+0000","last_change":"2024-01-24T17:44:09.511291+0000","last_active":"2024-01-24T17:44:10.984928+0000","last_peered":"2024-01-24T17:44:10.984928+0000","last_clean":"2024-01-24T17:44:10.984928+0000","last_became_active":"2024-01-24T17:44:09.510821+0000","last_became_peered":"2024-01-24T17:44:09.510821+0000","last_unstale":"2024-01-24T17:44:10.984928+0000","last_undegraded":"2024-01-24T17:44:10.984928+0000","last_fullsized":"2024-01-24T17:44:10.984928+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:12.417757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148091+0000","last_change":"2024-01-24T17:44:09.983510+0000","last_active":"2024-01-24T17:44:11.148091+0000","last_peered":"2024-01-24T17:44:11.148091+0000","last_clean":"2024-01-24T17:44:11.148091+0000","last_became_active":"2024-01-24T17:44:09.982911+0000","last_became_peered":"2024-01-24T17:44:09.982911+0000","last_unstale":"2024-01-24T17:44:11.148091+0000","last_undegraded":"2024-01-24T17:44:11.148091+0000","last_fullsized":"2024-01-24T17:44:11.148091+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:44.246072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":19,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149615+0000","last_change":"2024-01-24T17:44:09.513971+0000","last_active":"2024-01-24T17:44:11.149615+0000","last_peered":"2024-01-24T17:44:11.149615+0000","last_clean":"2024-01-24T17:44:11.149615+0000","last_became_active":"2024-01-24T17:44:09.513487+0000","last_became_peered":"2024-01-24T17:44:09.513487+0000","last_unstale":"2024-01-24T17:44:11.149615+0000","last_undegraded":"2024-01-24T17:44:11.149615+0000","last_fullsized":"2024-01-24T17:44:11.149615+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:52.557043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150683+0000","last_change":"2024-01-24T17:44:10.001468+0000","last_active":"2024-01-24T17:44:11.150683+0000","last_peered":"2024-01-24T17:44:11.150683+0000","last_clean":"2024-01-24T17:44:11.150683+0000","last_became_active":"2024-01-24T17:44:10.001189+0000","last_became_peered":"2024-01-24T17:44:10.001189+0000","last_unstale":"2024-01-24T17:44:11.150683+0000","last_undegraded":"2024-01-24T17:44:11.150683+0000","last_fullsized":"2024-01-24T17:44:11.150683+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:39:07.025772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989401+0000","last_change":"2024-01-24T17:44:09.516892+0000","last_active":"2024-01-24T17:44:10.989401+0000","last_peered":"2024-01-24T17:44:10.989401+0000","last_clean":"2024-01-24T17:44:10.989401+0000","last_became_active":"2024-01-24T17:44:09.513860+0000","last_became_peered":"2024-01-24T17:44:09.513860+0000","last_unstale":"2024-01-24T17:44:10.989401+0000","last_undegraded":"2024-01-24T17:44:10.989401+0000","last_fullsized":"2024-01-24T17:44:10.989401+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:12.417757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149564+0000","last_change":"2024-01-24T17:44:10.009941+0000","last_active":"2024-01-24T17:44:11.149564+0000","last_peered":"2024-01-24T17:44:11.149564+0000","last_clean":"2024-01-24T17:44:11.149564+0000","last_became_active":"2024-01-24T17:44:10.009473+0000","last_became_peered":"2024-01-24T17:44:10.009473+0000","last_unstale":"2024-01-24T17:44:11.149564+0000","last_undegraded":"2024-01-24T17:44:11.149564+0000","last_fullsized":"2024-01-24T17:44:11.149564+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:53.577142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149346+0000","last_change":"2024-01-24T17:44:09.509070+0000","last_active":"2024-01-24T17:44:11.149346+0000","last_peered":"2024-01-24T17:44:11.149346+0000","last_clean":"2024-01-24T17:44:11.149346+0000","last_became_active":"2024-01-24T17:44:09.508731+0000","last_became_peered":"2024-01-24T17:44:09.508731+0000","last_unstale":"2024-01-24T17:44:11.149346+0000","last_undegraded":"2024-01-24T17:44:11.149346+0000","last_fullsized":"2024-01-24T17:44:11.149346+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:20:14.765740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149288+0000","last_change":"2024-01-24T17:44:10.013975+0000","last_active":"2024-01-24T17:44:11.149288+0000","last_peered":"2024-01-24T17:44:11.149288+0000","last_clean":"2024-01-24T17:44:11.149288+0000","last_became_active":"2024-01-24T17:44:10.013749+0000","last_became_peered":"2024-01-24T17:44:10.013749+0000","last_unstale":"2024-01-24T17:44:11.149288+0000","last_undegraded":"2024-01-24T17:44:11.149288+0000","last_fullsized":"2024-01-24T17:44:11.149288+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:26:07.520763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149439+0000","last_change":"2024-01-24T17:44:09.511864+0000","last_active":"2024-01-24T17:44:11.149439+0000","last_peered":"2024-01-24T17:44:11.149439+0000","last_clean":"2024-01-24T17:44:11.149439+0000","last_became_active":"2024-01-24T17:44:09.511481+0000","last_became_peered":"2024-01-24T17:44:09.511481+0000","last_unstale":"2024-01-24T17:44:11.149439+0000","last_undegraded":"2024-01-24T17:44:11.149439+0000","last_fullsized":"2024-01-24T17:44:11.149439+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:14:29.720521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149305+0000","last_change":"2024-01-24T17:44:10.007141+0000","last_active":"2024-01-24T17:44:11.149305+0000","last_peered":"2024-01-24T17:44:11.149305+0000","last_clean":"2024-01-24T17:44:11.149305+0000","last_became_active":"2024-01-24T17:44:10.006297+0000","last_became_peered":"2024-01-24T17:44:10.006297+0000","last_unstale":"2024-01-24T17:44:11.149305+0000","last_undegraded":"2024-01-24T17:44:11.149305+0000","last_fullsized":"2024-01-24T17:44:11.149305+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:33:47.593357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985606+0000","last_change":"2024-01-24T17:44:09.506825+0000","last_active":"2024-01-24T17:44:10.985606+0000","last_peered":"2024-01-24T17:44:10.985606+0000","last_clean":"2024-01-24T17:44:10.985606+0000","last_became_active":"2024-01-24T17:44:09.505893+0000","last_became_peered":"2024-01-24T17:44:09.505893+0000","last_unstale":"2024-01-24T17:44:10.985606+0000","last_undegraded":"2024-01-24T17:44:10.985606+0000","last_fullsized":"2024-01-24T17:44:10.985606+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:24:38.218141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149824+0000","last_change":"2024-01-24T17:44:10.006503+0000","last_active":"2024-01-24T17:44:11.149824+0000","last_peered":"2024-01-24T17:44:11.149824+0000","last_clean":"2024-01-24T17:44:11.149824+0000","last_became_active":"2024-01-24T17:44:10.006033+0000","last_became_peered":"2024-01-24T17:44:10.006033+0000","last_unstale":"2024-01-24T17:44:11.149824+0000","last_undegraded":"2024-01-24T17:44:11.149824+0000","last_fullsized":"2024-01-24T17:44:11.149824+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:04:34.182176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150142+0000","last_change":"2024-01-24T17:44:09.513431+0000","last_active":"2024-01-24T17:44:11.150142+0000","last_peered":"2024-01-24T17:44:11.150142+0000","last_clean":"2024-01-24T17:44:11.150142+0000","last_became_active":"2024-01-24T17:44:09.513127+0000","last_became_peered":"2024-01-24T17:44:09.513127+0000","last_unstale":"2024-01-24T17:44:11.150142+0000","last_undegraded":"2024-01-24T17:44:11.150142+0000","last_fullsized":"2024-01-24T17:44:11.150142+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:36:45.944016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985622+0000","last_change":"2024-01-24T17:44:09.999680+0000","last_active":"2024-01-24T17:44:10.985622+0000","last_peered":"2024-01-24T17:44:10.985622+0000","last_clean":"2024-01-24T17:44:10.985622+0000","last_became_active":"2024-01-24T17:44:09.999261+0000","last_became_peered":"2024-01-24T17:44:09.999261+0000","last_unstale":"2024-01-24T17:44:10.985622+0000","last_undegraded":"2024-01-24T17:44:10.985622+0000","last_fullsized":"2024-01-24T17:44:10.985622+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:16:00.666938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147422+0000","last_change":"2024-01-24T17:44:09.512707+0000","last_active":"2024-01-24T17:44:11.147422+0000","last_peered":"2024-01-24T17:44:11.147422+0000","last_clean":"2024-01-24T17:44:11.147422+0000","last_became_active":"2024-01-24T17:44:09.511352+0000","last_became_peered":"2024-01-24T17:44:09.511352+0000","last_unstale":"2024-01-24T17:44:11.147422+0000","last_undegraded":"2024-01-24T17:44:11.147422+0000","last_fullsized":"2024-01-24T17:44:11.147422+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:59:46.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988665+0000","last_change":"2024-01-24T17:44:10.005348+0000","last_active":"2024-01-24T17:44:10.988665+0000","last_peered":"2024-01-24T17:44:10.988665+0000","last_clean":"2024-01-24T17:44:10.988665+0000","last_became_active":"2024-01-24T17:44:10.005038+0000","last_became_peered":"2024-01-24T17:44:10.005038+0000","last_unstale":"2024-01-24T17:44:10.988665+0000","last_undegraded":"2024-01-24T17:44:10.988665+0000","last_fullsized":"2024-01-24T17:44:10.988665+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:57:21.200159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149488+0000","last_change":"2024-01-24T17:44:09.513261+0000","last_active":"2024-01-24T17:44:11.149488+0000","last_peered":"2024-01-24T17:44:11.149488+0000","last_clean":"2024-01-24T17:44:11.149488+0000","last_became_active":"2024-01-24T17:44:09.511938+0000","last_became_peered":"2024-01-24T17:44:09.511938+0000","last_unstale":"2024-01-24T17:44:11.149488+0000","last_undegraded":"2024-01-24T17:44:11.149488+0000","last_fullsized":"2024-01-24T17:44:11.149488+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:27:15.941486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988457+0000","last_change":"2024-01-24T17:44:10.009146+0000","last_active":"2024-01-24T17:44:10.988457+0000","last_peered":"2024-01-24T17:44:10.988457+0000","last_clean":"2024-01-24T17:44:10.988457+0000","last_became_active":"2024-01-24T17:44:10.008107+0000","last_became_peered":"2024-01-24T17:44:10.008107+0000","last_unstale":"2024-01-24T17:44:10.988457+0000","last_undegraded":"2024-01-24T17:44:10.988457+0000","last_fullsized":"2024-01-24T17:44:10.988457+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:43:46.194621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984203+0000","last_change":"2024-01-24T17:44:09.511280+0000","last_active":"2024-01-24T17:44:10.984203+0000","last_peered":"2024-01-24T17:44:10.984203+0000","last_clean":"2024-01-24T17:44:10.984203+0000","last_became_active":"2024-01-24T17:44:09.511034+0000","last_became_peered":"2024-01-24T17:44:09.511034+0000","last_unstale":"2024-01-24T17:44:10.984203+0000","last_undegraded":"2024-01-24T17:44:10.984203+0000","last_fullsized":"2024-01-24T17:44:10.984203+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:24:29.185403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984731+0000","last_change":"2024-01-24T17:44:10.005054+0000","last_active":"2024-01-24T17:44:10.984731+0000","last_peered":"2024-01-24T17:44:10.984731+0000","last_clean":"2024-01-24T17:44:10.984731+0000","last_became_active":"2024-01-24T17:44:10.004484+0000","last_became_peered":"2024-01-24T17:44:10.004484+0000","last_unstale":"2024-01-24T17:44:10.984731+0000","last_undegraded":"2024-01-24T17:44:10.984731+0000","last_fullsized":"2024-01-24T17:44:10.984731+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:37:01.164705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989459+0000","last_change":"2024-01-24T17:44:09.514456+0000","last_active":"2024-01-24T17:44:10.989459+0000","last_peered":"2024-01-24T17:44:10.989459+0000","last_clean":"2024-01-24T17:44:10.989459+0000","last_became_active":"2024-01-24T17:44:09.514131+0000","last_became_peered":"2024-01-24T17:44:09.514131+0000","last_unstale":"2024-01-24T17:44:10.989459+0000","last_undegraded":"2024-01-24T17:44:10.989459+0000","last_fullsized":"2024-01-24T17:44:10.989459+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:24:38.218141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.150467+0000","last_change":"2024-01-24T17:44:10.006999+0000","last_active":"2024-01-24T17:44:11.150467+0000","last_peered":"2024-01-24T17:44:11.150467+0000","last_clean":"2024-01-24T17:44:11.150467+0000","last_became_active":"2024-01-24T17:44:10.006619+0000","last_became_peered":"2024-01-24T17:44:10.006619+0000","last_unstale":"2024-01-24T17:44:11.150467+0000","last_undegraded":"2024-01-24T17:44:11.150467+0000","last_fullsized":"2024-01-24T17:44:11.150467+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:24:30.205503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147752+0000","last_change":"2024-01-24T17:44:09.510989+0000","last_active":"2024-01-24T17:44:11.147752+0000","last_peered":"2024-01-24T17:44:11.147752+0000","last_clean":"2024-01-24T17:44:11.147752+0000","last_became_active":"2024-01-24T17:44:09.510665+0000","last_became_peered":"2024-01-24T17:44:09.510665+0000","last_unstale":"2024-01-24T17:44:11.147752+0000","last_undegraded":"2024-01-24T17:44:11.147752+0000","last_fullsized":"2024-01-24T17:44:11.147752+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:39:06.005673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984662+0000","last_change":"2024-01-24T17:44:10.015028+0000","last_active":"2024-01-24T17:44:10.984662+0000","last_peered":"2024-01-24T17:44:10.984662+0000","last_clean":"2024-01-24T17:44:10.984662+0000","last_became_active":"2024-01-24T17:44:10.014716+0000","last_became_peered":"2024-01-24T17:44:10.014716+0000","last_unstale":"2024-01-24T17:44:10.984662+0000","last_undegraded":"2024-01-24T17:44:10.984662+0000","last_fullsized":"2024-01-24T17:44:10.984662+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:19:35.252541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149741+0000","last_change":"2024-01-24T17:44:09.519257+0000","last_active":"2024-01-24T17:44:11.149741+0000","last_peered":"2024-01-24T17:44:11.149741+0000","last_clean":"2024-01-24T17:44:11.149741+0000","last_became_active":"2024-01-24T17:44:09.518939+0000","last_became_peered":"2024-01-24T17:44:09.518939+0000","last_unstale":"2024-01-24T17:44:11.149741+0000","last_undegraded":"2024-01-24T17:44:11.149741+0000","last_fullsized":"2024-01-24T17:44:11.149741+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:35.222103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149745+0000","last_change":"2024-01-24T17:44:10.006287+0000","last_active":"2024-01-24T17:44:11.149745+0000","last_peered":"2024-01-24T17:44:11.149745+0000","last_clean":"2024-01-24T17:44:11.149745+0000","last_became_active":"2024-01-24T17:44:10.005637+0000","last_became_peered":"2024-01-24T17:44:10.005637+0000","last_unstale":"2024-01-24T17:44:11.149745+0000","last_undegraded":"2024-01-24T17:44:11.149745+0000","last_fullsized":"2024-01-24T17:44:11.149745+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:43:46.194621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.147780+0000","last_change":"2024-01-24T17:44:09.511553+0000","last_active":"2024-01-24T17:44:11.147780+0000","last_peered":"2024-01-24T17:44:11.147780+0000","last_clean":"2024-01-24T17:44:11.147780+0000","last_became_active":"2024-01-24T17:44:09.511237+0000","last_became_peered":"2024-01-24T17:44:09.511237+0000","last_unstale":"2024-01-24T17:44:11.147780+0000","last_undegraded":"2024-01-24T17:44:11.147780+0000","last_fullsized":"2024-01-24T17:44:11.147780+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:42.086704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.988995+0000","last_change":"2024-01-24T17:44:10.002700+0000","last_active":"2024-01-24T17:44:10.988995+0000","last_peered":"2024-01-24T17:44:10.988995+0000","last_clean":"2024-01-24T17:44:10.988995+0000","last_became_active":"2024-01-24T17:44:10.002169+0000","last_became_peered":"2024-01-24T17:44:10.002169+0000","last_unstale":"2024-01-24T17:44:10.988995+0000","last_undegraded":"2024-01-24T17:44:10.988995+0000","last_fullsized":"2024-01-24T17:44:10.988995+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:03:16.674585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984479+0000","last_change":"2024-01-24T17:44:09.514378+0000","last_active":"2024-01-24T17:44:10.984479+0000","last_peered":"2024-01-24T17:44:10.984479+0000","last_clean":"2024-01-24T17:44:10.984479+0000","last_became_active":"2024-01-24T17:44:09.514073+0000","last_became_peered":"2024-01-24T17:44:09.514073+0000","last_unstale":"2024-01-24T17:44:10.984479+0000","last_undegraded":"2024-01-24T17:44:10.984479+0000","last_fullsized":"2024-01-24T17:44:10.984479+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:03:54.963771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148435+0000","last_change":"2024-01-24T17:44:10.015660+0000","last_active":"2024-01-24T17:44:11.148435+0000","last_peered":"2024-01-24T17:44:11.148435+0000","last_clean":"2024-01-24T17:44:11.148435+0000","last_became_active":"2024-01-24T17:44:10.015193+0000","last_became_peered":"2024-01-24T17:44:10.015193+0000","last_unstale":"2024-01-24T17:44:11.148435+0000","last_undegraded":"2024-01-24T17:44:11.148435+0000","last_fullsized":"2024-01-24T17:44:11.148435+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:34:01.458169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.985124+0000","last_change":"2024-01-24T17:44:09.511479+0000","last_active":"2024-01-24T17:44:10.985124+0000","last_peered":"2024-01-24T17:44:10.985124+0000","last_clean":"2024-01-24T17:44:10.985124+0000","last_became_active":"2024-01-24T17:44:09.511203+0000","last_became_peered":"2024-01-24T17:44:09.511203+0000","last_unstale":"2024-01-24T17:44:10.985124+0000","last_undegraded":"2024-01-24T17:44:10.985124+0000","last_fullsized":"2024-01-24T17:44:10.985124+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:33:46.573258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.148292+0000","last_change":"2024-01-24T17:44:10.000607+0000","last_active":"2024-01-24T17:44:11.148292+0000","last_peered":"2024-01-24T17:44:11.148292+0000","last_clean":"2024-01-24T17:44:11.148292+0000","last_became_active":"2024-01-24T17:44:10.000359+0000","last_became_peered":"2024-01-24T17:44:10.000359+0000","last_unstale":"2024-01-24T17:44:11.148292+0000","last_undegraded":"2024-01-24T17:44:11.148292+0000","last_fullsized":"2024-01-24T17:44:11.148292+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:03:16.674585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:11.149265+0000","last_change":"2024-01-24T17:44:09.522112+0000","last_active":"2024-01-24T17:44:11.149265+0000","last_peered":"2024-01-24T17:44:11.149265+0000","last_clean":"2024-01-24T17:44:11.149265+0000","last_became_active":"2024-01-24T17:44:09.521810+0000","last_became_peered":"2024-01-24T17:44:09.521810+0000","last_unstale":"2024-01-24T17:44:11.149265+0000","last_undegraded":"2024-01-24T17:44:11.149265+0000","last_fullsized":"2024-01-24T17:44:11.149265+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:21:53.583579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.989385+0000","last_change":"2024-01-24T17:44:10.007584+0000","last_active":"2024-01-24T17:44:10.989385+0000","last_peered":"2024-01-24T17:44:10.989385+0000","last_clean":"2024-01-24T17:44:10.989385+0000","last_became_active":"2024-01-24T17:44:10.005905+0000","last_became_peered":"2024-01-24T17:44:10.005905+0000","last_unstale":"2024-01-24T17:44:10.989385+0000","last_undegraded":"2024-01-24T17:44:10.989385+0000","last_fullsized":"2024-01-24T17:44:10.989385+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:09.461162+0000","last_clean_scrub_stamp":"2024-01-24T17:44:09.461162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:45:03.278906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":13,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-01-24T17:44:10.984785+0000","last_change":"2024-01-24T17:44:09.517800+0000","last_active":"2024-01-24T17:44:10.984785+0000","last_peered":"2024-01-24T17:44:10.984785+0000","last_clean":"2024-01-24T17:44:10.984785+0000","last_became_active":"2024-01-24T17:44:09.517546+0000","last_became_peered":"2024-01-24T17:44:09.517546+0000","last_unstale":"2024-01-24T17:44:10.984785+0000","last_undegraded":"2024-01-24T17:44:10.984785+0000","last_fullsized":"2024-01-24T17:44:10.984785+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:44:08.441063+0000","last_clean_scrub_stamp":"2024-01-24T17:44:08.441063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:03:15.397443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-24T17:44:09.988868+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:44:09.988868+0000","last_peered":"2024-01-24T17:44:09.988868+0000","last_clean":"2024-01-24T17:44:09.988868+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:44:09.988868+0000","last_undegraded":"2024-01-24T17:44:09.988868+0000","last_fullsized":"2024-01-24T17:44:09.988868+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":569344,"data_stored":232448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.0329999999999999,"15min":1.0189999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.028,"15min":1.026},"min":{"1min":0.75800000000000001,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.94499999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.087,"15min":1.1000000000000001},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.446,"5min":1.655,"15min":1.655},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1839999999999999,"15min":1.1919999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.8109999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.649}]},{"osd":2,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.165,"15min":1.1559999999999999},"min":{"1min":0.879,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.5660000000000001,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.008},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.204,"15min":1.226},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.321,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.141}]},{"osd":3,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.244,"15min":1.252},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.5529999999999999,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.173},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3009999999999999,"15min":1.3160000000000001},"min":{"1min":0.93400000000000005,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.464,"5min":2.5760000000000001,"15min":2.5760000000000001},"last":1.242}]},{"osd":4,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.302,"15min":1.3220000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.6299999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.319},{"interface":"front","average":{"1min":1.431,"5min":1.4379999999999999,"15min":1.4390000000000001},"min":{"1min":0.92500000000000004,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":1.774,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.288}]},{"osd":5,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.347,"15min":1.361},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7270000000000001,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.4319999999999999,"15min":1.446},"min":{"1min":0.94599999999999995,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.9510000000000001,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.528}]},{"osd":6,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.5269999999999999,"15min":1.5449999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.0310000000000001,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.4910000000000001,"15min":1.494},"min":{"1min":1.0569999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.9330000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.3600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":573440,"data_stored":233030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0229999999999999,"15min":1.0209999999999999},"min":{"1min":0.623,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.77,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.018,"15min":1.01},"min":{"1min":0.69499999999999995,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.272}]},{"osd":1,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":0.997,"15min":0.98299999999999998},"min":{"1min":0.66700000000000004,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.1220000000000001,"15min":1.1140000000000001},"min":{"1min":0.78600000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.944,"5min":1.972,"15min":1.972},"last":1.234}]},{"osd":2,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1439999999999999,"15min":1.137},"min":{"1min":0.70399999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6850000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1579999999999999,"15min":1.163},"min":{"1min":0.78200000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.534,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.446}]},{"osd":3,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.2370000000000001,"15min":1.2529999999999999},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.0009999999999999,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.155},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2430000000000001,"15min":1.2569999999999999},"min":{"1min":0.81899999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5249999999999999,"5min":2.46,"15min":2.46},"last":1.357}]},{"osd":4,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.236,"15min":1.23},"min":{"1min":0.76000000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.643,"5min":1.988,"15min":1.988},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.3,"15min":1.3029999999999999},"min":{"1min":0.80500000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5349999999999999,"5min":1.849,"15min":1.849},"last":1.1850000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.288,"15min":1.29},"min":{"1min":0.78100000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.538,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.323},{"interface":"front","average":{"1min":1.419,"5min":1.4350000000000001,"15min":1.4379999999999999},"min":{"1min":1.091,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.8680000000000001,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.4970000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:43:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3600000000000001,"15min":1.369},"min":{"1min":1.0549999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.8,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.381},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4570000000000001,"15min":1.464},"min":{"1min":0.88900000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.9019999999999999,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.556}]}]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1163264,"data_stored":822804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.097,"15min":1.109},"min":{"1min":0.64100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":1.508,"15min":1.508},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0349999999999999,"15min":1.0329999999999999},"min":{"1min":0.74399999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.0469999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0760000000000001,"15min":1.0740000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.254,"15min":1.2749999999999999},"min":{"1min":0.77000000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.5549999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.306}]},{"osd":2,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.246,"15min":1.2370000000000001},"min":{"1min":0.79400000000000004,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.655,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.3,"15min":1.319},"min":{"1min":0.67600000000000005,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.8380000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.093}]},{"osd":3,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.3360000000000001,"15min":1.3560000000000001},"min":{"1min":0.81499999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.0150000000000001,"5min":3.036,"15min":3.036},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3320000000000001,"15min":1.3400000000000001},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.7290000000000001,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.5580000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.454,"15min":1.46},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":1.72},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.363,"15min":1.371},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":3.2810000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.2210000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3560000000000001,"15min":1.3620000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":1.425},{"interface":"front","average":{"1min":1.516,"5min":1.5289999999999999,"15min":1.5309999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1579999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.8089999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:43:26 2024","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.492,"15min":1.504},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":3.161,"5min":3.161,"15min":3.161},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.6240000000000001,"15min":1.6459999999999999},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.0139999999999998,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.5129999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":585728,"data_stored":234070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.024,"15min":1.03},"min":{"1min":0.69999999999999996,"5min":0.621,"15min":0.621},"max":{"1min":1.5449999999999999,"5min":1.698,"15min":1.698},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.097,"15min":1.097},"min":{"1min":0.75,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.82799999999999996}]},{"osd":1,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.048,"15min":1.0409999999999999},"min":{"1min":0.75,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.2190000000000001,"15min":1.2270000000000001},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.73599999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.21,"15min":1.214},"min":{"1min":0.84399999999999997,"5min":0.75,"15min":0.75},"max":{"1min":1.5509999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":2.105},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.3340000000000001,"15min":1.3440000000000001},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.986,"5min":1.986,"15min":1.986},"last":1.1659999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.1990000000000001,"15min":1.196},"min":{"1min":0.88100000000000001,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3500000000000001,"15min":1.3540000000000001},"min":{"1min":0.90300000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.7150000000000001,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.8029999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.371,"15min":1.381},"min":{"1min":0.71199999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.321,"15min":1.3129999999999999},"min":{"1min":0.98299999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.732,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.2310000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4690000000000001,"15min":1.4770000000000001},"min":{"1min":0.83399999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.9490000000000001,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":2.3159999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.482,"15min":1.486},"min":{"1min":0.91900000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":2.2610000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4350000000000001,"15min":1.427},"min":{"1min":0.879,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":2.0209999999999999},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6240000000000001,"15min":1.629},"min":{"1min":0.94399999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.1520000000000001,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":2.2309999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":577536,"data_stored":233978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":0.90100000000000002,"15min":0.86699999999999999},"min":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":5.1929999999999996,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.236,"5min":0.93500000000000005,"15min":0.88500000000000001},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":8.6989999999999998,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":1.1060000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.94599999999999995,"15min":0.95299999999999996},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.143,"5min":1.079,"15min":1.069},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":3.7240000000000002,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.4630000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.254,"5min":1.175,"15min":1.1619999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":4.2919999999999998,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":1.4350000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.323,"15min":1.2929999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.877},"max":{"1min":6.7469999999999999,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":1.349},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3600000000000001,"15min":1.357},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.4049999999999998,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.159}]},{"osd":5,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.448,"15min":1.4419999999999999},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":4.3789999999999996,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.431,"15min":1.429},"min":{"1min":1.1259999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.2949999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":1.5529999999999999,"15min":1.51},"min":{"1min":0.89800000000000002,"5min":0.871,"15min":0.871},"max":{"1min":9.5440000000000005,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.4830000000000001,"15min":1.4670000000000001},"min":{"1min":0.93500000000000005,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":4.8170000000000002,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.5009999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:43:24 2024","interfaces":[{"interface":"back","average":{"1min":1.7090000000000001,"5min":1.546,"15min":1.5189999999999999},"min":{"1min":1.006,"5min":1.006,"15min":1.006},"max":{"1min":6.6970000000000001,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.603},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.55,"15min":1.528},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":6.6529999999999996,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.401}]}]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":569344,"data_stored":232504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.36399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3069999999999999,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.93200000000000005,"15min":0.93899999999999995},"min":{"1min":0.44500000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.502,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.46500000000000002}]},{"osd":1,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.048,"15min":1.0589999999999999},"min":{"1min":0.55300000000000005,"5min":0.498,"15min":0.498},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.202,"15min":1.2250000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.706,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.51100000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.99199999999999999,"15min":0.999},"min":{"1min":0.55600000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.2110000000000001,"15min":1.22},"min":{"1min":0.60699999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":0.54000000000000004}]},{"osd":4,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.3240000000000001,"15min":1.335},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.754,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.3939999999999999,"15min":1.423},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.8069999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":0.84599999999999997}]},{"osd":5,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.397,"15min":1.421},"min":{"1min":0.83099999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.6279999999999999,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.3280000000000001,"15min":1.3420000000000001},"min":{"1min":0.80600000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6279999999999999,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.86299999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.393,"15min":1.4099999999999999},"min":{"1min":0.86499999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.6830000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.4690000000000001,"15min":1.4930000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.1629999999999998,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Wed Jan 24 17:43:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.5660000000000001,"15min":1.6000000000000001},"min":{"1min":0.82399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.819,"5min":3.343,"15min":3.343},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.353,"5min":1.5,"15min":1.524},"min":{"1min":0.96299999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.7849999999999999,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":0.78600000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1163264,"data_stored":822852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.879,"15min":0.86199999999999999},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.56100000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.538,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.014}]},{"osd":2,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.98699999999999999,"15min":0.995},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.876},{"interface":"front","average":{"1min":1.028,"5min":1.105,"15min":1.1180000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.778,"5min":1.895,"15min":1.895},"last":1.0549999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1639999999999999,"15min":1.171},"min":{"1min":0.57499999999999996,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.863,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.236,"15min":1.258},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.591,"5min":3.149,"15min":3.149},"last":1.2549999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3680000000000001,"15min":1.3700000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.9370000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.097},{"interface":"front","average":{"1min":1.286,"5min":1.327,"15min":1.3340000000000001},"min":{"1min":0.88700000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.627,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.1200000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3660000000000001,"15min":1.3720000000000001},"min":{"1min":0.84399999999999997,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.7490000000000001,"5min":2.577,"15min":2.577},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.3740000000000001,"15min":1.381},"min":{"1min":0.83199999999999996,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.998,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.196}]},{"osd":6,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.4379999999999999,"15min":1.4570000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.863,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.159},{"interface":"front","average":{"1min":1.359,"5min":1.5249999999999999,"15min":1.5529999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":2.077,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.226}]},{"osd":7,"last update":"Wed Jan 24 17:43:30 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.478,"15min":1.482},"min":{"1min":0.83299999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0710000000000002,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.472},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.556,"15min":1.5720000000000001},"min":{"1min":0.93700000000000006,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":1.8129999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.427}]}]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":573440,"data_stored":233086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.81299999999999994,"15min":0.82199999999999995},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2290000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.95299999999999996,"15min":0.97699999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.4399999999999999,"5min":1.494,"15min":1.494},"last":0.73099999999999998}]},{"osd":2,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82799999999999996,"15min":0.82899999999999996},"min":{"1min":0.44500000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.84899999999999998,"5min":1.0720000000000001,"15min":1.109},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.216,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.81999999999999995}]},{"osd":3,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.1890000000000001,"15min":1.2070000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.609,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.123,"5min":1.1839999999999999,"15min":1.194},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.605,"5min":2.407,"15min":2.407},"last":1.028}]},{"osd":4,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.397,"15min":1.4330000000000001},"min":{"1min":0.73299999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.544,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.119},{"interface":"front","average":{"1min":1.262,"5min":1.472,"15min":1.5069999999999999},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.675,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.1379999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.375,"15min":1.411},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.5249999999999999,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.01},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2609999999999999,"15min":1.266},"min":{"1min":0.87,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.7569999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.151}]},{"osd":6,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.3460000000000001,"15min":1.363},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.871,"5min":2.335,"15min":2.335},"last":1.25},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.4430000000000001,"15min":1.47},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.6990000000000001,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.1919999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:43:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.5169999999999999,"15min":1.552},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.8260000000000001,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.591,"15min":1.623},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":1.9379999999999999,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.218}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:44:15.018 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-24T17:44:15.019 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-24T17:44:15.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.290+0000 7fa6a9000640 1 Processor -- start 2024-01-24T17:44:15.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.290+0000 7fa6a9000640 1 -- start start 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.291+0000 7fa6a9000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa6a415c9b0 0x7fa6a40d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a9000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a40d05c0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a9000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a40d08a0 con 0x7fa6a40cf360 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a9000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a40d0b80 con 0x7fa6a415c9b0 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a1e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa6a415c9b0 0x7fa6a40d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a1e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa6a415c9b0 0x7fa6a40d1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38930/0 (socket says 172.21.15.39:38930) 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a1e00640 1 -- 172.21.15.39:0/851406961 learned_addr learned my addr 172.21.15.39:0/851406961 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a1e00640 1 -- 172.21.15.39:0/851406961 >> v1:172.21.15.100:6789/0 conn(0x7fa6a40cf360 legacy=0x7fa6a40b59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a1e00640 1 -- 172.21.15.39:0/851406961 >> v1:172.21.15.39:6789/0 conn(0x7fa6a40cddf0 legacy=0x7fa6a40d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.292+0000 7fa6a1e00640 1 -- 172.21.15.39:0/851406961 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a40d0e60 con 0x7fa6a415c9b0 2024-01-24T17:44:15.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.293+0000 7fa6a1e00640 1 --2- 172.21.15.39:0/851406961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a415c9b0 0x7fa6a40d1660 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa694009870 tx=0x7fa69402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22209f588ef8ff63 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:15.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.294+0000 7fa6a0a00640 1 -- 172.21.15.39:0/851406961 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694031870 con 0x7fa6a415c9b0 2024-01-24T17:44:15.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.294+0000 7fa6a0a00640 1 -- 172.21.15.39:0/851406961 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6940319d0 con 0x7fa6a415c9b0 2024-01-24T17:44:15.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.294+0000 7fa6a0a00640 1 -- 172.21.15.39:0/851406961 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694031d60 con 0x7fa6a415c9b0 2024-01-24T17:44:15.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.294+0000 7fa6a9000640 1 -- 172.21.15.39:0/851406961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a415c9b0 msgr2=0x7fa6a40d1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.294+0000 7fa6a9000640 1 --2- 172.21.15.39:0/851406961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a415c9b0 0x7fa6a40d1660 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa694009870 tx=0x7fa69402b910 comp rx=0 tx=0).stop 2024-01-24T17:44:15.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.295+0000 7fa6a9000640 1 -- 172.21.15.39:0/851406961 shutdown_connections 2024-01-24T17:44:15.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.295+0000 7fa6a9000640 1 --2- 172.21.15.39:0/851406961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a415c9b0 0x7fa6a40d1660 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.295+0000 7fa6a9000640 1 -- 172.21.15.39:0/851406961 >> 172.21.15.39:0/851406961 conn(0x7fa6a406ae70 msgr2=0x7fa6a406b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:15.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.295+0000 7fa6a9000640 1 -- 172.21.15.39:0/851406961 shutdown_connections 2024-01-24T17:44:15.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.295+0000 7fa6a9000640 1 -- 172.21.15.39:0/851406961 wait complete. 2024-01-24T17:44:15.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.296+0000 7fa6a9000640 1 Processor -- start 2024-01-24T17:44:15.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.297+0000 7fa6a9000640 1 -- start start 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a40cddf0 0x7fa6a40c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a9000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6a40cf360 0x7fa6a41f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a9000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6a415c9b0 0x7fa6a41f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a9000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a41f35b0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a9000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a41f2900 con 0x7fa6a415c9b0 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a9000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a41f2be0 con 0x7fa6a40cf360 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a40cddf0 0x7fa6a40c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a40cddf0 0x7fa6a40c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38944/0 (socket says 172.21.15.39:38944) 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a1e00640 1 -- 172.21.15.39:0/1233275054 learned_addr learned my addr 172.21.15.39:0/1233275054 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:15.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a1e00640 1 -- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6a40cf360 msgr2=0x7fa6a41f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a1e00640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6a40cf360 0x7fa6a41f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a1e00640 1 -- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6a415c9b0 msgr2=0x7fa6a41f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a1e00640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6a415c9b0 0x7fa6a41f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.298+0000 7fa6a1e00640 1 -- 172.21.15.39:0/1233275054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940094d0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.299+0000 7fa6a1e00640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a40cddf0 0x7fa6a40c6fe0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa694009870 tx=0x7fa6940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.299+0000 7fa69a000640 1 -- 172.21.15.39:0/1233275054 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694031820 con 0x7fa6a40cddf0 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.299+0000 7fa69a000640 1 -- 172.21.15.39:0/1233275054 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6940443f0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.299+0000 7fa69a000640 1 -- 172.21.15.39:0/1233275054 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6940447a0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.299+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a41f2ec0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.299+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a40c7520 con 0x7fa6a40cddf0 2024-01-24T17:44:15.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.300+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a41f36f0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.301+0000 7fa69a000640 1 -- 172.21.15.39:0/1233275054 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa69404d450 con 0x7fa6a40cddf0 2024-01-24T17:44:15.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.302+0000 7fa69a000640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa67c06a430 0x7fa67c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.302+0000 7fa69a000640 1 -- 172.21.15.39:0/1233275054 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa69402d040 con 0x7fa6a40cddf0 2024-01-24T17:44:15.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.303+0000 7fa6a1400640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa67c06a430 0x7fa67c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.304+0000 7fa6a1400640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa67c06a430 0x7fa67c06c8f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa68c000f60 tx=0x7fa68c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:15.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.306+0000 7fa69a000640 1 -- 172.21.15.39:0/1233275054 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa6a41f36f0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.568+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa6a41f36f0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.576+0000 7fa69a000640 1 -- 172.21.15.39:0/1233275054 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa6a41f36f0 con 0x7fa6a40cddf0 2024-01-24T17:44:15.577 INFO:teuthology.orchestra.run.smithi039.stderr:updated caps for client.0 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa67c06a430 msgr2=0x7fa67c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa67c06a430 0x7fa67c06c8f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa68c000f60 tx=0x7fa68c0079b0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a40cddf0 msgr2=0x7fa6a40c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a40cddf0 0x7fa6a40c6fe0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa694009870 tx=0x7fa6940046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 shutdown_connections 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6a415c9b0 0x7fa6a41f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6a40cf360 0x7fa6a41f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa67c06a430 0x7fa67c06c8f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 --2- 172.21.15.39:0/1233275054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6a40cddf0 0x7fa6a40c6fe0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 >> 172.21.15.39:0/1233275054 conn(0x7fa6a406ae70 msgr2=0x7fa6a40c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:15.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 shutdown_connections 2024-01-24T17:44:15.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.579+0000 7fa6a9000640 1 -- 172.21.15.39:0/1233275054 wait complete. 2024-01-24T17:44:15.599 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:44:15.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.817+0000 7f3ee2400640 1 Processor -- start 2024-01-24T17:44:15.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.817+0000 7f3ee2400640 1 -- start start 2024-01-24T17:44:15.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.819+0000 7f3ee2400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3edc15ab00 0x7f3edc0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.819+0000 7f3ee2400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3edc0ccd20 0x7f3edc0ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.819+0000 7f3ee2400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc0ce400 con 0x7f3edc15a720 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.819+0000 7f3ee2400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc15b3b0 con 0x7f3edc15ab00 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.819+0000 7f3ee2400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc15b690 con 0x7f3edc0ccd20 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.819+0000 7f3edbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3edc15ab00 0x7f3edc0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.820+0000 7f3edbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3edc15ab00 0x7f3edc0cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38948/0 (socket says 172.21.15.39:38948) 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.820+0000 7f3edbe00640 1 -- 172.21.15.39:0/2418846070 learned_addr learned my addr 172.21.15.39:0/2418846070 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.820+0000 7f3ee1a00640 1 --2- 172.21.15.39:0/2418846070 >> v2:172.21.15.100:3300/0 conn(0x7f3edc0ccd20 0x7f3edc0ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.820+0000 7f3edbe00640 1 -- 172.21.15.39:0/2418846070 >> v2:172.21.15.100:3300/0 conn(0x7f3edc0ccd20 msgr2=0x7f3edc0ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.820+0000 7f3edbe00640 1 --2- 172.21.15.39:0/2418846070 >> v2:172.21.15.100:3300/0 conn(0x7f3edc0ccd20 0x7f3edc0ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.820+0000 7f3edbe00640 1 -- 172.21.15.39:0/2418846070 >> v1:172.21.15.100:6789/0 conn(0x7f3edc15a720 legacy=0x7f3edc0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:15.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.820+0000 7f3edbe00640 1 -- 172.21.15.39:0/2418846070 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc15b970 con 0x7f3edc15ab00 2024-01-24T17:44:15.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.821+0000 7f3edbe00640 1 --2- 172.21.15.39:0/2418846070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0cc7e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3edc072270 tx=0x7f3ec402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a859b4391129fd5a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:15.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3edb400640 1 -- 172.21.15.39:0/2418846070 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec40317e0 con 0x7f3edc15ab00 2024-01-24T17:44:15.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3edb400640 1 -- 172.21.15.39:0/2418846070 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec4031940 con 0x7f3edc15ab00 2024-01-24T17:44:15.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3edb400640 1 -- 172.21.15.39:0/2418846070 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4031cd0 con 0x7f3edc15ab00 2024-01-24T17:44:15.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3ee2400640 1 -- 172.21.15.39:0/2418846070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 msgr2=0x7f3edc0cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3ee2400640 1 --2- 172.21.15.39:0/2418846070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0cc7e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3edc072270 tx=0x7f3ec402b880 comp rx=0 tx=0).stop 2024-01-24T17:44:15.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3ee2400640 1 -- 172.21.15.39:0/2418846070 shutdown_connections 2024-01-24T17:44:15.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3ee2400640 1 --2- 172.21.15.39:0/2418846070 >> v2:172.21.15.100:3300/0 conn(0x7f3edc0ccd20 0x7f3edc0ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3ee2400640 1 --2- 172.21.15.39:0/2418846070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0cc7e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3ee2400640 1 -- 172.21.15.39:0/2418846070 >> 172.21.15.39:0/2418846070 conn(0x7f3edc06a090 msgr2=0x7f3edc06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:15.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.822+0000 7f3ee2400640 1 -- 172.21.15.39:0/2418846070 shutdown_connections 2024-01-24T17:44:15.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.823+0000 7f3ee2400640 1 -- 172.21.15.39:0/2418846070 wait complete. 2024-01-24T17:44:15.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.824+0000 7f3ee2400640 1 Processor -- start 2024-01-24T17:44:15.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.824+0000 7f3ee2400640 1 -- start start 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.825+0000 7f3ee2400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3edc15a720 0x7f3edc157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.825+0000 7f3ee2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3ee2400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3edc0c44c0 0x7f3edc0c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3ee2400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0c4e80 con 0x7f3edc15ab00 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3ee2400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0b3b30 con 0x7f3edc0c44c0 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3ee2400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc0c74b0 con 0x7f3edc15a720 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38962/0 (socket says 172.21.15.39:38962) 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3edbe00640 1 -- 172.21.15.39:0/3187006152 learned_addr learned my addr 172.21.15.39:0/3187006152 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3ee1a00640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3edc0c44c0 0x7f3edc0c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3ee1000640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3edc15a720 0x7f3edc157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3edbe00640 1 -- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3edc15a720 msgr2=0x7f3edc157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3edbe00640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3edc15a720 0x7f3edc157840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3edbe00640 1 -- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3edc0c44c0 msgr2=0x7f3edc0c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:15.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3edbe00640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3edc0c44c0 0x7f3edc0c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:15.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.826+0000 7f3edbe00640 1 -- 172.21.15.39:0/3187006152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40094d0 con 0x7f3edc15ab00 2024-01-24T17:44:15.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.827+0000 7f3edbe00640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0b35f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009960 tx=0x7f3ec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:15.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.827+0000 7f3ed8c00640 1 -- 172.21.15.39:0/3187006152 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4032880 con 0x7f3edc15ab00 2024-01-24T17:44:15.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.827+0000 7f3ed8c00640 1 -- 172.21.15.39:0/3187006152 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec40329e0 con 0x7f3edc15ab00 2024-01-24T17:44:15.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.827+0000 7f3ed8c00640 1 -- 172.21.15.39:0/3187006152 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4032d90 con 0x7f3edc15ab00 2024-01-24T17:44:15.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.827+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc0c68a0 con 0x7f3edc15ab00 2024-01-24T17:44:15.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.827+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc0c6e60 con 0x7f3edc15ab00 2024-01-24T17:44:15.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.828+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc04dff0 con 0x7f3edc15ab00 2024-01-24T17:44:15.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.829+0000 7f3ed8c00640 1 -- 172.21.15.39:0/3187006152 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3ec40453f0 con 0x7f3edc15ab00 2024-01-24T17:44:15.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.830+0000 7f3ed8c00640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3eb406a460 0x7f3eb406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:15.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.830+0000 7f3ed8c00640 1 -- 172.21.15.39:0/3187006152 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec40aab10 con 0x7f3edc15ab00 2024-01-24T17:44:15.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.831+0000 7f3ee1000640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3eb406a460 0x7f3eb406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:15.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.832+0000 7f3ee1000640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3eb406a460 0x7f3eb406c920 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc003d90 tx=0x7f3ecc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:15.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:15.835+0000 7f3ed8c00640 1 -- 172.21.15.39:0/3187006152 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3ec4076180 con 0x7f3edc15ab00 2024-01-24T17:44:16.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.065+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3edc04dff0 con 0x7f3edc15ab00 2024-01-24T17:44:16.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.067+0000 7f3ed8c00640 1 -- 172.21.15.39:0/3187006152 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f3ec40797a0 con 0x7f3edc15ab00 2024-01-24T17:44:16.067 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:16.068 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5763,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1418568795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1418568795},{"type":"v1","addr":"172.21.15.100:6837","nonce":1418568795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:44:09.961260+0000","modified":"2024-01-24T17:44:12.985097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5769},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5769":{"gid":5769,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/1882733155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1882733155},{"type":"v1","addr":"172.21.15.39:6835","nonce":1882733155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-24T17:44:16.068 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 23 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3eb406a460 msgr2=0x7f3eb406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3eb406a460 0x7f3eb406c920 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc003d90 tx=0x7f3ecc009040 comp rx=0 tx=0).stop 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 msgr2=0x7f3edc0b35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0b35f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009960 tx=0x7f3ec40046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 shutdown_connections 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3edc0c44c0 0x7f3edc0c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3eb406a460 0x7f3eb406c920 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3edc15ab00 0x7f3edc0b35f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 --2- 172.21.15.39:0/3187006152 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3edc15a720 0x7f3edc157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 >> 172.21.15.39:0/3187006152 conn(0x7f3edc06a090 msgr2=0x7f3edc06a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:16.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 shutdown_connections 2024-01-24T17:44:16.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.070+0000 7f3ee2400640 1 -- 172.21.15.39:0/3187006152 wait complete. 2024-01-24T17:44:16.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:44:09.961260+0000', 'modified': '2024-01-24T17:44:12.985097+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5769}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5769': {'gid': 5769, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6835/1882733155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6833', 'nonce': 1882733155}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 1882733155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:44:16.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T17:44:16.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-24T17:44:16.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.721+0000 7f2ab4000640 1 Processor -- start 2024-01-24T17:44:16.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.721+0000 7f2ab4000640 1 -- start start 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.722+0000 7f2ab4000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2aac0cf990 0x7f2aac0cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.722+0000 7f2ab4000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2aac15d020 0x7f2aac15d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.722+0000 7f2ab4000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aac0d02b0 con 0x7f2aac160610 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.722+0000 7f2ab4000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac15e080 con 0x7f2aac0cf990 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.722+0000 7f2ab4000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac15dac0 con 0x7f2aac15d020 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2aac0cf990 0x7f2aac0cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2aac0cf990 0x7f2aac0cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38980/0 (socket says 172.21.15.39:38980) 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab1600640 1 -- 172.21.15.39:0/2866606344 learned_addr learned my addr 172.21.15.39:0/2866606344 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:16.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab2000640 1 --1- 172.21.15.39:0/2866606344 >> v1:172.21.15.39:6789/0 conn(0x7f2aac160610 0x7f2aac160a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:57390/0 (socket says 172.21.15.39:57390) 2024-01-24T17:44:16.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/2866606344 >> v2:172.21.15.100:3300/0 conn(0x7f2aac15d020 0x7f2aac15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab1600640 1 -- 172.21.15.39:0/2866606344 >> v2:172.21.15.100:3300/0 conn(0x7f2aac15d020 msgr2=0x7f2aac15d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab1600640 1 --2- 172.21.15.39:0/2866606344 >> v2:172.21.15.100:3300/0 conn(0x7f2aac15d020 0x7f2aac15d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab1600640 1 -- 172.21.15.39:0/2866606344 >> v1:172.21.15.39:6789/0 conn(0x7f2aac160610 legacy=0x7f2aac160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.723+0000 7f2ab1600640 1 -- 172.21.15.39:0/2866606344 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac15dda0 con 0x7f2aac0cf990 2024-01-24T17:44:16.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.724+0000 7f2ab1600640 1 --2- 172.21.15.39:0/2866606344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac0cf990 0x7f2aac0cfd70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0b52d0 tx=0x7f2a9c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf03a37f73f4fb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:16.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2aabe00640 1 -- 172.21.15.39:0/2866606344 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c02cb10 con 0x7f2aac0cf990 2024-01-24T17:44:16.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2aabe00640 1 -- 172.21.15.39:0/2866606344 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c02cc70 con 0x7f2aac0cf990 2024-01-24T17:44:16.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2aabe00640 1 -- 172.21.15.39:0/2866606344 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c0343f0 con 0x7f2aac0cf990 2024-01-24T17:44:16.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2ab4000640 1 -- 172.21.15.39:0/2866606344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac0cf990 msgr2=0x7f2aac0cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2ab4000640 1 --2- 172.21.15.39:0/2866606344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac0cf990 0x7f2aac0cfd70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0b52d0 tx=0x7f2a9c02bf70 comp rx=0 tx=0).stop 2024-01-24T17:44:16.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2ab4000640 1 -- 172.21.15.39:0/2866606344 shutdown_connections 2024-01-24T17:44:16.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2ab4000640 1 --2- 172.21.15.39:0/2866606344 >> v2:172.21.15.100:3300/0 conn(0x7f2aac15d020 0x7f2aac15d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2ab4000640 1 --2- 172.21.15.39:0/2866606344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac0cf990 0x7f2aac0cfd70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.725+0000 7f2ab4000640 1 -- 172.21.15.39:0/2866606344 >> 172.21.15.39:0/2866606344 conn(0x7f2aac071790 msgr2=0x7f2aac071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:16.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.726+0000 7f2ab4000640 1 -- 172.21.15.39:0/2866606344 shutdown_connections 2024-01-24T17:44:16.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.726+0000 7f2ab4000640 1 -- 172.21.15.39:0/2866606344 wait complete. 2024-01-24T17:44:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.727+0000 7f2ab4000640 1 Processor -- start 2024-01-24T17:44:16.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.727+0000 7f2ab4000640 1 -- start start 2024-01-24T17:44:16.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.728+0000 7f2ab4000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0cf990 0x7f2aac0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.728+0000 7f2ab4000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac15d020 0x7f2aac14e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.728+0000 7f2ab4000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac160610 0x7f2aac0cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.728+0000 7f2ab4000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0cb520 con 0x7f2aac15d020 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.728+0000 7f2ab4000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0c6f40 con 0x7f2aac0cf990 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.728+0000 7f2ab4000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0c7220 con 0x7f2aac160610 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac15d020 0x7f2aac14e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac15d020 0x7f2aac14e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38982/0 (socket says 172.21.15.39:38982) 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab0c00640 1 -- 172.21.15.39:0/1879497015 learned_addr learned my addr 172.21.15.39:0/1879497015 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab1600640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0cf990 0x7f2aac0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab2000640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac160610 0x7f2aac0cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab0c00640 1 -- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac160610 msgr2=0x7f2aac0cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac160610 0x7f2aac0cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab0c00640 1 -- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0cf990 msgr2=0x7f2aac0c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0cf990 0x7f2aac0c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.729+0000 7f2ab0c00640 1 -- 172.21.15.39:0/1879497015 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c009b30 con 0x7f2aac15d020 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.730+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac15d020 0x7f2aac14e960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400c7f0 tx=0x7f2a9400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.730+0000 7f2aa9600640 1 -- 172.21.15.39:0/1879497015 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94004220 con 0x7f2aac15d020 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.730+0000 7f2aa9600640 1 -- 172.21.15.39:0/1879497015 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94004380 con 0x7f2aac15d020 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.730+0000 7f2aa9600640 1 -- 172.21.15.39:0/1879497015 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94007840 con 0x7f2aac15d020 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.730+0000 7f2ab4000640 1 -- 172.21.15.39:0/1879497015 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac0c7500 con 0x7f2aac15d020 2024-01-24T17:44:16.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.730+0000 7f2ab4000640 1 -- 172.21.15.39:0/1879497015 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac14f280 con 0x7f2aac15d020 2024-01-24T17:44:16.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.732+0000 7f2aa9600640 1 -- 172.21.15.39:0/1879497015 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2a94007bb0 con 0x7f2aac15d020 2024-01-24T17:44:16.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.733+0000 7f2aa9600640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a8406a430 0x7f2a8406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.733+0000 7f2aa9600640 1 -- 172.21.15.39:0/1879497015 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2a940889b0 con 0x7f2aac15d020 2024-01-24T17:44:16.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.734+0000 7f2ab1600640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a8406a430 0x7f2a8406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.734+0000 7f2ab6d96740 1 Processor -- start 2024-01-24T17:44:16.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.734+0000 7f2ab6d96740 1 -- start start 2024-01-24T17:44:16.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8b913f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b60f90 0x557bb8b8e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8b8e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bb8b3c6c0 con 0x557bb8b60f90 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bb8ad1070 con 0x557bb8b91010 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bb8ace800 con 0x557bb8b64b90 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab1600640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a8406a430 0x7f2a8406c8f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c035000 tx=0x7f2a9c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b60f90 0x557bb8b8e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b60f90 0x557bb8b8e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38986/0 (socket says 172.21.15.39:38986) 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab0c00640 1 -- 172.21.15.39:0/670836569 learned_addr learned my addr 172.21.15.39:0/670836569 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.735+0000 7f2ab1600640 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8b913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2ab2000640 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8b8e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2ab0c00640 1 -- 172.21.15.39:0/670836569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 msgr2=0x557bb8b8e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8b8e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2ab0c00640 1 -- 172.21.15.39:0/670836569 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 msgr2=0x557bb8b913f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8b913f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2ab0c00640 1 -- 172.21.15.39:0/670836569 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bb8a91780 con 0x557bb8b60f90 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b60f90 0x557bb8b8e280 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2a94054030 tx=0x7f2a94007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af1240aaf266978 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2aa2000640 1 -- 172.21.15.39:0/670836569 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9405b960 con 0x557bb8b60f90 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2aa2000640 1 -- 172.21.15.39:0/670836569 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9405fb10 con 0x557bb8b60f90 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.736+0000 7f2aa2000640 1 -- 172.21.15.39:0/670836569 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94055ed0 con 0x557bb8b60f90 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 -- 172.21.15.39:0/670836569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b60f90 msgr2=0x557bb8b8e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b60f90 0x557bb8b8e280 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2a94054030 tx=0x7f2a94007e60 comp rx=0 tx=0).stop 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 -- 172.21.15.39:0/670836569 shutdown_connections 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8b8e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b60f90 0x557bb8b8e280 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/670836569 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8b913f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 -- 172.21.15.39:0/670836569 >> 172.21.15.39:0/670836569 conn(0x557bb8b632a0 msgr2=0x557bb8babe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:16.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 -- 172.21.15.39:0/670836569 shutdown_connections 2024-01-24T17:44:16.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.737+0000 7f2ab6d96740 1 -- 172.21.15.39:0/670836569 wait complete. 2024-01-24T17:44:16.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.738+0000 7f2ab6d96740 1 -- start start 2024-01-24T17:44:16.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.738+0000 7f2ab6d96740 1 Processor -- start 2024-01-24T17:44:16.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b64b90 0x557bb8b88430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b91010 0x557bb8babd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 0x557bb8b8b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bb8aca0f0 con 0x557bb8b64b90 2024-01-24T17:44:16.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bb8a6b8c0 con 0x557bb8bac2d0 2024-01-24T17:44:16.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bb8a6b560 con 0x557bb8b91010 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab6d96740 10 client.0 _ll_register_callbacks cb 0x557bb8ac3a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b64b90 0x557bb8b88430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b64b90 0x557bb8b88430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38998/0 (socket says 172.21.15.39:38998) 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.739+0000 7f2ab1600640 1 -- 172.21.15.39:0/3238074834 learned_addr learned my addr 172.21.15.39:0/3238074834 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b91010 0x557bb8babd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2ab1600640 1 -- 172.21.15.39:0/3238074834 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b91010 msgr2=0x557bb8babd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2ab1600640 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b91010 0x557bb8babd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2ab1600640 1 -- 172.21.15.39:0/3238074834 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 msgr2=0x557bb8b8b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2ab1600640 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 0x557bb8b8b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2ab1600640 1 -- 172.21.15.39:0/3238074834 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac14f280 con 0x557bb8b64b90 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2ab1600640 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b64b90 0x557bb8b88430 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009f40 tx=0x7f2a9c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba8dbee875e624e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2a9aa00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2a9aa00640 1 -- 172.21.15.39:0/3238074834 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c03eb90 con 0x557bb8b64b90 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2a9aa00640 1 -- 172.21.15.39:0/3238074834 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(49..49 src has 1..49) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2a9c03ecf0 con 0x557bb8b64b90 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.740+0000 7f2ab6d96740 1 -- 172.21.15.39:0/3238074834 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557bb8af4640 con 0x557bb8b64b90 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2a9aa00640 1 -- 172.21.15.39:0/3238074834 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a9c04a440 con 0x557bb8b64b90 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2ab6d96740 10 client.5910 fetch_fsmap learned FSMap version 23 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2ab6d96740 1 -- 172.21.15.39:0/3238074834 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x557bb8ac8770 con 0x557bb8b64b90 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2a9aa00640 1 -- 172.21.15.39:0/3238074834 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557bb8af4640 con 0x557bb8b64b90 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2a9aa00640 1 -- 172.21.15.39:0/3238074834 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2a9c03c040 con 0x557bb8b64b90 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2ab6d96740 10 client.5910 fetch_fsmap finished waiting for FSMap version 23 2024-01-24T17:44:16.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2ab6d96740 10 client.5910 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2ab6d96740 20 client.5910 populate_metadata read hostname 'smithi039' 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x557bb8bbec70 0x557bb8bbf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2ab6d96740 4 client.5910 mds_command: new command op to 5769 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.741+0000 7f2ab6d96740 1 -- 172.21.15.39:0/3238074834 --> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557bb8abc940 con 0x557bb8bbec70 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.742+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x557bb8bbec70 0x557bb8bbf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.742+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x557bb8bbec70 0x557bb8bbf050 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83334be72cad77e5 server_cookie=452fd482f993c46f in_seq=0 out_seq=0 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.742+0000 7f2a9aa00640 10 client.5910 ms_handle_connect on v2:172.21.15.39:6833/1882733155 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2a9aa00640 1 -- 172.21.15.39:0/3238074834 <== mds.0 v2:172.21.15.39:6833/1882733155 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x557bb8abc940 con 0x557bb8bbec70 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2a9aa00640 10 client.5910 handle_command_reply: tid=0 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 client.5910 shutdown 2024-01-24T17:44:16.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 -- 172.21.15.39:0/3238074834 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b64b90 msgr2=0x557bb8b88430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b64b90 0x557bb8b88430 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009f40 tx=0x7f2a9c004ab0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 -- 172.21.15.39:0/3238074834 shutdown_connections 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x557bb8bbec70 0x557bb8bbf050 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 0x557bb8b8b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b91010 0x557bb8babd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/3238074834 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b64b90 0x557bb8b88430 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2a9aa00640 0 client.5910 ms_handle_reset on v2:172.21.15.39:6833/1882733155 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 -- 172.21.15.39:0/3238074834 >> 172.21.15.39:0/3238074834 conn(0x557bb8b60f90 msgr2=0x557bb8b94d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 -- 172.21.15.39:0/3238074834 shutdown_connections 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.743+0000 7f2ab6d96740 1 -- 172.21.15.39:0/3238074834 wait complete. 2024-01-24T17:44:16.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.744+0000 7f2ab6d96740 20 client.5910 trim_cache size 0 max 16384 2024-01-24T17:44:16.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.760+0000 7f2ab6d96740 1 Processor -- start 2024-01-24T17:44:16.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.760+0000 7f2ab6d96740 1 -- start start 2024-01-24T17:44:16.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8bbe610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bb8831f20 con 0x557bb8b91010 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bb8b88be0 con 0x557bb8bac2d0 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bb8b8ac00 con 0x557bb8b64b90 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39000/0 (socket says 172.21.15.39:39000) 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.761+0000 7f2ab0c00640 1 -- 172.21.15.39:0/527737078 learned_addr learned my addr 172.21.15.39:0/527737078 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab0c00640 1 -- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 msgr2=0x557bb8bbe610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab1600640 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8bbe610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab2000640 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8bbe610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab0c00640 1 -- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 msgr2=0x557bb8baeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab0c00640 1 -- 172.21.15.39:0/527737078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bb8afe2f0 con 0x557bb8b91010 2024-01-24T17:44:16.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab1600640 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8bbe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:16.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2a94054030 tx=0x7f2a9406c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f23f71ec8077da server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:16.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2aa2a00640 1 -- 172.21.15.39:0/527737078 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9405b9e0 con 0x557bb8b91010 2024-01-24T17:44:16.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2aa2a00640 1 -- 172.21.15.39:0/527737078 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9406f080 con 0x557bb8b91010 2024-01-24T17:44:16.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.762+0000 7f2aa2a00640 1 -- 172.21.15.39:0/527737078 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94054240 con 0x557bb8b91010 2024-01-24T17:44:16.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 -- 172.21.15.39:0/527737078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b91010 msgr2=0x557bb8bae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2a94054030 tx=0x7f2a9406c000 comp rx=0 tx=0).stop 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 -- 172.21.15.39:0/527737078 shutdown_connections 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/527737078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8bbe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 -- 172.21.15.39:0/527737078 >> 172.21.15.39:0/527737078 conn(0x557bb8b60f90 msgr2=0x557bb8b61420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 -- 172.21.15.39:0/527737078 shutdown_connections 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.763+0000 7f2ab6d96740 1 -- 172.21.15.39:0/527737078 wait complete. 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.764+0000 7f2ab6d96740 1 -- start start 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.764+0000 7f2ab6d96740 1 Processor -- start 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8b94130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab6d96740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bb8b88db0 con 0x557bb8bac2d0 2024-01-24T17:44:16.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bb8b89210 con 0x557bb8b91010 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab6d96740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bb8b947e0 con 0x557bb8b64b90 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab6d96740 10 client.0 _ll_register_callbacks cb 0x557bb8b91880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39010/0 (socket says 172.21.15.39:39010) 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab2000640 1 -- 172.21.15.39:0/2888891090 learned_addr learned my addr 172.21.15.39:0/2888891090 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.765+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab1600640 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8b94130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab2000640 1 -- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 msgr2=0x557bb8b94130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab2000640 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8b94130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab2000640 1 -- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 msgr2=0x557bb8bae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab2000640 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab2000640 1 -- 172.21.15.39:0/2888891090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557bb8a91780 con 0x557bb8bac2d0 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab0c00640 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:16.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab2000640 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4000f60 tx=0x7f2aa40014c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3e17540fe57826 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:16.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2a9aa00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:44:16.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2a9aa00640 1 -- 172.21.15.39:0/2888891090 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4018070 con 0x557bb8bac2d0 2024-01-24T17:44:16.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2a9aa00640 1 -- 172.21.15.39:0/2888891090 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(49..49 src has 1..49) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2aa400c070 con 0x557bb8bac2d0 2024-01-24T17:44:16.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab6d96740 1 -- 172.21.15.39:0/2888891090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a9405fb10 con 0x557bb8bac2d0 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2a9aa00640 1 -- 172.21.15.39:0/2888891090 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2aa4001b70 con 0x557bb8bac2d0 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab6d96740 10 client.5916 fetch_fsmap learned FSMap version 23 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.766+0000 7f2ab6d96740 1 -- 172.21.15.39:0/2888891090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x557bb8a6aad0 con 0x557bb8bac2d0 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.767+0000 7f2a9aa00640 1 -- 172.21.15.39:0/2888891090 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9405fb10 con 0x557bb8bac2d0 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.767+0000 7f2a9aa00640 1 -- 172.21.15.39:0/2888891090 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2aa40070d0 con 0x557bb8bac2d0 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.767+0000 7f2ab6d96740 10 client.5916 fetch_fsmap finished waiting for FSMap version 23 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.767+0000 7f2ab6d96740 10 client.5916 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.767+0000 7f2ab6d96740 20 client.5916 populate_metadata read hostname 'smithi039' 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.767+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x557bb8c12130 0x557bb8c12510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.767+0000 7f2ab6d96740 4 client.5916 mds_command: new command op to 5769 tid=0[{"prefix": "status"}] 2024-01-24T17:44:16.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.767+0000 7f2ab6d96740 1 -- 172.21.15.39:0/2888891090 --> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557bb8ac3170 con 0x557bb8c12130 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.768+0000 7f2ab1600640 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x557bb8c12130 0x557bb8c12510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.768+0000 7f2ab1600640 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x557bb8c12130 0x557bb8c12510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6367306e099df801 server_cookie=b98646eb571dce11 in_seq=0 out_seq=0 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.768+0000 7f2a9aa00640 10 client.5916 ms_handle_connect on v2:172.21.15.39:6833/1882733155 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.768+0000 7f2a9aa00640 1 -- 172.21.15.39:0/2888891090 <== mds.0 v2:172.21.15.39:6833/1882733155 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x557bb8ac3170 con 0x557bb8c12130 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.768+0000 7f2a9aa00640 10 client.5916 handle_command_reply: tid=0 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.768+0000 7f2ab6d96740 1 client.5916 shutdown 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 -- 172.21.15.39:0/2888891090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8bac2d0 msgr2=0x557bb8baeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4000f60 tx=0x7f2aa40014c0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 -- 172.21.15.39:0/2888891090 shutdown_connections 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557bb8bac2d0 0x557bb8baeb30 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x557bb8c12130 0x557bb8c12510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x557bb8b91010 0x557bb8bae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 --2- 172.21.15.39:0/2888891090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x557bb8b64b90 0x557bb8b94130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2a9aa00640 0 client.5916 ms_handle_reset on v2:172.21.15.39:6833/1882733155 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 -- 172.21.15.39:0/2888891090 >> 172.21.15.39:0/2888891090 conn(0x557bb8b60f90 msgr2=0x557bb8baf330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 -- 172.21.15.39:0/2888891090 shutdown_connections 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 1 -- 172.21.15.39:0/2888891090 wait complete. 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.769+0000 7f2ab6d96740 20 client.5916 trim_cache size 0 max 16384 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5769, 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:44:16.773 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:44:16.774 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 6.8023505259999997, 2024-01-24T17:44:16.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 23, 2024-01-24T17:44:16.774 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 49, 2024-01-24T17:44:16.774 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 48, 2024-01-24T17:44:16.774 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 12.645215035 2024-01-24T17:44:16.774 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 -- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a8406a430 msgr2=0x7f2a8406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a8406a430 0x7f2a8406c8f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c035000 tx=0x7f2a9c030000 comp rx=0 tx=0).stop 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 -- 172.21.15.39:0/1879497015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac15d020 msgr2=0x7f2aac14e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac15d020 0x7f2aac14e960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400c7f0 tx=0x7f2a9400ccb0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 -- 172.21.15.39:0/1879497015 shutdown_connections 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2aac160610 0x7f2aac0cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a8406a430 0x7f2a8406c8f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2aac15d020 0x7f2aac14e960 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 --2- 172.21.15.39:0/1879497015 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2aac0cf990 0x7f2aac0c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 -- 172.21.15.39:0/1879497015 >> 172.21.15.39:0/1879497015 conn(0x7f2aac071790 msgr2=0x7f2aac15a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:16.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 -- 172.21.15.39:0/1879497015 shutdown_connections 2024-01-24T17:44:16.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.770+0000 7f2ab4000640 1 -- 172.21.15.39:0/1879497015 wait complete. 2024-01-24T17:44:16.790 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:44:16.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.998+0000 7f0092600640 1 Processor -- start 2024-01-24T17:44:16.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.998+0000 7f0092600640 1 -- start start 2024-01-24T17:44:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.999+0000 7f0092600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0caf50 0x7f008c0cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.999+0000 7f0092600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f008c0b1390 0x7f008c0b1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.999+0000 7f0092600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008c0cb8f0 con 0x7f008c15a950 2024-01-24T17:44:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.999+0000 7f0092600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c0b43d0 con 0x7f008c0b1390 2024-01-24T17:44:17.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:16.999+0000 7f0092600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0c8f70 con 0x7f008c0caf50 2024-01-24T17:44:17.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.000+0000 7f008aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0caf50 0x7f008c0cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:17.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.000+0000 7f008aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0caf50 0x7f008c0cb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:49472/0 (socket says 172.21.15.39:49472) 2024-01-24T17:44:17.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.000+0000 7f008aa00640 1 -- 172.21.15.39:0/347633215 learned_addr learned my addr 172.21.15.39:0/347633215 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:17.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.000+0000 7f008be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f008c0b1390 0x7f008c0b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:17.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.001+0000 7f008aa00640 1 -- 172.21.15.39:0/347633215 >> v2:172.21.15.100:3300/0 conn(0x7f008c0b1390 msgr2=0x7f008c0b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:17.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.001+0000 7f008aa00640 1 --2- 172.21.15.39:0/347633215 >> v2:172.21.15.100:3300/0 conn(0x7f008c0b1390 0x7f008c0b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.001+0000 7f008aa00640 1 -- 172.21.15.39:0/347633215 >> v1:172.21.15.100:6789/0 conn(0x7f008c15a950 legacy=0x7f008c0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:17.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.001+0000 7f008aa00640 1 -- 172.21.15.39:0/347633215 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c0c9180 con 0x7f008c0caf50 2024-01-24T17:44:17.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.002+0000 7f008aa00640 1 --2- 172.21.15.39:0/347633215 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0caf50 0x7f008c0cb3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0074009870 tx=0x7f007402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76653b5806a2f120 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:17.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f008a000640 1 -- 172.21.15.39:0/347633215 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0074030840 con 0x7f008c0caf50 2024-01-24T17:44:17.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f008a000640 1 -- 172.21.15.39:0/347633215 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00740309a0 con 0x7f008c0caf50 2024-01-24T17:44:17.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f008a000640 1 -- 172.21.15.39:0/347633215 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0074030d30 con 0x7f008c0caf50 2024-01-24T17:44:17.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f0092600640 1 -- 172.21.15.39:0/347633215 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0caf50 msgr2=0x7f008c0cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:17.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f0092600640 1 --2- 172.21.15.39:0/347633215 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0caf50 0x7f008c0cb3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0074009870 tx=0x7f007402b980 comp rx=0 tx=0).stop 2024-01-24T17:44:17.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f0092600640 1 -- 172.21.15.39:0/347633215 shutdown_connections 2024-01-24T17:44:17.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f0092600640 1 --2- 172.21.15.39:0/347633215 >> v2:172.21.15.100:3300/0 conn(0x7f008c0b1390 0x7f008c0b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f0092600640 1 --2- 172.21.15.39:0/347633215 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0caf50 0x7f008c0cb3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.003+0000 7f0092600640 1 -- 172.21.15.39:0/347633215 >> 172.21.15.39:0/347633215 conn(0x7f008c06a3e0 msgr2=0x7f008c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:17.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.004+0000 7f0092600640 1 -- 172.21.15.39:0/347633215 shutdown_connections 2024-01-24T17:44:17.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.004+0000 7f0092600640 1 -- 172.21.15.39:0/347633215 wait complete. 2024-01-24T17:44:17.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.005+0000 7f0092600640 1 Processor -- start 2024-01-24T17:44:17.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.005+0000 7f0092600640 1 -- start start 2024-01-24T17:44:17.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.006+0000 7f0092600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0b1390 0x7f008c148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:17.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.006+0000 7f0092600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c0caf50 0x7f008c0cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:17.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.006+0000 7f0092600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008c15a950 0x7f008c0ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.006+0000 7f0092600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c15dcf0 con 0x7f008c15a950 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.006+0000 7f0092600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c148e50 con 0x7f008c0caf50 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.006+0000 7f0092600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0b35f0 con 0x7f008c0b1390 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008c15a950 0x7f008c0ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008c15a950 0x7f008c0ac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39020/0 (socket says 172.21.15.39:39020) 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 -- 172.21.15.39:0/3428999044 learned_addr learned my addr 172.21.15.39:0/3428999044 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008b400640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0b1390 0x7f008c148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008aa00640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c0caf50 0x7f008c0cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 -- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0b1390 msgr2=0x7f008c148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0b1390 0x7f008c148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 -- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c0caf50 msgr2=0x7f008c0cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c0caf50 0x7f008c0cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 -- 172.21.15.39:0/3428999044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00740094d0 con 0x7f008c15a950 2024-01-24T17:44:17.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.007+0000 7f008be00640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008c15a950 0x7f008c0ac2d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f008400efc0 tx=0x7f008400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:17.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.008+0000 7f0083400640 1 -- 172.21.15.39:0/3428999044 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084014480 con 0x7f008c15a950 2024-01-24T17:44:17.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.008+0000 7f0083400640 1 -- 172.21.15.39:0/3428999044 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008400f040 con 0x7f008c15a950 2024-01-24T17:44:17.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.008+0000 7f0083400640 1 -- 172.21.15.39:0/3428999044 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084014820 con 0x7f008c15a950 2024-01-24T17:44:17.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.008+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c15d730 con 0x7f008c15a950 2024-01-24T17:44:17.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.008+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c0b3280 con 0x7f008c15a950 2024-01-24T17:44:17.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.009+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008c0b30e0 con 0x7f008c15a950 2024-01-24T17:44:17.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.010+0000 7f0083400640 1 -- 172.21.15.39:0/3428999044 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0084014980 con 0x7f008c15a950 2024-01-24T17:44:17.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.011+0000 7f0083400640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f006406a430 0x7f006406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:17.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.011+0000 7f0083400640 1 -- 172.21.15.39:0/3428999044 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f00840895e0 con 0x7f008c15a950 2024-01-24T17:44:17.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.011+0000 7f008b400640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f006406a430 0x7f006406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:17.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.013+0000 7f008b400640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f006406a430 0x7f006406c8f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f007c000f60 tx=0x7f007c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:17.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.016+0000 7f0083400640 1 -- 172.21.15.39:0/3428999044 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f008c0b30e0 con 0x7f008c15a950 2024-01-24T17:44:17.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.247+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f008c0b30e0 con 0x7f008c15a950 2024-01-24T17:44:17.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.248+0000 7f0083400640 1 -- 172.21.15.39:0/3428999044 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0084018060 con 0x7f008c15a950 2024-01-24T17:44:17.249 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f006406a430 msgr2=0x7f006406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f006406a430 0x7f006406c8f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f007c000f60 tx=0x7f007c009040 comp rx=0 tx=0).stop 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008c15a950 msgr2=0x7f008c0ac2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008c15a950 0x7f008c0ac2d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f008400efc0 tx=0x7f008400c490 comp rx=0 tx=0).stop 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 shutdown_connections 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f006406a430 0x7f006406c8f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f008c15a950 0x7f008c0ac2d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c0caf50 0x7f008c0cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 --2- 172.21.15.39:0/3428999044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f008c0b1390 0x7f008c148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:17.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 >> 172.21.15.39:0/3428999044 conn(0x7f008c06a3e0 msgr2=0x7f008c0b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:17.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 shutdown_connections 2024-01-24T17:44:17.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:17.251+0000 7f0092600640 1 -- 172.21.15.39:0/3428999044 wait complete. 2024-01-24T17:44:17.277 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:44:17.277 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:44:17.277 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:44:17.277 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:44:17.277 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:17.278 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:44:17.278 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:44:17.278 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:44:17.278 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:44:17.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:17.279 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:17.329 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:44:17.330 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:17.330 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:17.330 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:44:17.330 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42475sec preferred_lft 42475sec 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:17.331 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:44:17.332 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:44:17.333 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:17.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:17.334 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:44:17.422 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:44:17.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:17.427 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:44:17.483 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:44:17.483 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:44:17.483 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:44:17.483 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:44:17.483 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:44:17.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:17.484 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:44:17.484 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:44:17.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:44:17.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:44:17.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:44:17.484 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:44:17.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:17.667 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:44:17.719 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:44:17.719 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:44:17.719 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:44:17.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:17.719 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:44:17.815 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:44:17.816 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:44:17.816 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:17.873 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:44:17.874 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:44:17.874 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:17.928 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:44:18.029 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:44:18.029 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:44:18.123 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:44:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:44:18.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:18.128 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:44:18.187 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:44:18.187 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:44:18.194 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:44:18.272 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:44:18.273 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:44:18.345 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:44:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:44:18.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:18.348 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:44:18.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:44:18.382+0000 7fd6eba44480 -1 init, newargv = 0x55b1861efec0 newargc=15 2024-01-24T17:44:18.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[55156]: starting ceph client 2024-01-24T17:44:18.391 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[55156]: starting fuse 2024-01-24T17:44:19.490 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:44:19.603 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:44:19.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:44:19.706 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:44:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:44:19.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:19.712 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:44:19.769 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:44:19.769 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:44:19.769 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:44:19.769 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:44:19.770 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:44:19.771 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:44:19.952 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.55156.asok 2024-01-24T17:44:20.133 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55156.asok 2024-01-24T17:44:20.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:20.133 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55156.asok status 2024-01-24T17:44:20.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:44:20.363 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:44:20.383 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:44:20.383 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:44:20.383 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:44:20.383 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:44:20.383 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:44:20.383 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:44:20.383 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:44:20.383 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "55156", 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5925, 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5925 2024-01-24T17:44:20.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:20.385 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:44:20.385 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:44:20.385 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:44:20.385 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1994510967 2024-01-24T17:44:20.385 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:20.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:20.385 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:44:20.386 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:44:20.386 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:44:20.386 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1994510967 2024-01-24T17:44:20.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:20.386 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.5925 192.168.0.1:0/1994510967", 2024-01-24T17:44:20.386 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/1994510967", 2024-01-24T17:44:20.387 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:44:20.387 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 23, 2024-01-24T17:44:20.387 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 50, 2024-01-24T17:44:20.387 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:44:20.387 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:44:20.387 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:44:20.388 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:44:20.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:20.401 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:44:20.401 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:20.492 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:44:20.492 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:20.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:20.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:20.635 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:44:20.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.851+0000 7fc5a4c00640 1 Processor -- start 2024-01-24T17:44:20.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.852+0000 7fc5a4c00640 1 -- start start 2024-01-24T17:44:20.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.853+0000 7fc5a4c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc5a015cbf0 0x7fc5a00d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.853+0000 7fc5a4c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 0x7fc5a00ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.853+0000 7fc5a4c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a015dfe0 con 0x7fc5a00d0da0 2024-01-24T17:44:20.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.853+0000 7fc5a4c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5a00d0a90 con 0x7fc5a015cbf0 2024-01-24T17:44:20.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.853+0000 7fc5a4c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00cd760 con 0x7fc5a015db60 2024-01-24T17:44:20.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.854+0000 7fc59fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc5a00d0da0 0x7fc5a015e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:57402/0 (socket says 172.21.15.39:57402) 2024-01-24T17:44:20.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.854+0000 7fc59fe00640 1 -- 172.21.15.39:0/3245384651 learned_addr learned my addr 172.21.15.39:0/3245384651 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.854+0000 7fc596800640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 0x7fc5a00ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.854+0000 7fc59f400640 1 --2- 172.21.15.39:0/3245384651 >> v2:172.21.15.100:3300/0 conn(0x7fc5a015cbf0 0x7fc5a00d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.854+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740189227 0 0) 0x7fc5a015dfe0 con 0x7fc5a00d0da0 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.854+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc580003520 con 0x7fc5a00d0da0 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 457121014 0 0) 0x7fc580003520 con 0x7fc5a00d0da0 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 msgr2=0x7fc5a00ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 0x7fc5a00ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 >> v2:172.21.15.100:3300/0 conn(0x7fc5a015cbf0 msgr2=0x7fc5a00d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:20.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 --2- 172.21.15.39:0/3245384651 >> v2:172.21.15.100:3300/0 conn(0x7fc5a015cbf0 0x7fc5a00d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a00cda40 con 0x7fc5a00d0da0 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc5980019f0 con 0x7fc5a00d0da0 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 >> v1:172.21.15.39:6789/0 conn(0x7fc5a00d0da0 legacy=0x7fc5a015e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 0x7fc580003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580003ef0 0x7fc5800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc580004d00 0x7fc580005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00d0a90 con 0x7fc580004d00 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00cd760 con 0x7fc580003ef0 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58000b980 con 0x7fc5a015db60 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59fe00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc580004d00 0x7fc580005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59f400640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 0x7fc580003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc596800640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580003ef0 0x7fc5800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.855+0000 7fc59fe00640 1 -- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 msgr2=0x7fc580003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:20.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.856+0000 7fc59fe00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 0x7fc580003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:20.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.856+0000 7fc59fe00640 1 -- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580003ef0 msgr2=0x7fc5800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:20.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.856+0000 7fc59fe00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580003ef0 0x7fc5800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:20.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.856+0000 7fc59fe00640 1 -- 172.21.15.39:0/3245384651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc580003520 con 0x7fc580004d00 2024-01-24T17:44:20.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.856+0000 7fc596800640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580003ef0 0x7fc5800047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:20.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.856+0000 7fc59f400640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a015db60 0x7fc580003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:20.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.857+0000 7fc59fe00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc580004d00 0x7fc580005180 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc59800a8d0 tx=0x7fc5800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab8138c2b38523d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.857+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598034630 con 0x7fc580004d00 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.857+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59802f070 con 0x7fc580004d00 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.857+0000 7fc59ea00640 1 -- 172.21.15.39:0/3245384651 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5980349d0 con 0x7fc580004d00 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.857+0000 7fc5a4c00640 1 -- 172.21.15.39:0/3245384651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc580004d00 msgr2=0x7fc580005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.857+0000 7fc5a4c00640 1 --2- 172.21.15.39:0/3245384651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc580004d00 0x7fc580005180 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc59800a8d0 tx=0x7fc5800057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.857+0000 7fc59fe00640 1 -- 172.21.15.39:0/3245384651 reap_dead start 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.858+0000 7fc5a4c00640 1 -- 172.21.15.39:0/3245384651 shutdown_connections 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.858+0000 7fc5a4c00640 1 -- 172.21.15.39:0/3245384651 >> 172.21.15.39:0/3245384651 conn(0x7fc5a009bb90 msgr2=0x7fc5a00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.858+0000 7fc5a4c00640 1 -- 172.21.15.39:0/3245384651 shutdown_connections 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.858+0000 7fc5a4c00640 1 -- 172.21.15.39:0/3245384651 wait complete. 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.859+0000 7fc5a4c00640 1 Processor -- start 2024-01-24T17:44:20.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.860+0000 7fc5a4c00640 1 -- start start 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc5a4c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580004d00 0x7fc5a0148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc5a4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a00c6d20 0x7fc5a00cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc5a4c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a00b74e0 0x7fc5a00b7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc5a4c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00b7eb0 con 0x7fc5a00c6d20 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc5a4c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00c82a0 con 0x7fc580004d00 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc5a4c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a00c7200 con 0x7fc5a00b74e0 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc596800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a00c6d20 0x7fc5a00cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc596800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a00c6d20 0x7fc5a00cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39042/0 (socket says 172.21.15.39:39042) 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc596800640 1 -- 172.21.15.39:0/4279263487 learned_addr learned my addr 172.21.15.39:0/4279263487 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc59fe00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a00b74e0 0x7fc5a00b7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc59f400640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580004d00 0x7fc5a0148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc596800640 1 -- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a00b74e0 msgr2=0x7fc5a00b7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc596800640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a00b74e0 0x7fc5a00b7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc596800640 1 -- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580004d00 msgr2=0x7fc5a0148ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc596800640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580004d00 0x7fc5a0148ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc596800640 1 -- 172.21.15.39:0/4279263487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a00c74e0 con 0x7fc5a00c6d20 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.861+0000 7fc59fe00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a00b74e0 0x7fc5a00b7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.862+0000 7fc596800640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a00c6d20 0x7fc5a00cbb60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc5880108b0 tx=0x7fc588010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.862+0000 7fc597e00640 1 -- 172.21.15.39:0/4279263487 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58800eb00 con 0x7fc5a00c6d20 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.862+0000 7fc597e00640 1 -- 172.21.15.39:0/4279263487 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58800ec60 con 0x7fc5a00c6d20 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.862+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a00c77c0 con 0x7fc5a00c6d20 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.862+0000 7fc597e00640 1 -- 172.21.15.39:0/4279263487 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58801c400 con 0x7fc5a00c6d20 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.862+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a00c7d80 con 0x7fc5a00c6d20 2024-01-24T17:44:20.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.863+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a004dff0 con 0x7fc5a00c6d20 2024-01-24T17:44:20.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.864+0000 7fc597e00640 1 -- 172.21.15.39:0/4279263487 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc58801c560 con 0x7fc5a00c6d20 2024-01-24T17:44:20.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.865+0000 7fc597e00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc57806a460 0x7fc57806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:20.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.865+0000 7fc597e00640 1 -- 172.21.15.39:0/4279263487 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc588088fa0 con 0x7fc5a00c6d20 2024-01-24T17:44:20.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.866+0000 7fc59f400640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc57806a460 0x7fc57806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:20.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.867+0000 7fc59f400640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc57806a460 0x7fc57806c920 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc590009a30 tx=0x7fc590008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:20.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:20.870+0000 7fc597e00640 1 -- 172.21.15.39:0/4279263487 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc588054460 con 0x7fc5a00c6d20 2024-01-24T17:44:21.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.101+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc5a00b4f60 con 0x7fc5a00c6d20 2024-01-24T17:44:21.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.102+0000 7fc597e00640 1 -- 172.21.15.39:0/4279263487 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc588057a80 con 0x7fc5a00c6d20 2024-01-24T17:44:21.103 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:44:21.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc57806a460 msgr2=0x7fc57806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:21.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc57806a460 0x7fc57806c920 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc590009a30 tx=0x7fc590008040 comp rx=0 tx=0).stop 2024-01-24T17:44:21.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a00c6d20 msgr2=0x7fc5a00cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:21.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a00c6d20 0x7fc5a00cbb60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc5880108b0 tx=0x7fc588010d70 comp rx=0 tx=0).stop 2024-01-24T17:44:21.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 shutdown_connections 2024-01-24T17:44:21.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc5a00b74e0 0x7fc5a00b7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:21.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc57806a460 0x7fc57806c920 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:21.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5a00c6d20 0x7fc5a00cbb60 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:21.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 --2- 172.21.15.39:0/4279263487 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc580004d00 0x7fc5a0148ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:21.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 >> 172.21.15.39:0/4279263487 conn(0x7fc5a009bb90 msgr2=0x7fc5a00d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:21.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 shutdown_connections 2024-01-24T17:44:21.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:21.105+0000 7fc5a4c00640 1 -- 172.21.15.39:0/4279263487 wait complete. 2024-01-24T17:44:21.124 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:44:21.124 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:44:21.124 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:44:21.124 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-24T17:44:21.125 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:44:21.125 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:44:21.125 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:44:21.125 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:44:21.125 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:44:21.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:21.125 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:44:21.183 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42468sec preferred_lft 42468sec 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:21.184 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:44:21.185 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:21.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:21.186 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:44:21.273 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:44:21.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:21.278 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:44:21.333 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:44:21.333 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:44:21.333 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:44:21.333 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:44:21.333 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:44:21.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:21.334 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:44:21.334 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:44:21.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:44:21.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:44:21.334 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:44:21.334 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:44:21.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:21.512 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:44:21.561 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:44:21.561 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:44:21.561 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:44:21.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:21.562 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:44:21.649 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-24T17:44:21.649 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:44:21.649 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:44:21.700 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:44:21.700 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:44:21.701 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:44:21.750 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:44:21.840 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:44:21.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:44:21.933 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:44:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:44:21.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:21.938 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:44:21.994 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:44:21.994 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-24T17:44:21.999 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:44:22.073 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:44:22.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:44:22.141 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:44:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:44:22.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:22.144 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:44:22.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:44:22.185+0000 7f7afda3f480 -1 init, newargv = 0x55b3a4077180 newargc=15 2024-01-24T17:44:22.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[44571]: starting ceph client 2024-01-24T17:44:22.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[44571]: starting fuse 2024-01-24T17:44:23.285 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:44:23.385 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:44:23.385 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:44:23.475 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:44:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:44:23.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:23.480 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:44:23.534 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:44:23.535 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:44:23.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:44:23.535 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:44:23.535 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:44:23.535 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:44:23.535 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:44:23.535 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:44:23.535 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:44:23.535 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:44:23.536 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:44:23.537 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:44:23.537 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:44:23.537 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2024-01-24T17:44:23.537 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:44:23.708 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.44571.asok 2024-01-24T17:44:23.890 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44571.asok 2024-01-24T17:44:23.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:23.890 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44571.asok status 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "44571", 2024-01-24T17:44:24.124 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 5335, 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5335 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 75616696 2024-01-24T17:44:24.125 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 75616696 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.5335 192.168.0.1:0/75616696", 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/75616696", 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 23, 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 50, 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:44:24.126 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:44:24.127 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:44:24.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:24.138 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:44:24.138 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:44:24.192 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:44:24.192 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:44:24.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:24.192 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:44:24.276 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-01-24T17:44:24.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.508+0000 7f076ac00640 1 Processor -- start 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.508+0000 7f076ac00640 1 -- start start 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.509+0000 7f076ac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f076415ca40 0x7f0764161170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.509+0000 7f076ac00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f07640cce00 0x7f07640cd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.509+0000 7f076ac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076406d930 con 0x7f07640cdb70 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.509+0000 7f076ac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076406d460 con 0x7f076415ca40 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.509+0000 7f076ac00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07640cd7d0 con 0x7f07640cce00 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f0769800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f076415ca40 0x7f0764161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f0768e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f07640cdb70 0x7f076406cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53848/0 (socket says 172.21.15.39:53848) 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f0768e00640 1 -- 172.21.15.39:0/391904030 learned_addr learned my addr 172.21.15.39:0/391904030 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f076a200640 1 --2- 172.21.15.39:0/391904030 >> v2:172.21.15.100:3300/0 conn(0x7f07640cce00 0x7f07640cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f0769800640 1 -- 172.21.15.39:0/391904030 >> v2:172.21.15.100:3300/0 conn(0x7f07640cce00 msgr2=0x7f07640cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f0769800640 1 --2- 172.21.15.39:0/391904030 >> v2:172.21.15.100:3300/0 conn(0x7f07640cce00 0x7f07640cd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f0769800640 1 -- 172.21.15.39:0/391904030 >> v1:172.21.15.39:6789/0 conn(0x7f07640cdb70 legacy=0x7f076406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f0769800640 1 -- 172.21.15.39:0/391904030 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076406d650 con 0x7f076415ca40 2024-01-24T17:44:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.510+0000 7f076a200640 1 --2- 172.21.15.39:0/391904030 >> v2:172.21.15.100:3300/0 conn(0x7f07640cce00 0x7f07640cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.511+0000 7f0769800640 1 --2- 172.21.15.39:0/391904030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f076415ca40 0x7f0764161170 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0764160530 tx=0x7f075402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ab8ebc1ba3d648 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f0763e00640 1 -- 172.21.15.39:0/391904030 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754031830 con 0x7f076415ca40 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f0763e00640 1 -- 172.21.15.39:0/391904030 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0754031990 con 0x7f076415ca40 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f0763e00640 1 -- 172.21.15.39:0/391904030 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754031d20 con 0x7f076415ca40 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f076ac00640 1 -- 172.21.15.39:0/391904030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f076415ca40 msgr2=0x7f0764161170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f076ac00640 1 --2- 172.21.15.39:0/391904030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f076415ca40 0x7f0764161170 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0764160530 tx=0x7f075402b8d0 comp rx=0 tx=0).stop 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f076ac00640 1 -- 172.21.15.39:0/391904030 shutdown_connections 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f076ac00640 1 --2- 172.21.15.39:0/391904030 >> v2:172.21.15.100:3300/0 conn(0x7f07640cce00 0x7f07640cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f076ac00640 1 --2- 172.21.15.39:0/391904030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f076415ca40 0x7f0764161170 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f076ac00640 1 -- 172.21.15.39:0/391904030 >> 172.21.15.39:0/391904030 conn(0x7f0764069e50 msgr2=0x7f076406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.512+0000 7f076ac00640 1 -- 172.21.15.39:0/391904030 shutdown_connections 2024-01-24T17:44:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.513+0000 7f076ac00640 1 -- 172.21.15.39:0/391904030 wait complete. 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.514+0000 7f076ac00640 1 Processor -- start 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.514+0000 7f076ac00640 1 -- start start 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.515+0000 7f076ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07640cce00 0x7f076415dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.515+0000 7f076ac00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07640cdb70 0x7f076415b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.515+0000 7f076ac00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f076415ca40 0x7f07641f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.515+0000 7f076ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f3530 con 0x7f07640cce00 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.515+0000 7f076ac00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f31e0 con 0x7f076415ca40 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.515+0000 7f076ac00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07641f2900 con 0x7f07640cdb70 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07640cce00 0x7f076415dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07640cce00 0x7f076415dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46366/0 (socket says 172.21.15.39:46366) 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 -- 172.21.15.39:0/2891525155 learned_addr learned my addr 172.21.15.39:0/2891525155 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f076a200640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f076415ca40 0x7f07641f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 -- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07640cdb70 msgr2=0x7f076415b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0768e00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07640cdb70 0x7f076415b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07640cdb70 0x7f076415b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 -- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f076415ca40 msgr2=0x7f07641f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f076415ca40 0x7f07641f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 -- 172.21.15.39:0/2891525155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07540093a0 con 0x7f07640cce00 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f076a200640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f076415ca40 0x7f07641f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.516+0000 7f0769800640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07640cce00 0x7f076415dcc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0754009880 tx=0x7f07540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.517+0000 7f0761600640 1 -- 172.21.15.39:0/2891525155 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075403ba70 con 0x7f07640cce00 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.517+0000 7f0761600640 1 -- 172.21.15.39:0/2891525155 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075403bbd0 con 0x7f07640cce00 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.517+0000 7f0761600640 1 -- 172.21.15.39:0/2891525155 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754005990 con 0x7f07640cce00 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.517+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07641f2be0 con 0x7f07640cce00 2024-01-24T17:44:24.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.517+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07641f2510 con 0x7f07640cce00 2024-01-24T17:44:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.518+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07641f26a0 con 0x7f07640cce00 2024-01-24T17:44:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.519+0000 7f0761600640 1 -- 172.21.15.39:0/2891525155 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f075404e440 con 0x7f07640cce00 2024-01-24T17:44:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.520+0000 7f0761600640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f073c06a430 0x7f073c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.520+0000 7f0761600640 1 -- 172.21.15.39:0/2891525155 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f07540aad40 con 0x7f07640cce00 2024-01-24T17:44:24.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.521+0000 7f0768e00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f073c06a430 0x7f073c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:24.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.522+0000 7f0768e00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f073c06a430 0x7f073c06c8f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f07640ab0e0 tx=0x7f074c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:24.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.525+0000 7f0761600640 1 -- 172.21.15.39:0/2891525155 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0754076290 con 0x7f07640cce00 2024-01-24T17:44:24.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:24.756+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f076406d280 con 0x7f07640cce00 2024-01-24T17:44:25.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.254+0000 7f0761600640 1 -- 172.21.15.39:0/2891525155 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v24) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0754035020 con 0x7f07640cce00 2024-01-24T17:44:25.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.257+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f073c06a430 msgr2=0x7f073c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:25.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f073c06a430 0x7f073c06c8f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f07640ab0e0 tx=0x7f074c009040 comp rx=0 tx=0).stop 2024-01-24T17:44:25.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07640cce00 msgr2=0x7f076415dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:25.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07640cce00 0x7f076415dcc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0754009880 tx=0x7f07540046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 shutdown_connections 2024-01-24T17:44:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f076415ca40 0x7f07641f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07640cdb70 0x7f076415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f073c06a430 0x7f073c06c8f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 --2- 172.21.15.39:0/2891525155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07640cce00 0x7f076415dcc0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 >> 172.21.15.39:0/2891525155 conn(0x7f0764069e50 msgr2=0x7f07640ae080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 shutdown_connections 2024-01-24T17:44:25.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.258+0000 7f076ac00640 1 -- 172.21.15.39:0/2891525155 wait complete. 2024-01-24T17:44:25.286 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:44:25.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.490+0000 7fc527e00640 1 Processor -- start 2024-01-24T17:44:25.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.490+0000 7fc527e00640 1 -- start start 2024-01-24T17:44:25.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.492+0000 7fc527e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815a720 0x7fc5280cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:25.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.492+0000 7fc527e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc5280ce8e0 0x7fc5280ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:25.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.492+0000 7fc527e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5280cf290 con 0x7fc52815ab00 2024-01-24T17:44:25.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.492+0000 7fc527e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5280ce400 con 0x7fc5280ce8e0 2024-01-24T17:44:25.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.492+0000 7fc527e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280ccef0 con 0x7fc52815a720 2024-01-24T17:44:25.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.492+0000 7fc527400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc5280ce8e0 0x7fc5280ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:25.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.493+0000 7fc527400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc5280ce8e0 0x7fc5280ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46368/0 (socket says 172.21.15.39:46368) 2024-01-24T17:44:25.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.493+0000 7fc527400640 1 -- 172.21.15.39:0/2173093008 learned_addr learned my addr 172.21.15.39:0/2173093008 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:25.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.493+0000 7fc526a00640 1 --2- 172.21.15.39:0/2173093008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815a720 0x7fc5280cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:25.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.493+0000 7fc527400640 1 -- 172.21.15.39:0/2173093008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815a720 msgr2=0x7fc5280cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:25.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.493+0000 7fc527400640 1 --2- 172.21.15.39:0/2173093008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815a720 0x7fc5280cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.493+0000 7fc527400640 1 -- 172.21.15.39:0/2173093008 >> v1:172.21.15.100:6789/0 conn(0x7fc52815ab00 legacy=0x7fc5280cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:25.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.493+0000 7fc527400640 1 -- 172.21.15.39:0/2173093008 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280ce110 con 0x7fc5280ce8e0 2024-01-24T17:44:25.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.495+0000 7fc527400640 1 --2- 172.21.15.39:0/2173093008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280ced50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc520009e50 tx=0x7fc52002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d2926cedcf8967 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:25.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.495+0000 7fc526000640 1 -- 172.21.15.39:0/2173093008 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52002c9e0 con 0x7fc5280ce8e0 2024-01-24T17:44:25.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.495+0000 7fc526000640 1 -- 172.21.15.39:0/2173093008 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52002cb40 con 0x7fc5280ce8e0 2024-01-24T17:44:25.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.495+0000 7fc526000640 1 -- 172.21.15.39:0/2173093008 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5200343f0 con 0x7fc5280ce8e0 2024-01-24T17:44:25.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.496+0000 7fc527e00640 1 -- 172.21.15.39:0/2173093008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 msgr2=0x7fc5280ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:25.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.496+0000 7fc527e00640 1 --2- 172.21.15.39:0/2173093008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280ced50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc520009e50 tx=0x7fc52002be40 comp rx=0 tx=0).stop 2024-01-24T17:44:25.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.496+0000 7fc527e00640 1 -- 172.21.15.39:0/2173093008 shutdown_connections 2024-01-24T17:44:25.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.496+0000 7fc527e00640 1 --2- 172.21.15.39:0/2173093008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280ced50 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.496+0000 7fc527e00640 1 --2- 172.21.15.39:0/2173093008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815a720 0x7fc5280cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.496+0000 7fc527e00640 1 -- 172.21.15.39:0/2173093008 >> 172.21.15.39:0/2173093008 conn(0x7fc52806a090 msgr2=0x7fc52806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:25.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.496+0000 7fc527e00640 1 -- 172.21.15.39:0/2173093008 shutdown_connections 2024-01-24T17:44:25.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.496+0000 7fc527e00640 1 -- 172.21.15.39:0/2173093008 wait complete. 2024-01-24T17:44:25.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.498+0000 7fc527e00640 1 Processor -- start 2024-01-24T17:44:25.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.498+0000 7fc527e00640 1 -- start start 2024-01-24T17:44:25.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc527e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:25.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc527e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52815a720 0x7fc5281572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc527e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815ab00 0x7fc528157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc527e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc528157d70 con 0x7fc5280ce8e0 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc527e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b3bc0 con 0x7fc52815a720 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc527e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b3040 con 0x7fc52815ab00 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc526a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc526a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280c9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46372/0 (socket says 172.21.15.39:46372) 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.499+0000 7fc526a00640 1 -- 172.21.15.39:0/2058155361 learned_addr learned my addr 172.21.15.39:0/2058155361 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc527400640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815ab00 0x7fc528157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc51fe00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52815a720 0x7fc5281572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc526a00640 1 -- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815ab00 msgr2=0x7fc528157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc526a00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815ab00 0x7fc528157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc526a00640 1 -- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52815a720 msgr2=0x7fc5281572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:25.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc526a00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52815a720 0x7fc5281572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc526a00640 1 -- 172.21.15.39:0/2058155361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc520009b30 con 0x7fc5280ce8e0 2024-01-24T17:44:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc51fe00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52815a720 0x7fc5281572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:25.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc526a00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280c9550 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc51800d800 tx=0x7fc51800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc51ea00640 1 -- 172.21.15.39:0/2058155361 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51800b6c0 con 0x7fc5280ce8e0 2024-01-24T17:44:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc51ea00640 1 -- 172.21.15.39:0/2058155361 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51800b820 con 0x7fc5280ce8e0 2024-01-24T17:44:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc51ea00640 1 -- 172.21.15.39:0/2058155361 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51800bbb0 con 0x7fc5280ce8e0 2024-01-24T17:44:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.500+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5280b3320 con 0x7fc5280ce8e0 2024-01-24T17:44:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.501+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5280b38e0 con 0x7fc5280ce8e0 2024-01-24T17:44:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.502+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52804dff0 con 0x7fc5280ce8e0 2024-01-24T17:44:25.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.503+0000 7fc51ea00640 1 -- 172.21.15.39:0/2058155361 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc51800bd10 con 0x7fc5280ce8e0 2024-01-24T17:44:25.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.503+0000 7fc51ea00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50006a430 0x7fc50006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:25.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.504+0000 7fc51ea00640 1 -- 172.21.15.39:0/2058155361 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc518088f90 con 0x7fc5280ce8e0 2024-01-24T17:44:25.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.504+0000 7fc51fe00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50006a430 0x7fc50006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:25.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.506+0000 7fc51fe00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50006a430 0x7fc50006c8f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc5280c8cf0 tx=0x7fc510009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:25.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.509+0000 7fc51ea00640 1 -- 172.21.15.39:0/2058155361 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc518054450 con 0x7fc5280ce8e0 2024-01-24T17:44:25.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.743+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc52806c8a0 con 0x7fc5280ce8e0 2024-01-24T17:44:25.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.744+0000 7fc51ea00640 1 -- 172.21.15.39:0/2058155361 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4756 (secure 0 0 0) 0x7fc518057a70 con 0x7fc5280ce8e0 2024-01-24T17:44:25.745 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:25.745 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:44:09.961260+0000","modified":"2024-01-24T17:44:25.254942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5769,"mds_1":5763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5763":{"gid":5763,"name":"b","rank":1,"incarnation":25,"state":"up:creating","state_seq":1,"addr":"172.21.15.100:6837/1418568795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1418568795},{"type":"v1","addr":"172.21.15.100:6837","nonce":1418568795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5769":{"gid":5769,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/1882733155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1882733155},{"type":"v1","addr":"172.21.15.39:6835","nonce":1882733155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-24T17:44:25.746 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 25 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50006a430 msgr2=0x7fc50006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50006a430 0x7fc50006c8f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc5280c8cf0 tx=0x7fc510009040 comp rx=0 tx=0).stop 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 msgr2=0x7fc5280c9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280c9550 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc51800d800 tx=0x7fc51800dcc0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 shutdown_connections 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52815ab00 0x7fc528157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52815a720 0x7fc5281572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50006a430 0x7fc50006c8f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 --2- 172.21.15.39:0/2058155361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5280ce8e0 0x7fc5280c9550 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 >> 172.21.15.39:0/2058155361 conn(0x7fc52806a090 msgr2=0x7fc528158260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 shutdown_connections 2024-01-24T17:44:25.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:25.748+0000 7fc527e00640 1 -- 172.21.15.39:0/2058155361 wait complete. 2024-01-24T17:44:26.170 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:44:09.961260+0000', 'modified': '2024-01-24T17:44:25.254942+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5769, 'mds_1': 5763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5763': {'gid': 5763, 'name': 'b', 'rank': 1, 'incarnation': 25, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.100:6837/1418568795', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6835', 'nonce': 1418568795}, {'type': 'v1', 'addr': '172.21.15.100:6837', 'nonce': 1418568795}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_5769': {'gid': 5769, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6835/1882733155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6833', 'nonce': 1882733155}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 1882733155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:44:26.170 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5763:up:creating 2024-01-24T17:44:27.172 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:44:27.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.405+0000 7f6b7d000640 1 Processor -- start 2024-01-24T17:44:27.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.405+0000 7f6b7d000640 1 -- start start 2024-01-24T17:44:27.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.407+0000 7f6b7d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b780c9470 0x7f6b78160410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:27.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.407+0000 7f6b7d000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6b780cb460 0x7f6b78160950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:27.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.407+0000 7f6b7d000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7815cce0 con 0x7f6b7815c900 2024-01-24T17:44:27.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.407+0000 7f6b7d000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b780cd410 con 0x7f6b780cb460 2024-01-24T17:44:27.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.407+0000 7f6b7d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780cc650 con 0x7f6b780c9470 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.407+0000 7f6b77400640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f6b7815c900 0x7f6b780cad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:36410/0 (socket says 172.21.15.39:36410) 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.407+0000 7f6b77400640 1 -- 172.21.15.39:0/618036033 learned_addr learned my addr 172.21.15.39:0/618036033 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.407+0000 7f6b76a00640 1 --2- 172.21.15.39:0/618036033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b780c9470 0x7f6b78160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.408+0000 7f6b77e00640 1 --2- 172.21.15.39:0/618036033 >> v2:172.21.15.100:3300/0 conn(0x7f6b780cb460 0x7f6b78160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.408+0000 7f6b76a00640 1 -- 172.21.15.39:0/618036033 >> v2:172.21.15.100:3300/0 conn(0x7f6b780cb460 msgr2=0x7f6b78160950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.408+0000 7f6b76a00640 1 --2- 172.21.15.39:0/618036033 >> v2:172.21.15.100:3300/0 conn(0x7f6b780cb460 0x7f6b78160950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.408+0000 7f6b76a00640 1 -- 172.21.15.39:0/618036033 >> v1:172.21.15.100:6789/0 conn(0x7f6b7815c900 legacy=0x7f6b780cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.408+0000 7f6b76a00640 1 -- 172.21.15.39:0/618036033 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b780cc930 con 0x7f6b780c9470 2024-01-24T17:44:27.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.408+0000 7f6b77e00640 1 --2- 172.21.15.39:0/618036033 >> v2:172.21.15.100:3300/0 conn(0x7f6b780cb460 0x7f6b78160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:44:27.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.410+0000 7f6b76a00640 1 --2- 172.21.15.39:0/618036033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b780c9470 0x7f6b78160410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6b600097a0 tx=0x7f6b6002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=344d146a0fbadd96 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:27.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.410+0000 7f6b76000640 1 -- 172.21.15.39:0/618036033 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60031780 con 0x7f6b780c9470 2024-01-24T17:44:27.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.410+0000 7f6b76000640 1 -- 172.21.15.39:0/618036033 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b600318e0 con 0x7f6b780c9470 2024-01-24T17:44:27.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.410+0000 7f6b76000640 1 -- 172.21.15.39:0/618036033 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60031c70 con 0x7f6b780c9470 2024-01-24T17:44:27.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.411+0000 7f6b7d000640 1 -- 172.21.15.39:0/618036033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b780c9470 msgr2=0x7f6b78160410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:27.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.411+0000 7f6b7d000640 1 --2- 172.21.15.39:0/618036033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b780c9470 0x7f6b78160410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6b600097a0 tx=0x7f6b6002b820 comp rx=0 tx=0).stop 2024-01-24T17:44:27.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.411+0000 7f6b7d000640 1 -- 172.21.15.39:0/618036033 shutdown_connections 2024-01-24T17:44:27.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.411+0000 7f6b7d000640 1 --2- 172.21.15.39:0/618036033 >> v2:172.21.15.100:3300/0 conn(0x7f6b780cb460 0x7f6b78160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.411+0000 7f6b7d000640 1 --2- 172.21.15.39:0/618036033 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b780c9470 0x7f6b78160410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.411+0000 7f6b7d000640 1 -- 172.21.15.39:0/618036033 >> 172.21.15.39:0/618036033 conn(0x7f6b780b0570 msgr2=0x7f6b78069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:27.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.411+0000 7f6b7d000640 1 -- 172.21.15.39:0/618036033 shutdown_connections 2024-01-24T17:44:27.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.411+0000 7f6b7d000640 1 -- 172.21.15.39:0/618036033 wait complete. 2024-01-24T17:44:27.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.412+0000 7f6b7d000640 1 Processor -- start 2024-01-24T17:44:27.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.413+0000 7f6b7d000640 1 -- start start 2024-01-24T17:44:27.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.413+0000 7f6b7d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b780cb460 0x7f6b780b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:27.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.413+0000 7f6b7d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b7815c900 0x7f6b780acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:27.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.413+0000 7f6b7d000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b780c1aa0 0x7f6b78158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:27.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.413+0000 7f6b7d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780ae1c0 con 0x7f6b780cb460 2024-01-24T17:44:27.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.413+0000 7f6b7d000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780c2060 con 0x7f6b780c1aa0 2024-01-24T17:44:27.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b7d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b780c2340 con 0x7f6b7815c900 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b780cb460 0x7f6b780b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b780cb460 0x7f6b780b3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46388/0 (socket says 172.21.15.39:46388) 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77400640 1 -- 172.21.15.39:0/2715773701 learned_addr learned my addr 172.21.15.39:0/2715773701 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b76a00640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b7815c900 0x7f6b780acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77e00640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b780c1aa0 0x7f6b78158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77400640 1 -- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b7815c900 msgr2=0x7f6b780acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77400640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b7815c900 0x7f6b780acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77400640 1 -- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b780c1aa0 msgr2=0x7f6b78158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77400640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b780c1aa0 0x7f6b78158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77400640 1 -- 172.21.15.39:0/2715773701 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b70009a70 con 0x7f6b780cb460 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.414+0000 7f6b77e00640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b780c1aa0 0x7f6b78158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:27.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.415+0000 7f6b77400640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b780cb460 0x7f6b780b3970 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c4a0 tx=0x7f6b6800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:27.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.415+0000 7f6b6f400640 1 -- 172.21.15.39:0/2715773701 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68005ce0 con 0x7f6b780cb460 2024-01-24T17:44:27.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.415+0000 7f6b6f400640 1 -- 172.21.15.39:0/2715773701 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b68005e40 con 0x7f6b780cb460 2024-01-24T17:44:27.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.415+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b60009400 con 0x7f6b780cb460 2024-01-24T17:44:27.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.415+0000 7f6b6f400640 1 -- 172.21.15.39:0/2715773701 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68007270 con 0x7f6b780cb460 2024-01-24T17:44:27.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.415+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b780c2900 con 0x7f6b780cb460 2024-01-24T17:44:27.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.417+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7804dff0 con 0x7f6b780cb460 2024-01-24T17:44:27.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.417+0000 7f6b6f400640 1 -- 172.21.15.39:0/2715773701 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6b680073d0 con 0x7f6b780cb460 2024-01-24T17:44:27.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.419+0000 7f6b6f400640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6b5006a430 0x7f6b5006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:27.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.419+0000 7f6b6f400640 1 -- 172.21.15.39:0/2715773701 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6b68087e00 con 0x7f6b780cb460 2024-01-24T17:44:27.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.419+0000 7f6b76a00640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6b5006a430 0x7f6b5006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:27.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.420+0000 7f6b76a00640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6b5006a430 0x7f6b5006c8f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6b60036000 tx=0x7f6b6003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:27.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.424+0000 7f6b6f400640 1 -- 172.21.15.39:0/2715773701 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6b68053410 con 0x7f6b780cb460 2024-01-24T17:44:27.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.656+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b780ce190 con 0x7f6b780cb460 2024-01-24T17:44:27.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.657+0000 7f6b6f400640 1 -- 172.21.15.39:0/2715773701 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4754 (secure 0 0 0) 0x7f6b68017090 con 0x7f6b780cb460 2024-01-24T17:44:27.659 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:27.659 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:44:09.961260+0000","modified":"2024-01-24T17:44:26.259619+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5769,"mds_1":5763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5763":{"gid":5763,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.100:6837/1418568795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1418568795},{"type":"v1","addr":"172.21.15.100:6837","nonce":1418568795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5769":{"gid":5769,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/1882733155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1882733155},{"type":"v1","addr":"172.21.15.39:6835","nonce":1882733155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-24T17:44:27.659 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 26 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6b5006a430 msgr2=0x7f6b5006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6b5006a430 0x7f6b5006c8f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6b60036000 tx=0x7f6b6003d040 comp rx=0 tx=0).stop 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b780cb460 msgr2=0x7f6b780b3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b780cb460 0x7f6b780b3970 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c4a0 tx=0x7f6b6800c960 comp rx=0 tx=0).stop 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 shutdown_connections 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b780c1aa0 0x7f6b78158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b7815c900 0x7f6b780acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6b5006a430 0x7f6b5006c8f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 --2- 172.21.15.39:0/2715773701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b780cb460 0x7f6b780b3970 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:27.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.660+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 >> 172.21.15.39:0/2715773701 conn(0x7f6b780b0570 msgr2=0x7f6b78069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.661+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 shutdown_connections 2024-01-24T17:44:27.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:27.661+0000 7f6b7d000640 1 -- 172.21.15.39:0/2715773701 wait complete. 2024-01-24T17:44:28.085 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:44:09.961260+0000', 'modified': '2024-01-24T17:44:26.259619+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5769, 'mds_1': 5763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5763': {'gid': 5763, 'name': 'b', 'rank': 1, 'incarnation': 25, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.100:6837/1418568795', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6835', 'nonce': 1418568795}, {'type': 'v1', 'addr': '172.21.15.100:6837', 'nonce': 1418568795}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_5769': {'gid': 5769, 'name': 'c', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6835/1882733155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6833', 'nonce': 1882733155}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 1882733155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:44:28.085 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-01-24T17:44:28.085 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-24T17:44:28.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.322+0000 7fbbe5a00640 1 Processor -- start 2024-01-24T17:44:28.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.323+0000 7fbbe5a00640 1 -- start start 2024-01-24T17:44:28.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.324+0000 7fbbe5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 0x7fbbe015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.324+0000 7fbbe5a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe015dac0 con 0x7fbbe015eb80 2024-01-24T17:44:28.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.324+0000 7fbbe5a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe015dda0 con 0x7fbbe00cf990 2024-01-24T17:44:28.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.324+0000 7fbbe5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe015e080 con 0x7fbbe015d020 2024-01-24T17:44:28.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.325+0000 7fbbdf400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fbbe015eb80 0x7fbbe015f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53864/0 (socket says 172.21.15.39:53864) 2024-01-24T17:44:28.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.325+0000 7fbbdf400640 1 -- 172.21.15.39:0/1877640971 learned_addr learned my addr 172.21.15.39:0/1877640971 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:28.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.325+0000 7fbbde000640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 0x7fbbe015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.325+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665574360 0 0) 0x7fbbe015dac0 con 0x7fbbe015eb80 2024-01-24T17:44:28.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.325+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbc0003520 con 0x7fbbe015eb80 2024-01-24T17:44:28.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2006484270 0 0) 0x7fbbc0003520 con 0x7fbbe015eb80 2024-01-24T17:44:28.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 msgr2=0x7fbbe015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 0x7fbbe015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 >> v1:172.21.15.100:6789/0 conn(0x7fbbe00cf990 legacy=0x7fbbe00cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe015e380 con 0x7fbbe015eb80 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fbbd80019f0 con 0x7fbbe015eb80 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 >> v1:172.21.15.39:6789/0 conn(0x7fbbe015eb80 legacy=0x7fbbe015f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbc00039a0 0x7fbbc0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc00047c0 0x7fbbc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 0x7fbbc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe015e080 con 0x7fbbc00047c0 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc000ba30 con 0x7fbbc00039a0 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc000bbf0 con 0x7fbbe015d020 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.326+0000 7fbbde000640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc00047c0 0x7fbbc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbdea00640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbc00039a0 0x7fbbc0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbdf400640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 0x7fbbc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbde000640 1 -- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 msgr2=0x7fbbc0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbde000640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 0x7fbbc0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbde000640 1 -- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbc00039a0 msgr2=0x7fbbc0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbde000640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbc00039a0 0x7fbbc0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbde000640 1 -- 172.21.15.39:0/1877640971 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc0003520 con 0x7fbbc00047c0 2024-01-24T17:44:28.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbdea00640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbc00039a0 0x7fbbc0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:28.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.327+0000 7fbbdf400640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbe015d020 0x7fbbc0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:28.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.328+0000 7fbbde000640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc00047c0 0x7fbbc0004c40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009a90 tx=0x7fbbc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0b917a50ca1c5b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.328+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc802c680 con 0x7fbbc00047c0 2024-01-24T17:44:28.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.328+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc802c7e0 con 0x7fbbc00047c0 2024-01-24T17:44:28.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.328+0000 7fbbdd600640 1 -- 172.21.15.39:0/1877640971 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc802cb70 con 0x7fbbc00047c0 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.329+0000 7fbbe5a00640 1 -- 172.21.15.39:0/1877640971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc00047c0 msgr2=0x7fbbc0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.329+0000 7fbbe5a00640 1 --2- 172.21.15.39:0/1877640971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc00047c0 0x7fbbc0004c40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009a90 tx=0x7fbbc00057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.329+0000 7fbbdf400640 1 -- 172.21.15.39:0/1877640971 reap_dead start 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.329+0000 7fbbe5a00640 1 -- 172.21.15.39:0/1877640971 shutdown_connections 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.329+0000 7fbbe5a00640 1 -- 172.21.15.39:0/1877640971 >> 172.21.15.39:0/1877640971 conn(0x7fbbe0071790 msgr2=0x7fbbe0071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.329+0000 7fbbe5a00640 1 -- 172.21.15.39:0/1877640971 shutdown_connections 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.329+0000 7fbbe5a00640 1 -- 172.21.15.39:0/1877640971 wait complete. 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.330+0000 7fbbe5a00640 1 Processor -- start 2024-01-24T17:44:28.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.331+0000 7fbbe5a00640 1 -- start start 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.331+0000 7fbbe5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbc00047c0 0x7fbbe0155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbe5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe00d16f0 0x7fbbe00d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbe5a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbe00b33b0 0x7fbbe00b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbe5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe01559c0 con 0x7fbbe00d16f0 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbe5a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe00b3d70 con 0x7fbbe00b33b0 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbe5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe0156c60 con 0x7fbbc00047c0 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbde000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe00d16f0 0x7fbbe00d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbde000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe00d16f0 0x7fbbe00d1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46406/0 (socket says 172.21.15.39:46406) 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbde000640 1 -- 172.21.15.39:0/746979415 learned_addr learned my addr 172.21.15.39:0/746979415 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbdea00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbc00047c0 0x7fbbe0155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbde000640 1 -- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbc00047c0 msgr2=0x7fbbe0155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbde000640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbc00047c0 0x7fbbe0155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbde000640 1 -- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbe00b33b0 msgr2=0x7fbbe00b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbde000640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbe00b33b0 0x7fbbe00b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.332+0000 7fbbde000640 1 -- 172.21.15.39:0/746979415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe00c66d0 con 0x7fbbe00d16f0 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.333+0000 7fbbde000640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe00d16f0 0x7fbbe00d1b70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8037040 tx=0x7fbbc8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.333+0000 7fbbd6a00640 1 -- 172.21.15.39:0/746979415 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8003810 con 0x7fbbe00d16f0 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.333+0000 7fbbd6a00640 1 -- 172.21.15.39:0/746979415 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc8003970 con 0x7fbbe00d16f0 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.333+0000 7fbbe5a00640 1 -- 172.21.15.39:0/746979415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe0156980 con 0x7fbbe00d16f0 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.333+0000 7fbbd6a00640 1 -- 172.21.15.39:0/746979415 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8003ce0 con 0x7fbbe00d16f0 2024-01-24T17:44:28.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.333+0000 7fbbe5a00640 1 -- 172.21.15.39:0/746979415 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe00c6c50 con 0x7fbbe00d16f0 2024-01-24T17:44:28.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.335+0000 7fbbd6a00640 1 -- 172.21.15.39:0/746979415 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbbc8034440 con 0x7fbbe00d16f0 2024-01-24T17:44:28.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.336+0000 7fbbd6a00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbbb806a460 0x7fbbb806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.336+0000 7fbbd6a00640 1 -- 172.21.15.39:0/746979415 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fbbc80aa940 con 0x7fbbe00d16f0 2024-01-24T17:44:28.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.337+0000 7fbbdea00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbbb806a460 0x7fbbb806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.337+0000 7fbbe8805740 1 Processor -- start 2024-01-24T17:44:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.337+0000 7fbbe8805740 1 -- start start 2024-01-24T17:44:28.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.338+0000 7fbbdea00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbbb806a460 0x7fbbb806c920 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0001630 tx=0x7fbbd0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.338+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc2010 0x55f639bc23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.338+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b91f90 0x55f639bbf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639b95b90 0x55f639bbf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbe8805740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f639b6d6c0 con 0x55f639bc2010 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbe8805740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f639b02070 con 0x55f639b91f90 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbe8805740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f639aff800 con 0x55f639b95b90 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc2010 0x55f639bc23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc2010 0x55f639bc23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46418/0 (socket says 172.21.15.39:46418) 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdea00640 1 -- 172.21.15.39:0/468759444 learned_addr learned my addr 172.21.15.39:0/468759444 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdf400640 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639b95b90 0x55f639bbf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbde000640 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b91f90 0x55f639bbf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdea00640 1 -- 172.21.15.39:0/468759444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639b95b90 msgr2=0x55f639bbf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdea00640 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639b95b90 0x55f639bbf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdea00640 1 -- 172.21.15.39:0/468759444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b91f90 msgr2=0x55f639bbf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdea00640 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b91f90 0x55f639bbf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.339+0000 7fbbdea00640 1 -- 172.21.15.39:0/468759444 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f639ac2780 con 0x55f639bc2010 2024-01-24T17:44:28.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.340+0000 7fbbdea00640 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc2010 0x55f639bc23f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0007270 tx=0x7fbbd0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b104ceb2a3702d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.340+0000 7fbbcf400640 1 -- 172.21.15.39:0/468759444 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd000c050 con 0x55f639bc2010 2024-01-24T17:44:28.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.340+0000 7fbbcf400640 1 -- 172.21.15.39:0/468759444 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd000d040 con 0x55f639bc2010 2024-01-24T17:44:28.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.340+0000 7fbbcf400640 1 -- 172.21.15.39:0/468759444 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd000e9b0 con 0x55f639bc2010 2024-01-24T17:44:28.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 -- 172.21.15.39:0/468759444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc2010 msgr2=0x55f639bc23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc2010 0x55f639bc23f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0007270 tx=0x7fbbd0011000 comp rx=0 tx=0).stop 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 -- 172.21.15.39:0/468759444 shutdown_connections 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639b95b90 0x55f639bbf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b91f90 0x55f639bbf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 --2- 172.21.15.39:0/468759444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc2010 0x55f639bc23f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 -- 172.21.15.39:0/468759444 >> 172.21.15.39:0/468759444 conn(0x55f639b942a0 msgr2=0x55f639bdce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 -- 172.21.15.39:0/468759444 shutdown_connections 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.341+0000 7fbbe8805740 1 -- 172.21.15.39:0/468759444 wait complete. 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.342+0000 7fbbe8805740 1 -- start start 2024-01-24T17:44:28.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.342+0000 7fbbe8805740 1 Processor -- start 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbe0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 0x55f639bc6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc72d0 0x55f639bc96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbe8805740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f639afb0f0 con 0x55f639b95b90 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbe8805740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f639a9c8c0 con 0x55f639bc72d0 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbe8805740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f639a9c560 con 0x55f639bc2010 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbe8805740 10 client.0 _ll_register_callbacks cb 0x55f639af4a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbdea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbe0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbdea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbe0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46434/0 (socket says 172.21.15.39:46434) 2024-01-24T17:44:28.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbdea00640 1 -- 172.21.15.39:0/390740462 learned_addr learned my addr 172.21.15.39:0/390740462 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbdea00640 1 -- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 msgr2=0x55f639bc6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbdea00640 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 0x55f639bc6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbdea00640 1 -- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc72d0 msgr2=0x55f639bc96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbdea00640 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc72d0 0x55f639bc96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.343+0000 7fbbdea00640 1 -- 172.21.15.39:0/390740462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe015dda0 con 0x55f639b95b90 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.344+0000 7fbbdea00640 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbe0c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0007dc0 tx=0x7fbbd0001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58fcba5b0c9e97a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.344+0000 7fbbc7e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.344+0000 7fbbc7e00640 1 -- 172.21.15.39:0/390740462 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd000c070 con 0x55f639b95b90 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.344+0000 7fbbc7e00640 1 -- 172.21.15.39:0/390740462 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fbbd0005470 con 0x55f639b95b90 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.344+0000 7fbbe8805740 1 -- 172.21.15.39:0/390740462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f639b25640 con 0x55f639b95b90 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.344+0000 7fbbc7e00640 1 -- 172.21.15.39:0/390740462 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbd0003db0 con 0x55f639b95b90 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.344+0000 7fbbe8805740 10 client.5961 fetch_fsmap learned FSMap version 26 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.344+0000 7fbbe8805740 1 -- 172.21.15.39:0/390740462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55f639af9770 con 0x55f639b95b90 2024-01-24T17:44:28.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.345+0000 7fbbc7e00640 1 -- 172.21.15.39:0/390740462 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f639b25640 con 0x55f639b95b90 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.345+0000 7fbbc7e00640 1 -- 172.21.15.39:0/390740462 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 3126+0+0 (secure 0 0 0) 0x7fbbd000e590 con 0x55f639b95b90 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.345+0000 7fbbe8805740 10 client.5961 fetch_fsmap finished waiting for FSMap version 26 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.345+0000 7fbbe8805740 10 client.5961 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.345+0000 7fbbe8805740 20 client.5961 populate_metadata read hostname 'smithi039' 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.345+0000 7fbbe8805740 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] conn(0x55f639bed100 0x55f639bc39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.345+0000 7fbbe8805740 4 client.5961 mds_command: new command op to 5763 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.345+0000 7fbbe8805740 1 -- 172.21.15.39:0/390740462 --> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f639aed940 con 0x55f639bed100 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.346+0000 7fbbde000640 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] conn(0x55f639bed100 0x55f639bc39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.347+0000 7fbbde000640 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] conn(0x55f639bed100 0x55f639bc39a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6e2a7c1795c99e8a server_cookie=7857daef3e6f9b6b in_seq=0 out_seq=0 2024-01-24T17:44:28.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.347+0000 7fbbc7e00640 10 client.5961 ms_handle_connect on v2:172.21.15.100:6835/1418568795 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.348+0000 7fbbc7e00640 1 -- 172.21.15.39:0/390740462 <== mds.1 v2:172.21.15.100:6835/1418568795 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55f639aed940 con 0x55f639bed100 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.348+0000 7fbbc7e00640 10 client.5961 handle_command_reply: tid=0 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 client.5961 shutdown 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 -- 172.21.15.39:0/390740462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 msgr2=0x55f639bbe0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbe0c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0007dc0 tx=0x7fbbd0001190 comp rx=0 tx=0).stop 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 -- 172.21.15.39:0/390740462 shutdown_connections 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] conn(0x55f639bed100 0x55f639bc39a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc72d0 0x55f639bc96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 0x55f639bc6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 --2- 172.21.15.39:0/390740462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbe0c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbc7e00640 0 client.5961 ms_handle_reset on v2:172.21.15.100:6835/1418568795 2024-01-24T17:44:28.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 -- 172.21.15.39:0/390740462 >> 172.21.15.39:0/390740462 conn(0x55f639b91f90 msgr2=0x55f639b96560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 -- 172.21.15.39:0/390740462 shutdown_connections 2024-01-24T17:44:28.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 1 -- 172.21.15.39:0/390740462 wait complete. 2024-01-24T17:44:28.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.349+0000 7fbbe8805740 20 client.5961 trim_cache size 0 max 16384 2024-01-24T17:44:28.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.366+0000 7fbbe8805740 1 Processor -- start 2024-01-24T17:44:28.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.366+0000 7fbbe8805740 1 -- start start 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b95b90 0x55f639bbeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 0x55f639beb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc72d0 0x55f639beb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbe8805740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f639862f20 con 0x55f639bc72d0 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbe8805740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f639bcadd0 con 0x55f639b95b90 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbe8805740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f639bc8280 con 0x55f639bc2010 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbdf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc72d0 0x55f639beb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbdf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc72d0 0x55f639beb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46438/0 (socket says 172.21.15.39:46438) 2024-01-24T17:44:28.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.367+0000 7fbbdf400640 1 -- 172.21.15.39:0/275077446 learned_addr learned my addr 172.21.15.39:0/275077446 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbdea00640 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b95b90 0x55f639bbeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbde000640 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 0x55f639beb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbdf400640 1 -- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 msgr2=0x55f639beb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbdf400640 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 0x55f639beb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbdf400640 1 -- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b95b90 msgr2=0x55f639bbeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbdf400640 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b95b90 0x55f639bbeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbdf400640 1 -- 172.21.15.39:0/275077446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f639bbe470 con 0x55f639bc72d0 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbdea00640 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b95b90 0x55f639bbeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbde000640 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 0x55f639beb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbdf400640 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc72d0 0x55f639beb580 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbbe015ef60 tx=0x7fbbd8005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff7b92ba160915b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbcfe00640 1 -- 172.21.15.39:0/275077446 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80077d0 con 0x55f639bc72d0 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbcfe00640 1 -- 172.21.15.39:0/275077446 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd800d070 con 0x55f639bc72d0 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.368+0000 7fbbcfe00640 1 -- 172.21.15.39:0/275077446 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80133f0 con 0x55f639bc72d0 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 -- 172.21.15.39:0/275077446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc72d0 msgr2=0x55f639beb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc72d0 0x55f639beb580 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbbe015ef60 tx=0x7fbbd8005f40 comp rx=0 tx=0).stop 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 -- 172.21.15.39:0/275077446 shutdown_connections 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639bc72d0 0x55f639beb580 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc2010 0x55f639beb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 --2- 172.21.15.39:0/275077446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639b95b90 0x55f639bbeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 -- 172.21.15.39:0/275077446 >> 172.21.15.39:0/275077446 conn(0x55f639b91f90 msgr2=0x55f639bc9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 -- 172.21.15.39:0/275077446 shutdown_connections 2024-01-24T17:44:28.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.369+0000 7fbbe8805740 1 -- 172.21.15.39:0/275077446 wait complete. 2024-01-24T17:44:28.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.370+0000 7fbbe8805740 1 -- start start 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.370+0000 7fbbe8805740 1 Processor -- start 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc2010 0x55f639bbd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbe8805740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc72d0 0x55f639bc85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbe8805740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f639bc94a0 con 0x55f639b95b90 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbe8805740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f639beba50 con 0x55f639bc2010 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbe8805740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f639bf6c70 con 0x55f639bc72d0 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbe8805740 10 client.0 _ll_register_callbacks cb 0x7fbbd0023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbdea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbdea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbcb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46440/0 (socket says 172.21.15.39:46440) 2024-01-24T17:44:28.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.371+0000 7fbbdea00640 1 -- 172.21.15.39:0/3082477081 learned_addr learned my addr 172.21.15.39:0/3082477081 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbdea00640 1 -- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc72d0 msgr2=0x55f639bc85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbdf400640 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc72d0 0x55f639bc85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbde000640 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc2010 0x55f639bbd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbdea00640 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc72d0 0x55f639bc85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbdea00640 1 -- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc2010 msgr2=0x55f639bbd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbdea00640 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc2010 0x55f639bbd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbdea00640 1 -- 172.21.15.39:0/3082477081 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe015e380 con 0x55f639b95b90 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbdf400640 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc72d0 0x55f639bc85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbdea00640 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbcb90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000e4a0 tx=0x7fbbd0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2307572be1bb1c4d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbc7e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbc7e00640 1 -- 172.21.15.39:0/3082477081 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd000eaa0 con 0x55f639b95b90 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbc7e00640 1 -- 172.21.15.39:0/3082477081 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fbbd0003bb0 con 0x55f639b95b90 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.372+0000 7fbbe8805740 1 -- 172.21.15.39:0/3082477081 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbd000d040 con 0x55f639b95b90 2024-01-24T17:44:28.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbc7e00640 1 -- 172.21.15.39:0/3082477081 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbd0003d70 con 0x55f639b95b90 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbe8805740 10 client.5967 fetch_fsmap learned FSMap version 26 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbe8805740 1 -- 172.21.15.39:0/3082477081 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55f639b2f2f0 con 0x55f639b95b90 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbc7e00640 1 -- 172.21.15.39:0/3082477081 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd000d040 con 0x55f639b95b90 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbc7e00640 1 -- 172.21.15.39:0/3082477081 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 3126+0+0 (secure 0 0 0) 0x7fbbd0013740 con 0x55f639b95b90 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbe8805740 10 client.5967 fetch_fsmap finished waiting for FSMap version 26 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbe8805740 10 client.5967 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbe8805740 20 client.5967 populate_metadata read hostname 'smithi039' 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbe8805740 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] conn(0x55f639bfecf0 0x55f639bc8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbe8805740 4 client.5967 mds_command: new command op to 5763 tid=0[{"prefix": "status"}] 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.373+0000 7fbbe8805740 1 -- 172.21.15.39:0/3082477081 --> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f639af4170 con 0x55f639bfecf0 2024-01-24T17:44:28.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.374+0000 7fbbde000640 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] conn(0x55f639bfecf0 0x55f639bc8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbde000640 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] conn(0x55f639bfecf0 0x55f639bc8b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=37389c9dee82db9d server_cookie=e77fc36e4bf38764 in_seq=0 out_seq=0 2024-01-24T17:44:28.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbc7e00640 10 client.5967 ms_handle_connect on v2:172.21.15.100:6835/1418568795 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbc7e00640 1 -- 172.21.15.39:0/3082477081 <== mds.1 v2:172.21.15.100:6835/1418568795 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55f639af4170 con 0x55f639bfecf0 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbc7e00640 10 client.5967 handle_command_reply: tid=0 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 client.5967 shutdown 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 -- 172.21.15.39:0/3082477081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 msgr2=0x55f639bbcb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbcb90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000e4a0 tx=0x7fbbd0014000 comp rx=0 tx=0).stop 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 -- 172.21.15.39:0/3082477081 shutdown_connections 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:6835/1418568795,v1:172.21.15.100:6837/1418568795] conn(0x55f639bfecf0 0x55f639bc8b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55f639bc72d0 0x55f639bc85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55f639bc2010 0x55f639bbd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 --2- 172.21.15.39:0/3082477081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f639b95b90 0x55f639bbcb90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbc7e00640 0 client.5967 ms_handle_reset on v2:172.21.15.100:6835/1418568795 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.375+0000 7fbbe8805740 1 -- 172.21.15.39:0/3082477081 >> 172.21.15.39:0/3082477081 conn(0x55f639b91f90 msgr2=0x55f639bc4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe8805740 1 -- 172.21.15.39:0/3082477081 shutdown_connections 2024-01-24T17:44:28.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe8805740 1 -- 172.21.15.39:0/3082477081 wait complete. 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe8805740 20 client.5967 trim_cache size 0 max 16384 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 1, 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5763, 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 3.1159886860000001, 2024-01-24T17:44:28.380 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 26, 2024-01-24T17:44:28.381 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 50, 2024-01-24T17:44:28.381 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 50, 2024-01-24T17:44:28.381 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 24.254008102 2024-01-24T17:44:28.381 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:44:28.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 -- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbbb806a460 msgr2=0x7fbbb806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbbb806a460 0x7fbbb806c920 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0001630 tx=0x7fbbd0009040 comp rx=0 tx=0).stop 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 -- 172.21.15.39:0/746979415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe00d16f0 msgr2=0x7fbbe00d1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe00d16f0 0x7fbbe00d1b70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8037040 tx=0x7fbbc8004730 comp rx=0 tx=0).stop 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 -- 172.21.15.39:0/746979415 shutdown_connections 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbbe00b33b0 0x7fbbe00b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbbb806a460 0x7fbbb806c920 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe00d16f0 0x7fbbe00d1b70 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 --2- 172.21.15.39:0/746979415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbbc00047c0 0x7fbbe0155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.376+0000 7fbbe5a00640 1 -- 172.21.15.39:0/746979415 >> 172.21.15.39:0/746979415 conn(0x7fbbe0071790 msgr2=0x7fbbe0159140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.377+0000 7fbbe5a00640 1 -- 172.21.15.39:0/746979415 shutdown_connections 2024-01-24T17:44:28.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.377+0000 7fbbe5a00640 1 -- 172.21.15.39:0/746979415 wait complete. 2024-01-24T17:44:28.396 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-24T17:44:28.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.616+0000 7fc049a00640 1 Processor -- start 2024-01-24T17:44:28.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.616+0000 7fc049a00640 1 -- start start 2024-01-24T17:44:28.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.618+0000 7fc049a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 0x7fc04415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.618+0000 7fc049a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04415b240 con 0x7fc0440c8c10 2024-01-24T17:44:28.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.618+0000 7fc049a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04415b520 con 0x7fc0440cb400 2024-01-24T17:44:28.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.618+0000 7fc049a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04415b800 con 0x7fc0440cb020 2024-01-24T17:44:28.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc049000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc0440c8c10 0x7fc04415e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53874/0 (socket says 172.21.15.39:53874) 2024-01-24T17:44:28.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc049000640 1 -- 172.21.15.39:0/1895471150 learned_addr learned my addr 172.21.15.39:0/1895471150 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:28.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc043e00640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 0x7fc04415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266692226 0 0) 0x7fc04415b240 con 0x7fc0440c8c10 2024-01-24T17:44:28.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc024003520 con 0x7fc0440c8c10 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2721125224 0 0) 0x7fc024003520 con 0x7fc0440c8c10 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 msgr2=0x7fc04415ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc043400640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 0x7fc04415ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.619+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 >> v1:172.21.15.100:6789/0 conn(0x7fc0440cb400 legacy=0x7fc0440ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04415bae0 con 0x7fc0440c8c10 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc03c0019f0 con 0x7fc0440c8c10 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 >> v1:172.21.15.39:6789/0 conn(0x7fc0440c8c10 legacy=0x7fc04415e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0240039a0 0x7fc024003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 0x7fc0240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc024004d00 0x7fc024005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04415b800 con 0x7fc024004d00 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02400ba60 con 0x7fc0240039a0 2024-01-24T17:44:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02400bc20 con 0x7fc0440cb020 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc049000640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc024004d00 0x7fc024005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.620+0000 7fc043e00640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0240039a0 0x7fc024003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.621+0000 7fc049000640 1 -- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 msgr2=0x7fc0240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.621+0000 7fc03be00640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 0x7fc0240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.621+0000 7fc049000640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 0x7fc0240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.621+0000 7fc049000640 1 -- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0240039a0 msgr2=0x7fc024003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.621+0000 7fc049000640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0240039a0 0x7fc024003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.621+0000 7fc049000640 1 -- 172.21.15.39:0/1895471150 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024003520 con 0x7fc024004d00 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.621+0000 7fc043e00640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0240039a0 0x7fc024003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:28.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.621+0000 7fc03be00640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0440cb020 0x7fc0240047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:28.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.622+0000 7fc049000640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc024004d00 0x7fc024005180 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc03c002230 tx=0x7fc0240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580ddc588bc72e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.622+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c034630 con 0x7fc024004d00 2024-01-24T17:44:28.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.622+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c02f070 con 0x7fc024004d00 2024-01-24T17:44:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.622+0000 7fc043400640 1 -- 172.21.15.39:0/1895471150 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0349b0 con 0x7fc024004d00 2024-01-24T17:44:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.623+0000 7fc049a00640 1 -- 172.21.15.39:0/1895471150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc024004d00 msgr2=0x7fc024005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.623+0000 7fc049a00640 1 --2- 172.21.15.39:0/1895471150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc024004d00 0x7fc024005180 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc03c002230 tx=0x7fc0240057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.623+0000 7fc049000640 1 -- 172.21.15.39:0/1895471150 reap_dead start 2024-01-24T17:44:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.623+0000 7fc049a00640 1 -- 172.21.15.39:0/1895471150 shutdown_connections 2024-01-24T17:44:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.623+0000 7fc049a00640 1 -- 172.21.15.39:0/1895471150 >> 172.21.15.39:0/1895471150 conn(0x7fc04406a2a0 msgr2=0x7fc0440afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.623+0000 7fc049a00640 1 -- 172.21.15.39:0/1895471150 shutdown_connections 2024-01-24T17:44:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.623+0000 7fc049a00640 1 -- 172.21.15.39:0/1895471150 wait complete. 2024-01-24T17:44:28.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.625+0000 7fc049a00640 1 Processor -- start 2024-01-24T17:44:28.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.625+0000 7fc049a00640 1 -- start start 2024-01-24T17:44:28.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc024004d00 0x7fc044157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044153970 0x7fc0440ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0440ccfc0 0x7fc0440cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440b1350 con 0x7fc0440ccfc0 2024-01-24T17:44:28.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440cd980 con 0x7fc044153970 2024-01-24T17:44:28.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440accc0 con 0x7fc024004d00 2024-01-24T17:44:28.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0440ccfc0 0x7fc0440cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0440ccfc0 0x7fc0440cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46444/0 (socket says 172.21.15.39:46444) 2024-01-24T17:44:28.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc049000640 1 -- 172.21.15.39:0/968701827 learned_addr learned my addr 172.21.15.39:0/968701827 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc043e00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc024004d00 0x7fc044157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.626+0000 7fc03be00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044153970 0x7fc0440ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc049000640 1 -- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc024004d00 msgr2=0x7fc044157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc049000640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc024004d00 0x7fc044157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc049000640 1 -- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044153970 msgr2=0x7fc0440ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc049000640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044153970 0x7fc0440ac4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc049000640 1 -- 172.21.15.39:0/968701827 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440aca20 con 0x7fc0440ccfc0 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc049000640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0440ccfc0 0x7fc0440cd440 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc03c02ca70 tx=0x7fc03c001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc040c00640 1 -- 172.21.15.39:0/968701827 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c001250 con 0x7fc0440ccfc0 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc040c00640 1 -- 172.21.15.39:0/968701827 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c034640 con 0x7fc0440ccfc0 2024-01-24T17:44:28.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc040c00640 1 -- 172.21.15.39:0/968701827 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0349b0 con 0x7fc0440ccfc0 2024-01-24T17:44:28.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc049a00640 1 -- 172.21.15.39:0/968701827 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc044154820 con 0x7fc0440ccfc0 2024-01-24T17:44:28.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.627+0000 7fc049a00640 1 -- 172.21.15.39:0/968701827 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc044154de0 con 0x7fc0440ccfc0 2024-01-24T17:44:28.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.630+0000 7fc040c00640 1 -- 172.21.15.39:0/968701827 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc03c031030 con 0x7fc0440ccfc0 2024-01-24T17:44:28.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.631+0000 7fc040c00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc01c06a460 0x7fc01c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.631+0000 7fc040c00640 1 -- 172.21.15.39:0/968701827 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc03c0aaca0 con 0x7fc0440ccfc0 2024-01-24T17:44:28.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.631+0000 7fc043e00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc01c06a460 0x7fc01c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.631+0000 7fc04dbbc740 1 Processor -- start 2024-01-24T17:44:28.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.632+0000 7fc04dbbc740 1 -- start start 2024-01-24T17:44:28.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.632+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e5f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.632+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e2f150 0x559309e5c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.632+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e32d50 0x559309e5c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.632+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559309e0a6c0 con 0x559309e32d50 2024-01-24T17:44:28.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.632+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559309d9f070 con 0x559309e5f1d0 2024-01-24T17:44:28.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.632+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559309d9c800 con 0x559309e2f150 2024-01-24T17:44:28.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.632+0000 7fc043e00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc01c06a460 0x7fc01c06c920 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc034009930 tx=0x7fc034008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e32d50 0x559309e5c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e32d50 0x559309e5c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46448/0 (socket says 172.21.15.39:46448) 2024-01-24T17:44:28.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 -- 172.21.15.39:0/740956123 learned_addr learned my addr 172.21.15.39:0/740956123 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc03be00640 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e2f150 0x559309e5c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 -- 172.21.15.39:0/740956123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e2f150 msgr2=0x559309e5c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e2f150 0x559309e5c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc043e00640 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e5f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 -- 172.21.15.39:0/740956123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 msgr2=0x559309e5f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e5f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 -- 172.21.15.39:0/740956123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559309d5f780 con 0x559309e32d50 2024-01-24T17:44:28.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.633+0000 7fc049000640 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e32d50 0x559309e5c980 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc03c081960 tx=0x7fc03c076320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f518b2787c97c68 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.634+0000 7fc038c00640 1 -- 172.21.15.39:0/740956123 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c081e90 con 0x559309e32d50 2024-01-24T17:44:28.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.634+0000 7fc038c00640 1 -- 172.21.15.39:0/740956123 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c077ff0 con 0x559309e32d50 2024-01-24T17:44:28.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.634+0000 7fc038c00640 1 -- 172.21.15.39:0/740956123 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c00a3b0 con 0x559309e32d50 2024-01-24T17:44:28.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.634+0000 7fc04dbbc740 1 -- 172.21.15.39:0/740956123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e32d50 msgr2=0x559309e5c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.634+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e32d50 0x559309e5c980 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc03c081960 tx=0x7fc03c076320 comp rx=0 tx=0).stop 2024-01-24T17:44:28.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.634+0000 7fc04dbbc740 1 -- 172.21.15.39:0/740956123 shutdown_connections 2024-01-24T17:44:28.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.634+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e32d50 0x559309e5c980 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.634+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e2f150 0x559309e5c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.635+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/740956123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e5f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.635+0000 7fc04dbbc740 1 -- 172.21.15.39:0/740956123 >> 172.21.15.39:0/740956123 conn(0x559309e31460 msgr2=0x559309e79fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.635+0000 7fc04dbbc740 1 -- 172.21.15.39:0/740956123 shutdown_connections 2024-01-24T17:44:28.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.635+0000 7fc04dbbc740 1 -- 172.21.15.39:0/740956123 wait complete. 2024-01-24T17:44:28.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.635+0000 7fc04dbbc740 1 -- start start 2024-01-24T17:44:28.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.636+0000 7fc04dbbc740 1 Processor -- start 2024-01-24T17:44:28.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.636+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e62410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.636+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e71830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e71d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559309d980f0 con 0x559309e59d30 2024-01-24T17:44:28.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559309d398c0 con 0x559309e5f1d0 2024-01-24T17:44:28.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559309d39560 con 0x559309e32d50 2024-01-24T17:44:28.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc04dbbc740 10 client.0 _ll_register_callbacks cb 0x559309d91a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:44:28.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc049000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e71d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc049000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e71d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46450/0 (socket says 172.21.15.39:46450) 2024-01-24T17:44:28.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc049000640 1 -- 172.21.15.39:0/636068230 learned_addr learned my addr 172.21.15.39:0/636068230 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc043e00640 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e62410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc049000640 1 -- 172.21.15.39:0/636068230 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 msgr2=0x559309e62410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc049000640 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e62410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc049000640 1 -- 172.21.15.39:0/636068230 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 msgr2=0x559309e71830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc03be00640 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e71830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc049000640 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e71830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.637+0000 7fc049000640 1 -- 172.21.15.39:0/636068230 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc044154de0 con 0x559309e59d30 2024-01-24T17:44:28.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc049000640 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e71d70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc03c07a070 tx=0x7fc03c0a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad16921095476901 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc031600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:44:28.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc031600640 1 -- 172.21.15.39:0/636068230 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c081e90 con 0x559309e59d30 2024-01-24T17:44:28.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc031600640 1 -- 172.21.15.39:0/636068230 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc03c0765c0 con 0x559309e59d30 2024-01-24T17:44:28.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc04dbbc740 1 -- 172.21.15.39:0/636068230 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559309dc2640 con 0x559309e59d30 2024-01-24T17:44:28.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc031600640 1 -- 172.21.15.39:0/636068230 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc03c076780 con 0x559309e59d30 2024-01-24T17:44:28.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc04dbbc740 10 client.5979 fetch_fsmap learned FSMap version 26 2024-01-24T17:44:28.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc04dbbc740 1 -- 172.21.15.39:0/636068230 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x559309d96770 con 0x559309e59d30 2024-01-24T17:44:28.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.638+0000 7fc031600640 1 -- 172.21.15.39:0/636068230 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559309dc2640 con 0x559309e59d30 2024-01-24T17:44:28.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc031600640 1 -- 172.21.15.39:0/636068230 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 3126+0+0 (secure 0 0 0) 0x7fc03c0982b0 con 0x559309e59d30 2024-01-24T17:44:28.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc04dbbc740 10 client.5979 fetch_fsmap finished waiting for FSMap version 26 2024-01-24T17:44:28.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc04dbbc740 10 client.5979 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:44:28.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc04dbbc740 20 client.5979 populate_metadata read hostname 'smithi039' 2024-01-24T17:44:28.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x559309e594b0 0x559309e96ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc04dbbc740 4 client.5979 mds_command: new command op to 5769 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:44:28.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc04dbbc740 1 -- 172.21.15.39:0/636068230 --> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559309d8a940 con 0x559309e594b0 2024-01-24T17:44:28.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc043e00640 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x559309e594b0 0x559309e96ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.639+0000 7fc043e00640 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x559309e594b0 0x559309e96ab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1460bd3464c8e53b server_cookie=11d30971a4d6233e in_seq=0 out_seq=0 2024-01-24T17:44:28.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.640+0000 7fc031600640 10 client.5979 ms_handle_connect on v2:172.21.15.39:6833/1882733155 2024-01-24T17:44:28.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.640+0000 7fc031600640 1 -- 172.21.15.39:0/636068230 <== mds.0 v2:172.21.15.39:6833/1882733155 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559309d8a940 con 0x559309e594b0 2024-01-24T17:44:28.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.640+0000 7fc031600640 10 client.5979 handle_command_reply: tid=0 2024-01-24T17:44:28.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.640+0000 7fc04dbbc740 1 client.5979 shutdown 2024-01-24T17:44:28.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.640+0000 7fc04dbbc740 1 -- 172.21.15.39:0/636068230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 msgr2=0x559309e71d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.640+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e71d70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc03c07a070 tx=0x7fc03c0a9000 comp rx=0 tx=0).stop 2024-01-24T17:44:28.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 1 -- 172.21.15.39:0/636068230 shutdown_connections 2024-01-24T17:44:28.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e71d70 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x559309e594b0 0x559309e96ab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e71830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/636068230 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e62410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 1 -- 172.21.15.39:0/636068230 >> 172.21.15.39:0/636068230 conn(0x559309e2f150 msgr2=0x559309e7b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc031600640 0 client.5979 ms_handle_reset on v2:172.21.15.39:6833/1882733155 2024-01-24T17:44:28.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 1 -- 172.21.15.39:0/636068230 shutdown_connections 2024-01-24T17:44:28.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 1 -- 172.21.15.39:0/636068230 wait complete. 2024-01-24T17:44:28.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.641+0000 7fc04dbbc740 20 client.5979 trim_cache size 0 max 16384 2024-01-24T17:44:28.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.657+0000 7fc04dbbc740 1 Processor -- start 2024-01-24T17:44:28.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.658+0000 7fc04dbbc740 1 -- start start 2024-01-24T17:44:28.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.658+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e5dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.658+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e7d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.658+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e7dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.658+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559309afff20 con 0x559309e59d30 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.658+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559309d38940 con 0x559309e5f1d0 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.658+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559309e5e550 con 0x559309e32d50 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc03be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e7d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc03be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e7d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46464/0 (socket says 172.21.15.39:46464) 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc03be00640 1 -- 172.21.15.39:0/1241110787 learned_addr learned my addr 172.21.15.39:0/1241110787 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc043e00640 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e5dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc049000640 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e7dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc03be00640 1 -- 172.21.15.39:0/1241110787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 msgr2=0x559309e5dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc03be00640 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e5dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc03be00640 1 -- 172.21.15.39:0/1241110787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 msgr2=0x559309e7dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc03be00640 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e7dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.659+0000 7fc03be00640 1 -- 172.21.15.39:0/1241110787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559309dcc2f0 con 0x559309e59d30 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.660+0000 7fc03be00640 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e7d720 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc02c001450 tx=0x7fc02c001910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b8d98b12c1af39 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.660+0000 7fc039600640 1 -- 172.21.15.39:0/1241110787 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c0052d0 con 0x559309e59d30 2024-01-24T17:44:28.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.660+0000 7fc039600640 1 -- 172.21.15.39:0/1241110787 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c005450 con 0x559309e59d30 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.660+0000 7fc039600640 1 -- 172.21.15.39:0/1241110787 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c007260 con 0x559309e59d30 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.660+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1241110787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 msgr2=0x559309e7d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.660+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e7d720 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc02c001450 tx=0x7fc02c001910 comp rx=0 tx=0).stop 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.660+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1241110787 shutdown_connections 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.661+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e5f1d0 0x559309e7dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.661+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e59d30 0x559309e7d720 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.661+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1241110787 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e5dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.661+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1241110787 >> 172.21.15.39:0/1241110787 conn(0x559309e2f150 msgr2=0x559309e59440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.661+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1241110787 shutdown_connections 2024-01-24T17:44:28.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.661+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1241110787 wait complete. 2024-01-24T17:44:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.661+0000 7fc04dbbc740 1 -- start start 2024-01-24T17:44:28.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.662+0000 7fc04dbbc740 1 Processor -- start 2024-01-24T17:44:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e7d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e59d30 0x559309e8bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc04dbbc740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e5f1d0 0x559309e5a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559309ee3810 con 0x559309e5f1d0 2024-01-24T17:44:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x559309e5ace0 con 0x559309e59d30 2024-01-24T17:44:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc04dbbc740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559309ee32e0 con 0x559309e32d50 2024-01-24T17:44:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc04dbbc740 10 client.0 _ll_register_callbacks cb 0x7fc03c0798a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:44:28.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc049000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e5f1d0 0x559309e5a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc049000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e5f1d0 0x559309e5a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46476/0 (socket says 172.21.15.39:46476) 2024-01-24T17:44:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc049000640 1 -- 172.21.15.39:0/1196651602 learned_addr learned my addr 172.21.15.39:0/1196651602 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc043e00640 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e7d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc03be00640 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e59d30 0x559309e8bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc049000640 1 -- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 msgr2=0x559309e7d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc049000640 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e7d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc049000640 1 -- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e59d30 msgr2=0x559309e8bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc049000640 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e59d30 0x559309e8bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc049000640 1 -- 172.21.15.39:0/1196651602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04415b520 con 0x559309e5f1d0 2024-01-24T17:44:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.663+0000 7fc03be00640 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e59d30 0x559309e8bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc049000640 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e5f1d0 0x559309e5a140 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc03c079e40 tx=0x7fc03c001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268c9e051983edf server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc031600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:44:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc031600640 1 -- 172.21.15.39:0/1196651602 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c079c80 con 0x559309e5f1d0 2024-01-24T17:44:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc031600640 1 -- 172.21.15.39:0/1196651602 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc03c081e90 con 0x559309e5f1d0 2024-01-24T17:44:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1196651602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559309e307d0 con 0x559309e5f1d0 2024-01-24T17:44:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc031600640 1 -- 172.21.15.39:0/1196651602 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc03c0799f0 con 0x559309e5f1d0 2024-01-24T17:44:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc04dbbc740 10 client.5985 fetch_fsmap learned FSMap version 26 2024-01-24T17:44:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1196651602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x559309d38ad0 con 0x559309e5f1d0 2024-01-24T17:44:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.664+0000 7fc031600640 1 -- 172.21.15.39:0/1196651602 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559309e307d0 con 0x559309e5f1d0 2024-01-24T17:44:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc031600640 1 -- 172.21.15.39:0/1196651602 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 3126+0+0 (secure 0 0 0) 0x7fc03c081a00 con 0x559309e5f1d0 2024-01-24T17:44:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc04dbbc740 10 client.5985 fetch_fsmap finished waiting for FSMap version 26 2024-01-24T17:44:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc04dbbc740 10 client.5985 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:44:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc04dbbc740 20 client.5985 populate_metadata read hostname 'smithi039' 2024-01-24T17:44:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x559309ede6d0 0x559309edeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc04dbbc740 4 client.5985 mds_command: new command op to 5769 tid=0[{"prefix": "status"}] 2024-01-24T17:44:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1196651602 --> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fc03c077ff0 con 0x559309ede6d0 2024-01-24T17:44:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc043e00640 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x559309ede6d0 0x559309edeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc043e00640 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x559309ede6d0 0x559309edeab0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7568c9a48f7b37ba server_cookie=b431747b64e9a439 in_seq=0 out_seq=0 2024-01-24T17:44:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.665+0000 7fc031600640 10 client.5985 ms_handle_connect on v2:172.21.15.39:6833/1882733155 2024-01-24T17:44:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc031600640 1 -- 172.21.15.39:0/1196651602 <== mds.0 v2:172.21.15.39:6833/1882733155 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7fc03c077ff0 con 0x559309ede6d0 2024-01-24T17:44:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc031600640 10 client.5985 handle_command_reply: tid=0 2024-01-24T17:44:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 client.5985 shutdown 2024-01-24T17:44:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1196651602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e5f1d0 msgr2=0x559309e5a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e5f1d0 0x559309e5a140 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc03c079e40 tx=0x7fc03c001ae0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1196651602 shutdown_connections 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559309e5f1d0 0x559309e5a140 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.39:6833/1882733155,v1:172.21.15.39:6835/1882733155] conn(0x559309ede6d0 0x559309edeab0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x559309e59d30 0x559309e8bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 --2- 172.21.15.39:0/1196651602 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x559309e32d50 0x559309e7d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc031600640 0 client.5985 ms_handle_reset on v2:172.21.15.39:6833/1882733155 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1196651602 >> 172.21.15.39:0/1196651602 conn(0x559309e2f150 msgr2=0x559309e7a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1196651602 shutdown_connections 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.666+0000 7fc04dbbc740 1 -- 172.21.15.39:0/1196651602 wait complete. 2024-01-24T17:44:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc04dbbc740 20 client.5985 trim_cache size 0 max 16384 2024-01-24T17:44:28.675 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:44:28.675 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:44:28.675 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:44:28.676 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5769, 2024-01-24T17:44:28.676 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:44:28.676 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:44:28.676 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:44:28.676 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 18.699646010999999, 2024-01-24T17:44:28.676 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 26, 2024-01-24T17:44:28.677 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 50, 2024-01-24T17:44:28.677 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 50, 2024-01-24T17:44:28.677 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 24.542515044999998 2024-01-24T17:44:28.677 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:44:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 -- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc01c06a460 msgr2=0x7fc01c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc01c06a460 0x7fc01c06c920 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc034009930 tx=0x7fc034008040 comp rx=0 tx=0).stop 2024-01-24T17:44:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 -- 172.21.15.39:0/968701827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0440ccfc0 msgr2=0x7fc0440cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0440ccfc0 0x7fc0440cd440 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc03c02ca70 tx=0x7fc03c001cb0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 -- 172.21.15.39:0/968701827 shutdown_connections 2024-01-24T17:44:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc01c06a460 0x7fc01c06c920 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0440ccfc0 0x7fc0440cd440 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc044153970 0x7fc0440ac4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 --2- 172.21.15.39:0/968701827 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc024004d00 0x7fc044157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 -- 172.21.15.39:0/968701827 >> 172.21.15.39:0/968701827 conn(0x7fc04406a2a0 msgr2=0x7fc04415da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 -- 172.21.15.39:0/968701827 shutdown_connections 2024-01-24T17:44:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:28.667+0000 7fc049a00640 1 -- 172.21.15.39:0/968701827 wait complete. 2024-01-24T17:44:28.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:28.685 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'mkdir {d0,d1} && touch {d0,d1}/file') 2024-01-24T17:44:28.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:28.768 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 d0) 2024-01-24T17:44:28.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:28.839 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 d1) 2024-01-24T17:44:28.898 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-01-24T17:44:29.134 INFO:teuthology.orchestra.run.smithi039.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"11","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-01-24T17:44:29.134 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "11" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-01-24T17:44:29.135 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d0', 0), ('/d1', 1)] 2024-01-24T17:44:29.431 INFO:tasks.ceph.mds.c.smithi039.stderr:2024-01-24T17:44:29.430+0000 7f1141400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-01-24T17:44:31.135 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-01-24T17:44:31.379 INFO:teuthology.orchestra.run.smithi039.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d0","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-01-24T17:44:31.380 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/d0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-01-24T17:44:31.380 INFO:tasks.cephfs.cephfs_test_case:[('/d0', 0)] =?= [('/d0', 0), ('/d1', 1)] 2024-01-24T17:44:33.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-01-24T17:44:33.633 INFO:teuthology.orchestra.run.smithi039.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d0","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-01-24T17:44:33.634 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/d0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-01-24T17:44:33.634 INFO:tasks.cephfs.cephfs_test_case:[('/d0', 0)] =?= [('/d0', 0), ('/d1', 1)] 2024-01-24T17:44:35.635 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-01-24T17:44:35.880 INFO:teuthology.orchestra.run.smithi039.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"14","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d0","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":1},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-01-24T17:44:35.880 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/d0", "pins": { "authpin": 1, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000001", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/d1", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-01-24T17:44:35.881 INFO:tasks.cephfs.cephfs_test_case:[('/d0', 0), ('/d1', 1)] =?= [('/d0', 0), ('/d1', 1)] 2024-01-24T17:44:35.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:35.881 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch d0/f0) 2024-01-24T17:44:35.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:35.942 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch d1/f0) 2024-01-24T17:44:36.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:36.019 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest/mnt.1 && exec touch d0/f1) 2024-01-24T17:44:36.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:36.102 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest/mnt.1 && exec touch d1/f1) 2024-01-24T17:44:36.193 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-01-24T17:44:36.441 INFO:teuthology.orchestra.run.smithi039.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":75616696}},"state":"open","num_leases":1,"num_caps":4,"request_load_avg":8,"uptime":14.234713157,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":7.9287751922230418,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":9,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5335 192.168.0.1:0/75616696","completed_requests":[],"prealloc_inos":[{"start":"0x100000005e1","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000001fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4e3182a65800aed6d4aa2f6b86da3cfd72188134","ceph_version":"ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)","entity_id":"1","hostname":"smithi100","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"44571","root":"/"}},{"id":5925,"entity":{"name":{"type":"client","num":5925},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":1994510967}},"state":"open","num_leases":3,"num_caps":5,"request_load_avg":11,"uptime":18.037636524,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":20.703111462612899,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":12,"last_trim_completed_flushes_tid":2,"delegated_inos":[{"start":"0x10000000003","length":500}],"inst":"client.5925 192.168.0.1:0/1994510967","completed_requests":[{"tid":12,"created_ino":"0x100000001f8"}],"prealloc_inos":[{"start":"0x10000000003","length":500},{"start":"0x100000001f9","length":999}],"client_metadata":{"client_features":{"feature_bits":"0x00000000001fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4e3182a65800aed6d4aa2f6b86da3cfd72188134","ceph_version":"ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)","entity_id":"0","hostname":"smithi039","mount_point":"/home/ubuntu/cephtest/mnt.0","pid":"55156","root":"/"}}] 2024-01-24T17:44:36.442 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "client_features": { "feature_bits": "0x00000000001fffff" }, "entity_id": "1", "hostname": "smithi100", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "44571", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 75616696, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 192.168.0.1:0/75616696", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 9, "num_caps": 4, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x100000005e1" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 8, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 7.928775192223042 }, "state": "open", "uptime": 14.234713157 }, { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "client_features": { "feature_bits": "0x00000000001fffff" }, "entity_id": "0", "hostname": "smithi039", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "55156", "root": "/" }, "completed_requests": [ { "created_ino": "0x100000001f8", "tid": 12 } ], "delegated_inos": [ { "length": 500, "start": "0x10000000003" } ], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 1994510967, "type": "any" }, "name": { "num": 5925, "type": "client" } }, "id": 5925, "inst": "client.5925 192.168.0.1:0/1994510967", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 12, "num_caps": 5, "num_completed_flushes": 1, "num_completed_requests": 1, "num_leases": 3, "prealloc_inos": [ { "length": 500, "start": "0x10000000003" }, { "length": 999, "start": "0x100000001f9" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 11, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 20.7031114626129 }, "state": "open", "uptime": 18.037636524 } ] 2024-01-24T17:44:36.443 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-01-24T17:44:36.677 INFO:teuthology.orchestra.run.smithi100.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":75616696}},"state":"open","num_leases":1,"num_caps":2,"request_load_avg":0,"uptime":0.49298738600000003,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":3.9956125882084348,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":9,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5335 192.168.0.1:0/75616696","completed_requests":[{"tid":10,"created_ino":"0x200000003e9"}],"prealloc_inos":[{"start":"0x200000003ea","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000001fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4e3182a65800aed6d4aa2f6b86da3cfd72188134","ceph_version":"ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)","entity_id":"1","hostname":"smithi100","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"44571","root":"/"}},{"id":5925,"entity":{"name":{"type":"client","num":5925},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":1994510967}},"state":"open","num_leases":1,"num_caps":3,"request_load_avg":2,"uptime":2.344940002,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":6.9779850914222274,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":12,"last_trim_completed_flushes_tid":3,"delegated_inos":[],"inst":"client.5925 192.168.0.1:0/1994510967","completed_requests":[{"tid":16,"created_ino":"0x20000000000"}],"prealloc_inos":[{"start":"0x20000000001","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000001fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4e3182a65800aed6d4aa2f6b86da3cfd72188134","ceph_version":"ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)","entity_id":"0","hostname":"smithi039","mount_point":"/home/ubuntu/cephtest/mnt.0","pid":"55156","root":"/"}}] 2024-01-24T17:44:36.678 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "client_features": { "feature_bits": "0x00000000001fffff" }, "entity_id": "1", "hostname": "smithi100", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "44571", "root": "/" }, "completed_requests": [ { "created_ino": "0x200000003e9", "tid": 10 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 75616696, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 192.168.0.1:0/75616696", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 9, "num_caps": 2, "num_completed_flushes": 0, "num_completed_requests": 1, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x200000003ea" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 3.995612588208435 }, "state": "open", "uptime": 0.492987386 }, { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "client_features": { "feature_bits": "0x00000000001fffff" }, "entity_id": "0", "hostname": "smithi039", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "55156", "root": "/" }, "completed_requests": [ { "created_ino": "0x20000000000", "tid": 16 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 1994510967, "type": "any" }, "name": { "num": 5925, "type": "client" } }, "id": 5925, "inst": "client.5925 192.168.0.1:0/1994510967", "last_trim_completed_flushes_tid": 3, "last_trim_completed_requests_tid": 12, "num_caps": 3, "num_completed_flushes": 1, "num_completed_requests": 1, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x20000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 2, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 6.9779850914222274 }, "state": "open", "uptime": 2.344940002 } ] 2024-01-24T17:44:36.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:44:36.678 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:44:36.678 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:44:36.678 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:44:36.679 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:44:36.680 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:44:36.680 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:44:36.680 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:44:36.680 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:44:36.680 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:44:36.680 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:36.680 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:44:36.680 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:44:36.881 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.55156.asok 2024-01-24T17:44:37.061 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55156.asok 2024-01-24T17:44:37.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:37.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55156.asok mds_sessions 2024-01-24T17:44:37.310 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5925, 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5925 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:44:37.311 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1994510967 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.5925 v1:192.168.0.1:0/1994510967", 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "v1:192.168.0.1:0/1994510967", 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions": [ 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": 0, 2024-01-24T17:44:37.312 INFO:teuthology.orchestra.run.smithi039.stdout: "addrs": { 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6833", 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1882733155 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6835", 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1882733155 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:37.313 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "seq": 10, 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_gen": 0, 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_ttl": "2024-01-24T17:45:19.385755+0000", 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "last_cap_renew_request": "2024-01-24T17:44:19.385755+0000", 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_renew_seq": 2, 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "num_caps": 5, 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "open" 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": 1, 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "addrs": { 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-01-24T17:44:37.314 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.100:6835", 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1418568795 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.100:6837", 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1418568795 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: "seq": 2, 2024-01-24T17:44:37.315 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_gen": 0, 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_ttl": "2024-01-24T17:45:34.331556+0000", 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout: "last_cap_renew_request": "2024-01-24T17:44:34.331556+0000", 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_renew_seq": 1, 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout: "num_caps": 3, 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "open" 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27 2024-01-24T17:44:37.316 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:44:37.327 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session evict 5925 2024-01-24T17:44:38.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:44:38.391+0000 7fd6d6000640 -1 client.5925 I was blocklisted at osd epoch 51 2024-01-24T17:44:38.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:44:38.392+0000 7fd6d6000640 -1 client.5925 remove_session_caps still has dirty|flushing caps on 0x100000001f8.head(faked_ino=0 nref=5 ll_ref=1 cap_refs={} open={2=0} mode=100644 size=0/4194304 nlink=1 btime=2024-01-24T17:44:35.937006+0000 mtime=2024-01-24T17:44:35.939001+0000 ctime=2024-01-24T17:44:35.940475+0000 change_attr=1 caps=- dirty_caps=AxFx objectset[0x100000001f8 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["f0"] 0x7fd6c000c3e0) 2024-01-24T17:44:38.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:44:38.392+0000 7fd6d6000640 -1 client.5925 remove_session_caps still has dirty|flushing caps on 0x20000000000.head(faked_ino=0 nref=5 ll_ref=1 cap_refs={} open={2=0} mode=100644 size=0/4194304 nlink=1 btime=2024-01-24T17:44:36.006880+0000 mtime=2024-01-24T17:44:36.014999+0000 ctime=2024-01-24T17:44:36.017002+0000 change_attr=1 caps=- dirty_caps=AxFx objectset[0x20000000000 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000001.head["f0"] 0x7fd6c000f5a0) 2024-01-24T17:44:38.431 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-01-24T17:44:38.431 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:44:38.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.610+0000 7f2581800640 1 Processor -- start 2024-01-24T17:44:38.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.611+0000 7f2581800640 1 -- start start 2024-01-24T17:44:38.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.612+0000 7f2581800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f257c0caf50 0x7f257c0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:38.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.612+0000 7f2581800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0c9180 con 0x7f257c15a950 2024-01-24T17:44:38.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.612+0000 7f2581800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0c9460 con 0x7f257c0b1390 2024-01-24T17:44:38.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.612+0000 7f2581800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c0cb920 con 0x7f257c0caf50 2024-01-24T17:44:38.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.612+0000 7f257aa00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f257c15a950 0x7f257c0ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:44518/0 (socket says 172.21.15.39:44518) 2024-01-24T17:44:38.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.612+0000 7f257aa00640 1 -- 172.21.15.39:0/478173796 learned_addr learned my addr 172.21.15.39:0/478173796 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:38.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225707451 0 0) 0x7f257c0c9180 con 0x7f257c15a950 2024-01-24T17:44:38.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f255c003520 con 0x7f257c15a950 2024-01-24T17:44:38.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f2573e00640 1 --2- 172.21.15.39:0/478173796 >> v2:172.21.15.100:3300/0 conn(0x7f257c0caf50 0x7f257c0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:38.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3789788167 0 0) 0x7f255c003520 con 0x7f257c15a950 2024-01-24T17:44:38.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 >> v2:172.21.15.100:3300/0 conn(0x7f257c0caf50 msgr2=0x7f257c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:38.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 --2- 172.21.15.39:0/478173796 >> v2:172.21.15.100:3300/0 conn(0x7f257c0caf50 0x7f257c0cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 >> v1:172.21.15.100:6789/0 conn(0x7f257c0b1390 legacy=0x7f257c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0c9740 con 0x7f257c15a950 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f256c001670 con 0x7f257c15a950 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 >> v1:172.21.15.39:6789/0 conn(0x7f257c15a950 legacy=0x7f257c0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f255c003970 0x7f255c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f255c0047c0 0x7f255c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f255c005180 0x7f255c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0cb920 con 0x7f255c003970 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c00c0d0 con 0x7f255c0047c0 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.613+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255c00c290 con 0x7f255c005180 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.614+0000 7f257aa00640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f255c003970 0x7f255c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.614+0000 7f257aa00640 1 -- 172.21.15.39:0/478173796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f255c005180 msgr2=0x7f255c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:38.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.614+0000 7f257aa00640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f255c005180 0x7f255c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.614+0000 7f257aa00640 1 -- 172.21.15.39:0/478173796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f255c0047c0 msgr2=0x7f255c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:38.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.614+0000 7f257aa00640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f255c0047c0 0x7f255c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.614+0000 7f257aa00640 1 -- 172.21.15.39:0/478173796 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c003520 con 0x7f255c003970 2024-01-24T17:44:38.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.614+0000 7f257b400640 1 -- 172.21.15.39:0/478173796 reap_dead start 2024-01-24T17:44:38.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.615+0000 7f257aa00640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f255c003970 0x7f255c003d90 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c240 tx=0x7f255c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78c640765c78a22 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:38.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.615+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c005ad0 con 0x7f255c003970 2024-01-24T17:44:38.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.615+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c005c30 con 0x7f255c003970 2024-01-24T17:44:38.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.615+0000 7f257a000640 1 -- 172.21.15.39:0/478173796 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c02caf0 con 0x7f255c003970 2024-01-24T17:44:38.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.616+0000 7f2581800640 1 -- 172.21.15.39:0/478173796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f255c003970 msgr2=0x7f255c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:38.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.616+0000 7f2581800640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f255c003970 0x7f255c003d90 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c240 tx=0x7f255c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.616+0000 7f2581800640 1 -- 172.21.15.39:0/478173796 shutdown_connections 2024-01-24T17:44:38.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.616+0000 7f2581800640 1 --2- 172.21.15.39:0/478173796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f255c003970 0x7f255c003d90 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.616+0000 7f2581800640 1 -- 172.21.15.39:0/478173796 >> 172.21.15.39:0/478173796 conn(0x7f257c06a3e0 msgr2=0x7f257c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:38.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.616+0000 7f2581800640 1 -- 172.21.15.39:0/478173796 shutdown_connections 2024-01-24T17:44:38.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.616+0000 7f2581800640 1 -- 172.21.15.39:0/478173796 wait complete. 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.618+0000 7f2581800640 1 Processor -- start 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.618+0000 7f2581800640 1 -- start start 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.619+0000 7f2581800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f255c003970 0x7f257c158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.619+0000 7f2581800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f257c158f80 0x7f257c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.619+0000 7f2581800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0af6d0 0x7f257c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.619+0000 7f2581800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0aed70 con 0x7f257c0af6d0 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.619+0000 7f2581800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0adea0 con 0x7f257c158f80 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.619+0000 7f2581800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c158d50 con 0x7f255c003970 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.619+0000 7f257b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0af6d0 0x7f257c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:38.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0af6d0 0x7f257c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56120/0 (socket says 172.21.15.39:56120) 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257b400640 1 -- 172.21.15.39:0/1317110954 learned_addr learned my addr 172.21.15.39:0/1317110954 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257aa00640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f255c003970 0x7f257c158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f2573e00640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f257c158f80 0x7f257c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257b400640 1 -- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f255c003970 msgr2=0x7f257c158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257b400640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f255c003970 0x7f257c158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257b400640 1 -- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f257c158f80 msgr2=0x7f257c0af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257b400640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f257c158f80 0x7f257c0af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257b400640 1 -- 172.21.15.39:0/1317110954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0ae0f0 con 0x7f257c0af6d0 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f2573e00640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f257c158f80 0x7f257c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:38.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.620+0000 7f257b400640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0af6d0 0x7f257c0c5610 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f257c06e0a0 tx=0x7f257400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:38.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.621+0000 7f2572a00640 1 -- 172.21.15.39:0/1317110954 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574005d40 con 0x7f257c0af6d0 2024-01-24T17:44:38.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.621+0000 7f2572a00640 1 -- 172.21.15.39:0/1317110954 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257400ced0 con 0x7f257c0af6d0 2024-01-24T17:44:38.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.621+0000 7f2572a00640 1 -- 172.21.15.39:0/1317110954 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257400a7f0 con 0x7f257c0af6d0 2024-01-24T17:44:38.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.621+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c0ae3d0 con 0x7f257c0af6d0 2024-01-24T17:44:38.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.621+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c0ae990 con 0x7f257c0af6d0 2024-01-24T17:44:38.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.622+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c0aeb50 con 0x7f257c0af6d0 2024-01-24T17:44:38.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.623+0000 7f2572a00640 1 -- 172.21.15.39:0/1317110954 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f257400a950 con 0x7f257c0af6d0 2024-01-24T17:44:38.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.624+0000 7f2572a00640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f255406a430 0x7f255406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:38.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.624+0000 7f2572a00640 1 -- 172.21.15.39:0/1317110954 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f2574087a20 con 0x7f257c0af6d0 2024-01-24T17:44:38.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.624+0000 7f257aa00640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f255406a430 0x7f255406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:38.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.626+0000 7f257aa00640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f255406a430 0x7f255406c8f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f256c034000 tx=0x7f256c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.629+0000 7f2572a00640 1 -- 172.21.15.39:0/1317110954 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2574052e20 con 0x7f257c0af6d0 2024-01-24T17:44:38.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.825+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f257c073ef0 con 0x7f257c0af6d0 2024-01-24T17:44:38.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.828+0000 7f2572a00640 1 -- 172.21.15.39:0/1317110954 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16392 (secure 0 0 0) 0x7f2574017020 con 0x7f257c0af6d0 2024-01-24T17:44:38.830 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:38.830 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:38.321132+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.9","mappings":[{"from":5,"to":7}]},{"pgid":"7.1d","mappings":[{"from":5,"to":7}]},{"pgid":"7.39","mappings":[{"from":5,"to":0}]},{"pgid":"7.3a","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1994510967":"2024-01-24T18:44:37.552044+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:44:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.830+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f255406a430 msgr2=0x7f255406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f255406a430 0x7f255406c8f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f256c034000 tx=0x7f256c009030 comp rx=0 tx=0).stop 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0af6d0 msgr2=0x7f257c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0af6d0 0x7f257c0c5610 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f257c06e0a0 tx=0x7f257400cb90 comp rx=0 tx=0).stop 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 shutdown_connections 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f255406a430 0x7f255406c8f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f257c0af6d0 0x7f257c0c5610 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f257c158f80 0x7f257c0af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 --2- 172.21.15.39:0/1317110954 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f255c003970 0x7f257c158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 >> 172.21.15.39:0/1317110954 conn(0x7f257c06a3e0 msgr2=0x7f257c0cc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 shutdown_connections 2024-01-24T17:44:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:38.831+0000 7f2581800640 1 -- 172.21.15.39:0/1317110954 wait complete. 2024-01-24T17:44:38.852 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-01-24T17:44:48.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-01-24T17:44:49.100 INFO:teuthology.orchestra.run.smithi039.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":75616696}},"state":"open","num_leases":1,"num_caps":4,"request_load_avg":7,"uptime":26.892458096999999,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":8.6821916296376092,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":9,"last_trim_completed_flushes_tid":2,"delegated_inos":[],"inst":"client.5335 192.168.0.1:0/75616696","completed_requests":[],"prealloc_inos":[{"start":"0x100000005e1","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000001fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4e3182a65800aed6d4aa2f6b86da3cfd72188134","ceph_version":"ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)","entity_id":"1","hostname":"smithi100","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"44571","root":"/"}}] 2024-01-24T17:44:49.100 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "client_features": { "feature_bits": "0x00000000001fffff" }, "entity_id": "1", "hostname": "smithi100", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "44571", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 75616696, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 192.168.0.1:0/75616696", "last_trim_completed_flushes_tid": 2, "last_trim_completed_requests_tid": 9, "num_caps": 4, "num_completed_flushes": 1, "num_completed_requests": 0, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x100000005e1" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 7, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 8.68219162963761 }, "state": "open", "uptime": 26.892458097 } ] 2024-01-24T17:44:49.101 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json session ls 2024-01-24T17:44:49.333 INFO:teuthology.orchestra.run.smithi100.stdout:[{"id":5335,"entity":{"name":{"type":"client","num":5335},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":75616696}},"state":"open","num_leases":1,"num_caps":2,"request_load_avg":0,"uptime":13.148663575,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":4.8618766237199154,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":9,"last_trim_completed_flushes_tid":3,"delegated_inos":[],"inst":"client.5335 192.168.0.1:0/75616696","completed_requests":[{"tid":10,"created_ino":"0x200000003e9"}],"prealloc_inos":[{"start":"0x200000003ea","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x00000000001fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4e3182a65800aed6d4aa2f6b86da3cfd72188134","ceph_version":"ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)","entity_id":"1","hostname":"smithi100","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"44571","root":"/"}}] 2024-01-24T17:44:49.333 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "client_features": { "feature_bits": "0x00000000001fffff" }, "entity_id": "1", "hostname": "smithi100", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "44571", "root": "/" }, "completed_requests": [ { "created_ino": "0x200000003e9", "tid": 10 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 75616696, "type": "any" }, "name": { "num": 5335, "type": "client" } }, "id": 5335, "inst": "client.5335 192.168.0.1:0/75616696", "last_trim_completed_flushes_tid": 3, "last_trim_completed_requests_tid": 9, "num_caps": 2, "num_completed_flushes": 1, "num_completed_requests": 1, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x200000003ea" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 4.861876623719915 }, "state": "open", "uptime": 13.148663575 } ] 2024-01-24T17:44:49.333 INFO:tasks.cephfs.mount:Cleaning up killed connection on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:44:49.334 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:44:49.334 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:49.397 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:44:49.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.612+0000 7fc7ffe00640 1 Processor -- start 2024-01-24T17:44:49.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.612+0000 7fc7ffe00640 1 -- start start 2024-01-24T17:44:49.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc80006c4f0 0x7fc8000715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:49.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7ffe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8000c9670 con 0x7fc80015c280 2024-01-24T17:44:49.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7ffe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8000ca210 con 0x7fc800071130 2024-01-24T17:44:49.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7ffe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000adab0 con 0x7fc80006c4f0 2024-01-24T17:44:49.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7fea00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc80015c280 0x7fc8000c9b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53566/0 (socket says 172.21.15.39:53566) 2024-01-24T17:44:49.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7fea00640 1 -- 172.21.15.39:0/3218371963 learned_addr learned my addr 172.21.15.39:0/3218371963 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:49.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170721004 0 0) 0x7fc8000c9670 con 0x7fc80015c280 2024-01-24T17:44:49.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e0003520 con 0x7fc80015c280 2024-01-24T17:44:49.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.614+0000 7fc7ff400640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc80006c4f0 0x7fc8000715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:49.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2904548131 0 0) 0x7fc7e0003520 con 0x7fc80015c280 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc80006c4f0 msgr2=0x7fc8000715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc80006c4f0 0x7fc8000715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 >> v1:172.21.15.100:6789/0 conn(0x7fc800071130 legacy=0x7fc80006bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8000c8a60 con 0x7fc80015c280 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc7f0001670 con 0x7fc80015c280 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 >> v1:172.21.15.39:6789/0 conn(0x7fc80015c280 legacy=0x7fc8000c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc80006c4f0 0x7fc7e00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7e0003fd0 0x7fc7e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7e0004d00 0x7fc7e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000adab0 con 0x7fc7e0003fd0 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e000dd50 con 0x7fc7e0004d00 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e000df10 con 0x7fc80006c4f0 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.615+0000 7fc7f5e00640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7e0003fd0 0x7fc7e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:49.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.616+0000 7fc7ff400640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7e0004d00 0x7fc7e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.616+0000 7fc7fea00640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc80006c4f0 0x7fc7e00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.616+0000 7fc7f5e00640 1 -- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc80006c4f0 msgr2=0x7fc7e00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.616+0000 7fc7f5e00640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc80006c4f0 0x7fc7e00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.616+0000 7fc7f5e00640 1 -- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7e0004d00 msgr2=0x7fc7e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.616+0000 7fc7f5e00640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7e0004d00 0x7fc7e0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.616+0000 7fc7f5e00640 1 -- 172.21.15.39:0/3218371963 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e0003520 con 0x7fc7e0003fd0 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.617+0000 7fc7f5e00640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7e0003fd0 0x7fc7e00047c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8001750 tx=0x7fc7e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8563db7fa83205b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.617+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e80052d0 con 0x7fc7e0003fd0 2024-01-24T17:44:49.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.617+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e802e050 con 0x7fc7e0003fd0 2024-01-24T17:44:49.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.617+0000 7fc7fe000640 1 -- 172.21.15.39:0/3218371963 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8003830 con 0x7fc7e0003fd0 2024-01-24T17:44:49.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.618+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3218371963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7e0003fd0 msgr2=0x7fc7e00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:49.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.618+0000 7fc7ffe00640 1 --2- 172.21.15.39:0/3218371963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7e0003fd0 0x7fc7e00047c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8001750 tx=0x7fc7e00057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.618+0000 7fc7ff400640 1 -- 172.21.15.39:0/3218371963 reap_dead start 2024-01-24T17:44:49.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.618+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3218371963 shutdown_connections 2024-01-24T17:44:49.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.618+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3218371963 >> 172.21.15.39:0/3218371963 conn(0x7fc80006a090 msgr2=0x7fc80006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:49.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.618+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3218371963 shutdown_connections 2024-01-24T17:44:49.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.618+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3218371963 wait complete. 2024-01-24T17:44:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.620+0000 7fc7ffe00640 1 Processor -- start 2024-01-24T17:44:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.620+0000 7fc7ffe00640 1 -- start start 2024-01-24T17:44:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7e0003fd0 0x7fc8000b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc800159970 0x7fc8000b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc800155660 0x7fc8000b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ffe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000b01b0 con 0x7fc800155660 2024-01-24T17:44:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ffe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc800156090 con 0x7fc800159970 2024-01-24T17:44:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ffe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc800159df0 con 0x7fc7e0003fd0 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc800155660 0x7fc8000b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc800155660 0x7fc8000b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46518/0 (socket says 172.21.15.39:46518) 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.621+0000 7fc7ff400640 1 -- 172.21.15.39:0/3676484979 learned_addr learned my addr 172.21.15.39:0/3676484979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7fea00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7e0003fd0 0x7fc8000b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7f5e00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc800159970 0x7fc8000b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7ff400640 1 -- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7e0003fd0 msgr2=0x7fc8000b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7ff400640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7e0003fd0 0x7fc8000b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7ff400640 1 -- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc800159970 msgr2=0x7fc8000b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7ff400640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc800159970 0x7fc8000b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7ff400640 1 -- 172.21.15.39:0/3676484979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8000b14e0 con 0x7fc800155660 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7ff400640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc800155660 0x7fc8000b1840 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc8000c1e90 tx=0x7fc7f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7f7400640 1 -- 172.21.15.39:0/3676484979 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8007de0 con 0x7fc800155660 2024-01-24T17:44:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7f7400640 1 -- 172.21.15.39:0/3676484979 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f800f050 con 0x7fc800155660 2024-01-24T17:44:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7f7400640 1 -- 172.21.15.39:0/3676484979 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8014650 con 0x7fc800155660 2024-01-24T17:44:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.622+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8000af910 con 0x7fc800155660 2024-01-24T17:44:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.623+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8000afed0 con 0x7fc800155660 2024-01-24T17:44:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.624+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8000cf060 con 0x7fc800155660 2024-01-24T17:44:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.625+0000 7fc7f7400640 1 -- 172.21.15.39:0/3676484979 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc7f80147b0 con 0x7fc800155660 2024-01-24T17:44:49.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.626+0000 7fc7f7400640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc7d806a430 0x7fc7d806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:49.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.626+0000 7fc7f7400640 1 -- 172.21.15.39:0/3676484979 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fc7f8089020 con 0x7fc800155660 2024-01-24T17:44:49.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.627+0000 7fc7fea00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc7d806a430 0x7fc7d806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:49.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.628+0000 7fc7fea00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc7d806a430 0x7fc7d806c8f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0001c90 tx=0x7fc7f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:49.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.631+0000 7fc7f7400640 1 -- 172.21.15.39:0/3676484979 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc7f8054550 con 0x7fc800155660 2024-01-24T17:44:49.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.827+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8000cbcb0 con 0x7fc800155660 2024-01-24T17:44:49.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.829+0000 7fc7f7400640 1 -- 172.21.15.39:0/3676484979 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16392 (secure 0 0 0) 0x7fc7f8057b70 con 0x7fc800155660 2024-01-24T17:44:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:49.831 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:38.321132+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.9","mappings":[{"from":5,"to":7}]},{"pgid":"7.1d","mappings":[{"from":5,"to":7}]},{"pgid":"7.39","mappings":[{"from":5,"to":0}]},{"pgid":"7.3a","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1994510967":"2024-01-24T18:44:37.552044+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc7d806a430 msgr2=0x7fc7d806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc7d806a430 0x7fc7d806c8f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0001c90 tx=0x7fc7f0009040 comp rx=0 tx=0).stop 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc800155660 msgr2=0x7fc8000b1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc800155660 0x7fc8000b1840 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc8000c1e90 tx=0x7fc7f8009da0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 shutdown_connections 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc7d806a430 0x7fc7d806c8f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc800155660 0x7fc8000b1840 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc800159970 0x7fc8000b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 --2- 172.21.15.39:0/3676484979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7e0003fd0 0x7fc8000b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 >> 172.21.15.39:0/3676484979 conn(0x7fc80006a090 msgr2=0x7fc800157960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 shutdown_connections 2024-01-24T17:44:49.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:49.832+0000 7fc7ffe00640 1 -- 172.21.15.39:0/3676484979 wait complete. 2024-01-24T17:44:49.853 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:44:07.660016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:44:08.985819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:44:49.853 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:44:50.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.085+0000 7ff419600640 1 Processor -- start 2024-01-24T17:44:50.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.085+0000 7ff419600640 1 -- start start 2024-01-24T17:44:50.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.086+0000 7ff419600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4140b0730 0x7ff41415da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.086+0000 7ff419600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff41415dfc0 0x7ff4140b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.086+0000 7ff419600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41415e450 con 0x7ff4140b0350 2024-01-24T17:44:50.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.086+0000 7ff419600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41406d4c0 con 0x7ff4140b0730 2024-01-24T17:44:50.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.086+0000 7ff419600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41406cf00 con 0x7ff41415dfc0 2024-01-24T17:44:50.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.087+0000 7ff412000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4140b0730 0x7ff41415da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.087+0000 7ff412000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4140b0730 0x7ff41415da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46534/0 (socket says 172.21.15.39:46534) 2024-01-24T17:44:50.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.087+0000 7ff412000640 1 -- 172.21.15.39:0/361296376 learned_addr learned my addr 172.21.15.39:0/361296376 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:50.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.087+0000 7ff413400640 1 --2- 172.21.15.39:0/361296376 >> v2:172.21.15.100:3300/0 conn(0x7ff41415dfc0 0x7ff4140b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.087+0000 7ff412000640 1 -- 172.21.15.39:0/361296376 >> v2:172.21.15.100:3300/0 conn(0x7ff41415dfc0 msgr2=0x7ff4140b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:50.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.087+0000 7ff412000640 1 --2- 172.21.15.39:0/361296376 >> v2:172.21.15.100:3300/0 conn(0x7ff41415dfc0 0x7ff4140b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.087+0000 7ff412000640 1 -- 172.21.15.39:0/361296376 >> v1:172.21.15.100:6789/0 conn(0x7ff4140b0350 legacy=0x7ff41406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.087+0000 7ff412000640 1 -- 172.21.15.39:0/361296376 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41406d1e0 con 0x7ff4140b0730 2024-01-24T17:44:50.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.089+0000 7ff412000640 1 --2- 172.21.15.39:0/361296376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4140b0730 0x7ff41415da80 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff4140ada80 tx=0x7ff3fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb56f673a774a37 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:50.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.089+0000 7ff411600640 1 -- 172.21.15.39:0/361296376 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc031870 con 0x7ff4140b0730 2024-01-24T17:44:50.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.089+0000 7ff411600640 1 -- 172.21.15.39:0/361296376 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc0319d0 con 0x7ff4140b0730 2024-01-24T17:44:50.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.089+0000 7ff411600640 1 -- 172.21.15.39:0/361296376 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc031d60 con 0x7ff4140b0730 2024-01-24T17:44:50.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.089+0000 7ff419600640 1 -- 172.21.15.39:0/361296376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4140b0730 msgr2=0x7ff41415da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.089+0000 7ff419600640 1 --2- 172.21.15.39:0/361296376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4140b0730 0x7ff41415da80 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff4140ada80 tx=0x7ff3fc02b910 comp rx=0 tx=0).stop 2024-01-24T17:44:50.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.090+0000 7ff419600640 1 -- 172.21.15.39:0/361296376 shutdown_connections 2024-01-24T17:44:50.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.090+0000 7ff419600640 1 --2- 172.21.15.39:0/361296376 >> v2:172.21.15.100:3300/0 conn(0x7ff41415dfc0 0x7ff4140b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.090+0000 7ff419600640 1 --2- 172.21.15.39:0/361296376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4140b0730 0x7ff41415da80 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.090+0000 7ff419600640 1 -- 172.21.15.39:0/361296376 >> 172.21.15.39:0/361296376 conn(0x7ff4140cb0e0 msgr2=0x7ff4140cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:50.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.090+0000 7ff419600640 1 -- 172.21.15.39:0/361296376 shutdown_connections 2024-01-24T17:44:50.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.090+0000 7ff419600640 1 -- 172.21.15.39:0/361296376 wait complete. 2024-01-24T17:44:50.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.091+0000 7ff419600640 1 Processor -- start 2024-01-24T17:44:50.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.092+0000 7ff419600640 1 -- start start 2024-01-24T17:44:50.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff419600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4140b0350 0x7ff4141e4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff419600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4140b0730 0x7ff4141e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff419600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41415dfc0 0x7ff4140cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff419600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4140c52e0 con 0x7ff41415dfc0 2024-01-24T17:44:50.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff419600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4141e3300 con 0x7ff4140b0350 2024-01-24T17:44:50.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff419600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4141e3e80 con 0x7ff4140b0730 2024-01-24T17:44:50.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff413400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41415dfc0 0x7ff4140cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff413400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41415dfc0 0x7ff4140cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46542/0 (socket says 172.21.15.39:46542) 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff413400640 1 -- 172.21.15.39:0/2282804936 learned_addr learned my addr 172.21.15.39:0/2282804936 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff412a00640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4140b0350 0x7ff4141e4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff412000640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4140b0730 0x7ff4141e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff413400640 1 -- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4140b0730 msgr2=0x7ff4141e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff413400640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4140b0730 0x7ff4141e4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff413400640 1 -- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4140b0350 msgr2=0x7ff4141e4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff413400640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4140b0350 0x7ff4141e4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff413400640 1 -- 172.21.15.39:0/2282804936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc0094d0 con 0x7ff41415dfc0 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.093+0000 7ff412000640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4140b0730 0x7ff4141e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:50.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.094+0000 7ff413400640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41415dfc0 0x7ff4140cd5e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff40c007e40 tx=0x7ff40c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.094+0000 7ff40aa00640 1 -- 172.21.15.39:0/2282804936 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c014620 con 0x7ff41415dfc0 2024-01-24T17:44:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.094+0000 7ff40aa00640 1 -- 172.21.15.39:0/2282804936 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c0147a0 con 0x7ff41415dfc0 2024-01-24T17:44:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.094+0000 7ff40aa00640 1 -- 172.21.15.39:0/2282804936 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c014b50 con 0x7ff41415dfc0 2024-01-24T17:44:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.094+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4141e3550 con 0x7ff41415dfc0 2024-01-24T17:44:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.094+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4141e3b10 con 0x7ff41415dfc0 2024-01-24T17:44:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.095+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff414073ef0 con 0x7ff41415dfc0 2024-01-24T17:44:50.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.096+0000 7ff40aa00640 1 -- 172.21.15.39:0/2282804936 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff40c014cb0 con 0x7ff41415dfc0 2024-01-24T17:44:50.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.098+0000 7ff40aa00640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3ec06a430 0x7ff3ec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.098+0000 7ff40aa00640 1 -- 172.21.15.39:0/2282804936 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7ff40c089310 con 0x7ff41415dfc0 2024-01-24T17:44:50.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.098+0000 7ff412a00640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3ec06a430 0x7ff3ec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.099+0000 7ff412a00640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3ec06a430 0x7ff3ec06c8f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff404000f60 tx=0x7ff404009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:50.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.102+0000 7ff40aa00640 1 -- 172.21.15.39:0/2282804936 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff40c054830 con 0x7ff41415dfc0 2024-01-24T17:44:50.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.298+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff4140b15b0 con 0x7ff41415dfc0 2024-01-24T17:44:50.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.299+0000 7ff40aa00640 1 -- 172.21.15.39:0/2282804936 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff40c057e50 con 0x7ff41415dfc0 2024-01-24T17:44:50.300 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.301+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3ec06a430 msgr2=0x7ff3ec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.301+0000 7ff419600640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3ec06a430 0x7ff3ec06c8f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff404000f60 tx=0x7ff404009040 comp rx=0 tx=0).stop 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.301+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41415dfc0 msgr2=0x7ff4140cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.301+0000 7ff419600640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41415dfc0 0x7ff4140cd5e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff40c007e40 tx=0x7ff40c00e4b0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.302+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 shutdown_connections 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.302+0000 7ff419600640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3ec06a430 0x7ff3ec06c8f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.302+0000 7ff419600640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41415dfc0 0x7ff4140cd5e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.302+0000 7ff419600640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4140b0730 0x7ff4141e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.302+0000 7ff419600640 1 --2- 172.21.15.39:0/2282804936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4140b0350 0x7ff4141e4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.302+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 >> 172.21.15.39:0/2282804936 conn(0x7ff4140cb0e0 msgr2=0x7ff41406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.302+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 shutdown_connections 2024-01-24T17:44:50.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.302+0000 7ff419600640 1 -- 172.21.15.39:0/2282804936 wait complete. 2024-01-24T17:44:50.323 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:44:50.500 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:44:50.508 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:44:50.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.542+0000 7f03d5400640 1 Processor -- start 2024-01-24T17:44:50.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.542+0000 7f03d5400640 1 -- start start 2024-01-24T17:44:50.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03d5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d015cb00 0x7f03d00c9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03d5400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f03d0160f70 0x7f03d00adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03d5400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f03d00ae030 0x7f03d00cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03d5400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00ae4c0 con 0x7f03d0160f70 2024-01-24T17:44:50.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03d5400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d015cee0 con 0x7f03d00ae030 2024-01-24T17:44:50.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03d5400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00c9b50 con 0x7f03d015cb00 2024-01-24T17:44:50.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03c7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f03d0160f70 0x7f03d00adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03c7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f03d0160f70 0x7f03d00adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46558/0 (socket says 172.21.15.39:46558) 2024-01-24T17:44:50.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03c7e00640 1 -- 172.21.15.39:0/1334801038 learned_addr learned my addr 172.21.15.39:0/1334801038 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:50.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03cf200640 1 --2- 172.21.15.39:0/1334801038 >> v2:172.21.15.100:3300/0 conn(0x7f03d00ae030 0x7f03d00cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.544+0000 7f03ce800640 1 --2- 172.21.15.39:0/1334801038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d015cb00 0x7f03d00c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.545+0000 7f03c7e00640 1 -- 172.21.15.39:0/1334801038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d015cb00 msgr2=0x7f03d00c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:50.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.545+0000 7f03c7e00640 1 --2- 172.21.15.39:0/1334801038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d015cb00 0x7f03d00c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.545+0000 7f03c7e00640 1 -- 172.21.15.39:0/1334801038 >> v2:172.21.15.100:3300/0 conn(0x7f03d00ae030 msgr2=0x7f03d00cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:50.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.545+0000 7f03c7e00640 1 --2- 172.21.15.39:0/1334801038 >> v2:172.21.15.100:3300/0 conn(0x7f03d00ae030 0x7f03d00cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.545+0000 7f03c7e00640 1 -- 172.21.15.39:0/1334801038 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00c9e30 con 0x7f03d0160f70 2024-01-24T17:44:50.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.545+0000 7f03cf200640 1 --2- 172.21.15.39:0/1334801038 >> v2:172.21.15.100:3300/0 conn(0x7f03d00ae030 0x7f03d00cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:50.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.545+0000 7f03ce800640 1 --2- 172.21.15.39:0/1334801038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d015cb00 0x7f03d00c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.546+0000 7f03c7e00640 1 --2- 172.21.15.39:0/1334801038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d0160f70 0x7f03d00adaf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f03d00ae6b0 tx=0x7f03b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3702eca2661993a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.546+0000 7f03cde00640 1 -- 172.21.15.39:0/1334801038 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8031640 con 0x7f03d0160f70 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.546+0000 7f03cde00640 1 -- 172.21.15.39:0/1334801038 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b80317a0 con 0x7f03d0160f70 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.546+0000 7f03cde00640 1 -- 172.21.15.39:0/1334801038 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8031b30 con 0x7f03d0160f70 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 -- 172.21.15.39:0/1334801038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d0160f70 msgr2=0x7f03d00adaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 --2- 172.21.15.39:0/1334801038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d0160f70 0x7f03d00adaf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f03d00ae6b0 tx=0x7f03b802b8f0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 -- 172.21.15.39:0/1334801038 shutdown_connections 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 --2- 172.21.15.39:0/1334801038 >> v2:172.21.15.100:3300/0 conn(0x7f03d00ae030 0x7f03d00cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 --2- 172.21.15.39:0/1334801038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d0160f70 0x7f03d00adaf0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 --2- 172.21.15.39:0/1334801038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d015cb00 0x7f03d00c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 -- 172.21.15.39:0/1334801038 >> 172.21.15.39:0/1334801038 conn(0x7f03d00b0910 msgr2=0x7f03d00b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:50.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 -- 172.21.15.39:0/1334801038 shutdown_connections 2024-01-24T17:44:50.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.547+0000 7f03d5400640 1 -- 172.21.15.39:0/1334801038 wait complete. 2024-01-24T17:44:50.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.549+0000 7f03d5400640 1 Processor -- start 2024-01-24T17:44:50.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.549+0000 7f03d5400640 1 -- start start 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.550+0000 7f03d5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d00ae030 0x7f03d015e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.550+0000 7f03d5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d015cb00 0x7f03d01f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.550+0000 7f03d5400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d0160f70 0x7f03d00c4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.550+0000 7f03d5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0152e60 con 0x7f03d015cb00 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.550+0000 7f03d5400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01f1da0 con 0x7f03d0160f70 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.550+0000 7f03d5400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d015d100 con 0x7f03d00ae030 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d015cb00 0x7f03d01f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d015cb00 0x7f03d01f1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46572/0 (socket says 172.21.15.39:46572) 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03c7e00640 1 -- 172.21.15.39:0/3926032975 learned_addr learned my addr 172.21.15.39:0/3926032975 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03ce800640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d00ae030 0x7f03d015e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03cf200640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d0160f70 0x7f03d00c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03c7e00640 1 -- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d00ae030 msgr2=0x7f03d015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03c7e00640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d00ae030 0x7f03d015e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03c7e00640 1 -- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d0160f70 msgr2=0x7f03d00c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03c7e00640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d0160f70 0x7f03d00c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03c7e00640 1 -- 172.21.15.39:0/3926032975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b80094d0 con 0x7f03d015cb00 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.551+0000 7f03cf200640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d0160f70 0x7f03d00c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.552+0000 7f03c7e00640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d015cb00 0x7f03d01f1860 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f03b8009870 tx=0x7f03b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.552+0000 7f03c6a00640 1 -- 172.21.15.39:0/3926032975 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b803ac60 con 0x7f03d015cb00 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.552+0000 7f03c6a00640 1 -- 172.21.15.39:0/3926032975 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b803adc0 con 0x7f03d015cb00 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.552+0000 7f03c6a00640 1 -- 172.21.15.39:0/3926032975 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8044630 con 0x7f03d015cb00 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.552+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d015d3e0 con 0x7f03d015cb00 2024-01-24T17:44:50.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.552+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d015d9a0 con 0x7f03d015cb00 2024-01-24T17:44:50.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.553+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d0073ef0 con 0x7f03d015cb00 2024-01-24T17:44:50.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.554+0000 7f03c6a00640 1 -- 172.21.15.39:0/3926032975 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f03b802f050 con 0x7f03d015cb00 2024-01-24T17:44:50.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.555+0000 7f03c6a00640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f03a806a460 0x7f03a806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.555+0000 7f03c6a00640 1 -- 172.21.15.39:0/3926032975 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f03b80aa750 con 0x7f03d015cb00 2024-01-24T17:44:50.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.556+0000 7f03ce800640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f03a806a460 0x7f03a806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:50.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.557+0000 7f03ce800640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f03a806a460 0x7f03a806c920 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f03c0006e80 tx=0x7f03c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:50.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.560+0000 7f03c6a00640 1 -- 172.21.15.39:0/3926032975 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f03b8075d10 con 0x7f03d015cb00 2024-01-24T17:44:50.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.752+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f03d00b4da0 con 0x7f03d015cb00 2024-01-24T17:44:50.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.753+0000 7f03c6a00640 1 -- 172.21.15.39:0/3926032975 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7f03b8031e60 con 0x7f03d015cb00 2024-01-24T17:44:50.755 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.755+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f03a806a460 msgr2=0x7f03a806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.755+0000 7f03d5400640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f03a806a460 0x7f03a806c920 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f03c0006e80 tx=0x7f03c0008040 comp rx=0 tx=0).stop 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.755+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d015cb00 msgr2=0x7f03d01f1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.755+0000 7f03d5400640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d015cb00 0x7f03d01f1860 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f03b8009870 tx=0x7f03b80046c0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.756+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 shutdown_connections 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.756+0000 7f03d5400640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03d0160f70 0x7f03d00c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.756+0000 7f03d5400640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f03a806a460 0x7f03a806c920 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.756+0000 7f03d5400640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03d015cb00 0x7f03d01f1860 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.756+0000 7f03d5400640 1 --2- 172.21.15.39:0/3926032975 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03d00ae030 0x7f03d015e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.756+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 >> 172.21.15.39:0/3926032975 conn(0x7f03d00b0910 msgr2=0x7f03d01571a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.756+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 shutdown_connections 2024-01-24T17:44:50.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.756+0000 7f03d5400640 1 -- 172.21.15.39:0/3926032975 wait complete. 2024-01-24T17:44:50.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:44:50.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.994+0000 7fda0ac00640 1 Processor -- start 2024-01-24T17:44:50.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.994+0000 7fda0ac00640 1 -- start start 2024-01-24T17:44:50.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.995+0000 7fda0ac00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fda040b3660 0x7fda040b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:50.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.995+0000 7fda0ac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040cda90 con 0x7fda0415fff0 2024-01-24T17:44:50.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.995+0000 7fda0ac00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda040d32c0 con 0x7fda040d14c0 2024-01-24T17:44:50.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.995+0000 7fda0ac00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda040b30a0 con 0x7fda040b3660 2024-01-24T17:44:50.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.996+0000 7fda03e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fda0415fff0 0x7fda040d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53576/0 (socket says 172.21.15.39:53576) 2024-01-24T17:44:50.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.996+0000 7fda03e00640 1 -- 172.21.15.39:0/1376133502 learned_addr learned my addr 172.21.15.39:0/1376133502 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:50.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.996+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766287680 0 0) 0x7fda040cda90 con 0x7fda0415fff0 2024-01-24T17:44:51.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.996+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e4003520 con 0x7fda0415fff0 2024-01-24T17:44:51.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.996+0000 7fda08c00640 1 --2- 172.21.15.39:0/1376133502 >> v2:172.21.15.100:3300/0 conn(0x7fda040b3660 0x7fda040b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632537671 0 0) 0x7fd9e4003520 con 0x7fda0415fff0 2024-01-24T17:44:51.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 >> v2:172.21.15.100:3300/0 conn(0x7fda040b3660 msgr2=0x7fda040b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:51.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 --2- 172.21.15.39:0/1376133502 >> v2:172.21.15.100:3300/0 conn(0x7fda040b3660 0x7fda040b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 >> v1:172.21.15.100:6789/0 conn(0x7fda040d14c0 legacy=0x7fda040cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda040b3380 con 0x7fda0415fff0 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fd9f4001670 con 0x7fda0415fff0 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 >> v1:172.21.15.39:6789/0 conn(0x7fda0415fff0 legacy=0x7fda040d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e40039c0 0x7fd9e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40047c0 0x7fd9e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e40051d0 0x7fd9e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda040b30a0 con 0x7fd9e40051d0 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e400c120 con 0x7fd9e40047c0 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e400c2e0 con 0x7fd9e40039c0 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.997+0000 7fda08c00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e40051d0 0x7fd9e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda03400640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40047c0 0x7fd9e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda03e00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e40039c0 0x7fd9e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda08c00640 1 -- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e40039c0 msgr2=0x7fd9e4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:51.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda08c00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e40039c0 0x7fd9e4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda08c00640 1 -- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40047c0 msgr2=0x7fd9e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:51.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda08c00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40047c0 0x7fd9e4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda08c00640 1 -- 172.21.15.39:0/1376133502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4003520 con 0x7fd9e40051d0 2024-01-24T17:44:51.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda03400640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40047c0 0x7fd9e4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:51.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.998+0000 7fda03e00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9e40039c0 0x7fd9e4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:51.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.999+0000 7fda08c00640 1 -- 172.21.15.39:0/1376133502 reap_dead start 2024-01-24T17:44:51.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.999+0000 7fda08c00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e40051d0 0x7fd9e400b980 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc007360 tx=0x7fd9e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0f0a60daa9517b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:51.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.999+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc02ccf0 con 0x7fd9e40051d0 2024-01-24T17:44:51.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.999+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc02ce90 con 0x7fd9e40051d0 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:50.999+0000 7fda02a00640 1 -- 172.21.15.39:0/1376133502 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc003bd0 con 0x7fd9e40051d0 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.000+0000 7fda0ac00640 1 -- 172.21.15.39:0/1376133502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e40051d0 msgr2=0x7fd9e400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.000+0000 7fda0ac00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e40051d0 0x7fd9e400b980 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc007360 tx=0x7fd9e40057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.000+0000 7fda0ac00640 1 -- 172.21.15.39:0/1376133502 shutdown_connections 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.000+0000 7fda0ac00640 1 --2- 172.21.15.39:0/1376133502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9e40051d0 0x7fd9e400b980 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.000+0000 7fda0ac00640 1 -- 172.21.15.39:0/1376133502 >> 172.21.15.39:0/1376133502 conn(0x7fda0409bcc0 msgr2=0x7fda040d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.000+0000 7fda0ac00640 1 -- 172.21.15.39:0/1376133502 shutdown_connections 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.000+0000 7fda0ac00640 1 -- 172.21.15.39:0/1376133502 wait complete. 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.002+0000 7fda0ac00640 1 Processor -- start 2024-01-24T17:44:51.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.002+0000 7fda0ac00640 1 -- start start 2024-01-24T17:44:51.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda0ac00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40051d0 0x7fda0415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda0ac00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda040cb130 0x7fda040a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda0ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda0409d920 0x7fda0409dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda0ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0409c630 con 0x7fda0409d920 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda0ac00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0409c880 con 0x7fd9e40051d0 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda0ac00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0409cb60 con 0x7fda040cb130 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda08c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda0409d920 0x7fda0409dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda08c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda0409d920 0x7fda0409dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46590/0 (socket says 172.21.15.39:46590) 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda08c00640 1 -- 172.21.15.39:0/1253303449 learned_addr learned my addr 172.21.15.39:0/1253303449 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.003+0000 7fda03e00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40051d0 0x7fda0415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda03400640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda040cb130 0x7fda040a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda08c00640 1 -- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda040cb130 msgr2=0x7fda040a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda08c00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda040cb130 0x7fda040a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda08c00640 1 -- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40051d0 msgr2=0x7fda0415c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda08c00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40051d0 0x7fda0415c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda08c00640 1 -- 172.21.15.39:0/1253303449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0409ce40 con 0x7fda0409d920 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda03e00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40051d0 0x7fda0415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:51.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda08c00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda0409d920 0x7fda0409dda0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc02c760 tx=0x7fd9fc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:51.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fd9fbe00640 1 -- 172.21.15.39:0/1253303449 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc004450 con 0x7fda0409d920 2024-01-24T17:44:51.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fd9fbe00640 1 -- 172.21.15.39:0/1253303449 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc0045b0 con 0x7fda0409d920 2024-01-24T17:44:51.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0409d120 con 0x7fda0409d920 2024-01-24T17:44:51.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fd9fbe00640 1 -- 172.21.15.39:0/1253303449 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc03f470 con 0x7fda0409d920 2024-01-24T17:44:51.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.004+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda041f2560 con 0x7fda0409d920 2024-01-24T17:44:51.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.006+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda04073ef0 con 0x7fda0409d920 2024-01-24T17:44:51.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.007+0000 7fd9fbe00640 1 -- 172.21.15.39:0/1253303449 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd9fc034440 con 0x7fda0409d920 2024-01-24T17:44:51.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.007+0000 7fd9fbe00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd9dc06a430 0x7fd9dc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.008+0000 7fd9fbe00640 1 -- 172.21.15.39:0/1253303449 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fd9fc03b070 con 0x7fda0409d920 2024-01-24T17:44:51.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.008+0000 7fda03e00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd9dc06a430 0x7fd9dc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.009+0000 7fda03e00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd9dc06a430 0x7fd9dc06c8f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007950 tx=0x7fd9f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:51.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.012+0000 7fd9fbe00640 1 -- 172.21.15.39:0/1253303449 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd9fc075750 con 0x7fda0409d920 2024-01-24T17:44:51.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.208+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fda0406c980 con 0x7fda0409d920 2024-01-24T17:44:51.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.208+0000 7fd9fbe00640 1 -- 172.21.15.39:0/1253303449 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd9fc078d70 con 0x7fda0409d920 2024-01-24T17:44:51.209 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:44:51.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd9dc06a430 msgr2=0x7fd9dc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd9dc06a430 0x7fd9dc06c8f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007950 tx=0x7fd9f4009040 comp rx=0 tx=0).stop 2024-01-24T17:44:51.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda0409d920 msgr2=0x7fda0409dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda0409d920 0x7fda0409dda0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc02c760 tx=0x7fd9fc004ab0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 shutdown_connections 2024-01-24T17:44:51.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd9dc06a430 0x7fd9dc06c8f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda0409d920 0x7fda0409dda0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda040cb130 0x7fda040a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 --2- 172.21.15.39:0/1253303449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9e40051d0 0x7fda0415c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 >> 172.21.15.39:0/1253303449 conn(0x7fda0409bcc0 msgr2=0x7fda0415df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:51.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 shutdown_connections 2024-01-24T17:44:51.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.211+0000 7fda0ac00640 1 -- 172.21.15.39:0/1253303449 wait complete. 2024-01-24T17:44:51.232 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:44:51.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.450+0000 7fa3f7200640 1 Processor -- start 2024-01-24T17:44:51.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.450+0000 7fa3f7200640 1 -- start start 2024-01-24T17:44:51.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.451+0000 7fa3f7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f015ca30 0x7fa3f00cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.451+0000 7fa3f7200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f00c9c10 con 0x7fa3f00cd3c0 2024-01-24T17:44:51.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.451+0000 7fa3f7200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f00c9650 con 0x7fa3f00caf00 2024-01-24T17:44:51.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.451+0000 7fa3f7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00c9930 con 0x7fa3f015ca30 2024-01-24T17:44:51.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.452+0000 7fa3ef400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa3f00cd3c0 0x7fa3f00c8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53588/0 (socket says 172.21.15.39:53588) 2024-01-24T17:44:51.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.452+0000 7fa3ef400640 1 -- 172.21.15.39:0/2398061734 learned_addr learned my addr 172.21.15.39:0/2398061734 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:51.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.452+0000 7fa3efe00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f015ca30 0x7fa3f00cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.452+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109685367 0 0) 0x7fa3f00c9c10 con 0x7fa3f00cd3c0 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.452+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d0003520 con 0x7fa3f00cd3c0 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3985739022 0 0) 0x7fa3d0003520 con 0x7fa3f00cd3c0 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f015ca30 msgr2=0x7fa3f00cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f015ca30 0x7fa3f00cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 >> v1:172.21.15.100:6789/0 conn(0x7fa3f00caf00 legacy=0x7fa3f0070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00cc680 con 0x7fa3f00cd3c0 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fa3d8001670 con 0x7fa3f00cd3c0 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 >> v1:172.21.15.39:6789/0 conn(0x7fa3f00cd3c0 legacy=0x7fa3f00c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3d00039a0 0x7fa3d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00047c0 0x7fa3d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f015ca30 0x7fa3d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00c9930 con 0x7fa3d00039a0 2024-01-24T17:44:51.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d000ba60 con 0x7fa3d00047c0 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d000bc20 con 0x7fa3f015ca30 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.453+0000 7fa3efe00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3d00039a0 0x7fa3d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.454+0000 7fa3ef400640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00047c0 0x7fa3d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.454+0000 7fa3efe00640 1 -- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f015ca30 msgr2=0x7fa3d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.454+0000 7fa3efe00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f015ca30 0x7fa3d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.454+0000 7fa3efe00640 1 -- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00047c0 msgr2=0x7fa3d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.454+0000 7fa3efe00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00047c0 0x7fa3d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.454+0000 7fa3efe00640 1 -- 172.21.15.39:0/2398061734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d0003520 con 0x7fa3d00039a0 2024-01-24T17:44:51.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.454+0000 7fa3ef400640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00047c0 0x7fa3d0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.455+0000 7fa3efe00640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3d00039a0 0x7fa3d0003dc0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009a90 tx=0x7fa3d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d775bd9f8a6164 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.455+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e002c6b0 con 0x7fa3d00039a0 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.455+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e002c810 con 0x7fa3d00039a0 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.455+0000 7fa3eea00640 1 -- 172.21.15.39:0/2398061734 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e002cba0 con 0x7fa3d00039a0 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.456+0000 7fa3f7200640 1 -- 172.21.15.39:0/2398061734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3d00039a0 msgr2=0x7fa3d0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.456+0000 7fa3f7200640 1 --2- 172.21.15.39:0/2398061734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3d00039a0 0x7fa3d0003dc0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009a90 tx=0x7fa3d00057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.456+0000 7fa3f5200640 1 -- 172.21.15.39:0/2398061734 reap_dead start 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.456+0000 7fa3f7200640 1 -- 172.21.15.39:0/2398061734 shutdown_connections 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.456+0000 7fa3f7200640 1 -- 172.21.15.39:0/2398061734 >> 172.21.15.39:0/2398061734 conn(0x7fa3f006a3e0 msgr2=0x7fa3f006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.456+0000 7fa3f7200640 1 -- 172.21.15.39:0/2398061734 shutdown_connections 2024-01-24T17:44:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.456+0000 7fa3f7200640 1 -- 172.21.15.39:0/2398061734 wait complete. 2024-01-24T17:44:51.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.458+0000 7fa3f7200640 1 Processor -- start 2024-01-24T17:44:51.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.458+0000 7fa3f7200640 1 -- start start 2024-01-24T17:44:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.459+0000 7fa3f7200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00039a0 0x7fa3f015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.459+0000 7fa3f7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f00b3ba0 0x7fa3f015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.459+0000 7fa3f7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f0158a00 0x7fa3f015db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.459+0000 7fa3f7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f01552f0 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.459+0000 7fa3f7200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00c5610 con 0x7fa3d00039a0 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.459+0000 7fa3f7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00c7d00 con 0x7fa3f0158a00 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3ef400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f00b3ba0 0x7fa3f015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3ef400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f00b3ba0 0x7fa3f015d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46598/0 (socket says 172.21.15.39:46598) 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3ef400640 1 -- 172.21.15.39:0/1302656819 learned_addr learned my addr 172.21.15.39:0/1302656819 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3efe00640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00039a0 0x7fa3f015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3f5200640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f0158a00 0x7fa3f015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3ef400640 1 -- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f0158a00 msgr2=0x7fa3f015db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3ef400640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f0158a00 0x7fa3f015db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3ef400640 1 -- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00039a0 msgr2=0x7fa3f015d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3ef400640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00039a0 0x7fa3f015d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3ef400640 1 -- 172.21.15.39:0/1302656819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00c7fc0 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.460+0000 7fa3f5200640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f0158a00 0x7fa3f015db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:51.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.461+0000 7fa3ef400640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f00b3ba0 0x7fa3f015d5f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800d8e0 tx=0x7fa3d800dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:51.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.461+0000 7fa3e7e00640 1 -- 172.21.15.39:0/1302656819 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80074e0 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.461+0000 7fa3e7e00640 1 -- 172.21.15.39:0/1302656819 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d8007640 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.461+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f00c82a0 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.461+0000 7fa3e7e00640 1 -- 172.21.15.39:0/1302656819 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80049e0 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.461+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f0159100 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.462+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f0160150 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.463+0000 7fa3e7e00640 1 -- 172.21.15.39:0/1302656819 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa3d8004b40 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.464+0000 7fa3e7e00640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3c806a430 0x7fa3c806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:51.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.465+0000 7fa3e7e00640 1 -- 172.21.15.39:0/1302656819 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fa3d80887e0 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.465+0000 7fa3efe00640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3c806a430 0x7fa3c806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.466+0000 7fa3efe00640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3c806a430 0x7fa3c806c8f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0036000 tx=0x7fa3e0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:51.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.469+0000 7fa3e7e00640 1 -- 172.21.15.39:0/1302656819 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa3d80577a0 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.674+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa3f0073ef0 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.675+0000 7fa3e7e00640 1 -- 172.21.15.39:0/1302656819 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v51) v1 ==== 80+0+57 (secure 0 0 0) 0x7fa3d8019020 con 0x7fa3f00b3ba0 2024-01-24T17:44:51.677 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.1:0/1994510967 2024-01-24T18:44:37.552044+0000 2024-01-24T17:44:51.677 INFO:teuthology.orchestra.run.smithi039.stderr:listed 1 entries 2024-01-24T17:44:51.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.677+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3c806a430 msgr2=0x7fa3c806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.677+0000 7fa3f7200640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3c806a430 0x7fa3c806c8f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0036000 tx=0x7fa3e0030000 comp rx=0 tx=0).stop 2024-01-24T17:44:51.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f00b3ba0 msgr2=0x7fa3f015d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:51.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f00b3ba0 0x7fa3f015d5f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800d8e0 tx=0x7fa3d800dda0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 shutdown_connections 2024-01-24T17:44:51.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f0158a00 0x7fa3f015db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3c806a430 0x7fa3c806c8f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f00b3ba0 0x7fa3f015d5f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 --2- 172.21.15.39:0/1302656819 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3d00039a0 0x7fa3f015d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:51.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 >> 172.21.15.39:0/1302656819 conn(0x7fa3f006a3e0 msgr2=0x7fa3f00c8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:51.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 shutdown_connections 2024-01-24T17:44:51.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:51.678+0000 7fa3f7200640 1 -- 172.21.15.39:0/1302656819 wait complete. 2024-01-24T17:44:51.700 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-01-24T17:44:51.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:51.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:51.799 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:44:51.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:51.799 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:44:51.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:44:52.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.068+0000 7f79cce00640 1 Processor -- start 2024-01-24T17:44:52.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.068+0000 7f79cce00640 1 -- start start 2024-01-24T17:44:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79cce00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f79c80ca670 0x7f79c80cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79cce00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f79c806bde0 0x7f79c815a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79cce00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c815a9b0 0x7f79c80cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79cce00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c806c260 con 0x7f79c80ca670 2024-01-24T17:44:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79cce00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c80cd050 con 0x7f79c806bde0 2024-01-24T17:44:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79cce00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c8159fe0 con 0x7f79c815a9b0 2024-01-24T17:44:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79c7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f79c80ca670 0x7f79c80cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79c7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f79c80ca670 0x7f79c80cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46610/0 (socket says 172.21.15.39:46610) 2024-01-24T17:44:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.070+0000 7f79c7400640 1 -- 172.21.15.39:0/1119062292 learned_addr learned my addr 172.21.15.39:0/1119062292 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.071+0000 7f79c7e00640 1 --2- 172.21.15.39:0/1119062292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c815a9b0 0x7f79c80cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.071+0000 7f79c6a00640 1 --2- 172.21.15.39:0/1119062292 >> v2:172.21.15.100:3300/0 conn(0x7f79c806bde0 0x7f79c815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.071+0000 7f79c7400640 1 -- 172.21.15.39:0/1119062292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c815a9b0 msgr2=0x7f79c80cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.071+0000 7f79c7400640 1 --2- 172.21.15.39:0/1119062292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c815a9b0 0x7f79c80cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.071+0000 7f79c7400640 1 -- 172.21.15.39:0/1119062292 >> v2:172.21.15.100:3300/0 conn(0x7f79c806bde0 msgr2=0x7f79c815a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.071+0000 7f79c7400640 1 --2- 172.21.15.39:0/1119062292 >> v2:172.21.15.100:3300/0 conn(0x7f79c806bde0 0x7f79c815a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.071+0000 7f79c7400640 1 -- 172.21.15.39:0/1119062292 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c80caa50 con 0x7f79c80ca670 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.072+0000 7f79c7400640 1 --2- 172.21.15.39:0/1119062292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c80ca670 0x7f79c80cdd70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f79c80cd200 tx=0x7f79b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd95c62bf1b69c60 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.072+0000 7f79c6000640 1 -- 172.21.15.39:0/1119062292 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b80317e0 con 0x7f79c80ca670 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.072+0000 7f79c6000640 1 -- 172.21.15.39:0/1119062292 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b8031940 con 0x7f79c80ca670 2024-01-24T17:44:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79c6000640 1 -- 172.21.15.39:0/1119062292 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8031cd0 con 0x7f79c80ca670 2024-01-24T17:44:52.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 -- 172.21.15.39:0/1119062292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c80ca670 msgr2=0x7f79c80cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:52.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 --2- 172.21.15.39:0/1119062292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c80ca670 0x7f79c80cdd70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f79c80cd200 tx=0x7f79b802b880 comp rx=0 tx=0).stop 2024-01-24T17:44:52.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 -- 172.21.15.39:0/1119062292 shutdown_connections 2024-01-24T17:44:52.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 --2- 172.21.15.39:0/1119062292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c815a9b0 0x7f79c80cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 --2- 172.21.15.39:0/1119062292 >> v2:172.21.15.100:3300/0 conn(0x7f79c806bde0 0x7f79c815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 --2- 172.21.15.39:0/1119062292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c80ca670 0x7f79c80cdd70 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 -- 172.21.15.39:0/1119062292 >> 172.21.15.39:0/1119062292 conn(0x7f79c806a090 msgr2=0x7f79c806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:52.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 -- 172.21.15.39:0/1119062292 shutdown_connections 2024-01-24T17:44:52.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.073+0000 7f79cce00640 1 -- 172.21.15.39:0/1119062292 wait complete. 2024-01-24T17:44:52.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.075+0000 7f79cce00640 1 Processor -- start 2024-01-24T17:44:52.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.075+0000 7f79cce00640 1 -- start start 2024-01-24T17:44:52.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79cce00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79c806bde0 0x7f79c8153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:52.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79cce00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c80ca670 0x7f79c81586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:52.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79cce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c81f30a0 0x7f79c8158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:52.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79cce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c81f23f0 con 0x7f79c81f30a0 2024-01-24T17:44:52.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79cce00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c81f25e0 con 0x7f79c806bde0 2024-01-24T17:44:52.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79cce00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c81f6710 con 0x7f79c80ca670 2024-01-24T17:44:52.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c81f30a0 0x7f79c8158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c81f30a0 0x7f79c8158c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46614/0 (socket says 172.21.15.39:46614) 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.076+0000 7f79c7e00640 1 -- 172.21.15.39:0/1608924273 learned_addr learned my addr 172.21.15.39:0/1608924273 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79c6a00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c80ca670 0x7f79c81586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79c7400640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79c806bde0 0x7f79c8153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79c7e00640 1 -- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c80ca670 msgr2=0x7f79c81586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79c7e00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c80ca670 0x7f79c81586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79c7e00640 1 -- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79c806bde0 msgr2=0x7f79c8153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79c7e00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79c806bde0 0x7f79c8153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79c7e00640 1 -- 172.21.15.39:0/1608924273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b80094d0 con 0x7f79c81f30a0 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79c7e00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c81f30a0 0x7f79c8158c20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f79c0007e30 tx=0x7f79c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.077+0000 7f79bf400640 1 -- 172.21.15.39:0/1608924273 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c0014620 con 0x7f79c81f30a0 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.078+0000 7f79bf400640 1 -- 172.21.15.39:0/1608924273 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c000ee90 con 0x7f79c81f30a0 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.078+0000 7f79bf400640 1 -- 172.21.15.39:0/1608924273 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c00149c0 con 0x7f79c81f30a0 2024-01-24T17:44:52.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.078+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c81f28a0 con 0x7f79c81f30a0 2024-01-24T17:44:52.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.078+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c8153940 con 0x7f79c81f30a0 2024-01-24T17:44:52.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.079+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c8153b00 con 0x7f79c81f30a0 2024-01-24T17:44:52.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.080+0000 7f79bf400640 1 -- 172.21.15.39:0/1608924273 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f79c0014b20 con 0x7f79c81f30a0 2024-01-24T17:44:52.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.081+0000 7f79bf400640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f79a006a430 0x7f79a006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:52.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.081+0000 7f79bf400640 1 -- 172.21.15.39:0/1608924273 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f79c00893c0 con 0x7f79c81f30a0 2024-01-24T17:44:52.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.081+0000 7f79c7400640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f79a006a430 0x7f79a006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:52.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.083+0000 7f79c7400640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f79a006a430 0x7f79a006c8f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f79b8036000 tx=0x7f79b803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:52.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.085+0000 7f79bf400640 1 -- 172.21.15.39:0/1608924273 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f79c0054850 con 0x7f79c81f30a0 2024-01-24T17:44:52.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.319+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f79c804dff0 con 0x7f79c81f30a0 2024-01-24T17:44:52.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.320+0000 7f79bf400640 1 -- 172.21.15.39:0/1608924273 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7f79c0057e70 con 0x7f79c81f30a0 2024-01-24T17:44:52.320 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.322+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f79a006a430 msgr2=0x7f79a006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.322+0000 7f79cce00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f79a006a430 0x7f79a006c8f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f79b8036000 tx=0x7f79b803d040 comp rx=0 tx=0).stop 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.322+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c81f30a0 msgr2=0x7f79c8158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.322+0000 7f79cce00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c81f30a0 0x7f79c8158c20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f79c0007e30 tx=0x7f79c000e4b0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.323+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 shutdown_connections 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.323+0000 7f79cce00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f79a006a430 0x7f79a006c8f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.323+0000 7f79cce00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79c81f30a0 0x7f79c8158c20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.323+0000 7f79cce00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f79c80ca670 0x7f79c81586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.323+0000 7f79cce00640 1 --2- 172.21.15.39:0/1608924273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79c806bde0 0x7f79c8153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.323+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 >> 172.21.15.39:0/1608924273 conn(0x7f79c806a090 msgr2=0x7f79c815aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.323+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 shutdown_connections 2024-01-24T17:44:52.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:52.323+0000 7f79cce00640 1 -- 172.21.15.39:0/1608924273 wait complete. 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:44:52.343 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:44:52.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:52.344 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:44:52.397 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:44:52.397 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:44:52.397 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:44:52.397 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:52.397 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42439sec preferred_lft 42439sec 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:44:52.398 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:44:52.399 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:44:52.400 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:44:52.400 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:44:52.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:52.400 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:44:52.494 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:44:52.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:52.498 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:44:52.556 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:44:52.556 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:44:52.556 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:44:52.556 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:44:52.556 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:44:52.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:52.560 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:44:52.560 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:44:52.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:44:52.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:44:52.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:44:52.560 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:44:52.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:52.754 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:44:52.808 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:44:52.808 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:44:52.809 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:44:52.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:52.809 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:44:52.910 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:44:52.910 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:44:52.910 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:52.961 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:44:52.961 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:44:52.961 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:53.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:44:53.114 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:44:53.114 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:44:53.215 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:44:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:44:53.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:53.220 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:44:53.278 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:44:53.278 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:44:53.283 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:44:53.359 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:44:53.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:44:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:44:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:44:53.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:53.440 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:44:53.468 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:44:53.468+0000 7fa0c620b480 -1 init, newargv = 0x55c1c57bcea0 newargc=15 2024-01-24T17:44:53.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[56529]: starting ceph client 2024-01-24T17:44:53.485 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[56529]: starting fuse 2024-01-24T17:44:53.486 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:44:53.486 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:44:53.486 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:44:53.486 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:44:53.487 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:44:53.488 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:44:53.691 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.56529.asok 2024-01-24T17:44:53.869 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56529.asok 2024-01-24T17:44:53.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:53.869 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56529.asok status 2024-01-24T17:44:54.118 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:44:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:44:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:44:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "56529", 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 6033, 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 6033 2024-01-24T17:44:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3166769649 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3166769649 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.6033 192.168.0.1:0/3166769649", 2024-01-24T17:44:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/3166769649", 2024-01-24T17:44:54.121 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:44:54.121 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 27, 2024-01-24T17:44:54.121 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 51, 2024-01-24T17:44:54.121 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:44:54.121 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:44:54.121 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:44:54.121 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:44:54.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:54.134 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:44:54.134 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:54.187 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:44:54.187 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:54.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:54.188 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:54.288 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:44:54.289 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:44:54.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:44:54.514 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:44:54.514 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:44:54.514 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:54.574 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:44:54.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.799+0000 7f5f81c00640 1 Processor -- start 2024-01-24T17:44:54.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.799+0000 7f5f81c00640 1 -- start start 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.800+0000 7f5f81c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5f7c0c8b90 0x7f5f7c0c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.800+0000 7f5f81c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f7c0caf00 0x7f5f7c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.800+0000 7f5f81c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7c0c9c10 con 0x7f5f7c15ca30 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f81c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c0cb8b0 con 0x7f5f7c0caf00 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f81c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c0cd590 con 0x7f5f7c0c8b90 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f7b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f7c0caf00 0x7f5f7c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f7b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f7c0caf00 0x7f5f7c0cb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43438/0 (socket says 172.21.15.39:43438) 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f7b400640 1 -- 172.21.15.39:0/844006319 learned_addr learned my addr 172.21.15.39:0/844006319 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:54.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f7a000640 1 --2- 172.21.15.39:0/844006319 >> v2:172.21.15.100:3300/0 conn(0x7f5f7c0c8b90 0x7f5f7c0c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:54.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f7b400640 1 -- 172.21.15.39:0/844006319 >> v2:172.21.15.100:3300/0 conn(0x7f5f7c0c8b90 msgr2=0x7f5f7c0c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:54.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f7b400640 1 --2- 172.21.15.39:0/844006319 >> v2:172.21.15.100:3300/0 conn(0x7f5f7c0c8b90 0x7f5f7c0c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:54.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f7b400640 1 -- 172.21.15.39:0/844006319 >> v1:172.21.15.100:6789/0 conn(0x7f5f7c15ca30 legacy=0x7f5f7c0cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:54.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.801+0000 7f5f7b400640 1 -- 172.21.15.39:0/844006319 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0c9650 con 0x7f5f7c0caf00 2024-01-24T17:44:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.803+0000 7f5f7b400640 1 --2- 172.21.15.39:0/844006319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c0caf00 0x7f5f7c0cb370 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c06e230 tx=0x7f5f7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77200a675497badb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.803+0000 7f5f79600640 1 -- 172.21.15.39:0/844006319 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7402ca80 con 0x7f5f7c0caf00 2024-01-24T17:44:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.803+0000 7f5f79600640 1 -- 172.21.15.39:0/844006319 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7402cbe0 con 0x7f5f7c0caf00 2024-01-24T17:44:54.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.803+0000 7f5f79600640 1 -- 172.21.15.39:0/844006319 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f740343f0 con 0x7f5f7c0caf00 2024-01-24T17:44:54.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.803+0000 7f5f81c00640 1 -- 172.21.15.39:0/844006319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c0caf00 msgr2=0x7f5f7c0cb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.804+0000 7f5f81c00640 1 --2- 172.21.15.39:0/844006319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c0caf00 0x7f5f7c0cb370 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c06e230 tx=0x7f5f7402bee0 comp rx=0 tx=0).stop 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.804+0000 7f5f81c00640 1 -- 172.21.15.39:0/844006319 shutdown_connections 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.804+0000 7f5f81c00640 1 --2- 172.21.15.39:0/844006319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c0caf00 0x7f5f7c0cb370 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.804+0000 7f5f81c00640 1 --2- 172.21.15.39:0/844006319 >> v2:172.21.15.100:3300/0 conn(0x7f5f7c0c8b90 0x7f5f7c0c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.804+0000 7f5f81c00640 1 -- 172.21.15.39:0/844006319 >> 172.21.15.39:0/844006319 conn(0x7f5f7c06a3e0 msgr2=0x7f5f7c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.804+0000 7f5f81c00640 1 -- 172.21.15.39:0/844006319 shutdown_connections 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.804+0000 7f5f81c00640 1 -- 172.21.15.39:0/844006319 wait complete. 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.806+0000 7f5f81c00640 1 Processor -- start 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.806+0000 7f5f81c00640 1 -- start start 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f81c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f7c0c8b90 0x7f5f7c0c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:54.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f81c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f7c0caf00 0x7f5f7c1523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f81c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c15ca30 0x7f5f7c0c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f81c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c1528e0 con 0x7f5f7c15ca30 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f81c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c15eb00 con 0x7f5f7c0c8b90 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f81c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0c4d20 con 0x7f5f7c0caf00 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f7b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c15ca30 0x7f5f7c0c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f7b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c15ca30 0x7f5f7c0c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43454/0 (socket says 172.21.15.39:43454) 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f7b400640 1 -- 172.21.15.39:0/3004924032 learned_addr learned my addr 172.21.15.39:0/3004924032 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f7b400640 1 -- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f7c0caf00 msgr2=0x7f5f7c1523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f7b400640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f7c0caf00 0x7f5f7c1523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f7b400640 1 -- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f7c0c8b90 msgr2=0x7f5f7c0c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f7b400640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f7c0c8b90 0x7f5f7c0c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.807+0000 7f5f7b400640 1 -- 172.21.15.39:0/3004924032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f74009b30 con 0x7f5f7c15ca30 2024-01-24T17:44:54.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.808+0000 7f5f7b400640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c15ca30 0x7f5f7c0c41f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009f40 tx=0x7f5f74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:54.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.808+0000 7f5f72a00640 1 -- 172.21.15.39:0/3004924032 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74009510 con 0x7f5f7c15ca30 2024-01-24T17:44:54.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.808+0000 7f5f72a00640 1 -- 172.21.15.39:0/3004924032 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7403e450 con 0x7f5f7c15ca30 2024-01-24T17:44:54.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.808+0000 7f5f72a00640 1 -- 172.21.15.39:0/3004924032 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7403e7e0 con 0x7f5f7c15ca30 2024-01-24T17:44:54.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.808+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7c15d590 con 0x7f5f7c15ca30 2024-01-24T17:44:54.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.808+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7c15cf40 con 0x7f5f7c15ca30 2024-01-24T17:44:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.809+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7c073ef0 con 0x7f5f7c15ca30 2024-01-24T17:44:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.810+0000 7f5f72a00640 1 -- 172.21.15.39:0/3004924032 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5f7402cd50 con 0x7f5f7c15ca30 2024-01-24T17:44:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.811+0000 7f5f72a00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f5406a460 0x7f5f5406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.811+0000 7f5f72a00640 1 -- 172.21.15.39:0/3004924032 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f5f740ab100 con 0x7f5f7c15ca30 2024-01-24T17:44:54.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.812+0000 7f5f7aa00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f5406a460 0x7f5f5406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:54.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.813+0000 7f5f7aa00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f5406a460 0x7f5f5406c920 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c006fd0 tx=0x7f5f6c006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:54.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:54.816+0000 7f5f72a00640 1 -- 172.21.15.39:0/3004924032 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5f74076620 con 0x7f5f7c15ca30 2024-01-24T17:44:55.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.011+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f7c0b44c0 con 0x7f5f7c15ca30 2024-01-24T17:44:55.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.014+0000 7f5f72a00640 1 -- 172.21.15.39:0/3004924032 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16392 (secure 0 0 0) 0x7f5f74079c40 con 0x7f5f7c15ca30 2024-01-24T17:44:55.015 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:55.016 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:38.321132+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.9","mappings":[{"from":5,"to":7}]},{"pgid":"7.1d","mappings":[{"from":5,"to":7}]},{"pgid":"7.39","mappings":[{"from":5,"to":0}]},{"pgid":"7.3a","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1994510967":"2024-01-24T18:44:37.552044+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:44:55.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f5406a460 msgr2=0x7f5f5406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f5406a460 0x7f5f5406c920 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c006fd0 tx=0x7f5f6c006bc0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c15ca30 msgr2=0x7f5f7c0c41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c15ca30 0x7f5f7c0c41f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009f40 tx=0x7f5f74004a40 comp rx=0 tx=0).stop 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 shutdown_connections 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f5406a460 0x7f5f5406c920 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f7c15ca30 0x7f5f7c0c41f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f7c0caf00 0x7f5f7c1523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 --2- 172.21.15.39:0/3004924032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f7c0c8b90 0x7f5f7c0c3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.017+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 >> 172.21.15.39:0/3004924032 conn(0x7f5f7c06a3e0 msgr2=0x7f5f7c0cdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.018+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 shutdown_connections 2024-01-24T17:44:55.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.018+0000 7f5f81c00640 1 -- 172.21.15.39:0/3004924032 wait complete. 2024-01-24T17:44:55.038 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:44:07.660016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:44:08.985819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:44:55.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:44:55.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.257+0000 7fe0dfe00640 1 Processor -- start 2024-01-24T17:44:55.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.258+0000 7fe0dfe00640 1 -- start start 2024-01-24T17:44:55.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.259+0000 7fe0dfe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe0e00cd810 0x7fe0e00cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.259+0000 7fe0dfe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e006ef10 con 0x7fe0e015df80 2024-01-24T17:44:55.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.259+0000 7fe0dfe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e00d1670 con 0x7fe0e006f9b0 2024-01-24T17:44:55.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.259+0000 7fe0dfe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e00cb350 con 0x7fe0e00cd810 2024-01-24T17:44:55.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.259+0000 7fe0dea00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe0e015df80 0x7fe0e00cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50616/0 (socket says 172.21.15.39:50616) 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.259+0000 7fe0dea00640 1 -- 172.21.15.39:0/460477981 learned_addr learned my addr 172.21.15.39:0/460477981 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34764636 0 0) 0x7fe0e006ef10 con 0x7fe0e015df80 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c0003520 con 0x7fe0e015df80 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0df400640 1 --2- 172.21.15.39:0/460477981 >> v2:172.21.15.100:3300/0 conn(0x7fe0e00cd810 0x7fe0e00cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3199153096 0 0) 0x7fe0c0003520 con 0x7fe0e015df80 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 >> v2:172.21.15.100:3300/0 conn(0x7fe0e00cd810 msgr2=0x7fe0e00cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 --2- 172.21.15.39:0/460477981 >> v2:172.21.15.100:3300/0 conn(0x7fe0e00cd810 0x7fe0e00cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 >> v1:172.21.15.100:6789/0 conn(0x7fe0e006f9b0 legacy=0x7fe0e00cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e006f160 con 0x7fe0e015df80 2024-01-24T17:44:55.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fe0d00017f0 con 0x7fe0e015df80 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 >> v1:172.21.15.39:6789/0 conn(0x7fe0e015df80 legacy=0x7fe0e00cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00039c0 0x7fe0c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c00047c0 0x7fe0c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0c00051d0 0x7fe0c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00cb350 con 0x7fe0c00047c0 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c000c120 con 0x7fe0c00051d0 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.260+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c000c2e0 con 0x7fe0c00039c0 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.261+0000 7fe0d7e00640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c00047c0 0x7fe0c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.261+0000 7fe0dea00640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00039c0 0x7fe0c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.261+0000 7fe0d7e00640 1 -- 172.21.15.39:0/460477981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00039c0 msgr2=0x7fe0c0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.261+0000 7fe0d7e00640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00039c0 0x7fe0c0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.261+0000 7fe0d7e00640 1 -- 172.21.15.39:0/460477981 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0c00051d0 msgr2=0x7fe0c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.261+0000 7fe0d7e00640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0c00051d0 0x7fe0c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.261+0000 7fe0d7e00640 1 -- 172.21.15.39:0/460477981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c0003520 con 0x7fe0c00047c0 2024-01-24T17:44:55.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.261+0000 7fe0df400640 1 -- 172.21.15.39:0/460477981 reap_dead start 2024-01-24T17:44:55.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.262+0000 7fe0d7e00640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c00047c0 0x7fe0c0004c40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8001750 tx=0x7fe0c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0de719f9474ccee server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:55.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c80052d0 con 0x7fe0c00047c0 2024-01-24T17:44:55.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c802e050 con 0x7fe0c00047c0 2024-01-24T17:44:55.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0de000640 1 -- 172.21.15.39:0/460477981 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8003830 con 0x7fe0c00047c0 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0dfe00640 1 -- 172.21.15.39:0/460477981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c00047c0 msgr2=0x7fe0c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0dfe00640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c00047c0 0x7fe0c0004c40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8001750 tx=0x7fe0c00057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0dfe00640 1 -- 172.21.15.39:0/460477981 shutdown_connections 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0dfe00640 1 --2- 172.21.15.39:0/460477981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c00047c0 0x7fe0c0004c40 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0dfe00640 1 -- 172.21.15.39:0/460477981 >> 172.21.15.39:0/460477981 conn(0x7fe0e0089fd0 msgr2=0x7fe0e00d06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0dfe00640 1 -- 172.21.15.39:0/460477981 shutdown_connections 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.263+0000 7fe0dfe00640 1 -- 172.21.15.39:0/460477981 wait complete. 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.265+0000 7fe0dfe00640 1 Processor -- start 2024-01-24T17:44:55.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.265+0000 7fe0dfe00640 1 -- start start 2024-01-24T17:44:55.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.266+0000 7fe0dfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00047c0 0x7fe0e00b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.266+0000 7fe0dfe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0e00c6b80 0x7fe0e0159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.266+0000 7fe0dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0e00b0d40 0x7fe0e00b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.266+0000 7fe0dfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00b2050 con 0x7fe0e00b0d40 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.266+0000 7fe0dfe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00b0260 con 0x7fe0e00c6b80 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.266+0000 7fe0dfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00b0540 con 0x7fe0c00047c0 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0e00b0d40 0x7fe0e00b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0e00b0d40 0x7fe0e00b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43476/0 (socket says 172.21.15.39:43476) 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 -- 172.21.15.39:0/490059281 learned_addr learned my addr 172.21.15.39:0/490059281 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0dea00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00047c0 0x7fe0e00b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 -- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00047c0 msgr2=0x7fe0e00b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0d7e00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0e00c6b80 0x7fe0e0159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00047c0 0x7fe0e00b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 -- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0e00c6b80 msgr2=0x7fe0e0159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0e00c6b80 0x7fe0e0159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 -- 172.21.15.39:0/490059281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e00b0820 con 0x7fe0e00b0d40 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.267+0000 7fe0df400640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0e00b0d40 0x7fe0e00b11c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00cde70 tx=0x7fe0d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.268+0000 7fe0d6a00640 1 -- 172.21.15.39:0/490059281 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d800ed90 con 0x7fe0e00b0d40 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.268+0000 7fe0d6a00640 1 -- 172.21.15.39:0/490059281 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d8014420 con 0x7fe0e00b0d40 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.268+0000 7fe0d6a00640 1 -- 172.21.15.39:0/490059281 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d80147d0 con 0x7fe0e00b0d40 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.268+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e00b2330 con 0x7fe0e00b0d40 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.268+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e0159970 con 0x7fe0e00b0d40 2024-01-24T17:44:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.269+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e0067040 con 0x7fe0e00b0d40 2024-01-24T17:44:55.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.270+0000 7fe0d6a00640 1 -- 172.21.15.39:0/490059281 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe0d8014930 con 0x7fe0e00b0d40 2024-01-24T17:44:55.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.271+0000 7fe0d6a00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0b806a430 0x7fe0b806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.271+0000 7fe0d6a00640 1 -- 172.21.15.39:0/490059281 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fe0d8088f80 con 0x7fe0e00b0d40 2024-01-24T17:44:55.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.271+0000 7fe0dea00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0b806a430 0x7fe0b806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.273+0000 7fe0dea00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0b806a430 0x7fe0b806c8f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe0e004f260 tx=0x7fe0d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:55.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.275+0000 7fe0d6a00640 1 -- 172.21.15.39:0/490059281 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe0d80544b0 con 0x7fe0e00b0d40 2024-01-24T17:44:55.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.467+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe0e00d02a0 con 0x7fe0e00b0d40 2024-01-24T17:44:55.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.468+0000 7fe0d6a00640 1 -- 172.21.15.39:0/490059281 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe0d8057ad0 con 0x7fe0e00b0d40 2024-01-24T17:44:55.469 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:44:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0b806a430 msgr2=0x7fe0b806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0b806a430 0x7fe0b806c8f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe0e004f260 tx=0x7fe0d0009040 comp rx=0 tx=0).stop 2024-01-24T17:44:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0e00b0d40 msgr2=0x7fe0e00b11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0e00b0d40 0x7fe0e00b11c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00cde70 tx=0x7fe0d800e4b0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 shutdown_connections 2024-01-24T17:44:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0b806a430 0x7fe0b806c8f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0e00b0d40 0x7fe0e00b11c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0e00c6b80 0x7fe0e0159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 --2- 172.21.15.39:0/490059281 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0c00047c0 0x7fe0e00b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 >> 172.21.15.39:0/490059281 conn(0x7fe0e0089fd0 msgr2=0x7fe0e00ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 shutdown_connections 2024-01-24T17:44:55.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.471+0000 7fe0dfe00640 1 -- 172.21.15.39:0/490059281 wait complete. 2024-01-24T17:44:55.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:44:55.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.699+0000 7fb2de800640 1 Processor -- start 2024-01-24T17:44:55.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.699+0000 7fb2de800640 1 -- start start 2024-01-24T17:44:55.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.701+0000 7fb2de800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb2d80cb540 0x7fb2d806d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.701+0000 7fb2de800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb2d806c3a0 0x7fb2d80ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.701+0000 7fb2de800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d80ad0b0 con 0x7fb2d80accd0 2024-01-24T17:44:55.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.701+0000 7fb2de800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d806d840 con 0x7fb2d806c3a0 2024-01-24T17:44:55.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.701+0000 7fb2de800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d80b2370 con 0x7fb2d80cb540 2024-01-24T17:44:55.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.701+0000 7fb2dde00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb2d806c3a0 0x7fb2d80ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.701+0000 7fb2dd400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb2d80accd0 0x7fb2d80cae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50632/0 (socket says 172.21.15.39:50632) 2024-01-24T17:44:55.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.701+0000 7fb2dd400640 1 -- 172.21.15.39:0/1951493533 learned_addr learned my addr 172.21.15.39:0/1951493533 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:55.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.702+0000 7fb2dca00640 1 --2- 172.21.15.39:0/1951493533 >> v2:172.21.15.100:3300/0 conn(0x7fb2d80cb540 0x7fb2d806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.702+0000 7fb2dde00640 1 -- 172.21.15.39:0/1951493533 >> v2:172.21.15.100:3300/0 conn(0x7fb2d80cb540 msgr2=0x7fb2d806d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:55.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.702+0000 7fb2dde00640 1 --2- 172.21.15.39:0/1951493533 >> v2:172.21.15.100:3300/0 conn(0x7fb2d80cb540 0x7fb2d806d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.702+0000 7fb2dde00640 1 -- 172.21.15.39:0/1951493533 >> v1:172.21.15.39:6789/0 conn(0x7fb2d80accd0 legacy=0x7fb2d80cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.702+0000 7fb2dde00640 1 -- 172.21.15.39:0/1951493533 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d80b2650 con 0x7fb2d806c3a0 2024-01-24T17:44:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.702+0000 7fb2d7e00640 1 -- 172.21.15.39:0/1951493533 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544825751 0 0) 0x7fb2d80ad0b0 con 0x7fb2d80accd0 2024-01-24T17:44:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.703+0000 7fb2dde00640 1 --2- 172.21.15.39:0/1951493533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d806c3a0 0x7fb2d80ab030 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb2d806def0 tx=0x7fb2d002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e833e19f2b775547 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2d7e00640 1 -- 172.21.15.39:0/1951493533 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d002cc70 con 0x7fb2d806c3a0 2024-01-24T17:44:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2d7e00640 1 -- 172.21.15.39:0/1951493533 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d002cdd0 con 0x7fb2d806c3a0 2024-01-24T17:44:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2d7e00640 1 -- 172.21.15.39:0/1951493533 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0034660 con 0x7fb2d806c3a0 2024-01-24T17:44:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2de800640 1 -- 172.21.15.39:0/1951493533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d806c3a0 msgr2=0x7fb2d80ab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2de800640 1 --2- 172.21.15.39:0/1951493533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d806c3a0 0x7fb2d80ab030 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb2d806def0 tx=0x7fb2d002bec0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2de800640 1 -- 172.21.15.39:0/1951493533 shutdown_connections 2024-01-24T17:44:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2de800640 1 --2- 172.21.15.39:0/1951493533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d806c3a0 0x7fb2d80ab030 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2de800640 1 --2- 172.21.15.39:0/1951493533 >> v2:172.21.15.100:3300/0 conn(0x7fb2d80cb540 0x7fb2d806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2de800640 1 -- 172.21.15.39:0/1951493533 >> 172.21.15.39:0/1951493533 conn(0x7fb2d8069e50 msgr2=0x7fb2d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2de800640 1 -- 172.21.15.39:0/1951493533 shutdown_connections 2024-01-24T17:44:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.704+0000 7fb2de800640 1 -- 172.21.15.39:0/1951493533 wait complete. 2024-01-24T17:44:55.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.706+0000 7fb2de800640 1 Processor -- start 2024-01-24T17:44:55.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.706+0000 7fb2de800640 1 -- start start 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.707+0000 7fb2de800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2d806c3a0 0x7fb2d8155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.707+0000 7fb2de800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2d80accd0 0x7fb2d80c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.707+0000 7fb2de800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d80cb540 0x7fb2d80c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.707+0000 7fb2de800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d815dac0 con 0x7fb2d80cb540 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.707+0000 7fb2de800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d815ccf0 con 0x7fb2d80accd0 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.707+0000 7fb2de800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d815cf40 con 0x7fb2d806c3a0 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d80cb540 0x7fb2d80c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d80cb540 0x7fb2d80c2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43494/0 (socket says 172.21.15.39:43494) 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 -- 172.21.15.39:0/3596505172 learned_addr learned my addr 172.21.15.39:0/3596505172 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dd400640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2d806c3a0 0x7fb2d8155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dca00640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2d80accd0 0x7fb2d80c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 -- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2d806c3a0 msgr2=0x7fb2d8155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2d806c3a0 0x7fb2d8155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 -- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2d80accd0 msgr2=0x7fb2d80c1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2d80accd0 0x7fb2d80c1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 -- 172.21.15.39:0/3596505172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0009a70 con 0x7fb2d80cb540 2024-01-24T17:44:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dca00640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2d80accd0 0x7fb2d80c1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.708+0000 7fb2dde00640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d80cb540 0x7fb2d80c2410 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009e80 tx=0x7fb2d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.709+0000 7fb2d5600640 1 -- 172.21.15.39:0/3596505172 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0040650 con 0x7fb2d80cb540 2024-01-24T17:44:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.709+0000 7fb2d5600640 1 -- 172.21.15.39:0/3596505172 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d00407d0 con 0x7fb2d80cb540 2024-01-24T17:44:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.709+0000 7fb2d5600640 1 -- 172.21.15.39:0/3596505172 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0040b80 con 0x7fb2d80cb540 2024-01-24T17:44:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.709+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d815d220 con 0x7fb2d80cb540 2024-01-24T17:44:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.709+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d815d7e0 con 0x7fb2d80cb540 2024-01-24T17:44:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.710+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d804dff0 con 0x7fb2d80cb540 2024-01-24T17:44:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.711+0000 7fb2d5600640 1 -- 172.21.15.39:0/3596505172 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb2d0040ce0 con 0x7fb2d80cb540 2024-01-24T17:44:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.712+0000 7fb2d5600640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2b006a430 0x7fb2b006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.713+0000 7fb2d5600640 1 -- 172.21.15.39:0/3596505172 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fb2d00ab720 con 0x7fb2d80cb540 2024-01-24T17:44:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.713+0000 7fb2dd400640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2b006a430 0x7fb2b006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:55.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.714+0000 7fb2dd400640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2b006a430 0x7fb2b006c8f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80010e0 tx=0x7fb2c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.717+0000 7fb2d5600640 1 -- 172.21.15.39:0/3596505172 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb2d0076ce0 con 0x7fb2d80cb540 2024-01-24T17:44:55.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.912+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb2d80b2be0 con 0x7fb2d80cb540 2024-01-24T17:44:55.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.913+0000 7fb2d5600640 1 -- 172.21.15.39:0/3596505172 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb2d007a300 con 0x7fb2d80cb540 2024-01-24T17:44:55.914 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2b006a430 msgr2=0x7fb2b006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2b006a430 0x7fb2b006c8f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80010e0 tx=0x7fb2c8009040 comp rx=0 tx=0).stop 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d80cb540 msgr2=0x7fb2d80c2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d80cb540 0x7fb2d80c2410 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009e80 tx=0x7fb2d0004a40 comp rx=0 tx=0).stop 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 shutdown_connections 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2b006a430 0x7fb2b006c8f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2d80cb540 0x7fb2d80c2410 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2d80accd0 0x7fb2d80c1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 --2- 172.21.15.39:0/3596505172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2d806c3a0 0x7fb2d8155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 >> 172.21.15.39:0/3596505172 conn(0x7fb2d8069e50 msgr2=0x7fb2d80c9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:55.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 shutdown_connections 2024-01-24T17:44:55.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:55.916+0000 7fb2de800640 1 -- 172.21.15.39:0/3596505172 wait complete. 2024-01-24T17:44:55.936 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:44:56.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.164+0000 7fe16ec00640 1 Processor -- start 2024-01-24T17:44:56.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.165+0000 7fe16ec00640 1 -- start start 2024-01-24T17:44:56.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.166+0000 7fe16ec00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe16806d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.166+0000 7fe16ec00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe16806c3a0 0x7fe1680cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.166+0000 7fe16ec00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1680ad0b0 con 0x7fe1680cb370 2024-01-24T17:44:56.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.166+0000 7fe16ec00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1680cb800 con 0x7fe16806c3a0 2024-01-24T17:44:56.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.166+0000 7fe16ec00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16806d840 con 0x7fe1680accd0 2024-01-24T17:44:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.167+0000 7fe16d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe16806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.167+0000 7fe16d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe16806d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:46320/0 (socket says 172.21.15.39:46320) 2024-01-24T17:44:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.167+0000 7fe16d800640 1 -- 172.21.15.39:0/554343891 learned_addr learned my addr 172.21.15.39:0/554343891 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.167+0000 7fe16ce00640 1 --2- 172.21.15.39:0/554343891 >> v2:172.21.15.100:3300/0 conn(0x7fe16806c3a0 0x7fe1680cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.167+0000 7fe16d800640 1 -- 172.21.15.39:0/554343891 >> v2:172.21.15.100:3300/0 conn(0x7fe16806c3a0 msgr2=0x7fe1680cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.167+0000 7fe16d800640 1 --2- 172.21.15.39:0/554343891 >> v2:172.21.15.100:3300/0 conn(0x7fe16806c3a0 0x7fe1680cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.167+0000 7fe16d800640 1 -- 172.21.15.39:0/554343891 >> v1:172.21.15.100:6789/0 conn(0x7fe1680cb370 legacy=0x7fe16815e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.167+0000 7fe16d800640 1 -- 172.21.15.39:0/554343891 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680b2370 con 0x7fe1680accd0 2024-01-24T17:44:56.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.169+0000 7fe16d800640 1 --2- 172.21.15.39:0/554343891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe16806d300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe158009850 tx=0x7fe15802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff58c73dbda229fb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:56.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.169+0000 7fe167e00640 1 -- 172.21.15.39:0/554343891 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158004230 con 0x7fe1680accd0 2024-01-24T17:44:56.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.169+0000 7fe167e00640 1 -- 172.21.15.39:0/554343891 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe158004390 con 0x7fe1680accd0 2024-01-24T17:44:56.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.169+0000 7fe167e00640 1 -- 172.21.15.39:0/554343891 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158003810 con 0x7fe1680accd0 2024-01-24T17:44:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.170+0000 7fe16ec00640 1 -- 172.21.15.39:0/554343891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 msgr2=0x7fe16806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.170+0000 7fe16ec00640 1 --2- 172.21.15.39:0/554343891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe16806d300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe158009850 tx=0x7fe15802b960 comp rx=0 tx=0).stop 2024-01-24T17:44:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.170+0000 7fe16ec00640 1 -- 172.21.15.39:0/554343891 shutdown_connections 2024-01-24T17:44:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.170+0000 7fe16ec00640 1 --2- 172.21.15.39:0/554343891 >> v2:172.21.15.100:3300/0 conn(0x7fe16806c3a0 0x7fe1680cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.170+0000 7fe16ec00640 1 --2- 172.21.15.39:0/554343891 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe16806d300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.170+0000 7fe16ec00640 1 -- 172.21.15.39:0/554343891 >> 172.21.15.39:0/554343891 conn(0x7fe168069e50 msgr2=0x7fe16806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.170+0000 7fe16ec00640 1 -- 172.21.15.39:0/554343891 shutdown_connections 2024-01-24T17:44:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.170+0000 7fe16ec00640 1 -- 172.21.15.39:0/554343891 wait complete. 2024-01-24T17:44:56.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.172+0000 7fe16ec00640 1 Processor -- start 2024-01-24T17:44:56.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.173+0000 7fe16ec00640 1 -- start start 2024-01-24T17:44:56.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16ec00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe16806c3a0 0x7fe168157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16ec00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe1680c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16ec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680cb370 0x7fe1681f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16ec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16815dcb0 con 0x7fe1680cb370 2024-01-24T17:44:56.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16ec00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1680ca720 con 0x7fe16806c3a0 2024-01-24T17:44:56.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16ec00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16815be70 con 0x7fe1680accd0 2024-01-24T17:44:56.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680cb370 0x7fe1681f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680cb370 0x7fe1681f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43504/0 (socket says 172.21.15.39:43504) 2024-01-24T17:44:56.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.174+0000 7fe16e200640 1 -- 172.21.15.39:0/2312122907 learned_addr learned my addr 172.21.15.39:0/2312122907 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:56.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.175+0000 7fe16d800640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe16806c3a0 0x7fe168157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.175+0000 7fe16ce00640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe1680c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.175+0000 7fe16e200640 1 -- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 msgr2=0x7fe1680c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.175+0000 7fe16e200640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe1680c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.175+0000 7fe16e200640 1 -- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe16806c3a0 msgr2=0x7fe168157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.175+0000 7fe16e200640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe16806c3a0 0x7fe168157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.175+0000 7fe16e200640 1 -- 172.21.15.39:0/2312122907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe158009460 con 0x7fe1680cb370 2024-01-24T17:44:56.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.175+0000 7fe16e200640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680cb370 0x7fe1681f1da0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe16000c6e0 tx=0x7fe16000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:56.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.176+0000 7fe165600640 1 -- 172.21.15.39:0/2312122907 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160003b70 con 0x7fe1680cb370 2024-01-24T17:44:56.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.176+0000 7fe165600640 1 -- 172.21.15.39:0/2312122907 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160003cf0 con 0x7fe1680cb370 2024-01-24T17:44:56.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.176+0000 7fe165600640 1 -- 172.21.15.39:0/2312122907 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160005310 con 0x7fe1680cb370 2024-01-24T17:44:56.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.176+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1681f22e0 con 0x7fe1680cb370 2024-01-24T17:44:56.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.176+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1681f2830 con 0x7fe1680cb370 2024-01-24T17:44:56.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.177+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16815c040 con 0x7fe1680cb370 2024-01-24T17:44:56.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.178+0000 7fe165600640 1 -- 172.21.15.39:0/2312122907 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe160005470 con 0x7fe1680cb370 2024-01-24T17:44:56.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.180+0000 7fe165600640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe14006a430 0x7fe14006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.180+0000 7fe165600640 1 -- 172.21.15.39:0/2312122907 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fe1600886c0 con 0x7fe1680cb370 2024-01-24T17:44:56.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.180+0000 7fe16d800640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe14006a430 0x7fe14006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.182+0000 7fe16d800640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe14006a430 0x7fe14006c8f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe158035000 tx=0x7fe158008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:56.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.184+0000 7fe165600640 1 -- 172.21.15.39:0/2312122907 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe16815c040 con 0x7fe1680cb370 2024-01-24T17:44:56.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.379+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe16815c040 con 0x7fe1680cb370 2024-01-24T17:44:56.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.380+0000 7fe165600640 1 -- 172.21.15.39:0/2312122907 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe16815c040 con 0x7fe1680cb370 2024-01-24T17:44:56.381 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe14006a430 msgr2=0x7fe14006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe14006a430 0x7fe14006c8f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe158035000 tx=0x7fe158008030 comp rx=0 tx=0).stop 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680cb370 msgr2=0x7fe1681f1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680cb370 0x7fe1681f1da0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fe16000c6e0 tx=0x7fe16000cba0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 shutdown_connections 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe14006a430 0x7fe14006c8f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1680cb370 0x7fe1681f1da0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1680accd0 0x7fe1680c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 --2- 172.21.15.39:0/2312122907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe16806c3a0 0x7fe168157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 >> 172.21.15.39:0/2312122907 conn(0x7fe168069e50 msgr2=0x7fe1680abd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:56.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 shutdown_connections 2024-01-24T17:44:56.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.383+0000 7fe16ec00640 1 -- 172.21.15.39:0/2312122907 wait complete. 2024-01-24T17:44:56.403 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:44:56.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.622+0000 7f53b6400640 1 Processor -- start 2024-01-24T17:44:56.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.622+0000 7f53b6400640 1 -- start start 2024-01-24T17:44:56.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53b6400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 0x7f53b00cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53b6400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f53b00caf00 0x7f53b00cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53b6400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b00c9c10 con 0x7f53b00cd3c0 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53b6400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b00cb8d0 con 0x7f53b00caf00 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53b6400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00c9650 con 0x7f53b015ca30 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53a6800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f53b00cd3c0 0x7f53b00c8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50636/0 (socket says 172.21.15.39:50636) 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53a6800640 1 -- 172.21.15.39:0/1165796931 learned_addr learned my addr 172.21.15.39:0/1165796931 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53af400640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 0x7f53b00cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53afe00640 1 --2- 172.21.15.39:0/1165796931 >> v2:172.21.15.100:3300/0 conn(0x7f53b00caf00 0x7f53b00cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.624+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853164934 0 0) 0x7f53b00c9c10 con 0x7f53b00cd3c0 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5390003520 con 0x7f53b00cd3c0 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386058547 0 0) 0x7f5390003520 con 0x7f53b00cd3c0 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 msgr2=0x7f53b00cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 0x7f53b00cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 >> v2:172.21.15.100:3300/0 conn(0x7f53b00caf00 msgr2=0x7f53b00cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 --2- 172.21.15.39:0/1165796931 >> v2:172.21.15.100:3300/0 conn(0x7f53b00caf00 0x7f53b00cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b00c9930 con 0x7f53b00cd3c0 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f5398001670 con 0x7f53b00cd3c0 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 >> v1:172.21.15.39:6789/0 conn(0x7f53b00cd3c0 legacy=0x7f53b00c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5390003970 0x7f5390003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 0x7f53900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5390004d00 0x7f5390005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00cb8d0 con 0x7f5390004d00 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00c9650 con 0x7f5390003970 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539000b980 con 0x7f53b015ca30 2024-01-24T17:44:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.625+0000 7f53afe00640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5390004d00 0x7f5390005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.626+0000 7f53af400640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5390003970 0x7f5390003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.626+0000 7f53a6800640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 0x7f53900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.626+0000 7f53afe00640 1 -- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 msgr2=0x7f53900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.626+0000 7f53afe00640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 0x7f53900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.626+0000 7f53afe00640 1 -- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5390003970 msgr2=0x7f5390003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.626+0000 7f53afe00640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5390003970 0x7f5390003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.626+0000 7f53afe00640 1 -- 172.21.15.39:0/1165796931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390003520 con 0x7f5390004d00 2024-01-24T17:44:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.626+0000 7f53a6800640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f53b015ca30 0x7f53900047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.627+0000 7f53afe00640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5390004d00 0x7f5390005180 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009fc0 tx=0x7f53900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da6c85faabfce4a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.627+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a802cc60 con 0x7f5390004d00 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.627+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a800f040 con 0x7f5390004d00 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.627+0000 7f53aea00640 1 -- 172.21.15.39:0/1165796931 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8034440 con 0x7f5390004d00 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.628+0000 7f53b6400640 1 -- 172.21.15.39:0/1165796931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5390004d00 msgr2=0x7f5390005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.628+0000 7f53b6400640 1 --2- 172.21.15.39:0/1165796931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5390004d00 0x7f5390005180 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009fc0 tx=0x7f53900057f0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.628+0000 7f53afe00640 1 -- 172.21.15.39:0/1165796931 reap_dead start 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.628+0000 7f53b6400640 1 -- 172.21.15.39:0/1165796931 shutdown_connections 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.628+0000 7f53b6400640 1 -- 172.21.15.39:0/1165796931 >> 172.21.15.39:0/1165796931 conn(0x7f53b006a3e0 msgr2=0x7f53b006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.628+0000 7f53b6400640 1 -- 172.21.15.39:0/1165796931 shutdown_connections 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.628+0000 7f53b6400640 1 -- 172.21.15.39:0/1165796931 wait complete. 2024-01-24T17:44:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.630+0000 7f53b6400640 1 Processor -- start 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.630+0000 7f53b6400640 1 -- start start 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.631+0000 7f53b6400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5390004d00 0x7f53b01f1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.631+0000 7f53b6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53b00adfa0 0x7f53b00ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.631+0000 7f53b6400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ed80 0x7f53b015f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.631+0000 7f53b6400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00ae960 con 0x7f53b00adfa0 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.631+0000 7f53b6400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b015f740 con 0x7f53b015ed80 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.631+0000 7f53b6400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b00b1200 con 0x7f5390004d00 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53a6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53b00adfa0 0x7f53b00ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53a6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53b00adfa0 0x7f53b00ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43524/0 (socket says 172.21.15.39:43524) 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53a6800640 1 -- 172.21.15.39:0/1850675695 learned_addr learned my addr 172.21.15.39:0/1850675695 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53af400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5390004d00 0x7f53b01f1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53afe00640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ed80 0x7f53b015f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53a6800640 1 -- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5390004d00 msgr2=0x7f53b01f1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53a6800640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5390004d00 0x7f53b01f1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53a6800640 1 -- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ed80 msgr2=0x7f53b015f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53a6800640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ed80 0x7f53b015f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.632+0000 7f53a6800640 1 -- 172.21.15.39:0/1850675695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b00b14e0 con 0x7f53b00adfa0 2024-01-24T17:44:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.633+0000 7f53a6800640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53b00adfa0 0x7f53b00ae420 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f53980017a0 tx=0x7f539800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:56.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.633+0000 7f53a7e00640 1 -- 172.21.15.39:0/1850675695 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980074e0 con 0x7f53b00adfa0 2024-01-24T17:44:56.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.633+0000 7f53a7e00640 1 -- 172.21.15.39:0/1850675695 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5398007640 con 0x7f53b00adfa0 2024-01-24T17:44:56.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.633+0000 7f53a7e00640 1 -- 172.21.15.39:0/1850675695 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398004a00 con 0x7f53b00adfa0 2024-01-24T17:44:56.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.633+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b00b17c0 con 0x7f53b00adfa0 2024-01-24T17:44:56.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.633+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b00b1d80 con 0x7f53b00adfa0 2024-01-24T17:44:56.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.634+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b0073ef0 con 0x7f53b00adfa0 2024-01-24T17:44:56.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.635+0000 7f53a7e00640 1 -- 172.21.15.39:0/1850675695 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5398004b60 con 0x7f53b00adfa0 2024-01-24T17:44:56.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.636+0000 7f53a7e00640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f538806a430 0x7f538806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:56.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.636+0000 7f53a7e00640 1 -- 172.21.15.39:0/1850675695 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f5398088450 con 0x7f53b00adfa0 2024-01-24T17:44:56.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.637+0000 7f53af400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f538806a430 0x7f538806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:56.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.638+0000 7f53af400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f538806a430 0x7f538806c8f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009a10 tx=0x7f53a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:56.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.641+0000 7f53a7e00640 1 -- 172.21.15.39:0/1850675695 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5398057400 con 0x7f53b00adfa0 2024-01-24T17:44:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.849+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f53b0160f30 con 0x7f53b00adfa0 2024-01-24T17:44:56.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.850+0000 7f53a7e00640 1 -- 172.21.15.39:0/1850675695 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v51) v1 ==== 80+0+57 (secure 0 0 0) 0x7f5398019050 con 0x7f53b00adfa0 2024-01-24T17:44:56.851 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.1:0/1994510967 2024-01-24T18:44:37.552044+0000 2024-01-24T17:44:56.851 INFO:teuthology.orchestra.run.smithi039.stderr:listed 1 entries 2024-01-24T17:44:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f538806a430 msgr2=0x7f538806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f538806a430 0x7f538806c8f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009a10 tx=0x7f53a0008040 comp rx=0 tx=0).stop 2024-01-24T17:44:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53b00adfa0 msgr2=0x7f53b00ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53b00adfa0 0x7f53b00ae420 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f53980017a0 tx=0x7f539800dd40 comp rx=0 tx=0).stop 2024-01-24T17:44:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 shutdown_connections 2024-01-24T17:44:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f53b015ed80 0x7f53b015f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f538806a430 0x7f538806c8f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53b00adfa0 0x7f53b00ae420 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 --2- 172.21.15.39:0/1850675695 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5390004d00 0x7f53b01f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:56.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.853+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 >> 172.21.15.39:0/1850675695 conn(0x7f53b006a3e0 msgr2=0x7f53b00ab8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:56.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.854+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 shutdown_connections 2024-01-24T17:44:56.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:56.854+0000 7f53b6400640 1 -- 172.21.15.39:0/1850675695 wait complete. 2024-01-24T17:44:56.874 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:44:56.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:56.874 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:44:56.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[56529]: fuse finished with error 0 and tester_r 0 2024-01-24T17:44:56.974 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:44:56.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:56.974 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:44:57.056 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:44:57.056 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:57.113 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:44:57.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.341+0000 7f40e3600640 1 Processor -- start 2024-01-24T17:44:57.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.341+0000 7f40e3600640 1 -- start start 2024-01-24T17:44:57.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40e3600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc0caf50 0x7f40dc0cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40e3600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f40dc0b1390 0x7f40dc0b1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40e3600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc0cb8f0 con 0x7f40dc15a950 2024-01-24T17:44:57.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40e3600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40dc0b43d0 con 0x7f40dc0b1390 2024-01-24T17:44:57.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40e3600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0c8f70 con 0x7f40dc0caf50 2024-01-24T17:44:57.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40e1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f40dc0b1390 0x7f40dc0b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40e1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f40dc0b1390 0x7f40dc0b1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43526/0 (socket says 172.21.15.39:43526) 2024-01-24T17:44:57.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40e1600640 1 -- 172.21.15.39:0/2594988211 learned_addr learned my addr 172.21.15.39:0/2594988211 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:57.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.343+0000 7f40d3e00640 1 --2- 172.21.15.39:0/2594988211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc0caf50 0x7f40dc0cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.344+0000 7f40e1600640 1 -- 172.21.15.39:0/2594988211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc0caf50 msgr2=0x7f40dc0cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:57.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.344+0000 7f40e1600640 1 --2- 172.21.15.39:0/2594988211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc0caf50 0x7f40dc0cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.344+0000 7f40e1600640 1 -- 172.21.15.39:0/2594988211 >> v1:172.21.15.100:6789/0 conn(0x7f40dc15a950 legacy=0x7f40dc0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:57.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.344+0000 7f40e1600640 1 -- 172.21.15.39:0/2594988211 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc0c9180 con 0x7f40dc0b1390 2024-01-24T17:44:57.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.344+0000 7f40d3e00640 1 --2- 172.21.15.39:0/2594988211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc0caf50 0x7f40dc0cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:57.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.345+0000 7f40e1600640 1 --2- 172.21.15.39:0/2594988211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc0b1820 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009e50 tx=0x7f40d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2c1d04f59b4a41 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:57.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.345+0000 7f40dbe00640 1 -- 172.21.15.39:0/2594988211 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d402ca70 con 0x7f40dc0b1390 2024-01-24T17:44:57.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.345+0000 7f40dbe00640 1 -- 172.21.15.39:0/2594988211 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d402cbd0 con 0x7f40dc0b1390 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.345+0000 7f40dbe00640 1 -- 172.21.15.39:0/2594988211 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d40343f0 con 0x7f40dc0b1390 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.346+0000 7f40e3600640 1 -- 172.21.15.39:0/2594988211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 msgr2=0x7f40dc0b1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.346+0000 7f40e3600640 1 --2- 172.21.15.39:0/2594988211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc0b1820 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f40d4009e50 tx=0x7f40d402bed0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.346+0000 7f40e3600640 1 -- 172.21.15.39:0/2594988211 shutdown_connections 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.346+0000 7f40e3600640 1 --2- 172.21.15.39:0/2594988211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc0b1820 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.346+0000 7f40e3600640 1 --2- 172.21.15.39:0/2594988211 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc0caf50 0x7f40dc0cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.346+0000 7f40e3600640 1 -- 172.21.15.39:0/2594988211 >> 172.21.15.39:0/2594988211 conn(0x7f40dc06a3e0 msgr2=0x7f40dc06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.346+0000 7f40e3600640 1 -- 172.21.15.39:0/2594988211 shutdown_connections 2024-01-24T17:44:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.346+0000 7f40e3600640 1 -- 172.21.15.39:0/2594988211 wait complete. 2024-01-24T17:44:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.348+0000 7f40e3600640 1 Processor -- start 2024-01-24T17:44:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.348+0000 7f40e3600640 1 -- start start 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e3600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc1584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e3600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40dc0caf50 0x7f40dc0abaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e3600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc15a950 0x7f40dc0ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e3600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc156e50 con 0x7f40dc0b1390 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e3600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0ac860 con 0x7f40dc0caf50 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e3600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc157080 con 0x7f40dc15a950 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc1584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc1584c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43538/0 (socket says 172.21.15.39:43538) 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e0c00640 1 -- 172.21.15.39:0/2572049770 learned_addr learned my addr 172.21.15.39:0/2572049770 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e0c00640 1 -- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc15a950 msgr2=0x7f40dc0ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e0c00640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc15a950 0x7f40dc0ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e0c00640 1 -- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40dc0caf50 msgr2=0x7f40dc0abaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e0c00640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40dc0caf50 0x7f40dc0abaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.349+0000 7f40e0c00640 1 -- 172.21.15.39:0/2572049770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4009b30 con 0x7f40dc0b1390 2024-01-24T17:44:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.350+0000 7f40e0c00640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc1584c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f40cc00b3f0 tx=0x7f40cc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.350+0000 7f40d9600640 1 -- 172.21.15.39:0/2572049770 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc011830 con 0x7f40dc0b1390 2024-01-24T17:44:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.350+0000 7f40d9600640 1 -- 172.21.15.39:0/2572049770 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc011990 con 0x7f40dc0b1390 2024-01-24T17:44:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.350+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc157360 con 0x7f40dc0b1390 2024-01-24T17:44:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.350+0000 7f40d9600640 1 -- 172.21.15.39:0/2572049770 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc011d40 con 0x7f40dc0b1390 2024-01-24T17:44:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.350+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc157920 con 0x7f40dc0b1390 2024-01-24T17:44:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.351+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc157b40 con 0x7f40dc0b1390 2024-01-24T17:44:57.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.352+0000 7f40d9600640 1 -- 172.21.15.39:0/2572049770 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f40cc01a400 con 0x7f40dc0b1390 2024-01-24T17:44:57.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.353+0000 7f40d9600640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f40b406a430 0x7f40b406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.353+0000 7f40d9600640 1 -- 172.21.15.39:0/2572049770 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f40cc088930 con 0x7f40dc0b1390 2024-01-24T17:44:57.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.354+0000 7f40d3e00640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f40b406a430 0x7f40b406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.355+0000 7f40d3e00640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f40b406a430 0x7f40b406c8f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f40c4006fd0 tx=0x7f40c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:57.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.357+0000 7f40d9600640 1 -- 172.21.15.39:0/2572049770 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f40cc053dc0 con 0x7f40dc0b1390 2024-01-24T17:44:57.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.558+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40dc0ac5d0 con 0x7f40dc0b1390 2024-01-24T17:44:57.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.561+0000 7f40d9600640 1 -- 172.21.15.39:0/2572049770 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16392 (secure 0 0 0) 0x7f40cc015090 con 0x7f40dc0b1390 2024-01-24T17:44:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:44:57.562 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:38.321132+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.9","mappings":[{"from":5,"to":7}]},{"pgid":"7.1d","mappings":[{"from":5,"to":7}]},{"pgid":"7.39","mappings":[{"from":5,"to":0}]},{"pgid":"7.3a","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1994510967":"2024-01-24T18:44:37.552044+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f40b406a430 msgr2=0x7f40b406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f40b406a430 0x7f40b406c8f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f40c4006fd0 tx=0x7f40c4008040 comp rx=0 tx=0).stop 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 msgr2=0x7f40dc1584c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc1584c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f40cc00b3f0 tx=0x7f40cc00b8b0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 shutdown_connections 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f40dc15a950 0x7f40dc0ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f40dc0caf50 0x7f40dc0abaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f40b406a430 0x7f40b406c8f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 --2- 172.21.15.39:0/2572049770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40dc0b1390 0x7f40dc1584c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 >> 172.21.15.39:0/2572049770 conn(0x7f40dc06a3e0 msgr2=0x7f40dc0b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 shutdown_connections 2024-01-24T17:44:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.564+0000 7f40e3600640 1 -- 172.21.15.39:0/2572049770 wait complete. 2024-01-24T17:44:57.583 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:44:07.660016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:44:08.985819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:44:57.583 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:44:57.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.821+0000 7f570be00640 1 Processor -- start 2024-01-24T17:44:57.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.821+0000 7f570be00640 1 -- start start 2024-01-24T17:44:57.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.822+0000 7f570be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f570c15fed0 0x7f570c0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.822+0000 7f570be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f570c06a380 0x7f570c0c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.822+0000 7f570be00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c071320 con 0x7f570c0cb830 2024-01-24T17:44:57.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570c071600 con 0x7f570c15fed0 2024-01-24T17:44:57.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570be00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570c0718e0 con 0x7f570c06a380 2024-01-24T17:44:57.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f570c15fed0 0x7f570c0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f570c15fed0 0x7f570c0cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43548/0 (socket says 172.21.15.39:43548) 2024-01-24T17:44:57.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570aa00640 1 -- 172.21.15.39:0/717130620 learned_addr learned my addr 172.21.15.39:0/717130620 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:57.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570b400640 1 --2- 172.21.15.39:0/717130620 >> v2:172.21.15.100:3300/0 conn(0x7f570c06a380 0x7f570c0c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570aa00640 1 -- 172.21.15.39:0/717130620 >> v2:172.21.15.100:3300/0 conn(0x7f570c06a380 msgr2=0x7f570c0c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:57.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570aa00640 1 --2- 172.21.15.39:0/717130620 >> v2:172.21.15.100:3300/0 conn(0x7f570c06a380 0x7f570c0c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570aa00640 1 -- 172.21.15.39:0/717130620 >> v1:172.21.15.100:6789/0 conn(0x7f570c0cb830 legacy=0x7f570c069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:57.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.823+0000 7f570aa00640 1 -- 172.21.15.39:0/717130620 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c071bc0 con 0x7f570c15fed0 2024-01-24T17:44:57.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.824+0000 7f570b400640 1 --2- 172.21.15.39:0/717130620 >> v2:172.21.15.100:3300/0 conn(0x7f570c06a380 0x7f570c0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:57.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.825+0000 7f570aa00640 1 --2- 172.21.15.39:0/717130620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c15fed0 0x7f570c0cdaf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f570c0c6560 tx=0x7f56fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9c74af0af2b957 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:57.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.825+0000 7f5709600640 1 -- 172.21.15.39:0/717130620 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc02ca80 con 0x7f570c15fed0 2024-01-24T17:44:57.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.825+0000 7f5709600640 1 -- 172.21.15.39:0/717130620 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc02cbe0 con 0x7f570c15fed0 2024-01-24T17:44:57.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.825+0000 7f5709600640 1 -- 172.21.15.39:0/717130620 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0343f0 con 0x7f570c15fed0 2024-01-24T17:44:57.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.826+0000 7f570be00640 1 -- 172.21.15.39:0/717130620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c15fed0 msgr2=0x7f570c0cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:57.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.826+0000 7f570be00640 1 --2- 172.21.15.39:0/717130620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c15fed0 0x7f570c0cdaf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f570c0c6560 tx=0x7f56fc02bee0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.826+0000 7f570be00640 1 -- 172.21.15.39:0/717130620 shutdown_connections 2024-01-24T17:44:57.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.826+0000 7f570be00640 1 --2- 172.21.15.39:0/717130620 >> v2:172.21.15.100:3300/0 conn(0x7f570c06a380 0x7f570c0c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.826+0000 7f570be00640 1 --2- 172.21.15.39:0/717130620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c15fed0 0x7f570c0cdaf0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.826+0000 7f570be00640 1 -- 172.21.15.39:0/717130620 >> 172.21.15.39:0/717130620 conn(0x7f570c0b10a0 msgr2=0x7f570c0b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:57.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.826+0000 7f570be00640 1 -- 172.21.15.39:0/717130620 shutdown_connections 2024-01-24T17:44:57.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.826+0000 7f570be00640 1 -- 172.21.15.39:0/717130620 wait complete. 2024-01-24T17:44:57.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.829+0000 7f570be00640 1 Processor -- start 2024-01-24T17:44:57.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.829+0000 7f570be00640 1 -- start start 2024-01-24T17:44:57.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c06a380 0x7f570c1ef930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f570c0cb830 0x7f570c1f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f570c15fed0 0x7f570c1f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c1efe70 con 0x7f570c06a380 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c1f6240 con 0x7f570c15fed0 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570c1f6520 con 0x7f570c0cb830 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c06a380 0x7f570c1ef930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c06a380 0x7f570c1ef930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43550/0 (socket says 172.21.15.39:43550) 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570aa00640 1 -- 172.21.15.39:0/2047418936 learned_addr learned my addr 172.21.15.39:0/2047418936 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.830+0000 7f570a000640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f570c0cb830 0x7f570c1f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570aa00640 1 -- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f570c0cb830 msgr2=0x7f570c1f59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570b400640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f570c15fed0 0x7f570c1f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570aa00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f570c0cb830 0x7f570c1f59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570aa00640 1 -- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f570c15fed0 msgr2=0x7f570c1f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570aa00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f570c15fed0 0x7f570c1f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:57.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570aa00640 1 -- 172.21.15.39:0/2047418936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc009b30 con 0x7f570c06a380 2024-01-24T17:44:57.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570b400640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f570c15fed0 0x7f570c1f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:57.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570aa00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c06a380 0x7f570c1ef930 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f56fc000f80 tx=0x7f56fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:57.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f5702a00640 1 -- 172.21.15.39:0/2047418936 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc03e450 con 0x7f570c06a380 2024-01-24T17:44:57.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f5702a00640 1 -- 172.21.15.39:0/2047418936 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc03e5f0 con 0x7f570c06a380 2024-01-24T17:44:57.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c1f6800 con 0x7f570c06a380 2024-01-24T17:44:57.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.831+0000 7f5702a00640 1 -- 172.21.15.39:0/2047418936 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc03e9a0 con 0x7f570c06a380 2024-01-24T17:44:57.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.832+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c1f2400 con 0x7f570c06a380 2024-01-24T17:44:57.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.833+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c1f6990 con 0x7f570c06a380 2024-01-24T17:44:57.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.834+0000 7f5702a00640 1 -- 172.21.15.39:0/2047418936 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f56fc02cd50 con 0x7f570c06a380 2024-01-24T17:44:57.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.835+0000 7f5702a00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f56e406a430 0x7f56e406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:57.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.835+0000 7f5702a00640 1 -- 172.21.15.39:0/2047418936 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f56fc0ab5f0 con 0x7f570c06a380 2024-01-24T17:44:57.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.836+0000 7f570a000640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f56e406a430 0x7f56e406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:57.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.837+0000 7f570a000640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f56e406a430 0x7f56e406c8f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f570c0c29e0 tx=0x7f56f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:57.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:57.839+0000 7f5702a00640 1 -- 172.21.15.39:0/2047418936 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f56fc076b10 con 0x7f570c06a380 2024-01-24T17:44:58.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.048+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f570c067040 con 0x7f570c06a380 2024-01-24T17:44:58.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.049+0000 7f5702a00640 1 -- 172.21.15.39:0/2047418936 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f56fc07a130 con 0x7f570c06a380 2024-01-24T17:44:58.050 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.051+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f56e406a430 msgr2=0x7f56e406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.051+0000 7f570be00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f56e406a430 0x7f56e406c8f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f570c0c29e0 tx=0x7f56f4009040 comp rx=0 tx=0).stop 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c06a380 msgr2=0x7f570c1ef930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c06a380 0x7f570c1ef930 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f56fc000f80 tx=0x7f56fc004a40 comp rx=0 tx=0).stop 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 shutdown_connections 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f570c15fed0 0x7f570c1f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f570c0cb830 0x7f570c1f59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f56e406a430 0x7f56e406c8f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 --2- 172.21.15.39:0/2047418936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f570c06a380 0x7f570c1ef930 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 >> 172.21.15.39:0/2047418936 conn(0x7f570c0b10a0 msgr2=0x7f570c0cf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 shutdown_connections 2024-01-24T17:44:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.052+0000 7f570be00640 1 -- 172.21.15.39:0/2047418936 wait complete. 2024-01-24T17:44:58.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:44:58.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.293+0000 7f29eda00640 1 Processor -- start 2024-01-24T17:44:58.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.293+0000 7f29eda00640 1 -- start start 2024-01-24T17:44:58.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.295+0000 7f29eda00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f29e815a830 0x7f29e8071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.295+0000 7f29eda00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f29e806a380 0x7f29e80b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.295+0000 7f29eda00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e8071370 con 0x7f29e80b4b30 2024-01-24T17:44:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.295+0000 7f29eda00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e80ccea0 con 0x7f29e815a830 2024-01-24T17:44:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.295+0000 7f29eda00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e80cc8e0 con 0x7f29e806a380 2024-01-24T17:44:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.295+0000 7f29e7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f29e815a830 0x7f29e8071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.295+0000 7f29e7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f29e815a830 0x7f29e8071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43552/0 (socket says 172.21.15.39:43552) 2024-01-24T17:44:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.295+0000 7f29e7e00640 1 -- 172.21.15.39:0/754702001 learned_addr learned my addr 172.21.15.39:0/754702001 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.296+0000 7f29e7e00640 1 -- 172.21.15.39:0/754702001 >> v2:172.21.15.100:3300/0 conn(0x7f29e806a380 msgr2=0x7f29e80b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.296+0000 7f29e7e00640 1 --2- 172.21.15.39:0/754702001 >> v2:172.21.15.100:3300/0 conn(0x7f29e806a380 0x7f29e80b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.296+0000 7f29e7e00640 1 -- 172.21.15.39:0/754702001 >> v1:172.21.15.100:6789/0 conn(0x7f29e80b4b30 legacy=0x7f29e8069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.296+0000 7f29e7e00640 1 -- 172.21.15.39:0/754702001 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e80ccbc0 con 0x7f29e815a830 2024-01-24T17:44:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.297+0000 7f29e7e00640 1 --2- 172.21.15.39:0/754702001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8071800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f29e8072430 tx=0x7f29d802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42010adb9cdcc040 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:58.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.297+0000 7f29e6a00640 1 -- 172.21.15.39:0/754702001 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d802ca80 con 0x7f29e815a830 2024-01-24T17:44:58.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.297+0000 7f29e6a00640 1 -- 172.21.15.39:0/754702001 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d802cbe0 con 0x7f29e815a830 2024-01-24T17:44:58.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.297+0000 7f29e6a00640 1 -- 172.21.15.39:0/754702001 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d80343f0 con 0x7f29e815a830 2024-01-24T17:44:58.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.298+0000 7f29eda00640 1 -- 172.21.15.39:0/754702001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 msgr2=0x7f29e8071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.298+0000 7f29eda00640 1 --2- 172.21.15.39:0/754702001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8071800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f29e8072430 tx=0x7f29d802bee0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.298+0000 7f29eda00640 1 -- 172.21.15.39:0/754702001 shutdown_connections 2024-01-24T17:44:58.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.298+0000 7f29eda00640 1 --2- 172.21.15.39:0/754702001 >> v2:172.21.15.100:3300/0 conn(0x7f29e806a380 0x7f29e80b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.298+0000 7f29eda00640 1 --2- 172.21.15.39:0/754702001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8071800 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.298+0000 7f29eda00640 1 -- 172.21.15.39:0/754702001 >> 172.21.15.39:0/754702001 conn(0x7f29e80b10a0 msgr2=0x7f29e80b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:58.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.298+0000 7f29eda00640 1 -- 172.21.15.39:0/754702001 shutdown_connections 2024-01-24T17:44:58.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.298+0000 7f29eda00640 1 -- 172.21.15.39:0/754702001 wait complete. 2024-01-24T17:44:58.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.300+0000 7f29eda00640 1 Processor -- start 2024-01-24T17:44:58.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.300+0000 7f29eda00640 1 -- start start 2024-01-24T17:44:58.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29eda00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e806a380 0x7f29e80addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29eda00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e80b4b30 0x7f29e80c3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29eda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29eda00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e815be10 con 0x7f29e815a830 2024-01-24T17:44:58.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29eda00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8158d80 con 0x7f29e80b4b30 2024-01-24T17:44:58.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29eda00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e815aeb0 con 0x7f29e806a380 2024-01-24T17:44:58.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29ed000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29ed000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43558/0 (socket says 172.21.15.39:43558) 2024-01-24T17:44:58.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.301+0000 7f29ed000640 1 -- 172.21.15.39:0/3874000600 learned_addr learned my addr 172.21.15.39:0/3874000600 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:58.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29e7400640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e80b4b30 0x7f29e80c3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29e7e00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e806a380 0x7f29e80addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29ed000640 1 -- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e806a380 msgr2=0x7f29e80addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29ed000640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e806a380 0x7f29e80addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29ed000640 1 -- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e80b4b30 msgr2=0x7f29e80c3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29ed000640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e80b4b30 0x7f29e80c3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29ed000640 1 -- 172.21.15.39:0/3874000600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d8009b30 con 0x7f29e815a830 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29ed000640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8151430 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f29e000b410 tx=0x7f29e000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.302+0000 7f29dfe00640 1 -- 172.21.15.39:0/3874000600 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0005970 con 0x7f29e815a830 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.303+0000 7f29dfe00640 1 -- 172.21.15.39:0/3874000600 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e0005af0 con 0x7f29e815a830 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.303+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e815b190 con 0x7f29e815a830 2024-01-24T17:44:58.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.303+0000 7f29dfe00640 1 -- 172.21.15.39:0/3874000600 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0005ea0 con 0x7f29e815a830 2024-01-24T17:44:58.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.303+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e815b750 con 0x7f29e815a830 2024-01-24T17:44:58.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.304+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e815bf50 con 0x7f29e815a830 2024-01-24T17:44:58.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.305+0000 7f29dfe00640 1 -- 172.21.15.39:0/3874000600 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f29e0006000 con 0x7f29e815a830 2024-01-24T17:44:58.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.306+0000 7f29dfe00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29c006a430 0x7f29c006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.306+0000 7f29dfe00640 1 -- 172.21.15.39:0/3874000600 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f29e00886d0 con 0x7f29e815a830 2024-01-24T17:44:58.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.306+0000 7f29e7e00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29c006a430 0x7f29c006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.307+0000 7f29e7e00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29c006a430 0x7f29c006c8f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f29d8035000 tx=0x7f29d8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:58.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.310+0000 7f29dfe00640 1 -- 172.21.15.39:0/3874000600 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f29e815bf50 con 0x7f29e815a830 2024-01-24T17:44:58.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.502+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f29e815bf50 con 0x7f29e815a830 2024-01-24T17:44:58.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.503+0000 7f29dfe00640 1 -- 172.21.15.39:0/3874000600 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29e815bf50 con 0x7f29e815a830 2024-01-24T17:44:58.504 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:44:58.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29c006a430 msgr2=0x7f29c006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29c006a430 0x7f29c006c8f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f29d8035000 tx=0x7f29d8030000 comp rx=0 tx=0).stop 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 msgr2=0x7f29e8151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8151430 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f29e000b410 tx=0x7f29e000b8d0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 shutdown_connections 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f29c006a430 0x7f29c006c8f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29e815a830 0x7f29e8151430 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29e80b4b30 0x7f29e80c3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.505+0000 7f29eda00640 1 --2- 172.21.15.39:0/3874000600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29e806a380 0x7f29e80addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.506+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 >> 172.21.15.39:0/3874000600 conn(0x7f29e80b10a0 msgr2=0x7f29e80b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.506+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 shutdown_connections 2024-01-24T17:44:58.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.506+0000 7f29eda00640 1 -- 172.21.15.39:0/3874000600 wait complete. 2024-01-24T17:44:58.526 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:44:58.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.748+0000 7f5b1fe00640 1 Processor -- start 2024-01-24T17:44:58.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.748+0000 7f5b1fe00640 1 -- start start 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.749+0000 7f5b1fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5b200ce390 0x7f5b200ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.749+0000 7f5b1fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5b200c8d50 0x7f5b200c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.749+0000 7f5b1fe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2015d090 con 0x7f5b2015ccb0 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1fe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b200cac20 con 0x7f5b200c8d50 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1fe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b200ced60 con 0x7f5b200ce390 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5b200c8d50 0x7f5b200c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5b200c8d50 0x7f5b200c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43566/0 (socket says 172.21.15.39:43566) 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1f400640 1 -- 172.21.15.39:0/1480081531 learned_addr learned my addr 172.21.15.39:0/1480081531 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1f400640 1 -- 172.21.15.39:0/1480081531 >> v2:172.21.15.100:3300/0 conn(0x7f5b200ce390 msgr2=0x7f5b200ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1f400640 1 --2- 172.21.15.39:0/1480081531 >> v2:172.21.15.100:3300/0 conn(0x7f5b200ce390 0x7f5b200ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1f400640 1 -- 172.21.15.39:0/1480081531 >> v1:172.21.15.100:6789/0 conn(0x7f5b2015ccb0 legacy=0x7f5b200cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:58.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.750+0000 7f5b1f400640 1 -- 172.21.15.39:0/1480081531 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b200ce1a0 con 0x7f5b200c8d50 2024-01-24T17:44:58.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.751+0000 7f5b1f400640 1 --2- 172.21.15.39:0/1480081531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b200c8d50 0x7f5b200c91a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5b2006b7f0 tx=0x7f5b1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f493ddf85d582058 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:58.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1e000640 1 -- 172.21.15.39:0/1480081531 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1802ca80 con 0x7f5b200c8d50 2024-01-24T17:44:58.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1e000640 1 -- 172.21.15.39:0/1480081531 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1802cbe0 con 0x7f5b200c8d50 2024-01-24T17:44:58.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1e000640 1 -- 172.21.15.39:0/1480081531 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180343f0 con 0x7f5b200c8d50 2024-01-24T17:44:58.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1fe00640 1 -- 172.21.15.39:0/1480081531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b200c8d50 msgr2=0x7f5b200c91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/1480081531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b200c8d50 0x7f5b200c91a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5b2006b7f0 tx=0x7f5b1802bee0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1fe00640 1 -- 172.21.15.39:0/1480081531 shutdown_connections 2024-01-24T17:44:58.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/1480081531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b200c8d50 0x7f5b200c91a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/1480081531 >> v2:172.21.15.100:3300/0 conn(0x7f5b200ce390 0x7f5b200ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1fe00640 1 -- 172.21.15.39:0/1480081531 >> 172.21.15.39:0/1480081531 conn(0x7f5b20069e50 msgr2=0x7f5b2006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:58.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.752+0000 7f5b1fe00640 1 -- 172.21.15.39:0/1480081531 shutdown_connections 2024-01-24T17:44:58.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.753+0000 7f5b1fe00640 1 -- 172.21.15.39:0/1480081531 wait complete. 2024-01-24T17:44:58.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.754+0000 7f5b1fe00640 1 Processor -- start 2024-01-24T17:44:58.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.754+0000 7f5b1fe00640 1 -- start start 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.755+0000 7f5b1fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b200c8d50 0x7f5b201499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.755+0000 7f5b1fe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b200ce390 0x7f5b2009b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.755+0000 7f5b1fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b2015ccb0 0x7f5b200ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.755+0000 7f5b1fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200c1b50 con 0x7f5b2015ccb0 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.755+0000 7f5b1fe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b20149ee0 con 0x7f5b200ce390 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.755+0000 7f5b1fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2014c0d0 con 0x7f5b200c8d50 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.755+0000 7f5b1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b2015ccb0 0x7f5b200ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b2015ccb0 0x7f5b200ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43582/0 (socket says 172.21.15.39:43582) 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1f400640 1 -- 172.21.15.39:0/3328857009 learned_addr learned my addr 172.21.15.39:0/3328857009 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1ea00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b200c8d50 0x7f5b201499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1f400640 1 -- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b200c8d50 msgr2=0x7f5b201499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b15e00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b200ce390 0x7f5b2009b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1f400640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b200c8d50 0x7f5b201499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1f400640 1 -- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b200ce390 msgr2=0x7f5b2009b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1f400640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b200ce390 0x7f5b2009b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1f400640 1 -- 172.21.15.39:0/3328857009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18009b30 con 0x7f5b2015ccb0 2024-01-24T17:44:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1ea00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b200c8d50 0x7f5b201499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b15e00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b200ce390 0x7f5b2009b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.756+0000 7f5b1f400640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b2015ccb0 0x7f5b200ae260 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5b18036040 tx=0x7f5b18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.757+0000 7f5b17400640 1 -- 172.21.15.39:0/3328857009 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1803e450 con 0x7f5b2015ccb0 2024-01-24T17:44:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.757+0000 7f5b17400640 1 -- 172.21.15.39:0/3328857009 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1803e5d0 con 0x7f5b2015ccb0 2024-01-24T17:44:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.757+0000 7f5b17400640 1 -- 172.21.15.39:0/3328857009 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1803e980 con 0x7f5b2015ccb0 2024-01-24T17:44:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.757+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2014b460 con 0x7f5b2015ccb0 2024-01-24T17:44:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.757+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2014ba20 con 0x7f5b2015ccb0 2024-01-24T17:44:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.758+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2009afa0 con 0x7f5b2015ccb0 2024-01-24T17:44:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.759+0000 7f5b17400640 1 -- 172.21.15.39:0/3328857009 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5b1802cd50 con 0x7f5b2015ccb0 2024-01-24T17:44:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.760+0000 7f5b17400640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5af806a430 0x7f5af806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.760+0000 7f5b17400640 1 -- 172.21.15.39:0/3328857009 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f5b180ab640 con 0x7f5b2015ccb0 2024-01-24T17:44:58.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.761+0000 7f5b1ea00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5af806a430 0x7f5af806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:58.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.762+0000 7f5b1ea00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5af806a430 0x7f5af806c8f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5b200af440 tx=0x7f5b10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:58.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.764+0000 7f5b17400640 1 -- 172.21.15.39:0/3328857009 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5b18076b60 con 0x7f5b2015ccb0 2024-01-24T17:44:58.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.955+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b200cea30 con 0x7f5b2015ccb0 2024-01-24T17:44:58.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.956+0000 7f5b17400640 1 -- 172.21.15.39:0/3328857009 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b1807a180 con 0x7f5b2015ccb0 2024-01-24T17:44:58.957 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:44:58.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.958+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5af806a430 msgr2=0x7f5af806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.958+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5af806a430 0x7f5af806c8f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5b200af440 tx=0x7f5b10008040 comp rx=0 tx=0).stop 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.958+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b2015ccb0 msgr2=0x7f5b200ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.958+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b2015ccb0 0x7f5b200ae260 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5b18036040 tx=0x7f5b18004a40 comp rx=0 tx=0).stop 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.959+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 shutdown_connections 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.959+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5af806a430 0x7f5af806c8f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.959+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b2015ccb0 0x7f5b200ae260 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.959+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b200ce390 0x7f5b2009b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.959+0000 7f5b1fe00640 1 --2- 172.21.15.39:0/3328857009 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b200c8d50 0x7f5b201499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.959+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 >> 172.21.15.39:0/3328857009 conn(0x7f5b20069e50 msgr2=0x7f5b200ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.959+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 shutdown_connections 2024-01-24T17:44:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:58.959+0000 7f5b1fe00640 1 -- 172.21.15.39:0/3328857009 wait complete. 2024-01-24T17:44:58.979 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:44:59.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.200+0000 7fdca5600640 1 Processor -- start 2024-01-24T17:44:59.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.201+0000 7fdca5600640 1 -- start start 2024-01-24T17:44:59.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.202+0000 7fdca5600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:59.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.202+0000 7fdca5600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdca015ec00 0x7fdca00b0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:59.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.202+0000 7fdca5600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca00cd750 con 0x7fdca00d07c0 2024-01-24T17:44:59.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.202+0000 7fdca5600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca015e490 con 0x7fdca015ec00 2024-01-24T17:44:59.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.202+0000 7fdca5600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca00cf8a0 con 0x7fdca00ce400 2024-01-24T17:44:59.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.203+0000 7fdc9f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdca015ec00 0x7fdca00b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:59.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.203+0000 7fdc9f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdca015ec00 0x7fdca00b0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43592/0 (socket says 172.21.15.39:43592) 2024-01-24T17:44:59.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.203+0000 7fdc9f400640 1 -- 172.21.15.39:0/1424149210 learned_addr learned my addr 172.21.15.39:0/1424149210 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:59.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.203+0000 7fdc97e00640 1 --2- 172.21.15.39:0/1424149210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:59.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.203+0000 7fdc9f400640 1 -- 172.21.15.39:0/1424149210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 msgr2=0x7fdca00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:44:59.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.203+0000 7fdc9f400640 1 --2- 172.21.15.39:0/1424149210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.203+0000 7fdc9f400640 1 -- 172.21.15.39:0/1424149210 >> v1:172.21.15.100:6789/0 conn(0x7fdca00d07c0 legacy=0x7fdca00d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:59.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.203+0000 7fdc9f400640 1 -- 172.21.15.39:0/1424149210 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca0158990 con 0x7fdca015ec00 2024-01-24T17:44:59.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.204+0000 7fdc9f400640 1 --2- 172.21.15.39:0/1424149210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca015ec00 0x7fdca00b0780 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009e50 tx=0x7fdc9802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e2d459703df728 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:59.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdc9e000640 1 -- 172.21.15.39:0/1424149210 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9802cb90 con 0x7fdca015ec00 2024-01-24T17:44:59.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdc9e000640 1 -- 172.21.15.39:0/1424149210 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9802ccf0 con 0x7fdca015ec00 2024-01-24T17:44:59.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdc9e000640 1 -- 172.21.15.39:0/1424149210 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc98034610 con 0x7fdca015ec00 2024-01-24T17:44:59.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdca5600640 1 -- 172.21.15.39:0/1424149210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca015ec00 msgr2=0x7fdca00b0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:59.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdca5600640 1 --2- 172.21.15.39:0/1424149210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca015ec00 0x7fdca00b0780 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009e50 tx=0x7fdc9802bff0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdca5600640 1 -- 172.21.15.39:0/1424149210 shutdown_connections 2024-01-24T17:44:59.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdca5600640 1 --2- 172.21.15.39:0/1424149210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca015ec00 0x7fdca00b0780 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdca5600640 1 --2- 172.21.15.39:0/1424149210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.205+0000 7fdca5600640 1 -- 172.21.15.39:0/1424149210 >> 172.21.15.39:0/1424149210 conn(0x7fdca0089fd0 msgr2=0x7fdca0099b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:59.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.206+0000 7fdca5600640 1 -- 172.21.15.39:0/1424149210 shutdown_connections 2024-01-24T17:44:59.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.206+0000 7fdca5600640 1 -- 172.21.15.39:0/1424149210 wait complete. 2024-01-24T17:44:59.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.207+0000 7fdca5600640 1 Processor -- start 2024-01-24T17:44:59.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.208+0000 7fdca5600640 1 -- start start 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.208+0000 7fdca5600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca0157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdca5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca00d07c0 0x7fdca015a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdca5600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdca015ec00 0x7fdca0158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdca5600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca0159fc0 con 0x7fdca00d07c0 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdca5600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca01590c0 con 0x7fdca015ec00 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdca5600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca0159340 con 0x7fdca00ce400 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc97e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca00d07c0 0x7fdca015a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc97e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca00d07c0 0x7fdca015a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43598/0 (socket says 172.21.15.39:43598) 2024-01-24T17:44:59.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc97e00640 1 -- 172.21.15.39:0/2501048358 learned_addr learned my addr 172.21.15.39:0/2501048358 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc9ea00640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca0157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc9f400640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdca015ec00 0x7fdca0158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc97e00640 1 -- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 msgr2=0x7fdca0157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc97e00640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca0157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc97e00640 1 -- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdca015ec00 msgr2=0x7fdca0158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc97e00640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdca015ec00 0x7fdca0158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc97e00640 1 -- 172.21.15.39:0/2501048358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc98009b30 con 0x7fdca00d07c0 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.209+0000 7fdc9ea00640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca0157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.210+0000 7fdc97e00640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca00d07c0 0x7fdca015a3e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800d7d0 tx=0x7fdc8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.210+0000 7fdc96a00640 1 -- 172.21.15.39:0/2501048358 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88018070 con 0x7fdca00d07c0 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.210+0000 7fdc96a00640 1 -- 172.21.15.39:0/2501048358 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88004020 con 0x7fdca00d07c0 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.210+0000 7fdc96a00640 1 -- 172.21.15.39:0/2501048358 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc880041b0 con 0x7fdca00d07c0 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.210+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca0159620 con 0x7fdca00d07c0 2024-01-24T17:44:59.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.210+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca0159be0 con 0x7fdca00d07c0 2024-01-24T17:44:59.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.211+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca015a100 con 0x7fdca00d07c0 2024-01-24T17:44:59.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.212+0000 7fdc96a00640 1 -- 172.21.15.39:0/2501048358 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdc88004310 con 0x7fdca00d07c0 2024-01-24T17:44:59.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.213+0000 7fdc96a00640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc7806a460 0x7fdc7806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:44:59.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.213+0000 7fdc96a00640 1 -- 172.21.15.39:0/2501048358 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fdc88088880 con 0x7fdca00d07c0 2024-01-24T17:44:59.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.214+0000 7fdc9ea00640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc7806a460 0x7fdc7806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:44:59.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.215+0000 7fdc9ea00640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc7806a460 0x7fdc7806c920 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdca00afe00 tx=0x7fdc90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:44:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.217+0000 7fdc96a00640 1 -- 172.21.15.39:0/2501048358 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdca015a100 con 0x7fdca00d07c0 2024-01-24T17:44:59.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.424+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdca015a100 con 0x7fdca00d07c0 2024-01-24T17:44:59.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.425+0000 7fdc96a00640 1 -- 172.21.15.39:0/2501048358 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 1 entries v51) v1 ==== 80+0+57 (secure 0 0 0) 0x7fdca015a100 con 0x7fdca00d07c0 2024-01-24T17:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.1:0/1994510967 2024-01-24T18:44:37.552044+0000 2024-01-24T17:44:59.426 INFO:teuthology.orchestra.run.smithi039.stderr:listed 1 entries 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc7806a460 msgr2=0x7fdc7806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc7806a460 0x7fdc7806c920 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdca00afe00 tx=0x7fdc90009040 comp rx=0 tx=0).stop 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca00d07c0 msgr2=0x7fdca015a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca00d07c0 0x7fdca015a3e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800d7d0 tx=0x7fdc8800dc90 comp rx=0 tx=0).stop 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 shutdown_connections 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdca015ec00 0x7fdca0158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdc7806a460 0x7fdc7806c920 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdca00d07c0 0x7fdca015a3e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 --2- 172.21.15.39:0/2501048358 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdca00ce400 0x7fdca0157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 >> 172.21.15.39:0/2501048358 conn(0x7fdca0089fd0 msgr2=0x7fdca00ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 shutdown_connections 2024-01-24T17:44:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:44:59.428+0000 7fdca5600640 1 -- 172.21.15.39:0/2501048358 wait complete. 2024-01-24T17:44:59.449 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:44:59.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:59.449 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:44:59.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[44571]: fuse finished with error 0 and tester_r 0 2024-01-24T17:44:59.550 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:44:59.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:59.550 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:44:59.627 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:44:59.627 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:59.687 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:44:59.687 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:44:59.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:59.687 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-24T17:44:59.743 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-24T17:44:59.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:44:59.745 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:44:59.745 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:44:59.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:44:59.803 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:44:59.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:44:59.803 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-24T17:44:59.853 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-24T17:44:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:44:59.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:45:00.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.084+0000 7f826b600640 1 Processor -- start 2024-01-24T17:45:00.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.084+0000 7f826b600640 1 -- start start 2024-01-24T17:45:00.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.085+0000 7f826b600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f826415a950 0x7f82640ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.085+0000 7f826b600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82640c9180 con 0x7f82640cb660 2024-01-24T17:45:00.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.085+0000 7f826b600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82640c9460 con 0x7f82640caaa0 2024-01-24T17:45:00.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.085+0000 7f826b600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82640b43d0 con 0x7f826415a950 2024-01-24T17:45:00.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.086+0000 7f8268c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f826415a950 0x7f82640ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.086+0000 7f8268c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f826415a950 0x7f82640ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43614/0 (socket says 172.21.15.39:43614) 2024-01-24T17:45:00.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.086+0000 7f8268c00640 1 -- 172.21.15.39:0/1153592275 learned_addr learned my addr 172.21.15.39:0/1153592275 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:00.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.086+0000 7f8269600640 1 --1- 172.21.15.39:0/1153592275 >> v1:172.21.15.39:6789/0 conn(0x7f82640cb660 0x7f82640b1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50648/0 (socket says 172.21.15.39:50648) 2024-01-24T17:45:00.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.086+0000 7f8268c00640 1 -- 172.21.15.39:0/1153592275 >> v1:172.21.15.100:6789/0 conn(0x7f82640caaa0 legacy=0x7f82640caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:00.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.086+0000 7f8268c00640 1 -- 172.21.15.39:0/1153592275 >> v1:172.21.15.39:6789/0 conn(0x7f82640cb660 legacy=0x7f82640b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.086+0000 7f8268c00640 1 -- 172.21.15.39:0/1153592275 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82640c9740 con 0x7f826415a950 2024-01-24T17:45:00.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.087+0000 7f8268c00640 1 --2- 172.21.15.39:0/1153592275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f826415a950 0x7f82640ca560 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8254009870 tx=0x7f825402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa8e4fc62012c8d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:00.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.088+0000 7f8263400640 1 -- 172.21.15.39:0/1153592275 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254030860 con 0x7f826415a950 2024-01-24T17:45:00.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.088+0000 7f8263400640 1 -- 172.21.15.39:0/1153592275 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82540309c0 con 0x7f826415a950 2024-01-24T17:45:00.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.088+0000 7f8263400640 1 -- 172.21.15.39:0/1153592275 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254030d50 con 0x7f826415a950 2024-01-24T17:45:00.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.088+0000 7f826b600640 1 -- 172.21.15.39:0/1153592275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f826415a950 msgr2=0x7f82640ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.088+0000 7f826b600640 1 --2- 172.21.15.39:0/1153592275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f826415a950 0x7f82640ca560 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8254009870 tx=0x7f825402b9a0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.088+0000 7f826b600640 1 -- 172.21.15.39:0/1153592275 shutdown_connections 2024-01-24T17:45:00.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.088+0000 7f826b600640 1 --2- 172.21.15.39:0/1153592275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f826415a950 0x7f82640ca560 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.088+0000 7f826b600640 1 -- 172.21.15.39:0/1153592275 >> 172.21.15.39:0/1153592275 conn(0x7f826406a3e0 msgr2=0x7f826406bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:00.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.089+0000 7f826b600640 1 -- 172.21.15.39:0/1153592275 shutdown_connections 2024-01-24T17:45:00.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.089+0000 7f826b600640 1 -- 172.21.15.39:0/1153592275 wait complete. 2024-01-24T17:45:00.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.090+0000 7f826b600640 1 Processor -- start 2024-01-24T17:45:00.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.090+0000 7f826b600640 1 -- start start 2024-01-24T17:45:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.091+0000 7f826b600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82640caaa0 0x7f82641f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.091+0000 7f826b600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82640cb660 0x7f82641ed530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.091+0000 7f826b600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f826415a950 0x7f8264159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.091+0000 7f826b600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82641466a0 con 0x7f82640caaa0 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.091+0000 7f826b600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826415d930 con 0x7f826415a950 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.091+0000 7f826b600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8264148ea0 con 0x7f82640cb660 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82640caaa0 0x7f82641f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82640caaa0 0x7f82641f1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43626/0 (socket says 172.21.15.39:43626) 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 -- 172.21.15.39:0/4031838393 learned_addr learned my addr 172.21.15.39:0/4031838393 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 -- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82640cb660 msgr2=0x7f82641ed530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82640cb660 0x7f82641ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 -- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f826415a950 msgr2=0x7f8264159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f826415a950 0x7f8264159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 -- 172.21.15.39:0/4031838393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82540094d0 con 0x7f82640caaa0 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.092+0000 7f8268c00640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82640caaa0 0x7f82641f1d10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8254009870 tx=0x7f8254004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.093+0000 7f8260c00640 1 -- 172.21.15.39:0/4031838393 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254030810 con 0x7f82640caaa0 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.093+0000 7f8260c00640 1 -- 172.21.15.39:0/4031838393 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8254039d90 con 0x7f82640caaa0 2024-01-24T17:45:00.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.093+0000 7f8260c00640 1 -- 172.21.15.39:0/4031838393 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254043630 con 0x7f82640caaa0 2024-01-24T17:45:00.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.093+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8264149180 con 0x7f82640caaa0 2024-01-24T17:45:00.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.093+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8264149740 con 0x7f82640caaa0 2024-01-24T17:45:00.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.094+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82640704d0 con 0x7f82640caaa0 2024-01-24T17:45:00.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.095+0000 7f8260c00640 1 -- 172.21.15.39:0/4031838393 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f825402f3b0 con 0x7f82640caaa0 2024-01-24T17:45:00.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.096+0000 7f8260c00640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f823c06a430 0x7f823c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.096+0000 7f8260c00640 1 -- 172.21.15.39:0/4031838393 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f82540aade0 con 0x7f82640caaa0 2024-01-24T17:45:00.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.097+0000 7f8263e00640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f823c06a430 0x7f823c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.098+0000 7f8263e00640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f823c06a430 0x7f823c06c8f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f82640c86a0 tx=0x7f824c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.100+0000 7f8260c00640 1 -- 172.21.15.39:0/4031838393 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8254076310 con 0x7f82640caaa0 2024-01-24T17:45:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.334+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82641498d0 con 0x7f82640caaa0 2024-01-24T17:45:00.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.335+0000 7f8260c00640 1 -- 172.21.15.39:0/4031838393 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4755 (secure 0 0 0) 0x7f8254034090 con 0x7f82640caaa0 2024-01-24T17:45:00.337 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:00.338 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:44:09.961260+0000","modified":"2024-01-24T17:44:34.305760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5769,"mds_1":5763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5763":{"gid":5763,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.100:6837/1418568795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1418568795},{"type":"v1","addr":"172.21.15.100:6837","nonce":1418568795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5769":{"gid":5769,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/1882733155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1882733155},{"type":"v1","addr":"172.21.15.39:6835","nonce":1882733155}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-24T17:45:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-01-24T17:45:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f823c06a430 msgr2=0x7f823c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f823c06a430 0x7f823c06c8f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f82640c86a0 tx=0x7f824c0079b0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82640caaa0 msgr2=0x7f82641f1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82640caaa0 0x7f82641f1d10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8254009870 tx=0x7f8254004730 comp rx=0 tx=0).stop 2024-01-24T17:45:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 shutdown_connections 2024-01-24T17:45:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f826415a950 0x7f8264159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82640cb660 0x7f82641ed530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f823c06a430 0x7f823c06c8f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 --2- 172.21.15.39:0/4031838393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82640caaa0 0x7f82641f1d10 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 >> 172.21.15.39:0/4031838393 conn(0x7f826406a3e0 msgr2=0x7f82640b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 shutdown_connections 2024-01-24T17:45:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.338+0000 7f826b600640 1 -- 172.21.15.39:0/4031838393 wait complete. 2024-01-24T17:45:00.358 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:00.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.581+0000 7fedf7a00640 1 Processor -- start 2024-01-24T17:45:00.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.581+0000 7fedf7a00640 1 -- start start 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.583+0000 7fedf7a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fedf00cf990 0x7fedf00cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.583+0000 7fedf7a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 0x7fedf015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.583+0000 7fedf7a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf00d02b0 con 0x7fedf0160610 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.583+0000 7fedf7a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf015e080 con 0x7fedf00cf990 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.583+0000 7fedf7a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf015dac0 con 0x7fedf015d020 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.583+0000 7fedf5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fedf00cf990 0x7fedf00cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.583+0000 7fedf5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fedf00cf990 0x7fedf00cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43632/0 (socket says 172.21.15.39:43632) 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.583+0000 7fedf5000640 1 -- 172.21.15.39:0/3154499321 learned_addr learned my addr 172.21.15.39:0/3154499321 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:00.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.584+0000 7fede7c00640 1 --2- 172.21.15.39:0/3154499321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 0x7fedf015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.584+0000 7fedf5000640 1 -- 172.21.15.39:0/3154499321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 msgr2=0x7fedf015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:00.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.584+0000 7fedf5000640 1 --2- 172.21.15.39:0/3154499321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 0x7fedf015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.584+0000 7fedf5000640 1 -- 172.21.15.39:0/3154499321 >> v1:172.21.15.100:6789/0 conn(0x7fedf0160610 legacy=0x7fedf0160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.584+0000 7fedf5000640 1 -- 172.21.15.39:0/3154499321 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf015dda0 con 0x7fedf00cf990 2024-01-24T17:45:00.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.586+0000 7fedf5000640 1 --2- 172.21.15.39:0/3154499321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf00cf990 0x7fedf00cfd70 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fede0009870 tx=0x7fede002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7609bc971d551d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:00.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.586+0000 7fedefe00640 1 -- 172.21.15.39:0/3154499321 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0030840 con 0x7fedf00cf990 2024-01-24T17:45:00.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.586+0000 7fedefe00640 1 -- 172.21.15.39:0/3154499321 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede00309a0 con 0x7fedf00cf990 2024-01-24T17:45:00.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.586+0000 7fedefe00640 1 -- 172.21.15.39:0/3154499321 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede0030d30 con 0x7fedf00cf990 2024-01-24T17:45:00.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.586+0000 7fedf7a00640 1 -- 172.21.15.39:0/3154499321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf00cf990 msgr2=0x7fedf00cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.586+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3154499321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf00cf990 0x7fedf00cfd70 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fede0009870 tx=0x7fede002b980 comp rx=0 tx=0).stop 2024-01-24T17:45:00.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.587+0000 7fedf7a00640 1 -- 172.21.15.39:0/3154499321 shutdown_connections 2024-01-24T17:45:00.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.587+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3154499321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 0x7fedf015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.587+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3154499321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf00cf990 0x7fedf00cfd70 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.587+0000 7fedf7a00640 1 -- 172.21.15.39:0/3154499321 >> 172.21.15.39:0/3154499321 conn(0x7fedf0071790 msgr2=0x7fedf0071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:00.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.587+0000 7fedf7a00640 1 -- 172.21.15.39:0/3154499321 shutdown_connections 2024-01-24T17:45:00.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.587+0000 7fedf7a00640 1 -- 172.21.15.39:0/3154499321 wait complete. 2024-01-24T17:45:00.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.588+0000 7fedf7a00640 1 Processor -- start 2024-01-24T17:45:00.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.589+0000 7fedf7a00640 1 -- start start 2024-01-24T17:45:00.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf7a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedf00cf990 0x7fedf0157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf7a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 0x7fedf00d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf7a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf0160610 0x7fedf00b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf7a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00caa80 con 0x7fedf0160610 2024-01-24T17:45:00.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf7a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00c9e70 con 0x7fedf00cf990 2024-01-24T17:45:00.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf7a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf00ca150 con 0x7fedf015d020 2024-01-24T17:45:00.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf0160610 0x7fedf00b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf0160610 0x7fedf00b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43646/0 (socket says 172.21.15.39:43646) 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5a00640 1 -- 172.21.15.39:0/3786821069 learned_addr learned my addr 172.21.15.39:0/3786821069 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5000640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedf00cf990 0x7fedf0157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fede7c00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 0x7fedf00d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5a00640 1 -- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 msgr2=0x7fedf00d2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 0x7fedf00d2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5a00640 1 -- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedf00cf990 msgr2=0x7fedf0157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedf00cf990 0x7fedf0157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5a00640 1 -- 172.21.15.39:0/3786821069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede00094d0 con 0x7fedf0160610 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.590+0000 7fedf5000640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedf00cf990 0x7fedf0157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:00.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.591+0000 7fedf5a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf0160610 0x7fedf00b7bd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fede8001eb0 tx=0x7fede800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:00.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.591+0000 7feded600640 1 -- 172.21.15.39:0/3786821069 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede80052d0 con 0x7fedf0160610 2024-01-24T17:45:00.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.591+0000 7feded600640 1 -- 172.21.15.39:0/3786821069 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede8005430 con 0x7fedf0160610 2024-01-24T17:45:00.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.591+0000 7feded600640 1 -- 172.21.15.39:0/3786821069 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8003810 con 0x7fedf0160610 2024-01-24T17:45:00.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.591+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf00ca430 con 0x7fedf0160610 2024-01-24T17:45:00.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.591+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf00c9ad0 con 0x7fedf0160610 2024-01-24T17:45:00.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.592+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf00cabf0 con 0x7fedf0160610 2024-01-24T17:45:00.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.593+0000 7feded600640 1 -- 172.21.15.39:0/3786821069 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fede8003970 con 0x7fedf0160610 2024-01-24T17:45:00.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.594+0000 7feded600640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fedc806a430 0x7fedc806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:00.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.595+0000 7feded600640 1 -- 172.21.15.39:0/3786821069 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fede8087fb0 con 0x7fedf0160610 2024-01-24T17:45:00.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.595+0000 7fedf5000640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fedc806a430 0x7fedc806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:00.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.596+0000 7fedf5000640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fedc806a430 0x7fedc806c8f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fede0036000 tx=0x7fede003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:00.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.599+0000 7feded600640 1 -- 172.21.15.39:0/3786821069 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fede80534d0 con 0x7fedf0160610 2024-01-24T17:45:00.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.796+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedf0073ef0 con 0x7fedf0160610 2024-01-24T17:45:00.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.799+0000 7feded600640 1 -- 172.21.15.39:0/3786821069 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16392 (secure 0 0 0) 0x7fede8056af0 con 0x7fedf0160610 2024-01-24T17:45:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:00.800 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:38.321132+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.9","mappings":[{"from":5,"to":7}]},{"pgid":"7.1d","mappings":[{"from":5,"to":7}]},{"pgid":"7.39","mappings":[{"from":5,"to":0}]},{"pgid":"7.3a","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1994510967":"2024-01-24T18:44:37.552044+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:00.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.801+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fedc806a430 msgr2=0x7fedc806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.801+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fedc806a430 0x7fedc806c8f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fede0036000 tx=0x7fede003b040 comp rx=0 tx=0).stop 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.801+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf0160610 msgr2=0x7fedf00b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.801+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf0160610 0x7fedf00b7bd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fede8001eb0 tx=0x7fede800c890 comp rx=0 tx=0).stop 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.802+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 shutdown_connections 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.802+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fedc806a430 0x7fedc806c8f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.802+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf0160610 0x7fedf00b7bd0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.802+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fedf015d020 0x7fedf00d2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.802+0000 7fedf7a00640 1 --2- 172.21.15.39:0/3786821069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fedf00cf990 0x7fedf0157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.802+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 >> 172.21.15.39:0/3786821069 conn(0x7fedf0071790 msgr2=0x7fedf015ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.802+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 shutdown_connections 2024-01-24T17:45:00.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:00.802+0000 7fedf7a00640 1 -- 172.21.15.39:0/3786821069 wait complete. 2024-01-24T17:45:00.822 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:44:07.660016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:44:08.985819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:45:00.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:45:01.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.054+0000 7fc8cc400640 1 Processor -- start 2024-01-24T17:45:01.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.054+0000 7fc8cc400640 1 -- start start 2024-01-24T17:45:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.056+0000 7fc8cc400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc8c40b0350 0x7fc8c406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.056+0000 7fc8cc400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c415e190 0x7fc8c40b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.056+0000 7fc8cc400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c406d4c0 con 0x7fc8c406c9f0 2024-01-24T17:45:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.056+0000 7fc8cc400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c406cf00 con 0x7fc8c40b0350 2024-01-24T17:45:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.056+0000 7fc8cc400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c406d1e0 con 0x7fc8c415e190 2024-01-24T17:45:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.056+0000 7fc8c9000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc8c406c9f0 0x7fc8c415da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50652/0 (socket says 172.21.15.39:50652) 2024-01-24T17:45:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.056+0000 7fc8c9000640 1 -- 172.21.15.39:0/2738127964 learned_addr learned my addr 172.21.15.39:0/2738127964 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c9a00640 1 --2- 172.21.15.39:0/2738127964 >> v2:172.21.15.100:3300/0 conn(0x7fc8c40b0350 0x7fc8c406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8ca400640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c415e190 0x7fc8c40b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190706463 0 0) 0x7fc8c406d4c0 con 0x7fc8c406c9f0 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a4003520 con 0x7fc8c406c9f0 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354156874 0 0) 0x7fc8a4003520 con 0x7fc8c406c9f0 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c415e190 msgr2=0x7fc8c40b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c415e190 0x7fc8c40b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 >> v2:172.21.15.100:3300/0 conn(0x7fc8c40b0350 msgr2=0x7fc8c406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 --2- 172.21.15.39:0/2738127964 >> v2:172.21.15.100:3300/0 conn(0x7fc8c40b0350 0x7fc8c406c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40b15e0 con 0x7fc8c406c9f0 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc8ac001670 con 0x7fc8c406c9f0 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 >> v1:172.21.15.39:6789/0 conn(0x7fc8c406c9f0 legacy=0x7fc8c415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8a4003970 0x7fc8a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c415e190 0x7fc8a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c406cf00 con 0x7fc8a4004d00 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c406d1e0 con 0x7fc8a4003970 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.057+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a400b980 con 0x7fc8c415e190 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.058+0000 7fc8ca400640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.058+0000 7fc8c9a00640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8a4003970 0x7fc8a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.058+0000 7fc8c9000640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c415e190 0x7fc8a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.058+0000 7fc8ca400640 1 -- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c415e190 msgr2=0x7fc8a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:01.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.058+0000 7fc8ca400640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c415e190 0x7fc8a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.058+0000 7fc8ca400640 1 -- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8a4003970 msgr2=0x7fc8a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:01.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.058+0000 7fc8ca400640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8a4003970 0x7fc8a4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.058+0000 7fc8ca400640 1 -- 172.21.15.39:0/2738127964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a4003520 con 0x7fc8a4004d00 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.059+0000 7fc8ca400640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8a4005180 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009f90 tx=0x7fc8a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2293c68a9d50ea4d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc02cbd0 con 0x7fc8a4004d00 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8bc00f040 con 0x7fc8a4004d00 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8c3e00640 1 -- 172.21.15.39:0/2738127964 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc034440 con 0x7fc8a4004d00 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8cc400640 1 -- 172.21.15.39:0/2738127964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 msgr2=0x7fc8a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8cc400640 1 --2- 172.21.15.39:0/2738127964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8a4005180 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009f90 tx=0x7fc8a40057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8ca400640 1 -- 172.21.15.39:0/2738127964 reap_dead start 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8cc400640 1 -- 172.21.15.39:0/2738127964 shutdown_connections 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8cc400640 1 -- 172.21.15.39:0/2738127964 >> 172.21.15.39:0/2738127964 conn(0x7fc8c40cb0e0 msgr2=0x7fc8c40cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8cc400640 1 -- 172.21.15.39:0/2738127964 shutdown_connections 2024-01-24T17:45:01.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.060+0000 7fc8cc400640 1 -- 172.21.15.39:0/2738127964 wait complete. 2024-01-24T17:45:01.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.062+0000 7fc8cc400640 1 Processor -- start 2024-01-24T17:45:01.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.062+0000 7fc8cc400640 1 -- start start 2024-01-24T17:45:01.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.063+0000 7fc8cc400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8c4155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.063+0000 7fc8cc400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c4153240 0x7fc8c41536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.063+0000 7fc8cc400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8c41566c0 0x7fc8c4156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.063+0000 7fc8cc400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40c2170 con 0x7fc8a4004d00 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.063+0000 7fc8cc400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40c4cf0 con 0x7fc8c41566c0 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.063+0000 7fc8cc400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c40c23c0 con 0x7fc8c4153240 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.063+0000 7fc8c9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8c4155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8c4155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43660/0 (socket says 172.21.15.39:43660) 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9a00640 1 -- 172.21.15.39:0/2443389284 learned_addr learned my addr 172.21.15.39:0/2443389284 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9a00640 1 -- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c4153240 msgr2=0x7fc8c41536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9000640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c4153240 0x7fc8c41536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8ca400640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8c41566c0 0x7fc8c4156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9a00640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c4153240 0x7fc8c41536c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9a00640 1 -- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8c41566c0 msgr2=0x7fc8c4156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9a00640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8c41566c0 0x7fc8c4156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9a00640 1 -- 172.21.15.39:0/2443389284 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40c26a0 con 0x7fc8a4004d00 2024-01-24T17:45:01.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.064+0000 7fc8c9a00640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8c4155650 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4011600 tx=0x7fc8b4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.065+0000 7fc8c1600640 1 -- 172.21.15.39:0/2443389284 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b400f920 con 0x7fc8a4004d00 2024-01-24T17:45:01.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.065+0000 7fc8c1600640 1 -- 172.21.15.39:0/2443389284 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b400fa80 con 0x7fc8a4004d00 2024-01-24T17:45:01.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.065+0000 7fc8c1600640 1 -- 172.21.15.39:0/2443389284 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b400fdf0 con 0x7fc8a4004d00 2024-01-24T17:45:01.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.065+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c40c2980 con 0x7fc8a4004d00 2024-01-24T17:45:01.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.065+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c40c3180 con 0x7fc8a4004d00 2024-01-24T17:45:01.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.066+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c40c2500 con 0x7fc8a4004d00 2024-01-24T17:45:01.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.067+0000 7fc8c1600640 1 -- 172.21.15.39:0/2443389284 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc8b4015460 con 0x7fc8a4004d00 2024-01-24T17:45:01.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.068+0000 7fc8c1600640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc89c06a430 0x7fc89c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.068+0000 7fc8c1600640 1 -- 172.21.15.39:0/2443389284 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fc8b4088f50 con 0x7fc8a4004d00 2024-01-24T17:45:01.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.069+0000 7fc8c9000640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc89c06a430 0x7fc89c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.070+0000 7fc8c9000640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc89c06a430 0x7fc89c06c8f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac001cf0 tx=0x7fc8ac001c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.072+0000 7fc8c1600640 1 -- 172.21.15.39:0/2443389284 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc8c40c2500 con 0x7fc8a4004d00 2024-01-24T17:45:01.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.265+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8c40c2500 con 0x7fc8a4004d00 2024-01-24T17:45:01.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.266+0000 7fc8c1600640 1 -- 172.21.15.39:0/2443389284 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8c40c2500 con 0x7fc8a4004d00 2024-01-24T17:45:01.267 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc89c06a430 msgr2=0x7fc89c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc89c06a430 0x7fc89c06c8f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac001cf0 tx=0x7fc8ac001c80 comp rx=0 tx=0).stop 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 msgr2=0x7fc8c4155650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8c4155650 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4011600 tx=0x7fc8b4011ac0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 shutdown_connections 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8c41566c0 0x7fc8c4156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc8c4153240 0x7fc8c41536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc89c06a430 0x7fc89c06c8f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 --2- 172.21.15.39:0/2443389284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8a4004d00 0x7fc8c4155650 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 >> 172.21.15.39:0/2443389284 conn(0x7fc8c40cb0e0 msgr2=0x7fc8c415cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:01.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 shutdown_connections 2024-01-24T17:45:01.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.269+0000 7fc8cc400640 1 -- 172.21.15.39:0/2443389284 wait complete. 2024-01-24T17:45:01.291 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:45:01.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.514+0000 7fa1bd000640 1 Processor -- start 2024-01-24T17:45:01.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.514+0000 7fa1bd000640 1 -- start start 2024-01-24T17:45:01.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1bd000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 0x7fa1b8161170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1bd000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1b80cdb70 0x7fa1b806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1bd000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b806d930 con 0x7fa1b80cce00 2024-01-24T17:45:01.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1bd000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b806d650 con 0x7fa1b80cdb70 2024-01-24T17:45:01.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1bd000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b8160690 con 0x7fa1b815ca40 2024-01-24T17:45:01.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1afe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1b80cdb70 0x7fa1b806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1b7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa1b80cce00 0x7fa1b80ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50666/0 (socket says 172.21.15.39:50666) 2024-01-24T17:45:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1b7e00640 1 -- 172.21.15.39:0/3599835754 learned_addr learned my addr 172.21.15.39:0/3599835754 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.516+0000 7fa1afe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1b80cdb70 0x7fa1b806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43670/0 (socket says 172.21.15.39:43670) 2024-01-24T17:45:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.517+0000 7fa1b7400640 1 --2- 172.21.15.39:0/3599835754 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 0x7fa1b8161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.517+0000 7fa1afe00640 1 -- 172.21.15.39:0/3599835754 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 msgr2=0x7fa1b8161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.517+0000 7fa1afe00640 1 --2- 172.21.15.39:0/3599835754 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 0x7fa1b8161170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.517+0000 7fa1afe00640 1 -- 172.21.15.39:0/3599835754 >> v1:172.21.15.39:6789/0 conn(0x7fa1b80cce00 legacy=0x7fa1b80ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.517+0000 7fa1afe00640 1 -- 172.21.15.39:0/3599835754 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b8160970 con 0x7fa1b80cdb70 2024-01-24T17:45:01.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.518+0000 7fa1afe00640 1 --2- 172.21.15.39:0/3599835754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cdb70 0x7fa1b806cd50 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00097a0 tx=0x7fa1a002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5fee086689471d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.518+0000 7fa1b6a00640 1 -- 172.21.15.39:0/3599835754 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a002d040 con 0x7fa1b80cdb70 2024-01-24T17:45:01.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.518+0000 7fa1b6a00640 1 -- 172.21.15.39:0/3599835754 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a0031810 con 0x7fa1b80cdb70 2024-01-24T17:45:01.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.518+0000 7fa1b6a00640 1 -- 172.21.15.39:0/3599835754 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0031ba0 con 0x7fa1b80cdb70 2024-01-24T17:45:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.519+0000 7fa1bd000640 1 -- 172.21.15.39:0/3599835754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cdb70 msgr2=0x7fa1b806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.519+0000 7fa1bd000640 1 --2- 172.21.15.39:0/3599835754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cdb70 0x7fa1b806cd50 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00097a0 tx=0x7fa1a002b8b0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.519+0000 7fa1bd000640 1 -- 172.21.15.39:0/3599835754 shutdown_connections 2024-01-24T17:45:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.519+0000 7fa1bd000640 1 --2- 172.21.15.39:0/3599835754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cdb70 0x7fa1b806cd50 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.519+0000 7fa1bd000640 1 --2- 172.21.15.39:0/3599835754 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 0x7fa1b8161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.519+0000 7fa1bd000640 1 -- 172.21.15.39:0/3599835754 >> 172.21.15.39:0/3599835754 conn(0x7fa1b8069e50 msgr2=0x7fa1b806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.519+0000 7fa1bd000640 1 -- 172.21.15.39:0/3599835754 shutdown_connections 2024-01-24T17:45:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.519+0000 7fa1bd000640 1 -- 172.21.15.39:0/3599835754 wait complete. 2024-01-24T17:45:01.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.521+0000 7fa1bd000640 1 Processor -- start 2024-01-24T17:45:01.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.521+0000 7fa1bd000640 1 -- start start 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.522+0000 7fa1bd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cce00 0x7fa1b80aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.522+0000 7fa1bd000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80cdb70 0x7fa1b80c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.522+0000 7fa1bd000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 0x7fa1b80c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.522+0000 7fa1bd000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80c3640 con 0x7fa1b80cce00 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.522+0000 7fa1bd000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b81f2710 con 0x7fa1b80cdb70 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.522+0000 7fa1bd000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80c3890 con 0x7fa1b815ca40 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.522+0000 7fa1b7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cce00 0x7fa1b80aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1b7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cce00 0x7fa1b80aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43682/0 (socket says 172.21.15.39:43682) 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1b7400640 1 -- 172.21.15.39:0/1466960061 learned_addr learned my addr 172.21.15.39:0/1466960061 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1b7e00640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 0x7fa1b80c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1afe00640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80cdb70 0x7fa1b80c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1b7400640 1 -- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 msgr2=0x7fa1b80c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1b7400640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 0x7fa1b80c7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1b7400640 1 -- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80cdb70 msgr2=0x7fa1b80c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1b7400640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80cdb70 0x7fa1b80c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1b7400640 1 -- 172.21.15.39:0/1466960061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a0009400 con 0x7fa1b80cce00 2024-01-24T17:45:01.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.523+0000 7fa1afe00640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80cdb70 0x7fa1b80c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:01.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.524+0000 7fa1b7400640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cce00 0x7fa1b80aeb80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800d800 tx=0x7fa1a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.524+0000 7fa1af400640 1 -- 172.21.15.39:0/1466960061 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a800b6c0 con 0x7fa1b80cce00 2024-01-24T17:45:01.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.524+0000 7fa1af400640 1 -- 172.21.15.39:0/1466960061 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a800b820 con 0x7fa1b80cce00 2024-01-24T17:45:01.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.524+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b80c3b70 con 0x7fa1b80cce00 2024-01-24T17:45:01.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.524+0000 7fa1af400640 1 -- 172.21.15.39:0/1466960061 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a800bbd0 con 0x7fa1b80cce00 2024-01-24T17:45:01.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.524+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b80c4130 con 0x7fa1b80cce00 2024-01-24T17:45:01.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.525+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b804dff0 con 0x7fa1b80cce00 2024-01-24T17:45:01.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.526+0000 7fa1af400640 1 -- 172.21.15.39:0/1466960061 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa1a800bd30 con 0x7fa1b80cce00 2024-01-24T17:45:01.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.527+0000 7fa1af400640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.527+0000 7fa1af400640 1 -- 172.21.15.39:0/1466960061 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fa1a8088e90 con 0x7fa1b80cce00 2024-01-24T17:45:01.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.528+0000 7fa1afe00640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.529+0000 7fa1afe00640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0036000 tx=0x7fa1a003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.531+0000 7fa1af400640 1 -- 172.21.15.39:0/1466960061 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa1a8054450 con 0x7fa1b80cce00 2024-01-24T17:45:01.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.727+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1b80ccbe0 con 0x7fa1b80cce00 2024-01-24T17:45:01.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.728+0000 7fa1af400640 1 -- 172.21.15.39:0/1466960061 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v51) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1a8057a70 con 0x7fa1b80cce00 2024-01-24T17:45:01.728 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 msgr2=0x7fa19006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0036000 tx=0x7fa1a003c040 comp rx=0 tx=0).stop 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cce00 msgr2=0x7fa1b80aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cce00 0x7fa1b80aeb80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800d800 tx=0x7fa1a800dcc0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 shutdown_connections 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b815ca40 0x7fa1b80c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80cdb70 0x7fa1b80c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 --2- 172.21.15.39:0/1466960061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80cce00 0x7fa1b80aeb80 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 >> 172.21.15.39:0/1466960061 conn(0x7fa1b8069e50 msgr2=0x7fa1b80ac730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:01.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 shutdown_connections 2024-01-24T17:45:01.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.731+0000 7fa1bd000640 1 -- 172.21.15.39:0/1466960061 wait complete. 2024-01-24T17:45:01.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:45:01.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.960+0000 7fd63b000640 1 Processor -- start 2024-01-24T17:45:01.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.961+0000 7fd63b000640 1 -- start start 2024-01-24T17:45:01.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.962+0000 7fd63b000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd6340b8250 0x7fd6340cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.962+0000 7fd63b000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340b7e30 con 0x7fd6340cdb90 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.962+0000 7fd63b000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6340ce040 con 0x7fd63415c9e0 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.962+0000 7fd63b000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6340b6d90 con 0x7fd6340b8250 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.962+0000 7fd639200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd6340b8250 0x7fd6340cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.963+0000 7fd639200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd6340b8250 0x7fd6340cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43698/0 (socket says 172.21.15.39:43698) 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.963+0000 7fd639200640 1 -- 172.21.15.39:0/785197636 learned_addr learned my addr 172.21.15.39:0/785197636 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.963+0000 7fd639200640 1 -- 172.21.15.39:0/785197636 >> v1:172.21.15.100:6789/0 conn(0x7fd63415c9e0 legacy=0x7fd6340b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.963+0000 7fd639200640 1 -- 172.21.15.39:0/785197636 >> v1:172.21.15.39:6789/0 conn(0x7fd6340cdb90 legacy=0x7fd6340d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.963+0000 7fd639200640 1 -- 172.21.15.39:0/785197636 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6340b7070 con 0x7fd6340b8250 2024-01-24T17:45:01.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.964+0000 7fd639200640 1 --2- 172.21.15.39:0/785197636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd6340cd650 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd61c009870 tx=0x7fd61c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead25491021667eb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.964+0000 7fd633e00640 1 -- 172.21.15.39:0/785197636 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c02f850 con 0x7fd6340b8250 2024-01-24T17:45:01.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd633e00640 1 -- 172.21.15.39:0/785197636 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61c02f9b0 con 0x7fd6340b8250 2024-01-24T17:45:01.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd633e00640 1 -- 172.21.15.39:0/785197636 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c02fd40 con 0x7fd6340b8250 2024-01-24T17:45:01.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd63b000640 1 -- 172.21.15.39:0/785197636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 msgr2=0x7fd6340cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd63b000640 1 --2- 172.21.15.39:0/785197636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd6340cd650 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd61c009870 tx=0x7fd61c02ba30 comp rx=0 tx=0).stop 2024-01-24T17:45:01.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd63b000640 1 -- 172.21.15.39:0/785197636 shutdown_connections 2024-01-24T17:45:01.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd63b000640 1 --2- 172.21.15.39:0/785197636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd6340cd650 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd63b000640 1 -- 172.21.15.39:0/785197636 >> 172.21.15.39:0/785197636 conn(0x7fd634071790 msgr2=0x7fd634071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:01.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd63b000640 1 -- 172.21.15.39:0/785197636 shutdown_connections 2024-01-24T17:45:01.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.965+0000 7fd63b000640 1 -- 172.21.15.39:0/785197636 wait complete. 2024-01-24T17:45:01.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.967+0000 7fd63b000640 1 Processor -- start 2024-01-24T17:45:01.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.967+0000 7fd63b000640 1 -- start start 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd63b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd63414d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd63b000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6340cdb90 0x7fd6340c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd63b000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd63415c9e0 0x7fd63414dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd63b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63415f7a0 con 0x7fd6340b8250 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd63b000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63415ec20 con 0x7fd6340cdb90 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd63b000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63415ef00 con 0x7fd63415c9e0 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd639c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd63414d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd639c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd63414d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43706/0 (socket says 172.21.15.39:43706) 2024-01-24T17:45:01.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.968+0000 7fd639c00640 1 -- 172.21.15.39:0/4254064401 learned_addr learned my addr 172.21.15.39:0/4254064401 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:01.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd639200640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6340cdb90 0x7fd6340c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd639c00640 1 -- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd63415c9e0 msgr2=0x7fd63414dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd63a600640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd63415c9e0 0x7fd63414dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd639c00640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd63415c9e0 0x7fd63414dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd639c00640 1 -- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6340cdb90 msgr2=0x7fd6340c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:01.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd639c00640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6340cdb90 0x7fd6340c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:01.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd639c00640 1 -- 172.21.15.39:0/4254064401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c0094d0 con 0x7fd6340b8250 2024-01-24T17:45:01.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd639200640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6340cdb90 0x7fd6340c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:01.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd639c00640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd63414d440 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd62400b3f0 tx=0x7fd62400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd631600640 1 -- 172.21.15.39:0/4254064401 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624011830 con 0x7fd6340b8250 2024-01-24T17:45:01.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd631600640 1 -- 172.21.15.39:0/4254064401 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd624011990 con 0x7fd6340b8250 2024-01-24T17:45:01.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.969+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63415f1e0 con 0x7fd6340b8250 2024-01-24T17:45:01.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.970+0000 7fd631600640 1 -- 172.21.15.39:0/4254064401 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624011d40 con 0x7fd6340b8250 2024-01-24T17:45:01.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.970+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63415e820 con 0x7fd6340b8250 2024-01-24T17:45:01.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.971+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6340c6d90 con 0x7fd6340b8250 2024-01-24T17:45:01.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.972+0000 7fd631600640 1 -- 172.21.15.39:0/4254064401 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd62401a400 con 0x7fd6340b8250 2024-01-24T17:45:01.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.973+0000 7fd631600640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd60c06a430 0x7fd60c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:01.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.973+0000 7fd631600640 1 -- 172.21.15.39:0/4254064401 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fd6240888d0 con 0x7fd6340b8250 2024-01-24T17:45:01.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.974+0000 7fd639200640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd60c06a430 0x7fd60c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:01.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.975+0000 7fd639200640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd60c06a430 0x7fd60c06c8f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd61c036000 tx=0x7fd61c039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:01.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:01.978+0000 7fd631600640 1 -- 172.21.15.39:0/4254064401 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd6340c6d90 con 0x7fd6340b8250 2024-01-24T17:45:02.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.164+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd6340c6d90 con 0x7fd6340b8250 2024-01-24T17:45:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.165+0000 7fd631600640 1 -- 172.21.15.39:0/4254064401 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v51) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd6340c6d90 con 0x7fd6340b8250 2024-01-24T17:45:02.166 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd60c06a430 msgr2=0x7fd60c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd60c06a430 0x7fd60c06c8f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd61c036000 tx=0x7fd61c039040 comp rx=0 tx=0).stop 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 msgr2=0x7fd63414d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd63414d440 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd62400b3f0 tx=0x7fd62400b8b0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 shutdown_connections 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd63415c9e0 0x7fd63414dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd6340cdb90 0x7fd6340c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd60c06a430 0x7fd60c06c8f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 --2- 172.21.15.39:0/4254064401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6340b8250 0x7fd63414d440 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 >> 172.21.15.39:0/4254064401 conn(0x7fd634071790 msgr2=0x7fd6340d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 shutdown_connections 2024-01-24T17:45:02.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.168+0000 7fd63b000640 1 -- 172.21.15.39:0/4254064401 wait complete. 2024-01-24T17:45:02.186 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:45:02.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.404+0000 7f7015800640 1 Processor -- start 2024-01-24T17:45:02.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.404+0000 7f7015800640 1 -- start start 2024-01-24T17:45:02.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f7015800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f701015a720 0x7f70100cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f7015800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f70100ce8e0 0x7f70100ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f7015800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70100cf290 con 0x7f701015ab00 2024-01-24T17:45:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f7015800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70100ce400 con 0x7f70100ce8e0 2024-01-24T17:45:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f7015800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100ccef0 con 0x7f701015a720 2024-01-24T17:45:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f700f400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f701015ab00 0x7f70100cc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50676/0 (socket says 172.21.15.39:50676) 2024-01-24T17:45:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f700f400640 1 -- 172.21.15.39:0/1998947341 learned_addr learned my addr 172.21.15.39:0/1998947341 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:02.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f700fe00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f701015a720 0x7f70100cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f7014e00640 1 --2- 172.21.15.39:0/1998947341 >> v2:172.21.15.100:3300/0 conn(0x7f70100ce8e0 0x7f70100ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.406+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627109971 0 0) 0x7f70100cf290 con 0x7f701015ab00 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff0003520 con 0x7f701015ab00 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3841229966 0 0) 0x7f6ff0003520 con 0x7f701015ab00 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f701015a720 msgr2=0x7f70100cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f701015a720 0x7f70100cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 >> v2:172.21.15.100:3300/0 conn(0x7f70100ce8e0 msgr2=0x7f70100ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 --2- 172.21.15.39:0/1998947341 >> v2:172.21.15.100:3300/0 conn(0x7f70100ce8e0 0x7f70100ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100ce110 con 0x7f701015ab00 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f6ff8001670 con 0x7f701015ab00 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 >> v1:172.21.15.39:6789/0 conn(0x7f701015ab00 legacy=0x7f70100cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0003970 0x7f6ff0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff00047c0 0x7f6ff0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f701015a720 0x7f6ff0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100ce400 con 0x7f6ff0003970 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100ccef0 con 0x7f6ff00047c0 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff000bae0 con 0x7f701015a720 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f7014e00640 1 --2- 172.21.15.39:0/1998947341 >> v2:172.21.15.100:3300/0 conn(0x7f70100ce8e0 0x7f70100ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.407+0000 7f700fe00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0003970 0x7f6ff0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.408+0000 7f7014e00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f701015a720 0x7f6ff0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.408+0000 7f700f400640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff00047c0 0x7f6ff0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.408+0000 7f700fe00640 1 -- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f701015a720 msgr2=0x7f6ff0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.408+0000 7f700fe00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f701015a720 0x7f6ff0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.408+0000 7f700fe00640 1 -- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff00047c0 msgr2=0x7f6ff0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.408+0000 7f700fe00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff00047c0 0x7f6ff0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.408+0000 7f700fe00640 1 -- 172.21.15.39:0/1998947341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff0003520 con 0x7f6ff0003970 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.408+0000 7f700f400640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff00047c0 0x7f6ff0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.409+0000 7f700fe00640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0003970 0x7f6ff0003d90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f700000ba00 tx=0x7f6ff00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bcee140a7122070 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.409+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000325f0 con 0x7f6ff0003970 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.409+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000032750 con 0x7f6ff0003970 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.409+0000 7f700ea00640 1 -- 172.21.15.39:0/1998947341 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000032ae0 con 0x7f6ff0003970 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.410+0000 7f7015800640 1 -- 172.21.15.39:0/1998947341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0003970 msgr2=0x7f6ff0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.410+0000 7f7015800640 1 --2- 172.21.15.39:0/1998947341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ff0003970 0x7f6ff0003d90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f700000ba00 tx=0x7f6ff00057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.410+0000 7f7014e00640 1 -- 172.21.15.39:0/1998947341 reap_dead start 2024-01-24T17:45:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.410+0000 7f7015800640 1 -- 172.21.15.39:0/1998947341 shutdown_connections 2024-01-24T17:45:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.410+0000 7f7015800640 1 -- 172.21.15.39:0/1998947341 >> 172.21.15.39:0/1998947341 conn(0x7f701006a090 msgr2=0x7f701006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.410+0000 7f7015800640 1 -- 172.21.15.39:0/1998947341 shutdown_connections 2024-01-24T17:45:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.410+0000 7f7015800640 1 -- 172.21.15.39:0/1998947341 wait complete. 2024-01-24T17:45:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.412+0000 7f7015800640 1 Processor -- start 2024-01-24T17:45:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.412+0000 7f7015800640 1 -- start start 2024-01-24T17:45:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f7015800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003970 0x7f70100b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f7015800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70100c4820 0x7f701014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f7015800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100c4c80 0x7f701014a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f7015800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701014ade0 con 0x7f70100c4820 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f7015800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100b13f0 con 0x7f6ff0003970 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f7015800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100b16d0 con 0x7f70100c4c80 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f700f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70100c4820 0x7f701014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f700f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70100c4820 0x7f701014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43722/0 (socket says 172.21.15.39:43722) 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.413+0000 7f700f400640 1 -- 172.21.15.39:0/4014040670 learned_addr learned my addr 172.21.15.39:0/4014040670 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f700fe00640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003970 0x7f70100b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f700f400640 1 -- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100c4c80 msgr2=0x7f701014a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f7014e00640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100c4c80 0x7f701014a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f700f400640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100c4c80 0x7f701014a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f700f400640 1 -- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003970 msgr2=0x7f70100b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f700f400640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003970 0x7f70100b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f700f400640 1 -- 172.21.15.39:0/4014040670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100b19b0 con 0x7f70100c4820 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f700f400640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70100c4820 0x7f701014a360 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800d8c0 tx=0x7f6ff800dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f7007e00640 1 -- 172.21.15.39:0/4014040670 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80074e0 con 0x7f70100c4820 2024-01-24T17:45:02.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f7007e00640 1 -- 172.21.15.39:0/4014040670 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff8007640 con 0x7f70100c4820 2024-01-24T17:45:02.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70100b1c90 con 0x7f70100c4820 2024-01-24T17:45:02.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f7007e00640 1 -- 172.21.15.39:0/4014040670 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff80049e0 con 0x7f70100c4820 2024-01-24T17:45:02.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.414+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70100b0c50 con 0x7f70100c4820 2024-01-24T17:45:02.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.416+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70100c5710 con 0x7f70100c4820 2024-01-24T17:45:02.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.417+0000 7f7007e00640 1 -- 172.21.15.39:0/4014040670 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6ff8004b40 con 0x7f70100c4820 2024-01-24T17:45:02.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.417+0000 7f7007e00640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fe806a430 0x7f6fe806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.418+0000 7f7007e00640 1 -- 172.21.15.39:0/4014040670 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f6ff8088660 con 0x7f70100c4820 2024-01-24T17:45:02.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.418+0000 7f700fe00640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fe806a430 0x7f6fe806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.419+0000 7f700fe00640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fe806a430 0x7f6fe806c8f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00057a0 tx=0x7f7000030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:02.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.423+0000 7f7007e00640 1 -- 172.21.15.39:0/4014040670 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6ff8053b80 con 0x7f70100c4820 2024-01-24T17:45:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.663+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70100b1810 con 0x7f70100c4820 2024-01-24T17:45:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.663+0000 7f7007e00640 1 -- 172.21.15.39:0/4014040670 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4755 (secure 0 0 0) 0x7f70100b1810 con 0x7f70100c4820 2024-01-24T17:45:02.664 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:02.664 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:44:09.961260+0000","modified":"2024-01-24T17:44:34.305760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5769,"mds_1":5763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5763":{"gid":5763,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.100:6837/1418568795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1418568795},{"type":"v1","addr":"172.21.15.100:6837","nonce":1418568795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5769":{"gid":5769,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/1882733155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1882733155},{"type":"v1","addr":"172.21.15.39:6835","nonce":1882733155}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-24T17:45:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-01-24T17:45:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fe806a430 msgr2=0x7f6fe806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fe806a430 0x7f6fe806c8f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00057a0 tx=0x7f7000030000 comp rx=0 tx=0).stop 2024-01-24T17:45:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70100c4820 msgr2=0x7f701014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70100c4820 0x7f701014a360 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800d8c0 tx=0x7f6ff800dd80 comp rx=0 tx=0).stop 2024-01-24T17:45:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 shutdown_connections 2024-01-24T17:45:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f70100c4c80 0x7f701014a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fe806a430 0x7f6fe806c8f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f70100c4820 0x7f701014a360 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 --2- 172.21.15.39:0/4014040670 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6ff0003970 0x7f70100b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 >> 172.21.15.39:0/4014040670 conn(0x7f701006a090 msgr2=0x7f70100cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.666+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 shutdown_connections 2024-01-24T17:45:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.667+0000 7f7015800640 1 -- 172.21.15.39:0/4014040670 wait complete. 2024-01-24T17:45:02.685 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:02.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.907+0000 7fb058400640 1 Processor -- start 2024-01-24T17:45:02.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.908+0000 7fb058400640 1 -- start start 2024-01-24T17:45:02.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.909+0000 7fb058400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05015d490 0x7fb05015f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.909+0000 7fb058400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05015dac0 con 0x7fb0500cf990 2024-01-24T17:45:02.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.909+0000 7fb058400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05015dda0 con 0x7fb05015d020 2024-01-24T17:45:02.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.909+0000 7fb058400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb05015e080 con 0x7fb05015d490 2024-01-24T17:45:02.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.910+0000 7fb055a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb0500cf990 0x7fb0500cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50680/0 (socket says 172.21.15.39:50680) 2024-01-24T17:45:02.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.910+0000 7fb055a00640 1 -- 172.21.15.39:0/1291643933 learned_addr learned my addr 172.21.15.39:0/1291643933 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:02.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.910+0000 7fb056400640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05015d490 0x7fb05015f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.910+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454245060 0 0) 0x7fb05015dac0 con 0x7fb0500cf990 2024-01-24T17:45:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb030003520 con 0x7fb0500cf990 2024-01-24T17:45:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760925228 0 0) 0x7fb05015dda0 con 0x7fb05015d020 2024-01-24T17:45:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05015dac0 con 0x7fb05015d020 2024-01-24T17:45:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628455868 0 0) 0x7fb030003520 con 0x7fb0500cf990 2024-01-24T17:45:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05015d490 msgr2=0x7fb05015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:02.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05015d490 0x7fb05015f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 >> v1:172.21.15.100:6789/0 conn(0x7fb05015d020 legacy=0x7fb05015eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05015e380 con 0x7fb0500cf990 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fb040001670 con 0x7fb0500cf990 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 >> v1:172.21.15.39:6789/0 conn(0x7fb0500cf990 legacy=0x7fb0500cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030003b30 0x7fb030003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05015d490 0x7fb0300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb030004d00 0x7fb030005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05015e080 con 0x7fb030004d00 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03000ba60 con 0x7fb030003b30 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03000bc20 con 0x7fb05015d490 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.911+0000 7fb056400640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb030004d00 0x7fb030005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.912+0000 7fb055000640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05015d490 0x7fb0300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.912+0000 7fb055a00640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030003b30 0x7fb030003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.912+0000 7fb056400640 1 -- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05015d490 msgr2=0x7fb0300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.912+0000 7fb056400640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb05015d490 0x7fb0300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.912+0000 7fb056400640 1 -- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030003b30 msgr2=0x7fb030003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.912+0000 7fb056400640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030003b30 0x7fb030003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.912+0000 7fb056400640 1 -- 172.21.15.39:0/1291643933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030003520 con 0x7fb030004d00 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.913+0000 7fb056400640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb030004d00 0x7fb030005180 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb048009fc0 tx=0x7fb0300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cdae0343827bb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.913+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04802cc20 con 0x7fb030004d00 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.913+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04800f040 con 0x7fb030004d00 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.913+0000 7fb04fe00640 1 -- 172.21.15.39:0/1291643933 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048034440 con 0x7fb030004d00 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.914+0000 7fb058400640 1 -- 172.21.15.39:0/1291643933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb030004d00 msgr2=0x7fb030005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.914+0000 7fb058400640 1 --2- 172.21.15.39:0/1291643933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb030004d00 0x7fb030005180 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fb048009fc0 tx=0x7fb0300057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.914+0000 7fb056400640 1 -- 172.21.15.39:0/1291643933 reap_dead start 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.914+0000 7fb058400640 1 -- 172.21.15.39:0/1291643933 shutdown_connections 2024-01-24T17:45:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.914+0000 7fb058400640 1 -- 172.21.15.39:0/1291643933 >> 172.21.15.39:0/1291643933 conn(0x7fb050071790 msgr2=0x7fb050071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:02.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.914+0000 7fb058400640 1 -- 172.21.15.39:0/1291643933 shutdown_connections 2024-01-24T17:45:02.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.914+0000 7fb058400640 1 -- 172.21.15.39:0/1291643933 wait complete. 2024-01-24T17:45:02.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.916+0000 7fb058400640 1 Processor -- start 2024-01-24T17:45:02.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.916+0000 7fb058400640 1 -- start start 2024-01-24T17:45:02.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb058400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030004d00 0x7fb0500d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb058400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0501f59a0 0x7fb0501f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb058400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0501f28c0 0x7fb0501f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb058400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0501f6360 con 0x7fb0501f59a0 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb058400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500d2540 con 0x7fb030004d00 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb058400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500d2820 con 0x7fb0501f28c0 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb055000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0501f59a0 0x7fb0501f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb055000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0501f59a0 0x7fb0501f5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43752/0 (socket says 172.21.15.39:43752) 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.917+0000 7fb055000640 1 -- 172.21.15.39:0/3531567501 learned_addr learned my addr 172.21.15.39:0/3531567501 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb055a00640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030004d00 0x7fb0500d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb056400640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0501f28c0 0x7fb0501f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb055000640 1 -- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0501f28c0 msgr2=0x7fb0501f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb055000640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0501f28c0 0x7fb0501f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb055000640 1 -- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030004d00 msgr2=0x7fb0500d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:02.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb055000640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030004d00 0x7fb0500d1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:02.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb055000640 1 -- 172.21.15.39:0/3531567501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0500d2b00 con 0x7fb0501f59a0 2024-01-24T17:45:02.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb056400640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0501f28c0 0x7fb0501f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:02.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb055000640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0501f59a0 0x7fb0501f5e20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb03800e7b0 tx=0x7fb03800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:02.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb04d600640 1 -- 172.21.15.39:0/3531567501 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03800ee60 con 0x7fb0501f59a0 2024-01-24T17:45:02.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb04d600640 1 -- 172.21.15.39:0/3531567501 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0380052e0 con 0x7fb0501f59a0 2024-01-24T17:45:02.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb04d600640 1 -- 172.21.15.39:0/3531567501 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03800c900 con 0x7fb0501f59a0 2024-01-24T17:45:02.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.918+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0500d2de0 con 0x7fb0501f59a0 2024-01-24T17:45:02.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.919+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0500d2250 con 0x7fb0501f59a0 2024-01-24T17:45:02.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.920+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb050073ef0 con 0x7fb0501f59a0 2024-01-24T17:45:02.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.921+0000 7fb04d600640 1 -- 172.21.15.39:0/3531567501 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb03800ca60 con 0x7fb0501f59a0 2024-01-24T17:45:02.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.921+0000 7fb04d600640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb02806a460 0x7fb02806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:02.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.922+0000 7fb04d600640 1 -- 172.21.15.39:0/3531567501 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fb0380888a0 con 0x7fb0501f59a0 2024-01-24T17:45:02.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.922+0000 7fb055a00640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb02806a460 0x7fb02806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:02.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.923+0000 7fb055a00640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb02806a460 0x7fb02806c920 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb040001eb0 tx=0x7fb040009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:02.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:02.926+0000 7fb04d600640 1 -- 172.21.15.39:0/3531567501 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb038053dc0 con 0x7fb0501f59a0 2024-01-24T17:45:03.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.130+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb05015ea80 con 0x7fb0501f59a0 2024-01-24T17:45:03.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.131+0000 7fb04d600640 1 -- 172.21.15.39:0/3531567501 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16392 (secure 0 0 0) 0x7fb0380573e0 con 0x7fb0501f59a0 2024-01-24T17:45:03.132 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:03.132 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:38.321132+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.9","mappings":[{"from":5,"to":7}]},{"pgid":"7.1d","mappings":[{"from":5,"to":7}]},{"pgid":"7.39","mappings":[{"from":5,"to":0}]},{"pgid":"7.3a","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1994510967":"2024-01-24T18:44:37.552044+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb02806a460 msgr2=0x7fb02806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb02806a460 0x7fb02806c920 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb040001eb0 tx=0x7fb040009040 comp rx=0 tx=0).stop 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0501f59a0 msgr2=0x7fb0501f5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0501f59a0 0x7fb0501f5e20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb03800e7b0 tx=0x7fb03800ec70 comp rx=0 tx=0).stop 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 shutdown_connections 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0501f28c0 0x7fb0501f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb02806a460 0x7fb02806c920 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0501f59a0 0x7fb0501f5e20 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 --2- 172.21.15.39:0/3531567501 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb030004d00 0x7fb0500d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 >> 172.21.15.39:0/3531567501 conn(0x7fb050071790 msgr2=0x7fb0500b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:03.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 shutdown_connections 2024-01-24T17:45:03.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.134+0000 7fb058400640 1 -- 172.21.15.39:0/3531567501 wait complete. 2024-01-24T17:45:03.156 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-24T17:45:03.156 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:45:03.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.360+0000 7fda4d000640 1 Processor -- start 2024-01-24T17:45:03.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.360+0000 7fda4d000640 1 -- start start 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda4d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481654d0 0x7fda480c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda4d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fda480c90d0 0x7fda48070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda4d000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fda481658b0 0x7fda480caf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda4d000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda480cc9d0 con 0x7fda480c90d0 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda4d000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda480cc630 con 0x7fda481658b0 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda4d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda480c9570 con 0x7fda481654d0 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda3fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fda480c90d0 0x7fda48070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda3fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fda480c90d0 0x7fda48070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43770/0 (socket says 172.21.15.39:43770) 2024-01-24T17:45:03.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.362+0000 7fda3fe00640 1 -- 172.21.15.39:0/1551504320 learned_addr learned my addr 172.21.15.39:0/1551504320 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:03.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.363+0000 7fda45e00640 1 --2- 172.21.15.39:0/1551504320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481654d0 0x7fda480c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.363+0000 7fda46800640 1 --2- 172.21.15.39:0/1551504320 >> v2:172.21.15.100:3300/0 conn(0x7fda481658b0 0x7fda480caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.363+0000 7fda3fe00640 1 -- 172.21.15.39:0/1551504320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481654d0 msgr2=0x7fda480c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:03.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.363+0000 7fda3fe00640 1 --2- 172.21.15.39:0/1551504320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481654d0 0x7fda480c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.363+0000 7fda3fe00640 1 -- 172.21.15.39:0/1551504320 >> v2:172.21.15.100:3300/0 conn(0x7fda481658b0 msgr2=0x7fda480caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.363+0000 7fda3fe00640 1 --2- 172.21.15.39:0/1551504320 >> v2:172.21.15.100:3300/0 conn(0x7fda481658b0 0x7fda480caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.363+0000 7fda3fe00640 1 -- 172.21.15.39:0/1551504320 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda480c9850 con 0x7fda480c90d0 2024-01-24T17:45:03.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.363+0000 7fda46800640 1 --2- 172.21.15.39:0/1551504320 >> v2:172.21.15.100:3300/0 conn(0x7fda481658b0 0x7fda480caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.364+0000 7fda3fe00640 1 --2- 172.21.15.39:0/1551504320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48070030 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fda4806cb10 tx=0x7fda3002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74123e12e35066 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda45400640 1 -- 172.21.15.39:0/1551504320 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30031740 con 0x7fda480c90d0 2024-01-24T17:45:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda45400640 1 -- 172.21.15.39:0/1551504320 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda300318a0 con 0x7fda480c90d0 2024-01-24T17:45:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda45400640 1 -- 172.21.15.39:0/1551504320 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30031c30 con 0x7fda480c90d0 2024-01-24T17:45:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda4d000640 1 -- 172.21.15.39:0/1551504320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 msgr2=0x7fda48070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda4d000640 1 --2- 172.21.15.39:0/1551504320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48070030 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fda4806cb10 tx=0x7fda3002b7e0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda4d000640 1 -- 172.21.15.39:0/1551504320 shutdown_connections 2024-01-24T17:45:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda4d000640 1 --2- 172.21.15.39:0/1551504320 >> v2:172.21.15.100:3300/0 conn(0x7fda481658b0 0x7fda480caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda4d000640 1 --2- 172.21.15.39:0/1551504320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48070030 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda4d000640 1 --2- 172.21.15.39:0/1551504320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481654d0 0x7fda480c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda4d000640 1 -- 172.21.15.39:0/1551504320 >> 172.21.15.39:0/1551504320 conn(0x7fda4806a3e0 msgr2=0x7fda4806bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.365+0000 7fda4d000640 1 -- 172.21.15.39:0/1551504320 shutdown_connections 2024-01-24T17:45:03.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.366+0000 7fda4d000640 1 -- 172.21.15.39:0/1551504320 wait complete. 2024-01-24T17:45:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.367+0000 7fda4d000640 1 Processor -- start 2024-01-24T17:45:03.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.367+0000 7fda4d000640 1 -- start start 2024-01-24T17:45:03.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.368+0000 7fda4d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.368+0000 7fda4d000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda481654d0 0x7fda480c41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.368+0000 7fda4d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481658b0 0x7fda480adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.368+0000 7fda4d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda48155d00 con 0x7fda480c90d0 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.368+0000 7fda4d000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda480b0470 con 0x7fda481654d0 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.368+0000 7fda4d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda480c2540 con 0x7fda481658b0 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda45e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda45e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43784/0 (socket says 172.21.15.39:43784) 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda45e00640 1 -- 172.21.15.39:0/143782636 learned_addr learned my addr 172.21.15.39:0/143782636 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda46800640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481658b0 0x7fda480adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda45e00640 1 -- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481658b0 msgr2=0x7fda480adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda45e00640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481658b0 0x7fda480adea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda3fe00640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda481654d0 0x7fda480c41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda45e00640 1 -- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda481654d0 msgr2=0x7fda480c41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda45e00640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda481654d0 0x7fda480c41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.369+0000 7fda45e00640 1 -- 172.21.15.39:0/143782636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300094d0 con 0x7fda480c90d0 2024-01-24T17:45:03.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.370+0000 7fda45e00640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48155390 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d830 tx=0x7fda3800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:03.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.370+0000 7fda3e000640 1 -- 172.21.15.39:0/143782636 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380096a0 con 0x7fda480c90d0 2024-01-24T17:45:03.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.370+0000 7fda3e000640 1 -- 172.21.15.39:0/143782636 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda38009800 con 0x7fda480c90d0 2024-01-24T17:45:03.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.370+0000 7fda3e000640 1 -- 172.21.15.39:0/143782636 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38009b90 con 0x7fda480c90d0 2024-01-24T17:45:03.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.370+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda480c2210 con 0x7fda480c90d0 2024-01-24T17:45:03.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.370+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda480b0190 con 0x7fda480c90d0 2024-01-24T17:45:03.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.371+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda480c2070 con 0x7fda480c90d0 2024-01-24T17:45:03.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.372+0000 7fda3e000640 1 -- 172.21.15.39:0/143782636 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fda38014410 con 0x7fda480c90d0 2024-01-24T17:45:03.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.373+0000 7fda3e000640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fda2006a430 0x7fda2006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.373+0000 7fda3e000640 1 -- 172.21.15.39:0/143782636 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7fda38089390 con 0x7fda480c90d0 2024-01-24T17:45:03.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.374+0000 7fda3fe00640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fda2006a430 0x7fda2006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.375+0000 7fda3fe00640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fda2006a430 0x7fda2006c8f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fda30035000 tx=0x7fda30008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:03.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.378+0000 7fda3e000640 1 -- 172.21.15.39:0/143782636 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fda480c2070 con 0x7fda480c90d0 2024-01-24T17:45:03.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.606+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda480c2070 con 0x7fda480c90d0 2024-01-24T17:45:03.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.607+0000 7fda3e000640 1 -- 172.21.15.39:0/143782636 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4755 (secure 0 0 0) 0x7fda480c2070 con 0x7fda480c90d0 2024-01-24T17:45:03.608 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:03.608 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:44:09.961260+0000","modified":"2024-01-24T17:44:34.305760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5769,"mds_1":5763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5763":{"gid":5763,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.100:6837/1418568795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1418568795},{"type":"v1","addr":"172.21.15.100:6837","nonce":1418568795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5769":{"gid":5769,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/1882733155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1882733155},{"type":"v1","addr":"172.21.15.39:6835","nonce":1882733155}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-24T17:45:03.608 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-01-24T17:45:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fda2006a430 msgr2=0x7fda2006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fda2006a430 0x7fda2006c8f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fda30035000 tx=0x7fda30008030 comp rx=0 tx=0).stop 2024-01-24T17:45:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 msgr2=0x7fda48155390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48155390 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d830 tx=0x7fda3800dcf0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 shutdown_connections 2024-01-24T17:45:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fda481658b0 0x7fda480adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fda481654d0 0x7fda480c41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fda2006a430 0x7fda2006c8f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 --2- 172.21.15.39:0/143782636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda480c90d0 0x7fda48155390 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 >> 172.21.15.39:0/143782636 conn(0x7fda4806a3e0 msgr2=0x7fda480caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 shutdown_connections 2024-01-24T17:45:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.610+0000 7fda4d000640 1 -- 172.21.15.39:0/143782636 wait complete. 2024-01-24T17:45:03.629 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:45:03.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.844+0000 7f19e3e00640 1 Processor -- start 2024-01-24T17:45:03.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.845+0000 7f19e3e00640 1 -- start start 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.846+0000 7f19e3e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f19dc15ca40 0x7f19dc0b8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.846+0000 7f19e3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f19dc0b69f0 0x7f19dc0b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.846+0000 7f19e3e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc0b7ce0 con 0x7f19dc0b4480 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.846+0000 7f19e3e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0b4e30 con 0x7f19dc0b69f0 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.846+0000 7f19e3e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0b6e40 con 0x7f19dc15ca40 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.847+0000 7f19e1e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f19dc0b69f0 0x7f19dc0b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.847+0000 7f19e1e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f19dc0b69f0 0x7f19dc0b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43794/0 (socket says 172.21.15.39:43794) 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.847+0000 7f19e1e00640 1 -- 172.21.15.39:0/1129144771 learned_addr learned my addr 172.21.15.39:0/1129144771 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.847+0000 7f19e1400640 1 --2- 172.21.15.39:0/1129144771 >> v2:172.21.15.100:3300/0 conn(0x7f19dc15ca40 0x7f19dc0b8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.847+0000 7f19e1e00640 1 -- 172.21.15.39:0/1129144771 >> v2:172.21.15.100:3300/0 conn(0x7f19dc15ca40 msgr2=0x7f19dc0b8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:03.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.847+0000 7f19e1e00640 1 --2- 172.21.15.39:0/1129144771 >> v2:172.21.15.100:3300/0 conn(0x7f19dc15ca40 0x7f19dc0b8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.847+0000 7f19e1e00640 1 -- 172.21.15.39:0/1129144771 >> v1:172.21.15.100:6789/0 conn(0x7f19dc0b4480 legacy=0x7f19dc0d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:03.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.847+0000 7f19e1e00640 1 -- 172.21.15.39:0/1129144771 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0b70d0 con 0x7f19dc0b69f0 2024-01-24T17:45:03.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.848+0000 7f19e1e00640 1 --2- 172.21.15.39:0/1129144771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b69f0 0x7f19dc0b48f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0d0760 tx=0x7f19d402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229d3260723bedb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:03.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.849+0000 7f19dbe00640 1 -- 172.21.15.39:0/1129144771 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d402ca80 con 0x7f19dc0b69f0 2024-01-24T17:45:03.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.849+0000 7f19dbe00640 1 -- 172.21.15.39:0/1129144771 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d402cbe0 con 0x7f19dc0b69f0 2024-01-24T17:45:03.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.849+0000 7f19dbe00640 1 -- 172.21.15.39:0/1129144771 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d40343f0 con 0x7f19dc0b69f0 2024-01-24T17:45:03.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.849+0000 7f19e3e00640 1 -- 172.21.15.39:0/1129144771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b69f0 msgr2=0x7f19dc0b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.849+0000 7f19e3e00640 1 --2- 172.21.15.39:0/1129144771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b69f0 0x7f19dc0b48f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0d0760 tx=0x7f19d402bee0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.850+0000 7f19e3e00640 1 -- 172.21.15.39:0/1129144771 shutdown_connections 2024-01-24T17:45:03.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.850+0000 7f19e3e00640 1 --2- 172.21.15.39:0/1129144771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b69f0 0x7f19dc0b48f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.850+0000 7f19e3e00640 1 --2- 172.21.15.39:0/1129144771 >> v2:172.21.15.100:3300/0 conn(0x7f19dc15ca40 0x7f19dc0b8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.850+0000 7f19e3e00640 1 -- 172.21.15.39:0/1129144771 >> 172.21.15.39:0/1129144771 conn(0x7f19dc071790 msgr2=0x7f19dc071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:03.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.850+0000 7f19e3e00640 1 -- 172.21.15.39:0/1129144771 shutdown_connections 2024-01-24T17:45:03.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.850+0000 7f19e3e00640 1 -- 172.21.15.39:0/1129144771 wait complete. 2024-01-24T17:45:03.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.851+0000 7f19e3e00640 1 Processor -- start 2024-01-24T17:45:03.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.852+0000 7f19e3e00640 1 -- start start 2024-01-24T17:45:03.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.852+0000 7f19e3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b4480 0x7f19dc15b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.852+0000 7f19e3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19dc0b69f0 0x7f19dc152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.852+0000 7f19e3e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19dc15ca40 0x7f19dc0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.852+0000 7f19e3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc15c0f0 con 0x7f19dc0b4480 2024-01-24T17:45:03.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.852+0000 7f19e3e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc152dd0 con 0x7f19dc15ca40 2024-01-24T17:45:03.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.852+0000 7f19e3e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0ca240 con 0x7f19dc0b69f0 2024-01-24T17:45:03.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b4480 0x7f19dc15b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b4480 0x7f19dc15b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43796/0 (socket says 172.21.15.39:43796) 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1400640 1 -- 172.21.15.39:0/2543981005 learned_addr learned my addr 172.21.15.39:0/2543981005 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1400640 1 -- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19dc0b69f0 msgr2=0x7f19dc152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e0a00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19dc0b69f0 0x7f19dc152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1e00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19dc15ca40 0x7f19dc0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1400640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19dc0b69f0 0x7f19dc152890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1400640 1 -- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19dc15ca40 msgr2=0x7f19dc0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1400640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19dc15ca40 0x7f19dc0c9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.853+0000 7f19e1400640 1 -- 172.21.15.39:0/2543981005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d4009b30 con 0x7f19dc0b4480 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.854+0000 7f19e1400640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b4480 0x7f19dc15b190 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00d5f0 tx=0x7f19cc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.854+0000 7f19d9600640 1 -- 172.21.15.39:0/2543981005 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0125f0 con 0x7f19dc0b4480 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.854+0000 7f19d9600640 1 -- 172.21.15.39:0/2543981005 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc012750 con 0x7f19dc0b4480 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.854+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc15b850 con 0x7f19dc0b4480 2024-01-24T17:45:03.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.854+0000 7f19d9600640 1 -- 172.21.15.39:0/2543981005 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc012b00 con 0x7f19dc0b4480 2024-01-24T17:45:03.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.854+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc15be10 con 0x7f19dc0b4480 2024-01-24T17:45:03.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.855+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc073ef0 con 0x7f19dc0b4480 2024-01-24T17:45:03.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.856+0000 7f19d9600640 1 -- 172.21.15.39:0/2543981005 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f19cc012c60 con 0x7f19dc0b4480 2024-01-24T17:45:03.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.857+0000 7f19d9600640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f19b406a430 0x7f19b406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:03.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.857+0000 7f19d9600640 1 -- 172.21.15.39:0/2543981005 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f19cc088e70 con 0x7f19dc0b4480 2024-01-24T17:45:03.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.858+0000 7f19e0a00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f19b406a430 0x7f19b406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:03.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.859+0000 7f19e0a00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f19b406a430 0x7f19b406c8f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0d2560 tx=0x7f19c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:03.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:03.861+0000 7f19d9600640 1 -- 172.21.15.39:0/2543981005 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f19cc054390 con 0x7f19dc0b4480 2024-01-24T17:45:04.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.092+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19dc0c8d30 con 0x7f19dc0b4480 2024-01-24T17:45:04.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.093+0000 7f19d9600640 1 -- 172.21.15.39:0/2543981005 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4755 (secure 0 0 0) 0x7f19cc0579b0 con 0x7f19dc0b4480 2024-01-24T17:45:04.094 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:04.094 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:44:09.961260+0000","modified":"2024-01-24T17:44:34.305760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5769,"mds_1":5763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5763":{"gid":5763,"name":"b","rank":1,"incarnation":25,"state":"up:active","state_seq":7,"addr":"172.21.15.100:6837/1418568795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1418568795},{"type":"v1","addr":"172.21.15.100:6837","nonce":1418568795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5769":{"gid":5769,"name":"c","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/1882733155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1882733155},{"type":"v1","addr":"172.21.15.39:6835","nonce":1882733155}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-24T17:45:04.094 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-01-24T17:45:04.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f19b406a430 msgr2=0x7f19b406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f19b406a430 0x7f19b406c8f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0d2560 tx=0x7f19c4009040 comp rx=0 tx=0).stop 2024-01-24T17:45:04.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b4480 msgr2=0x7f19dc15b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b4480 0x7f19dc15b190 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00d5f0 tx=0x7f19cc00dab0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 shutdown_connections 2024-01-24T17:45:04.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19dc15ca40 0x7f19dc0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19dc0b69f0 0x7f19dc152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f19b406a430 0x7f19b406c8f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 --2- 172.21.15.39:0/2543981005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19dc0b4480 0x7f19dc15b190 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 >> 172.21.15.39:0/2543981005 conn(0x7f19dc071790 msgr2=0x7f19dc0c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:04.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 shutdown_connections 2024-01-24T17:45:04.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.096+0000 7f19e3e00640 1 -- 172.21.15.39:0/2543981005 wait complete. 2024-01-24T17:45:04.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.336+0000 7f9e42800640 1 Processor -- start 2024-01-24T17:45:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.336+0000 7f9e42800640 1 -- start start 2024-01-24T17:45:04.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.338+0000 7f9e42800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9e3c0caf50 0x7f9e3c0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.338+0000 7f9e42800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c0b1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.338+0000 7f9e42800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3c0c9180 con 0x7f9e3c15a950 2024-01-24T17:45:04.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.338+0000 7f9e42800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e3c0cb920 con 0x7f9e3c0caf50 2024-01-24T17:45:04.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.338+0000 7f9e42800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c0b43d0 con 0x7f9e3c0b1390 2024-01-24T17:45:04.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.338+0000 7f9e3be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c0b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.339+0000 7f9e33e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9e3c0caf50 0x7f9e3c0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.339+0000 7f9e3be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c0b1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:33566/0 (socket says 172.21.15.39:33566) 2024-01-24T17:45:04.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.339+0000 7f9e3be00640 1 -- 172.21.15.39:0/984549748 learned_addr learned my addr 172.21.15.39:0/984549748 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:04.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.339+0000 7f9e3be00640 1 -- 172.21.15.39:0/984549748 >> v2:172.21.15.100:3300/0 conn(0x7f9e3c0caf50 msgr2=0x7f9e3c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:04.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.339+0000 7f9e3be00640 1 --2- 172.21.15.39:0/984549748 >> v2:172.21.15.100:3300/0 conn(0x7f9e3c0caf50 0x7f9e3c0cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.339+0000 7f9e3be00640 1 -- 172.21.15.39:0/984549748 >> v1:172.21.15.100:6789/0 conn(0x7f9e3c15a950 legacy=0x7f9e3c0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.339+0000 7f9e3be00640 1 -- 172.21.15.39:0/984549748 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c0c9460 con 0x7f9e3c0b1390 2024-01-24T17:45:04.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.341+0000 7f9e3be00640 1 --2- 172.21.15.39:0/984549748 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c0b1800 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009e50 tx=0x7f9e3402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4cc4754cb95888 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:04.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.341+0000 7f9e3aa00640 1 -- 172.21.15.39:0/984549748 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3402cb00 con 0x7f9e3c0b1390 2024-01-24T17:45:04.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.341+0000 7f9e3aa00640 1 -- 172.21.15.39:0/984549748 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3402cc60 con 0x7f9e3c0b1390 2024-01-24T17:45:04.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.341+0000 7f9e3aa00640 1 -- 172.21.15.39:0/984549748 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e340343f0 con 0x7f9e3c0b1390 2024-01-24T17:45:04.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.342+0000 7f9e42800640 1 -- 172.21.15.39:0/984549748 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 msgr2=0x7f9e3c0b1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.342+0000 7f9e42800640 1 --2- 172.21.15.39:0/984549748 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c0b1800 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009e50 tx=0x7f9e3402bf60 comp rx=0 tx=0).stop 2024-01-24T17:45:04.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.342+0000 7f9e42800640 1 -- 172.21.15.39:0/984549748 shutdown_connections 2024-01-24T17:45:04.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.342+0000 7f9e42800640 1 --2- 172.21.15.39:0/984549748 >> v2:172.21.15.100:3300/0 conn(0x7f9e3c0caf50 0x7f9e3c0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.342+0000 7f9e42800640 1 --2- 172.21.15.39:0/984549748 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c0b1800 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.342+0000 7f9e42800640 1 -- 172.21.15.39:0/984549748 >> 172.21.15.39:0/984549748 conn(0x7f9e3c06a3e0 msgr2=0x7f9e3c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:04.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.342+0000 7f9e42800640 1 -- 172.21.15.39:0/984549748 shutdown_connections 2024-01-24T17:45:04.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.342+0000 7f9e42800640 1 -- 172.21.15.39:0/984549748 wait complete. 2024-01-24T17:45:04.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.344+0000 7f9e42800640 1 Processor -- start 2024-01-24T17:45:04.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.344+0000 7f9e42800640 1 -- start start 2024-01-24T17:45:04.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e42800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c1ef830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e42800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e3c0caf50 0x7f9e3c0b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e42800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e3c15a950 0x7f9e3c0b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e42800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c14f0e0 con 0x7f9e3c15a950 2024-01-24T17:45:04.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e42800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c15d930 con 0x7f9e3c0caf50 2024-01-24T17:45:04.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e42800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c0a75e0 con 0x7f9e3c0b1390 2024-01-24T17:45:04.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e3be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e3c15a950 0x7f9e3c0b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e3be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e3c15a950 0x7f9e3c0b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43802/0 (socket says 172.21.15.39:43802) 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e3be00640 1 -- 172.21.15.39:0/2240616001 learned_addr learned my addr 172.21.15.39:0/2240616001 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e3b400640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c1ef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.346+0000 7f9e33e00640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e3c0caf50 0x7f9e3c0b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e3be00640 1 -- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 msgr2=0x7f9e3c1ef830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e3be00640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c1ef830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e3be00640 1 -- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e3c0caf50 msgr2=0x7f9e3c0b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e3be00640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e3c0caf50 0x7f9e3c0b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e3be00640 1 -- 172.21.15.39:0/2240616001 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e34009b30 con 0x7f9e3c15a950 2024-01-24T17:45:04.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e3be00640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e3c15a950 0x7f9e3c0b07b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009e50 tx=0x7f9e34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:04.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e33400640 1 -- 172.21.15.39:0/2240616001 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e34009510 con 0x7f9e3c15a950 2024-01-24T17:45:04.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e33400640 1 -- 172.21.15.39:0/2240616001 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3403e450 con 0x7f9e3c15a950 2024-01-24T17:45:04.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e33400640 1 -- 172.21.15.39:0/2240616001 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3403e800 con 0x7f9e3c15a950 2024-01-24T17:45:04.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3c1efd70 con 0x7f9e3c15a950 2024-01-24T17:45:04.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.347+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3c15d650 con 0x7f9e3c15a950 2024-01-24T17:45:04.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.349+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3c073ef0 con 0x7f9e3c15a950 2024-01-24T17:45:04.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.350+0000 7f9e33400640 1 -- 172.21.15.39:0/2240616001 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9e3403e960 con 0x7f9e3c15a950 2024-01-24T17:45:04.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.351+0000 7f9e33400640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e1406a430 0x7f9e1406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.351+0000 7f9e33400640 1 -- 172.21.15.39:0/2240616001 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7f9e340ab740 con 0x7f9e3c15a950 2024-01-24T17:45:04.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.351+0000 7f9e3b400640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e1406a430 0x7f9e1406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.352+0000 7f9e3b400640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e1406a430 0x7f9e1406c8f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c000f60 tx=0x7f9e2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.355+0000 7f9e33400640 1 -- 172.21.15.39:0/2240616001 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9e34076c60 con 0x7f9e3c15a950 2024-01-24T17:45:04.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.549+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e3c073490 con 0x7f9e3c15a950 2024-01-24T17:45:04.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.552+0000 7f9e33400640 1 -- 172.21.15.39:0/2240616001 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16392 (secure 0 0 0) 0x7f9e3407a280 con 0x7f9e3c15a950 2024-01-24T17:45:04.554 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:04.554 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:44:38.321132+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:44:07.660016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-24T17:44:08.985819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":7}]},{"pgid":"7.9","mappings":[{"from":5,"to":7}]},{"pgid":"7.1d","mappings":[{"from":5,"to":7}]},{"pgid":"7.39","mappings":[{"from":5,"to":0}]},{"pgid":"7.3a","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"192.168.0.1:0/1994510967":"2024-01-24T18:44:37.552044+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:04.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e1406a430 msgr2=0x7f9e1406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e1406a430 0x7f9e1406c8f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c000f60 tx=0x7f9e2c009040 comp rx=0 tx=0).stop 2024-01-24T17:45:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e3c15a950 msgr2=0x7f9e3c0b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e3c15a950 0x7f9e3c0b07b0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009e50 tx=0x7f9e34004a40 comp rx=0 tx=0).stop 2024-01-24T17:45:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 shutdown_connections 2024-01-24T17:45:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e1406a430 0x7f9e1406c8f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9e3c15a950 0x7f9e3c0b07b0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9e3c0caf50 0x7f9e3c0b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 --2- 172.21.15.39:0/2240616001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9e3c0b1390 0x7f9e3c1ef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 >> 172.21.15.39:0/2240616001 conn(0x7f9e3c06a3e0 msgr2=0x7f9e3c0b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 shutdown_connections 2024-01-24T17:45:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.555+0000 7f9e42800640 1 -- 172.21.15.39:0/2240616001 wait complete. 2024-01-24T17:45:04.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-24T17:45:04.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.791+0000 7ff747200640 1 Processor -- start 2024-01-24T17:45:04.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.792+0000 7ff747200640 1 -- start start 2024-01-24T17:45:04.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.793+0000 7ff747200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff7480cb020 0x7ff74815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.793+0000 7ff747200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7480c8c10 0x7ff7480c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.793+0000 7ff747200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74815b240 con 0x7ff7480cb400 2024-01-24T17:45:04.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.793+0000 7ff747200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74815b520 con 0x7ff7480c8c10 2024-01-24T17:45:04.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.793+0000 7ff747200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74815b800 con 0x7ff7480cb020 2024-01-24T17:45:04.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.794+0000 7ff746800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7480c8c10 0x7ff7480c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.794+0000 7ff746800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7480c8c10 0x7ff7480c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43812/0 (socket says 172.21.15.39:43812) 2024-01-24T17:45:04.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.794+0000 7ff746800640 1 -- 172.21.15.39:0/1094079825 learned_addr learned my addr 172.21.15.39:0/1094079825 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:04.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.794+0000 7ff745e00640 1 --2- 172.21.15.39:0/1094079825 >> v2:172.21.15.100:3300/0 conn(0x7ff7480cb020 0x7ff74815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.794+0000 7ff746800640 1 -- 172.21.15.39:0/1094079825 >> v2:172.21.15.100:3300/0 conn(0x7ff7480cb020 msgr2=0x7ff74815ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:04.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.794+0000 7ff746800640 1 --2- 172.21.15.39:0/1094079825 >> v2:172.21.15.100:3300/0 conn(0x7ff7480cb020 0x7ff74815ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.794+0000 7ff746800640 1 -- 172.21.15.39:0/1094079825 >> v1:172.21.15.100:6789/0 conn(0x7ff7480cb400 legacy=0x7ff7480ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.794+0000 7ff746800640 1 -- 172.21.15.39:0/1094079825 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74815bae0 con 0x7ff7480c8c10 2024-01-24T17:45:04.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.795+0000 7ff746800640 1 --2- 172.21.15.39:0/1094079825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c8c10 0x7ff7480c90a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff7480b08f0 tx=0x7ff74002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dde904e8643cd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:04.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.796+0000 7ff745400640 1 -- 172.21.15.39:0/1094079825 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74002cb20 con 0x7ff7480c8c10 2024-01-24T17:45:04.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.796+0000 7ff745400640 1 -- 172.21.15.39:0/1094079825 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74002cc80 con 0x7ff7480c8c10 2024-01-24T17:45:04.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.796+0000 7ff745400640 1 -- 172.21.15.39:0/1094079825 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7400343f0 con 0x7ff7480c8c10 2024-01-24T17:45:04.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.796+0000 7ff747200640 1 -- 172.21.15.39:0/1094079825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c8c10 msgr2=0x7ff7480c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.796+0000 7ff747200640 1 --2- 172.21.15.39:0/1094079825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c8c10 0x7ff7480c90a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff7480b08f0 tx=0x7ff74002bf80 comp rx=0 tx=0).stop 2024-01-24T17:45:04.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.797+0000 7ff747200640 1 -- 172.21.15.39:0/1094079825 shutdown_connections 2024-01-24T17:45:04.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.797+0000 7ff747200640 1 --2- 172.21.15.39:0/1094079825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480c8c10 0x7ff7480c90a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.797+0000 7ff747200640 1 --2- 172.21.15.39:0/1094079825 >> v2:172.21.15.100:3300/0 conn(0x7ff7480cb020 0x7ff74815ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.797+0000 7ff747200640 1 -- 172.21.15.39:0/1094079825 >> 172.21.15.39:0/1094079825 conn(0x7ff74806a2a0 msgr2=0x7ff7480afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:04.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.797+0000 7ff747200640 1 -- 172.21.15.39:0/1094079825 shutdown_connections 2024-01-24T17:45:04.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.797+0000 7ff747200640 1 -- 172.21.15.39:0/1094079825 wait complete. 2024-01-24T17:45:04.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.798+0000 7ff747200640 1 Processor -- start 2024-01-24T17:45:04.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.798+0000 7ff747200640 1 -- start start 2024-01-24T17:45:04.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.799+0000 7ff747200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7480c8c10 0x7ff7480c1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.799+0000 7ff747200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480cb020 0x7ff7480c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.799+0000 7ff747200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7480cb400 0x7ff7481f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.799+0000 7ff747200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7480c4eb0 con 0x7ff7480cb020 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.799+0000 7ff747200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7480c59b0 con 0x7ff7480cb400 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.799+0000 7ff747200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7481f4f30 con 0x7ff7480c8c10 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480cb020 0x7ff7480c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480cb020 0x7ff7480c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43820/0 (socket says 172.21.15.39:43820) 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 -- 172.21.15.39:0/3026651873 learned_addr learned my addr 172.21.15.39:0/3026651873 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff745e00640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7480c8c10 0x7ff7480c1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff746800640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7480cb400 0x7ff7481f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 -- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7480c8c10 msgr2=0x7ff7480c1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7480c8c10 0x7ff7480c1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 -- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7480cb400 msgr2=0x7ff7481f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7480cb400 0x7ff7481f49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 -- 172.21.15.39:0/3026651873 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740009b30 con 0x7ff7480cb020 2024-01-24T17:45:04.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff746800640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7480cb400 0x7ff7481f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:04.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.800+0000 7ff73fe00640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480cb020 0x7ff7480c5470 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff73000c430 tx=0x7ff73000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:04.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.801+0000 7ff73e000640 1 -- 172.21.15.39:0/3026651873 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7300052d0 con 0x7ff7480cb020 2024-01-24T17:45:04.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.801+0000 7ff73e000640 1 -- 172.21.15.39:0/3026651873 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff730005430 con 0x7ff7480cb020 2024-01-24T17:45:04.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.801+0000 7ff73e000640 1 -- 172.21.15.39:0/3026651873 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730003830 con 0x7ff7480cb020 2024-01-24T17:45:04.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.801+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7480c7680 con 0x7ff7480cb020 2024-01-24T17:45:04.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.801+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7480c4900 con 0x7ff7480cb020 2024-01-24T17:45:04.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.802+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7480c7810 con 0x7ff7480cb020 2024-01-24T17:45:04.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.803+0000 7ff73e000640 1 -- 172.21.15.39:0/3026651873 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff730003990 con 0x7ff7480cb020 2024-01-24T17:45:04.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.804+0000 7ff73e000640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72006a430 0x7ff72006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:04.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.804+0000 7ff73e000640 1 -- 172.21.15.39:0/3026651873 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5897+0+0 (secure 0 0 0) 0x7ff730087f60 con 0x7ff7480cb020 2024-01-24T17:45:04.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.805+0000 7ff745e00640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72006a430 0x7ff72006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:04.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.806+0000 7ff745e00640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72006a430 0x7ff72006c8f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff738009ad0 tx=0x7ff738008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:04.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:04.809+0000 7ff73e000640 1 -- 172.21.15.39:0/3026651873 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff730053470 con 0x7ff7480cb020 2024-01-24T17:45:05.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.040+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff74804dff0 con 0x7ff7480cb020 2024-01-24T17:45:05.043 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:45:05.042+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:45:05.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.057+0000 7ff73e000640 1 -- 172.21.15.39:0/3026651873 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff730056a90 con 0x7ff7480cb020 2024-01-24T17:45:05.058 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-24T17:45:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72006a430 msgr2=0x7ff72006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72006a430 0x7ff72006c8f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff738009ad0 tx=0x7ff738008040 comp rx=0 tx=0).stop 2024-01-24T17:45:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480cb020 msgr2=0x7ff7480c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480cb020 0x7ff7480c5470 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff73000c430 tx=0x7ff73000c8f0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 shutdown_connections 2024-01-24T17:45:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff7480cb400 0x7ff7481f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72006a430 0x7ff72006c8f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7480cb020 0x7ff7480c5470 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 --2- 172.21.15.39:0/3026651873 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7480c8c10 0x7ff7480c1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 >> 172.21.15.39:0/3026651873 conn(0x7ff74806a2a0 msgr2=0x7ff74815db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.060+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 shutdown_connections 2024-01-24T17:45:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.061+0000 7ff747200640 1 -- 172.21.15.39:0/3026651873 wait complete. 2024-01-24T17:45:05.072 INFO:tasks.ceph.mds.b.smithi100.stderr: -3703> 2024-01-24T17:44:04.118+0000 7f523e600640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-01-24T17:45:05.080 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-24T17:45:05.086 INFO:tasks.ceph.mds.c.smithi039.stderr: -8524> 2024-01-24T17:44:14.429+0000 7f1141400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:45:05.094 INFO:tasks.ceph.mds.c.smithi039.stderr: -5744> 2024-01-24T17:44:29.430+0000 7f1141400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-01-24T17:45:05.165 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:45:05.166 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:45:05.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.284+0000 7f2c27000640 1 Processor -- start 2024-01-24T17:45:05.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.284+0000 7f2c27000640 1 -- start start 2024-01-24T17:45:05.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.285+0000 7f2c27000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015ab30 0x7f2c200cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.285+0000 7f2c27000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2c200ce760 0x7f2c200b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.285+0000 7f2c27000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c200cebf0 con 0x7f2c200b2230 2024-01-24T17:45:05.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.285+0000 7f2c27000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c200ceed0 con 0x7f2c200ce760 2024-01-24T17:45:05.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.285+0000 7f2c27000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c200cf1b0 con 0x7f2c2015ab30 2024-01-24T17:45:05.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.286+0000 7f2c25000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2c200b2230 0x7f2c200b3600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53678/0 (socket says 172.21.15.39:53678) 2024-01-24T17:45:05.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.286+0000 7f2c25000640 1 -- 172.21.15.39:0/3392406646 learned_addr learned my addr 172.21.15.39:0/3392406646 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:05.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.286+0000 7f2c1fe00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015ab30 0x7f2c200cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.286+0000 7f2c1f400640 1 --2- 172.21.15.39:0/3392406646 >> v2:172.21.15.100:3300/0 conn(0x7f2c200ce760 0x7f2c200b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514581072 0 0) 0x7f2c200cebf0 con 0x7f2c200b2230 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c00003520 con 0x7f2c200b2230 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1885026224 0 0) 0x7f2c00003520 con 0x7f2c200b2230 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015ab30 msgr2=0x7f2c200cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015ab30 0x7f2c200cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 >> v2:172.21.15.100:3300/0 conn(0x7f2c200ce760 msgr2=0x7f2c200b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 --2- 172.21.15.39:0/3392406646 >> v2:172.21.15.100:3300/0 conn(0x7f2c200ce760 0x7f2c200b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c200cf490 con 0x7f2c200b2230 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2c18001670 con 0x7f2c200b2230 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 >> v1:172.21.15.39:6789/0 conn(0x7f2c200b2230 legacy=0x7f2c200b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c00003970 0x7f2c00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c000047c0 0x7f2c00004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015ab30 0x7f2c00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c200ceed0 con 0x7f2c00003970 2024-01-24T17:45:05.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c200cf1b0 con 0x7f2c000047c0 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0000b980 con 0x7f2c2015ab30 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.287+0000 7f2c1fe00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c00003970 0x7f2c00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.288+0000 7f2c25000640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015ab30 0x7f2c00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.288+0000 7f2c1fe00640 1 -- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015ab30 msgr2=0x7f2c00005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.288+0000 7f2c1f400640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c000047c0 0x7f2c00004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.288+0000 7f2c1fe00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015ab30 0x7f2c00005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.288+0000 7f2c1fe00640 1 -- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c000047c0 msgr2=0x7f2c00004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.288+0000 7f2c1fe00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c000047c0 0x7f2c00004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.288+0000 7f2c1fe00640 1 -- 172.21.15.39:0/3392406646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c00003520 con 0x7f2c00003970 2024-01-24T17:45:05.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.289+0000 7f2c1fe00640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c00003970 0x7f2c00003d90 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009ac0 tx=0x7f2c000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d4f35c16182893 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:05.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.289+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0802c6b0 con 0x7f2c00003970 2024-01-24T17:45:05.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.289+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0802c810 con 0x7f2c00003970 2024-01-24T17:45:05.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.289+0000 7f2c1ea00640 1 -- 172.21.15.39:0/3392406646 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0802cba0 con 0x7f2c00003970 2024-01-24T17:45:05.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.290+0000 7f2c27000640 1 -- 172.21.15.39:0/3392406646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c00003970 msgr2=0x7f2c00003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:05.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.290+0000 7f2c27000640 1 --2- 172.21.15.39:0/3392406646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c00003970 0x7f2c00003d90 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009ac0 tx=0x7f2c000057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.290+0000 7f2c25000640 1 -- 172.21.15.39:0/3392406646 reap_dead start 2024-01-24T17:45:05.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.290+0000 7f2c27000640 1 -- 172.21.15.39:0/3392406646 shutdown_connections 2024-01-24T17:45:05.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.290+0000 7f2c27000640 1 -- 172.21.15.39:0/3392406646 >> 172.21.15.39:0/3392406646 conn(0x7f2c200cd230 msgr2=0x7f2c200cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:05.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.290+0000 7f2c27000640 1 -- 172.21.15.39:0/3392406646 shutdown_connections 2024-01-24T17:45:05.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.290+0000 7f2c27000640 1 -- 172.21.15.39:0/3392406646 wait complete. 2024-01-24T17:45:05.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.292+0000 7f2c27000640 1 Processor -- start 2024-01-24T17:45:05.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.292+0000 7f2c27000640 1 -- start start 2024-01-24T17:45:05.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c27000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c00003970 0x7f2c2006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c27000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c2006aef0 0x7f2c20158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c27000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015dac0 0x7f2c200ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c27000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2015d6a0 con 0x7f2c2006aef0 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c27000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c200ca780 con 0x7f2c00003970 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c27000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c200ca1c0 con 0x7f2c2015dac0 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c2006aef0 0x7f2c20158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c2006aef0 0x7f2c20158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43866/0 (socket says 172.21.15.39:43866) 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.293+0000 7f2c1f400640 1 -- 172.21.15.39:0/1503535260 learned_addr learned my addr 172.21.15.39:0/1503535260 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c25000640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015dac0 0x7f2c200ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c1fe00640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c00003970 0x7f2c2006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c1f400640 1 -- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015dac0 msgr2=0x7f2c200ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c1f400640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015dac0 0x7f2c200ada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c1f400640 1 -- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c00003970 msgr2=0x7f2c2006b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c1f400640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c00003970 0x7f2c2006b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:05.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c1f400640 1 -- 172.21.15.39:0/1503535260 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c200ca4a0 con 0x7f2c2006aef0 2024-01-24T17:45:05.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c1f400640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c2006aef0 0x7f2c20158b10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000f3d0 tx=0x7f2c1000f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:05.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c17e00640 1 -- 172.21.15.39:0/1503535260 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10016620 con 0x7f2c2006aef0 2024-01-24T17:45:05.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c17e00640 1 -- 172.21.15.39:0/1503535260 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c100167a0 con 0x7f2c2006aef0 2024-01-24T17:45:05.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c17e00640 1 -- 172.21.15.39:0/1503535260 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10016b50 con 0x7f2c2006aef0 2024-01-24T17:45:05.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.294+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2015d2c0 con 0x7f2c2006aef0 2024-01-24T17:45:05.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.295+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2015df40 con 0x7f2c2006aef0 2024-01-24T17:45:05.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.296+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c20067040 con 0x7f2c2006aef0 2024-01-24T17:45:05.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.297+0000 7f2c17e00640 1 -- 172.21.15.39:0/1503535260 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2c10016cb0 con 0x7f2c2006aef0 2024-01-24T17:45:05.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.298+0000 7f2c17e00640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2bf806a430 0x7f2bf806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:05.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.298+0000 7f2c17e00640 1 -- 172.21.15.39:0/1503535260 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6069+0+0 (secure 0 0 0) 0x7f2c10089990 con 0x7f2c2006aef0 2024-01-24T17:45:05.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.298+0000 7f2c1fe00640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2bf806a430 0x7f2bf806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:05.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.299+0000 7f2c1fe00640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2bf806a430 0x7f2bf806c8f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2c08036000 tx=0x7f2c08030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:05.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.303+0000 7f2c17e00640 1 -- 172.21.15.39:0/1503535260 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2c10054e00 con 0x7f2c2006aef0 2024-01-24T17:45:05.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:05.537+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2c200aeec0 con 0x7f2c2006aef0 2024-01-24T17:45:06.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.062+0000 7f2c17e00640 1 -- 172.21.15.39:0/1503535260 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2c10058420 con 0x7f2c2006aef0 2024-01-24T17:45:06.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2bf806a430 msgr2=0x7f2bf806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:06.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2bf806a430 0x7f2bf806c8f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2c08036000 tx=0x7f2c08030000 comp rx=0 tx=0).stop 2024-01-24T17:45:06.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c2006aef0 msgr2=0x7f2c20158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:06.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c2006aef0 0x7f2c20158b10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000f3d0 tx=0x7f2c1000f890 comp rx=0 tx=0).stop 2024-01-24T17:45:06.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 shutdown_connections 2024-01-24T17:45:06.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c2015dac0 0x7f2c200ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2bf806a430 0x7f2bf806c8f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c2006aef0 0x7f2c20158b10 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 --2- 172.21.15.39:0/1503535260 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c00003970 0x7f2c2006b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 >> 172.21.15.39:0/1503535260 conn(0x7f2c200cd230 msgr2=0x7f2c200b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:06.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 shutdown_connections 2024-01-24T17:45:06.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.066+0000 7f2c27000640 1 -- 172.21.15.39:0/1503535260 wait complete. 2024-01-24T17:45:06.089 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-24T17:45:06.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.357+0000 7fb6e1800640 1 Processor -- start 2024-01-24T17:45:06.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.357+0000 7fb6e1800640 1 -- start start 2024-01-24T17:45:06.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.359+0000 7fb6e1800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc0b0730 0x7fb6dc15da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:06.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.359+0000 7fb6e1800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc15e450 con 0x7fb6dc15dfc0 2024-01-24T17:45:06.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.359+0000 7fb6e1800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc06d4c0 con 0x7fb6dc0b0350 2024-01-24T17:45:06.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.359+0000 7fb6e1800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc06cf00 con 0x7fb6dc0b0730 2024-01-24T17:45:06.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.359+0000 7fb6db400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb6dc15dfc0 0x7fb6dc0b1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53682/0 (socket says 172.21.15.39:53682) 2024-01-24T17:45:06.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.359+0000 7fb6db400640 1 -- 172.21.15.39:0/4177797859 learned_addr learned my addr 172.21.15.39:0/4177797859 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:06.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6da000640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc0b0730 0x7fb6dc15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:06.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173048549 0 0) 0x7fb6d4001490 con 0x7fb6dc15dfc0 2024-01-24T17:45:06.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6bc003520 con 0x7fb6dc15dfc0 2024-01-24T17:45:06.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27105456 0 0) 0x7fb6bc003520 con 0x7fb6dc15dfc0 2024-01-24T17:45:06.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc0b0730 msgr2=0x7fb6dc15da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:06.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc0b0730 0x7fb6dc15da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 >> v1:172.21.15.100:6789/0 conn(0x7fb6dc0b0350 legacy=0x7fb6dc06c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:06.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc06d1e0 con 0x7fb6dc15dfc0 2024-01-24T17:45:06.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fb6d4001b80 con 0x7fb6dc15dfc0 2024-01-24T17:45:06.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 >> v1:172.21.15.39:6789/0 conn(0x7fb6dc15dfc0 legacy=0x7fb6dc0b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:06.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc0039a0 0x7fb6bc003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:06.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6bc004350 0x7fb6bc004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:06.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc0b0730 0x7fb6bc005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:06.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc06cf00 con 0x7fb6bc004350 2024-01-24T17:45:06.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc00ba60 con 0x7fb6bc0039a0 2024-01-24T17:45:06.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.360+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc00bc20 con 0x7fb6dc0b0730 2024-01-24T17:45:06.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.361+0000 7fb6da000640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6bc004350 0x7fb6bc004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:06.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.361+0000 7fb6daa00640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc0039a0 0x7fb6bc003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:06.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.361+0000 7fb6db400640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc0b0730 0x7fb6bc005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:06.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.361+0000 7fb6da000640 1 -- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc0b0730 msgr2=0x7fb6bc005120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:06.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.361+0000 7fb6da000640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc0b0730 0x7fb6bc005120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.361+0000 7fb6da000640 1 -- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc0039a0 msgr2=0x7fb6bc003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:06.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.361+0000 7fb6da000640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc0039a0 0x7fb6bc003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.361+0000 7fb6da000640 1 -- 172.21.15.39:0/4177797859 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc003520 con 0x7fb6bc004350 2024-01-24T17:45:06.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.362+0000 7fb6da000640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6bc004350 0x7fb6bc004820 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009ac0 tx=0x7fb6bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83759bb32eaded21 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:06.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.363+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c402c6b0 con 0x7fb6bc004350 2024-01-24T17:45:06.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.363+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c402c810 con 0x7fb6bc004350 2024-01-24T17:45:06.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.363+0000 7fb6d9600640 1 -- 172.21.15.39:0/4177797859 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c402cba0 con 0x7fb6bc004350 2024-01-24T17:45:06.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.363+0000 7fb6e1800640 1 -- 172.21.15.39:0/4177797859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6bc004350 msgr2=0x7fb6bc004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:06.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.363+0000 7fb6e1800640 1 --2- 172.21.15.39:0/4177797859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6bc004350 0x7fb6bc004820 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009ac0 tx=0x7fb6bc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.363+0000 7fb6db400640 1 -- 172.21.15.39:0/4177797859 reap_dead start 2024-01-24T17:45:06.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.363+0000 7fb6e1800640 1 -- 172.21.15.39:0/4177797859 shutdown_connections 2024-01-24T17:45:06.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.363+0000 7fb6e1800640 1 -- 172.21.15.39:0/4177797859 >> 172.21.15.39:0/4177797859 conn(0x7fb6dc0cb0e0 msgr2=0x7fb6dc0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:06.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.364+0000 7fb6e1800640 1 -- 172.21.15.39:0/4177797859 shutdown_connections 2024-01-24T17:45:06.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.364+0000 7fb6e1800640 1 -- 172.21.15.39:0/4177797859 wait complete. 2024-01-24T17:45:06.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.365+0000 7fb6e1800640 1 Processor -- start 2024-01-24T17:45:06.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.366+0000 7fb6e1800640 1 -- start start 2024-01-24T17:45:06.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6e1800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc004350 0x7fb6dc0c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:06.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6e1800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc1f32d0 0x7fb6dc1e5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:06.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6e1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6dc1e3c20 0x7fb6dc1e40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:06.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6e1800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc1e71e0 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6e1800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc1e45e0 con 0x7fb6bc004350 2024-01-24T17:45:06.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6e1800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc1e6560 con 0x7fb6dc1f32d0 2024-01-24T17:45:06.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6db400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6dc1e3c20 0x7fb6dc1e40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:06.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6db400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6dc1e3c20 0x7fb6dc1e40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43880/0 (socket says 172.21.15.39:43880) 2024-01-24T17:45:06.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6db400640 1 -- 172.21.15.39:0/2311840479 learned_addr learned my addr 172.21.15.39:0/2311840479 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:06.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6daa00640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc004350 0x7fb6dc0c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:06.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6da000640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc1f32d0 0x7fb6dc1e5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:06.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6db400640 1 -- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc1f32d0 msgr2=0x7fb6dc1e5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:06.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6db400640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc1f32d0 0x7fb6dc1e5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.367+0000 7fb6db400640 1 -- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc004350 msgr2=0x7fb6dc0c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:06.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6db400640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc004350 0x7fb6dc0c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:06.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6db400640 1 -- 172.21.15.39:0/2311840479 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc1e6840 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6da000640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc1f32d0 0x7fb6dc1e5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:06.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6db400640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6dc1e3c20 0x7fb6dc1e40a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0ada80 tx=0x7fb6d4007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:06.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6d2a00640 1 -- 172.21.15.39:0/2311840479 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d400eb00 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6d2a00640 1 -- 172.21.15.39:0/2311840479 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d400ec60 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6d2a00640 1 -- 172.21.15.39:0/2311840479 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4013420 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc1e6b20 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.368+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc1e6190 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.369+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc1e6320 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.370+0000 7fb6d2a00640 1 -- 172.21.15.39:0/2311840479 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb6d4013580 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.371+0000 7fb6d2a00640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb6b406a460 0x7fb6b406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:06.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.371+0000 7fb6d2a00640 1 -- 172.21.15.39:0/2311840479 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6069+0+0 (secure 0 0 0) 0x7fb6d4088d00 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.372+0000 7fb6daa00640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb6b406a460 0x7fb6b406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:06.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.373+0000 7fb6daa00640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb6b406a460 0x7fb6b406c920 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000c00 tx=0x7fb6cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:06.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.376+0000 7fb6d2a00640 1 -- 172.21.15.39:0/2311840479 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb6dc1e6320 con 0x7fb6dc1e3c20 2024-01-24T17:45:06.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:06.573+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb6dc15e450 con 0x7fb6dc1e3c20 2024-01-24T17:45:07.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.345+0000 7fb6d2a00640 1 -- 172.21.15.39:0/2311840479 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v53) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb6dc15e450 con 0x7fb6dc1e3c20 2024-01-24T17:45:07.346 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-01-24T17:45:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb6b406a460 msgr2=0x7fb6b406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb6b406a460 0x7fb6b406c920 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000c00 tx=0x7fb6cc009040 comp rx=0 tx=0).stop 2024-01-24T17:45:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6dc1e3c20 msgr2=0x7fb6dc1e40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6dc1e3c20 0x7fb6dc1e40a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0ada80 tx=0x7fb6d4007bc0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 shutdown_connections 2024-01-24T17:45:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb6b406a460 0x7fb6b406c920 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6dc1e3c20 0x7fb6dc1e40a0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb6dc1f32d0 0x7fb6dc1e5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 --2- 172.21.15.39:0/2311840479 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb6bc004350 0x7fb6dc0c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.350+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 >> 172.21.15.39:0/2311840479 conn(0x7fb6dc0cb0e0 msgr2=0x7fb6dc0c8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.351+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 shutdown_connections 2024-01-24T17:45:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.351+0000 7fb6e1800640 1 -- 172.21.15.39:0/2311840479 wait complete. 2024-01-24T17:45:07.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-24T17:45:07.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.590+0000 7efe1bc00640 1 Processor -- start 2024-01-24T17:45:07.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.590+0000 7efe1bc00640 1 -- start start 2024-01-24T17:45:07.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.591+0000 7efe1bc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 0x7efe1406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:07.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.591+0000 7efe1bc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efe1415e190 0x7efe140b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:07.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe1bc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1406d4c0 con 0x7efe1406c9f0 2024-01-24T17:45:07.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe1bc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1406cf00 con 0x7efe1415e190 2024-01-24T17:45:07.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe1bc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1406d1e0 con 0x7efe140b0350 2024-01-24T17:45:07.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe19c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efe1415e190 0x7efe140b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:07.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe13e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efe1406c9f0 0x7efe1415da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53684/0 (socket says 172.21.15.39:53684) 2024-01-24T17:45:07.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe13e00640 1 -- 172.21.15.39:0/3266687537 learned_addr learned my addr 172.21.15.39:0/3266687537 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:07.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe19200640 1 --2- 172.21.15.39:0/3266687537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 0x7efe1406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:07.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe13400640 1 -- 172.21.15.39:0/3266687537 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914895538 0 0) 0x7efe1406d4c0 con 0x7efe1406c9f0 2024-01-24T17:45:07.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe13400640 1 -- 172.21.15.39:0/3266687537 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdf4003520 con 0x7efe1406c9f0 2024-01-24T17:45:07.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe19c00640 1 -- 172.21.15.39:0/3266687537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 msgr2=0x7efe1406c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:07.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.592+0000 7efe19c00640 1 --2- 172.21.15.39:0/3266687537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 0x7efe1406c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.593+0000 7efe19c00640 1 -- 172.21.15.39:0/3266687537 >> v1:172.21.15.39:6789/0 conn(0x7efe1406c9f0 legacy=0x7efe1415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:07.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.593+0000 7efe19c00640 1 -- 172.21.15.39:0/3266687537 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe140b15e0 con 0x7efe1415e190 2024-01-24T17:45:07.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.594+0000 7efe19c00640 1 --2- 172.21.15.39:0/3266687537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1415e190 0x7efe140b1de0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009d90 tx=0x7efe0c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93810f70ae376add server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.594+0000 7efe13400640 1 -- 172.21.15.39:0/3266687537 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c02cbd0 con 0x7efe1415e190 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.594+0000 7efe13400640 1 -- 172.21.15.39:0/3266687537 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0c02cd30 con 0x7efe1415e190 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.594+0000 7efe13400640 1 -- 172.21.15.39:0/3266687537 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c02cea0 con 0x7efe1415e190 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.595+0000 7efe1bc00640 1 -- 172.21.15.39:0/3266687537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1415e190 msgr2=0x7efe140b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.595+0000 7efe1bc00640 1 --2- 172.21.15.39:0/3266687537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1415e190 0x7efe140b1de0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009d90 tx=0x7efe0c02be20 comp rx=0 tx=0).stop 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.595+0000 7efe1bc00640 1 -- 172.21.15.39:0/3266687537 shutdown_connections 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.595+0000 7efe1bc00640 1 --2- 172.21.15.39:0/3266687537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1415e190 0x7efe140b1de0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.595+0000 7efe1bc00640 1 --2- 172.21.15.39:0/3266687537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 0x7efe1406c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.595+0000 7efe1bc00640 1 -- 172.21.15.39:0/3266687537 >> 172.21.15.39:0/3266687537 conn(0x7efe140cb0e0 msgr2=0x7efe140cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.595+0000 7efe1bc00640 1 -- 172.21.15.39:0/3266687537 shutdown_connections 2024-01-24T17:45:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.595+0000 7efe1bc00640 1 -- 172.21.15.39:0/3266687537 wait complete. 2024-01-24T17:45:07.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.597+0000 7efe1bc00640 1 Processor -- start 2024-01-24T17:45:07.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.597+0000 7efe1bc00640 1 -- start start 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe1bc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1406c9f0 0x7efe140cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe1bc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 0x7efe140ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe1bc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe1415e190 0x7efe141f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe1bc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe140cc770 con 0x7efe1406c9f0 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe1bc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe141596c0 con 0x7efe1415e190 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe1bc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe141599a0 con 0x7efe140b0350 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe19200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1406c9f0 0x7efe140cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe19200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1406c9f0 0x7efe140cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43902/0 (socket says 172.21.15.39:43902) 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe19200640 1 -- 172.21.15.39:0/1323166191 learned_addr learned my addr 172.21.15.39:0/1323166191 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe19200640 1 -- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 msgr2=0x7efe140ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:07.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe19200640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 0x7efe140ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe19200640 1 -- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe1415e190 msgr2=0x7efe141f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:07.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe19200640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe1415e190 0x7efe141f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:07.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.598+0000 7efe19200640 1 -- 172.21.15.39:0/1323166191 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c009a70 con 0x7efe1406c9f0 2024-01-24T17:45:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.599+0000 7efe19200640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1406c9f0 0x7efe140cc230 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7efe0400d800 tx=0x7efe0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.599+0000 7efe10c00640 1 -- 172.21.15.39:0/1323166191 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0400b6c0 con 0x7efe1406c9f0 2024-01-24T17:45:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.599+0000 7efe10c00640 1 -- 172.21.15.39:0/1323166191 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0400b820 con 0x7efe1406c9f0 2024-01-24T17:45:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.599+0000 7efe10c00640 1 -- 172.21.15.39:0/1323166191 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0400bbb0 con 0x7efe1406c9f0 2024-01-24T17:45:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.599+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe14159c80 con 0x7efe1406c9f0 2024-01-24T17:45:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.599+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe140ad070 con 0x7efe1406c9f0 2024-01-24T17:45:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.600+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe14073ef0 con 0x7efe1406c9f0 2024-01-24T17:45:07.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.601+0000 7efe10c00640 1 -- 172.21.15.39:0/1323166191 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efe0400bd10 con 0x7efe1406c9f0 2024-01-24T17:45:07.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.602+0000 7efe10c00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdec06a430 0x7efdec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:07.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.602+0000 7efe10c00640 1 -- 172.21.15.39:0/1323166191 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5631+0+0 (secure 0 0 0) 0x7efe04088eb0 con 0x7efe1406c9f0 2024-01-24T17:45:07.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.603+0000 7efe13e00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdec06a430 0x7efdec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:07.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.604+0000 7efe13e00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdec06a430 0x7efdec06c8f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efdfc000f60 tx=0x7efdfc0078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:07.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.607+0000 7efe10c00640 1 -- 172.21.15.39:0/1323166191 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efe04054450 con 0x7efe1406c9f0 2024-01-24T17:45:07.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:07.801+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efe14071050 con 0x7efe1406c9f0 2024-01-24T17:45:08.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.359+0000 7efe10c00640 1 -- 172.21.15.39:0/1323166191 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v54) v1 ==== 169+0+0 (secure 0 0 0) 0x7efe04057a70 con 0x7efe1406c9f0 2024-01-24T17:45:08.360 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-01-24T17:45:08.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.363+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdec06a430 msgr2=0x7efdec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:08.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.363+0000 7efe1bc00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdec06a430 0x7efdec06c8f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efdfc000f60 tx=0x7efdfc0078f0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.363+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1406c9f0 msgr2=0x7efe140cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:08.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.363+0000 7efe1bc00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1406c9f0 0x7efe140cc230 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7efe0400d800 tx=0x7efe0400dcc0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.364+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 shutdown_connections 2024-01-24T17:45:08.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.364+0000 7efe1bc00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe1415e190 0x7efe141f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.364+0000 7efe1bc00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe140b0350 0x7efe140ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.364+0000 7efe1bc00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdec06a430 0x7efdec06c8f0 secure :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efdfc000f60 tx=0x7efdfc0078f0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.364+0000 7efe1bc00640 1 --2- 172.21.15.39:0/1323166191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe1406c9f0 0x7efe140cc230 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.364+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 >> 172.21.15.39:0/1323166191 conn(0x7efe140cb0e0 msgr2=0x7efe140c20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:08.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.364+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 shutdown_connections 2024-01-24T17:45:08.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.364+0000 7efe1bc00640 1 -- 172.21.15.39:0/1323166191 wait complete. 2024-01-24T17:45:08.388 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-24T17:45:08.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.598+0000 7fb93fe00640 1 Processor -- start 2024-01-24T17:45:08.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.599+0000 7fb93fe00640 1 -- start start 2024-01-24T17:45:08.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.600+0000 7fb93fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb93815fff0 0x7fb9380b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:08.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.600+0000 7fb93fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb9380d14c0 0x7fb9380cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:08.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.600+0000 7fb93fe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9380cda90 con 0x7fb9380b2a10 2024-01-24T17:45:08.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.600+0000 7fb93fe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9380b2e90 con 0x7fb9380d14c0 2024-01-24T17:45:08.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.600+0000 7fb93fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9380d32c0 con 0x7fb93815fff0 2024-01-24T17:45:08.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.600+0000 7fb93de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb9380d14c0 0x7fb9380cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:08.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.601+0000 7fb93de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb9380d14c0 0x7fb9380cdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43906/0 (socket says 172.21.15.39:43906) 2024-01-24T17:45:08.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.601+0000 7fb93de00640 1 -- 172.21.15.39:0/2052525613 learned_addr learned my addr 172.21.15.39:0/2052525613 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:08.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.601+0000 7fb93d400640 1 --2- 172.21.15.39:0/2052525613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb93815fff0 0x7fb9380b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:08.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.601+0000 7fb93de00640 1 -- 172.21.15.39:0/2052525613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb93815fff0 msgr2=0x7fb9380b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:08.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.601+0000 7fb93de00640 1 --2- 172.21.15.39:0/2052525613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb93815fff0 0x7fb9380b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.601+0000 7fb93de00640 1 -- 172.21.15.39:0/2052525613 >> v1:172.21.15.100:6789/0 conn(0x7fb9380b2a10 legacy=0x7fb9380d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:08.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.601+0000 7fb93de00640 1 -- 172.21.15.39:0/2052525613 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9380b30a0 con 0x7fb9380d14c0 2024-01-24T17:45:08.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.603+0000 7fb93de00640 1 --2- 172.21.15.39:0/2052525613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9380d14c0 0x7fb9380cdf20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb930009870 tx=0x7fb93002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fff586aaa101b78 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:08.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.603+0000 7fb937e00640 1 -- 172.21.15.39:0/2052525613 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930031640 con 0x7fb9380d14c0 2024-01-24T17:45:08.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.603+0000 7fb937e00640 1 -- 172.21.15.39:0/2052525613 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9300317a0 con 0x7fb9380d14c0 2024-01-24T17:45:08.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.603+0000 7fb937e00640 1 -- 172.21.15.39:0/2052525613 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930031b30 con 0x7fb9380d14c0 2024-01-24T17:45:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.604+0000 7fb93fe00640 1 -- 172.21.15.39:0/2052525613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9380d14c0 msgr2=0x7fb9380cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.604+0000 7fb93fe00640 1 --2- 172.21.15.39:0/2052525613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9380d14c0 0x7fb9380cdf20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb930009870 tx=0x7fb93002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.604+0000 7fb93fe00640 1 -- 172.21.15.39:0/2052525613 shutdown_connections 2024-01-24T17:45:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.604+0000 7fb93fe00640 1 --2- 172.21.15.39:0/2052525613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9380d14c0 0x7fb9380cdf20 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.604+0000 7fb93fe00640 1 --2- 172.21.15.39:0/2052525613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb93815fff0 0x7fb9380b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.604+0000 7fb93fe00640 1 -- 172.21.15.39:0/2052525613 >> 172.21.15.39:0/2052525613 conn(0x7fb93809bcc0 msgr2=0x7fb9380d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.604+0000 7fb93fe00640 1 -- 172.21.15.39:0/2052525613 shutdown_connections 2024-01-24T17:45:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.604+0000 7fb93fe00640 1 -- 172.21.15.39:0/2052525613 wait complete. 2024-01-24T17:45:08.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.606+0000 7fb93fe00640 1 Processor -- start 2024-01-24T17:45:08.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.606+0000 7fb93fe00640 1 -- start start 2024-01-24T17:45:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9380b2a10 0x7fb938150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93fe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9380d14c0 0x7fb93806b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb93815fff0 0x7fb93806bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:08.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93809d2e0 con 0x7fb93815fff0 2024-01-24T17:45:08.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93fe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938151520 con 0x7fb9380d14c0 2024-01-24T17:45:08.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93809c6d0 con 0x7fb9380b2a10 2024-01-24T17:45:08.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb93815fff0 0x7fb93806bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:08.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb93815fff0 0x7fb93806bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43920/0 (socket says 172.21.15.39:43920) 2024-01-24T17:45:08.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93de00640 1 -- 172.21.15.39:0/3989180902 learned_addr learned my addr 172.21.15.39:0/3989180902 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:08.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93d400640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9380b2a10 0x7fb938150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:08.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93de00640 1 -- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9380b2a10 msgr2=0x7fb938150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:08.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.607+0000 7fb93de00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9380b2a10 0x7fb938150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb93de00640 1 -- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9380d14c0 msgr2=0x7fb93806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:08.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb93de00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9380d14c0 0x7fb93806b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb93de00640 1 -- 172.21.15.39:0/3989180902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300094d0 con 0x7fb93815fff0 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb93de00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb93815fff0 0x7fb93806bc60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb930009870 tx=0x7fb9300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb935600640 1 -- 172.21.15.39:0/3989180902 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93003ac60 con 0x7fb93815fff0 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb935600640 1 -- 172.21.15.39:0/3989180902 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93003ade0 con 0x7fb93815fff0 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb935600640 1 -- 172.21.15.39:0/3989180902 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930044630 con 0x7fb93815fff0 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93809c9b0 con 0x7fb93815fff0 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.608+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93809cf70 con 0x7fb93815fff0 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.610+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93809d490 con 0x7fb93815fff0 2024-01-24T17:45:08.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.611+0000 7fb935600640 1 -- 172.21.15.39:0/3989180902 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb93002f050 con 0x7fb93815fff0 2024-01-24T17:45:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.611+0000 7fb935600640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb91006a460 0x7fb91006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.612+0000 7fb935600640 1 -- 172.21.15.39:0/3989180902 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5072+0+0 (secure 0 0 0) 0x7fb9300aa570 con 0x7fb93815fff0 2024-01-24T17:45:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.612+0000 7fb93d400640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb91006a460 0x7fb91006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.613+0000 7fb93d400640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb91006a460 0x7fb91006c920 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb9380a0690 tx=0x7fb92800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:08.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.616+0000 7fb935600640 1 -- 172.21.15.39:0/3989180902 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb93809d490 con 0x7fb93815fff0 2024-01-24T17:45:08.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.822+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb938073ef0 con 0x7fb93815fff0 2024-01-24T17:45:08.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.828+0000 7fb935600640 1 -- 172.21.15.39:0/3989180902 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb930075d60 con 0x7fb93815fff0 2024-01-24T17:45:08.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.830+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb91006a460 msgr2=0x7fb91006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:08.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.830+0000 7fb93fe00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb91006a460 0x7fb91006c920 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb9380a0690 tx=0x7fb92800b040 comp rx=0 tx=0).stop 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.830+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb93815fff0 msgr2=0x7fb93806bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.830+0000 7fb93fe00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb93815fff0 0x7fb93806bc60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb930009870 tx=0x7fb9300046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.831+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 shutdown_connections 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.831+0000 7fb93fe00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb91006a460 0x7fb91006c920 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.831+0000 7fb93fe00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb93815fff0 0x7fb93806bc60 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.831+0000 7fb93fe00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9380d14c0 0x7fb93806b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.831+0000 7fb93fe00640 1 --2- 172.21.15.39:0/3989180902 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9380b2a10 0x7fb938150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.831+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 >> 172.21.15.39:0/3989180902 conn(0x7fb93809bcc0 msgr2=0x7fb93806af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.831+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 shutdown_connections 2024-01-24T17:45:08.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:08.831+0000 7fb93fe00640 1 -- 172.21.15.39:0/3989180902 wait complete. 2024-01-24T17:45:08.851 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted' 2024-01-24T17:45:09.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.070+0000 7f26d5a00640 1 Processor -- start 2024-01-24T17:45:09.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.070+0000 7f26d5a00640 1 -- start start 2024-01-24T17:45:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.071+0000 7f26d5a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f26d015a950 0x7f26d00ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.071+0000 7f26d5a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f26d00caaa0 0x7f26d00caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.071+0000 7f26d5a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d00cb870 con 0x7f26d00cb490 2024-01-24T17:45:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.071+0000 7f26d5a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d00b43d0 con 0x7f26d00caaa0 2024-01-24T17:45:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.071+0000 7f26d5a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d00c8f70 con 0x7f26d015a950 2024-01-24T17:45:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.071+0000 7f26ce000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f26d00caaa0 0x7f26d00caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.071+0000 7f26ce000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f26d00caaa0 0x7f26d00caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43922/0 (socket says 172.21.15.39:43922) 2024-01-24T17:45:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.071+0000 7f26ce000640 1 -- 172.21.15.39:0/2074165593 learned_addr learned my addr 172.21.15.39:0/2074165593 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:09.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.072+0000 7f26cea00640 1 --2- 172.21.15.39:0/2074165593 >> v2:172.21.15.100:3300/0 conn(0x7f26d015a950 0x7f26d00ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:09.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.072+0000 7f26ce000640 1 -- 172.21.15.39:0/2074165593 >> v2:172.21.15.100:3300/0 conn(0x7f26d015a950 msgr2=0x7f26d00ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:09.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.072+0000 7f26ce000640 1 --2- 172.21.15.39:0/2074165593 >> v2:172.21.15.100:3300/0 conn(0x7f26d015a950 0x7f26d00ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:09.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.072+0000 7f26ce000640 1 -- 172.21.15.39:0/2074165593 >> v1:172.21.15.39:6789/0 conn(0x7f26d00cb490 legacy=0x7f26d00affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:09.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.072+0000 7f26ce000640 1 -- 172.21.15.39:0/2074165593 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d00c9180 con 0x7f26d00caaa0 2024-01-24T17:45:09.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.073+0000 7f26ce000640 1 --2- 172.21.15.39:0/2074165593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00caaa0 0x7f26d00caf50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f26d00ce470 tx=0x7f26c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c8c44e327df7c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:09.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.073+0000 7f26cd600640 1 -- 172.21.15.39:0/2074165593 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0030860 con 0x7f26d00caaa0 2024-01-24T17:45:09.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.073+0000 7f26cd600640 1 -- 172.21.15.39:0/2074165593 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c00309c0 con 0x7f26d00caaa0 2024-01-24T17:45:09.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.073+0000 7f26cd600640 1 -- 172.21.15.39:0/2074165593 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0030d50 con 0x7f26d00caaa0 2024-01-24T17:45:09.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.074+0000 7f26d5a00640 1 -- 172.21.15.39:0/2074165593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00caaa0 msgr2=0x7f26d00caf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:09.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.074+0000 7f26d5a00640 1 --2- 172.21.15.39:0/2074165593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00caaa0 0x7f26d00caf50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f26d00ce470 tx=0x7f26c002b9a0 comp rx=0 tx=0).stop 2024-01-24T17:45:09.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.074+0000 7f26d5a00640 1 -- 172.21.15.39:0/2074165593 shutdown_connections 2024-01-24T17:45:09.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.074+0000 7f26d5a00640 1 --2- 172.21.15.39:0/2074165593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00caaa0 0x7f26d00caf50 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:09.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.074+0000 7f26d5a00640 1 --2- 172.21.15.39:0/2074165593 >> v2:172.21.15.100:3300/0 conn(0x7f26d015a950 0x7f26d00ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:09.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.074+0000 7f26d5a00640 1 -- 172.21.15.39:0/2074165593 >> 172.21.15.39:0/2074165593 conn(0x7f26d006a3e0 msgr2=0x7f26d006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:09.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.074+0000 7f26d5a00640 1 -- 172.21.15.39:0/2074165593 shutdown_connections 2024-01-24T17:45:09.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.074+0000 7f26d5a00640 1 -- 172.21.15.39:0/2074165593 wait complete. 2024-01-24T17:45:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.076+0000 7f26d5a00640 1 Processor -- start 2024-01-24T17:45:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.076+0000 7f26d5a00640 1 -- start start 2024-01-24T17:45:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26d5a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26d00caaa0 0x7f26d00adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26d5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00cb490 0x7f26d00b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26d5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26d015a950 0x7f26d00b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26d5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d00ae730 con 0x7f26d00cb490 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26d5a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d00aea10 con 0x7f26d00caaa0 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26d5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d00aecf0 con 0x7f26d015a950 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26ce000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00cb490 0x7f26d00b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26ce000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00cb490 0x7f26d00b01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43934/0 (socket says 172.21.15.39:43934) 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.077+0000 7f26ce000640 1 -- 172.21.15.39:0/4118446699 learned_addr learned my addr 172.21.15.39:0/4118446699 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26cea00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26d00caaa0 0x7f26d00adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26ce000640 1 -- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26d015a950 msgr2=0x7f26d00b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26ce000640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26d015a950 0x7f26d00b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26ce000640 1 -- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26d00caaa0 msgr2=0x7f26d00adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26ce000640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26d00caaa0 0x7f26d00adea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26ce000640 1 -- 172.21.15.39:0/4118446699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c00094d0 con 0x7f26d00cb490 2024-01-24T17:45:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26ce000640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00cb490 0x7f26d00b01c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f26c002b9e0 tx=0x7f26c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:09.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26c6a00640 1 -- 172.21.15.39:0/4118446699 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0030850 con 0x7f26d00cb490 2024-01-24T17:45:09.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26c6a00640 1 -- 172.21.15.39:0/4118446699 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c0043400 con 0x7f26d00cb490 2024-01-24T17:45:09.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26c6a00640 1 -- 172.21.15.39:0/4118446699 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c00437b0 con 0x7f26d00cb490 2024-01-24T17:45:09.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.078+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d00aefd0 con 0x7f26d00cb490 2024-01-24T17:45:09.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.079+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d00ae3e0 con 0x7f26d00cb490 2024-01-24T17:45:09.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.080+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d00af4f0 con 0x7f26d00cb490 2024-01-24T17:45:09.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.080+0000 7f26c6a00640 1 -- 172.21.15.39:0/4118446699 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f26c0043910 con 0x7f26d00cb490 2024-01-24T17:45:09.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.082+0000 7f26c6a00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f26a806a430 0x7f26a806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:09.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.082+0000 7f26c6a00640 1 -- 172.21.15.39:0/4118446699 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5072+0+0 (secure 0 0 0) 0x7f26c003e070 con 0x7f26d00cb490 2024-01-24T17:45:09.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.082+0000 7f26cea00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f26a806a430 0x7f26a806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:09.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.084+0000 7f26cea00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f26a806a430 0x7f26a806c8f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f26b800b7c0 tx=0x7f26b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:09.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.086+0000 7f26c6a00640 1 -- 172.21.15.39:0/4118446699 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f26d00af4f0 con 0x7f26d00cb490 2024-01-24T17:45:09.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:09.351+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted"]} v 0) v1 -- 0x7f26d00af4f0 con 0x7f26d00cb490 2024-01-24T17:45:10.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.359+0000 7f26c6a00640 1 -- 172.21.15.39:0/4118446699 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted"]}]=0 v200) v1 ==== 153+0+0 (secure 0 0 0) 0x7f26d00af4f0 con 0x7f26d00cb490 2024-01-24T17:45:10.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f26a806a430 msgr2=0x7f26a806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:10.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f26a806a430 0x7f26a806c8f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f26b800b7c0 tx=0x7f26b8008040 comp rx=0 tx=0).stop 2024-01-24T17:45:10.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00cb490 msgr2=0x7f26d00b01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:10.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00cb490 0x7f26d00b01c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f26c002b9e0 tx=0x7f26c0004730 comp rx=0 tx=0).stop 2024-01-24T17:45:10.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 shutdown_connections 2024-01-24T17:45:10.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f26d015a950 0x7f26d00b0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f26a806a430 0x7f26a806c8f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26d00cb490 0x7f26d00b01c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 --2- 172.21.15.39:0/4118446699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f26d00caaa0 0x7f26d00adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 >> 172.21.15.39:0/4118446699 conn(0x7f26d006a3e0 msgr2=0x7f26d00cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:10.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 shutdown_connections 2024-01-24T17:45:10.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.363+0000 7f26d5a00640 1 -- 172.21.15.39:0/4118446699 wait complete. 2024-01-24T17:45:10.386 INFO:tasks.cephfs_test_runner:Check that mds evicts blocklisted client ... ok 2024-01-24T17:45:10.387 INFO:tasks.cephfs_test_runner:Starting test: test_session_reject (tasks.cephfs.test_sessionmap.TestSessionMap) 2024-01-24T17:45:10.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_reject' 2024-01-24T17:45:10.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.607+0000 7f2a2f200640 1 Processor -- start 2024-01-24T17:45:10.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.607+0000 7f2a2f200640 1 -- start start 2024-01-24T17:45:10.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.608+0000 7f2a2f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3015d320 0x7f2a300c9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:10.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.608+0000 7f2a2f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2a300cab70 0x7f2a300cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:10.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.608+0000 7f2a2f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a300c9940 con 0x7f2a30158bd0 2024-01-24T17:45:10.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.608+0000 7f2a2f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3015d780 con 0x7f2a300cab70 2024-01-24T17:45:10.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.608+0000 7f2a2f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300c9b70 con 0x7f2a3015d320 2024-01-24T17:45:10.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.608+0000 7f2a2de00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2a30158bd0 0x7f2a300702b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53702/0 (socket says 172.21.15.39:53702) 2024-01-24T17:45:10.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.608+0000 7f2a2de00640 1 -- 172.21.15.39:0/330329776 learned_addr learned my addr 172.21.15.39:0/330329776 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:10.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.608+0000 7f2a2e800640 1 --2- 172.21.15.39:0/330329776 >> v2:172.21.15.39:3300/0 conn(0x7f2a300cab70 0x7f2a300cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:10.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.609+0000 7f2a27e00640 1 --2- 172.21.15.39:0/330329776 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3015d320 0x7f2a300c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:10.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.609+0000 7f2a2d400640 1 -- 172.21.15.39:0/330329776 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122345370 0 0) 0x7f2a300c9940 con 0x7f2a30158bd0 2024-01-24T17:45:10.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.609+0000 7f2a2d400640 1 -- 172.21.15.39:0/330329776 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a10003520 con 0x7f2a30158bd0 2024-01-24T17:45:10.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.609+0000 7f2a2e800640 1 -- 172.21.15.39:0/330329776 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3015d320 msgr2=0x7f2a300c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:10.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.609+0000 7f2a2e800640 1 --2- 172.21.15.39:0/330329776 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3015d320 0x7f2a300c9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.609+0000 7f2a2e800640 1 -- 172.21.15.39:0/330329776 >> v1:172.21.15.39:6789/0 conn(0x7f2a30158bd0 legacy=0x7f2a300702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:10.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.609+0000 7f2a2e800640 1 -- 172.21.15.39:0/330329776 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a300c9e50 con 0x7f2a300cab70 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.610+0000 7f2a2e800640 1 --2- 172.21.15.39:0/330329776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a300cab70 0x7f2a300cafc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009d90 tx=0x7f2a2802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff739f7a820ba113 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.610+0000 7f2a2d400640 1 -- 172.21.15.39:0/330329776 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2802cc50 con 0x7f2a300cab70 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2d400640 1 -- 172.21.15.39:0/330329776 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2802cdb0 con 0x7f2a300cab70 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2d400640 1 -- 172.21.15.39:0/330329776 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28034640 con 0x7f2a300cab70 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2f200640 1 -- 172.21.15.39:0/330329776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a300cab70 msgr2=0x7f2a300cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2f200640 1 --2- 172.21.15.39:0/330329776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a300cab70 0x7f2a300cafc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009d90 tx=0x7f2a2802bea0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2f200640 1 -- 172.21.15.39:0/330329776 shutdown_connections 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2f200640 1 --2- 172.21.15.39:0/330329776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a300cab70 0x7f2a300cafc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2f200640 1 --2- 172.21.15.39:0/330329776 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a3015d320 0x7f2a300c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2f200640 1 -- 172.21.15.39:0/330329776 >> 172.21.15.39:0/330329776 conn(0x7f2a3006a2a0 msgr2=0x7f2a300afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2f200640 1 -- 172.21.15.39:0/330329776 shutdown_connections 2024-01-24T17:45:10.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.611+0000 7f2a2f200640 1 -- 172.21.15.39:0/330329776 wait complete. 2024-01-24T17:45:10.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.613+0000 7f2a2f200640 1 Processor -- start 2024-01-24T17:45:10.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.613+0000 7f2a2f200640 1 -- start start 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a2f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a300cab70 0x7f2a300af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a2f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a30158bd0 0x7f2a301460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a2f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3015d320 0x7f2a30146620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a2f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300b0500 con 0x7f2a30158bd0 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a2f200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300af8f0 con 0x7f2a3015d320 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a2f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300afbd0 con 0x7f2a300cab70 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a27e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a30158bd0 0x7f2a301460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a27e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a30158bd0 0x7f2a301460e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43962/0 (socket says 172.21.15.39:43962) 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.614+0000 7f2a27e00640 1 -- 172.21.15.39:0/2725242513 learned_addr learned my addr 172.21.15.39:0/2725242513 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:10.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a2de00640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a300cab70 0x7f2a300af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a2e800640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3015d320 0x7f2a30146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a27e00640 1 -- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a300cab70 msgr2=0x7f2a300af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a27e00640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a300cab70 0x7f2a300af0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a27e00640 1 -- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3015d320 msgr2=0x7f2a30146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a27e00640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3015d320 0x7f2a30146620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a27e00640 1 -- 172.21.15.39:0/2725242513 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28009a70 con 0x7f2a30158bd0 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a2e800640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3015d320 0x7f2a30146620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a2de00640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a300cab70 0x7f2a300af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:10.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a27e00640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a30158bd0 0x7f2a301460e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800d5f0 tx=0x7f2a1800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:10.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a26000640 1 -- 172.21.15.39:0/2725242513 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a180125f0 con 0x7f2a30158bd0 2024-01-24T17:45:10.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.615+0000 7f2a26000640 1 -- 172.21.15.39:0/2725242513 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a18012750 con 0x7f2a30158bd0 2024-01-24T17:45:10.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.616+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a300afeb0 con 0x7f2a30158bd0 2024-01-24T17:45:10.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.616+0000 7f2a26000640 1 -- 172.21.15.39:0/2725242513 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18012b00 con 0x7f2a30158bd0 2024-01-24T17:45:10.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.616+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a300af600 con 0x7f2a30158bd0 2024-01-24T17:45:10.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.617+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3006e130 con 0x7f2a30158bd0 2024-01-24T17:45:10.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.618+0000 7f2a26000640 1 -- 172.21.15.39:0/2725242513 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2a18012c60 con 0x7f2a30158bd0 2024-01-24T17:45:10.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.619+0000 7f2a26000640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a0806a430 0x7f2a0806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:10.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.619+0000 7f2a26000640 1 -- 172.21.15.39:0/2725242513 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5072+0+0 (secure 0 0 0) 0x7f2a18088b60 con 0x7f2a30158bd0 2024-01-24T17:45:10.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.620+0000 7f2a2de00640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a0806a430 0x7f2a0806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:10.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.621+0000 7f2a2de00640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a0806a430 0x7f2a0806c8f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2a20005880 tx=0x7f2a20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:10.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.624+0000 7f2a26000640 1 -- 172.21.15.39:0/2725242513 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2a180543d0 con 0x7f2a30158bd0 2024-01-24T17:45:10.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:10.886+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_reject"]} v 0) v1 -- 0x7f2a3004dff0 con 0x7f2a30158bd0 2024-01-24T17:45:11.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.365+0000 7f2a26000640 1 -- 172.21.15.39:0/2725242513 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_reject"]}]=0 v201) v1 ==== 145+0+0 (secure 0 0 0) 0x7f2a180579f0 con 0x7f2a30158bd0 2024-01-24T17:45:11.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.368+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a0806a430 msgr2=0x7f2a0806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:11.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.368+0000 7f2a2f200640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a0806a430 0x7f2a0806c8f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2a20005880 tx=0x7f2a20009040 comp rx=0 tx=0).stop 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.368+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a30158bd0 msgr2=0x7f2a301460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.368+0000 7f2a2f200640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a30158bd0 0x7f2a301460e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800d5f0 tx=0x7f2a1800dab0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.369+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 shutdown_connections 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.369+0000 7f2a2f200640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a3015d320 0x7f2a30146620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.369+0000 7f2a2f200640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2a0806a430 0x7f2a0806c8f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.369+0000 7f2a2f200640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a30158bd0 0x7f2a301460e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.369+0000 7f2a2f200640 1 --2- 172.21.15.39:0/2725242513 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2a300cab70 0x7f2a300af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.369+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 >> 172.21.15.39:0/2725242513 conn(0x7f2a3006a2a0 msgr2=0x7f2a300710c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.369+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 shutdown_connections 2024-01-24T17:45:11.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.369+0000 7f2a2f200640 1 -- 172.21.15.39:0/2725242513 wait complete. 2024-01-24T17:45:11.391 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-24T17:45:11.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.610+0000 7f4eede00640 1 Processor -- start 2024-01-24T17:45:11.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.610+0000 7f4eede00640 1 -- start start 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4eede00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4eede00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4ee80cd810 0x7f4ee80cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4eede00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee806ef10 con 0x7f4ee815df80 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4eede00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee80d1670 con 0x7f4ee80cd810 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4eede00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee80cb350 con 0x7f4ee806f9b0 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4eed400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4ee80cd810 0x7f4ee80cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4eeca00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4ee815df80 0x7f4ee80cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53710/0 (socket says 172.21.15.39:53710) 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4eeca00640 1 -- 172.21.15.39:0/2807552507 learned_addr learned my addr 172.21.15.39:0/2807552507 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.612+0000 7f4ee7e00640 1 --2- 172.21.15.39:0/2807552507 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.613+0000 7f4eed400640 1 -- 172.21.15.39:0/2807552507 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 msgr2=0x7f4ee80cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.613+0000 7f4eed400640 1 --2- 172.21.15.39:0/2807552507 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.613+0000 7f4eed400640 1 -- 172.21.15.39:0/2807552507 >> v1:172.21.15.39:6789/0 conn(0x7f4ee815df80 legacy=0x7f4ee80cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:11.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.613+0000 7f4ee7400640 1 -- 172.21.15.39:0/2807552507 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193765428 0 0) 0x7f4ee806ef10 con 0x7f4ee815df80 2024-01-24T17:45:11.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.613+0000 7f4eed400640 1 -- 172.21.15.39:0/2807552507 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee806f160 con 0x7f4ee80cd810 2024-01-24T17:45:11.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.613+0000 7f4ee7e00640 1 --2- 172.21.15.39:0/2807552507 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:45:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.614+0000 7f4eed400640 1 --2- 172.21.15.39:0/2807552507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee80cd810 0x7f4ee80cbd20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009d90 tx=0x7f4ee002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b47c6a23d98cd9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.614+0000 7f4ee7400640 1 -- 172.21.15.39:0/2807552507 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee002cc50 con 0x7f4ee80cd810 2024-01-24T17:45:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4ee7400640 1 -- 172.21.15.39:0/2807552507 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee002cdb0 con 0x7f4ee80cd810 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4ee7400640 1 -- 172.21.15.39:0/2807552507 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0034660 con 0x7f4ee80cd810 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4eede00640 1 -- 172.21.15.39:0/2807552507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee80cd810 msgr2=0x7f4ee80cbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4eede00640 1 --2- 172.21.15.39:0/2807552507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee80cd810 0x7f4ee80cbd20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009d90 tx=0x7f4ee002bea0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4eede00640 1 -- 172.21.15.39:0/2807552507 shutdown_connections 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4eede00640 1 --2- 172.21.15.39:0/2807552507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee80cd810 0x7f4ee80cbd20 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4eede00640 1 --2- 172.21.15.39:0/2807552507 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4eede00640 1 -- 172.21.15.39:0/2807552507 >> 172.21.15.39:0/2807552507 conn(0x7f4ee8089fd0 msgr2=0x7f4ee80d06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4eede00640 1 -- 172.21.15.39:0/2807552507 shutdown_connections 2024-01-24T17:45:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.615+0000 7f4eede00640 1 -- 172.21.15.39:0/2807552507 wait complete. 2024-01-24T17:45:11.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.617+0000 7f4eede00640 1 Processor -- start 2024-01-24T17:45:11.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.617+0000 7f4eede00640 1 -- start start 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eede00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eede00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ee80cd810 0x7f4ee8159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eede00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee815df80 0x7f4ee80b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eede00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee80c7060 con 0x7f4ee815df80 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eede00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee80c4c20 con 0x7f4ee80cd810 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eede00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee80c4360 con 0x7f4ee806f9b0 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eed400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee815df80 0x7f4ee80b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eed400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee815df80 0x7f4ee80b4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43970/0 (socket says 172.21.15.39:43970) 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.618+0000 7f4eed400640 1 -- 172.21.15.39:0/2930010196 learned_addr learned my addr 172.21.15.39:0/2930010196 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4eeca00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4ee7e00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ee80cd810 0x7f4ee8159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:11.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4eed400640 1 -- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 msgr2=0x7f4ee80c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4eed400640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80c6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4eed400640 1 -- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ee80cd810 msgr2=0x7f4ee8159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4eed400640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ee80cd810 0x7f4ee8159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4eed400640 1 -- 172.21.15.39:0/2930010196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0009a70 con 0x7f4ee815df80 2024-01-24T17:45:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4eeca00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.619+0000 7f4eed400640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee815df80 0x7f4ee80b4ba0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009eb0 tx=0x7f4ee0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.620+0000 7f4ee4c00640 1 -- 172.21.15.39:0/2930010196 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0003b90 con 0x7f4ee815df80 2024-01-24T17:45:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.620+0000 7f4ee4c00640 1 -- 172.21.15.39:0/2930010196 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee0003cf0 con 0x7f4ee815df80 2024-01-24T17:45:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.620+0000 7f4ee4c00640 1 -- 172.21.15.39:0/2930010196 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0004270 con 0x7f4ee815df80 2024-01-24T17:45:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.620+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee81f6320 con 0x7f4ee815df80 2024-01-24T17:45:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.620+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee80c48b0 con 0x7f4ee815df80 2024-01-24T17:45:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.621+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee80c4db0 con 0x7f4ee815df80 2024-01-24T17:45:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.622+0000 7f4ee4c00640 1 -- 172.21.15.39:0/2930010196 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4ee003b050 con 0x7f4ee815df80 2024-01-24T17:45:11.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.623+0000 7f4ee4c00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4ec006a430 0x7f4ec006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:11.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.623+0000 7f4ee4c00640 1 -- 172.21.15.39:0/2930010196 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5072+0+0 (secure 0 0 0) 0x7f4ee00aac10 con 0x7f4ee815df80 2024-01-24T17:45:11.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.623+0000 7f4eeca00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4ec006a430 0x7f4ec006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:11.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.625+0000 7f4eeca00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4ec006a430 0x7f4ec006c8f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80b2a00 tx=0x7f4ed8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:11.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.627+0000 7f4ee4c00640 1 -- 172.21.15.39:0/2930010196 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4ee0076490 con 0x7f4ee815df80 2024-01-24T17:45:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.823+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4ee8067040 con 0x7f4ee815df80 2024-01-24T17:45:11.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.829+0000 7f4ee4c00640 1 -- 172.21.15.39:0/2930010196 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4ee0079ab0 con 0x7f4ee815df80 2024-01-24T17:45:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4ec006a430 msgr2=0x7f4ec006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4ec006a430 0x7f4ec006c8f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80b2a00 tx=0x7f4ed8009040 comp rx=0 tx=0).stop 2024-01-24T17:45:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee815df80 msgr2=0x7f4ee80b4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee815df80 0x7f4ee80b4ba0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009eb0 tx=0x7f4ee0004a40 comp rx=0 tx=0).stop 2024-01-24T17:45:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 shutdown_connections 2024-01-24T17:45:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4ec006a430 0x7f4ec006c8f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ee815df80 0x7f4ee80b4ba0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4ee80cd810 0x7f4ee8159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 --2- 172.21.15.39:0/2930010196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4ee806f9b0 0x7f4ee80c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 >> 172.21.15.39:0/2930010196 conn(0x7f4ee8089fd0 msgr2=0x7f4ee80af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 shutdown_connections 2024-01-24T17:45:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:11.832+0000 7f4eede00640 1 -- 172.21.15.39:0/2930010196 wait complete. 2024-01-24T17:45:11.853 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:45:11.853 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:45:11.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:45:12.082 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:45:12.082 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:45:12.082 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:45:12.142 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:45:12.142 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:45:12.142 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:45:12.199 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:45:12.200 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:45:12.200 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:45:12.260 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:45:12.260 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:45:12.260 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:45:12.317 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:45:12.317 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-24T17:45:12.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:45:12.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.547+0000 7faf3be00640 1 Processor -- start 2024-01-24T17:45:12.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.547+0000 7faf3be00640 1 -- start start 2024-01-24T17:45:12.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7faf3c0ca670 0x7faf3c0cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:12.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c06bde0 0x7faf3c15a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:12.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faf3c15a9b0 0x7faf3c0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:12.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3c06c260 con 0x7faf3c15a9b0 2024-01-24T17:45:12.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3be00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf3c0cd050 con 0x7faf3c0ca670 2024-01-24T17:45:12.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c159fe0 con 0x7faf3c06bde0 2024-01-24T17:45:12.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faf3c15a9b0 0x7faf3c0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:12.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faf3c15a9b0 0x7faf3c0cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43980/0 (socket says 172.21.15.39:43980) 2024-01-24T17:45:12.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3b400640 1 -- 172.21.15.39:0/2294141422 learned_addr learned my addr 172.21.15.39:0/2294141422 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:12.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf3aa00640 1 --2- 172.21.15.39:0/2294141422 >> v2:172.21.15.100:3300/0 conn(0x7faf3c0ca670 0x7faf3c0cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:12.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.549+0000 7faf33e00640 1 --2- 172.21.15.39:0/2294141422 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c06bde0 0x7faf3c15a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:12.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.550+0000 7faf3b400640 1 -- 172.21.15.39:0/2294141422 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c06bde0 msgr2=0x7faf3c15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:12.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.550+0000 7faf3b400640 1 --2- 172.21.15.39:0/2294141422 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c06bde0 0x7faf3c15a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.550+0000 7faf3b400640 1 -- 172.21.15.39:0/2294141422 >> v2:172.21.15.100:3300/0 conn(0x7faf3c0ca670 msgr2=0x7faf3c0cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:12.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.550+0000 7faf3b400640 1 --2- 172.21.15.39:0/2294141422 >> v2:172.21.15.100:3300/0 conn(0x7faf3c0ca670 0x7faf3c0cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.550+0000 7faf3b400640 1 -- 172.21.15.39:0/2294141422 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c0caa50 con 0x7faf3c15a9b0 2024-01-24T17:45:12.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.551+0000 7faf3b400640 1 --2- 172.21.15.39:0/2294141422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c15a9b0 0x7faf3c0cb2b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0cbf40 tx=0x7faf3402e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b5629c0fe6c69 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.551+0000 7faf3a000640 1 -- 172.21.15.39:0/2294141422 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3402ea00 con 0x7faf3c15a9b0 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.551+0000 7faf3a000640 1 -- 172.21.15.39:0/2294141422 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3402eb60 con 0x7faf3c15a9b0 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.551+0000 7faf3a000640 1 -- 172.21.15.39:0/2294141422 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf34009600 con 0x7faf3c15a9b0 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 -- 172.21.15.39:0/2294141422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c15a9b0 msgr2=0x7faf3c0cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 --2- 172.21.15.39:0/2294141422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c15a9b0 0x7faf3c0cb2b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0cbf40 tx=0x7faf3402e060 comp rx=0 tx=0).stop 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 -- 172.21.15.39:0/2294141422 shutdown_connections 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 --2- 172.21.15.39:0/2294141422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c15a9b0 0x7faf3c0cb2b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 --2- 172.21.15.39:0/2294141422 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c06bde0 0x7faf3c15a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 --2- 172.21.15.39:0/2294141422 >> v2:172.21.15.100:3300/0 conn(0x7faf3c0ca670 0x7faf3c0cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 -- 172.21.15.39:0/2294141422 >> 172.21.15.39:0/2294141422 conn(0x7faf3c06a090 msgr2=0x7faf3c06a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 -- 172.21.15.39:0/2294141422 shutdown_connections 2024-01-24T17:45:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.552+0000 7faf3be00640 1 -- 172.21.15.39:0/2294141422 wait complete. 2024-01-24T17:45:12.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.554+0000 7faf3be00640 1 Processor -- start 2024-01-24T17:45:12.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.554+0000 7faf3be00640 1 -- start start 2024-01-24T17:45:12.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf3be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf3c06bde0 0x7faf3c1e6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:12.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf3be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c0ca670 0x7faf3c0c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:12.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf3be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c0c0d50 0x7faf3c1583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:12.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf3be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c1546b0 con 0x7faf3c0ca670 2024-01-24T17:45:12.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf3be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c0c1200 con 0x7faf3c06bde0 2024-01-24T17:45:12.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf3be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c1e4230 con 0x7faf3c0c0d50 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf33e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c0ca670 0x7faf3c0c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf33e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c0ca670 0x7faf3c0c0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43992/0 (socket says 172.21.15.39:43992) 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf33e00640 1 -- 172.21.15.39:0/1656409734 learned_addr learned my addr 172.21.15.39:0/1656409734 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf3aa00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf3c06bde0 0x7faf3c1e6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.555+0000 7faf3b400640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c0c0d50 0x7faf3c1583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf33e00640 1 -- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c0c0d50 msgr2=0x7faf3c1583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf33e00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c0c0d50 0x7faf3c1583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf33e00640 1 -- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf3c06bde0 msgr2=0x7faf3c1e6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf33e00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf3c06bde0 0x7faf3c1e6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf33e00640 1 -- 172.21.15.39:0/1656409734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3400bd40 con 0x7faf3c0ca670 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf3b400640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c0c0d50 0x7faf3c1583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf3aa00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf3c06bde0 0x7faf3c1e6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf33e00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c0ca670 0x7faf3c0c0810 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7faf2400d830 tx=0x7faf2400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:12.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf32a00640 1 -- 172.21.15.39:0/1656409734 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2400b6c0 con 0x7faf3c0ca670 2024-01-24T17:45:12.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf32a00640 1 -- 172.21.15.39:0/1656409734 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf2400b820 con 0x7faf3c0ca670 2024-01-24T17:45:12.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf32a00640 1 -- 172.21.15.39:0/1656409734 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2400bbb0 con 0x7faf3c0ca670 2024-01-24T17:45:12.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3c159140 con 0x7faf3c0ca670 2024-01-24T17:45:12.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.556+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3c159700 con 0x7faf3c0ca670 2024-01-24T17:45:12.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.558+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3c1547f0 con 0x7faf3c0ca670 2024-01-24T17:45:12.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.558+0000 7faf32a00640 1 -- 172.21.15.39:0/1656409734 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7faf2401e460 con 0x7faf3c0ca670 2024-01-24T17:45:12.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.559+0000 7faf32a00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf1406a430 0x7faf1406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:12.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.559+0000 7faf32a00640 1 -- 172.21.15.39:0/1656409734 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5072+0+0 (secure 0 0 0) 0x7faf24088e90 con 0x7faf3c0ca670 2024-01-24T17:45:12.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.560+0000 7faf3aa00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf1406a430 0x7faf1406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:12.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.561+0000 7faf3aa00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf1406a430 0x7faf1406c8f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7faf2c006fd0 tx=0x7faf2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:12.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.564+0000 7faf32a00640 1 -- 172.21.15.39:0/1656409734 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7faf3c1547f0 con 0x7faf3c0ca670 2024-01-24T17:45:12.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.799+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf3c1547f0 con 0x7faf3c0ca670 2024-01-24T17:45:12.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.800+0000 7faf32a00640 1 -- 172.21.15.39:0/1656409734 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3535 (secure 0 0 0) 0x7faf2408ade0 con 0x7faf3c0ca670 2024-01-24T17:45:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:12.801 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/1925481596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":1925481596},{"type":"v1","addr":"172.21.15.100:6836","nonce":1925481596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2843554867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2843554867},{"type":"v1","addr":"172.21.15.39:6834","nonce":2843554867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18},{"gid":5458,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1095519433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1095519433},{"type":"v1","addr":"172.21.15.100:6837","nonce":1095519433}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":6177,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/1316090337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":1316090337},{"type":"v1","addr":"172.21.15.39:6835","nonce":1316090337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[]} 2024-01-24T17:45:12.801 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 29 2024-01-24T17:45:12.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf1406a430 msgr2=0x7faf1406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:12.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf1406a430 0x7faf1406c8f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7faf2c006fd0 tx=0x7faf2c008040 comp rx=0 tx=0).stop 2024-01-24T17:45:12.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c0ca670 msgr2=0x7faf3c0c0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:12.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c0ca670 0x7faf3c0c0810 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7faf2400d830 tx=0x7faf2400dcf0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 shutdown_connections 2024-01-24T17:45:12.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faf3c0c0d50 0x7faf3c1583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf1406a430 0x7faf1406c8f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf3c0ca670 0x7faf3c0c0810 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 --2- 172.21.15.39:0/1656409734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf3c06bde0 0x7faf3c1e6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:12.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.803+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 >> 172.21.15.39:0/1656409734 conn(0x7faf3c06a090 msgr2=0x7faf3c15c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:12.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.804+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 shutdown_connections 2024-01-24T17:45:12.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:12.804+0000 7faf3be00640 1 -- 172.21.15.39:0/1656409734 wait complete. 2024-01-24T17:45:12.824 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-24T17:45:12.824 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-24T17:45:12.825 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-24T17:45:12.825 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:45:12.825 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-24T17:45:12.825 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-24T17:45:12.825 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-24T17:45:12.825 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:45:12.826 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-24T17:45:12.826 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-24T17:45:12.826 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-24T17:45:12.826 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:45:12.826 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-24T17:45:12.826 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-24T17:45:12.827 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-24T17:45:12.827 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:45:12.844 INFO:tasks.ceph.mds.a:Stopped 2024-01-24T17:45:12.844 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-24T17:45:12.847 INFO:tasks.ceph.mds.d:Stopped 2024-01-24T17:45:12.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-24T17:45:12.848 INFO:tasks.ceph.mds.c:Stopped 2024-01-24T17:45:12.849 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-24T17:45:12.850 INFO:tasks.ceph.mds.b:Stopped 2024-01-24T17:45:12.850 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-24T17:45:12.895 INFO:tasks.ceph.mds.a:Started 2024-01-24T17:45:12.896 DEBUG:teuthology.parallel:result is None 2024-01-24T17:45:12.897 INFO:tasks.ceph.mds.d:Started 2024-01-24T17:45:12.897 DEBUG:teuthology.parallel:result is None 2024-01-24T17:45:12.902 INFO:tasks.ceph.mds.c:Started 2024-01-24T17:45:12.902 DEBUG:teuthology.parallel:result is None 2024-01-24T17:45:12.903 INFO:tasks.ceph.mds.b:Started 2024-01-24T17:45:12.903 DEBUG:teuthology.parallel:result is None 2024-01-24T17:45:12.903 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-24T17:45:13.053 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:45:13.057 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:45:13.059 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:45:13.059 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:45:13.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.097+0000 7f9f41c00640 1 Processor -- start 2024-01-24T17:45:13.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.097+0000 7f9f41c00640 1 -- start start 2024-01-24T17:45:13.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.098+0000 7f9f41c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9f3c0cb610 0x7f9f3c0c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:13.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.098+0000 7f9f41c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0c9c10 con 0x7f9f3c0c8b90 2024-01-24T17:45:13.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.098+0000 7f9f41c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0cd590 con 0x7f9f3c15ca30 2024-01-24T17:45:13.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.098+0000 7f9f41c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f3c0c9650 con 0x7f9f3c0cb610 2024-01-24T17:45:13.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.098+0000 7f9f3b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9f3c0cb610 0x7f9f3c0c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:13.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.098+0000 7f9f3b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9f3c0cb610 0x7f9f3c0c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51224/0 (socket says 172.21.15.39:51224) 2024-01-24T17:45:13.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.098+0000 7f9f3b400640 1 -- 172.21.15.39:0/979455686 learned_addr learned my addr 172.21.15.39:0/979455686 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:13.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.099+0000 7f9f3b400640 1 -- 172.21.15.39:0/979455686 >> v1:172.21.15.100:6789/0 conn(0x7f9f3c15ca30 legacy=0x7f9f3c0cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:13.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.099+0000 7f9f3b400640 1 -- 172.21.15.39:0/979455686 >> v1:172.21.15.39:6789/0 conn(0x7f9f3c0c8b90 legacy=0x7f9f3c0caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:13.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.099+0000 7f9f3b400640 1 -- 172.21.15.39:0/979455686 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c0c9930 con 0x7f9f3c0cb610 2024-01-24T17:45:13.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.100+0000 7f9f3b400640 1 --2- 172.21.15.39:0/979455686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c0cb610 0x7f9f3c0c8ff0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009e50 tx=0x7f9f3402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0947ed6e7d82600 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:13.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.100+0000 7f9f3a000640 1 -- 172.21.15.39:0/979455686 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3402cbb0 con 0x7f9f3c0cb610 2024-01-24T17:45:13.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.100+0000 7f9f3a000640 1 -- 172.21.15.39:0/979455686 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3402cd10 con 0x7f9f3c0cb610 2024-01-24T17:45:13.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.100+0000 7f9f3a000640 1 -- 172.21.15.39:0/979455686 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f34034610 con 0x7f9f3c0cb610 2024-01-24T17:45:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.101+0000 7f9f41c00640 1 -- 172.21.15.39:0/979455686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c0cb610 msgr2=0x7f9f3c0c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.101+0000 7f9f41c00640 1 --2- 172.21.15.39:0/979455686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c0cb610 0x7f9f3c0c8ff0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009e50 tx=0x7f9f3402c010 comp rx=0 tx=0).stop 2024-01-24T17:45:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.101+0000 7f9f41c00640 1 -- 172.21.15.39:0/979455686 shutdown_connections 2024-01-24T17:45:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.101+0000 7f9f41c00640 1 --2- 172.21.15.39:0/979455686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c0cb610 0x7f9f3c0c8ff0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.101+0000 7f9f41c00640 1 -- 172.21.15.39:0/979455686 >> 172.21.15.39:0/979455686 conn(0x7f9f3c06a3e0 msgr2=0x7f9f3c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.101+0000 7f9f41c00640 1 -- 172.21.15.39:0/979455686 shutdown_connections 2024-01-24T17:45:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.101+0000 7f9f41c00640 1 -- 172.21.15.39:0/979455686 wait complete. 2024-01-24T17:45:13.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.104+0000 7f9f41c00640 1 Processor -- start 2024-01-24T17:45:13.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.104+0000 7f9f41c00640 1 -- start start 2024-01-24T17:45:13.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f41c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f3c0c8b90 0x7f9f3c0afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:13.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f41c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f3c0cb610 0x7f9f3c0c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:13.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f41c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c15ca30 0x7f9f3c14e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:13.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f41c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c0afa30 con 0x7f9f3c15ca30 2024-01-24T17:45:13.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f41c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c0aedb0 con 0x7f9f3c0cb610 2024-01-24T17:45:13.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f41c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3c0af090 con 0x7f9f3c0c8b90 2024-01-24T17:45:13.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c15ca30 0x7f9f3c14e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c15ca30 0x7f9f3c14e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51240/0 (socket says 172.21.15.39:51240) 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3b400640 1 -- 172.21.15.39:0/1325669111 learned_addr learned my addr 172.21.15.39:0/1325669111 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3aa00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f3c0c8b90 0x7f9f3c0afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f31e00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f3c0cb610 0x7f9f3c0c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3b400640 1 -- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f3c0c8b90 msgr2=0x7f9f3c0afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3b400640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f3c0c8b90 0x7f9f3c0afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3b400640 1 -- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f3c0cb610 msgr2=0x7f9f3c0c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3b400640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f3c0cb610 0x7f9f3c0c7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.105+0000 7f9f3b400640 1 -- 172.21.15.39:0/1325669111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f34009b30 con 0x7f9f3c15ca30 2024-01-24T17:45:13.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.106+0000 7f9f3b400640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c15ca30 0x7f9f3c14e570 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009e50 tx=0x7f9f34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:13.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.106+0000 7f9f33400640 1 -- 172.21.15.39:0/1325669111 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f34009510 con 0x7f9f3c15ca30 2024-01-24T17:45:13.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.106+0000 7f9f33400640 1 -- 172.21.15.39:0/1325669111 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3403e450 con 0x7f9f3c15ca30 2024-01-24T17:45:13.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.106+0000 7f9f33400640 1 -- 172.21.15.39:0/1325669111 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3403e800 con 0x7f9f3c15ca30 2024-01-24T17:45:13.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.106+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3c0af370 con 0x7f9f3c15ca30 2024-01-24T17:45:13.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.106+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3c0b0390 con 0x7f9f3c15ca30 2024-01-24T17:45:13.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.107+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3c0afb70 con 0x7f9f3c15ca30 2024-01-24T17:45:13.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.108+0000 7f9f33400640 1 -- 172.21.15.39:0/1325669111 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9f3403e960 con 0x7f9f3c15ca30 2024-01-24T17:45:13.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.109+0000 7f9f33400640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f1406a460 0x7f9f1406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:13.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.109+0000 7f9f33400640 1 -- 172.21.15.39:0/1325669111 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5072+0+0 (secure 0 0 0) 0x7f9f340ab390 con 0x7f9f3c15ca30 2024-01-24T17:45:13.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.110+0000 7f9f3aa00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f1406a460 0x7f9f1406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:13.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.110+0000 7f9f3aa00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f1406a460 0x7f9f1406c920 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c001780 tx=0x7f9f2c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:13.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.113+0000 7f9f33400640 1 -- 172.21.15.39:0/1325669111 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9f3c0afb70 con 0x7f9f3c15ca30 2024-01-24T17:45:13.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:13.323+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9f3c0afb70 con 0x7f9f3c15ca30 2024-01-24T17:45:14.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.097+0000 7f9f33400640 1 -- 172.21.15.39:0/1325669111 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v59) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9f3c0afb70 con 0x7f9f3c15ca30 2024-01-24T17:45:14.098 INFO:teuthology.orchestra.run.smithi039.stderr: removed all blocklist entries 2024-01-24T17:45:14.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.100+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f1406a460 msgr2=0x7f9f1406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.100+0000 7f9f41c00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f1406a460 0x7f9f1406c920 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c001780 tx=0x7f9f2c009040 comp rx=0 tx=0).stop 2024-01-24T17:45:14.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.100+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c15ca30 msgr2=0x7f9f3c14e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.100+0000 7f9f41c00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c15ca30 0x7f9f3c14e570 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009e50 tx=0x7f9f34004a40 comp rx=0 tx=0).stop 2024-01-24T17:45:14.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.100+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 shutdown_connections 2024-01-24T17:45:14.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.101+0000 7f9f41c00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f1406a460 0x7f9f1406c920 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.101+0000 7f9f41c00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f3c15ca30 0x7f9f3c14e570 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.101+0000 7f9f41c00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f3c0cb610 0x7f9f3c0c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.101+0000 7f9f41c00640 1 --2- 172.21.15.39:0/1325669111 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f3c0c8b90 0x7f9f3c0afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.101+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 >> 172.21.15.39:0/1325669111 conn(0x7f9f3c06a3e0 msgr2=0x7f9f3c0caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:14.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.101+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 shutdown_connections 2024-01-24T17:45:14.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.101+0000 7f9f41c00640 1 -- 172.21.15.39:0/1325669111 wait complete. 2024-01-24T17:45:14.123 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-24T17:45:14.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.348+0000 7f20c5000640 1 Processor -- start 2024-01-24T17:45:14.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.348+0000 7f20c5000640 1 -- start start 2024-01-24T17:45:14.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.350+0000 7f20c5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00cb020 0x7f20c015ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.350+0000 7f20c5000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f20c00cb400 0x7f20c00ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.350+0000 7f20c5000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c015b240 con 0x7f20c00c8c10 2024-01-24T17:45:14.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.350+0000 7f20c5000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c015b520 con 0x7f20c00cb400 2024-01-24T17:45:14.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.350+0000 7f20c5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c015b800 con 0x7f20c00cb020 2024-01-24T17:45:14.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.350+0000 7f20bfe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f20c00c8c10 0x7f20c015e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56098/0 (socket says 172.21.15.39:56098) 2024-01-24T17:45:14.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.350+0000 7f20bfe00640 1 -- 172.21.15.39:0/450267363 learned_addr learned my addr 172.21.15.39:0/450267363 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:14.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20bf400640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00cb020 0x7f20c015ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20bea00640 1 --2- 172.21.15.39:0/450267363 >> v2:172.21.15.100:3300/0 conn(0x7f20c00cb400 0x7f20c00ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653764830 0 0) 0x7f20c015b240 con 0x7f20c00c8c10 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a0003520 con 0x7f20c00c8c10 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649685062 0 0) 0x7f20a0003520 con 0x7f20c00c8c10 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00cb020 msgr2=0x7f20c015ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00cb020 0x7f20c015ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 >> v2:172.21.15.100:3300/0 conn(0x7f20c00cb400 msgr2=0x7f20c00ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 --2- 172.21.15.39:0/450267363 >> v2:172.21.15.100:3300/0 conn(0x7f20c00cb400 0x7f20c00ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c015bae0 con 0x7f20c00c8c10 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f20b80019f0 con 0x7f20c00c8c10 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 >> v1:172.21.15.39:6789/0 conn(0x7f20c00c8c10 legacy=0x7f20c015e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0003970 0x7f20a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00cb020 0x7f20a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.351+0000 7f20be000640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a0004d00 0x7f20a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c015b520 con 0x7f20a0004d00 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c015b800 con 0x7f20a0003970 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a000b980 con 0x7f20c00cb020 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bfe00640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a0004d00 0x7f20a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bea00640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00cb020 0x7f20a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bf400640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0003970 0x7f20a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bfe00640 1 -- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00cb020 msgr2=0x7f20a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bfe00640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00cb020 0x7f20a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bfe00640 1 -- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0003970 msgr2=0x7f20a0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bfe00640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0003970 0x7f20a0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bfe00640 1 -- 172.21.15.39:0/450267363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a0003520 con 0x7f20a0004d00 2024-01-24T17:45:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.352+0000 7f20bf400640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0003970 0x7f20a0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.353+0000 7f20bfe00640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a0004d00 0x7f20a0005180 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f20b800a860 tx=0x7f20a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd1c99facd4aebc server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b8034630 con 0x7f20a0004d00 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b802f070 con 0x7f20a0004d00 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20be000640 1 -- 172.21.15.39:0/450267363 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b80349f0 con 0x7f20a0004d00 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20c5000640 1 -- 172.21.15.39:0/450267363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a0004d00 msgr2=0x7f20a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20c5000640 1 --2- 172.21.15.39:0/450267363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a0004d00 0x7f20a0005180 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f20b800a860 tx=0x7f20a00057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20bfe00640 1 -- 172.21.15.39:0/450267363 reap_dead start 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20c5000640 1 -- 172.21.15.39:0/450267363 shutdown_connections 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20c5000640 1 -- 172.21.15.39:0/450267363 >> 172.21.15.39:0/450267363 conn(0x7f20c006a2a0 msgr2=0x7f20c00afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.354+0000 7f20c5000640 1 -- 172.21.15.39:0/450267363 shutdown_connections 2024-01-24T17:45:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.355+0000 7f20c5000640 1 -- 172.21.15.39:0/450267363 wait complete. 2024-01-24T17:45:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.356+0000 7f20c5000640 1 Processor -- start 2024-01-24T17:45:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.356+0000 7f20c5000640 1 -- start start 2024-01-24T17:45:14.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.357+0000 7f20c5000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0004d00 0x7f20c00cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.357+0000 7f20c5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20c00c59a0 0x7f20c00c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.357+0000 7f20c5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00c79a0 0x7f20c00aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.357+0000 7f20c5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00b2070 con 0x7f20c00c59a0 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.357+0000 7f20c5000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00c7f40 con 0x7f20a0004d00 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.357+0000 7f20c5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c00b2310 con 0x7f20c00c79a0 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20c00c59a0 0x7f20c00c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20c00c59a0 0x7f20c00c7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51260/0 (socket says 172.21.15.39:51260) 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 -- 172.21.15.39:0/3486721147 learned_addr learned my addr 172.21.15.39:0/3486721147 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bfe00640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00c79a0 0x7f20c00aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bf400640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0004d00 0x7f20c00cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 -- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00c79a0 msgr2=0x7f20c00aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00c79a0 0x7f20c00aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 -- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0004d00 msgr2=0x7f20c00cbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0004d00 0x7f20c00cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 -- 172.21.15.39:0/3486721147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c00b25f0 con 0x7f20c00c59a0 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bfe00640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00c79a0 0x7f20c00aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.358+0000 7f20bea00640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20c00c59a0 0x7f20c00c7460 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f20a80116d0 tx=0x7f20a8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.359+0000 7f20b7400640 1 -- 172.21.15.39:0/3486721147 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a800f9c0 con 0x7f20c00c59a0 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.359+0000 7f20b7400640 1 -- 172.21.15.39:0/3486721147 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a800fb20 con 0x7f20c00c59a0 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.359+0000 7f20b7400640 1 -- 172.21.15.39:0/3486721147 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a800fe90 con 0x7f20c00c59a0 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.359+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c00c8220 con 0x7f20c00c59a0 2024-01-24T17:45:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.359+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c00c5e20 con 0x7f20c00c59a0 2024-01-24T17:45:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.360+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c004dff0 con 0x7f20c00c59a0 2024-01-24T17:45:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.361+0000 7f20b7400640 1 -- 172.21.15.39:0/3486721147 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f20a8015460 con 0x7f20c00c59a0 2024-01-24T17:45:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.362+0000 7f20b7400640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f209806a430 0x7f209806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.362+0000 7f20b7400640 1 -- 172.21.15.39:0/3486721147 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20a8088bf0 con 0x7f20c00c59a0 2024-01-24T17:45:14.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.363+0000 7f20bf400640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f209806a430 0x7f209806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.364+0000 7f20bf400640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f209806a430 0x7f209806c8f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f20b00099c0 tx=0x7f20b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:14.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.367+0000 7f20b7400640 1 -- 172.21.15.39:0/3486721147 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f20a8054490 con 0x7f20c00c59a0 2024-01-24T17:45:14.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.629+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f20c00b2780 con 0x7f20c00c59a0 2024-01-24T17:45:14.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.631+0000 7f20b7400640 1 -- 172.21.15.39:0/3486721147 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f20a8057ab0 con 0x7f20c00c59a0 2024-01-24T17:45:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_dump": [ 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.a", 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlZZEtLBAAUxf4chL3tgy3zHlZoraXlg==", 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.b", 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Fln4utEhAA52ngJFqrdfa9bx56mVhRVA==", 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.c", 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flo8iGARAAf+29RXWr4RHkILIz7em3mA==", 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.d", 2024-01-24T17:45:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flh+08IRAARhZxy//j8c9b3Hg9uV9ajg==", 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.0", 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCQS7FlOj/pLBAAzi89tAacOfDKdlU0BUJvvA==", 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:45:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.1", 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCUS7FlbRAKDRAAAuAy47MDTtvmh8qNA3ZLTg==", 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.2", 2024-01-24T17:45:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCXS7Fl7QmRGRAAVECH7oYTdvDPSHBy9WmHwA==", 2024-01-24T17:45:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:45:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:45:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.3", 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCaS7FlCQHIMxAA9OchsMzsKFxs110SeJ3Atw==", 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.4", 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCiS7FlDsBKBRAAXF2HafraT5nRbB5tBRUgWQ==", 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.5", 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQClS7FlfrXBHhAAy8GP3TLX/iok3OYPGTP5Mw==", 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.6", 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCoS7Fla11hNhAA/3oMP7cFtPSTdJl7eSnPTA==", 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.7", 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCsS7FlKFMtEhAAPmaFzr5XHVQaKZYkJiIarA==", 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.0", 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlTi/3LhAAc8FwXtLH2XjlFMO7wDMcSQ==", 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow r", 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.1", 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlH5s7ChAAlFfFDziTgfv4ibWO08N+4Q==", 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.2", 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlIISlOhAAoVxxez7UjXDxyHQNNPnn2g==", 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:45:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.3", 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlO/U/FBAA2pI9aGSS2cZ3AMnn+W1fQA==", 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.admin", 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCGS7FlxKo7HhAAlu037DIc1WmWkz81BG3j7Q==", 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.x", 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlRbF6HxAAoFjlyWb5oqIkC3L1iPRmHw==", 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.y", 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7Fl512kFRAAdCpU+QjRmGJmMD5uTdFmHQ==", 2024-01-24T17:45:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:45:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:45:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:45:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:45:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:45:14.647 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:45:14.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f209806a430 msgr2=0x7f209806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f209806a430 0x7f209806c8f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f20b00099c0 tx=0x7f20b0008040 comp rx=0 tx=0).stop 2024-01-24T17:45:14.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20c00c59a0 msgr2=0x7f20c00c7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20c00c59a0 0x7f20c00c7460 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f20a80116d0 tx=0x7f20a8011b90 comp rx=0 tx=0).stop 2024-01-24T17:45:14.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 shutdown_connections 2024-01-24T17:45:14.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20c00c79a0 0x7f20c00aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f209806a430 0x7f209806c8f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20c00c59a0 0x7f20c00c7460 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 --2- 172.21.15.39:0/3486721147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a0004d00 0x7f20c00cbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 >> 172.21.15.39:0/3486721147 conn(0x7f20c006a2a0 msgr2=0x7f20c0157f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 shutdown_connections 2024-01-24T17:45:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.633+0000 7f20c5000640 1 -- 172.21.15.39:0/3486721147 wait complete. 2024-01-24T17:45:14.652 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:14.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.871+0000 7f59a3000640 1 Processor -- start 2024-01-24T17:45:14.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.871+0000 7f59a3000640 1 -- start start 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f59a3000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f599c1654d0 0x7f599c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f59a3000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f599c0c90d0 0x7f599c070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f59a3000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c1658b0 0x7f599c0caf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f59a3000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f599c0cc9d0 con 0x7f599c1654d0 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f59a3000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f599c0cc630 con 0x7f599c0c90d0 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f59a3000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599c0c9570 con 0x7f599c1658b0 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f599be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f599c1654d0 0x7f599c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f599be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f599c1654d0 0x7f599c0c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51274/0 (socket says 172.21.15.39:51274) 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.873+0000 7f599be00640 1 -- 172.21.15.39:0/1544215785 learned_addr learned my addr 172.21.15.39:0/1544215785 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.874+0000 7f5993e00640 1 --2- 172.21.15.39:0/1544215785 >> v2:172.21.15.100:3300/0 conn(0x7f599c0c90d0 0x7f599c070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.874+0000 7f59a1000640 1 --2- 172.21.15.39:0/1544215785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c1658b0 0x7f599c0caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.874+0000 7f599be00640 1 -- 172.21.15.39:0/1544215785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c1658b0 msgr2=0x7f599c0caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:14.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.874+0000 7f599be00640 1 --2- 172.21.15.39:0/1544215785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c1658b0 0x7f599c0caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.874+0000 7f599be00640 1 -- 172.21.15.39:0/1544215785 >> v2:172.21.15.100:3300/0 conn(0x7f599c0c90d0 msgr2=0x7f599c070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:14.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.874+0000 7f599be00640 1 --2- 172.21.15.39:0/1544215785 >> v2:172.21.15.100:3300/0 conn(0x7f599c0c90d0 0x7f599c070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.874+0000 7f599be00640 1 -- 172.21.15.39:0/1544215785 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c0c9850 con 0x7f599c1654d0 2024-01-24T17:45:14.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.874+0000 7f5993e00640 1 --2- 172.21.15.39:0/1544215785 >> v2:172.21.15.100:3300/0 conn(0x7f599c0c90d0 0x7f599c070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:14.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.876+0000 7f599be00640 1 --2- 172.21.15.39:0/1544215785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c0c8b90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f599c164b00 tx=0x7f598c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08338090b9e8608 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:14.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.876+0000 7f599b400640 1 -- 172.21.15.39:0/1544215785 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c031750 con 0x7f599c1654d0 2024-01-24T17:45:14.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.876+0000 7f599b400640 1 -- 172.21.15.39:0/1544215785 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598c0318b0 con 0x7f599c1654d0 2024-01-24T17:45:14.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.876+0000 7f599b400640 1 -- 172.21.15.39:0/1544215785 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c031c40 con 0x7f599c1654d0 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 -- 172.21.15.39:0/1544215785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 msgr2=0x7f599c0c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 --2- 172.21.15.39:0/1544215785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c0c8b90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f599c164b00 tx=0x7f598c02b7f0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 -- 172.21.15.39:0/1544215785 shutdown_connections 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 --2- 172.21.15.39:0/1544215785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c1658b0 0x7f599c0caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 --2- 172.21.15.39:0/1544215785 >> v2:172.21.15.100:3300/0 conn(0x7f599c0c90d0 0x7f599c070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 --2- 172.21.15.39:0/1544215785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c0c8b90 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 -- 172.21.15.39:0/1544215785 >> 172.21.15.39:0/1544215785 conn(0x7f599c06a3e0 msgr2=0x7f599c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 -- 172.21.15.39:0/1544215785 shutdown_connections 2024-01-24T17:45:14.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.877+0000 7f59a3000640 1 -- 172.21.15.39:0/1544215785 wait complete. 2024-01-24T17:45:14.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.879+0000 7f59a3000640 1 Processor -- start 2024-01-24T17:45:14.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.880+0000 7f59a3000640 1 -- start start 2024-01-24T17:45:14.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f59a3000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c0c90d0 0x7f599c158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f59a3000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c1f9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f59a3000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f599c1658b0 0x7f599c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f59a3000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c0ae020 con 0x7f599c1654d0 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f59a3000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c1fa8a0 con 0x7f599c1658b0 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f59a3000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599c1fab80 con 0x7f599c0c90d0 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f5993e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c1f9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f5993e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c1f9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51284/0 (socket says 172.21.15.39:51284) 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.881+0000 7f5993e00640 1 -- 172.21.15.39:0/2593146511 learned_addr learned my addr 172.21.15.39:0/2593146511 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f59a1000640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f599c1658b0 0x7f599c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5993e00640 1 -- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c0c90d0 msgr2=0x7f599c158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f599be00640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c0c90d0 0x7f599c158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5993e00640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c0c90d0 0x7f599c158630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5993e00640 1 -- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f599c1658b0 msgr2=0x7f599c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5993e00640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f599c1658b0 0x7f599c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5993e00640 1 -- 172.21.15.39:0/2593146511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0094d0 con 0x7f599c1654d0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f59a1000640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f599c1658b0 0x7f599c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5993e00640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c1f9e50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f598400c7f0 tx=0x7f598400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5998c00640 1 -- 172.21.15.39:0/2593146511 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984004220 con 0x7f599c1654d0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5998c00640 1 -- 172.21.15.39:0/2593146511 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5984004380 con 0x7f599c1654d0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.882+0000 7f5998c00640 1 -- 172.21.15.39:0/2593146511 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984007860 con 0x7f599c1654d0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.883+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c0ae2e0 con 0x7f599c1654d0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.883+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c0c5cd0 con 0x7f599c1654d0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.884+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c0ade20 con 0x7f599c1654d0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.885+0000 7f5998c00640 1 -- 172.21.15.39:0/2593146511 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5984007bd0 con 0x7f599c1654d0 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.885+0000 7f5998c00640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f597406a430 0x7f597406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:14.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.885+0000 7f5998c00640 1 -- 172.21.15.39:0/2593146511 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5984088670 con 0x7f599c1654d0 2024-01-24T17:45:14.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.886+0000 7f599be00640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f597406a430 0x7f597406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:14.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.887+0000 7f599be00640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f597406a430 0x7f597406c8f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f598c035000 tx=0x7f598c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:14.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:14.890+0000 7f5998c00640 1 -- 172.21.15.39:0/2593146511 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5984053e80 con 0x7f599c1654d0 2024-01-24T17:45:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.078+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f599c073ef0 con 0x7f599c1654d0 2024-01-24T17:45:15.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.080+0000 7f5998c00640 1 -- 172.21.15.39:0/2593146511 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f598401e090 con 0x7f599c1654d0 2024-01-24T17:45:15.081 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:15.081 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":59,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:14.088389+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:15.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f597406a430 msgr2=0x7f597406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f597406a430 0x7f597406c8f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f598c035000 tx=0x7f598c008030 comp rx=0 tx=0).stop 2024-01-24T17:45:15.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 msgr2=0x7f599c1f9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c1f9e50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f598400c7f0 tx=0x7f598400ccb0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 shutdown_connections 2024-01-24T17:45:15.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f599c1658b0 0x7f599c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f597406a430 0x7f597406c8f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f599c1654d0 0x7f599c1f9e50 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 --2- 172.21.15.39:0/2593146511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f599c0c90d0 0x7f599c158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 >> 172.21.15.39:0/2593146511 conn(0x7f599c06a3e0 msgr2=0x7f599c161670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:15.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 shutdown_connections 2024-01-24T17:45:15.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.083+0000 7f59a3000640 1 -- 172.21.15.39:0/2593146511 wait complete. 2024-01-24T17:45:15.101 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:45:15.101 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:45:15.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.321+0000 7fd5ae200640 1 Processor -- start 2024-01-24T17:45:15.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.321+0000 7fd5ae200640 1 -- start start 2024-01-24T17:45:15.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ae200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd5a80accd0 0x7fd5a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ae200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a806c3a0 0x7fd5a80cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ae200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd5a80cb370 0x7fd5a815e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ae200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a80cb800 con 0x7fd5a80accd0 2024-01-24T17:45:15.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ae200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a806d840 con 0x7fd5a80cb370 2024-01-24T17:45:15.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ae200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80ad0b0 con 0x7fd5a806c3a0 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ace00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd5a80accd0 0x7fd5a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ace00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd5a80accd0 0x7fd5a806d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51296/0 (socket says 172.21.15.39:51296) 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.323+0000 7fd5ace00640 1 -- 172.21.15.39:0/815913797 learned_addr learned my addr 172.21.15.39:0/815913797 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.324+0000 7fd5ad800640 1 --2- 172.21.15.39:0/815913797 >> v2:172.21.15.100:3300/0 conn(0x7fd5a80cb370 0x7fd5a815e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.324+0000 7fd59fe00640 1 --2- 172.21.15.39:0/815913797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a806c3a0 0x7fd5a80cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.324+0000 7fd5ace00640 1 -- 172.21.15.39:0/815913797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a806c3a0 msgr2=0x7fd5a80cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.324+0000 7fd5ace00640 1 --2- 172.21.15.39:0/815913797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a806c3a0 0x7fd5a80cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.324+0000 7fd5ace00640 1 -- 172.21.15.39:0/815913797 >> v2:172.21.15.100:3300/0 conn(0x7fd5a80cb370 msgr2=0x7fd5a815e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.324+0000 7fd5ace00640 1 --2- 172.21.15.39:0/815913797 >> v2:172.21.15.100:3300/0 conn(0x7fd5a80cb370 0x7fd5a815e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.324+0000 7fd5ace00640 1 -- 172.21.15.39:0/815913797 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a80b2370 con 0x7fd5a80accd0 2024-01-24T17:45:15.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.325+0000 7fd5ace00640 1 --2- 172.21.15.39:0/815913797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80accd0 0x7fd5a806d300 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80c5470 tx=0x7fd59802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e15ba52d0a387b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:15.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.325+0000 7fd5a7e00640 1 -- 172.21.15.39:0/815913797 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598004230 con 0x7fd5a80accd0 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.325+0000 7fd5a7e00640 1 -- 172.21.15.39:0/815913797 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd598004390 con 0x7fd5a80accd0 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.325+0000 7fd5a7e00640 1 -- 172.21.15.39:0/815913797 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598003810 con 0x7fd5a80accd0 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 -- 172.21.15.39:0/815913797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80accd0 msgr2=0x7fd5a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 --2- 172.21.15.39:0/815913797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80accd0 0x7fd5a806d300 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80c5470 tx=0x7fd59802b8f0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 -- 172.21.15.39:0/815913797 shutdown_connections 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 --2- 172.21.15.39:0/815913797 >> v2:172.21.15.100:3300/0 conn(0x7fd5a80cb370 0x7fd5a815e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 --2- 172.21.15.39:0/815913797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a806c3a0 0x7fd5a80cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 --2- 172.21.15.39:0/815913797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80accd0 0x7fd5a806d300 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 -- 172.21.15.39:0/815913797 >> 172.21.15.39:0/815913797 conn(0x7fd5a8069e50 msgr2=0x7fd5a806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 -- 172.21.15.39:0/815913797 shutdown_connections 2024-01-24T17:45:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.326+0000 7fd5ae200640 1 -- 172.21.15.39:0/815913797 wait complete. 2024-01-24T17:45:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.328+0000 7fd5ae200640 1 Processor -- start 2024-01-24T17:45:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.328+0000 7fd5ae200640 1 -- start start 2024-01-24T17:45:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ae200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5a806c3a0 0x7fd5a80c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ae200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a80accd0 0x7fd5a81f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ae200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80cb370 0x7fd5a81f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ae200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a815c710 con 0x7fd5a80cb370 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ae200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a81f5150 con 0x7fd5a806c3a0 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ae200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80c3fc0 con 0x7fd5a80accd0 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ad800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80cb370 0x7fd5a81f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ad800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80cb370 0x7fd5a81f4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51312/0 (socket says 172.21.15.39:51312) 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ad800640 1 -- 172.21.15.39:0/3575796578 learned_addr learned my addr 172.21.15.39:0/3575796578 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ace00640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5a806c3a0 0x7fd5a80c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ad800640 1 -- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a80accd0 msgr2=0x7fd5a81f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd59fe00640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a80accd0 0x7fd5a81f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ad800640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a80accd0 0x7fd5a81f1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ad800640 1 -- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5a806c3a0 msgr2=0x7fd5a80c4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ad800640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5a806c3a0 0x7fd5a80c4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.329+0000 7fd5ad800640 1 -- 172.21.15.39:0/3575796578 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd598009460 con 0x7fd5a80cb370 2024-01-24T17:45:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.330+0000 7fd5ace00640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5a806c3a0 0x7fd5a80c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.330+0000 7fd5ad800640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80cb370 0x7fd5a81f4c10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0007e40 tx=0x7fd5a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.330+0000 7fd5a5600640 1 -- 172.21.15.39:0/3575796578 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0014620 con 0x7fd5a80cb370 2024-01-24T17:45:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.330+0000 7fd5a5600640 1 -- 172.21.15.39:0/3575796578 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a000ee90 con 0x7fd5a80cb370 2024-01-24T17:45:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.330+0000 7fd5a5600640 1 -- 172.21.15.39:0/3575796578 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a00149e0 con 0x7fd5a80cb370 2024-01-24T17:45:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.330+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a80c3720 con 0x7fd5a80cb370 2024-01-24T17:45:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.330+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a80c3ce0 con 0x7fd5a80cb370 2024-01-24T17:45:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.332+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a804dff0 con 0x7fd5a80cb370 2024-01-24T17:45:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.332+0000 7fd5a5600640 1 -- 172.21.15.39:0/3575796578 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd5a0014b40 con 0x7fd5a80cb370 2024-01-24T17:45:15.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.333+0000 7fd5a5600640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd58006a430 0x7fd58006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.333+0000 7fd5a5600640 1 -- 172.21.15.39:0/3575796578 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd5a0088fd0 con 0x7fd5a80cb370 2024-01-24T17:45:15.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.334+0000 7fd5ace00640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd58006a430 0x7fd58006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.335+0000 7fd5ace00640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd58006a430 0x7fd58006c8f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd598035000 tx=0x7fd598008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:15.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.337+0000 7fd5a5600640 1 -- 172.21.15.39:0/3575796578 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd5a0054910 con 0x7fd5a80cb370 2024-01-24T17:45:15.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.529+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5a80cca30 con 0x7fd5a80cb370 2024-01-24T17:45:15.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.530+0000 7fd5a5600640 1 -- 172.21.15.39:0/3575796578 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5a0057f30 con 0x7fd5a80cb370 2024-01-24T17:45:15.531 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:45:15.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd58006a430 msgr2=0x7fd58006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd58006a430 0x7fd58006c8f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd598035000 tx=0x7fd598008030 comp rx=0 tx=0).stop 2024-01-24T17:45:15.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80cb370 msgr2=0x7fd5a81f4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80cb370 0x7fd5a81f4c10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0007e40 tx=0x7fd5a000e4b0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 shutdown_connections 2024-01-24T17:45:15.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd58006a430 0x7fd58006c8f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5a80cb370 0x7fd5a81f4c10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd5a80accd0 0x7fd5a81f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 --2- 172.21.15.39:0/3575796578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5a806c3a0 0x7fd5a80c4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 >> 172.21.15.39:0/3575796578 conn(0x7fd5a8069e50 msgr2=0x7fd5a80b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:15.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 shutdown_connections 2024-01-24T17:45:15.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.533+0000 7fd5ae200640 1 -- 172.21.15.39:0/3575796578 wait complete. 2024-01-24T17:45:15.552 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T17:45:15.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.769+0000 7fb0c3200640 1 Processor -- start 2024-01-24T17:45:15.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.770+0000 7fb0c3200640 1 -- start start 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.771+0000 7fb0c3200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc15c900 0x7fb0bc0d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.771+0000 7fb0c3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0bc0d2640 0x7fb0bc0d2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.771+0000 7fb0c3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0bc0d03e0 con 0x7fb0bc06c2e0 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.771+0000 7fb0c3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc0d3010 con 0x7fb0bc0d2640 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.771+0000 7fb0c3200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0d0630 con 0x7fb0bc15c900 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c1400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb0bc06c2e0 0x7fb0bc0cf950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56106/0 (socket says 172.21.15.39:56106) 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c1400640 1 -- 172.21.15.39:0/3869632115 learned_addr learned my addr 172.21.15.39:0/3869632115 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c2800640 1 --2- 172.21.15.39:0/3869632115 >> v2:172.21.15.39:3300/0 conn(0x7fb0bc0d2640 0x7fb0bc0d2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c1e00640 1 --2- 172.21.15.39:0/3869632115 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc15c900 0x7fb0bc0d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c0a00640 1 -- 172.21.15.39:0/3869632115 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134132370 0 0) 0x7fb0bc0d03e0 con 0x7fb0bc06c2e0 2024-01-24T17:45:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c0a00640 1 -- 172.21.15.39:0/3869632115 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb09c003520 con 0x7fb0bc06c2e0 2024-01-24T17:45:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c2800640 1 -- 172.21.15.39:0/3869632115 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc15c900 msgr2=0x7fb0bc0d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c2800640 1 --2- 172.21.15.39:0/3869632115 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc15c900 0x7fb0bc0d13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c2800640 1 -- 172.21.15.39:0/3869632115 >> v1:172.21.15.39:6789/0 conn(0x7fb0bc06c2e0 legacy=0x7fb0bc0cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.772+0000 7fb0c2800640 1 -- 172.21.15.39:0/3869632115 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc0d0910 con 0x7fb0bc0d2640 2024-01-24T17:45:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.773+0000 7fb0c2800640 1 --2- 172.21.15.39:0/3869632115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc0d2640 0x7fb0bc0d2ad0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009dc0 tx=0x7fb0b402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616bee476422e1b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.774+0000 7fb0c0a00640 1 -- 172.21.15.39:0/3869632115 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b402cc40 con 0x7fb0bc0d2640 2024-01-24T17:45:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.774+0000 7fb0c0a00640 1 -- 172.21.15.39:0/3869632115 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b402cda0 con 0x7fb0bc0d2640 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.774+0000 7fb0c0a00640 1 -- 172.21.15.39:0/3869632115 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4034640 con 0x7fb0bc0d2640 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.774+0000 7fb0c3200640 1 -- 172.21.15.39:0/3869632115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc0d2640 msgr2=0x7fb0bc0d2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.774+0000 7fb0c3200640 1 --2- 172.21.15.39:0/3869632115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc0d2640 0x7fb0bc0d2ad0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009dc0 tx=0x7fb0b402be90 comp rx=0 tx=0).stop 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.774+0000 7fb0c3200640 1 -- 172.21.15.39:0/3869632115 shutdown_connections 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.775+0000 7fb0c3200640 1 --2- 172.21.15.39:0/3869632115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc0d2640 0x7fb0bc0d2ad0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.775+0000 7fb0c3200640 1 --2- 172.21.15.39:0/3869632115 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc15c900 0x7fb0bc0d13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.775+0000 7fb0c3200640 1 -- 172.21.15.39:0/3869632115 >> 172.21.15.39:0/3869632115 conn(0x7fb0bc09bb30 msgr2=0x7fb0bc089fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.775+0000 7fb0c3200640 1 -- 172.21.15.39:0/3869632115 shutdown_connections 2024-01-24T17:45:15.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.775+0000 7fb0c3200640 1 -- 172.21.15.39:0/3869632115 wait complete. 2024-01-24T17:45:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.776+0000 7fb0c3200640 1 Processor -- start 2024-01-24T17:45:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.777+0000 7fb0c3200640 1 -- start start 2024-01-24T17:45:15.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.777+0000 7fb0c3200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0bc06c2e0 0x7fb0bc0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.777+0000 7fb0c3200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc0d2640 0x7fb0bc1f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc15c900 0x7fb0bc1f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0cc200 con 0x7fb0bc15c900 2024-01-24T17:45:15.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c3200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc1f6820 con 0x7fb0bc06c2e0 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c3200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc0c9260 con 0x7fb0bc0d2640 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc15c900 0x7fb0bc1f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc15c900 0x7fb0bc1f5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51338/0 (socket says 172.21.15.39:51338) 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c2800640 1 -- 172.21.15.39:0/1640692344 learned_addr learned my addr 172.21.15.39:0/1640692344 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c1e00640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0bc06c2e0 0x7fb0bc0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c1400640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc0d2640 0x7fb0bc1f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c2800640 1 -- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc0d2640 msgr2=0x7fb0bc1f59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c2800640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc0d2640 0x7fb0bc1f59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c2800640 1 -- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0bc06c2e0 msgr2=0x7fb0bc0c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c2800640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0bc06c2e0 0x7fb0bc0c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c2800640 1 -- 172.21.15.39:0/1640692344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b4009a70 con 0x7fb0bc15c900 2024-01-24T17:45:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.778+0000 7fb0c1400640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc0d2640 0x7fb0bc1f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.779+0000 7fb0c2800640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc15c900 0x7fb0bc1f5f00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009eb0 tx=0x7fb0b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.779+0000 7fb0ba000640 1 -- 172.21.15.39:0/1640692344 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b403e610 con 0x7fb0bc15c900 2024-01-24T17:45:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.779+0000 7fb0ba000640 1 -- 172.21.15.39:0/1640692344 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b403e770 con 0x7fb0bc15c900 2024-01-24T17:45:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.779+0000 7fb0ba000640 1 -- 172.21.15.39:0/1640692344 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b403eb20 con 0x7fb0bc15c900 2024-01-24T17:45:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.779+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0bc0cb580 con 0x7fb0bc15c900 2024-01-24T17:45:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.779+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc0cbb40 con 0x7fb0bc15c900 2024-01-24T17:45:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.781+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc04dff0 con 0x7fb0bc15c900 2024-01-24T17:45:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.781+0000 7fb0ba000640 1 -- 172.21.15.39:0/1640692344 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb0b403b050 con 0x7fb0bc15c900 2024-01-24T17:45:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.782+0000 7fb0ba000640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb09406a430 0x7fb09406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.782+0000 7fb0ba000640 1 -- 172.21.15.39:0/1640692344 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb0b40aaff0 con 0x7fb0bc15c900 2024-01-24T17:45:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.783+0000 7fb0c1e00640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb09406a430 0x7fb09406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:15.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.784+0000 7fb0c1e00640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb09406a430 0x7fb09406c8f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0cff00 tx=0x7fb0ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:15.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.786+0000 7fb0ba000640 1 -- 172.21.15.39:0/1640692344 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb0b4076a10 con 0x7fb0bc15c900 2024-01-24T17:45:15.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.983+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb0bc0c9650 con 0x7fb0bc15c900 2024-01-24T17:45:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.984+0000 7fb0ba000640 1 -- 172.21.15.39:0/1640692344 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v59) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb0b404e060 con 0x7fb0bc15c900 2024-01-24T17:45:15.985 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:15.985 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-01-24T17:45:15.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:45:15.985 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-01-24T17:45:15.985 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-01-24T17:45:15.985 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:15.986 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb09406a430 msgr2=0x7fb09406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb09406a430 0x7fb09406c8f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0cff00 tx=0x7fb0ac008040 comp rx=0 tx=0).stop 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc15c900 msgr2=0x7fb0bc1f5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc15c900 0x7fb0bc1f5f00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009eb0 tx=0x7fb0b4004a40 comp rx=0 tx=0).stop 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 shutdown_connections 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb09406a430 0x7fb09406c8f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0bc15c900 0x7fb0bc1f5f00 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0bc0d2640 0x7fb0bc1f59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 --2- 172.21.15.39:0/1640692344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0bc06c2e0 0x7fb0bc0c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 >> 172.21.15.39:0/1640692344 conn(0x7fb0bc09bb30 msgr2=0x7fb0bc0b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 shutdown_connections 2024-01-24T17:45:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:15.987+0000 7fb0c3200640 1 -- 172.21.15.39:0/1640692344 wait complete. 2024-01-24T17:45:16.008 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T17:45:16.008 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T17:45:16.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.232+0000 7f4e50a00640 1 Processor -- start 2024-01-24T17:45:16.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.232+0000 7f4e50a00640 1 -- start start 2024-01-24T17:45:16.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.233+0000 7f4e50a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e4c15a7c0 0x7f4e4c06fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:16.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.233+0000 7f4e50a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4e4c15ea00 0x7f4e4c0ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:16.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.233+0000 7f4e50a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4c070790 con 0x7f4e4c070330 2024-01-24T17:45:16.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e50a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4c15f1b0 con 0x7f4e4c15a7c0 2024-01-24T17:45:16.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e50a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4c0cbca0 con 0x7f4e4c15ea00 2024-01-24T17:45:16.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e4c15a7c0 0x7f4e4c06fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:16.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e4c15a7c0 0x7f4e4c06fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51342/0 (socket says 172.21.15.39:51342) 2024-01-24T17:45:16.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4b400640 1 -- 172.21.15.39:0/2505546083 learned_addr learned my addr 172.21.15.39:0/2505546083 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:16.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4be00640 1 --2- 172.21.15.39:0/2505546083 >> v2:172.21.15.100:3300/0 conn(0x7f4e4c15ea00 0x7f4e4c0ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:16.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4b400640 1 -- 172.21.15.39:0/2505546083 >> v2:172.21.15.100:3300/0 conn(0x7f4e4c15ea00 msgr2=0x7f4e4c0ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:16.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4b400640 1 --2- 172.21.15.39:0/2505546083 >> v2:172.21.15.100:3300/0 conn(0x7f4e4c15ea00 0x7f4e4c0ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4b400640 1 -- 172.21.15.39:0/2505546083 >> v1:172.21.15.39:6789/0 conn(0x7f4e4c070330 legacy=0x7f4e4c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:16.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4b400640 1 -- 172.21.15.39:0/2505546083 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c15eed0 con 0x7f4e4c15a7c0 2024-01-24T17:45:16.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.234+0000 7f4e4be00640 1 --2- 172.21.15.39:0/2505546083 >> v2:172.21.15.100:3300/0 conn(0x7f4e4c15ea00 0x7f4e4c0ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:16.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.236+0000 7f4e4b400640 1 --2- 172.21.15.39:0/2505546083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c15a7c0 0x7f4e4c06fdf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c06d110 tx=0x7f4e3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f95c797466e9a58 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:16.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.236+0000 7f4e4aa00640 1 -- 172.21.15.39:0/2505546083 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c031870 con 0x7f4e4c15a7c0 2024-01-24T17:45:16.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.236+0000 7f4e4aa00640 1 -- 172.21.15.39:0/2505546083 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c0319d0 con 0x7f4e4c15a7c0 2024-01-24T17:45:16.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.236+0000 7f4e4aa00640 1 -- 172.21.15.39:0/2505546083 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c031d60 con 0x7f4e4c15a7c0 2024-01-24T17:45:16.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.237+0000 7f4e50a00640 1 -- 172.21.15.39:0/2505546083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c15a7c0 msgr2=0x7f4e4c06fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:16.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.237+0000 7f4e50a00640 1 --2- 172.21.15.39:0/2505546083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c15a7c0 0x7f4e4c06fdf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c06d110 tx=0x7f4e3c02b910 comp rx=0 tx=0).stop 2024-01-24T17:45:16.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.237+0000 7f4e50a00640 1 -- 172.21.15.39:0/2505546083 shutdown_connections 2024-01-24T17:45:16.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.237+0000 7f4e50a00640 1 --2- 172.21.15.39:0/2505546083 >> v2:172.21.15.100:3300/0 conn(0x7f4e4c15ea00 0x7f4e4c0ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.237+0000 7f4e50a00640 1 --2- 172.21.15.39:0/2505546083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c15a7c0 0x7f4e4c06fdf0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.237+0000 7f4e50a00640 1 -- 172.21.15.39:0/2505546083 >> 172.21.15.39:0/2505546083 conn(0x7f4e4c0caeb0 msgr2=0x7f4e4c0cb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:16.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.237+0000 7f4e50a00640 1 -- 172.21.15.39:0/2505546083 shutdown_connections 2024-01-24T17:45:16.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.237+0000 7f4e50a00640 1 -- 172.21.15.39:0/2505546083 wait complete. 2024-01-24T17:45:16.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.239+0000 7f4e50a00640 1 Processor -- start 2024-01-24T17:45:16.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.239+0000 7f4e50a00640 1 -- start start 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.240+0000 7f4e50a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c070330 0x7f4e4c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.240+0000 7f4e50a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e4c15a7c0 0x7f4e4c0ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.240+0000 7f4e50a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e4c15ea00 0x7f4e4c0c9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.240+0000 7f4e50a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0b1140 con 0x7f4e4c070330 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.240+0000 7f4e50a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0c9c10 con 0x7f4e4c15ea00 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.240+0000 7f4e50a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0b0850 con 0x7f4e4c15a7c0 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c070330 0x7f4e4c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c070330 0x7f4e4c15aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51346/0 (socket says 172.21.15.39:51346) 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 -- 172.21.15.39:0/1119095128 learned_addr learned my addr 172.21.15.39:0/1119095128 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 -- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e4c15a7c0 msgr2=0x7f4e4c0ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e4c15a7c0 0x7f4e4c0ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 -- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e4c15ea00 msgr2=0x7f4e4c0c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:16.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e4c15ea00 0x7f4e4c0c9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 -- 172.21.15.39:0/1119095128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c0094d0 con 0x7f4e4c070330 2024-01-24T17:45:16.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.241+0000 7f4e4b400640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c070330 0x7f4e4c15aba0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c02b950 tx=0x7f4e3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.242+0000 7f4e43e00640 1 -- 172.21.15.39:0/1119095128 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c031860 con 0x7f4e4c070330 2024-01-24T17:45:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.242+0000 7f4e43e00640 1 -- 172.21.15.39:0/1119095128 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c0443f0 con 0x7f4e4c070330 2024-01-24T17:45:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.242+0000 7f4e43e00640 1 -- 172.21.15.39:0/1119095128 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0447a0 con 0x7f4e4c070330 2024-01-24T17:45:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.242+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c0c9970 con 0x7f4e4c070330 2024-01-24T17:45:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.242+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c0b0dd0 con 0x7f4e4c070330 2024-01-24T17:45:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.243+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c0cc440 con 0x7f4e4c070330 2024-01-24T17:45:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.244+0000 7f4e43e00640 1 -- 172.21.15.39:0/1119095128 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4e3c04d450 con 0x7f4e4c070330 2024-01-24T17:45:16.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.245+0000 7f4e43e00640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4e2406a430 0x7f4e2406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:16.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.246+0000 7f4e43e00640 1 -- 172.21.15.39:0/1119095128 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4e3c02d040 con 0x7f4e4c070330 2024-01-24T17:45:16.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.246+0000 7f4e42800640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4e2406a430 0x7f4e2406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:16.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.247+0000 7f4e42800640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4e2406a430 0x7f4e2406c8f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4e340012e0 tx=0x7f4e34007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:16.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.250+0000 7f4e43e00640 1 -- 172.21.15.39:0/1119095128 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4e4c0cc440 con 0x7f4e4c070330 2024-01-24T17:45:16.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.447+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4e4c0cc440 con 0x7f4e4c070330 2024-01-24T17:45:16.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.874+0000 7f4e43e00640 1 -- 172.21.15.39:0/1119095128 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v60) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4e4c0cc440 con 0x7f4e4c070330 2024-01-24T17:45:16.876 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.879+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4e2406a430 msgr2=0x7f4e2406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.879+0000 7f4e50a00640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4e2406a430 0x7f4e2406c8f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4e340012e0 tx=0x7f4e34007d30 comp rx=0 tx=0).stop 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.879+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c070330 msgr2=0x7f4e4c15aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.879+0000 7f4e50a00640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c070330 0x7f4e4c15aba0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c02b950 tx=0x7f4e3c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.880+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 shutdown_connections 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.880+0000 7f4e50a00640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e4c15ea00 0x7f4e4c0c9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.880+0000 7f4e50a00640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e4c15a7c0 0x7f4e4c0ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.880+0000 7f4e50a00640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4e2406a430 0x7f4e2406c8f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.880+0000 7f4e50a00640 1 --2- 172.21.15.39:0/1119095128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e4c070330 0x7f4e4c15aba0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.880+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 >> 172.21.15.39:0/1119095128 conn(0x7f4e4c0caeb0 msgr2=0x7f4e4c0c2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.880+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 shutdown_connections 2024-01-24T17:45:16.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:16.880+0000 7f4e50a00640 1 -- 172.21.15.39:0/1119095128 wait complete. 2024-01-24T17:45:16.905 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T17:45:17.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.121+0000 7ffafe600640 1 Processor -- start 2024-01-24T17:45:17.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.121+0000 7ffafe600640 1 -- start start 2024-01-24T17:45:17.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.123+0000 7ffafe600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 0x7ffaf80affa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:17.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.123+0000 7ffafe600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf8160b80 con 0x7ffaf80cb3c0 2024-01-24T17:45:17.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.123+0000 7ffafe600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf815cf00 con 0x7ffaf80b3210 2024-01-24T17:45:17.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.123+0000 7ffafe600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80b0eb0 con 0x7ffaf80ca360 2024-01-24T17:45:17.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.123+0000 7ffaf7400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ffaf80cb3c0 0x7ffaf80cb7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56112/0 (socket says 172.21.15.39:56112) 2024-01-24T17:45:17.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.123+0000 7ffaf7400640 1 -- 172.21.15.39:0/2338987123 learned_addr learned my addr 172.21.15.39:0/2338987123 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.123+0000 7ffaf7e00640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 0x7ffaf80affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067328829 0 0) 0x7ffaf8160b80 con 0x7ffaf80cb3c0 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8003520 con 0x7ffaf80cb3c0 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3783722677 0 0) 0x7ffad8003520 con 0x7ffaf80cb3c0 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 msgr2=0x7ffaf80affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 0x7ffaf80affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 >> v1:172.21.15.100:6789/0 conn(0x7ffaf80b3210 legacy=0x7ffaf8161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf80b08f0 con 0x7ffaf80cb3c0 2024-01-24T17:45:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ffae8001b70 con 0x7ffaf80cb3c0 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf7e00640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 0x7ffaf80affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 >> v1:172.21.15.39:6789/0 conn(0x7ffaf80cb3c0 legacy=0x7ffaf80cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad80039a0 0x7ffad8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad8004350 0x7ffad8004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 0x7ffad8005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80b0eb0 con 0x7ffad8004350 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.124+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad800b980 con 0x7ffad80039a0 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad800bb40 con 0x7ffaf80ca360 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf6a00640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad8004350 0x7ffad8004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf7400640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad80039a0 0x7ffad8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf7e00640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 0x7ffad8005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf6a00640 1 -- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 msgr2=0x7ffad8005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf6a00640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf80ca360 0x7ffad8005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf6a00640 1 -- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad80039a0 msgr2=0x7ffad8003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf6a00640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad80039a0 0x7ffad8003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf6a00640 1 -- 172.21.15.39:0/2338987123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf0009400 con 0x7ffad8004350 2024-01-24T17:45:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.125+0000 7ffaf7400640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad80039a0 0x7ffad8003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:45:17.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.127+0000 7ffaf6a00640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad8004350 0x7ffad8004820 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ffae0001860 tx=0x7ffad80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd37b8bfd5a90a7e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.127+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0033410 con 0x7ffad8004350 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.127+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae0033590 con 0x7ffad8004350 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.127+0000 7ffaf6000640 1 -- 172.21.15.39:0/2338987123 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0033940 con 0x7ffad8004350 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.128+0000 7ffafe600640 1 -- 172.21.15.39:0/2338987123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad8004350 msgr2=0x7ffad8004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.128+0000 7ffafe600640 1 --2- 172.21.15.39:0/2338987123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffad8004350 0x7ffad8004820 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ffae0001860 tx=0x7ffad80057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.128+0000 7ffaf7e00640 1 -- 172.21.15.39:0/2338987123 reap_dead start 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.128+0000 7ffafe600640 1 -- 172.21.15.39:0/2338987123 shutdown_connections 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.128+0000 7ffafe600640 1 -- 172.21.15.39:0/2338987123 >> 172.21.15.39:0/2338987123 conn(0x7ffaf806a090 msgr2=0x7ffaf806a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.128+0000 7ffafe600640 1 -- 172.21.15.39:0/2338987123 shutdown_connections 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.128+0000 7ffafe600640 1 -- 172.21.15.39:0/2338987123 wait complete. 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.130+0000 7ffafe600640 1 Processor -- start 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.130+0000 7ffafe600640 1 -- start start 2024-01-24T17:45:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffafe600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad8004350 0x7ffaf80ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffafe600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf81561c0 0x7ffaf8156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffafe600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaf81e7540 0x7ffaf81e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffafe600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815d0f0 con 0x7ffaf81e7540 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffafe600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815d340 con 0x7ffad8004350 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffafe600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815d620 con 0x7ffaf81561c0 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffaf7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaf81e7540 0x7ffaf81e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffaf7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaf81e7540 0x7ffaf81e79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51362/0 (socket says 172.21.15.39:51362) 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.131+0000 7ffaf7e00640 1 -- 172.21.15.39:0/287713406 learned_addr learned my addr 172.21.15.39:0/287713406 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaf7400640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad8004350 0x7ffaf80ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaf6a00640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf81561c0 0x7ffaf8156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaf7e00640 1 -- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf81561c0 msgr2=0x7ffaf8156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaf7e00640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf81561c0 0x7ffaf8156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaf7e00640 1 -- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad8004350 msgr2=0x7ffaf80ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaf7e00640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad8004350 0x7ffaf80ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaf7e00640 1 -- 172.21.15.39:0/287713406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf815d900 con 0x7ffaf81e7540 2024-01-24T17:45:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaf7e00640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaf81e7540 0x7ffaf81e79c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ffaf806ba60 tx=0x7ffaf0011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.132+0000 7ffaef400640 1 -- 172.21.15.39:0/287713406 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0004440 con 0x7ffaf81e7540 2024-01-24T17:45:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.133+0000 7ffaef400640 1 -- 172.21.15.39:0/287713406 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf0003610 con 0x7ffaf81e7540 2024-01-24T17:45:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.133+0000 7ffaef400640 1 -- 172.21.15.39:0/287713406 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0003980 con 0x7ffaf81e7540 2024-01-24T17:45:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.133+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf815dbe0 con 0x7ffaf81e7540 2024-01-24T17:45:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.133+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf81e7f00 con 0x7ffaf81e7540 2024-01-24T17:45:17.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.134+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf815ddd0 con 0x7ffaf81e7540 2024-01-24T17:45:17.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.135+0000 7ffaef400640 1 -- 172.21.15.39:0/287713406 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ffaf0003ae0 con 0x7ffaf81e7540 2024-01-24T17:45:17.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.136+0000 7ffaef400640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffad006a460 0x7ffad006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:17.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.137+0000 7ffaef400640 1 -- 172.21.15.39:0/287713406 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ffaf0088dc0 con 0x7ffaf81e7540 2024-01-24T17:45:17.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.137+0000 7ffaf7400640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffad006a460 0x7ffad006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:17.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.138+0000 7ffaf7400640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffad006a460 0x7ffad006c920 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffae8000f80 tx=0x7ffae8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:17.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.142+0000 7ffaef400640 1 -- 172.21.15.39:0/287713406 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ffaf00545a0 con 0x7ffaf81e7540 2024-01-24T17:45:17.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.335+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ffaf806c730 con 0x7ffaf81e7540 2024-01-24T17:45:17.898 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-01-24T17:45:17.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.894+0000 7ffaef400640 1 -- 172.21.15.39:0/287713406 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v61) v1 ==== 170+0+0 (secure 0 0 0) 0x7ffaf0057bc0 con 0x7ffaf81e7540 2024-01-24T17:45:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.902+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffad006a460 msgr2=0x7ffad006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.902+0000 7ffafe600640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffad006a460 0x7ffad006c920 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffae8000f80 tx=0x7ffae8009040 comp rx=0 tx=0).stop 2024-01-24T17:45:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.902+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaf81e7540 msgr2=0x7ffaf81e79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:17.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.902+0000 7ffafe600640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaf81e7540 0x7ffaf81e79c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ffaf806ba60 tx=0x7ffaf0011920 comp rx=0 tx=0).stop 2024-01-24T17:45:17.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.906+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 shutdown_connections 2024-01-24T17:45:17.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.906+0000 7ffafe600640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffad006a460 0x7ffad006c920 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.906+0000 7ffafe600640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffaf81e7540 0x7ffaf81e79c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.906+0000 7ffafe600640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffaf81561c0 0x7ffaf8156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.906+0000 7ffafe600640 1 --2- 172.21.15.39:0/287713406 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffad8004350 0x7ffaf80ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:17.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.907+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 >> 172.21.15.39:0/287713406 conn(0x7ffaf806a090 msgr2=0x7ffaf80ad3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:17.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.907+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 shutdown_connections 2024-01-24T17:45:17.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:17.907+0000 7ffafe600640 1 -- 172.21.15.39:0/287713406 wait complete. 2024-01-24T17:45:17.934 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T17:45:18.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.126+0000 7ff32e800640 1 Processor -- start 2024-01-24T17:45:18.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.127+0000 7ff32e800640 1 -- start start 2024-01-24T17:45:18.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.128+0000 7ff32e800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 0x7ff3281656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.128+0000 7ff32e800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32806d4c0 con 0x7ff3280b0730 2024-01-24T17:45:18.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.128+0000 7ff32e800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32806cf00 con 0x7ff3280b0350 2024-01-24T17:45:18.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.128+0000 7ff32e800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32806d1e0 con 0x7ff32815e190 2024-01-24T17:45:18.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.128+0000 7ff326a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff3280b0730 0x7ff32815da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56118/0 (socket says 172.21.15.39:56118) 2024-01-24T17:45:18.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.129+0000 7ff326a00640 1 -- 172.21.15.39:0/1173531666 learned_addr learned my addr 172.21.15.39:0/1173531666 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:18.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.129+0000 7ff327e00640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 0x7ff3281656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.129+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675187740 0 0) 0x7ff32806d4c0 con 0x7ff3280b0730 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.129+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff308003520 con 0x7ff3280b0730 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976175910 0 0) 0x7ff308003520 con 0x7ff3280b0730 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 msgr2=0x7ff3281656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 0x7ff3281656a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 >> v1:172.21.15.100:6789/0 conn(0x7ff3280b0350 legacy=0x7ff32806c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3280b15e0 con 0x7ff3280b0730 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff310001670 con 0x7ff3280b0730 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 >> v1:172.21.15.39:6789/0 conn(0x7ff3280b0730 legacy=0x7ff32815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3080039a0 0x7ff308003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 0x7ff3080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff308004d00 0x7ff308005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32806d1e0 con 0x7ff3080039a0 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30800ba30 con 0x7ff308004d00 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30800bbf0 con 0x7ff32815e190 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff327400640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3080039a0 0x7ff308003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff326a00640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 0x7ff3080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff327400640 1 -- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 msgr2=0x7ff3080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.130+0000 7ff327e00640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff308004d00 0x7ff308005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.131+0000 7ff327400640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 0x7ff3080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.131+0000 7ff327400640 1 -- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff308004d00 msgr2=0x7ff308005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:18.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.131+0000 7ff327400640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff308004d00 0x7ff308005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.131+0000 7ff327400640 1 -- 172.21.15.39:0/1173531666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff308003520 con 0x7ff3080039a0 2024-01-24T17:45:18.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.131+0000 7ff326a00640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff32815e190 0x7ff3080047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:18.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.131+0000 7ff327e00640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff308004d00 0x7ff308005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:18.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.132+0000 7ff327400640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3080039a0 0x7ff308003d80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff318001f60 tx=0x7ff3080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37b9e2ea3b04a3c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:18.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.132+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31802c710 con 0x7ff3080039a0 2024-01-24T17:45:18.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.132+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31802c890 con 0x7ff3080039a0 2024-01-24T17:45:18.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.132+0000 7ff326000640 1 -- 172.21.15.39:0/1173531666 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31802cc40 con 0x7ff3080039a0 2024-01-24T17:45:18.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.133+0000 7ff32e800640 1 -- 172.21.15.39:0/1173531666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3080039a0 msgr2=0x7ff308003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.133+0000 7ff32e800640 1 --2- 172.21.15.39:0/1173531666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3080039a0 0x7ff308003d80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff318001f60 tx=0x7ff3080057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.133+0000 7ff327e00640 1 -- 172.21.15.39:0/1173531666 reap_dead start 2024-01-24T17:45:18.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.133+0000 7ff32e800640 1 -- 172.21.15.39:0/1173531666 shutdown_connections 2024-01-24T17:45:18.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.133+0000 7ff32e800640 1 -- 172.21.15.39:0/1173531666 >> 172.21.15.39:0/1173531666 conn(0x7ff3280cb0e0 msgr2=0x7ff3280cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:18.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.133+0000 7ff32e800640 1 -- 172.21.15.39:0/1173531666 shutdown_connections 2024-01-24T17:45:18.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.133+0000 7ff32e800640 1 -- 172.21.15.39:0/1173531666 wait complete. 2024-01-24T17:45:18.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.135+0000 7ff32e800640 1 Processor -- start 2024-01-24T17:45:18.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.135+0000 7ff32e800640 1 -- start start 2024-01-24T17:45:18.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff32e800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3080039a0 0x7ff328150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff32e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff328158a60 0x7ff3281e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff32e800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff328158ee0 0x7ff3281e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff32e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff328159af0 con 0x7ff328158a60 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff32e800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3281579c0 con 0x7ff328158ee0 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff32e800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff328157ca0 con 0x7ff3080039a0 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff326a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff328158a60 0x7ff3281e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff326a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff328158a60 0x7ff3281e4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51374/0 (socket says 172.21.15.39:51374) 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.136+0000 7ff326a00640 1 -- 172.21.15.39:0/2277131942 learned_addr learned my addr 172.21.15.39:0/2277131942 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff327400640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3080039a0 0x7ff328150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff327e00640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff328158ee0 0x7ff3281e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff326a00640 1 -- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3080039a0 msgr2=0x7ff328150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff326a00640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3080039a0 0x7ff328150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff326a00640 1 -- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff328158ee0 msgr2=0x7ff3281e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff326a00640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff328158ee0 0x7ff3281e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff326a00640 1 -- 172.21.15.39:0/2277131942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff328157f80 con 0x7ff328158a60 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff326a00640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff328158a60 0x7ff3281e4480 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff310000f60 tx=0x7ff31000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff31f400640 1 -- 172.21.15.39:0/2277131942 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3100074e0 con 0x7ff328158a60 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff31f400640 1 -- 172.21.15.39:0/2277131942 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff310007640 con 0x7ff328158a60 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff31f400640 1 -- 172.21.15.39:0/2277131942 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310004a00 con 0x7ff328158a60 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.137+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff328158260 con 0x7ff328158a60 2024-01-24T17:45:18.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.138+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff328156d50 con 0x7ff328158a60 2024-01-24T17:45:18.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.139+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff328073ef0 con 0x7ff328158a60 2024-01-24T17:45:18.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.140+0000 7ff31f400640 1 -- 172.21.15.39:0/2277131942 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff310004b60 con 0x7ff328158a60 2024-01-24T17:45:18.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.142+0000 7ff31f400640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff30006a430 0x7ff30006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.142+0000 7ff31f400640 1 -- 172.21.15.39:0/2277131942 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff3100881e0 con 0x7ff328158a60 2024-01-24T17:45:18.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.142+0000 7ff327400640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff30006a430 0x7ff30006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.144+0000 7ff327400640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff30006a430 0x7ff30006c8f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff318036000 tx=0x7ff318009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:18.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.147+0000 7ff31f400640 1 -- 172.21.15.39:0/2277131942 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff310053820 con 0x7ff328158a60 2024-01-24T17:45:18.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.379+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff3280af650 con 0x7ff328158a60 2024-01-24T17:45:18.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.387+0000 7ff31f400640 1 -- 172.21.15.39:0/2277131942 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v31) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff310056e40 con 0x7ff328158a60 2024-01-24T17:45:18.388 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 8 and data pool 9 2024-01-24T17:45:18.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.390+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff30006a430 msgr2=0x7ff30006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.390+0000 7ff32e800640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff30006a430 0x7ff30006c8f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff318036000 tx=0x7ff318009030 comp rx=0 tx=0).stop 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.390+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff328158a60 msgr2=0x7ff3281e4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.390+0000 7ff32e800640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff328158a60 0x7ff3281e4480 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff310000f60 tx=0x7ff31000dc90 comp rx=0 tx=0).stop 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.391+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 shutdown_connections 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.391+0000 7ff32e800640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff328158ee0 0x7ff3281e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.391+0000 7ff32e800640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff30006a430 0x7ff30006c8f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.391+0000 7ff32e800640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff328158a60 0x7ff3281e4480 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.391+0000 7ff32e800640 1 --2- 172.21.15.39:0/2277131942 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3080039a0 0x7ff328150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.391+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 >> 172.21.15.39:0/2277131942 conn(0x7ff3280cb0e0 msgr2=0x7ff328152ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.391+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 shutdown_connections 2024-01-24T17:45:18.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.391+0000 7ff32e800640 1 -- 172.21.15.39:0/2277131942 wait complete. 2024-01-24T17:45:18.424 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:18.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.605+0000 7f2738800640 1 Processor -- start 2024-01-24T17:45:18.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.606+0000 7f2738800640 1 -- start start 2024-01-24T17:45:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.607+0000 7f2738800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 0x7f27300ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.607+0000 7f2738800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27300c9180 con 0x7f27300caaa0 2024-01-24T17:45:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.607+0000 7f2738800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27300c9460 con 0x7f27300cb660 2024-01-24T17:45:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.607+0000 7f2738800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27300b43d0 con 0x7f273015a950 2024-01-24T17:45:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.607+0000 7f2735400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f27300caaa0 0x7f27300caf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56122/0 (socket says 172.21.15.39:56122) 2024-01-24T17:45:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.607+0000 7f2735400640 1 -- 172.21.15.39:0/1420512061 learned_addr learned my addr 172.21.15.39:0/1420512061 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:18.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2735e00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 0x7f27300ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850150943 0 0) 0x7f27300c9180 con 0x7f27300caaa0 2024-01-24T17:45:18.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710003520 con 0x7f27300caaa0 2024-01-24T17:45:18.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2775058002 0 0) 0x7f2710003520 con 0x7f27300caaa0 2024-01-24T17:45:18.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 msgr2=0x7f27300ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:18.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 0x7f27300ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 >> v1:172.21.15.100:6789/0 conn(0x7f27300cb660 legacy=0x7f27300b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27300c9740 con 0x7f27300caaa0 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2718001670 con 0x7f27300caaa0 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 >> v1:172.21.15.39:6789/0 conn(0x7f27300caaa0 legacy=0x7f27300caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27100039a0 0x7f2710003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100047c0 0x7f2710004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 0x7f2710005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27300b43d0 con 0x7f27100039a0 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271000ba60 con 0x7f27100047c0 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.608+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271000bc20 con 0x7f273015a950 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2735e00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27100039a0 0x7f2710003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2735400640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100047c0 0x7f2710004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2736800640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 0x7f2710005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2735e00640 1 -- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 msgr2=0x7f2710005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2735e00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 0x7f2710005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2735e00640 1 -- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100047c0 msgr2=0x7f2710004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2735e00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100047c0 0x7f2710004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2735e00640 1 -- 172.21.15.39:0/1420512061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2710003520 con 0x7f27100039a0 2024-01-24T17:45:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2735400640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100047c0 0x7f2710004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.609+0000 7f2736800640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f273015a950 0x7f2710005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.610+0000 7f2735e00640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27100039a0 0x7f2710003dc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2720009a90 tx=0x7f27100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16164250465eb514 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272002c660 con 0x7f27100039a0 2024-01-24T17:45:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272002c7c0 con 0x7f27100039a0 2024-01-24T17:45:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2734a00640 1 -- 172.21.15.39:0/1420512061 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272002cb50 con 0x7f27100039a0 2024-01-24T17:45:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2738800640 1 -- 172.21.15.39:0/1420512061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27100039a0 msgr2=0x7f2710003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2738800640 1 --2- 172.21.15.39:0/1420512061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27100039a0 0x7f2710003dc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2720009a90 tx=0x7f27100057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2736800640 1 -- 172.21.15.39:0/1420512061 reap_dead start 2024-01-24T17:45:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2738800640 1 -- 172.21.15.39:0/1420512061 shutdown_connections 2024-01-24T17:45:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2738800640 1 -- 172.21.15.39:0/1420512061 >> 172.21.15.39:0/1420512061 conn(0x7f273006a3e0 msgr2=0x7f273006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2738800640 1 -- 172.21.15.39:0/1420512061 shutdown_connections 2024-01-24T17:45:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.611+0000 7f2738800640 1 -- 172.21.15.39:0/1420512061 wait complete. 2024-01-24T17:45:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.613+0000 7f2738800640 1 Processor -- start 2024-01-24T17:45:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.613+0000 7f2738800640 1 -- start start 2024-01-24T17:45:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.614+0000 7f2738800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100039a0 0x7f27300c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.614+0000 7f2738800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27300c7040 0x7f27300c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.614+0000 7f2738800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27300b24e0 0x7f27300b2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.614+0000 7f2738800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27300c7b30 con 0x7f27300c7040 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.614+0000 7f2738800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27300c7e10 con 0x7f27100039a0 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.614+0000 7f2738800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2730144860 con 0x7f27300b24e0 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27300c7040 0x7f27300c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27300c7040 0x7f27300c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51396/0 (socket says 172.21.15.39:51396) 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 -- 172.21.15.39:0/1406025559 learned_addr learned my addr 172.21.15.39:0/1406025559 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2736800640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27300b24e0 0x7f27300b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 -- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27300b24e0 msgr2=0x7f27300b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27300b24e0 0x7f27300b2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735e00640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100039a0 0x7f27300c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 -- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100039a0 msgr2=0x7f27300c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100039a0 0x7f27300c44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 -- 172.21.15.39:0/1406025559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2730144b40 con 0x7f27300c7040 2024-01-24T17:45:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2736800640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27300b24e0 0x7f27300b2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:18.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.615+0000 7f2735400640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27300c7040 0x7f27300c74c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f271800d7e0 tx=0x7f271800dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:18.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.616+0000 7f272e000640 1 -- 172.21.15.39:0/1406025559 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27180074e0 con 0x7f27300c7040 2024-01-24T17:45:18.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.616+0000 7f272e000640 1 -- 172.21.15.39:0/1406025559 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2718007640 con 0x7f27300c7040 2024-01-24T17:45:18.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.616+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27300b2ed0 con 0x7f27300c7040 2024-01-24T17:45:18.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.616+0000 7f272e000640 1 -- 172.21.15.39:0/1406025559 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27180049e0 con 0x7f27300c7040 2024-01-24T17:45:18.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.616+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2730155f10 con 0x7f27300c7040 2024-01-24T17:45:18.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.617+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27301446c0 con 0x7f27300c7040 2024-01-24T17:45:18.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.618+0000 7f272e000640 1 -- 172.21.15.39:0/1406025559 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2718004b40 con 0x7f27300c7040 2024-01-24T17:45:18.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.619+0000 7f272e000640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f270806a430 0x7f270806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:18.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.619+0000 7f272e000640 1 -- 172.21.15.39:0/1406025559 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2718088630 con 0x7f27300c7040 2024-01-24T17:45:18.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.619+0000 7f2735e00640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f270806a430 0x7f270806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.621+0000 7f2735e00640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f270806a430 0x7f270806c8f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2720036000 tx=0x7f2720030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:18.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.623+0000 7f272e000640 1 -- 172.21.15.39:0/1406025559 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f27301446c0 con 0x7f27300c7040 2024-01-24T17:45:18.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.816+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27301446c0 con 0x7f27300c7040 2024-01-24T17:45:18.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.819+0000 7f272e000640 1 -- 172.21.15.39:0/1406025559 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16110 (secure 0 0 0) 0x7f27301446c0 con 0x7f27300c7040 2024-01-24T17:45:18.820 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:18.820 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":62,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:18.381090+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:18.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f270806a430 msgr2=0x7f270806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f270806a430 0x7f270806c8f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2720036000 tx=0x7f2720030000 comp rx=0 tx=0).stop 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27300c7040 msgr2=0x7f27300c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27300c7040 0x7f27300c74c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f271800d7e0 tx=0x7f271800dca0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 shutdown_connections 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f27300b24e0 0x7f27300b2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f270806a430 0x7f270806c8f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27300c7040 0x7f27300c74c0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 --2- 172.21.15.39:0/1406025559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27100039a0 0x7f27300c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.821+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 >> 172.21.15.39:0/1406025559 conn(0x7f273006a3e0 msgr2=0x7f273006fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.822+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 shutdown_connections 2024-01-24T17:45:18.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:18.822+0000 7f2738800640 1 -- 172.21.15.39:0/1406025559 wait complete. 2024-01-24T17:45:18.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:19.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.058+0000 7f89d3200640 1 Processor -- start 2024-01-24T17:45:19.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.059+0000 7f89d3200640 1 -- start start 2024-01-24T17:45:19.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.060+0000 7f89d3200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f89cc0cf990 0x7f89cc0cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.060+0000 7f89d3200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89cc15fb90 0x7f89cc15d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.060+0000 7f89d3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc0d02b0 con 0x7f89cc15d020 2024-01-24T17:45:19.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.060+0000 7f89d3200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89cc15e080 con 0x7f89cc0cf990 2024-01-24T17:45:19.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.060+0000 7f89d3200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc15dac0 con 0x7f89cc15fb90 2024-01-24T17:45:19.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.061+0000 7f89cb400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f89cc15d020 0x7f89cc160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56128/0 (socket says 172.21.15.39:56128) 2024-01-24T17:45:19.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.061+0000 7f89cb400640 1 -- 172.21.15.39:0/4216339683 learned_addr learned my addr 172.21.15.39:0/4216339683 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:19.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.061+0000 7f89cbe00640 1 --2- 172.21.15.39:0/4216339683 >> v2:172.21.15.100:3300/0 conn(0x7f89cc0cf990 0x7f89cc0cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.061+0000 7f89d1200640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89cc15fb90 0x7f89cc15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.061+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209827595 0 0) 0x7f89cc0d02b0 con 0x7f89cc15d020 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.061+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89ac003520 con 0x7f89cc15d020 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3122786290 0 0) 0x7f89ac003520 con 0x7f89cc15d020 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89cc15fb90 msgr2=0x7f89cc15d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89cc15fb90 0x7f89cc15d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 >> v2:172.21.15.100:3300/0 conn(0x7f89cc0cf990 msgr2=0x7f89cc0cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 --2- 172.21.15.39:0/4216339683 >> v2:172.21.15.100:3300/0 conn(0x7f89cc0cf990 0x7f89cc0cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc15dda0 con 0x7f89cc15d020 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f89b4001670 con 0x7f89cc15d020 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 >> v1:172.21.15.39:6789/0 conn(0x7f89cc15d020 legacy=0x7f89cc160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89ac003970 0x7f89ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89ac0047c0 0x7f89ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89cc15fb90 0x7f89ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc15e080 con 0x7f89ac0047c0 2024-01-24T17:45:19.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc15dac0 con 0x7f89ac003970 2024-01-24T17:45:19.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac00b980 con 0x7f89cc15fb90 2024-01-24T17:45:19.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89cb400640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89ac0047c0 0x7f89ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89cb400640 1 -- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89cc15fb90 msgr2=0x7f89ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:19.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89cb400640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89cc15fb90 0x7f89ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89cb400640 1 -- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89ac003970 msgr2=0x7f89ac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:19.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89cb400640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89ac003970 0x7f89ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.062+0000 7f89cb400640 1 -- 172.21.15.39:0/4216339683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac003520 con 0x7f89ac0047c0 2024-01-24T17:45:19.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.063+0000 7f89cb400640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89ac0047c0 0x7f89ac004c40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f89b400c240 tx=0x7f89ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9533fa8d3a3f39fe server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:19.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.064+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b402cb00 con 0x7f89ac0047c0 2024-01-24T17:45:19.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.064+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b402cc60 con 0x7f89ac0047c0 2024-01-24T17:45:19.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.064+0000 7f89caa00640 1 -- 172.21.15.39:0/4216339683 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b40058d0 con 0x7f89ac0047c0 2024-01-24T17:45:19.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.064+0000 7f89d3200640 1 -- 172.21.15.39:0/4216339683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89ac0047c0 msgr2=0x7f89ac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.064+0000 7f89d3200640 1 --2- 172.21.15.39:0/4216339683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89ac0047c0 0x7f89ac004c40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f89b400c240 tx=0x7f89ac0057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.064+0000 7f89d1200640 1 -- 172.21.15.39:0/4216339683 reap_dead start 2024-01-24T17:45:19.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.064+0000 7f89d3200640 1 -- 172.21.15.39:0/4216339683 shutdown_connections 2024-01-24T17:45:19.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.064+0000 7f89d3200640 1 -- 172.21.15.39:0/4216339683 >> 172.21.15.39:0/4216339683 conn(0x7f89cc071790 msgr2=0x7f89cc071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:19.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.065+0000 7f89d3200640 1 -- 172.21.15.39:0/4216339683 shutdown_connections 2024-01-24T17:45:19.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.065+0000 7f89d3200640 1 -- 172.21.15.39:0/4216339683 wait complete. 2024-01-24T17:45:19.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.066+0000 7f89d3200640 1 Processor -- start 2024-01-24T17:45:19.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.066+0000 7f89d3200640 1 -- start start 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.067+0000 7f89d3200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89ac0047c0 0x7f89cc148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.067+0000 7f89d3200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89cc149320 0x7f89cc0b6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.067+0000 7f89d3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89cc1497a0 0x7f89cc0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.067+0000 7f89d3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc14c3a0 con 0x7f89cc1497a0 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.067+0000 7f89d3200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0cbd30 con 0x7f89cc149320 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.067+0000 7f89d3200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc149010 con 0x7f89ac0047c0 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89d1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89cc1497a0 0x7f89cc0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89d1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89cc1497a0 0x7f89cc0b7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51414/0 (socket says 172.21.15.39:51414) 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89d1200640 1 -- 172.21.15.39:0/837740456 learned_addr learned my addr 172.21.15.39:0/837740456 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89cbe00640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89ac0047c0 0x7f89cc148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89cb400640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89cc149320 0x7f89cc0b6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89d1200640 1 -- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89ac0047c0 msgr2=0x7f89cc148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89d1200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89ac0047c0 0x7f89cc148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89d1200640 1 -- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89cc149320 msgr2=0x7f89cc0b6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89d1200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89cc149320 0x7f89cc0b6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89d1200640 1 -- 172.21.15.39:0/837740456 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc14c0c0 con 0x7f89cc1497a0 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.068+0000 7f89cb400640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89cc149320 0x7f89cc0b6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.069+0000 7f89d1200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89cc1497a0 0x7f89cc0b7090 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0b5270 tx=0x7f89c400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.069+0000 7f89c3e00640 1 -- 172.21.15.39:0/837740456 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c4010040 con 0x7f89cc1497a0 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.069+0000 7f89c3e00640 1 -- 172.21.15.39:0/837740456 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c400bae0 con 0x7f89cc1497a0 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.069+0000 7f89c3e00640 1 -- 172.21.15.39:0/837740456 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c400be70 con 0x7f89cc1497a0 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.069+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc0cbfd0 con 0x7f89cc1497a0 2024-01-24T17:45:19.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.069+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc0cc590 con 0x7f89cc1497a0 2024-01-24T17:45:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.070+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc073ef0 con 0x7f89cc1497a0 2024-01-24T17:45:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.071+0000 7f89c3e00640 1 -- 172.21.15.39:0/837740456 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f89c401e410 con 0x7f89cc1497a0 2024-01-24T17:45:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.072+0000 7f89c3e00640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89a406a460 0x7f89a406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.072+0000 7f89c3e00640 1 -- 172.21.15.39:0/837740456 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89c4088bc0 con 0x7f89cc1497a0 2024-01-24T17:45:19.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.073+0000 7f89cbe00640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89a406a460 0x7f89a406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.074+0000 7f89cbe00640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89a406a460 0x7f89a406c920 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0047b0 tx=0x7f89bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:19.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.076+0000 7f89c3e00640 1 -- 172.21.15.39:0/837740456 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f89c4057ba0 con 0x7f89cc1497a0 2024-01-24T17:45:19.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.272+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89cc0b2140 con 0x7f89cc1497a0 2024-01-24T17:45:19.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.275+0000 7f89c3e00640 1 -- 172.21.15.39:0/837740456 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16110 (secure 0 0 0) 0x7f89c4057730 con 0x7f89cc1497a0 2024-01-24T17:45:19.276 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:19.276 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":62,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:18.381090+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:19.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.277+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89a406a460 msgr2=0x7f89a406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.277+0000 7f89d3200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89a406a460 0x7f89a406c920 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f89bc0047b0 tx=0x7f89bc008040 comp rx=0 tx=0).stop 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.277+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89cc1497a0 msgr2=0x7f89cc0b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.277+0000 7f89d3200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89cc1497a0 0x7f89cc0b7090 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0b5270 tx=0x7f89c400b480 comp rx=0 tx=0).stop 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.278+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 shutdown_connections 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.278+0000 7f89d3200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f89a406a460 0x7f89a406c920 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.278+0000 7f89d3200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89cc1497a0 0x7f89cc0b7090 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.278+0000 7f89d3200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89cc149320 0x7f89cc0b6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.278+0000 7f89d3200640 1 --2- 172.21.15.39:0/837740456 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89ac0047c0 0x7f89cc148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.278+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 >> 172.21.15.39:0/837740456 conn(0x7f89cc071790 msgr2=0x7f89cc0c5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.278+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 shutdown_connections 2024-01-24T17:45:19.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.278+0000 7f89d3200640 1 -- 172.21.15.39:0/837740456 wait complete. 2024-01-24T17:45:19.299 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T17:45:19.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.513+0000 7f801f200640 1 Processor -- start 2024-01-24T17:45:19.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.514+0000 7f801f200640 1 -- start start 2024-01-24T17:45:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.515+0000 7f801f200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f802015cad0 0x7f80200d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.515+0000 7f801f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200b16e0 0x7f80200d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801f200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200d37b0 con 0x7f80200d1350 2024-01-24T17:45:19.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801f200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80200cfe10 con 0x7f802015cad0 2024-01-24T17:45:19.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80200d1780 con 0x7f80200b16e0 2024-01-24T17:45:19.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f802015cad0 0x7f80200d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f802015cad0 0x7f80200d0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51418/0 (socket says 172.21.15.39:51418) 2024-01-24T17:45:19.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801de00640 1 -- 172.21.15.39:0/3588565547 learned_addr learned my addr 172.21.15.39:0/3588565547 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:19.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801d400640 1 --2- 172.21.15.39:0/3588565547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200b16e0 0x7f80200d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801de00640 1 -- 172.21.15.39:0/3588565547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200b16e0 msgr2=0x7f80200d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:19.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.516+0000 7f801de00640 1 --2- 172.21.15.39:0/3588565547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200b16e0 0x7f80200d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.517+0000 7f801de00640 1 -- 172.21.15.39:0/3588565547 >> v1:172.21.15.100:6789/0 conn(0x7f80200d1350 legacy=0x7f8020160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:19.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.517+0000 7f801de00640 1 -- 172.21.15.39:0/3588565547 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80200cecf0 con 0x7f802015cad0 2024-01-24T17:45:19.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.518+0000 7f801de00640 1 --2- 172.21.15.39:0/3588565547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f80200d0610 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8010009850 tx=0x7f801002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b33b3faf20d2dc3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:19.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.518+0000 7f801ca00640 1 -- 172.21.15.39:0/3588565547 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010004230 con 0x7f802015cad0 2024-01-24T17:45:19.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.518+0000 7f801ca00640 1 -- 172.21.15.39:0/3588565547 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801002f050 con 0x7f802015cad0 2024-01-24T17:45:19.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.518+0000 7f801ca00640 1 -- 172.21.15.39:0/3588565547 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010003610 con 0x7f802015cad0 2024-01-24T17:45:19.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.519+0000 7f801f200640 1 -- 172.21.15.39:0/3588565547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 msgr2=0x7f80200d0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.519+0000 7f801f200640 1 --2- 172.21.15.39:0/3588565547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f80200d0610 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8010009850 tx=0x7f801002b7b0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.519+0000 7f801f200640 1 -- 172.21.15.39:0/3588565547 shutdown_connections 2024-01-24T17:45:19.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.519+0000 7f801f200640 1 --2- 172.21.15.39:0/3588565547 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200b16e0 0x7f80200d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.519+0000 7f801f200640 1 --2- 172.21.15.39:0/3588565547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f80200d0610 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.519+0000 7f801f200640 1 -- 172.21.15.39:0/3588565547 >> 172.21.15.39:0/3588565547 conn(0x7f80200d2830 msgr2=0x7f80200d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:19.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.519+0000 7f801f200640 1 -- 172.21.15.39:0/3588565547 shutdown_connections 2024-01-24T17:45:19.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.519+0000 7f801f200640 1 -- 172.21.15.39:0/3588565547 wait complete. 2024-01-24T17:45:19.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.521+0000 7f801f200640 1 Processor -- start 2024-01-24T17:45:19.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.521+0000 7f801f200640 1 -- start start 2024-01-24T17:45:19.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80200b16e0 0x7f802014ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200d1350 0x7f8020148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f802014ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200cb150 con 0x7f802015cad0 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801f200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200b7320 con 0x7f80200b16e0 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80200b7600 con 0x7f80200d1350 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f802014ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f802014ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51426/0 (socket says 172.21.15.39:51426) 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.522+0000 7f801e800640 1 -- 172.21.15.39:0/2297149083 learned_addr learned my addr 172.21.15.39:0/2297149083 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801d400640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200d1350 0x7f8020148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801de00640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80200b16e0 0x7f802014ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801e800640 1 -- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200d1350 msgr2=0x7f8020148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801e800640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200d1350 0x7f8020148340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801e800640 1 -- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80200b16e0 msgr2=0x7f802014ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801e800640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80200b16e0 0x7f802014ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801e800640 1 -- 172.21.15.39:0/2297149083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8010009460 con 0x7f802015cad0 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801d400640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200d1350 0x7f8020148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801e800640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f802014ace0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f801800b410 tx=0x7f801800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f8016000640 1 -- 172.21.15.39:0/2297149083 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018004020 con 0x7f802015cad0 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f8016000640 1 -- 172.21.15.39:0/2297149083 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8018004180 con 0x7f802015cad0 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80200b78e0 con 0x7f802015cad0 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f8016000640 1 -- 172.21.15.39:0/2297149083 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018011630 con 0x7f802015cad0 2024-01-24T17:45:19.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.523+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80200c9ba0 con 0x7f802015cad0 2024-01-24T17:45:19.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.524+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80200cb350 con 0x7f802015cad0 2024-01-24T17:45:19.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.525+0000 7f8016000640 1 -- 172.21.15.39:0/2297149083 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8018011790 con 0x7f802015cad0 2024-01-24T17:45:19.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.526+0000 7f8016000640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ff806a430 0x7f7ff806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:19.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.526+0000 7f8016000640 1 -- 172.21.15.39:0/2297149083 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8018088630 con 0x7f802015cad0 2024-01-24T17:45:19.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.527+0000 7f801de00640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ff806a430 0x7f7ff806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:19.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.528+0000 7f801de00640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ff806a430 0x7f7ff806c8f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8010035000 tx=0x7f8010008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:19.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.531+0000 7f8016000640 1 -- 172.21.15.39:0/2297149083 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f80200cb350 con 0x7f802015cad0 2024-01-24T17:45:19.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:19.769+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f80200cb320 con 0x7f802015cad0 2024-01-24T17:45:20.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.395+0000 7f8016000640 1 -- 172.21.15.39:0/2297149083 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8018053c10 con 0x7f802015cad0 2024-01-24T17:45:20.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ff806a430 msgr2=0x7f7ff806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ff806a430 0x7f7ff806c8f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8010035000 tx=0x7f8010008030 comp rx=0 tx=0).stop 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 msgr2=0x7f802014ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f802014ace0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f801800b410 tx=0x7f801800b8d0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 shutdown_connections 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ff806a430 0x7f7ff806c8f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f802015cad0 0x7f802014ace0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80200d1350 0x7f8020148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 --2- 172.21.15.39:0/2297149083 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80200b16e0 0x7f802014ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 >> 172.21.15.39:0/2297149083 conn(0x7f80200d2830 msgr2=0x7f80200b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 shutdown_connections 2024-01-24T17:45:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.399+0000 7f801f200640 1 -- 172.21.15.39:0/2297149083 wait complete. 2024-01-24T17:45:20.419 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-24T17:45:20.420 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-24T17:45:20.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.642+0000 7fbd8ae00640 1 Processor -- start 2024-01-24T17:45:20.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.643+0000 7fbd8ae00640 1 -- start start 2024-01-24T17:45:20.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.644+0000 7fbd8ae00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbd8415a6d0 0x7fbd840c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:20.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.644+0000 7fbd8ae00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415aab0 0x7fbd840c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:20.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.644+0000 7fbd8ae00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd840caaa0 con 0x7fbd8415e480 2024-01-24T17:45:20.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.644+0000 7fbd8ae00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd840ca800 con 0x7fbd8415a6d0 2024-01-24T17:45:20.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.644+0000 7fbd8ae00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840c94d0 con 0x7fbd8415aab0 2024-01-24T17:45:20.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.644+0000 7fbd89000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415aab0 0x7fbd840c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:20.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.645+0000 7fbd89a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbd8415a6d0 0x7fbd840c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:20.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.645+0000 7fbd89000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415aab0 0x7fbd840c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:32780/0 (socket says 172.21.15.39:32780) 2024-01-24T17:45:20.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.645+0000 7fbd89000640 1 -- 172.21.15.39:0/1145137483 learned_addr learned my addr 172.21.15.39:0/1145137483 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:20.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.645+0000 7fbd89000640 1 -- 172.21.15.39:0/1145137483 >> v2:172.21.15.100:3300/0 conn(0x7fbd8415a6d0 msgr2=0x7fbd840c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:20.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.645+0000 7fbd89000640 1 --2- 172.21.15.39:0/1145137483 >> v2:172.21.15.100:3300/0 conn(0x7fbd8415a6d0 0x7fbd840c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.645+0000 7fbd89000640 1 -- 172.21.15.39:0/1145137483 >> v1:172.21.15.100:6789/0 conn(0x7fbd8415e480 legacy=0x7fbd8406fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:20.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.645+0000 7fbd89000640 1 -- 172.21.15.39:0/1145137483 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd840c9750 con 0x7fbd8415aab0 2024-01-24T17:45:20.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.647+0000 7fbd89000640 1 --2- 172.21.15.39:0/1145137483 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415aab0 0x7fbd840c1f80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0098a0 tx=0x7fbd6c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2f0c730af8ace61 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:20.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.647+0000 7fbd83e00640 1 -- 172.21.15.39:0/1145137483 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0317c0 con 0x7fbd8415aab0 2024-01-24T17:45:20.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.647+0000 7fbd83e00640 1 -- 172.21.15.39:0/1145137483 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c031920 con 0x7fbd8415aab0 2024-01-24T17:45:20.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.647+0000 7fbd83e00640 1 -- 172.21.15.39:0/1145137483 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c031cb0 con 0x7fbd8415aab0 2024-01-24T17:45:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.648+0000 7fbd8ae00640 1 -- 172.21.15.39:0/1145137483 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415aab0 msgr2=0x7fbd840c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.648+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/1145137483 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415aab0 0x7fbd840c1f80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0098a0 tx=0x7fbd6c02b860 comp rx=0 tx=0).stop 2024-01-24T17:45:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.648+0000 7fbd8ae00640 1 -- 172.21.15.39:0/1145137483 shutdown_connections 2024-01-24T17:45:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.648+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/1145137483 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415aab0 0x7fbd840c1f80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.648+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/1145137483 >> v2:172.21.15.100:3300/0 conn(0x7fbd8415a6d0 0x7fbd840c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.648+0000 7fbd8ae00640 1 -- 172.21.15.39:0/1145137483 >> 172.21.15.39:0/1145137483 conn(0x7fbd8406a2a0 msgr2=0x7fbd840aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.648+0000 7fbd8ae00640 1 -- 172.21.15.39:0/1145137483 shutdown_connections 2024-01-24T17:45:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.648+0000 7fbd8ae00640 1 -- 172.21.15.39:0/1145137483 wait complete. 2024-01-24T17:45:20.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.650+0000 7fbd8ae00640 1 Processor -- start 2024-01-24T17:45:20.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.650+0000 7fbd8ae00640 1 -- start start 2024-01-24T17:45:20.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.651+0000 7fbd8ae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd8415a6d0 0x7fbd841f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.651+0000 7fbd8ae00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd8415aab0 0x7fbd8415cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.651+0000 7fbd8ae00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415e480 0x7fbd8415d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.651+0000 7fbd8ae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840abfb0 con 0x7fbd8415a6d0 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.651+0000 7fbd8ae00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8415db90 con 0x7fbd8415aab0 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.651+0000 7fbd8ae00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8415de70 con 0x7fbd8415e480 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd8415a6d0 0x7fbd841f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd8415a6d0 0x7fbd841f5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51438/0 (socket says 172.21.15.39:51438) 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89a00640 1 -- 172.21.15.39:0/571643947 learned_addr learned my addr 172.21.15.39:0/571643947 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89000640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd8415aab0 0x7fbd8415cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd8a400640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415e480 0x7fbd8415d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89a00640 1 -- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415e480 msgr2=0x7fbd8415d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89a00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415e480 0x7fbd8415d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89a00640 1 -- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd8415aab0 msgr2=0x7fbd8415cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:20.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89a00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd8415aab0 0x7fbd8415cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:20.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.652+0000 7fbd89a00640 1 -- 172.21.15.39:0/571643947 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c0094d0 con 0x7fbd8415a6d0 2024-01-24T17:45:20.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.653+0000 7fbd89a00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd8415a6d0 0x7fbd841f5aa0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400e870 tx=0x7fbd7400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:20.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.653+0000 7fbd81600640 1 -- 172.21.15.39:0/571643947 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd740094f0 con 0x7fbd8415a6d0 2024-01-24T17:45:20.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.653+0000 7fbd81600640 1 -- 172.21.15.39:0/571643947 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7400c940 con 0x7fbd8415a6d0 2024-01-24T17:45:20.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.653+0000 7fbd81600640 1 -- 172.21.15.39:0/571643947 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7400ccf0 con 0x7fbd8415a6d0 2024-01-24T17:45:20.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.653+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd840ab960 con 0x7fbd8415a6d0 2024-01-24T17:45:20.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.653+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8415d850 con 0x7fbd8415a6d0 2024-01-24T17:45:20.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.654+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8404dff0 con 0x7fbd8415a6d0 2024-01-24T17:45:20.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.655+0000 7fbd81600640 1 -- 172.21.15.39:0/571643947 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbd74013470 con 0x7fbd8415a6d0 2024-01-24T17:45:20.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.656+0000 7fbd81600640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd5c06a430 0x7fbd5c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:20.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.656+0000 7fbd81600640 1 -- 172.21.15.39:0/571643947 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd74088a30 con 0x7fbd8415a6d0 2024-01-24T17:45:20.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.657+0000 7fbd89000640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd5c06a430 0x7fbd5c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:20.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.658+0000 7fbd89000640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd5c06a430 0x7fbd5c06c8f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c036000 tx=0x7fbd6c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:20.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.660+0000 7fbd81600640 1 -- 172.21.15.39:0/571643947 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbd74054030 con 0x7fbd8415a6d0 2024-01-24T17:45:20.687 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:45:20.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:45:20.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:20.897+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbd840b0d80 con 0x7fbd8415a6d0 2024-01-24T17:45:21.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.399+0000 7fbd81600640 1 -- 172.21.15.39:0/571643947 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbd740210c0 con 0x7fbd8415a6d0 2024-01-24T17:45:21.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.402+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd5c06a430 msgr2=0x7fbd5c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:21.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.402+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd5c06a430 0x7fbd5c06c8f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c036000 tx=0x7fbd6c03d040 comp rx=0 tx=0).stop 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.402+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd8415a6d0 msgr2=0x7fbd841f5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.402+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd8415a6d0 0x7fbd841f5aa0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400e870 tx=0x7fbd7400ed30 comp rx=0 tx=0).stop 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.403+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 shutdown_connections 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.403+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbd8415e480 0x7fbd8415d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.403+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbd8415aab0 0x7fbd8415cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.403+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd5c06a430 0x7fbd5c06c8f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.403+0000 7fbd8ae00640 1 --2- 172.21.15.39:0/571643947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd8415a6d0 0x7fbd841f5aa0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.403+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 >> 172.21.15.39:0/571643947 conn(0x7fbd8406a2a0 msgr2=0x7fbd840c40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.403+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 shutdown_connections 2024-01-24T17:45:21.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.403+0000 7fbd8ae00640 1 -- 172.21.15.39:0/571643947 wait complete. 2024-01-24T17:45:21.424 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-24T17:45:21.424 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:45:21.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.640+0000 7f2039400640 1 Processor -- start 2024-01-24T17:45:21.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.640+0000 7f2039400640 1 -- start start 2024-01-24T17:45:21.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.642+0000 7f2039400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f203415a720 0x7f20340cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:21.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.642+0000 7f2039400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 0x7f20340cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:21.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.642+0000 7f2039400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20340ce400 con 0x7f20340ccd20 2024-01-24T17:45:21.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.642+0000 7f2039400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20340ce110 con 0x7f203415a720 2024-01-24T17:45:21.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.642+0000 7f2039400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203415b3b0 con 0x7f203415ab00 2024-01-24T17:45:21.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.642+0000 7f2033e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f203415a720 0x7f20340cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:21.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.642+0000 7f2033e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f203415a720 0x7f20340cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51442/0 (socket says 172.21.15.39:51442) 2024-01-24T17:45:21.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.642+0000 7f2033e00640 1 -- 172.21.15.39:0/4104869050 learned_addr learned my addr 172.21.15.39:0/4104869050 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:21.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.643+0000 7f2033400640 1 --2- 172.21.15.39:0/4104869050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 0x7f20340cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:21.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.643+0000 7f2033e00640 1 -- 172.21.15.39:0/4104869050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 msgr2=0x7f20340cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:21.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.643+0000 7f2033e00640 1 --2- 172.21.15.39:0/4104869050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 0x7f20340cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.643+0000 7f2033e00640 1 -- 172.21.15.39:0/4104869050 >> v1:172.21.15.100:6789/0 conn(0x7f20340ccd20 legacy=0x7f20340ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:21.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.643+0000 7f2033e00640 1 -- 172.21.15.39:0/4104869050 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203415b690 con 0x7f203415a720 2024-01-24T17:45:21.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.644+0000 7f2033e00640 1 --2- 172.21.15.39:0/4104869050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20340cdbd0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2024009870 tx=0x7f202402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0f29f3d4dcc999 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:21.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.644+0000 7f2032a00640 1 -- 172.21.15.39:0/4104869050 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20240317c0 con 0x7f203415a720 2024-01-24T17:45:21.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.644+0000 7f2032a00640 1 -- 172.21.15.39:0/4104869050 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2024031920 con 0x7f203415a720 2024-01-24T17:45:21.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.644+0000 7f2032a00640 1 -- 172.21.15.39:0/4104869050 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024031cb0 con 0x7f203415a720 2024-01-24T17:45:21.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.645+0000 7f2039400640 1 -- 172.21.15.39:0/4104869050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 msgr2=0x7f20340cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:21.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.645+0000 7f2039400640 1 --2- 172.21.15.39:0/4104869050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20340cdbd0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2024009870 tx=0x7f202402b860 comp rx=0 tx=0).stop 2024-01-24T17:45:21.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.645+0000 7f2039400640 1 -- 172.21.15.39:0/4104869050 shutdown_connections 2024-01-24T17:45:21.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.645+0000 7f2039400640 1 --2- 172.21.15.39:0/4104869050 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 0x7f20340cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.645+0000 7f2039400640 1 --2- 172.21.15.39:0/4104869050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20340cdbd0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.645+0000 7f2039400640 1 -- 172.21.15.39:0/4104869050 >> 172.21.15.39:0/4104869050 conn(0x7f203406a090 msgr2=0x7f203406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:21.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.645+0000 7f2039400640 1 -- 172.21.15.39:0/4104869050 shutdown_connections 2024-01-24T17:45:21.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.645+0000 7f2039400640 1 -- 172.21.15.39:0/4104869050 wait complete. 2024-01-24T17:45:21.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.647+0000 7f2039400640 1 Processor -- start 2024-01-24T17:45:21.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.647+0000 7f2039400640 1 -- start start 2024-01-24T17:45:21.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2039400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20341ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:21.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2039400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 0x7f20340b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:21.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2039400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f203414c570 0x7f20340b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:21.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2039400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20340b3da0 con 0x7f203415a720 2024-01-24T17:45:21.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2039400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203414c360 con 0x7f203414c570 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2039400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20341efef0 con 0x7f203415ab00 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2033e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20341ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2033e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20341ef9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51452/0 (socket says 172.21.15.39:51452) 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.648+0000 7f2033e00640 1 -- 172.21.15.39:0/233890642 learned_addr learned my addr 172.21.15.39:0/233890642 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2033400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 0x7f20340b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2038a00640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f203414c570 0x7f20340b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2033e00640 1 -- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 msgr2=0x7f20340b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2033e00640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 0x7f20340b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2033e00640 1 -- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f203414c570 msgr2=0x7f20340b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2033e00640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f203414c570 0x7f20340b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2033e00640 1 -- 172.21.15.39:0/233890642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20240094d0 con 0x7f203415a720 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2038a00640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f203414c570 0x7f20340b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f2033e00640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20341ef9b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f202402b8a0 tx=0x7f20240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:21.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.649+0000 7f202be00640 1 -- 172.21.15.39:0/233890642 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024032860 con 0x7f203415a720 2024-01-24T17:45:21.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.650+0000 7f202be00640 1 -- 172.21.15.39:0/233890642 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20240329c0 con 0x7f203415a720 2024-01-24T17:45:21.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.650+0000 7f202be00640 1 -- 172.21.15.39:0/233890642 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024032d70 con 0x7f203415a720 2024-01-24T17:45:21.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.650+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20340c6d20 con 0x7f203415a720 2024-01-24T17:45:21.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.650+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20340c66d0 con 0x7f203415a720 2024-01-24T17:45:21.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.651+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20340c6530 con 0x7f203415a720 2024-01-24T17:45:21.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.652+0000 7f202be00640 1 -- 172.21.15.39:0/233890642 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f202404e450 con 0x7f203415a720 2024-01-24T17:45:21.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.652+0000 7f202be00640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f200c06a430 0x7f200c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:21.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.653+0000 7f202be00640 1 -- 172.21.15.39:0/233890642 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20240aab20 con 0x7f203415a720 2024-01-24T17:45:21.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.653+0000 7f2033400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f200c06a430 0x7f200c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:21.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.654+0000 7f2033400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f200c06a430 0x7f200c06c8f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f20340b01d0 tx=0x7f201c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:21.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.657+0000 7f202be00640 1 -- 172.21.15.39:0/233890642 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f20340c6530 con 0x7f203415a720 2024-01-24T17:45:21.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.888+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20340c6530 con 0x7f203415a720 2024-01-24T17:45:21.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.889+0000 7f202be00640 1 -- 172.21.15.39:0/233890642 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f20340c6530 con 0x7f203415a720 2024-01-24T17:45:21.890 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:21.890 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3045325513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3045325513},{"type":"v1","addr":"172.21.15.100:6836","nonce":3045325513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5126,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/858868211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":858868211},{"type":"v1","addr":"172.21.15.100:6837","nonce":858868211}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3121826441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3121826441},{"type":"v1","addr":"172.21.15.39:6833","nonce":3121826441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:45:18.381640+0000","modified":"2024-01-24T17:45:21.392053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6261":{"gid":6261,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/527429915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":527429915},{"type":"v1","addr":"172.21.15.39:6835","nonce":527429915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-24T17:45:21.890 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f200c06a430 msgr2=0x7f200c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f200c06a430 0x7f200c06c8f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f20340b01d0 tx=0x7f201c008040 comp rx=0 tx=0).stop 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 msgr2=0x7f20341ef9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20341ef9b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f202402b8a0 tx=0x7f20240046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 shutdown_connections 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f203414c570 0x7f20340b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f203415ab00 0x7f20340b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f200c06a430 0x7f200c06c8f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 --2- 172.21.15.39:0/233890642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f203415a720 0x7f20341ef9b0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 >> 172.21.15.39:0/233890642 conn(0x7f203406a090 msgr2=0x7f20340c1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:21.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 shutdown_connections 2024-01-24T17:45:21.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:21.892+0000 7f2039400640 1 -- 172.21.15.39:0/233890642 wait complete. 2024-01-24T17:45:21.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:22.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.138+0000 7f85a6a00640 1 Processor -- start 2024-01-24T17:45:22.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.139+0000 7f85a6a00640 1 -- start start 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.140+0000 7f85a6a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f85a015ca30 0x7f85a00cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.140+0000 7f85a6a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a00cd3c0 0x7f85a00c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.140+0000 7f85a6a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a00c9c10 con 0x7f85a00c90d0 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.140+0000 7f85a6a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a00c9650 con 0x7f85a015ca30 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.140+0000 7f85a6a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a00c9930 con 0x7f85a00cd3c0 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.141+0000 7f859f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a00cd3c0 0x7f85a00c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.141+0000 7f859fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f85a015ca30 0x7f85a00cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.141+0000 7f859f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a00cd3c0 0x7f85a00c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:32820/0 (socket says 172.21.15.39:32820) 2024-01-24T17:45:22.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.141+0000 7f859f400640 1 -- 172.21.15.39:0/4011006061 learned_addr learned my addr 172.21.15.39:0/4011006061 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:22.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.142+0000 7f859f400640 1 -- 172.21.15.39:0/4011006061 >> v2:172.21.15.100:3300/0 conn(0x7f85a015ca30 msgr2=0x7f85a00cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:22.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.142+0000 7f859f400640 1 --2- 172.21.15.39:0/4011006061 >> v2:172.21.15.100:3300/0 conn(0x7f85a015ca30 0x7f85a00cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.142+0000 7f859f400640 1 -- 172.21.15.39:0/4011006061 >> v1:172.21.15.100:6789/0 conn(0x7f85a00c90d0 legacy=0x7f85a00caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.142+0000 7f859f400640 1 -- 172.21.15.39:0/4011006061 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a00cc680 con 0x7f85a00cd3c0 2024-01-24T17:45:22.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.143+0000 7f859f400640 1 --2- 172.21.15.39:0/4011006061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a00cd3c0 0x7f85a00c8b90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8588009870 tx=0x7f858802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad9e09d40edf069 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:22.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f859ea00640 1 -- 172.21.15.39:0/4011006061 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588030840 con 0x7f85a00cd3c0 2024-01-24T17:45:22.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f859ea00640 1 -- 172.21.15.39:0/4011006061 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85880309a0 con 0x7f85a00cd3c0 2024-01-24T17:45:22.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f859ea00640 1 -- 172.21.15.39:0/4011006061 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588030d30 con 0x7f85a00cd3c0 2024-01-24T17:45:22.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f85a6a00640 1 -- 172.21.15.39:0/4011006061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a00cd3c0 msgr2=0x7f85a00c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f85a6a00640 1 --2- 172.21.15.39:0/4011006061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a00cd3c0 0x7f85a00c8b90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8588009870 tx=0x7f858802b980 comp rx=0 tx=0).stop 2024-01-24T17:45:22.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f85a6a00640 1 -- 172.21.15.39:0/4011006061 shutdown_connections 2024-01-24T17:45:22.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f85a6a00640 1 --2- 172.21.15.39:0/4011006061 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a00cd3c0 0x7f85a00c8b90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f85a6a00640 1 --2- 172.21.15.39:0/4011006061 >> v2:172.21.15.100:3300/0 conn(0x7f85a015ca30 0x7f85a00cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.144+0000 7f85a6a00640 1 -- 172.21.15.39:0/4011006061 >> 172.21.15.39:0/4011006061 conn(0x7f85a006a3e0 msgr2=0x7f85a006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:22.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.145+0000 7f85a6a00640 1 -- 172.21.15.39:0/4011006061 shutdown_connections 2024-01-24T17:45:22.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.145+0000 7f85a6a00640 1 -- 172.21.15.39:0/4011006061 wait complete. 2024-01-24T17:45:22.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.146+0000 7f85a6a00640 1 Processor -- start 2024-01-24T17:45:22.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.146+0000 7f85a6a00640 1 -- start start 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.147+0000 7f85a6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85a00c90d0 0x7f85a015d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.147+0000 7f85a6a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a015ca30 0x7f85a015a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.147+0000 7f85a6a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85a00b3680 0x7f85a00b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.147+0000 7f85a6a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a00c2050 con 0x7f85a00c90d0 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.147+0000 7f85a6a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a015b650 con 0x7f85a00b3680 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.147+0000 7f85a6a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a00c2260 con 0x7f85a015ca30 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f859fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85a00c90d0 0x7f85a015d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f859fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85a00c90d0 0x7f85a015d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51464/0 (socket says 172.21.15.39:51464) 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f859fe00640 1 -- 172.21.15.39:0/55222139 learned_addr learned my addr 172.21.15.39:0/55222139 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f85a4a00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85a00b3680 0x7f85a00b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f859fe00640 1 -- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a015ca30 msgr2=0x7f85a015a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f859fe00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a015ca30 0x7f85a015a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f859fe00640 1 -- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85a00b3680 msgr2=0x7f85a00b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f859fe00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85a00b3680 0x7f85a00b3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.148+0000 7f859fe00640 1 -- 172.21.15.39:0/55222139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85880094d0 con 0x7f85a00c90d0 2024-01-24T17:45:22.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.149+0000 7f859fe00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85a00c90d0 0x7f85a015d160 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f859000e870 tx=0x7f859000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:22.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.149+0000 7f8597e00640 1 -- 172.21.15.39:0/55222139 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859000eec0 con 0x7f85a00c90d0 2024-01-24T17:45:22.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.149+0000 7f8597e00640 1 -- 172.21.15.39:0/55222139 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859000c940 con 0x7f85a00c90d0 2024-01-24T17:45:22.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.149+0000 7f8597e00640 1 -- 172.21.15.39:0/55222139 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859000ccd0 con 0x7f85a00c90d0 2024-01-24T17:45:22.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.149+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a00c2540 con 0x7f85a00c90d0 2024-01-24T17:45:22.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.149+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a00c2b00 con 0x7f85a00c90d0 2024-01-24T17:45:22.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.150+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a00c2cf0 con 0x7f85a00c90d0 2024-01-24T17:45:22.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.151+0000 7f8597e00640 1 -- 172.21.15.39:0/55222139 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8590013470 con 0x7f85a00c90d0 2024-01-24T17:45:22.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.152+0000 7f8597e00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f857806a430 0x7f857806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.152+0000 7f8597e00640 1 -- 172.21.15.39:0/55222139 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8590088b30 con 0x7f85a00c90d0 2024-01-24T17:45:22.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.153+0000 7f859f400640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f857806a430 0x7f857806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.154+0000 7f859f400640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f857806a430 0x7f857806c8f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8588036000 tx=0x7f858803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:22.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.156+0000 7f8597e00640 1 -- 172.21.15.39:0/55222139 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8590054070 con 0x7f85a00c90d0 2024-01-24T17:45:22.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.355+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85a0073ef0 con 0x7f85a00c90d0 2024-01-24T17:45:22.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.358+0000 7f8597e00640 1 -- 172.21.15.39:0/55222139 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f8590021090 con 0x7f85a00c90d0 2024-01-24T17:45:22.359 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:22.359 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.360+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f857806a430 msgr2=0x7f857806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.360+0000 7f85a6a00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f857806a430 0x7f857806c8f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8588036000 tx=0x7f858803b040 comp rx=0 tx=0).stop 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.360+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85a00c90d0 msgr2=0x7f85a015d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.360+0000 7f85a6a00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85a00c90d0 0x7f85a015d160 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f859000e870 tx=0x7f859000ec70 comp rx=0 tx=0).stop 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.361+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 shutdown_connections 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.361+0000 7f85a6a00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f85a00b3680 0x7f85a00b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.361+0000 7f85a6a00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f85a015ca30 0x7f85a015a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.361+0000 7f85a6a00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f857806a430 0x7f857806c8f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.361+0000 7f85a6a00640 1 --2- 172.21.15.39:0/55222139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f85a00c90d0 0x7f85a015d160 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.361+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 >> 172.21.15.39:0/55222139 conn(0x7f85a006a3e0 msgr2=0x7f85a00ab8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.361+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 shutdown_connections 2024-01-24T17:45:22.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.361+0000 7f85a6a00640 1 -- 172.21.15.39:0/55222139 wait complete. 2024-01-24T17:45:22.379 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-24T17:45:22.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:45:22.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.599+0000 7fa1e1e00640 1 Processor -- start 2024-01-24T17:45:22.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.599+0000 7fa1e1e00640 1 -- start start 2024-01-24T17:45:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.601+0000 7fa1e1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc15a950 0x7fa1dc0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.601+0000 7fa1e1e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa1dc0cb660 0x7fa1dc0b1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.601+0000 7fa1e1e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1dc0c9180 con 0x7fa1dc0caaa0 2024-01-24T17:45:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.601+0000 7fa1e1e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1dc0b43d0 con 0x7fa1dc0cb660 2024-01-24T17:45:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.601+0000 7fa1e1e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc0c8f70 con 0x7fa1dc15a950 2024-01-24T17:45:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.601+0000 7fa1d1e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa1dc0caaa0 0x7fa1dc0caf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56138/0 (socket says 172.21.15.39:56138) 2024-01-24T17:45:22.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.601+0000 7fa1d1e00640 1 -- 172.21.15.39:0/3292475665 learned_addr learned my addr 172.21.15.39:0/3292475665 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1daa00640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc15a950 0x7fa1dc0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1db400640 1 --2- 172.21.15.39:0/3292475665 >> v2:172.21.15.100:3300/0 conn(0x7fa1dc0cb660 0x7fa1dc0b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050863791 0 0) 0x7fa1dc0c9180 con 0x7fa1dc0caaa0 2024-01-24T17:45:22.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1bc003520 con 0x7fa1dc0caaa0 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2444695973 0 0) 0x7fa1bc003520 con 0x7fa1dc0caaa0 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc15a950 msgr2=0x7fa1dc0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc15a950 0x7fa1dc0ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 >> v2:172.21.15.100:3300/0 conn(0x7fa1dc0cb660 msgr2=0x7fa1dc0b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 --2- 172.21.15.39:0/3292475665 >> v2:172.21.15.100:3300/0 conn(0x7fa1dc0cb660 0x7fa1dc0b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc0c9460 con 0x7fa1dc0caaa0 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fa1c4001670 con 0x7fa1dc0caaa0 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 >> v1:172.21.15.39:6789/0 conn(0x7fa1dc0caaa0 legacy=0x7fa1dc0caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc15a950 0x7fa1bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc004d00 0x7fa1bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.602+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc0b43d0 con 0x7fa1bc003970 2024-01-24T17:45:22.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc0c8f70 con 0x7fa1bc004d00 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc00b980 con 0x7fa1dc15a950 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1daa00640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1d1e00640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc15a950 0x7fa1bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1db400640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc004d00 0x7fa1bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1daa00640 1 -- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc15a950 msgr2=0x7fa1bc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1daa00640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc15a950 0x7fa1bc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1daa00640 1 -- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc004d00 msgr2=0x7fa1bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1daa00640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1bc004d00 0x7fa1bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.603+0000 7fa1daa00640 1 -- 172.21.15.39:0/3292475665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc003520 con 0x7fa1bc003970 2024-01-24T17:45:22.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.604+0000 7fa1daa00640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1bc003d90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009ac0 tx=0x7fa1bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd5f732fd5b6e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:22.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc02c6b0 con 0x7fa1bc003970 2024-01-24T17:45:22.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc02c810 con 0x7fa1bc003970 2024-01-24T17:45:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1da000640 1 -- 172.21.15.39:0/3292475665 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc02cba0 con 0x7fa1bc003970 2024-01-24T17:45:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3292475665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 msgr2=0x7fa1bc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3292475665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1bc003d90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009ac0 tx=0x7fa1bc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1db400640 1 -- 172.21.15.39:0/3292475665 reap_dead start 2024-01-24T17:45:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3292475665 shutdown_connections 2024-01-24T17:45:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3292475665 >> 172.21.15.39:0/3292475665 conn(0x7fa1dc06a3e0 msgr2=0x7fa1dc06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3292475665 shutdown_connections 2024-01-24T17:45:22.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.605+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3292475665 wait complete. 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.607+0000 7fa1e1e00640 1 Processor -- start 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.607+0000 7fa1e1e00640 1 -- start start 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.608+0000 7fa1e1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1dc1efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.608+0000 7fa1e1e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1dc153a90 0x7fa1dc153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.608+0000 7fa1e1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc1504a0 0x7fa1dc150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.608+0000 7fa1e1e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc1502a0 con 0x7fa1bc003970 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.608+0000 7fa1e1e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc155410 con 0x7fa1dc153a90 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.608+0000 7fa1e1e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc154800 con 0x7fa1dc1504a0 2024-01-24T17:45:22.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1daa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1dc1efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1daa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1dc1efdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51478/0 (socket says 172.21.15.39:51478) 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1daa00640 1 -- 172.21.15.39:0/3541651503 learned_addr learned my addr 172.21.15.39:0/3541651503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1db400640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc1504a0 0x7fa1dc150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1daa00640 1 -- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc1504a0 msgr2=0x7fa1dc150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1daa00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc1504a0 0x7fa1dc150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1d1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1dc153a90 0x7fa1dc153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1daa00640 1 -- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1dc153a90 msgr2=0x7fa1dc153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1daa00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1dc153a90 0x7fa1dc153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.609+0000 7fa1daa00640 1 -- 172.21.15.39:0/3541651503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc154ae0 con 0x7fa1bc003970 2024-01-24T17:45:22.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.610+0000 7fa1daa00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1dc1efdb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc037040 tx=0x7fa1cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:22.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.610+0000 7fa1d3e00640 1 -- 172.21.15.39:0/3541651503 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc003810 con 0x7fa1bc003970 2024-01-24T17:45:22.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.610+0000 7fa1d3e00640 1 -- 172.21.15.39:0/3541651503 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc003970 con 0x7fa1bc003970 2024-01-24T17:45:22.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.610+0000 7fa1d3e00640 1 -- 172.21.15.39:0/3541651503 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc003ce0 con 0x7fa1bc003970 2024-01-24T17:45:22.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.610+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1dc154dc0 con 0x7fa1bc003970 2024-01-24T17:45:22.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.610+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1dc154450 con 0x7fa1bc003970 2024-01-24T17:45:22.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.611+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1dc073ef0 con 0x7fa1bc003970 2024-01-24T17:45:22.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.612+0000 7fa1d3e00640 1 -- 172.21.15.39:0/3541651503 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa1cc034440 con 0x7fa1bc003970 2024-01-24T17:45:22.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.613+0000 7fa1d3e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1b406a430 0x7fa1b406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:22.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.613+0000 7fa1d3e00640 1 -- 172.21.15.39:0/3541651503 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1cc0aa840 con 0x7fa1bc003970 2024-01-24T17:45:22.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.614+0000 7fa1d1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1b406a430 0x7fa1b406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:22.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.615+0000 7fa1d1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1b406a430 0x7fa1b406c8f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc159210 tx=0x7fa1c4001430 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:22.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.617+0000 7fa1d3e00640 1 -- 172.21.15.39:0/3541651503 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa1cc075ee0 con 0x7fa1bc003970 2024-01-24T17:45:22.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.806+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1dc0c9a20 con 0x7fa1b406a430 2024-01-24T17:45:22.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.830+0000 7fa1d3e00640 1 -- 172.21.15.39:0/3541651503 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276345 (secure 0 0 0) 0x7fa1c4005e60 con 0x7fa1b406a430 2024-01-24T17:45:22.834 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:22.836 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1b406a430 msgr2=0x7fa1b406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1b406a430 0x7fa1b406c8f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc159210 tx=0x7fa1c4001430 comp rx=0 tx=0).stop 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 msgr2=0x7fa1dc1efdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1dc1efdb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc037040 tx=0x7fa1cc004730 comp rx=0 tx=0).stop 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 shutdown_connections 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1dc1504a0 0x7fa1dc150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1dc153a90 0x7fa1dc153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1b406a430 0x7fa1b406c8f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 --2- 172.21.15.39:0/3541651503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1bc003970 0x7fa1dc1efdb0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 >> 172.21.15.39:0/3541651503 conn(0x7fa1dc06a3e0 msgr2=0x7fa1dc0c6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.836+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 shutdown_connections 2024-01-24T17:45:22.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:22.837+0000 7fa1e1e00640 1 -- 172.21.15.39:0/3541651503 wait complete. 2024-01-24T17:45:22.860 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-01-24T17:45:22.293556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243380,"kb_used_data":6720,"kb_used_omap":13,"kb_used_meta":236530,"kb_avail":754731340,"statfs":{"total":773094113280,"available":772844892160,"internally_reserved":0,"allocated":6881280,"data_stored":3557554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":242206808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.426115"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883136+0000","last_change":"2024-01-24T17:45:16.883136+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883136+0000","last_undegraded":"2024-01-24T17:45:16.883136+0000","last_fullsized":"2024-01-24T17:45:16.883136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882151+0000","last_change":"2024-01-24T17:45:16.882151+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882151+0000","last_undegraded":"2024-01-24T17:45:16.882151+0000","last_fullsized":"2024-01-24T17:45:16.882151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:49:04.288996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879725+0000","last_change":"2024-01-24T17:45:16.879725+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879725+0000","last_undegraded":"2024-01-24T17:45:16.879725+0000","last_fullsized":"2024-01-24T17:45:16.879725+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:05:02.184752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883414+0000","last_change":"2024-01-24T17:45:16.883414+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883414+0000","last_undegraded":"2024-01-24T17:45:16.883414+0000","last_fullsized":"2024-01-24T17:45:16.883414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:41:31.769012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879587+0000","last_change":"2024-01-24T17:45:16.879587+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879587+0000","last_undegraded":"2024-01-24T17:45:16.879587+0000","last_fullsized":"2024-01-24T17:45:16.879587+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:16.899578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878094+0000","last_change":"2024-01-24T17:45:16.878094+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878094+0000","last_undegraded":"2024-01-24T17:45:16.878094+0000","last_fullsized":"2024-01-24T17:45:16.878094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:26.416880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871981+0000","last_change":"2024-01-24T17:45:16.871981+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871981+0000","last_undegraded":"2024-01-24T17:45:16.871981+0000","last_fullsized":"2024-01-24T17:45:16.871981+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:34:38.987748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879299+0000","last_change":"2024-01-24T17:45:16.879299+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879299+0000","last_undegraded":"2024-01-24T17:45:16.879299+0000","last_fullsized":"2024-01-24T17:45:16.879299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871869+0000","last_change":"2024-01-24T17:45:16.871869+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871869+0000","last_undegraded":"2024-01-24T17:45:16.871869+0000","last_fullsized":"2024-01-24T17:45:16.871869+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:30:30.289664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.876053+0000","last_change":"2024-01-24T17:45:16.876053+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.876053+0000","last_undegraded":"2024-01-24T17:45:16.876053+0000","last_fullsized":"2024-01-24T17:45:16.876053+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:12.547194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882928+0000","last_change":"2024-01-24T17:45:16.882928+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882928+0000","last_undegraded":"2024-01-24T17:45:16.882928+0000","last_fullsized":"2024-01-24T17:45:16.882928+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:22.676649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.886168+0000","last_change":"2024-01-24T17:45:16.886168+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.886168+0000","last_undegraded":"2024-01-24T17:45:16.886168+0000","last_fullsized":"2024-01-24T17:45:16.886168+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874285+0000","last_change":"2024-01-24T17:45:16.874285+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874285+0000","last_undegraded":"2024-01-24T17:45:16.874285+0000","last_fullsized":"2024-01-24T17:45:16.874285+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:44.801426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882687+0000","last_change":"2024-01-24T17:45:16.882687+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882687+0000","last_undegraded":"2024-01-24T17:45:16.882687+0000","last_fullsized":"2024-01-24T17:45:16.882687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:00.154641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882836+0000","last_change":"2024-01-24T17:45:16.882836+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882836+0000","last_undegraded":"2024-01-24T17:45:16.882836+0000","last_fullsized":"2024-01-24T17:45:16.882836+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:32.202115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882612+0000","last_change":"2024-01-24T17:45:16.882612+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882612+0000","last_undegraded":"2024-01-24T17:45:16.882612+0000","last_fullsized":"2024-01-24T17:45:16.882612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:36.108099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.886030+0000","last_change":"2024-01-24T17:45:16.886030+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.886030+0000","last_undegraded":"2024-01-24T17:45:16.886030+0000","last_fullsized":"2024-01-24T17:45:16.886030+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874899+0000","last_change":"2024-01-24T17:45:16.874899+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874899+0000","last_undegraded":"2024-01-24T17:45:16.874899+0000","last_fullsized":"2024-01-24T17:45:16.874899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:03.155346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878536+0000","last_change":"2024-01-24T17:45:16.878536+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878536+0000","last_undegraded":"2024-01-24T17:45:16.878536+0000","last_fullsized":"2024-01-24T17:45:16.878536+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874615+0000","last_change":"2024-01-24T17:45:16.874615+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874615+0000","last_undegraded":"2024-01-24T17:45:16.874615+0000","last_fullsized":"2024-01-24T17:45:16.874615+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:23:00.228501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883038+0000","last_change":"2024-01-24T17:45:16.883038+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883038+0000","last_undegraded":"2024-01-24T17:45:16.883038+0000","last_fullsized":"2024-01-24T17:45:16.883038+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874831+0000","last_change":"2024-01-24T17:45:16.874831+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874831+0000","last_undegraded":"2024-01-24T17:45:16.874831+0000","last_fullsized":"2024-01-24T17:45:16.874831+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:03.318893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871818+0000","last_change":"2024-01-24T17:45:16.871818+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871818+0000","last_undegraded":"2024-01-24T17:45:16.871818+0000","last_fullsized":"2024-01-24T17:45:16.871818+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:21:59.679470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.884892+0000","last_change":"2024-01-24T17:45:16.884892+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.884892+0000","last_undegraded":"2024-01-24T17:45:16.884892+0000","last_fullsized":"2024-01-24T17:45:16.884892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878682+0000","last_change":"2024-01-24T17:45:16.878682+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878682+0000","last_undegraded":"2024-01-24T17:45:16.878682+0000","last_fullsized":"2024-01-24T17:45:16.878682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:17.910448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871862+0000","last_change":"2024-01-24T17:45:16.871862+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871862+0000","last_undegraded":"2024-01-24T17:45:16.871862+0000","last_fullsized":"2024-01-24T17:45:16.871862+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:35:45.998203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878591+0000","last_change":"2024-01-24T17:45:16.878591+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878591+0000","last_undegraded":"2024-01-24T17:45:16.878591+0000","last_fullsized":"2024-01-24T17:45:16.878591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.884282+0000","last_change":"2024-01-24T17:45:16.884282+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.884282+0000","last_undegraded":"2024-01-24T17:45:16.884282+0000","last_fullsized":"2024-01-24T17:45:16.884282+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878068+0000","last_change":"2024-01-24T17:45:16.878068+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878068+0000","last_undegraded":"2024-01-24T17:45:16.878068+0000","last_fullsized":"2024-01-24T17:45:16.878068+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:09:47.749409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878139+0000","last_change":"2024-01-24T17:45:16.878139+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878139+0000","last_undegraded":"2024-01-24T17:45:16.878139+0000","last_fullsized":"2024-01-24T17:45:16.878139+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:30.797138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874263+0000","last_change":"2024-01-24T17:45:16.874263+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874263+0000","last_undegraded":"2024-01-24T17:45:16.874263+0000","last_fullsized":"2024-01-24T17:45:16.874263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:12.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.885971+0000","last_change":"2024-01-24T17:45:16.885971+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.885971+0000","last_undegraded":"2024-01-24T17:45:16.885971+0000","last_fullsized":"2024-01-24T17:45:16.885971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":199,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-24T17:45:14.842315+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:45:14.842315+0000","last_peered":"2024-01-24T17:45:14.842315+0000","last_clean":"2024-01-24T17:45:14.842315+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:45:14.842315+0000","last_undegraded":"2024-01-24T17:45:14.842315+0000","last_fullsized":"2024-01-24T17:45:14.842315+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.05,"15min":1.0249999999999999},"min":{"1min":0.80300000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.23},{"interface":"front","average":{"1min":1.127,"5min":1.048,"15min":1.0329999999999999},"min":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.599,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.4510000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0860000000000001,"15min":1.1000000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1859999999999999,"15min":1.1919999999999999},"min":{"1min":0.81499999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.649,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1000000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.171,"15min":1.1579999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.764,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.119,"5min":1.181,"15min":1.2190000000000001},"min":{"1min":0.745,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.484,"5min":2.484,"15min":2.484},"last":0.96699999999999997}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8779999999999999,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.282,"15min":1.3100000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.6269999999999998,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.1739999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.296,"15min":1.3200000000000001},"min":{"1min":0.84699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.1190000000000002,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.331},{"interface":"front","average":{"1min":1.383,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5449999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.589}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.319,"15min":1.3520000000000001},"min":{"1min":0.70799999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.264},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3979999999999999,"15min":1.4350000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.966,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.5489999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4990000000000001,"15min":1.5349999999999999},"min":{"1min":0.72599999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.5720000000000001,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.635},{"interface":"front","average":{"1min":1.399,"5min":1.472,"15min":1.488},"min":{"1min":0.86099999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.2450000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.679}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":720896,"data_stored":301465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.034,"15min":1.024},"min":{"1min":0.70699999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6919999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.583},{"interface":"front","average":{"1min":1.141,"5min":1.044,"15min":1.0189999999999999},"min":{"1min":0.749,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.2050000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.022,"15min":0.99199999999999999},"min":{"1min":0.70199999999999996,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4890000000000001,"5min":1.73,"15min":1.73},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1180000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.722,"5min":1.972,"15min":1.972},"last":1.1379999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.79300000000000004,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5369999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.169,"15min":1.167},"min":{"1min":0.90700000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.514,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.6559999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.242,"15min":1.254},"min":{"1min":0.83799999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.542,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.387},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2310000000000001,"15min":1.2529999999999999},"min":{"1min":0.84199999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5980000000000001,"5min":2.46,"15min":2.46},"last":1.758}]},{"osd":4,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.2569999999999999,"15min":1.2370000000000001},"min":{"1min":0.76000000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.661,"5min":1.988,"15min":1.988},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2989999999999999,"15min":1.3029999999999999},"min":{"1min":0.73899999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.8420000000000001,"5min":1.849,"15min":1.849},"last":1.784}]},{"osd":5,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3080000000000001,"15min":1.296},"min":{"1min":0.97999999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.645,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.8140000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.89800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9319999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":2.0030000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.355,"15min":1.367},"min":{"1min":0.85399999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.978,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.694},{"interface":"front","average":{"1min":1.45,"5min":1.454,"15min":1.4630000000000001},"min":{"1min":1.008,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.899,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.8859999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1302528,"data_stored":886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0669999999999999,"15min":1.099},"min":{"1min":0.67500000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.47,"5min":1.508,"15min":1.508},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.998,"5min":1.0289999999999999,"15min":1.0309999999999999},"min":{"1min":0.67900000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5,"5min":1.889,"15min":1.889},"last":1.0820000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.077,"15min":1.0740000000000001},"min":{"1min":0.72899999999999998,"5min":0.622,"15min":0.622},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.22,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.552,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.6040000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2230000000000001,"15min":1.2290000000000001},"min":{"1min":0.80600000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.28,"15min":1.3120000000000001},"min":{"1min":0.79800000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.9870000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.97299999999999998}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2909999999999999,"15min":1.341},"min":{"1min":0.77800000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6020000000000001,"5min":3.036,"15min":3.036},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3320000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.641,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.4550000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4339999999999999,"15min":1.4530000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.915,"5min":2.069,"15min":2.069},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.329,"15min":1.3600000000000001},"min":{"1min":0.77600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5780000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.2010000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.339,"15min":1.3560000000000001},"min":{"1min":0.91100000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.724,"5min":2.504,"15min":2.504},"last":1.645},{"interface":"front","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":0.88600000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.044,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.8839999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.458,"15min":1.4930000000000001},"min":{"1min":0.63800000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.6719999999999999,"5min":3.161,"15min":3.161},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.5900000000000001,"15min":1.6339999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.1389999999999998,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.7569999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.048,"15min":1.038},"min":{"1min":0.73499999999999999,"5min":0.621,"15min":0.621},"max":{"1min":1.673,"5min":1.698,"15min":1.698},"last":1.038},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.093,"15min":1.0960000000000001},"min":{"1min":0.71199999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5529999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.071}]},{"osd":1,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.0800000000000001,"15min":1.052},"min":{"1min":0.78500000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5049999999999999,"5min":1.633,"15min":1.633},"last":1.25},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1899999999999999,"15min":1.2170000000000001},"min":{"1min":0.67800000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.4139999999999999,"5min":1.849,"15min":1.849},"last":1.2010000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.206,"15min":1.212},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.304,"15min":1.3340000000000001},"min":{"1min":0.71199999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.724,"5min":1.986,"15min":1.986},"last":1.1200000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1850000000000001,"15min":1.1910000000000001},"min":{"1min":0.81000000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.456,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.319,"5min":1.343,"15min":1.351},"min":{"1min":0.89400000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.8029999999999999,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":0.95299999999999996}]},{"osd":5,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3480000000000001,"15min":1.373},"min":{"1min":1.014,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.613,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.298,"15min":1.306},"min":{"1min":0.85799999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5600000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":0.98799999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4550000000000001,"15min":1.4730000000000001},"min":{"1min":0.93799999999999994,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.3159999999999998,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.4610000000000001,"15min":1.4790000000000001},"min":{"1min":0.86899999999999999,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.91600000000000004}]},{"osd":7,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4330000000000001,"15min":1.4259999999999999},"min":{"1min":0.92000000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0209999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.6000000000000001,"15min":1.621},"min":{"1min":0.97299999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.2309999999999999,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.411}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.91600000000000004,"15min":0.872},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.792,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95499999999999996,"15min":0.89100000000000001},"min":{"1min":0.64200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5680000000000001,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":1.0980000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.93899999999999995,"15min":0.95099999999999996},"min":{"1min":0.53100000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.819,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0720000000000001,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.663,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.0620000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.105,"15min":1.111},"min":{"1min":0.629,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.571,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.181,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.593,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.69199999999999995}]},{"osd":4,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3200000000000001,"15min":1.292},"min":{"1min":0.90800000000000003,"5min":0.877,"15min":0.877},"max":{"1min":1.8240000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9790000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.262}]},{"osd":5,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4350000000000001,"15min":1.4370000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2189999999999999,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":0.995},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4159999999999999,"15min":1.4239999999999999},"min":{"1min":1.024,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.0579999999999998,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.288}]},{"osd":6,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.532,"15min":1.5029999999999999},"min":{"1min":1.0429999999999999,"5min":0.871,"15min":0.871},"max":{"1min":1.984,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.486},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4730000000000001,"15min":1.4630000000000001},"min":{"1min":0.90200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.1120000000000001,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.7450000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5389999999999999,"15min":1.5169999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.012,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.5249999999999999},"min":{"1min":1.0649999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.1160000000000001,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.444}]}]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.439,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.24,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93600000000000005,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.99299999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0369999999999999,"15min":1.0549999999999999},"min":{"1min":0.51300000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.5509999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.173,"15min":1.2150000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":1.5600000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.98199999999999998,"15min":0.996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6910000000000001,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2050000000000001,"15min":1.218},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7629999999999999,"5min":2.996,"15min":2.996},"last":1.327}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.306,"15min":1.329},"min":{"1min":0.72899999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6950000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4099999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.7,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.29}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3600000000000001,"15min":1.409},"min":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.762,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.379},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2969999999999999,"15min":1.331},"min":{"1min":0.86299999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.625,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.0680000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.369,"15min":1.4019999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9350000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.266},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4399999999999999,"15min":1.4830000000000001},"min":{"1min":0.80200000000000005,"5min":0.753,"15min":0.753},"max":{"1min":2.2530000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.4810000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.526,"15min":1.5860000000000001},"min":{"1min":0.82499999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0209999999999999,"5min":3.343,"15min":3.343},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4630000000000001,"15min":1.512},"min":{"1min":0.755,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9510000000000001,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.177}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1302528,"data_stored":886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.874,"15min":0.85999999999999999},"min":{"1min":0.45500000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4419999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.93000000000000005,"15min":0.95299999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.155,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.95399999999999996,"15min":0.98399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.4790000000000001,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5680000000000001,"5min":1.895,"15min":1.895},"last":2.0270000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.143,"15min":1.1639999999999999},"min":{"1min":0.61299999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4930000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.194,"15min":1.244},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.4019999999999999,"5min":3.149,"15min":3.149},"last":0.67600000000000005}]},{"osd":4,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.353,"15min":1.365},"min":{"1min":0.88700000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.77,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.175},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.296,"15min":1.3240000000000001},"min":{"1min":0.83599999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.698,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.3999999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.323,"15min":1.357},"min":{"1min":0.73499999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.806,"5min":2.577,"15min":2.577},"last":1.075},{"interface":"front","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.9730000000000001,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.4850000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.4019999999999999,"15min":1.4450000000000001},"min":{"1min":0.80000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.653,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.335},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.46,"15min":1.5309999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.921,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.145}]},{"osd":7,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4570000000000001,"15min":1.4750000000000001},"min":{"1min":0.91400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0979999999999999,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.76},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.5049999999999999,"15min":1.5549999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.641,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.5740000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":720896,"data_stored":301465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80000000000000004,"15min":0.81799999999999995},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.242,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.93000000000000005,"15min":0.96999999999999997},"min":{"1min":0.52900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.2410000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.84199999999999997,"15min":0.83399999999999996},"min":{"1min":0.45200000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.997,"5min":1.046,"15min":1.1000000000000001},"min":{"1min":0.46800000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4430000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.359}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.1359999999999999,"15min":1.1890000000000001},"min":{"1min":0.51000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.639,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.105,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.60699999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6040000000000001,"5min":2.407,"15min":2.407},"last":1.202}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4179999999999999},"min":{"1min":0.752,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.056,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.4139999999999999,"15min":1.488},"min":{"1min":0.89000000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.5840000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.173}]},{"osd":5,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.333,"15min":1.397},"min":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.7490000000000001,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.256,"15min":1.264},"min":{"1min":0.85599999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.694,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.2929999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.337,"15min":1.3600000000000001},"min":{"1min":0.93500000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":2.335,"15min":2.335},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.4139999999999999,"15min":1.46},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.877,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.5429999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.462,"15min":1.534},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.694,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.325},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.5369999999999999,"15min":1.605},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.806,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.119}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:45:22.865 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:45:23.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.126+0000 7ff265600640 1 Processor -- start 2024-01-24T17:45:23.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.126+0000 7ff265600640 1 -- start start 2024-01-24T17:45:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff265600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff26015a720 0x7ff2600cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff265600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff2600ce8e0 0x7ff2600ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff265600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2600cf260 con 0x7ff26015ab00 2024-01-24T17:45:23.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff265600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2600ce400 con 0x7ff2600ce8e0 2024-01-24T17:45:23.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff265600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2600ccef0 con 0x7ff26015a720 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff264c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff2600ce8e0 0x7ff2600ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff257e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff26015ab00 0x7ff2600cc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:36738/0 (socket says 172.21.15.39:36738) 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff257e00640 1 -- 172.21.15.39:0/1533878020 learned_addr learned my addr 172.21.15.39:0/1533878020 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.128+0000 7ff264c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff2600ce8e0 0x7ff2600ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47846/0 (socket says 172.21.15.39:47846) 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.129+0000 7ff25fe00640 1 --2- 172.21.15.39:0/1533878020 >> v2:172.21.15.100:3300/0 conn(0x7ff26015a720 0x7ff2600cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.129+0000 7ff264c00640 1 -- 172.21.15.39:0/1533878020 >> v2:172.21.15.100:3300/0 conn(0x7ff26015a720 msgr2=0x7ff2600cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.129+0000 7ff264c00640 1 --2- 172.21.15.39:0/1533878020 >> v2:172.21.15.100:3300/0 conn(0x7ff26015a720 0x7ff2600cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.129+0000 7ff264c00640 1 -- 172.21.15.39:0/1533878020 >> v1:172.21.15.39:6789/0 conn(0x7ff26015ab00 legacy=0x7ff2600cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.129+0000 7ff264c00640 1 -- 172.21.15.39:0/1533878020 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2600ce110 con 0x7ff2600ce8e0 2024-01-24T17:45:23.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.130+0000 7ff264c00640 1 --2- 172.21.15.39:0/1533878020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2600ce8e0 0x7ff2600ced20 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff26006c3e0 tx=0x7ff25802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350e170d8be0112 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:23.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.131+0000 7ff25f400640 1 -- 172.21.15.39:0/1533878020 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25802cbe0 con 0x7ff2600ce8e0 2024-01-24T17:45:23.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.131+0000 7ff25f400640 1 -- 172.21.15.39:0/1533878020 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25802cd40 con 0x7ff2600ce8e0 2024-01-24T17:45:23.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.131+0000 7ff25f400640 1 -- 172.21.15.39:0/1533878020 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258034660 con 0x7ff2600ce8e0 2024-01-24T17:45:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.131+0000 7ff265600640 1 -- 172.21.15.39:0/1533878020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2600ce8e0 msgr2=0x7ff2600ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.131+0000 7ff265600640 1 --2- 172.21.15.39:0/1533878020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2600ce8e0 0x7ff2600ced20 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff26006c3e0 tx=0x7ff25802be30 comp rx=0 tx=0).stop 2024-01-24T17:45:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.132+0000 7ff265600640 1 -- 172.21.15.39:0/1533878020 shutdown_connections 2024-01-24T17:45:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.132+0000 7ff265600640 1 --2- 172.21.15.39:0/1533878020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2600ce8e0 0x7ff2600ced20 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.132+0000 7ff265600640 1 --2- 172.21.15.39:0/1533878020 >> v2:172.21.15.100:3300/0 conn(0x7ff26015a720 0x7ff2600cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.132+0000 7ff265600640 1 -- 172.21.15.39:0/1533878020 >> 172.21.15.39:0/1533878020 conn(0x7ff26006a090 msgr2=0x7ff26006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:23.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.132+0000 7ff265600640 1 -- 172.21.15.39:0/1533878020 shutdown_connections 2024-01-24T17:45:23.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.132+0000 7ff265600640 1 -- 172.21.15.39:0/1533878020 wait complete. 2024-01-24T17:45:23.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.133+0000 7ff265600640 1 Processor -- start 2024-01-24T17:45:23.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.134+0000 7ff265600640 1 -- start start 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff265600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2600ce8e0 0x7ff2601f1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff265600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff26015a720 0x7ff2601e41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff265600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff26015ab00 0x7ff2600c5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff265600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2601e4c70 con 0x7ff26015a720 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff265600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2601e4f50 con 0x7ff26015ab00 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff265600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2601e5230 con 0x7ff2600ce8e0 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff257e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff26015a720 0x7ff2601e41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff257e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff26015a720 0x7ff2601e41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47848/0 (socket says 172.21.15.39:47848) 2024-01-24T17:45:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff257e00640 1 -- 172.21.15.39:0/4112378959 learned_addr learned my addr 172.21.15.39:0/4112378959 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff257e00640 1 -- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2600ce8e0 msgr2=0x7ff2601f1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff257e00640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2600ce8e0 0x7ff2601f1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff257e00640 1 -- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff26015ab00 msgr2=0x7ff2600c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff257e00640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff26015ab00 0x7ff2600c5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.135+0000 7ff257e00640 1 -- 172.21.15.39:0/4112378959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff258009a70 con 0x7ff26015a720 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.136+0000 7ff257e00640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff26015a720 0x7ff2601e41b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff24800c480 tx=0x7ff24800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.136+0000 7ff25cc00640 1 -- 172.21.15.39:0/4112378959 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff248005ce0 con 0x7ff26015a720 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.136+0000 7ff25cc00640 1 -- 172.21.15.39:0/4112378959 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff248005e40 con 0x7ff26015a720 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.136+0000 7ff25cc00640 1 -- 172.21.15.39:0/4112378959 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2480072f0 con 0x7ff26015a720 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.136+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2601e5510 con 0x7ff26015a720 2024-01-24T17:45:23.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.136+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2601f1e90 con 0x7ff26015a720 2024-01-24T17:45:23.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.137+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2601e5a30 con 0x7ff26015a720 2024-01-24T17:45:23.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.139+0000 7ff25cc00640 1 -- 172.21.15.39:0/4112378959 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff248007510 con 0x7ff26015a720 2024-01-24T17:45:23.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.140+0000 7ff25cc00640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff23806a430 0x7ff23806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.140+0000 7ff25cc00640 1 -- 172.21.15.39:0/4112378959 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff248087dd0 con 0x7ff26015a720 2024-01-24T17:45:23.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.141+0000 7ff25fe00640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff23806a430 0x7ff23806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.142+0000 7ff25fe00640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff23806a430 0x7ff23806c8f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff250006a90 tx=0x7ff250006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:23.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.145+0000 7ff25cc00640 1 -- 172.21.15.39:0/4112378959 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff2601e5a30 con 0x7ff26015a720 2024-01-24T17:45:23.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.335+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff26015b690 con 0x7ff23806a430 2024-01-24T17:45:23.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.356+0000 7ff25cc00640 1 -- 172.21.15.39:0/4112378959 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276345 (secure 0 0 0) 0x7ff25000e860 con 0x7ff23806a430 2024-01-24T17:45:23.363 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:23.365 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:45:23.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff23806a430 msgr2=0x7ff23806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff23806a430 0x7ff23806c8f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff250006a90 tx=0x7ff250006a20 comp rx=0 tx=0).stop 2024-01-24T17:45:23.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff26015a720 msgr2=0x7ff2601e41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff26015a720 0x7ff2601e41b0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff24800c480 tx=0x7ff24800c940 comp rx=0 tx=0).stop 2024-01-24T17:45:23.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 shutdown_connections 2024-01-24T17:45:23.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff26015ab00 0x7ff2600c5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff23806a430 0x7ff23806c8f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff26015a720 0x7ff2601e41b0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 --2- 172.21.15.39:0/4112378959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff2600ce8e0 0x7ff2601f1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.362+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 >> 172.21.15.39:0/4112378959 conn(0x7ff26006a090 msgr2=0x7ff2601598e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:23.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.363+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 shutdown_connections 2024-01-24T17:45:23.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.363+0000 7ff265600640 1 -- 172.21.15.39:0/4112378959 wait complete. 2024-01-24T17:45:23.383 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-01-24T17:45:22.293556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243380,"kb_used_data":6720,"kb_used_omap":13,"kb_used_meta":236530,"kb_avail":754731340,"statfs":{"total":773094113280,"available":772844892160,"internally_reserved":0,"allocated":6881280,"data_stored":3557554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":242206808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.426115"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883136+0000","last_change":"2024-01-24T17:45:16.883136+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883136+0000","last_undegraded":"2024-01-24T17:45:16.883136+0000","last_fullsized":"2024-01-24T17:45:16.883136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882151+0000","last_change":"2024-01-24T17:45:16.882151+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882151+0000","last_undegraded":"2024-01-24T17:45:16.882151+0000","last_fullsized":"2024-01-24T17:45:16.882151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:49:04.288996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879725+0000","last_change":"2024-01-24T17:45:16.879725+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879725+0000","last_undegraded":"2024-01-24T17:45:16.879725+0000","last_fullsized":"2024-01-24T17:45:16.879725+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:05:02.184752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883414+0000","last_change":"2024-01-24T17:45:16.883414+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883414+0000","last_undegraded":"2024-01-24T17:45:16.883414+0000","last_fullsized":"2024-01-24T17:45:16.883414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:41:31.769012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879587+0000","last_change":"2024-01-24T17:45:16.879587+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879587+0000","last_undegraded":"2024-01-24T17:45:16.879587+0000","last_fullsized":"2024-01-24T17:45:16.879587+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:16.899578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878094+0000","last_change":"2024-01-24T17:45:16.878094+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878094+0000","last_undegraded":"2024-01-24T17:45:16.878094+0000","last_fullsized":"2024-01-24T17:45:16.878094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:26.416880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871981+0000","last_change":"2024-01-24T17:45:16.871981+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871981+0000","last_undegraded":"2024-01-24T17:45:16.871981+0000","last_fullsized":"2024-01-24T17:45:16.871981+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:34:38.987748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879299+0000","last_change":"2024-01-24T17:45:16.879299+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879299+0000","last_undegraded":"2024-01-24T17:45:16.879299+0000","last_fullsized":"2024-01-24T17:45:16.879299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871869+0000","last_change":"2024-01-24T17:45:16.871869+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871869+0000","last_undegraded":"2024-01-24T17:45:16.871869+0000","last_fullsized":"2024-01-24T17:45:16.871869+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:30:30.289664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.876053+0000","last_change":"2024-01-24T17:45:16.876053+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.876053+0000","last_undegraded":"2024-01-24T17:45:16.876053+0000","last_fullsized":"2024-01-24T17:45:16.876053+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:12.547194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882928+0000","last_change":"2024-01-24T17:45:16.882928+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882928+0000","last_undegraded":"2024-01-24T17:45:16.882928+0000","last_fullsized":"2024-01-24T17:45:16.882928+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:22.676649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.886168+0000","last_change":"2024-01-24T17:45:16.886168+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.886168+0000","last_undegraded":"2024-01-24T17:45:16.886168+0000","last_fullsized":"2024-01-24T17:45:16.886168+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874285+0000","last_change":"2024-01-24T17:45:16.874285+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874285+0000","last_undegraded":"2024-01-24T17:45:16.874285+0000","last_fullsized":"2024-01-24T17:45:16.874285+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:44.801426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882687+0000","last_change":"2024-01-24T17:45:16.882687+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882687+0000","last_undegraded":"2024-01-24T17:45:16.882687+0000","last_fullsized":"2024-01-24T17:45:16.882687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:00.154641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882836+0000","last_change":"2024-01-24T17:45:16.882836+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882836+0000","last_undegraded":"2024-01-24T17:45:16.882836+0000","last_fullsized":"2024-01-24T17:45:16.882836+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:32.202115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882612+0000","last_change":"2024-01-24T17:45:16.882612+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882612+0000","last_undegraded":"2024-01-24T17:45:16.882612+0000","last_fullsized":"2024-01-24T17:45:16.882612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:36.108099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.886030+0000","last_change":"2024-01-24T17:45:16.886030+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.886030+0000","last_undegraded":"2024-01-24T17:45:16.886030+0000","last_fullsized":"2024-01-24T17:45:16.886030+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874899+0000","last_change":"2024-01-24T17:45:16.874899+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874899+0000","last_undegraded":"2024-01-24T17:45:16.874899+0000","last_fullsized":"2024-01-24T17:45:16.874899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:03.155346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878536+0000","last_change":"2024-01-24T17:45:16.878536+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878536+0000","last_undegraded":"2024-01-24T17:45:16.878536+0000","last_fullsized":"2024-01-24T17:45:16.878536+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874615+0000","last_change":"2024-01-24T17:45:16.874615+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874615+0000","last_undegraded":"2024-01-24T17:45:16.874615+0000","last_fullsized":"2024-01-24T17:45:16.874615+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:23:00.228501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883038+0000","last_change":"2024-01-24T17:45:16.883038+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883038+0000","last_undegraded":"2024-01-24T17:45:16.883038+0000","last_fullsized":"2024-01-24T17:45:16.883038+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874831+0000","last_change":"2024-01-24T17:45:16.874831+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874831+0000","last_undegraded":"2024-01-24T17:45:16.874831+0000","last_fullsized":"2024-01-24T17:45:16.874831+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:03.318893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871818+0000","last_change":"2024-01-24T17:45:16.871818+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871818+0000","last_undegraded":"2024-01-24T17:45:16.871818+0000","last_fullsized":"2024-01-24T17:45:16.871818+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:21:59.679470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.884892+0000","last_change":"2024-01-24T17:45:16.884892+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.884892+0000","last_undegraded":"2024-01-24T17:45:16.884892+0000","last_fullsized":"2024-01-24T17:45:16.884892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878682+0000","last_change":"2024-01-24T17:45:16.878682+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878682+0000","last_undegraded":"2024-01-24T17:45:16.878682+0000","last_fullsized":"2024-01-24T17:45:16.878682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:17.910448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871862+0000","last_change":"2024-01-24T17:45:16.871862+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871862+0000","last_undegraded":"2024-01-24T17:45:16.871862+0000","last_fullsized":"2024-01-24T17:45:16.871862+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:35:45.998203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878591+0000","last_change":"2024-01-24T17:45:16.878591+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878591+0000","last_undegraded":"2024-01-24T17:45:16.878591+0000","last_fullsized":"2024-01-24T17:45:16.878591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.884282+0000","last_change":"2024-01-24T17:45:16.884282+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.884282+0000","last_undegraded":"2024-01-24T17:45:16.884282+0000","last_fullsized":"2024-01-24T17:45:16.884282+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878068+0000","last_change":"2024-01-24T17:45:16.878068+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878068+0000","last_undegraded":"2024-01-24T17:45:16.878068+0000","last_fullsized":"2024-01-24T17:45:16.878068+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:09:47.749409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878139+0000","last_change":"2024-01-24T17:45:16.878139+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878139+0000","last_undegraded":"2024-01-24T17:45:16.878139+0000","last_fullsized":"2024-01-24T17:45:16.878139+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:30.797138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874263+0000","last_change":"2024-01-24T17:45:16.874263+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874263+0000","last_undegraded":"2024-01-24T17:45:16.874263+0000","last_fullsized":"2024-01-24T17:45:16.874263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:12.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.885971+0000","last_change":"2024-01-24T17:45:16.885971+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.885971+0000","last_undegraded":"2024-01-24T17:45:16.885971+0000","last_fullsized":"2024-01-24T17:45:16.885971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":199,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-24T17:45:14.842315+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:45:14.842315+0000","last_peered":"2024-01-24T17:45:14.842315+0000","last_clean":"2024-01-24T17:45:14.842315+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:45:14.842315+0000","last_undegraded":"2024-01-24T17:45:14.842315+0000","last_fullsized":"2024-01-24T17:45:14.842315+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.05,"15min":1.0249999999999999},"min":{"1min":0.80300000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.23},{"interface":"front","average":{"1min":1.127,"5min":1.048,"15min":1.0329999999999999},"min":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.599,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.4510000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0860000000000001,"15min":1.1000000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1859999999999999,"15min":1.1919999999999999},"min":{"1min":0.81499999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.649,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1000000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.171,"15min":1.1579999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.764,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.119,"5min":1.181,"15min":1.2190000000000001},"min":{"1min":0.745,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.484,"5min":2.484,"15min":2.484},"last":0.96699999999999997}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8779999999999999,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.282,"15min":1.3100000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.6269999999999998,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.1739999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.296,"15min":1.3200000000000001},"min":{"1min":0.84699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.1190000000000002,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.331},{"interface":"front","average":{"1min":1.383,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5449999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.589}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.319,"15min":1.3520000000000001},"min":{"1min":0.70799999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.264},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3979999999999999,"15min":1.4350000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.966,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.5489999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4990000000000001,"15min":1.5349999999999999},"min":{"1min":0.72599999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.5720000000000001,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.635},{"interface":"front","average":{"1min":1.399,"5min":1.472,"15min":1.488},"min":{"1min":0.86099999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.2450000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.679}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":720896,"data_stored":301465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.034,"15min":1.024},"min":{"1min":0.70699999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6919999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.583},{"interface":"front","average":{"1min":1.141,"5min":1.044,"15min":1.0189999999999999},"min":{"1min":0.749,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.2050000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.022,"15min":0.99199999999999999},"min":{"1min":0.70199999999999996,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4890000000000001,"5min":1.73,"15min":1.73},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1180000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.722,"5min":1.972,"15min":1.972},"last":1.1379999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.79300000000000004,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5369999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.169,"15min":1.167},"min":{"1min":0.90700000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.514,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.6559999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.242,"15min":1.254},"min":{"1min":0.83799999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.542,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.387},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2310000000000001,"15min":1.2529999999999999},"min":{"1min":0.84199999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5980000000000001,"5min":2.46,"15min":2.46},"last":1.758}]},{"osd":4,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.2569999999999999,"15min":1.2370000000000001},"min":{"1min":0.76000000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.661,"5min":1.988,"15min":1.988},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2989999999999999,"15min":1.3029999999999999},"min":{"1min":0.73899999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.8420000000000001,"5min":1.849,"15min":1.849},"last":1.784}]},{"osd":5,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3080000000000001,"15min":1.296},"min":{"1min":0.97999999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.645,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.8140000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.89800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9319999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":2.0030000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.355,"15min":1.367},"min":{"1min":0.85399999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.978,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.694},{"interface":"front","average":{"1min":1.45,"5min":1.454,"15min":1.4630000000000001},"min":{"1min":1.008,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.899,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.8859999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1302528,"data_stored":886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0669999999999999,"15min":1.099},"min":{"1min":0.67500000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.47,"5min":1.508,"15min":1.508},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.998,"5min":1.0289999999999999,"15min":1.0309999999999999},"min":{"1min":0.67900000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5,"5min":1.889,"15min":1.889},"last":1.0820000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.077,"15min":1.0740000000000001},"min":{"1min":0.72899999999999998,"5min":0.622,"15min":0.622},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.22,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.552,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.6040000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2230000000000001,"15min":1.2290000000000001},"min":{"1min":0.80600000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.28,"15min":1.3120000000000001},"min":{"1min":0.79800000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.9870000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.97299999999999998}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2909999999999999,"15min":1.341},"min":{"1min":0.77800000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6020000000000001,"5min":3.036,"15min":3.036},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3320000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.641,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.4550000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4339999999999999,"15min":1.4530000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.915,"5min":2.069,"15min":2.069},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.329,"15min":1.3600000000000001},"min":{"1min":0.77600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5780000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.2010000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.339,"15min":1.3560000000000001},"min":{"1min":0.91100000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.724,"5min":2.504,"15min":2.504},"last":1.645},{"interface":"front","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":0.88600000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.044,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.8839999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.458,"15min":1.4930000000000001},"min":{"1min":0.63800000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.6719999999999999,"5min":3.161,"15min":3.161},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.5900000000000001,"15min":1.6339999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.1389999999999998,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.7569999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.048,"15min":1.038},"min":{"1min":0.73499999999999999,"5min":0.621,"15min":0.621},"max":{"1min":1.673,"5min":1.698,"15min":1.698},"last":1.038},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.093,"15min":1.0960000000000001},"min":{"1min":0.71199999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5529999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.071}]},{"osd":1,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.0800000000000001,"15min":1.052},"min":{"1min":0.78500000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5049999999999999,"5min":1.633,"15min":1.633},"last":1.25},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1899999999999999,"15min":1.2170000000000001},"min":{"1min":0.67800000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.4139999999999999,"5min":1.849,"15min":1.849},"last":1.2010000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.206,"15min":1.212},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.304,"15min":1.3340000000000001},"min":{"1min":0.71199999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.724,"5min":1.986,"15min":1.986},"last":1.1200000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1850000000000001,"15min":1.1910000000000001},"min":{"1min":0.81000000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.456,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.319,"5min":1.343,"15min":1.351},"min":{"1min":0.89400000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.8029999999999999,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":0.95299999999999996}]},{"osd":5,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3480000000000001,"15min":1.373},"min":{"1min":1.014,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.613,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.298,"15min":1.306},"min":{"1min":0.85799999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5600000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":0.98799999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4550000000000001,"15min":1.4730000000000001},"min":{"1min":0.93799999999999994,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.3159999999999998,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.4610000000000001,"15min":1.4790000000000001},"min":{"1min":0.86899999999999999,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.91600000000000004}]},{"osd":7,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4330000000000001,"15min":1.4259999999999999},"min":{"1min":0.92000000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0209999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.6000000000000001,"15min":1.621},"min":{"1min":0.97299999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.2309999999999999,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.411}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.91600000000000004,"15min":0.872},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.792,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95499999999999996,"15min":0.89100000000000001},"min":{"1min":0.64200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5680000000000001,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":1.0980000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.93899999999999995,"15min":0.95099999999999996},"min":{"1min":0.53100000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.819,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0720000000000001,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.663,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.0620000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.105,"15min":1.111},"min":{"1min":0.629,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.571,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.181,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.593,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.69199999999999995}]},{"osd":4,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3200000000000001,"15min":1.292},"min":{"1min":0.90800000000000003,"5min":0.877,"15min":0.877},"max":{"1min":1.8240000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9790000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.262}]},{"osd":5,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4350000000000001,"15min":1.4370000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2189999999999999,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":0.995},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4159999999999999,"15min":1.4239999999999999},"min":{"1min":1.024,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.0579999999999998,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.288}]},{"osd":6,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.532,"15min":1.5029999999999999},"min":{"1min":1.0429999999999999,"5min":0.871,"15min":0.871},"max":{"1min":1.984,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.486},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4730000000000001,"15min":1.4630000000000001},"min":{"1min":0.90200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.1120000000000001,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.7450000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5389999999999999,"15min":1.5169999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.012,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.5249999999999999},"min":{"1min":1.0649999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.1160000000000001,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.444}]}]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.439,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.24,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93600000000000005,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.99299999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0369999999999999,"15min":1.0549999999999999},"min":{"1min":0.51300000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.5509999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.173,"15min":1.2150000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":1.5600000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.98199999999999998,"15min":0.996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6910000000000001,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2050000000000001,"15min":1.218},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7629999999999999,"5min":2.996,"15min":2.996},"last":1.327}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.306,"15min":1.329},"min":{"1min":0.72899999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6950000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4099999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.7,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.29}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3600000000000001,"15min":1.409},"min":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.762,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.379},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2969999999999999,"15min":1.331},"min":{"1min":0.86299999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.625,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.0680000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.369,"15min":1.4019999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9350000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.266},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4399999999999999,"15min":1.4830000000000001},"min":{"1min":0.80200000000000005,"5min":0.753,"15min":0.753},"max":{"1min":2.2530000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.4810000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.526,"15min":1.5860000000000001},"min":{"1min":0.82499999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0209999999999999,"5min":3.343,"15min":3.343},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4630000000000001,"15min":1.512},"min":{"1min":0.755,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9510000000000001,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.177}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1302528,"data_stored":886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.874,"15min":0.85999999999999999},"min":{"1min":0.45500000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4419999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.93000000000000005,"15min":0.95299999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.155,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.95399999999999996,"15min":0.98399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.4790000000000001,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5680000000000001,"5min":1.895,"15min":1.895},"last":2.0270000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.143,"15min":1.1639999999999999},"min":{"1min":0.61299999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4930000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.194,"15min":1.244},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.4019999999999999,"5min":3.149,"15min":3.149},"last":0.67600000000000005}]},{"osd":4,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.353,"15min":1.365},"min":{"1min":0.88700000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.77,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.175},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.296,"15min":1.3240000000000001},"min":{"1min":0.83599999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.698,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.3999999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.323,"15min":1.357},"min":{"1min":0.73499999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.806,"5min":2.577,"15min":2.577},"last":1.075},{"interface":"front","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.9730000000000001,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.4850000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.4019999999999999,"15min":1.4450000000000001},"min":{"1min":0.80000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.653,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.335},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.46,"15min":1.5309999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.921,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.145}]},{"osd":7,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4570000000000001,"15min":1.4750000000000001},"min":{"1min":0.91400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0979999999999999,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.76},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.5049999999999999,"15min":1.5549999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.641,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.5740000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":720896,"data_stored":301465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80000000000000004,"15min":0.81799999999999995},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.242,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.93000000000000005,"15min":0.96999999999999997},"min":{"1min":0.52900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.2410000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.84199999999999997,"15min":0.83399999999999996},"min":{"1min":0.45200000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.997,"5min":1.046,"15min":1.1000000000000001},"min":{"1min":0.46800000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4430000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.359}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.1359999999999999,"15min":1.1890000000000001},"min":{"1min":0.51000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.639,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.105,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.60699999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6040000000000001,"5min":2.407,"15min":2.407},"last":1.202}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4179999999999999},"min":{"1min":0.752,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.056,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.4139999999999999,"15min":1.488},"min":{"1min":0.89000000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.5840000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.173}]},{"osd":5,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.333,"15min":1.397},"min":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.7490000000000001,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.256,"15min":1.264},"min":{"1min":0.85599999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.694,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.2929999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.337,"15min":1.3600000000000001},"min":{"1min":0.93500000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":2.335,"15min":2.335},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.4139999999999999,"15min":1.46},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.877,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.5429999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.462,"15min":1.534},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.694,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.325},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.5369999999999999,"15min":1.605},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.806,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.119}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:45:23.388 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:45:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.649+0000 7ff4c6600640 1 Processor -- start 2024-01-24T17:45:23.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.650+0000 7ff4c6600640 1 -- start start 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.651+0000 7ff4c6600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff4c00ca670 0x7ff4c00cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.651+0000 7ff4c6600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4c006bde0 0x7ff4c015a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.651+0000 7ff4c6600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c00cd050 con 0x7ff4c015a9b0 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.651+0000 7ff4c6600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c006c260 con 0x7ff4c006bde0 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.651+0000 7ff4c6600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c0159fe0 con 0x7ff4c00ca670 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.651+0000 7ff4b7c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4c006bde0 0x7ff4c015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.652+0000 7ff4b7c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff4c006bde0 0x7ff4c015a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47862/0 (socket says 172.21.15.39:47862) 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.652+0000 7ff4b7c00640 1 -- 172.21.15.39:0/1257673457 learned_addr learned my addr 172.21.15.39:0/1257673457 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.652+0000 7ff4c5200640 1 --2- 172.21.15.39:0/1257673457 >> v2:172.21.15.100:3300/0 conn(0x7ff4c00ca670 0x7ff4c00cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.652+0000 7ff4b7c00640 1 -- 172.21.15.39:0/1257673457 >> v2:172.21.15.100:3300/0 conn(0x7ff4c00ca670 msgr2=0x7ff4c00cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:23.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.652+0000 7ff4b7c00640 1 --2- 172.21.15.39:0/1257673457 >> v2:172.21.15.100:3300/0 conn(0x7ff4c00ca670 0x7ff4c00cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.652+0000 7ff4b7c00640 1 -- 172.21.15.39:0/1257673457 >> v1:172.21.15.100:6789/0 conn(0x7ff4c015a9b0 legacy=0x7ff4c00cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:23.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.652+0000 7ff4b7c00640 1 -- 172.21.15.39:0/1257673457 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00caa50 con 0x7ff4c006bde0 2024-01-24T17:45:23.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.652+0000 7ff4c5200640 1 --2- 172.21.15.39:0/1257673457 >> v2:172.21.15.100:3300/0 conn(0x7ff4c00ca670 0x7ff4c00cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.653+0000 7ff4b7c00640 1 --2- 172.21.15.39:0/1257673457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c006bde0 0x7ff4c015a470 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00cbf40 tx=0x7ff4a802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d6c241097d2cf5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.653+0000 7ff4bfe00640 1 -- 172.21.15.39:0/1257673457 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8031800 con 0x7ff4c006bde0 2024-01-24T17:45:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.653+0000 7ff4bfe00640 1 -- 172.21.15.39:0/1257673457 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a8031960 con 0x7ff4c006bde0 2024-01-24T17:45:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.653+0000 7ff4bfe00640 1 -- 172.21.15.39:0/1257673457 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8031cf0 con 0x7ff4c006bde0 2024-01-24T17:45:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.654+0000 7ff4c6600640 1 -- 172.21.15.39:0/1257673457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c006bde0 msgr2=0x7ff4c015a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.654+0000 7ff4c6600640 1 --2- 172.21.15.39:0/1257673457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c006bde0 0x7ff4c015a470 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00cbf40 tx=0x7ff4a802b8a0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.654+0000 7ff4c6600640 1 -- 172.21.15.39:0/1257673457 shutdown_connections 2024-01-24T17:45:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.654+0000 7ff4c6600640 1 --2- 172.21.15.39:0/1257673457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c006bde0 0x7ff4c015a470 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.654+0000 7ff4c6600640 1 --2- 172.21.15.39:0/1257673457 >> v2:172.21.15.100:3300/0 conn(0x7ff4c00ca670 0x7ff4c00cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.654+0000 7ff4c6600640 1 -- 172.21.15.39:0/1257673457 >> 172.21.15.39:0/1257673457 conn(0x7ff4c006a090 msgr2=0x7ff4c006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.654+0000 7ff4c6600640 1 -- 172.21.15.39:0/1257673457 shutdown_connections 2024-01-24T17:45:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.654+0000 7ff4c6600640 1 -- 172.21.15.39:0/1257673457 wait complete. 2024-01-24T17:45:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.656+0000 7ff4c6600640 1 Processor -- start 2024-01-24T17:45:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.656+0000 7ff4c6600640 1 -- start start 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c6600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c006bde0 0x7ff4c00c4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c6600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c00ca670 0x7ff4c0149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c6600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c01556f0 0x7ff4c014c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c6600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c35c0 con 0x7ff4c01556f0 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c6600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c38a0 con 0x7ff4c006bde0 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c6600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c00c3b80 con 0x7ff4c00ca670 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c01556f0 0x7ff4c014c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c01556f0 0x7ff4c014c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47874/0 (socket says 172.21.15.39:47874) 2024-01-24T17:45:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.657+0000 7ff4c5c00640 1 -- 172.21.15.39:0/866271309 learned_addr learned my addr 172.21.15.39:0/866271309 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:23.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4c5200640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c006bde0 0x7ff4c00c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4b7c00640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c00ca670 0x7ff4c0149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4c5c00640 1 -- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c00ca670 msgr2=0x7ff4c0149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4c5c00640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c00ca670 0x7ff4c0149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4c5c00640 1 -- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c006bde0 msgr2=0x7ff4c00c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4c5c00640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c006bde0 0x7ff4c00c4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4c5c00640 1 -- 172.21.15.39:0/866271309 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a80094d0 con 0x7ff4c01556f0 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4b7c00640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c00ca670 0x7ff4c0149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4c5c00640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c01556f0 0x7ff4c014c2a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800c700 tx=0x7ff4b800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4bd600640 1 -- 172.21.15.39:0/866271309 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800aa50 con 0x7ff4c01556f0 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4bd600640 1 -- 172.21.15.39:0/866271309 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b800abb0 con 0x7ff4c01556f0 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4bd600640 1 -- 172.21.15.39:0/866271309 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800ce10 con 0x7ff4c01556f0 2024-01-24T17:45:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.658+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c00c3e60 con 0x7ff4c01556f0 2024-01-24T17:45:23.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.659+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c0149a50 con 0x7ff4c01556f0 2024-01-24T17:45:23.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.660+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c00c4380 con 0x7ff4c01556f0 2024-01-24T17:45:23.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.661+0000 7ff4bd600640 1 -- 172.21.15.39:0/866271309 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff4b8012420 con 0x7ff4c01556f0 2024-01-24T17:45:23.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.662+0000 7ff4bd600640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff49806a430 0x7ff49806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.662+0000 7ff4bd600640 1 -- 172.21.15.39:0/866271309 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4b8088e50 con 0x7ff4c01556f0 2024-01-24T17:45:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.662+0000 7ff4c5200640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff49806a430 0x7ff49806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.663+0000 7ff4c5200640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff49806a430 0x7ff49806c8f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000a990 tx=0x7ff4b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:23.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.666+0000 7ff4bd600640 1 -- 172.21.15.39:0/866271309 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff4c00c4380 con 0x7ff4c01556f0 2024-01-24T17:45:23.850 INFO:tasks.ceph.mds.c.smithi039.stderr:2024-01-24T17:45:23.849+0000 7f1927400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:45:23.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.856+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4c006c470 con 0x7ff49806a430 2024-01-24T17:45:23.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.878+0000 7ff4bd600640 1 -- 172.21.15.39:0/866271309 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276345 (secure 0 0 0) 0x7ff4b000a710 con 0x7ff49806a430 2024-01-24T17:45:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:23.884 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff49806a430 msgr2=0x7ff49806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff49806a430 0x7ff49806c8f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000a990 tx=0x7ff4b0008040 comp rx=0 tx=0).stop 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c01556f0 msgr2=0x7ff4c014c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c01556f0 0x7ff4c014c2a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800c700 tx=0x7ff4b800cbc0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 shutdown_connections 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff49806a430 0x7ff49806c8f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4c01556f0 0x7ff4c014c2a0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4c00ca670 0x7ff4c0149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 --2- 172.21.15.39:0/866271309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4c006bde0 0x7ff4c00c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 >> 172.21.15.39:0/866271309 conn(0x7ff4c006a090 msgr2=0x7ff4c00ae510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 shutdown_connections 2024-01-24T17:45:23.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:23.884+0000 7ff4c6600640 1 -- 172.21.15.39:0/866271309 wait complete. 2024-01-24T17:45:23.904 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-01-24T17:45:22.293556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243380,"kb_used_data":6720,"kb_used_omap":13,"kb_used_meta":236530,"kb_avail":754731340,"statfs":{"total":773094113280,"available":772844892160,"internally_reserved":0,"allocated":6881280,"data_stored":3557554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":242206808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.426115"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883136+0000","last_change":"2024-01-24T17:45:16.883136+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883136+0000","last_undegraded":"2024-01-24T17:45:16.883136+0000","last_fullsized":"2024-01-24T17:45:16.883136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882151+0000","last_change":"2024-01-24T17:45:16.882151+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882151+0000","last_undegraded":"2024-01-24T17:45:16.882151+0000","last_fullsized":"2024-01-24T17:45:16.882151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:49:04.288996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879725+0000","last_change":"2024-01-24T17:45:16.879725+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879725+0000","last_undegraded":"2024-01-24T17:45:16.879725+0000","last_fullsized":"2024-01-24T17:45:16.879725+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:05:02.184752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883414+0000","last_change":"2024-01-24T17:45:16.883414+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883414+0000","last_undegraded":"2024-01-24T17:45:16.883414+0000","last_fullsized":"2024-01-24T17:45:16.883414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:41:31.769012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879587+0000","last_change":"2024-01-24T17:45:16.879587+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879587+0000","last_undegraded":"2024-01-24T17:45:16.879587+0000","last_fullsized":"2024-01-24T17:45:16.879587+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:16.899578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878094+0000","last_change":"2024-01-24T17:45:16.878094+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878094+0000","last_undegraded":"2024-01-24T17:45:16.878094+0000","last_fullsized":"2024-01-24T17:45:16.878094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:26.416880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871981+0000","last_change":"2024-01-24T17:45:16.871981+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871981+0000","last_undegraded":"2024-01-24T17:45:16.871981+0000","last_fullsized":"2024-01-24T17:45:16.871981+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:34:38.987748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.879299+0000","last_change":"2024-01-24T17:45:16.879299+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.879299+0000","last_undegraded":"2024-01-24T17:45:16.879299+0000","last_fullsized":"2024-01-24T17:45:16.879299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871869+0000","last_change":"2024-01-24T17:45:16.871869+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871869+0000","last_undegraded":"2024-01-24T17:45:16.871869+0000","last_fullsized":"2024-01-24T17:45:16.871869+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:30:30.289664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.876053+0000","last_change":"2024-01-24T17:45:16.876053+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.876053+0000","last_undegraded":"2024-01-24T17:45:16.876053+0000","last_fullsized":"2024-01-24T17:45:16.876053+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:12.547194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882928+0000","last_change":"2024-01-24T17:45:16.882928+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882928+0000","last_undegraded":"2024-01-24T17:45:16.882928+0000","last_fullsized":"2024-01-24T17:45:16.882928+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:22.676649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.886168+0000","last_change":"2024-01-24T17:45:16.886168+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.886168+0000","last_undegraded":"2024-01-24T17:45:16.886168+0000","last_fullsized":"2024-01-24T17:45:16.886168+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874285+0000","last_change":"2024-01-24T17:45:16.874285+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874285+0000","last_undegraded":"2024-01-24T17:45:16.874285+0000","last_fullsized":"2024-01-24T17:45:16.874285+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:44.801426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882687+0000","last_change":"2024-01-24T17:45:16.882687+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882687+0000","last_undegraded":"2024-01-24T17:45:16.882687+0000","last_fullsized":"2024-01-24T17:45:16.882687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:00.154641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882836+0000","last_change":"2024-01-24T17:45:16.882836+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882836+0000","last_undegraded":"2024-01-24T17:45:16.882836+0000","last_fullsized":"2024-01-24T17:45:16.882836+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:32.202115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.882612+0000","last_change":"2024-01-24T17:45:16.882612+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.882612+0000","last_undegraded":"2024-01-24T17:45:16.882612+0000","last_fullsized":"2024-01-24T17:45:16.882612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:36.108099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.886030+0000","last_change":"2024-01-24T17:45:16.886030+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.886030+0000","last_undegraded":"2024-01-24T17:45:16.886030+0000","last_fullsized":"2024-01-24T17:45:16.886030+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874899+0000","last_change":"2024-01-24T17:45:16.874899+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874899+0000","last_undegraded":"2024-01-24T17:45:16.874899+0000","last_fullsized":"2024-01-24T17:45:16.874899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:03.155346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878536+0000","last_change":"2024-01-24T17:45:16.878536+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878536+0000","last_undegraded":"2024-01-24T17:45:16.878536+0000","last_fullsized":"2024-01-24T17:45:16.878536+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874615+0000","last_change":"2024-01-24T17:45:16.874615+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874615+0000","last_undegraded":"2024-01-24T17:45:16.874615+0000","last_fullsized":"2024-01-24T17:45:16.874615+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:23:00.228501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.883038+0000","last_change":"2024-01-24T17:45:16.883038+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.883038+0000","last_undegraded":"2024-01-24T17:45:16.883038+0000","last_fullsized":"2024-01-24T17:45:16.883038+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874831+0000","last_change":"2024-01-24T17:45:16.874831+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874831+0000","last_undegraded":"2024-01-24T17:45:16.874831+0000","last_fullsized":"2024-01-24T17:45:16.874831+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:03.318893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871818+0000","last_change":"2024-01-24T17:45:16.871818+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871818+0000","last_undegraded":"2024-01-24T17:45:16.871818+0000","last_fullsized":"2024-01-24T17:45:16.871818+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:21:59.679470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.884892+0000","last_change":"2024-01-24T17:45:16.884892+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.884892+0000","last_undegraded":"2024-01-24T17:45:16.884892+0000","last_fullsized":"2024-01-24T17:45:16.884892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878682+0000","last_change":"2024-01-24T17:45:16.878682+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878682+0000","last_undegraded":"2024-01-24T17:45:16.878682+0000","last_fullsized":"2024-01-24T17:45:16.878682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:17.910448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.871862+0000","last_change":"2024-01-24T17:45:16.871862+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.871862+0000","last_undegraded":"2024-01-24T17:45:16.871862+0000","last_fullsized":"2024-01-24T17:45:16.871862+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:35:45.998203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878591+0000","last_change":"2024-01-24T17:45:16.878591+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878591+0000","last_undegraded":"2024-01-24T17:45:16.878591+0000","last_fullsized":"2024-01-24T17:45:16.878591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.884282+0000","last_change":"2024-01-24T17:45:16.884282+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.884282+0000","last_undegraded":"2024-01-24T17:45:16.884282+0000","last_fullsized":"2024-01-24T17:45:16.884282+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878068+0000","last_change":"2024-01-24T17:45:16.878068+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878068+0000","last_undegraded":"2024-01-24T17:45:16.878068+0000","last_fullsized":"2024-01-24T17:45:16.878068+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:09:47.749409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.878139+0000","last_change":"2024-01-24T17:45:16.878139+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.878139+0000","last_undegraded":"2024-01-24T17:45:16.878139+0000","last_fullsized":"2024-01-24T17:45:16.878139+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:30.797138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.874263+0000","last_change":"2024-01-24T17:45:16.874263+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.874263+0000","last_undegraded":"2024-01-24T17:45:16.874263+0000","last_fullsized":"2024-01-24T17:45:16.874263+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:12.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-24T17:45:17.864345+0000","last_change":"2024-01-24T17:45:17.864345+0000","last_active":"2024-01-24T17:45:17.864345+0000","last_peered":"2024-01-24T17:45:17.864345+0000","last_clean":"2024-01-24T17:45:17.864345+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:17.864345+0000","last_undegraded":"2024-01-24T17:45:17.864345+0000","last_fullsized":"2024-01-24T17:45:17.864345+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":60,"state":"creating+peering","last_fresh":"2024-01-24T17:45:16.885971+0000","last_change":"2024-01-24T17:45:16.885971+0000","last_active":"2024-01-24T17:45:16.848013+0000","last_peered":"2024-01-24T17:45:16.848013+0000","last_clean":"2024-01-24T17:45:16.848013+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-24T17:45:16.885971+0000","last_undegraded":"2024-01-24T17:45:16.885971+0000","last_fullsized":"2024-01-24T17:45:16.885971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":199,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-24T17:45:14.842315+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:45:14.842315+0000","last_peered":"2024-01-24T17:45:14.842315+0000","last_clean":"2024-01-24T17:45:14.842315+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:45:14.842315+0000","last_undegraded":"2024-01-24T17:45:14.842315+0000","last_fullsized":"2024-01-24T17:45:14.842315+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.05,"15min":1.0249999999999999},"min":{"1min":0.80300000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.23},{"interface":"front","average":{"1min":1.127,"5min":1.048,"15min":1.0329999999999999},"min":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.599,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.4510000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0860000000000001,"15min":1.1000000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1859999999999999,"15min":1.1919999999999999},"min":{"1min":0.81499999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.649,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1000000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.171,"15min":1.1579999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.764,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.119,"5min":1.181,"15min":1.2190000000000001},"min":{"1min":0.745,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.484,"5min":2.484,"15min":2.484},"last":0.96699999999999997}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8779999999999999,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.282,"15min":1.3100000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.6269999999999998,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.1739999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.296,"15min":1.3200000000000001},"min":{"1min":0.84699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.1190000000000002,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.331},{"interface":"front","average":{"1min":1.383,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5449999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.589}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.319,"15min":1.3520000000000001},"min":{"1min":0.70799999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.264},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3979999999999999,"15min":1.4350000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.966,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.5489999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4990000000000001,"15min":1.5349999999999999},"min":{"1min":0.72599999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.5720000000000001,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.635},{"interface":"front","average":{"1min":1.399,"5min":1.472,"15min":1.488},"min":{"1min":0.86099999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.2450000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.679}]}]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":720896,"data_stored":301465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.034,"15min":1.024},"min":{"1min":0.70699999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6919999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.583},{"interface":"front","average":{"1min":1.141,"5min":1.044,"15min":1.0189999999999999},"min":{"1min":0.749,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.2050000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.022,"15min":0.99199999999999999},"min":{"1min":0.70199999999999996,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4890000000000001,"5min":1.73,"15min":1.73},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1180000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.722,"5min":1.972,"15min":1.972},"last":1.1379999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.79300000000000004,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5369999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.169,"15min":1.167},"min":{"1min":0.90700000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.514,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.6559999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.242,"15min":1.254},"min":{"1min":0.83799999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.542,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.387},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2310000000000001,"15min":1.2529999999999999},"min":{"1min":0.84199999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5980000000000001,"5min":2.46,"15min":2.46},"last":1.758}]},{"osd":4,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.2569999999999999,"15min":1.2370000000000001},"min":{"1min":0.76000000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.661,"5min":1.988,"15min":1.988},"last":1.9179999999999999},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2989999999999999,"15min":1.3029999999999999},"min":{"1min":0.73899999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.8420000000000001,"5min":1.849,"15min":1.849},"last":1.784}]},{"osd":5,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3080000000000001,"15min":1.296},"min":{"1min":0.97999999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.645,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.8140000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.89800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9319999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":2.0030000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.355,"15min":1.367},"min":{"1min":0.85399999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.978,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.694},{"interface":"front","average":{"1min":1.45,"5min":1.454,"15min":1.4630000000000001},"min":{"1min":1.008,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.899,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.8859999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1302528,"data_stored":886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0669999999999999,"15min":1.099},"min":{"1min":0.67500000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.47,"5min":1.508,"15min":1.508},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.998,"5min":1.0289999999999999,"15min":1.0309999999999999},"min":{"1min":0.67900000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5,"5min":1.889,"15min":1.889},"last":1.0820000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.077,"15min":1.0740000000000001},"min":{"1min":0.72899999999999998,"5min":0.622,"15min":0.622},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.22,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.552,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.6040000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2230000000000001,"15min":1.2290000000000001},"min":{"1min":0.80600000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.28,"15min":1.3120000000000001},"min":{"1min":0.79800000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.9870000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.97299999999999998}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2909999999999999,"15min":1.341},"min":{"1min":0.77800000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6020000000000001,"5min":3.036,"15min":3.036},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3320000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.641,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.4550000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4339999999999999,"15min":1.4530000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.915,"5min":2.069,"15min":2.069},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.329,"15min":1.3600000000000001},"min":{"1min":0.77600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5780000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.2010000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.339,"15min":1.3560000000000001},"min":{"1min":0.91100000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.724,"5min":2.504,"15min":2.504},"last":1.645},{"interface":"front","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":0.88600000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.044,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.8839999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.458,"15min":1.4930000000000001},"min":{"1min":0.63800000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.6719999999999999,"5min":3.161,"15min":3.161},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.5900000000000001,"15min":1.6339999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.1389999999999998,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.7569999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.048,"15min":1.038},"min":{"1min":0.73499999999999999,"5min":0.621,"15min":0.621},"max":{"1min":1.673,"5min":1.698,"15min":1.698},"last":1.038},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.093,"15min":1.0960000000000001},"min":{"1min":0.71199999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5529999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.071}]},{"osd":1,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.0800000000000001,"15min":1.052},"min":{"1min":0.78500000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5049999999999999,"5min":1.633,"15min":1.633},"last":1.25},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1899999999999999,"15min":1.2170000000000001},"min":{"1min":0.67800000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.4139999999999999,"5min":1.849,"15min":1.849},"last":1.2010000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.206,"15min":1.212},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.304,"15min":1.3340000000000001},"min":{"1min":0.71199999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.724,"5min":1.986,"15min":1.986},"last":1.1200000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1850000000000001,"15min":1.1910000000000001},"min":{"1min":0.81000000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.456,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.319,"5min":1.343,"15min":1.351},"min":{"1min":0.89400000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.8029999999999999,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":0.95299999999999996}]},{"osd":5,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3480000000000001,"15min":1.373},"min":{"1min":1.014,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.613,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.298,"15min":1.306},"min":{"1min":0.85799999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5600000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":0.98799999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4550000000000001,"15min":1.4730000000000001},"min":{"1min":0.93799999999999994,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.3159999999999998,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.4610000000000001,"15min":1.4790000000000001},"min":{"1min":0.86899999999999999,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.91600000000000004}]},{"osd":7,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4330000000000001,"15min":1.4259999999999999},"min":{"1min":0.92000000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0209999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.6000000000000001,"15min":1.621},"min":{"1min":0.97299999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.2309999999999999,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.411}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.91600000000000004,"15min":0.872},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.792,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95499999999999996,"15min":0.89100000000000001},"min":{"1min":0.64200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5680000000000001,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":1.0980000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.93899999999999995,"15min":0.95099999999999996},"min":{"1min":0.53100000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.819,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0720000000000001,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.663,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.0620000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.105,"15min":1.111},"min":{"1min":0.629,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.571,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.181,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.593,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.69199999999999995}]},{"osd":4,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3200000000000001,"15min":1.292},"min":{"1min":0.90800000000000003,"5min":0.877,"15min":0.877},"max":{"1min":1.8240000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9790000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.262}]},{"osd":5,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4350000000000001,"15min":1.4370000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2189999999999999,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":0.995},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4159999999999999,"15min":1.4239999999999999},"min":{"1min":1.024,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.0579999999999998,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.288}]},{"osd":6,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.532,"15min":1.5029999999999999},"min":{"1min":1.0429999999999999,"5min":0.871,"15min":0.871},"max":{"1min":1.984,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.486},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4730000000000001,"15min":1.4630000000000001},"min":{"1min":0.90200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.1120000000000001,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.7450000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5389999999999999,"15min":1.5169999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.012,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.6859999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.5249999999999999},"min":{"1min":1.0649999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.1160000000000001,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.444}]}]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31812,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340028,"statfs":{"total":96636764160,"available":96604188672,"internally_reserved":0,"allocated":708608,"data_stored":295648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.439,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.24,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93600000000000005,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.99299999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0369999999999999,"15min":1.0549999999999999},"min":{"1min":0.51300000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.5509999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.173,"15min":1.2150000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":1.5600000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.98199999999999998,"15min":0.996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6910000000000001,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2050000000000001,"15min":1.218},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7629999999999999,"5min":2.996,"15min":2.996},"last":1.327}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.306,"15min":1.329},"min":{"1min":0.72899999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6950000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4099999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.7,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.29}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3600000000000001,"15min":1.409},"min":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.762,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.379},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2969999999999999,"15min":1.331},"min":{"1min":0.86299999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.625,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.0680000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.369,"15min":1.4019999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9350000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.266},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4399999999999999,"15min":1.4830000000000001},"min":{"1min":0.80200000000000005,"5min":0.753,"15min":0.753},"max":{"1min":2.2530000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.4810000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.526,"15min":1.5860000000000001},"min":{"1min":0.82499999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0209999999999999,"5min":3.343,"15min":3.343},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4630000000000001,"15min":1.512},"min":{"1min":0.755,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9510000000000001,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.177}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1302528,"data_stored":886016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.874,"15min":0.85999999999999999},"min":{"1min":0.45500000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4419999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.93000000000000005,"15min":0.95299999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.155,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.95399999999999996,"15min":0.98399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.4790000000000001,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5680000000000001,"5min":1.895,"15min":1.895},"last":2.0270000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.143,"15min":1.1639999999999999},"min":{"1min":0.61299999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4930000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.194,"15min":1.244},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.4019999999999999,"5min":3.149,"15min":3.149},"last":0.67600000000000005}]},{"osd":4,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.353,"15min":1.365},"min":{"1min":0.88700000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.77,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.175},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.296,"15min":1.3240000000000001},"min":{"1min":0.83599999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.698,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.3999999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.323,"15min":1.357},"min":{"1min":0.73499999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.806,"5min":2.577,"15min":2.577},"last":1.075},{"interface":"front","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.9730000000000001,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.4850000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.4019999999999999,"15min":1.4450000000000001},"min":{"1min":0.80000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.653,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.335},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.46,"15min":1.5309999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.921,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.145}]},{"osd":7,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4570000000000001,"15min":1.4750000000000001},"min":{"1min":0.91400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0979999999999999,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.76},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.5049999999999999,"15min":1.5549999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.641,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.5740000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":720896,"data_stored":301465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80000000000000004,"15min":0.81799999999999995},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.242,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.93000000000000005,"15min":0.96999999999999997},"min":{"1min":0.52900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.2410000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.84199999999999997,"15min":0.83399999999999996},"min":{"1min":0.45200000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.997,"5min":1.046,"15min":1.1000000000000001},"min":{"1min":0.46800000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4430000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.359}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.1359999999999999,"15min":1.1890000000000001},"min":{"1min":0.51000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.639,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.105,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.60699999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6040000000000001,"5min":2.407,"15min":2.407},"last":1.202}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4179999999999999},"min":{"1min":0.752,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.056,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.4139999999999999,"15min":1.488},"min":{"1min":0.89000000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.5840000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.173}]},{"osd":5,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.333,"15min":1.397},"min":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.7490000000000001,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.256,"15min":1.264},"min":{"1min":0.85599999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.694,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.2929999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.337,"15min":1.3600000000000001},"min":{"1min":0.93500000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":2.335,"15min":2.335},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.341,"5min":1.4139999999999999,"15min":1.46},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.877,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.5429999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.462,"15min":1.534},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.694,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.325},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.5369999999999999,"15min":1.605},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.806,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.119}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:45:23.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3f is not active+clean 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3e is not active+clean 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3d is not active+clean 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3b is not active+clean 2024-01-24T17:45:23.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.39 is not active+clean 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.38 is not active+clean 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.37 is not active+clean 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.36 is not active+clean 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.35 is not active+clean 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.33 is not active+clean 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.32 is not active+clean 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.31 is not active+clean 2024-01-24T17:45:23.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-01-24T17:45:23.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2f is not active+clean 2024-01-24T17:45:23.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2e is not active+clean 2024-01-24T17:45:23.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2d is not active+clean 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2c is not active+clean 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2b is not active+clean 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2a is not active+clean 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.29 is not active+clean 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.28 is not active+clean 2024-01-24T17:45:23.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.27 is not active+clean 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.24 is not active+clean 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.23 is not active+clean 2024-01-24T17:45:23.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.22 is not active+clean 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.21 is not active+clean 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-01-24T17:45:23.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.883136+0000', 'last_change': '2024-01-24T17:45:16.883136+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.883136+0000', 'last_undegraded': '2024-01-24T17:45:16.883136+0000', 'last_fullsized': '2024-01-24T17:45:16.883136+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-24T17:45:23.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1f is not active+clean 2024-01-24T17:45:23.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-01-24T17:45:23.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.882151+0000', 'last_change': '2024-01-24T17:45:16.882151+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.882151+0000', 'last_undegraded': '2024-01-24T17:45:16.882151+0000', 'last_fullsized': '2024-01-24T17:45:16.882151+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T04:49:04.288996+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-24T17:45:23.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-01-24T17:45:23.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-01-24T17:45:23.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.879725+0000', 'last_change': '2024-01-24T17:45:16.879725+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.879725+0000', 'last_undegraded': '2024-01-24T17:45:16.879725+0000', 'last_fullsized': '2024-01-24T17:45:16.879725+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T20:05:02.184752+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-24T17:45:23.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-01-24T17:45:23.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-01-24T17:45:23.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.883414+0000', 'last_change': '2024-01-24T17:45:16.883414+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.883414+0000', 'last_undegraded': '2024-01-24T17:45:16.883414+0000', 'last_fullsized': '2024-01-24T17:45:16.883414+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T03:41:31.769012+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-24T17:45:23.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-01-24T17:45:23.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.879587+0000', 'last_change': '2024-01-24T17:45:16.879587+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.879587+0000', 'last_undegraded': '2024-01-24T17:45:16.879587+0000', 'last_fullsized': '2024-01-24T17:45:16.879587+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T01:00:16.899578+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.878094+0000', 'last_change': '2024-01-24T17:45:16.878094+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.878094+0000', 'last_undegraded': '2024-01-24T17:45:16.878094+0000', 'last_fullsized': '2024-01-24T17:45:16.878094+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T21:49:26.416880+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-01-24T17:45:23.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.871981+0000', 'last_change': '2024-01-24T17:45:16.871981+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.871981+0000', 'last_undegraded': '2024-01-24T17:45:16.871981+0000', 'last_fullsized': '2024-01-24T17:45:16.871981+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T20:34:38.987748+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.879299+0000', 'last_change': '2024-01-24T17:45:16.879299+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.879299+0000', 'last_undegraded': '2024-01-24T17:45:16.879299+0000', 'last_fullsized': '2024-01-24T17:45:16.879299+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.871869+0000', 'last_change': '2024-01-24T17:45:16.871869+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.871869+0000', 'last_undegraded': '2024-01-24T17:45:16.871869+0000', 'last_fullsized': '2024-01-24T17:45:16.871869+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T18:30:30.289664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2024-01-24T17:45:23.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.876053+0000', 'last_change': '2024-01-24T17:45:16.876053+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.876053+0000', 'last_undegraded': '2024-01-24T17:45:16.876053+0000', 'last_fullsized': '2024-01-24T17:45:16.876053+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T01:18:12.547194+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.14 is not active+clean 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.882928+0000', 'last_change': '2024-01-24T17:45:16.882928+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.882928+0000', 'last_undegraded': '2024-01-24T17:45:16.882928+0000', 'last_fullsized': '2024-01-24T17:45:16.882928+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T03:57:22.676649+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.886168+0000', 'last_change': '2024-01-24T17:45:16.886168+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.886168+0000', 'last_undegraded': '2024-01-24T17:45:16.886168+0000', 'last_fullsized': '2024-01-24T17:45:16.886168+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.874285+0000', 'last_change': '2024-01-24T17:45:16.874285+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.874285+0000', 'last_undegraded': '2024-01-24T17:45:16.874285+0000', 'last_fullsized': '2024-01-24T17:45:16.874285+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T02:17:44.801426+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-01-24T17:45:23.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.882687+0000', 'last_change': '2024-01-24T17:45:16.882687+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.882687+0000', 'last_undegraded': '2024-01-24T17:45:16.882687+0000', 'last_fullsized': '2024-01-24T17:45:16.882687+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T20:58:00.154641+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.882836+0000', 'last_change': '2024-01-24T17:45:16.882836+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.882836+0000', 'last_undegraded': '2024-01-24T17:45:16.882836+0000', 'last_fullsized': '2024-01-24T17:45:16.882836+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T02:35:32.202115+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.882612+0000', 'last_change': '2024-01-24T17:45:16.882612+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.882612+0000', 'last_undegraded': '2024-01-24T17:45:16.882612+0000', 'last_fullsized': '2024-01-24T17:45:16.882612+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T19:38:36.108099+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2024-01-24T17:45:23.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.886030+0000', 'last_change': '2024-01-24T17:45:16.886030+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.886030+0000', 'last_undegraded': '2024-01-24T17:45:16.886030+0000', 'last_fullsized': '2024-01-24T17:45:16.886030+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.874899+0000', 'last_change': '2024-01-24T17:45:16.874899+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.874899+0000', 'last_undegraded': '2024-01-24T17:45:16.874899+0000', 'last_fullsized': '2024-01-24T17:45:16.874899+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T01:31:03.155346+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-01-24T17:45:23.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.878536+0000', 'last_change': '2024-01-24T17:45:16.878536+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.878536+0000', 'last_undegraded': '2024-01-24T17:45:16.878536+0000', 'last_fullsized': '2024-01-24T17:45:16.878536+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.874615+0000', 'last_change': '2024-01-24T17:45:16.874615+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.874615+0000', 'last_undegraded': '2024-01-24T17:45:16.874615+0000', 'last_fullsized': '2024-01-24T17:45:16.874615+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T21:23:00.228501+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.883038+0000', 'last_change': '2024-01-24T17:45:16.883038+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.883038+0000', 'last_undegraded': '2024-01-24T17:45:16.883038+0000', 'last_fullsized': '2024-01-24T17:45:16.883038+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-24T17:45:23.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.874831+0000', 'last_change': '2024-01-24T17:45:16.874831+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.874831+0000', 'last_undegraded': '2024-01-24T17:45:16.874831+0000', 'last_fullsized': '2024-01-24T17:45:16.874831+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T00:06:03.318893+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.871818+0000', 'last_change': '2024-01-24T17:45:16.871818+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.871818+0000', 'last_undegraded': '2024-01-24T17:45:16.871818+0000', 'last_fullsized': '2024-01-24T17:45:16.871818+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T05:21:59.679470+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-01-24T17:45:23.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.884892+0000', 'last_change': '2024-01-24T17:45:16.884892+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.884892+0000', 'last_undegraded': '2024-01-24T17:45:16.884892+0000', 'last_fullsized': '2024-01-24T17:45:16.884892+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.878682+0000', 'last_change': '2024-01-24T17:45:16.878682+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.878682+0000', 'last_undegraded': '2024-01-24T17:45:16.878682+0000', 'last_fullsized': '2024-01-24T17:45:16.878682+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T19:56:17.910448+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.4 is not active+clean 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-01-24T17:45:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.871862+0000', 'last_change': '2024-01-24T17:45:16.871862+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.871862+0000', 'last_undegraded': '2024-01-24T17:45:16.871862+0000', 'last_fullsized': '2024-01-24T17:45:16.871862+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T05:35:45.998203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.878591+0000', 'last_change': '2024-01-24T17:45:16.878591+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.878591+0000', 'last_undegraded': '2024-01-24T17:45:16.878591+0000', 'last_fullsized': '2024-01-24T17:45:16.878591+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.884282+0000', 'last_change': '2024-01-24T17:45:16.884282+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.884282+0000', 'last_undegraded': '2024-01-24T17:45:16.884282+0000', 'last_fullsized': '2024-01-24T17:45:16.884282+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-24T17:45:23.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.878068+0000', 'last_change': '2024-01-24T17:45:16.878068+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.878068+0000', 'last_undegraded': '2024-01-24T17:45:16.878068+0000', 'last_fullsized': '2024-01-24T17:45:16.878068+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T01:09:47.749409+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.0 is not active+clean 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.878139+0000', 'last_change': '2024-01-24T17:45:16.878139+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.878139+0000', 'last_undegraded': '2024-01-24T17:45:16.878139+0000', 'last_fullsized': '2024-01-24T17:45:16.878139+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T23:55:30.797138+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-01-24T17:45:23.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-01-24T17:45:23.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.874263+0000', 'last_change': '2024-01-24T17:45:16.874263+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.874263+0000', 'last_undegraded': '2024-01-24T17:45:16.874263+0000', 'last_fullsized': '2024-01-24T17:45:16.874263+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-26T00:01:12.641089+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-24T17:45:23.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-01-24T17:45:23.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-24T17:45:17.864345+0000', 'last_change': '2024-01-24T17:45:17.864345+0000', 'last_active': '2024-01-24T17:45:17.864345+0000', 'last_peered': '2024-01-24T17:45:17.864345+0000', 'last_clean': '2024-01-24T17:45:17.864345+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:17.864345+0000', 'last_undegraded': '2024-01-24T17:45:17.864345+0000', 'last_fullsized': '2024-01-24T17:45:17.864345+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:17.864345+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-24T17:45:23.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-01-24T17:45:23.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 60, 'state': 'creating+peering', 'last_fresh': '2024-01-24T17:45:16.885971+0000', 'last_change': '2024-01-24T17:45:16.885971+0000', 'last_active': '2024-01-24T17:45:16.848013+0000', 'last_peered': '2024-01-24T17:45:16.848013+0000', 'last_clean': '2024-01-24T17:45:16.848013+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-24T17:45:16.885971+0000', 'last_undegraded': '2024-01-24T17:45:16.885971+0000', 'last_fullsized': '2024-01-24T17:45:16.885971+0000', 'mapping_epoch': 60, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 60, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'last_clean_scrub_stamp': '2024-01-24T17:45:16.848013+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-24T17:45:23.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-24T17:45:24.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.168+0000 7f74a1200640 1 Processor -- start 2024-01-24T17:45:24.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.168+0000 7f74a1200640 1 -- start start 2024-01-24T17:45:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.170+0000 7f74a1200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f749c15d020 0x7f749c15d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.170+0000 7f74a1200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c15dac0 con 0x7f749c0cf990 2024-01-24T17:45:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.170+0000 7f74a1200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c15dda0 con 0x7f749c15eb80 2024-01-24T17:45:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.170+0000 7f74a1200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c15e080 con 0x7f749c15d020 2024-01-24T17:45:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.170+0000 7f7499e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f749c0cf990 0x7f749c0cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:36748/0 (socket says 172.21.15.39:36748) 2024-01-24T17:45:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.170+0000 7f7499e00640 1 -- 172.21.15.39:0/2673288309 learned_addr learned my addr 172.21.15.39:0/2673288309 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:24.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7491200640 1 --2- 172.21.15.39:0/2673288309 >> v2:172.21.15.100:3300/0 conn(0x7f749c15d020 0x7f749c15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710971950 0 0) 0x7f749c15dac0 con 0x7f749c0cf990 2024-01-24T17:45:24.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c003520 con 0x7f749c0cf990 2024-01-24T17:45:24.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1083342702 0 0) 0x7f747c003520 con 0x7f749c0cf990 2024-01-24T17:45:24.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 >> v2:172.21.15.100:3300/0 conn(0x7f749c15d020 msgr2=0x7f749c15d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 --2- 172.21.15.39:0/2673288309 >> v2:172.21.15.100:3300/0 conn(0x7f749c15d020 0x7f749c15d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 >> v1:172.21.15.100:6789/0 conn(0x7f749c15eb80 legacy=0x7f749c15f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c15e380 con 0x7f749c0cf990 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f748c0019f0 con 0x7f749c0cf990 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 >> v1:172.21.15.39:6789/0 conn(0x7f749c0cf990 legacy=0x7f749c0cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c003970 0x7f747c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f747c0047c0 0x7f747c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.171+0000 7f7499400640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f747c005180 0x7f747c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c15e080 con 0x7f747c005180 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00c000 con 0x7f747c0047c0 2024-01-24T17:45:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00c1c0 con 0x7f747c003970 2024-01-24T17:45:24.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f749a800640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f747c005180 0x7f747c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f7491200640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f747c0047c0 0x7f747c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f7499e00640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c003970 0x7f747c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f749a800640 1 -- 172.21.15.39:0/2673288309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c003970 msgr2=0x7f747c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:24.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f749a800640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c003970 0x7f747c003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f749a800640 1 -- 172.21.15.39:0/2673288309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f747c0047c0 msgr2=0x7f747c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:24.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f749a800640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f747c0047c0 0x7f747c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.172+0000 7f749a800640 1 -- 172.21.15.39:0/2673288309 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c003520 con 0x7f747c005180 2024-01-24T17:45:24.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.173+0000 7f749a800640 1 -- 172.21.15.39:0/2673288309 reap_dead start 2024-01-24T17:45:24.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f749a800640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f747c005180 0x7f747c00b980 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f7494003db0 tx=0x7f747c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3299e8af62cb054 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:24.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494032600 con 0x7f747c005180 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749402e050 con 0x7f747c005180 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f7499400640 1 -- 172.21.15.39:0/2673288309 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494032980 con 0x7f747c005180 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f74a1200640 1 -- 172.21.15.39:0/2673288309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f747c005180 msgr2=0x7f747c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f74a1200640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f747c005180 0x7f747c00b980 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f7494003db0 tx=0x7f747c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f74a1200640 1 -- 172.21.15.39:0/2673288309 shutdown_connections 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f74a1200640 1 --2- 172.21.15.39:0/2673288309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f747c005180 0x7f747c00b980 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.174+0000 7f74a1200640 1 -- 172.21.15.39:0/2673288309 >> 172.21.15.39:0/2673288309 conn(0x7f749c071790 msgr2=0x7f749c071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.175+0000 7f74a1200640 1 -- 172.21.15.39:0/2673288309 shutdown_connections 2024-01-24T17:45:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.175+0000 7f74a1200640 1 -- 172.21.15.39:0/2673288309 wait complete. 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.176+0000 7f74a1200640 1 Processor -- start 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.177+0000 7f74a1200640 1 -- start start 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.177+0000 7f74a1200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c005180 0x7f749c0c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f74a1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f749c0c7f30 0x7f749c1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f74a1200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f749c15f010 0x7f749c15f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f74a1200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c15f9d0 con 0x7f749c0c7f30 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f74a1200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c6b10 con 0x7f749c15f010 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f74a1200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c6df0 con 0x7f747c005180 2024-01-24T17:45:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7491200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f749c0c7f30 0x7f749c1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7491200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f749c0c7f30 0x7f749c1591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47902/0 (socket says 172.21.15.39:47902) 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7491200640 1 -- 172.21.15.39:0/1616890452 learned_addr learned my addr 172.21.15.39:0/1616890452 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7499e00640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c005180 0x7f749c0c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f749a800640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f749c15f010 0x7f749c15f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7491200640 1 -- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c005180 msgr2=0x7f749c0c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7491200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c005180 0x7f749c0c6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7491200640 1 -- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f749c15f010 msgr2=0x7f749c15f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7491200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f749c15f010 0x7f749c15f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.178+0000 7f7491200640 1 -- 172.21.15.39:0/1616890452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c0c70d0 con 0x7f749c0c7f30 2024-01-24T17:45:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.179+0000 7f7491200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f749c0c7f30 0x7f749c1591b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7484011540 tx=0x7f7484011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.179+0000 7f7492a00640 1 -- 172.21.15.39:0/1616890452 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484015930 con 0x7f749c0c7f30 2024-01-24T17:45:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.179+0000 7f7492a00640 1 -- 172.21.15.39:0/1616890452 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7484015a90 con 0x7f749c0c7f30 2024-01-24T17:45:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.179+0000 7f7492a00640 1 -- 172.21.15.39:0/1616890452 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484015e60 con 0x7f749c0c7f30 2024-01-24T17:45:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.179+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c0c73b0 con 0x7f749c0c7f30 2024-01-24T17:45:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.179+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c1596f0 con 0x7f749c0c7f30 2024-01-24T17:45:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.180+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c073ef0 con 0x7f749c0c7f30 2024-01-24T17:45:24.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.181+0000 7f7492a00640 1 -- 172.21.15.39:0/1616890452 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f748400f460 con 0x7f749c0c7f30 2024-01-24T17:45:24.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.182+0000 7f7492a00640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f747406a430 0x7f747406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.182+0000 7f7492a00640 1 -- 172.21.15.39:0/1616890452 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7484089140 con 0x7f749c0c7f30 2024-01-24T17:45:24.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.183+0000 7f7499e00640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f747406a430 0x7f747406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.184+0000 7f7499e00640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f747406a430 0x7f747406c8f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f748c001b20 tx=0x7f748c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:24.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.187+0000 7f7492a00640 1 -- 172.21.15.39:0/1616890452 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7484054730 con 0x7f749c0c7f30 2024-01-24T17:45:24.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.459+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f749c0b2140 con 0x7f749c0c7f30 2024-01-24T17:45:24.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.460+0000 7f7492a00640 1 -- 172.21.15.39:0/1616890452 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1398 (secure 0 0 0) 0x7f7484057d50 con 0x7f749c0c7f30 2024-01-24T17:45:24.461 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:24.461 INFO:teuthology.orchestra.run.smithi039.stdout:{"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":255,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":63,"num_osds":8,"num_up_osds":8,"osd_up_since":1706118081,"num_in_osds":8,"osd_in_since":1706118072,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":249221120,"bytes_avail":772844892160,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":34,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":6261}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-01-24T17:45:06.288944+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-01-24T17:45:24.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f747406a430 msgr2=0x7f747406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f747406a430 0x7f747406c8f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f748c001b20 tx=0x7f748c009040 comp rx=0 tx=0).stop 2024-01-24T17:45:24.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f749c0c7f30 msgr2=0x7f749c1591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f749c0c7f30 0x7f749c1591b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7484011540 tx=0x7f7484011a00 comp rx=0 tx=0).stop 2024-01-24T17:45:24.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 shutdown_connections 2024-01-24T17:45:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f749c15f010 0x7f749c15f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f747406a430 0x7f747406c8f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f749c0c7f30 0x7f749c1591b0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 --2- 172.21.15.39:0/1616890452 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f747c005180 0x7f749c0c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 >> 172.21.15.39:0/1616890452 conn(0x7f749c071790 msgr2=0x7f749c15c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.463+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 shutdown_connections 2024-01-24T17:45:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.464+0000 7f74a1200640 1 -- 172.21.15.39:0/1616890452 wait complete. 2024-01-24T17:45:24.484 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-24T17:45:24.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:45:24.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.702+0000 7f0ccfe00640 1 Processor -- start 2024-01-24T17:45:24.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.703+0000 7f0ccfe00640 1 -- start start 2024-01-24T17:45:24.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.704+0000 7f0ccfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0cd00d03b0 0x7f0cd015db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.704+0000 7f0ccfe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd00d0860 con 0x7f0cd015cbf0 2024-01-24T17:45:24.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.704+0000 7f0ccfe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd00d0a90 con 0x7f0cd00d0da0 2024-01-24T17:45:24.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.704+0000 7f0ccfe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd00cd760 con 0x7f0cd00d03b0 2024-01-24T17:45:24.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.704+0000 7f0ccea00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0cd015cbf0 0x7f0cd00ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:36750/0 (socket says 172.21.15.39:36750) 2024-01-24T17:45:24.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.704+0000 7f0ccea00640 1 -- 172.21.15.39:0/2613270529 learned_addr learned my addr 172.21.15.39:0/2613270529 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:24.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.704+0000 7f0cc7e00640 1 --2- 172.21.15.39:0/2613270529 >> v2:172.21.15.39:3300/0 conn(0x7f0cd00d03b0 0x7f0cd015db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.705+0000 7f0cc7e00640 1 -- 172.21.15.39:0/2613270529 >> v1:172.21.15.100:6789/0 conn(0x7f0cd00d0da0 legacy=0x7f0cd015e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:24.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.705+0000 7f0cc7e00640 1 -- 172.21.15.39:0/2613270529 >> v1:172.21.15.39:6789/0 conn(0x7f0cd015cbf0 legacy=0x7f0cd00ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.705+0000 7f0cce000640 1 -- 172.21.15.39:0/2613270529 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144010607 0 0) 0x7f0cd00d0860 con 0x7f0cd015cbf0 2024-01-24T17:45:24.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.705+0000 7f0cc7e00640 1 -- 172.21.15.39:0/2613270529 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd00cda40 con 0x7f0cd00d03b0 2024-01-24T17:45:24.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.706+0000 7f0cc7e00640 1 --2- 172.21.15.39:0/2613270529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd00d03b0 0x7f0cd015db60 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f0cb80097d0 tx=0x7f0cb802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4f58b17555441b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:24.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.706+0000 7f0cce000640 1 -- 172.21.15.39:0/2613270529 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8031830 con 0x7f0cd00d03b0 2024-01-24T17:45:24.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0cce000640 1 -- 172.21.15.39:0/2613270529 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb8031990 con 0x7f0cd00d03b0 2024-01-24T17:45:24.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0cce000640 1 -- 172.21.15.39:0/2613270529 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8031d20 con 0x7f0cd00d03b0 2024-01-24T17:45:24.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0ccfe00640 1 -- 172.21.15.39:0/2613270529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd00d03b0 msgr2=0x7f0cd015db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0ccfe00640 1 --2- 172.21.15.39:0/2613270529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd00d03b0 0x7f0cd015db60 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f0cb80097d0 tx=0x7f0cb802b8d0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0ccfe00640 1 -- 172.21.15.39:0/2613270529 shutdown_connections 2024-01-24T17:45:24.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0ccfe00640 1 --2- 172.21.15.39:0/2613270529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd00d03b0 0x7f0cd015db60 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0ccfe00640 1 -- 172.21.15.39:0/2613270529 >> 172.21.15.39:0/2613270529 conn(0x7f0cd009bb90 msgr2=0x7f0cd00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:24.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0ccfe00640 1 -- 172.21.15.39:0/2613270529 shutdown_connections 2024-01-24T17:45:24.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.707+0000 7f0ccfe00640 1 -- 172.21.15.39:0/2613270529 wait complete. 2024-01-24T17:45:24.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.709+0000 7f0ccfe00640 1 Processor -- start 2024-01-24T17:45:24.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.709+0000 7f0ccfe00640 1 -- start start 2024-01-24T17:45:24.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccfe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cd00d03b0 0x7f0cd01552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0cd00d0da0 0x7f0cd0150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd015cbf0 0x7f0cd00cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd0155ef0 con 0x7f0cd015cbf0 2024-01-24T17:45:24.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccfe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00c6820 con 0x7f0cd00d03b0 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd0155930 con 0x7f0cd00d0da0 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd015cbf0 0x7f0cd00cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd015cbf0 0x7f0cd00cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47916/0 (socket says 172.21.15.39:47916) 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.710+0000 7f0ccf400640 1 -- 172.21.15.39:0/1056082424 learned_addr learned my addr 172.21.15.39:0/1056082424 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.711+0000 7f0cc7e00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0cd00d0da0 0x7f0cd0150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.711+0000 7f0ccea00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cd00d03b0 0x7f0cd01552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.711+0000 7f0ccf400640 1 -- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0cd00d0da0 msgr2=0x7f0cd0150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.711+0000 7f0ccf400640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0cd00d0da0 0x7f0cd0150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.711+0000 7f0ccf400640 1 -- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cd00d03b0 msgr2=0x7f0cd01552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.711+0000 7f0ccf400640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cd00d03b0 0x7f0cd01552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.711+0000 7f0ccf400640 1 -- 172.21.15.39:0/1056082424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8009400 con 0x7f0cd015cbf0 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.711+0000 7f0ccf400640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd015cbf0 0x7f0cd00cba80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800c7b0 tx=0x7f0cc800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.712+0000 7f0cc6a00640 1 -- 172.21.15.39:0/1056082424 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8004b60 con 0x7f0cd015cbf0 2024-01-24T17:45:24.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.712+0000 7f0cc6a00640 1 -- 172.21.15.39:0/1056082424 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc8004cc0 con 0x7f0cd015cbf0 2024-01-24T17:45:24.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.712+0000 7f0cc6a00640 1 -- 172.21.15.39:0/1056082424 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc800aa90 con 0x7f0cd015cbf0 2024-01-24T17:45:24.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.712+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd0155c10 con 0x7f0cd015cbf0 2024-01-24T17:45:24.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.712+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd0151f40 con 0x7f0cd015cbf0 2024-01-24T17:45:24.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.713+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd00c6310 con 0x7f0cd015cbf0 2024-01-24T17:45:24.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.714+0000 7f0cc6a00640 1 -- 172.21.15.39:0/1056082424 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0cc800abf0 con 0x7f0cd015cbf0 2024-01-24T17:45:24.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.715+0000 7f0cc6a00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ca806a430 0x7f0ca806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:24.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.715+0000 7f0cc6a00640 1 -- 172.21.15.39:0/1056082424 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cc8088aa0 con 0x7f0cd015cbf0 2024-01-24T17:45:24.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.716+0000 7f0ccea00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ca806a430 0x7f0ca806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:24.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.717+0000 7f0ccea00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ca806a430 0x7f0ca806c8f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0001000 tx=0x7f0cc0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:24.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.719+0000 7f0cc6a00640 1 -- 172.21.15.39:0/1056082424 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0cd00c6310 con 0x7f0cd015cbf0 2024-01-24T17:45:24.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.909+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cd00cdd20 con 0x7f0ca806a430 2024-01-24T17:45:24.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.932+0000 7f0cc6a00640 1 -- 172.21.15.39:0/1056082424 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287083 (secure 0 0 0) 0x7f0cc00012c0 con 0x7f0ca806a430 2024-01-24T17:45:24.936 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:24.938 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:45:24.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.936+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ca806a430 msgr2=0x7f0ca806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.936+0000 7f0ccfe00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ca806a430 0x7f0ca806c8f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0001000 tx=0x7f0cc0009040 comp rx=0 tx=0).stop 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.936+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd015cbf0 msgr2=0x7f0cd00cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.936+0000 7f0ccfe00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd015cbf0 0x7f0cd00cba80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800c7b0 tx=0x7f0cc800cc70 comp rx=0 tx=0).stop 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.937+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 shutdown_connections 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.937+0000 7f0ccfe00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0ca806a430 0x7f0ca806c8f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.937+0000 7f0ccfe00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cd015cbf0 0x7f0cd00cba80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.937+0000 7f0ccfe00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0cd00d0da0 0x7f0cd0150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.937+0000 7f0ccfe00640 1 --2- 172.21.15.39:0/1056082424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cd00d03b0 0x7f0cd01552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.937+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 >> 172.21.15.39:0/1056082424 conn(0x7f0cd009bb90 msgr2=0x7f0cd006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.937+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 shutdown_connections 2024-01-24T17:45:24.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:24.937+0000 7f0ccfe00640 1 -- 172.21.15.39:0/1056082424 wait complete. 2024-01-24T17:45:24.958 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-01-24T17:45:24.294298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252056,"kb_used_data":7144,"kb_used_omap":13,"kb_used_meta":244722,"kb_avail":754722664,"statfs":{"total":773094113280,"available":772836007936,"internally_reserved":0,"allocated":7315456,"data_stored":3751980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":250595416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.408601"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396720+0000","last_change":"2024-01-24T17:45:18.428192+0000","last_active":"2024-01-24T17:45:19.396720+0000","last_peered":"2024-01-24T17:45:19.396720+0000","last_clean":"2024-01-24T17:45:19.396720+0000","last_became_active":"2024-01-24T17:45:18.421242+0000","last_became_peered":"2024-01-24T17:45:18.421242+0000","last_unstale":"2024-01-24T17:45:19.396720+0000","last_undegraded":"2024-01-24T17:45:19.396720+0000","last_fullsized":"2024-01-24T17:45:19.396720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:04.171678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397973+0000","last_change":"2024-01-24T17:45:18.434096+0000","last_active":"2024-01-24T17:45:19.397973+0000","last_peered":"2024-01-24T17:45:19.397973+0000","last_clean":"2024-01-24T17:45:19.397973+0000","last_became_active":"2024-01-24T17:45:18.433813+0000","last_became_peered":"2024-01-24T17:45:18.433813+0000","last_unstale":"2024-01-24T17:45:19.397973+0000","last_undegraded":"2024-01-24T17:45:19.397973+0000","last_fullsized":"2024-01-24T17:45:19.397973+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:14:51.606606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631230+0000","last_change":"2024-01-24T17:45:18.416822+0000","last_active":"2024-01-24T17:45:19.631230+0000","last_peered":"2024-01-24T17:45:19.631230+0000","last_clean":"2024-01-24T17:45:19.631230+0000","last_became_active":"2024-01-24T17:45:18.416525+0000","last_became_peered":"2024-01-24T17:45:18.416525+0000","last_unstale":"2024-01-24T17:45:19.631230+0000","last_undegraded":"2024-01-24T17:45:19.631230+0000","last_fullsized":"2024-01-24T17:45:19.631230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:58:02.429970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.632549+0000","last_change":"2024-01-24T17:45:18.410425+0000","last_active":"2024-01-24T17:45:19.632549+0000","last_peered":"2024-01-24T17:45:19.632549+0000","last_clean":"2024-01-24T17:45:19.632549+0000","last_became_active":"2024-01-24T17:45:18.408436+0000","last_became_peered":"2024-01-24T17:45:18.408436+0000","last_unstale":"2024-01-24T17:45:19.632549+0000","last_undegraded":"2024-01-24T17:45:19.632549+0000","last_fullsized":"2024-01-24T17:45:19.632549+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:11:53.119290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396552+0000","last_change":"2024-01-24T17:45:18.419250+0000","last_active":"2024-01-24T17:45:19.396552+0000","last_peered":"2024-01-24T17:45:19.396552+0000","last_clean":"2024-01-24T17:45:19.396552+0000","last_became_active":"2024-01-24T17:45:18.418924+0000","last_became_peered":"2024-01-24T17:45:18.418924+0000","last_unstale":"2024-01-24T17:45:19.396552+0000","last_undegraded":"2024-01-24T17:45:19.396552+0000","last_fullsized":"2024-01-24T17:45:19.396552+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:04.335225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397721+0000","last_change":"2024-01-24T17:45:18.437862+0000","last_active":"2024-01-24T17:45:19.397721+0000","last_peered":"2024-01-24T17:45:19.397721+0000","last_clean":"2024-01-24T17:45:19.397721+0000","last_became_active":"2024-01-24T17:45:18.437581+0000","last_became_peered":"2024-01-24T17:45:18.437581+0000","last_unstale":"2024-01-24T17:45:19.397721+0000","last_undegraded":"2024-01-24T17:45:19.397721+0000","last_fullsized":"2024-01-24T17:45:19.397721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:05:11.682782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631554+0000","last_change":"2024-01-24T17:45:18.408418+0000","last_active":"2024-01-24T17:45:19.631554+0000","last_peered":"2024-01-24T17:45:19.631554+0000","last_clean":"2024-01-24T17:45:19.631554+0000","last_became_active":"2024-01-24T17:45:18.408145+0000","last_became_peered":"2024-01-24T17:45:18.408145+0000","last_unstale":"2024-01-24T17:45:19.631554+0000","last_undegraded":"2024-01-24T17:45:19.631554+0000","last_fullsized":"2024-01-24T17:45:19.631554+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:13:32.005579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629503+0000","last_change":"2024-01-24T17:45:18.416365+0000","last_active":"2024-01-24T17:45:19.629503+0000","last_peered":"2024-01-24T17:45:19.629503+0000","last_clean":"2024-01-24T17:45:19.629503+0000","last_became_active":"2024-01-24T17:45:18.416080+0000","last_became_peered":"2024-01-24T17:45:18.416080+0000","last_unstale":"2024-01-24T17:45:19.629503+0000","last_undegraded":"2024-01-24T17:45:19.629503+0000","last_fullsized":"2024-01-24T17:45:19.629503+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:13:50.434214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.395146+0000","last_change":"2024-01-24T17:45:18.419555+0000","last_active":"2024-01-24T17:45:19.395146+0000","last_peered":"2024-01-24T17:45:19.395146+0000","last_clean":"2024-01-24T17:45:19.395146+0000","last_became_active":"2024-01-24T17:45:18.418203+0000","last_became_peered":"2024-01-24T17:45:18.418203+0000","last_unstale":"2024-01-24T17:45:19.395146+0000","last_undegraded":"2024-01-24T17:45:19.395146+0000","last_fullsized":"2024-01-24T17:45:19.395146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:54:57.788190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397031+0000","last_change":"2024-01-24T17:45:18.414408+0000","last_active":"2024-01-24T17:45:19.397031+0000","last_peered":"2024-01-24T17:45:19.397031+0000","last_clean":"2024-01-24T17:45:19.397031+0000","last_became_active":"2024-01-24T17:45:18.413540+0000","last_became_peered":"2024-01-24T17:45:18.413540+0000","last_unstale":"2024-01-24T17:45:19.397031+0000","last_undegraded":"2024-01-24T17:45:19.397031+0000","last_fullsized":"2024-01-24T17:45:19.397031+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:49:33.451268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628310+0000","last_change":"2024-01-24T17:45:18.433178+0000","last_active":"2024-01-24T17:45:19.628310+0000","last_peered":"2024-01-24T17:45:19.628310+0000","last_clean":"2024-01-24T17:45:19.628310+0000","last_became_active":"2024-01-24T17:45:18.432838+0000","last_became_peered":"2024-01-24T17:45:18.432838+0000","last_unstale":"2024-01-24T17:45:19.628310+0000","last_undegraded":"2024-01-24T17:45:19.628310+0000","last_fullsized":"2024-01-24T17:45:19.628310+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:13.563526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.395285+0000","last_change":"2024-01-24T17:45:18.412494+0000","last_active":"2024-01-24T17:45:19.395285+0000","last_peered":"2024-01-24T17:45:19.395285+0000","last_clean":"2024-01-24T17:45:19.395285+0000","last_became_active":"2024-01-24T17:45:18.412189+0000","last_became_peered":"2024-01-24T17:45:18.412189+0000","last_unstale":"2024-01-24T17:45:19.395285+0000","last_undegraded":"2024-01-24T17:45:19.395285+0000","last_fullsized":"2024-01-24T17:45:19.395285+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:05:03.201083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630518+0000","last_change":"2024-01-24T17:45:18.419893+0000","last_active":"2024-01-24T17:45:19.630518+0000","last_peered":"2024-01-24T17:45:19.630518+0000","last_clean":"2024-01-24T17:45:19.630518+0000","last_became_active":"2024-01-24T17:45:18.419585+0000","last_became_peered":"2024-01-24T17:45:18.419585+0000","last_unstale":"2024-01-24T17:45:19.630518+0000","last_undegraded":"2024-01-24T17:45:19.630518+0000","last_fullsized":"2024-01-24T17:45:19.630518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:04.171678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628827+0000","last_change":"2024-01-24T17:45:18.421833+0000","last_active":"2024-01-24T17:45:19.628827+0000","last_peered":"2024-01-24T17:45:19.628827+0000","last_clean":"2024-01-24T17:45:19.628827+0000","last_became_active":"2024-01-24T17:45:18.421603+0000","last_became_peered":"2024-01-24T17:45:18.421603+0000","last_unstale":"2024-01-24T17:45:19.628827+0000","last_undegraded":"2024-01-24T17:45:19.628827+0000","last_fullsized":"2024-01-24T17:45:19.628827+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:08:30.981439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630163+0000","last_change":"2024-01-24T17:45:18.404163+0000","last_active":"2024-01-24T17:45:19.630163+0000","last_peered":"2024-01-24T17:45:19.630163+0000","last_clean":"2024-01-24T17:45:19.630163+0000","last_became_active":"2024-01-24T17:45:18.403613+0000","last_became_peered":"2024-01-24T17:45:18.403613+0000","last_unstale":"2024-01-24T17:45:19.630163+0000","last_undegraded":"2024-01-24T17:45:19.630163+0000","last_fullsized":"2024-01-24T17:45:19.630163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:45.817758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.395440+0000","last_change":"2024-01-24T17:45:18.424370+0000","last_active":"2024-01-24T17:45:19.395440+0000","last_peered":"2024-01-24T17:45:19.395440+0000","last_clean":"2024-01-24T17:45:19.395440+0000","last_became_active":"2024-01-24T17:45:18.419616+0000","last_became_peered":"2024-01-24T17:45:18.419616+0000","last_unstale":"2024-01-24T17:45:19.395440+0000","last_undegraded":"2024-01-24T17:45:19.395440+0000","last_fullsized":"2024-01-24T17:45:19.395440+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:45:55.342357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397197+0000","last_change":"2024-01-24T17:45:18.414883+0000","last_active":"2024-01-24T17:45:19.397197+0000","last_peered":"2024-01-24T17:45:19.397197+0000","last_clean":"2024-01-24T17:45:19.397197+0000","last_became_active":"2024-01-24T17:45:18.413671+0000","last_became_peered":"2024-01-24T17:45:18.413671+0000","last_unstale":"2024-01-24T17:45:19.397197+0000","last_undegraded":"2024-01-24T17:45:19.397197+0000","last_fullsized":"2024-01-24T17:45:19.397197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:13.563526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.632296+0000","last_change":"2024-01-24T17:45:18.423276+0000","last_active":"2024-01-24T17:45:19.632296+0000","last_peered":"2024-01-24T17:45:19.632296+0000","last_clean":"2024-01-24T17:45:19.632296+0000","last_became_active":"2024-01-24T17:45:18.423068+0000","last_became_peered":"2024-01-24T17:45:18.423068+0000","last_unstale":"2024-01-24T17:45:19.632296+0000","last_undegraded":"2024-01-24T17:45:19.632296+0000","last_fullsized":"2024-01-24T17:45:19.632296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:33:02.565560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397557+0000","last_change":"2024-01-24T17:45:18.438606+0000","last_active":"2024-01-24T17:45:19.397557+0000","last_peered":"2024-01-24T17:45:19.397557+0000","last_clean":"2024-01-24T17:45:19.397557+0000","last_became_active":"2024-01-24T17:45:18.438232+0000","last_became_peered":"2024-01-24T17:45:18.438232+0000","last_unstale":"2024-01-24T17:45:19.397557+0000","last_undegraded":"2024-01-24T17:45:19.397557+0000","last_fullsized":"2024-01-24T17:45:19.397557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:46:56.750634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628816+0000","last_change":"2024-01-24T17:45:18.434304+0000","last_active":"2024-01-24T17:45:19.628816+0000","last_peered":"2024-01-24T17:45:19.628816+0000","last_clean":"2024-01-24T17:45:19.628816+0000","last_became_active":"2024-01-24T17:45:18.432897+0000","last_became_peered":"2024-01-24T17:45:18.432897+0000","last_unstale":"2024-01-24T17:45:19.628816+0000","last_undegraded":"2024-01-24T17:45:19.628816+0000","last_fullsized":"2024-01-24T17:45:19.628816+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:46:56.750634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630570+0000","last_change":"2024-01-24T17:45:18.412717+0000","last_active":"2024-01-24T17:45:19.630570+0000","last_peered":"2024-01-24T17:45:19.630570+0000","last_clean":"2024-01-24T17:45:19.630570+0000","last_became_active":"2024-01-24T17:45:18.412447+0000","last_became_peered":"2024-01-24T17:45:18.412447+0000","last_unstale":"2024-01-24T17:45:19.630570+0000","last_undegraded":"2024-01-24T17:45:19.630570+0000","last_fullsized":"2024-01-24T17:45:19.630570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:13.563526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628295+0000","last_change":"2024-01-24T17:45:18.426865+0000","last_active":"2024-01-24T17:45:19.628295+0000","last_peered":"2024-01-24T17:45:19.628295+0000","last_clean":"2024-01-24T17:45:19.628295+0000","last_became_active":"2024-01-24T17:45:18.426589+0000","last_became_peered":"2024-01-24T17:45:18.426589+0000","last_unstale":"2024-01-24T17:45:19.628295+0000","last_undegraded":"2024-01-24T17:45:19.628295+0000","last_fullsized":"2024-01-24T17:45:19.628295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:08:30.981439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631790+0000","last_change":"2024-01-24T17:45:18.426201+0000","last_active":"2024-01-24T17:45:19.631790+0000","last_peered":"2024-01-24T17:45:19.631790+0000","last_clean":"2024-01-24T17:45:19.631790+0000","last_became_active":"2024-01-24T17:45:18.425939+0000","last_became_peered":"2024-01-24T17:45:18.425939+0000","last_unstale":"2024-01-24T17:45:19.631790+0000","last_undegraded":"2024-01-24T17:45:19.631790+0000","last_fullsized":"2024-01-24T17:45:19.631790+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:47:11.363408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394549+0000","last_change":"2024-01-24T17:45:18.420014+0000","last_active":"2024-01-24T17:45:19.394549+0000","last_peered":"2024-01-24T17:45:19.394549+0000","last_clean":"2024-01-24T17:45:19.394549+0000","last_became_active":"2024-01-24T17:45:18.419578+0000","last_became_peered":"2024-01-24T17:45:18.419578+0000","last_unstale":"2024-01-24T17:45:19.394549+0000","last_undegraded":"2024-01-24T17:45:19.394549+0000","last_fullsized":"2024-01-24T17:45:19.394549+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:45:46.357529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.398218+0000","last_change":"2024-01-24T17:45:18.435364+0000","last_active":"2024-01-24T17:45:19.398218+0000","last_peered":"2024-01-24T17:45:19.398218+0000","last_clean":"2024-01-24T17:45:19.398218+0000","last_became_active":"2024-01-24T17:45:18.435072+0000","last_became_peered":"2024-01-24T17:45:18.435072+0000","last_unstale":"2024-01-24T17:45:19.398218+0000","last_undegraded":"2024-01-24T17:45:19.398218+0000","last_fullsized":"2024-01-24T17:45:19.398218+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:08:30.981439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631466+0000","last_change":"2024-01-24T17:45:18.399862+0000","last_active":"2024-01-24T17:45:19.631466+0000","last_peered":"2024-01-24T17:45:19.631466+0000","last_clean":"2024-01-24T17:45:19.631466+0000","last_became_active":"2024-01-24T17:45:18.399525+0000","last_became_peered":"2024-01-24T17:45:18.399525+0000","last_unstale":"2024-01-24T17:45:19.631466+0000","last_undegraded":"2024-01-24T17:45:19.631466+0000","last_fullsized":"2024-01-24T17:45:19.631466+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:13.657421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397944+0000","last_change":"2024-01-24T17:45:18.434414+0000","last_active":"2024-01-24T17:45:19.397944+0000","last_peered":"2024-01-24T17:45:19.397944+0000","last_clean":"2024-01-24T17:45:19.397944+0000","last_became_active":"2024-01-24T17:45:18.433941+0000","last_became_peered":"2024-01-24T17:45:18.433941+0000","last_unstale":"2024-01-24T17:45:19.397944+0000","last_undegraded":"2024-01-24T17:45:19.397944+0000","last_fullsized":"2024-01-24T17:45:19.397944+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:49:33.451268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630068+0000","last_change":"2024-01-24T17:45:18.407428+0000","last_active":"2024-01-24T17:45:19.630068+0000","last_peered":"2024-01-24T17:45:19.630068+0000","last_clean":"2024-01-24T17:45:19.630068+0000","last_became_active":"2024-01-24T17:45:18.407022+0000","last_became_peered":"2024-01-24T17:45:18.407022+0000","last_unstale":"2024-01-24T17:45:19.630068+0000","last_undegraded":"2024-01-24T17:45:19.630068+0000","last_fullsized":"2024-01-24T17:45:19.630068+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:01.170973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629935+0000","last_change":"2024-01-24T17:45:18.414712+0000","last_active":"2024-01-24T17:45:19.629935+0000","last_peered":"2024-01-24T17:45:19.629935+0000","last_clean":"2024-01-24T17:45:19.629935+0000","last_became_active":"2024-01-24T17:45:18.414419+0000","last_became_peered":"2024-01-24T17:45:18.414419+0000","last_unstale":"2024-01-24T17:45:19.629935+0000","last_undegraded":"2024-01-24T17:45:19.629935+0000","last_fullsized":"2024-01-24T17:45:19.629935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:58:02.429970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633006+0000","last_change":"2024-01-24T17:45:18.400760+0000","last_active":"2024-01-24T17:45:19.633006+0000","last_peered":"2024-01-24T17:45:19.633006+0000","last_clean":"2024-01-24T17:45:19.633006+0000","last_became_active":"2024-01-24T17:45:18.400461+0000","last_became_peered":"2024-01-24T17:45:18.400461+0000","last_unstale":"2024-01-24T17:45:19.633006+0000","last_undegraded":"2024-01-24T17:45:19.633006+0000","last_fullsized":"2024-01-24T17:45:19.633006+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:05:01.096215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396455+0000","last_change":"2024-01-24T17:45:18.429868+0000","last_active":"2024-01-24T17:45:19.396455+0000","last_peered":"2024-01-24T17:45:19.396455+0000","last_clean":"2024-01-24T17:45:19.396455+0000","last_became_active":"2024-01-24T17:45:18.429573+0000","last_became_peered":"2024-01-24T17:45:18.429573+0000","last_unstale":"2024-01-24T17:45:19.396455+0000","last_undegraded":"2024-01-24T17:45:19.396455+0000","last_fullsized":"2024-01-24T17:45:19.396455+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:04.335225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628022+0000","last_change":"2024-01-24T17:45:18.431055+0000","last_active":"2024-01-24T17:45:19.628022+0000","last_peered":"2024-01-24T17:45:19.628022+0000","last_clean":"2024-01-24T17:45:19.628022+0000","last_became_active":"2024-01-24T17:45:18.430791+0000","last_became_peered":"2024-01-24T17:45:18.430791+0000","last_unstale":"2024-01-24T17:45:19.628022+0000","last_undegraded":"2024-01-24T17:45:19.628022+0000","last_fullsized":"2024-01-24T17:45:19.628022+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:05:11.682782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397283+0000","last_change":"2024-01-24T17:45:18.414027+0000","last_active":"2024-01-24T17:45:19.397283+0000","last_peered":"2024-01-24T17:45:19.397283+0000","last_clean":"2024-01-24T17:45:19.397283+0000","last_became_active":"2024-01-24T17:45:18.413452+0000","last_became_peered":"2024-01-24T17:45:18.413452+0000","last_unstale":"2024-01-24T17:45:19.397283+0000","last_undegraded":"2024-01-24T17:45:19.397283+0000","last_fullsized":"2024-01-24T17:45:19.397283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:39:17.384315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"62'4","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.434749+0000","last_change":"2024-01-24T17:45:17.919800+0000","last_active":"2024-01-24T17:45:18.434749+0000","last_peered":"2024-01-24T17:45:18.434749+0000","last_clean":"2024-01-24T17:45:18.434749+0000","last_became_active":"2024-01-24T17:45:17.918994+0000","last_became_peered":"2024-01-24T17:45:17.918994+0000","last_unstale":"2024-01-24T17:45:18.434749+0000","last_undegraded":"2024-01-24T17:45:18.434749+0000","last_fullsized":"2024-01-24T17:45:18.434749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397389+0000","last_change":"2024-01-24T17:45:18.437398+0000","last_active":"2024-01-24T17:45:19.397389+0000","last_peered":"2024-01-24T17:45:19.397389+0000","last_clean":"2024-01-24T17:45:19.397389+0000","last_became_active":"2024-01-24T17:45:18.437095+0000","last_became_peered":"2024-01-24T17:45:18.437095+0000","last_unstale":"2024-01-24T17:45:19.397389+0000","last_undegraded":"2024-01-24T17:45:19.397389+0000","last_fullsized":"2024-01-24T17:45:19.397389+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:13.657421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.436986+0000","last_change":"2024-01-24T17:45:17.919211+0000","last_active":"2024-01-24T17:45:18.436986+0000","last_peered":"2024-01-24T17:45:18.436986+0000","last_clean":"2024-01-24T17:45:18.436986+0000","last_became_active":"2024-01-24T17:45:17.918942+0000","last_became_peered":"2024-01-24T17:45:17.918942+0000","last_unstale":"2024-01-24T17:45:18.436986+0000","last_undegraded":"2024-01-24T17:45:18.436986+0000","last_fullsized":"2024-01-24T17:45:18.436986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:49:04.288996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630291+0000","last_change":"2024-01-24T17:45:18.424280+0000","last_active":"2024-01-24T17:45:19.630291+0000","last_peered":"2024-01-24T17:45:19.630291+0000","last_clean":"2024-01-24T17:45:19.630291+0000","last_became_active":"2024-01-24T17:45:18.422789+0000","last_became_peered":"2024-01-24T17:45:18.422789+0000","last_unstale":"2024-01-24T17:45:19.630291+0000","last_undegraded":"2024-01-24T17:45:19.630291+0000","last_fullsized":"2024-01-24T17:45:19.630291+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:23.267943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.438612+0000","last_change":"2024-01-24T17:45:17.922512+0000","last_active":"2024-01-24T17:45:18.438612+0000","last_peered":"2024-01-24T17:45:18.438612+0000","last_clean":"2024-01-24T17:45:18.438612+0000","last_became_active":"2024-01-24T17:45:17.920772+0000","last_became_peered":"2024-01-24T17:45:17.920772+0000","last_unstale":"2024-01-24T17:45:18.438612+0000","last_undegraded":"2024-01-24T17:45:18.438612+0000","last_fullsized":"2024-01-24T17:45:18.438612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:05:02.184752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396198+0000","last_change":"2024-01-24T17:45:18.420475+0000","last_active":"2024-01-24T17:45:19.396198+0000","last_peered":"2024-01-24T17:45:19.396198+0000","last_clean":"2024-01-24T17:45:19.396198+0000","last_became_active":"2024-01-24T17:45:18.419965+0000","last_became_peered":"2024-01-24T17:45:18.419965+0000","last_unstale":"2024-01-24T17:45:19.396198+0000","last_undegraded":"2024-01-24T17:45:19.396198+0000","last_fullsized":"2024-01-24T17:45:19.396198+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:46:56.750634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.429670+0000","last_change":"2024-01-24T17:45:17.925679+0000","last_active":"2024-01-24T17:45:18.429670+0000","last_peered":"2024-01-24T17:45:18.429670+0000","last_clean":"2024-01-24T17:45:18.429670+0000","last_became_active":"2024-01-24T17:45:17.925384+0000","last_became_peered":"2024-01-24T17:45:17.925384+0000","last_unstale":"2024-01-24T17:45:18.429670+0000","last_undegraded":"2024-01-24T17:45:18.429670+0000","last_fullsized":"2024-01-24T17:45:18.429670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:41:31.769012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629796+0000","last_change":"2024-01-24T17:45:18.429943+0000","last_active":"2024-01-24T17:45:19.629796+0000","last_peered":"2024-01-24T17:45:19.629796+0000","last_clean":"2024-01-24T17:45:19.629796+0000","last_became_active":"2024-01-24T17:45:18.429477+0000","last_became_peered":"2024-01-24T17:45:18.429477+0000","last_unstale":"2024-01-24T17:45:19.629796+0000","last_undegraded":"2024-01-24T17:45:19.629796+0000","last_fullsized":"2024-01-24T17:45:19.629796+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:32:36.644421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.440256+0000","last_change":"2024-01-24T17:45:17.923006+0000","last_active":"2024-01-24T17:45:18.440256+0000","last_peered":"2024-01-24T17:45:18.440256+0000","last_clean":"2024-01-24T17:45:18.440256+0000","last_became_active":"2024-01-24T17:45:17.922289+0000","last_became_peered":"2024-01-24T17:45:17.922289+0000","last_unstale":"2024-01-24T17:45:18.440256+0000","last_undegraded":"2024-01-24T17:45:18.440256+0000","last_fullsized":"2024-01-24T17:45:18.440256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:16.899578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628295+0000","last_change":"2024-01-24T17:45:18.428697+0000","last_active":"2024-01-24T17:45:19.628295+0000","last_peered":"2024-01-24T17:45:19.628295+0000","last_clean":"2024-01-24T17:45:19.628295+0000","last_became_active":"2024-01-24T17:45:18.428426+0000","last_became_peered":"2024-01-24T17:45:18.428426+0000","last_unstale":"2024-01-24T17:45:19.628295+0000","last_undegraded":"2024-01-24T17:45:19.628295+0000","last_fullsized":"2024-01-24T17:45:19.628295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:49:33.451268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.435349+0000","last_change":"2024-01-24T17:45:17.923077+0000","last_active":"2024-01-24T17:45:18.435349+0000","last_peered":"2024-01-24T17:45:18.435349+0000","last_clean":"2024-01-24T17:45:18.435349+0000","last_became_active":"2024-01-24T17:45:17.922774+0000","last_became_peered":"2024-01-24T17:45:17.922774+0000","last_unstale":"2024-01-24T17:45:18.435349+0000","last_undegraded":"2024-01-24T17:45:18.435349+0000","last_fullsized":"2024-01-24T17:45:18.435349+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:26.416880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628862+0000","last_change":"2024-01-24T17:45:18.418235+0000","last_active":"2024-01-24T17:45:19.628862+0000","last_peered":"2024-01-24T17:45:19.628862+0000","last_clean":"2024-01-24T17:45:19.628862+0000","last_became_active":"2024-01-24T17:45:18.417811+0000","last_became_peered":"2024-01-24T17:45:18.417811+0000","last_unstale":"2024-01-24T17:45:19.628862+0000","last_undegraded":"2024-01-24T17:45:19.628862+0000","last_fullsized":"2024-01-24T17:45:19.628862+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:19:01.175473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.408175+0000","last_change":"2024-01-24T17:45:17.912250+0000","last_active":"2024-01-24T17:45:18.408175+0000","last_peered":"2024-01-24T17:45:18.408175+0000","last_clean":"2024-01-24T17:45:18.408175+0000","last_became_active":"2024-01-24T17:45:17.911960+0000","last_became_peered":"2024-01-24T17:45:17.911960+0000","last_unstale":"2024-01-24T17:45:18.408175+0000","last_undegraded":"2024-01-24T17:45:18.408175+0000","last_fullsized":"2024-01-24T17:45:18.408175+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:34:38.987748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.395540+0000","last_change":"2024-01-24T17:45:18.410406+0000","last_active":"2024-01-24T17:45:19.395540+0000","last_peered":"2024-01-24T17:45:19.395540+0000","last_clean":"2024-01-24T17:45:19.395540+0000","last_became_active":"2024-01-24T17:45:18.409406+0000","last_became_peered":"2024-01-24T17:45:18.409406+0000","last_unstale":"2024-01-24T17:45:19.395540+0000","last_undegraded":"2024-01-24T17:45:19.395540+0000","last_fullsized":"2024-01-24T17:45:19.395540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:45.817758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.390230+0000","last_change":"2024-01-24T17:45:17.916659+0000","last_active":"2024-01-24T17:45:18.390230+0000","last_peered":"2024-01-24T17:45:18.390230+0000","last_clean":"2024-01-24T17:45:18.390230+0000","last_became_active":"2024-01-24T17:45:17.916394+0000","last_became_peered":"2024-01-24T17:45:17.916394+0000","last_unstale":"2024-01-24T17:45:18.390230+0000","last_undegraded":"2024-01-24T17:45:18.390230+0000","last_fullsized":"2024-01-24T17:45:18.390230+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631547+0000","last_change":"2024-01-24T17:45:18.431208+0000","last_active":"2024-01-24T17:45:19.631547+0000","last_peered":"2024-01-24T17:45:19.631547+0000","last_clean":"2024-01-24T17:45:19.631547+0000","last_became_active":"2024-01-24T17:45:18.430737+0000","last_became_peered":"2024-01-24T17:45:18.430737+0000","last_unstale":"2024-01-24T17:45:19.631547+0000","last_undegraded":"2024-01-24T17:45:19.631547+0000","last_fullsized":"2024-01-24T17:45:19.631547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:04.335225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.409308+0000","last_change":"2024-01-24T17:45:17.912678+0000","last_active":"2024-01-24T17:45:18.409308+0000","last_peered":"2024-01-24T17:45:18.409308+0000","last_clean":"2024-01-24T17:45:18.409308+0000","last_became_active":"2024-01-24T17:45:17.912008+0000","last_became_peered":"2024-01-24T17:45:17.912008+0000","last_unstale":"2024-01-24T17:45:18.409308+0000","last_undegraded":"2024-01-24T17:45:18.409308+0000","last_fullsized":"2024-01-24T17:45:18.409308+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:30:30.289664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633552+0000","last_change":"2024-01-24T17:45:18.423553+0000","last_active":"2024-01-24T17:45:19.633552+0000","last_peered":"2024-01-24T17:45:19.633552+0000","last_clean":"2024-01-24T17:45:19.633552+0000","last_became_active":"2024-01-24T17:45:18.420777+0000","last_became_peered":"2024-01-24T17:45:18.420777+0000","last_unstale":"2024-01-24T17:45:19.633552+0000","last_undegraded":"2024-01-24T17:45:19.633552+0000","last_fullsized":"2024-01-24T17:45:19.633552+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:23.692981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.398732+0000","last_change":"2024-01-24T17:45:17.922147+0000","last_active":"2024-01-24T17:45:18.398732+0000","last_peered":"2024-01-24T17:45:18.398732+0000","last_clean":"2024-01-24T17:45:18.398732+0000","last_became_active":"2024-01-24T17:45:17.921599+0000","last_became_peered":"2024-01-24T17:45:17.921599+0000","last_unstale":"2024-01-24T17:45:18.398732+0000","last_undegraded":"2024-01-24T17:45:18.398732+0000","last_fullsized":"2024-01-24T17:45:18.398732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:12.547194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630533+0000","last_change":"2024-01-24T17:45:18.426483+0000","last_active":"2024-01-24T17:45:19.630533+0000","last_peered":"2024-01-24T17:45:19.630533+0000","last_clean":"2024-01-24T17:45:19.630533+0000","last_became_active":"2024-01-24T17:45:18.420858+0000","last_became_peered":"2024-01-24T17:45:18.420858+0000","last_unstale":"2024-01-24T17:45:19.630533+0000","last_undegraded":"2024-01-24T17:45:19.630533+0000","last_fullsized":"2024-01-24T17:45:19.630533+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:22:32.072368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"62'3","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.428360+0000","last_change":"2024-01-24T17:45:17.925319+0000","last_active":"2024-01-24T17:45:18.428360+0000","last_peered":"2024-01-24T17:45:18.428360+0000","last_clean":"2024-01-24T17:45:18.428360+0000","last_became_active":"2024-01-24T17:45:17.925026+0000","last_became_peered":"2024-01-24T17:45:17.925026+0000","last_unstale":"2024-01-24T17:45:18.428360+0000","last_undegraded":"2024-01-24T17:45:18.428360+0000","last_fullsized":"2024-01-24T17:45:18.428360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:22.676649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.632142+0000","last_change":"2024-01-24T17:45:18.409641+0000","last_active":"2024-01-24T17:45:19.632142+0000","last_peered":"2024-01-24T17:45:19.632142+0000","last_clean":"2024-01-24T17:45:19.632142+0000","last_became_active":"2024-01-24T17:45:18.409314+0000","last_became_peered":"2024-01-24T17:45:18.409314+0000","last_unstale":"2024-01-24T17:45:19.632142+0000","last_undegraded":"2024-01-24T17:45:19.632142+0000","last_fullsized":"2024-01-24T17:45:19.632142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:42:55.776505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"62'6","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.441892+0000","last_change":"2024-01-24T17:45:17.924919+0000","last_active":"2024-01-24T17:45:18.441892+0000","last_peered":"2024-01-24T17:45:18.441892+0000","last_clean":"2024-01-24T17:45:18.441892+0000","last_became_active":"2024-01-24T17:45:17.924653+0000","last_became_peered":"2024-01-24T17:45:17.924653+0000","last_unstale":"2024-01-24T17:45:18.441892+0000","last_undegraded":"2024-01-24T17:45:18.441892+0000","last_fullsized":"2024-01-24T17:45:18.441892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394970+0000","last_change":"2024-01-24T17:45:18.424098+0000","last_active":"2024-01-24T17:45:19.394970+0000","last_peered":"2024-01-24T17:45:19.394970+0000","last_clean":"2024-01-24T17:45:19.394970+0000","last_became_active":"2024-01-24T17:45:18.423052+0000","last_became_peered":"2024-01-24T17:45:18.423052+0000","last_unstale":"2024-01-24T17:45:19.394970+0000","last_undegraded":"2024-01-24T17:45:19.394970+0000","last_fullsized":"2024-01-24T17:45:19.394970+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:18.926779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.440287+0000","last_change":"2024-01-24T17:45:17.921223+0000","last_active":"2024-01-24T17:45:18.440287+0000","last_peered":"2024-01-24T17:45:18.440287+0000","last_clean":"2024-01-24T17:45:18.440287+0000","last_became_active":"2024-01-24T17:45:17.919530+0000","last_became_peered":"2024-01-24T17:45:17.919530+0000","last_unstale":"2024-01-24T17:45:18.440287+0000","last_undegraded":"2024-01-24T17:45:18.440287+0000","last_fullsized":"2024-01-24T17:45:18.440287+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:44.801426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629876+0000","last_change":"2024-01-24T17:45:18.414202+0000","last_active":"2024-01-24T17:45:19.629876+0000","last_peered":"2024-01-24T17:45:19.629876+0000","last_clean":"2024-01-24T17:45:19.629876+0000","last_became_active":"2024-01-24T17:45:18.413959+0000","last_became_peered":"2024-01-24T17:45:18.413959+0000","last_unstale":"2024-01-24T17:45:19.629876+0000","last_undegraded":"2024-01-24T17:45:19.629876+0000","last_fullsized":"2024-01-24T17:45:19.629876+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:29:15.124598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.396399+0000","last_change":"2024-01-24T17:45:17.916370+0000","last_active":"2024-01-24T17:45:18.396399+0000","last_peered":"2024-01-24T17:45:18.396399+0000","last_clean":"2024-01-24T17:45:18.396399+0000","last_became_active":"2024-01-24T17:45:17.916091+0000","last_became_peered":"2024-01-24T17:45:17.916091+0000","last_unstale":"2024-01-24T17:45:18.396399+0000","last_undegraded":"2024-01-24T17:45:18.396399+0000","last_fullsized":"2024-01-24T17:45:18.396399+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:00.154641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394743+0000","last_change":"2024-01-24T17:45:18.412289+0000","last_active":"2024-01-24T17:45:19.394743+0000","last_peered":"2024-01-24T17:45:19.394743+0000","last_clean":"2024-01-24T17:45:19.394743+0000","last_became_active":"2024-01-24T17:45:18.412031+0000","last_became_peered":"2024-01-24T17:45:18.412031+0000","last_unstale":"2024-01-24T17:45:19.394743+0000","last_undegraded":"2024-01-24T17:45:19.394743+0000","last_fullsized":"2024-01-24T17:45:19.394743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:39:31.796345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.396585+0000","last_change":"2024-01-24T17:45:17.914759+0000","last_active":"2024-01-24T17:45:18.396585+0000","last_peered":"2024-01-24T17:45:18.396585+0000","last_clean":"2024-01-24T17:45:18.396585+0000","last_became_active":"2024-01-24T17:45:17.914448+0000","last_became_peered":"2024-01-24T17:45:17.914448+0000","last_unstale":"2024-01-24T17:45:18.396585+0000","last_undegraded":"2024-01-24T17:45:18.396585+0000","last_fullsized":"2024-01-24T17:45:18.396585+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:32.202115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631963+0000","last_change":"2024-01-24T17:45:18.426059+0000","last_active":"2024-01-24T17:45:19.631963+0000","last_peered":"2024-01-24T17:45:19.631963+0000","last_clean":"2024-01-24T17:45:19.631963+0000","last_became_active":"2024-01-24T17:45:18.425160+0000","last_became_peered":"2024-01-24T17:45:18.425160+0000","last_unstale":"2024-01-24T17:45:19.631963+0000","last_undegraded":"2024-01-24T17:45:19.631963+0000","last_fullsized":"2024-01-24T17:45:19.631963+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:54.914651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"62'8","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.431993+0000","last_change":"2024-01-24T17:45:17.919777+0000","last_active":"2024-01-24T17:45:18.431993+0000","last_peered":"2024-01-24T17:45:18.431993+0000","last_clean":"2024-01-24T17:45:18.431993+0000","last_became_active":"2024-01-24T17:45:17.919426+0000","last_became_peered":"2024-01-24T17:45:17.919426+0000","last_unstale":"2024-01-24T17:45:18.431993+0000","last_undegraded":"2024-01-24T17:45:18.431993+0000","last_fullsized":"2024-01-24T17:45:18.431993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:36.108099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633391+0000","last_change":"2024-01-24T17:45:18.431490+0000","last_active":"2024-01-24T17:45:19.633391+0000","last_peered":"2024-01-24T17:45:19.633391+0000","last_clean":"2024-01-24T17:45:19.633391+0000","last_became_active":"2024-01-24T17:45:18.429776+0000","last_became_peered":"2024-01-24T17:45:18.429776+0000","last_unstale":"2024-01-24T17:45:19.633391+0000","last_undegraded":"2024-01-24T17:45:19.633391+0000","last_fullsized":"2024-01-24T17:45:19.633391+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:22:30.250177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.439375+0000","last_change":"2024-01-24T17:45:17.922557+0000","last_active":"2024-01-24T17:45:18.439375+0000","last_peered":"2024-01-24T17:45:18.439375+0000","last_clean":"2024-01-24T17:45:18.439375+0000","last_became_active":"2024-01-24T17:45:17.921938+0000","last_became_peered":"2024-01-24T17:45:17.921938+0000","last_unstale":"2024-01-24T17:45:18.439375+0000","last_undegraded":"2024-01-24T17:45:18.439375+0000","last_fullsized":"2024-01-24T17:45:18.439375+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633829+0000","last_change":"2024-01-24T17:45:18.423319+0000","last_active":"2024-01-24T17:45:19.633829+0000","last_peered":"2024-01-24T17:45:19.633829+0000","last_clean":"2024-01-24T17:45:19.633829+0000","last_became_active":"2024-01-24T17:45:18.420403+0000","last_became_peered":"2024-01-24T17:45:18.420403+0000","last_unstale":"2024-01-24T17:45:19.633829+0000","last_undegraded":"2024-01-24T17:45:19.633829+0000","last_fullsized":"2024-01-24T17:45:19.633829+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:33.218446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.436502+0000","last_change":"2024-01-24T17:45:17.920941+0000","last_active":"2024-01-24T17:45:18.436502+0000","last_peered":"2024-01-24T17:45:18.436502+0000","last_clean":"2024-01-24T17:45:18.436502+0000","last_became_active":"2024-01-24T17:45:17.920695+0000","last_became_peered":"2024-01-24T17:45:17.920695+0000","last_unstale":"2024-01-24T17:45:18.436502+0000","last_undegraded":"2024-01-24T17:45:18.436502+0000","last_fullsized":"2024-01-24T17:45:18.436502+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:03.155346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397112+0000","last_change":"2024-01-24T17:45:18.431983+0000","last_active":"2024-01-24T17:45:19.397112+0000","last_peered":"2024-01-24T17:45:19.397112+0000","last_clean":"2024-01-24T17:45:19.397112+0000","last_became_active":"2024-01-24T17:45:18.431708+0000","last_became_peered":"2024-01-24T17:45:18.431708+0000","last_unstale":"2024-01-24T17:45:19.397112+0000","last_undegraded":"2024-01-24T17:45:19.397112+0000","last_fullsized":"2024-01-24T17:45:19.397112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:13.563526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.438350+0000","last_change":"2024-01-24T17:45:17.922564+0000","last_active":"2024-01-24T17:45:18.438350+0000","last_peered":"2024-01-24T17:45:18.438350+0000","last_clean":"2024-01-24T17:45:18.438350+0000","last_became_active":"2024-01-24T17:45:17.921964+0000","last_became_peered":"2024-01-24T17:45:17.921964+0000","last_unstale":"2024-01-24T17:45:18.438350+0000","last_undegraded":"2024-01-24T17:45:18.438350+0000","last_fullsized":"2024-01-24T17:45:18.438350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.632387+0000","last_change":"2024-01-24T17:45:18.409031+0000","last_active":"2024-01-24T17:45:19.632387+0000","last_peered":"2024-01-24T17:45:19.632387+0000","last_clean":"2024-01-24T17:45:19.632387+0000","last_became_active":"2024-01-24T17:45:18.408776+0000","last_became_peered":"2024-01-24T17:45:18.408776+0000","last_unstale":"2024-01-24T17:45:19.632387+0000","last_undegraded":"2024-01-24T17:45:19.632387+0000","last_fullsized":"2024-01-24T17:45:19.632387+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:10:00.077325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.398988+0000","last_change":"2024-01-24T17:45:17.916140+0000","last_active":"2024-01-24T17:45:18.398988+0000","last_peered":"2024-01-24T17:45:18.398988+0000","last_clean":"2024-01-24T17:45:18.398988+0000","last_became_active":"2024-01-24T17:45:17.915075+0000","last_became_peered":"2024-01-24T17:45:17.915075+0000","last_unstale":"2024-01-24T17:45:18.398988+0000","last_undegraded":"2024-01-24T17:45:18.398988+0000","last_fullsized":"2024-01-24T17:45:18.398988+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:23:00.228501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394675+0000","last_change":"2024-01-24T17:45:18.414477+0000","last_active":"2024-01-24T17:45:19.394675+0000","last_peered":"2024-01-24T17:45:19.394675+0000","last_clean":"2024-01-24T17:45:19.394675+0000","last_became_active":"2024-01-24T17:45:18.414238+0000","last_became_peered":"2024-01-24T17:45:18.414238+0000","last_unstale":"2024-01-24T17:45:19.394675+0000","last_undegraded":"2024-01-24T17:45:19.394675+0000","last_fullsized":"2024-01-24T17:45:19.394675+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:27.433212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.435767+0000","last_change":"2024-01-24T17:45:17.921634+0000","last_active":"2024-01-24T17:45:18.435767+0000","last_peered":"2024-01-24T17:45:18.435767+0000","last_clean":"2024-01-24T17:45:18.435767+0000","last_became_active":"2024-01-24T17:45:17.920658+0000","last_became_peered":"2024-01-24T17:45:17.920658+0000","last_unstale":"2024-01-24T17:45:18.435767+0000","last_undegraded":"2024-01-24T17:45:18.435767+0000","last_fullsized":"2024-01-24T17:45:18.435767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629969+0000","last_change":"2024-01-24T17:45:18.424893+0000","last_active":"2024-01-24T17:45:19.629969+0000","last_peered":"2024-01-24T17:45:19.629969+0000","last_clean":"2024-01-24T17:45:19.629969+0000","last_became_active":"2024-01-24T17:45:18.424613+0000","last_became_peered":"2024-01-24T17:45:18.424613+0000","last_unstale":"2024-01-24T17:45:19.629969+0000","last_undegraded":"2024-01-24T17:45:19.629969+0000","last_fullsized":"2024-01-24T17:45:19.629969+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:25:02.027211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.395532+0000","last_change":"2024-01-24T17:45:17.920125+0000","last_active":"2024-01-24T17:45:18.395532+0000","last_peered":"2024-01-24T17:45:18.395532+0000","last_clean":"2024-01-24T17:45:18.395532+0000","last_became_active":"2024-01-24T17:45:17.919831+0000","last_became_peered":"2024-01-24T17:45:17.919831+0000","last_unstale":"2024-01-24T17:45:18.395532+0000","last_undegraded":"2024-01-24T17:45:18.395532+0000","last_fullsized":"2024-01-24T17:45:18.395532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:03.318893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629603+0000","last_change":"2024-01-24T17:45:18.406061+0000","last_active":"2024-01-24T17:45:19.629603+0000","last_peered":"2024-01-24T17:45:19.629603+0000","last_clean":"2024-01-24T17:45:19.629603+0000","last_became_active":"2024-01-24T17:45:18.404842+0000","last_became_peered":"2024-01-24T17:45:18.404842+0000","last_unstale":"2024-01-24T17:45:19.629603+0000","last_undegraded":"2024-01-24T17:45:19.629603+0000","last_fullsized":"2024-01-24T17:45:19.629603+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:37.124431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.407605+0000","last_change":"2024-01-24T17:45:17.908034+0000","last_active":"2024-01-24T17:45:18.407605+0000","last_peered":"2024-01-24T17:45:18.407605+0000","last_clean":"2024-01-24T17:45:18.407605+0000","last_became_active":"2024-01-24T17:45:17.907755+0000","last_became_peered":"2024-01-24T17:45:17.907755+0000","last_unstale":"2024-01-24T17:45:18.407605+0000","last_undegraded":"2024-01-24T17:45:18.407605+0000","last_fullsized":"2024-01-24T17:45:18.407605+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:21:59.679470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633058+0000","last_change":"2024-01-24T17:45:18.420321+0000","last_active":"2024-01-24T17:45:19.633058+0000","last_peered":"2024-01-24T17:45:19.633058+0000","last_clean":"2024-01-24T17:45:19.633058+0000","last_became_active":"2024-01-24T17:45:18.420048+0000","last_became_peered":"2024-01-24T17:45:18.420048+0000","last_unstale":"2024-01-24T17:45:19.633058+0000","last_undegraded":"2024-01-24T17:45:19.633058+0000","last_fullsized":"2024-01-24T17:45:19.633058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:13:50.434214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.439378+0000","last_change":"2024-01-24T17:45:17.922727+0000","last_active":"2024-01-24T17:45:18.439378+0000","last_peered":"2024-01-24T17:45:18.439378+0000","last_clean":"2024-01-24T17:45:18.439378+0000","last_became_active":"2024-01-24T17:45:17.922288+0000","last_became_peered":"2024-01-24T17:45:17.922288+0000","last_unstale":"2024-01-24T17:45:18.439378+0000","last_undegraded":"2024-01-24T17:45:18.439378+0000","last_fullsized":"2024-01-24T17:45:18.439378+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633893+0000","last_change":"2024-01-24T17:45:18.424353+0000","last_active":"2024-01-24T17:45:19.633893+0000","last_peered":"2024-01-24T17:45:19.633893+0000","last_clean":"2024-01-24T17:45:19.633893+0000","last_became_active":"2024-01-24T17:45:18.424056+0000","last_became_peered":"2024-01-24T17:45:18.424056+0000","last_unstale":"2024-01-24T17:45:19.633893+0000","last_undegraded":"2024-01-24T17:45:19.633893+0000","last_fullsized":"2024-01-24T17:45:19.633893+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:03.006906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.410170+0000","last_change":"2024-01-24T17:45:17.920500+0000","last_active":"2024-01-24T17:45:18.410170+0000","last_peered":"2024-01-24T17:45:18.410170+0000","last_clean":"2024-01-24T17:45:18.410170+0000","last_became_active":"2024-01-24T17:45:17.920207+0000","last_became_peered":"2024-01-24T17:45:17.920207+0000","last_unstale":"2024-01-24T17:45:18.410170+0000","last_undegraded":"2024-01-24T17:45:18.410170+0000","last_fullsized":"2024-01-24T17:45:18.410170+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:17.910448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397363+0000","last_change":"2024-01-24T17:45:18.436099+0000","last_active":"2024-01-24T17:45:19.397363+0000","last_peered":"2024-01-24T17:45:19.397363+0000","last_clean":"2024-01-24T17:45:19.397363+0000","last_became_active":"2024-01-24T17:45:18.435809+0000","last_became_peered":"2024-01-24T17:45:18.435809+0000","last_unstale":"2024-01-24T17:45:19.397363+0000","last_undegraded":"2024-01-24T17:45:19.397363+0000","last_fullsized":"2024-01-24T17:45:19.397363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:04.171678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.408606+0000","last_change":"2024-01-24T17:45:17.923999+0000","last_active":"2024-01-24T17:45:18.408606+0000","last_peered":"2024-01-24T17:45:18.408606+0000","last_clean":"2024-01-24T17:45:18.408606+0000","last_became_active":"2024-01-24T17:45:17.923727+0000","last_became_peered":"2024-01-24T17:45:17.923727+0000","last_unstale":"2024-01-24T17:45:18.408606+0000","last_undegraded":"2024-01-24T17:45:18.408606+0000","last_fullsized":"2024-01-24T17:45:18.408606+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:35:45.998203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630604+0000","last_change":"2024-01-24T17:45:18.406857+0000","last_active":"2024-01-24T17:45:19.630604+0000","last_peered":"2024-01-24T17:45:19.630604+0000","last_clean":"2024-01-24T17:45:19.630604+0000","last_became_active":"2024-01-24T17:45:18.404804+0000","last_became_peered":"2024-01-24T17:45:18.404804+0000","last_unstale":"2024-01-24T17:45:19.630604+0000","last_undegraded":"2024-01-24T17:45:19.630604+0000","last_fullsized":"2024-01-24T17:45:19.630604+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:33.218446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.406718+0000","last_change":"2024-01-24T17:45:17.917767+0000","last_active":"2024-01-24T17:45:18.406718+0000","last_peered":"2024-01-24T17:45:18.406718+0000","last_clean":"2024-01-24T17:45:18.406718+0000","last_became_active":"2024-01-24T17:45:17.917485+0000","last_became_peered":"2024-01-24T17:45:17.917485+0000","last_unstale":"2024-01-24T17:45:18.406718+0000","last_undegraded":"2024-01-24T17:45:18.406718+0000","last_fullsized":"2024-01-24T17:45:18.406718+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396273+0000","last_change":"2024-01-24T17:45:18.437810+0000","last_active":"2024-01-24T17:45:19.396273+0000","last_peered":"2024-01-24T17:45:19.396273+0000","last_clean":"2024-01-24T17:45:19.396273+0000","last_became_active":"2024-01-24T17:45:18.437208+0000","last_became_peered":"2024-01-24T17:45:18.437208+0000","last_unstale":"2024-01-24T17:45:19.396273+0000","last_undegraded":"2024-01-24T17:45:19.396273+0000","last_fullsized":"2024-01-24T17:45:19.396273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:17.915909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.422057+0000","last_change":"2024-01-24T17:45:17.925427+0000","last_active":"2024-01-24T17:45:18.422057+0000","last_peered":"2024-01-24T17:45:18.422057+0000","last_clean":"2024-01-24T17:45:18.422057+0000","last_became_active":"2024-01-24T17:45:17.925156+0000","last_became_peered":"2024-01-24T17:45:17.925156+0000","last_unstale":"2024-01-24T17:45:18.422057+0000","last_undegraded":"2024-01-24T17:45:18.422057+0000","last_fullsized":"2024-01-24T17:45:18.422057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630813+0000","last_change":"2024-01-24T17:45:18.409314+0000","last_active":"2024-01-24T17:45:19.630813+0000","last_peered":"2024-01-24T17:45:19.630813+0000","last_clean":"2024-01-24T17:45:19.630813+0000","last_became_active":"2024-01-24T17:45:18.409031+0000","last_became_peered":"2024-01-24T17:45:18.409031+0000","last_unstale":"2024-01-24T17:45:19.630813+0000","last_undegraded":"2024-01-24T17:45:19.630813+0000","last_fullsized":"2024-01-24T17:45:19.630813+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:17.915909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.406944+0000","last_change":"2024-01-24T17:45:17.921375+0000","last_active":"2024-01-24T17:45:18.406944+0000","last_peered":"2024-01-24T17:45:18.406944+0000","last_clean":"2024-01-24T17:45:18.406944+0000","last_became_active":"2024-01-24T17:45:17.921094+0000","last_became_peered":"2024-01-24T17:45:17.921094+0000","last_unstale":"2024-01-24T17:45:18.406944+0000","last_undegraded":"2024-01-24T17:45:18.406944+0000","last_fullsized":"2024-01-24T17:45:18.406944+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:09:47.749409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629511+0000","last_change":"2024-01-24T17:45:18.416295+0000","last_active":"2024-01-24T17:45:19.629511+0000","last_peered":"2024-01-24T17:45:19.629511+0000","last_clean":"2024-01-24T17:45:19.629511+0000","last_became_active":"2024-01-24T17:45:18.415734+0000","last_became_peered":"2024-01-24T17:45:18.415734+0000","last_unstale":"2024-01-24T17:45:19.629511+0000","last_undegraded":"2024-01-24T17:45:19.629511+0000","last_fullsized":"2024-01-24T17:45:19.629511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:23.692981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.407138+0000","last_change":"2024-01-24T17:45:17.921933+0000","last_active":"2024-01-24T17:45:18.407138+0000","last_peered":"2024-01-24T17:45:18.407138+0000","last_clean":"2024-01-24T17:45:18.407138+0000","last_became_active":"2024-01-24T17:45:17.921674+0000","last_became_peered":"2024-01-24T17:45:17.921674+0000","last_unstale":"2024-01-24T17:45:18.407138+0000","last_undegraded":"2024-01-24T17:45:18.407138+0000","last_fullsized":"2024-01-24T17:45:18.407138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:30.797138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396509+0000","last_change":"2024-01-24T17:45:18.431790+0000","last_active":"2024-01-24T17:45:19.396509+0000","last_peered":"2024-01-24T17:45:19.396509+0000","last_clean":"2024-01-24T17:45:19.396509+0000","last_became_active":"2024-01-24T17:45:18.431405+0000","last_became_peered":"2024-01-24T17:45:18.431405+0000","last_unstale":"2024-01-24T17:45:19.396509+0000","last_undegraded":"2024-01-24T17:45:19.396509+0000","last_fullsized":"2024-01-24T17:45:19.396509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:05:11.682782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.395382+0000","last_change":"2024-01-24T17:45:17.914736+0000","last_active":"2024-01-24T17:45:18.395382+0000","last_peered":"2024-01-24T17:45:18.395382+0000","last_clean":"2024-01-24T17:45:18.395382+0000","last_became_active":"2024-01-24T17:45:17.914309+0000","last_became_peered":"2024-01-24T17:45:17.914309+0000","last_unstale":"2024-01-24T17:45:18.395382+0000","last_undegraded":"2024-01-24T17:45:18.395382+0000","last_fullsized":"2024-01-24T17:45:18.395382+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:12.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630025+0000","last_change":"2024-01-24T17:45:18.425885+0000","last_active":"2024-01-24T17:45:19.630025+0000","last_peered":"2024-01-24T17:45:19.630025+0000","last_clean":"2024-01-24T17:45:19.630025+0000","last_became_active":"2024-01-24T17:45:18.425596+0000","last_became_peered":"2024-01-24T17:45:18.425596+0000","last_unstale":"2024-01-24T17:45:19.630025+0000","last_undegraded":"2024-01-24T17:45:19.630025+0000","last_fullsized":"2024-01-24T17:45:19.630025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:41:32.785344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.390567+0000","last_change":"2024-01-24T17:45:17.919042+0000","last_active":"2024-01-24T17:45:18.390567+0000","last_peered":"2024-01-24T17:45:18.390567+0000","last_clean":"2024-01-24T17:45:18.390567+0000","last_became_active":"2024-01-24T17:45:17.918689+0000","last_became_peered":"2024-01-24T17:45:17.918689+0000","last_unstale":"2024-01-24T17:45:18.390567+0000","last_undegraded":"2024-01-24T17:45:18.390567+0000","last_fullsized":"2024-01-24T17:45:18.390567+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":202,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.407708+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:45:18.407708+0000","last_peered":"2024-01-24T17:45:18.407708+0000","last_clean":"2024-01-24T17:45:18.407708+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:45:18.407708+0000","last_undegraded":"2024-01-24T17:45:18.407708+0000","last_fullsized":"2024-01-24T17:45:18.407708+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":765952,"data_stored":320907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.05,"15min":1.0249999999999999},"min":{"1min":0.80300000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.048,"15min":1.0329999999999999},"min":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.599,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.341}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0860000000000001,"15min":1.1000000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.319},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1859999999999999,"15min":1.1919999999999999},"min":{"1min":0.81499999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.649,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.494}]},{"osd":2,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.171,"15min":1.1579999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.764,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.181,"15min":1.2190000000000001},"min":{"1min":0.745,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.484,"5min":2.484,"15min":2.484},"last":1.2549999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8779999999999999,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.282,"15min":1.3100000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.6269999999999998,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.3660000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.296,"15min":1.3200000000000001},"min":{"1min":0.84699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.1190000000000002,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5449999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.468}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.319,"15min":1.3520000000000001},"min":{"1min":0.70799999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.133},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3979999999999999,"15min":1.4350000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.966,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.1759999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4990000000000001,"15min":1.5349999999999999},"min":{"1min":0.72599999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.5720000000000001,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.427},{"interface":"front","average":{"1min":1.399,"5min":1.472,"15min":1.488},"min":{"1min":0.86099999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.2450000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.22}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":761856,"data_stored":320837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.034,"15min":1.024},"min":{"1min":0.70699999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6919999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.044,"15min":1.0189999999999999},"min":{"1min":0.749,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.94099999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.022,"15min":0.99199999999999999},"min":{"1min":0.70199999999999996,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4890000000000001,"5min":1.73,"15min":1.73},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1180000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.722,"5min":1.972,"15min":1.972},"last":1.206}]},{"osd":2,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.79300000000000004,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5369999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.169,"15min":1.167},"min":{"1min":0.90700000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.514,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.26}]},{"osd":3,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.242,"15min":1.254},"min":{"1min":0.83799999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.542,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2310000000000001,"15min":1.2529999999999999},"min":{"1min":0.84199999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5980000000000001,"5min":2.46,"15min":2.46},"last":0.97499999999999998}]},{"osd":4,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.2569999999999999,"15min":1.2370000000000001},"min":{"1min":0.76000000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.661,"5min":1.988,"15min":1.988},"last":1.105},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2989999999999999,"15min":1.3029999999999999},"min":{"1min":0.73899999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.8420000000000001,"5min":1.849,"15min":1.849},"last":1.5760000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3080000000000001,"15min":1.296},"min":{"1min":0.97999999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.645,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.351},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.89800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9319999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.389}]},{"osd":7,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.355,"15min":1.367},"min":{"1min":0.85399999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.978,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.534},{"interface":"front","average":{"1min":1.45,"5min":1.454,"15min":1.4630000000000001},"min":{"1min":1.008,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.899,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.6399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574900,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1355776,"data_stored":911217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0669999999999999,"15min":1.099},"min":{"1min":0.67500000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.47,"5min":1.508,"15min":1.508},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.998,"5min":1.0289999999999999,"15min":1.0309999999999999},"min":{"1min":0.67900000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5,"5min":1.889,"15min":1.889},"last":0.94099999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.077,"15min":1.0740000000000001},"min":{"1min":0.72899999999999998,"5min":0.622,"15min":0.622},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.22,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.552,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.1120000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2230000000000001,"15min":1.2290000000000001},"min":{"1min":0.80600000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.28,"15min":1.3120000000000001},"min":{"1min":0.79800000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.9870000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.381}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2909999999999999,"15min":1.341},"min":{"1min":0.77800000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6020000000000001,"5min":3.036,"15min":3.036},"last":1.208},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3320000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.641,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.2549999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4339999999999999,"15min":1.4530000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.915,"5min":2.069,"15min":2.069},"last":1.429},{"interface":"front","average":{"1min":1.206,"5min":1.329,"15min":1.3600000000000001},"min":{"1min":0.77600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5780000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.056}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.339,"15min":1.3560000000000001},"min":{"1min":0.91100000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.724,"5min":2.504,"15min":2.504},"last":1.298},{"interface":"front","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":0.88600000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.044,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.5509999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.458,"15min":1.4930000000000001},"min":{"1min":0.63800000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.6719999999999999,"5min":3.161,"15min":3.161},"last":1.337},{"interface":"front","average":{"1min":1.486,"5min":1.5900000000000001,"15min":1.6339999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.1389999999999998,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.6060000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":765952,"data_stored":321431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.048,"15min":1.038},"min":{"1min":0.73499999999999999,"5min":0.621,"15min":0.621},"max":{"1min":1.673,"5min":1.698,"15min":1.698},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.093,"15min":1.0960000000000001},"min":{"1min":0.71199999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5529999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.85599999999999998}]},{"osd":1,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.0800000000000001,"15min":1.052},"min":{"1min":0.78500000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5049999999999999,"5min":1.633,"15min":1.633},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1899999999999999,"15min":1.2170000000000001},"min":{"1min":0.67800000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.4139999999999999,"5min":1.849,"15min":1.849},"last":1.4950000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.206,"15min":1.212},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.202,"5min":1.304,"15min":1.3340000000000001},"min":{"1min":0.71199999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.724,"5min":1.986,"15min":1.986},"last":1.794}]},{"osd":3,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1850000000000001,"15min":1.1910000000000001},"min":{"1min":0.81000000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.456,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.343,"15min":1.351},"min":{"1min":0.89400000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.8029999999999999,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.216}]},{"osd":5,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3480000000000001,"15min":1.373},"min":{"1min":1.014,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.613,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.8759999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.298,"15min":1.306},"min":{"1min":0.85799999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5600000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.1639999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4550000000000001,"15min":1.4730000000000001},"min":{"1min":0.93799999999999994,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.3159999999999998,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":1.077},{"interface":"front","average":{"1min":1.385,"5min":1.4610000000000001,"15min":1.4790000000000001},"min":{"1min":0.86899999999999999,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.619}]},{"osd":7,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4330000000000001,"15min":1.4259999999999999},"min":{"1min":0.92000000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0209999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.6000000000000001,"15min":1.621},"min":{"1min":0.97299999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.2309999999999999,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.4379999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":761856,"data_stored":320905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.91600000000000004,"15min":0.872},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.792,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95499999999999996,"15min":0.89100000000000001},"min":{"1min":0.64200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5680000000000001,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.55300000000000005}]},{"osd":1,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.93899999999999995,"15min":0.95099999999999996},"min":{"1min":0.53100000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.819,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0720000000000001,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.663,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.82999999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.105,"15min":1.111},"min":{"1min":0.629,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.571,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.181,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.593,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.877}]},{"osd":4,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3200000000000001,"15min":1.292},"min":{"1min":0.90800000000000003,"5min":0.877,"15min":0.877},"max":{"1min":1.8240000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9790000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":0.99099999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4350000000000001,"15min":1.4370000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2189999999999999,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4159999999999999,"15min":1.4239999999999999},"min":{"1min":1.024,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.0579999999999998,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.1879999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.532,"15min":1.5029999999999999},"min":{"1min":1.0429999999999999,"5min":0.871,"15min":0.871},"max":{"1min":1.984,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.147},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4730000000000001,"15min":1.4630000000000001},"min":{"1min":0.90200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.1120000000000001,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":0.81899999999999995}]},{"osd":7,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5389999999999999,"15min":1.5169999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.012,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.5249999999999999},"min":{"1min":1.0649999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.1160000000000001,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.0580000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":778240,"data_stored":322539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.439,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.24,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93600000000000005,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.78500000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0369999999999999,"15min":1.0549999999999999},"min":{"1min":0.51300000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.5509999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.173,"15min":1.2150000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.66900000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.98199999999999998,"15min":0.996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6910000000000001,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2050000000000001,"15min":1.218},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7629999999999999,"5min":2.996,"15min":2.996},"last":0.72799999999999998}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.306,"15min":1.329},"min":{"1min":0.72899999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6950000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4099999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.7,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.1339999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3600000000000001,"15min":1.409},"min":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.762,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2969999999999999,"15min":1.331},"min":{"1min":0.86299999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.625,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.92500000000000004}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.369,"15min":1.4019999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9350000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4399999999999999,"15min":1.4830000000000001},"min":{"1min":0.80200000000000005,"5min":0.753,"15min":0.753},"max":{"1min":2.2530000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.85899999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.526,"15min":1.5860000000000001},"min":{"1min":0.82499999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0209999999999999,"5min":3.343,"15min":3.343},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4630000000000001,"15min":1.512},"min":{"1min":0.755,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9510000000000001,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.0069999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1355776,"data_stored":911765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.874,"15min":0.85999999999999999},"min":{"1min":0.45500000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4419999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.115},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.93000000000000005,"15min":0.95299999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.155,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.36399999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.95399999999999996,"15min":0.98399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.4790000000000001,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5680000000000001,"5min":1.895,"15min":1.895},"last":1.5960000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.143,"15min":1.1639999999999999},"min":{"1min":0.61299999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4930000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.194,"15min":1.244},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.4019999999999999,"5min":3.149,"15min":3.149},"last":1.246}]},{"osd":4,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.353,"15min":1.365},"min":{"1min":0.88700000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.77,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.296,"15min":1.3240000000000001},"min":{"1min":0.83599999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.698,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.3200000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.323,"15min":1.357},"min":{"1min":0.73499999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.806,"5min":2.577,"15min":2.577},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.9730000000000001,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.3999999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.4019999999999999,"15min":1.4450000000000001},"min":{"1min":0.80000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.653,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.637},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.46,"15min":1.5309999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.921,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.675}]},{"osd":7,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4570000000000001,"15min":1.4750000000000001},"min":{"1min":0.91400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0979999999999999,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.5049999999999999,"15min":1.5549999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.641,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.4339999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":770048,"data_stored":322379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80000000000000004,"15min":0.81799999999999995},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.242,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.875,"5min":0.93000000000000005,"15min":0.96999999999999997},"min":{"1min":0.52900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.1879999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.84199999999999997,"15min":0.83399999999999996},"min":{"1min":0.45200000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.997,"5min":1.046,"15min":1.1000000000000001},"min":{"1min":0.46800000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4430000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.80300000000000005}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.1359999999999999,"15min":1.1890000000000001},"min":{"1min":0.51000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.639,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.105,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.60699999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6040000000000001,"5min":2.407,"15min":2.407},"last":1.133}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4179999999999999},"min":{"1min":0.752,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.056,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.4139999999999999,"15min":1.488},"min":{"1min":0.89000000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.5840000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.161}]},{"osd":5,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.333,"15min":1.397},"min":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.7490000000000001,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.256,"15min":1.264},"min":{"1min":0.85599999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.694,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.6659999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.337,"15min":1.3600000000000001},"min":{"1min":0.93500000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":2.335,"15min":2.335},"last":1.752},{"interface":"front","average":{"1min":1.341,"5min":1.4139999999999999,"15min":1.46},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.877,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.04}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.462,"15min":1.534},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.694,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.5369999999999999,"15min":1.605},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.806,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.403}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:45:27.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:45:28.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.191+0000 7f914f200640 1 Processor -- start 2024-01-24T17:45:28.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.191+0000 7f914f200640 1 -- start start 2024-01-24T17:45:28.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.192+0000 7f914f200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f915015ab00 0x7f91500cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.192+0000 7f914f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91500ce400 con 0x7f91500ccd20 2024-01-24T17:45:28.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.192+0000 7f914f200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915015b3b0 con 0x7f915015a720 2024-01-24T17:45:28.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.192+0000 7f914f200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915015b690 con 0x7f915015ab00 2024-01-24T17:45:28.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.193+0000 7f914e800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f91500ccd20 0x7f91500ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:36754/0 (socket says 172.21.15.39:36754) 2024-01-24T17:45:28.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.193+0000 7f914e800640 1 -- 172.21.15.39:0/3146326301 learned_addr learned my addr 172.21.15.39:0/3146326301 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:28.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.193+0000 7f914d400640 1 --2- 172.21.15.39:0/3146326301 >> v2:172.21.15.100:3300/0 conn(0x7f915015ab00 0x7f91500cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.193+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745572832 0 0) 0x7f91500ce400 con 0x7f91500ccd20 2024-01-24T17:45:28.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.193+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9130003520 con 0x7f91500ccd20 2024-01-24T17:45:28.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.193+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288529911 0 0) 0x7f9130003520 con 0x7f91500ccd20 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.193+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 >> v2:172.21.15.100:3300/0 conn(0x7f915015ab00 msgr2=0x7f91500cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 --2- 172.21.15.39:0/3146326301 >> v2:172.21.15.100:3300/0 conn(0x7f915015ab00 0x7f91500cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 >> v1:172.21.15.100:6789/0 conn(0x7f915015a720 legacy=0x7f91500cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915015b970 con 0x7f91500ccd20 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f91480019f0 con 0x7f91500ccd20 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 >> v1:172.21.15.39:6789/0 conn(0x7f91500ccd20 legacy=0x7f91500ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9130003970 0x7f9130003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f9130004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9130005180 0x7f913000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915015b690 con 0x7f91300047c0 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913000c120 con 0x7f9130005180 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913000c2e0 con 0x7f9130003970 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914d400640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f9130004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914e800640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9130005180 0x7f913000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914de00640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9130003970 0x7f9130003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.194+0000 7f914d400640 1 -- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9130003970 msgr2=0x7f9130003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.195+0000 7f914d400640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9130003970 0x7f9130003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.195+0000 7f914d400640 1 -- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9130005180 msgr2=0x7f913000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.195+0000 7f914d400640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9130005180 0x7f913000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.195+0000 7f914d400640 1 -- 172.21.15.39:0/3146326301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9130003520 con 0x7f91300047c0 2024-01-24T17:45:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.195+0000 7f914e800640 1 -- 172.21.15.39:0/3146326301 reap_dead start 2024-01-24T17:45:28.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.195+0000 7f914de00640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9130003970 0x7f9130003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:28.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.196+0000 7f914d400640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f9130004c40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9138009ac0 tx=0x7f91300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37147848e578dbe6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:28.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.196+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913802c6b0 con 0x7f91300047c0 2024-01-24T17:45:28.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.196+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913802c810 con 0x7f91300047c0 2024-01-24T17:45:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.196+0000 7f914ca00640 1 -- 172.21.15.39:0/3146326301 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913802cba0 con 0x7f91300047c0 2024-01-24T17:45:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.197+0000 7f914f200640 1 -- 172.21.15.39:0/3146326301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 msgr2=0x7f9130004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.197+0000 7f914f200640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f9130004c40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9138009ac0 tx=0x7f91300057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.197+0000 7f914f200640 1 -- 172.21.15.39:0/3146326301 shutdown_connections 2024-01-24T17:45:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.197+0000 7f914f200640 1 --2- 172.21.15.39:0/3146326301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f9130004c40 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.197+0000 7f914f200640 1 -- 172.21.15.39:0/3146326301 >> 172.21.15.39:0/3146326301 conn(0x7f915006a090 msgr2=0x7f915006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.197+0000 7f914f200640 1 -- 172.21.15.39:0/3146326301 shutdown_connections 2024-01-24T17:45:28.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.197+0000 7f914f200640 1 -- 172.21.15.39:0/3146326301 wait complete. 2024-01-24T17:45:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.199+0000 7f914f200640 1 Processor -- start 2024-01-24T17:45:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.199+0000 7f914f200640 1 -- start start 2024-01-24T17:45:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f91501579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91500c7a80 0x7f91500c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91500c9d30 0x7f91500ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91500ca6f0 con 0x7f91300047c0 2024-01-24T17:45:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914f200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9150157ef0 con 0x7f91500c7a80 2024-01-24T17:45:28.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91500c6260 con 0x7f91500c9d30 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f91501579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f91501579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47936/0 (socket says 172.21.15.39:47936) 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914de00640 1 -- 172.21.15.39:0/4003296418 learned_addr learned my addr 172.21.15.39:0/4003296418 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914d400640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91500c7a80 0x7f91500c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914e800640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91500c9d30 0x7f91500ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914de00640 1 -- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91500c9d30 msgr2=0x7f91500ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914de00640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91500c9d30 0x7f91500ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914de00640 1 -- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91500c7a80 msgr2=0x7f91500c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914de00640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91500c7a80 0x7f91500c7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.200+0000 7f914de00640 1 -- 172.21.15.39:0/4003296418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91500c6540 con 0x7f91300047c0 2024-01-24T17:45:28.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.201+0000 7f914de00640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f91501579b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f914000d4d0 tx=0x7f914000d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.201+0000 7f9146000640 1 -- 172.21.15.39:0/4003296418 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9140005480 con 0x7f91300047c0 2024-01-24T17:45:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.201+0000 7f9146000640 1 -- 172.21.15.39:0/4003296418 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9140007030 con 0x7f91300047c0 2024-01-24T17:45:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.201+0000 7f9146000640 1 -- 172.21.15.39:0/4003296418 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91400073c0 con 0x7f91300047c0 2024-01-24T17:45:28.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.201+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91500c6820 con 0x7f91300047c0 2024-01-24T17:45:28.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.201+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91500c3450 con 0x7f91300047c0 2024-01-24T17:45:28.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.202+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91500c6d40 con 0x7f91300047c0 2024-01-24T17:45:28.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.203+0000 7f9146000640 1 -- 172.21.15.39:0/4003296418 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9140007520 con 0x7f91300047c0 2024-01-24T17:45:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.204+0000 7f9146000640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f912806a430 0x7f912806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.205+0000 7f9146000640 1 -- 172.21.15.39:0/4003296418 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9140087ea0 con 0x7f91300047c0 2024-01-24T17:45:28.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.205+0000 7f914d400640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f912806a430 0x7f912806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.206+0000 7f914d400640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f912806a430 0x7f912806c8f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9138036000 tx=0x7f9138030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:28.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.209+0000 7f9146000640 1 -- 172.21.15.39:0/4003296418 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9140053480 con 0x7f91300047c0 2024-01-24T17:45:28.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.407+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f915015bc50 con 0x7f912806a430 2024-01-24T17:45:28.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.429+0000 7f9146000640 1 -- 172.21.15.39:0/4003296418 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286991 (secure 0 0 0) 0x7f91380041c0 con 0x7f912806a430 2024-01-24T17:45:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:28.436 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:45:28.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f912806a430 msgr2=0x7f912806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f912806a430 0x7f912806c8f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9138036000 tx=0x7f9138030000 comp rx=0 tx=0).stop 2024-01-24T17:45:28.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 msgr2=0x7f91501579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f91501579b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f914000d4d0 tx=0x7f914000d990 comp rx=0 tx=0).stop 2024-01-24T17:45:28.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 shutdown_connections 2024-01-24T17:45:28.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f91500c9d30 0x7f91500ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f91500c7a80 0x7f91500c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f912806a430 0x7f912806c8f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 --2- 172.21.15.39:0/4003296418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91300047c0 0x7f91501579b0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.436+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 >> 172.21.15.39:0/4003296418 conn(0x7f915006a090 msgr2=0x7f91500cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:28.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.437+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 shutdown_connections 2024-01-24T17:45:28.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.437+0000 7f914f200640 1 -- 172.21.15.39:0/4003296418 wait complete. 2024-01-24T17:45:28.460 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-01-24T17:45:28.295874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252056,"kb_used_data":7144,"kb_used_omap":13,"kb_used_meta":244722,"kb_avail":754722664,"statfs":{"total":773094113280,"available":772836007936,"internally_reserved":0,"allocated":7315456,"data_stored":3751980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":250595416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.905574"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396720+0000","last_change":"2024-01-24T17:45:18.428192+0000","last_active":"2024-01-24T17:45:19.396720+0000","last_peered":"2024-01-24T17:45:19.396720+0000","last_clean":"2024-01-24T17:45:19.396720+0000","last_became_active":"2024-01-24T17:45:18.421242+0000","last_became_peered":"2024-01-24T17:45:18.421242+0000","last_unstale":"2024-01-24T17:45:19.396720+0000","last_undegraded":"2024-01-24T17:45:19.396720+0000","last_fullsized":"2024-01-24T17:45:19.396720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:04.171678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397973+0000","last_change":"2024-01-24T17:45:18.434096+0000","last_active":"2024-01-24T17:45:19.397973+0000","last_peered":"2024-01-24T17:45:19.397973+0000","last_clean":"2024-01-24T17:45:19.397973+0000","last_became_active":"2024-01-24T17:45:18.433813+0000","last_became_peered":"2024-01-24T17:45:18.433813+0000","last_unstale":"2024-01-24T17:45:19.397973+0000","last_undegraded":"2024-01-24T17:45:19.397973+0000","last_fullsized":"2024-01-24T17:45:19.397973+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:14:51.606606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631230+0000","last_change":"2024-01-24T17:45:18.416822+0000","last_active":"2024-01-24T17:45:19.631230+0000","last_peered":"2024-01-24T17:45:19.631230+0000","last_clean":"2024-01-24T17:45:19.631230+0000","last_became_active":"2024-01-24T17:45:18.416525+0000","last_became_peered":"2024-01-24T17:45:18.416525+0000","last_unstale":"2024-01-24T17:45:19.631230+0000","last_undegraded":"2024-01-24T17:45:19.631230+0000","last_fullsized":"2024-01-24T17:45:19.631230+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:58:02.429970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.632549+0000","last_change":"2024-01-24T17:45:18.410425+0000","last_active":"2024-01-24T17:45:19.632549+0000","last_peered":"2024-01-24T17:45:19.632549+0000","last_clean":"2024-01-24T17:45:19.632549+0000","last_became_active":"2024-01-24T17:45:18.408436+0000","last_became_peered":"2024-01-24T17:45:18.408436+0000","last_unstale":"2024-01-24T17:45:19.632549+0000","last_undegraded":"2024-01-24T17:45:19.632549+0000","last_fullsized":"2024-01-24T17:45:19.632549+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:11:53.119290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396552+0000","last_change":"2024-01-24T17:45:18.419250+0000","last_active":"2024-01-24T17:45:19.396552+0000","last_peered":"2024-01-24T17:45:19.396552+0000","last_clean":"2024-01-24T17:45:19.396552+0000","last_became_active":"2024-01-24T17:45:18.418924+0000","last_became_peered":"2024-01-24T17:45:18.418924+0000","last_unstale":"2024-01-24T17:45:19.396552+0000","last_undegraded":"2024-01-24T17:45:19.396552+0000","last_fullsized":"2024-01-24T17:45:19.396552+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:04.335225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397721+0000","last_change":"2024-01-24T17:45:18.437862+0000","last_active":"2024-01-24T17:45:19.397721+0000","last_peered":"2024-01-24T17:45:19.397721+0000","last_clean":"2024-01-24T17:45:19.397721+0000","last_became_active":"2024-01-24T17:45:18.437581+0000","last_became_peered":"2024-01-24T17:45:18.437581+0000","last_unstale":"2024-01-24T17:45:19.397721+0000","last_undegraded":"2024-01-24T17:45:19.397721+0000","last_fullsized":"2024-01-24T17:45:19.397721+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:05:11.682782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631554+0000","last_change":"2024-01-24T17:45:18.408418+0000","last_active":"2024-01-24T17:45:19.631554+0000","last_peered":"2024-01-24T17:45:19.631554+0000","last_clean":"2024-01-24T17:45:19.631554+0000","last_became_active":"2024-01-24T17:45:18.408145+0000","last_became_peered":"2024-01-24T17:45:18.408145+0000","last_unstale":"2024-01-24T17:45:19.631554+0000","last_undegraded":"2024-01-24T17:45:19.631554+0000","last_fullsized":"2024-01-24T17:45:19.631554+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:13:32.005579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629503+0000","last_change":"2024-01-24T17:45:18.416365+0000","last_active":"2024-01-24T17:45:19.629503+0000","last_peered":"2024-01-24T17:45:19.629503+0000","last_clean":"2024-01-24T17:45:19.629503+0000","last_became_active":"2024-01-24T17:45:18.416080+0000","last_became_peered":"2024-01-24T17:45:18.416080+0000","last_unstale":"2024-01-24T17:45:19.629503+0000","last_undegraded":"2024-01-24T17:45:19.629503+0000","last_fullsized":"2024-01-24T17:45:19.629503+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:13:50.434214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.395146+0000","last_change":"2024-01-24T17:45:18.419555+0000","last_active":"2024-01-24T17:45:19.395146+0000","last_peered":"2024-01-24T17:45:19.395146+0000","last_clean":"2024-01-24T17:45:19.395146+0000","last_became_active":"2024-01-24T17:45:18.418203+0000","last_became_peered":"2024-01-24T17:45:18.418203+0000","last_unstale":"2024-01-24T17:45:19.395146+0000","last_undegraded":"2024-01-24T17:45:19.395146+0000","last_fullsized":"2024-01-24T17:45:19.395146+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:54:57.788190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397031+0000","last_change":"2024-01-24T17:45:18.414408+0000","last_active":"2024-01-24T17:45:19.397031+0000","last_peered":"2024-01-24T17:45:19.397031+0000","last_clean":"2024-01-24T17:45:19.397031+0000","last_became_active":"2024-01-24T17:45:18.413540+0000","last_became_peered":"2024-01-24T17:45:18.413540+0000","last_unstale":"2024-01-24T17:45:19.397031+0000","last_undegraded":"2024-01-24T17:45:19.397031+0000","last_fullsized":"2024-01-24T17:45:19.397031+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:49:33.451268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628310+0000","last_change":"2024-01-24T17:45:18.433178+0000","last_active":"2024-01-24T17:45:19.628310+0000","last_peered":"2024-01-24T17:45:19.628310+0000","last_clean":"2024-01-24T17:45:19.628310+0000","last_became_active":"2024-01-24T17:45:18.432838+0000","last_became_peered":"2024-01-24T17:45:18.432838+0000","last_unstale":"2024-01-24T17:45:19.628310+0000","last_undegraded":"2024-01-24T17:45:19.628310+0000","last_fullsized":"2024-01-24T17:45:19.628310+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:13.563526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.395285+0000","last_change":"2024-01-24T17:45:18.412494+0000","last_active":"2024-01-24T17:45:19.395285+0000","last_peered":"2024-01-24T17:45:19.395285+0000","last_clean":"2024-01-24T17:45:19.395285+0000","last_became_active":"2024-01-24T17:45:18.412189+0000","last_became_peered":"2024-01-24T17:45:18.412189+0000","last_unstale":"2024-01-24T17:45:19.395285+0000","last_undegraded":"2024-01-24T17:45:19.395285+0000","last_fullsized":"2024-01-24T17:45:19.395285+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:05:03.201083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630518+0000","last_change":"2024-01-24T17:45:18.419893+0000","last_active":"2024-01-24T17:45:19.630518+0000","last_peered":"2024-01-24T17:45:19.630518+0000","last_clean":"2024-01-24T17:45:19.630518+0000","last_became_active":"2024-01-24T17:45:18.419585+0000","last_became_peered":"2024-01-24T17:45:18.419585+0000","last_unstale":"2024-01-24T17:45:19.630518+0000","last_undegraded":"2024-01-24T17:45:19.630518+0000","last_fullsized":"2024-01-24T17:45:19.630518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:04.171678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628827+0000","last_change":"2024-01-24T17:45:18.421833+0000","last_active":"2024-01-24T17:45:19.628827+0000","last_peered":"2024-01-24T17:45:19.628827+0000","last_clean":"2024-01-24T17:45:19.628827+0000","last_became_active":"2024-01-24T17:45:18.421603+0000","last_became_peered":"2024-01-24T17:45:18.421603+0000","last_unstale":"2024-01-24T17:45:19.628827+0000","last_undegraded":"2024-01-24T17:45:19.628827+0000","last_fullsized":"2024-01-24T17:45:19.628827+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:08:30.981439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630163+0000","last_change":"2024-01-24T17:45:18.404163+0000","last_active":"2024-01-24T17:45:19.630163+0000","last_peered":"2024-01-24T17:45:19.630163+0000","last_clean":"2024-01-24T17:45:19.630163+0000","last_became_active":"2024-01-24T17:45:18.403613+0000","last_became_peered":"2024-01-24T17:45:18.403613+0000","last_unstale":"2024-01-24T17:45:19.630163+0000","last_undegraded":"2024-01-24T17:45:19.630163+0000","last_fullsized":"2024-01-24T17:45:19.630163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:45.817758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.395440+0000","last_change":"2024-01-24T17:45:18.424370+0000","last_active":"2024-01-24T17:45:19.395440+0000","last_peered":"2024-01-24T17:45:19.395440+0000","last_clean":"2024-01-24T17:45:19.395440+0000","last_became_active":"2024-01-24T17:45:18.419616+0000","last_became_peered":"2024-01-24T17:45:18.419616+0000","last_unstale":"2024-01-24T17:45:19.395440+0000","last_undegraded":"2024-01-24T17:45:19.395440+0000","last_fullsized":"2024-01-24T17:45:19.395440+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:45:55.342357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397197+0000","last_change":"2024-01-24T17:45:18.414883+0000","last_active":"2024-01-24T17:45:19.397197+0000","last_peered":"2024-01-24T17:45:19.397197+0000","last_clean":"2024-01-24T17:45:19.397197+0000","last_became_active":"2024-01-24T17:45:18.413671+0000","last_became_peered":"2024-01-24T17:45:18.413671+0000","last_unstale":"2024-01-24T17:45:19.397197+0000","last_undegraded":"2024-01-24T17:45:19.397197+0000","last_fullsized":"2024-01-24T17:45:19.397197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:13.563526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.632296+0000","last_change":"2024-01-24T17:45:18.423276+0000","last_active":"2024-01-24T17:45:19.632296+0000","last_peered":"2024-01-24T17:45:19.632296+0000","last_clean":"2024-01-24T17:45:19.632296+0000","last_became_active":"2024-01-24T17:45:18.423068+0000","last_became_peered":"2024-01-24T17:45:18.423068+0000","last_unstale":"2024-01-24T17:45:19.632296+0000","last_undegraded":"2024-01-24T17:45:19.632296+0000","last_fullsized":"2024-01-24T17:45:19.632296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:33:02.565560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397557+0000","last_change":"2024-01-24T17:45:18.438606+0000","last_active":"2024-01-24T17:45:19.397557+0000","last_peered":"2024-01-24T17:45:19.397557+0000","last_clean":"2024-01-24T17:45:19.397557+0000","last_became_active":"2024-01-24T17:45:18.438232+0000","last_became_peered":"2024-01-24T17:45:18.438232+0000","last_unstale":"2024-01-24T17:45:19.397557+0000","last_undegraded":"2024-01-24T17:45:19.397557+0000","last_fullsized":"2024-01-24T17:45:19.397557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:46:56.750634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628816+0000","last_change":"2024-01-24T17:45:18.434304+0000","last_active":"2024-01-24T17:45:19.628816+0000","last_peered":"2024-01-24T17:45:19.628816+0000","last_clean":"2024-01-24T17:45:19.628816+0000","last_became_active":"2024-01-24T17:45:18.432897+0000","last_became_peered":"2024-01-24T17:45:18.432897+0000","last_unstale":"2024-01-24T17:45:19.628816+0000","last_undegraded":"2024-01-24T17:45:19.628816+0000","last_fullsized":"2024-01-24T17:45:19.628816+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:46:56.750634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630570+0000","last_change":"2024-01-24T17:45:18.412717+0000","last_active":"2024-01-24T17:45:19.630570+0000","last_peered":"2024-01-24T17:45:19.630570+0000","last_clean":"2024-01-24T17:45:19.630570+0000","last_became_active":"2024-01-24T17:45:18.412447+0000","last_became_peered":"2024-01-24T17:45:18.412447+0000","last_unstale":"2024-01-24T17:45:19.630570+0000","last_undegraded":"2024-01-24T17:45:19.630570+0000","last_fullsized":"2024-01-24T17:45:19.630570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:13.563526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628295+0000","last_change":"2024-01-24T17:45:18.426865+0000","last_active":"2024-01-24T17:45:19.628295+0000","last_peered":"2024-01-24T17:45:19.628295+0000","last_clean":"2024-01-24T17:45:19.628295+0000","last_became_active":"2024-01-24T17:45:18.426589+0000","last_became_peered":"2024-01-24T17:45:18.426589+0000","last_unstale":"2024-01-24T17:45:19.628295+0000","last_undegraded":"2024-01-24T17:45:19.628295+0000","last_fullsized":"2024-01-24T17:45:19.628295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:08:30.981439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631790+0000","last_change":"2024-01-24T17:45:18.426201+0000","last_active":"2024-01-24T17:45:19.631790+0000","last_peered":"2024-01-24T17:45:19.631790+0000","last_clean":"2024-01-24T17:45:19.631790+0000","last_became_active":"2024-01-24T17:45:18.425939+0000","last_became_peered":"2024-01-24T17:45:18.425939+0000","last_unstale":"2024-01-24T17:45:19.631790+0000","last_undegraded":"2024-01-24T17:45:19.631790+0000","last_fullsized":"2024-01-24T17:45:19.631790+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:47:11.363408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394549+0000","last_change":"2024-01-24T17:45:18.420014+0000","last_active":"2024-01-24T17:45:19.394549+0000","last_peered":"2024-01-24T17:45:19.394549+0000","last_clean":"2024-01-24T17:45:19.394549+0000","last_became_active":"2024-01-24T17:45:18.419578+0000","last_became_peered":"2024-01-24T17:45:18.419578+0000","last_unstale":"2024-01-24T17:45:19.394549+0000","last_undegraded":"2024-01-24T17:45:19.394549+0000","last_fullsized":"2024-01-24T17:45:19.394549+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:45:46.357529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.398218+0000","last_change":"2024-01-24T17:45:18.435364+0000","last_active":"2024-01-24T17:45:19.398218+0000","last_peered":"2024-01-24T17:45:19.398218+0000","last_clean":"2024-01-24T17:45:19.398218+0000","last_became_active":"2024-01-24T17:45:18.435072+0000","last_became_peered":"2024-01-24T17:45:18.435072+0000","last_unstale":"2024-01-24T17:45:19.398218+0000","last_undegraded":"2024-01-24T17:45:19.398218+0000","last_fullsized":"2024-01-24T17:45:19.398218+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:08:30.981439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631466+0000","last_change":"2024-01-24T17:45:18.399862+0000","last_active":"2024-01-24T17:45:19.631466+0000","last_peered":"2024-01-24T17:45:19.631466+0000","last_clean":"2024-01-24T17:45:19.631466+0000","last_became_active":"2024-01-24T17:45:18.399525+0000","last_became_peered":"2024-01-24T17:45:18.399525+0000","last_unstale":"2024-01-24T17:45:19.631466+0000","last_undegraded":"2024-01-24T17:45:19.631466+0000","last_fullsized":"2024-01-24T17:45:19.631466+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:13.657421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397944+0000","last_change":"2024-01-24T17:45:18.434414+0000","last_active":"2024-01-24T17:45:19.397944+0000","last_peered":"2024-01-24T17:45:19.397944+0000","last_clean":"2024-01-24T17:45:19.397944+0000","last_became_active":"2024-01-24T17:45:18.433941+0000","last_became_peered":"2024-01-24T17:45:18.433941+0000","last_unstale":"2024-01-24T17:45:19.397944+0000","last_undegraded":"2024-01-24T17:45:19.397944+0000","last_fullsized":"2024-01-24T17:45:19.397944+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:49:33.451268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630068+0000","last_change":"2024-01-24T17:45:18.407428+0000","last_active":"2024-01-24T17:45:19.630068+0000","last_peered":"2024-01-24T17:45:19.630068+0000","last_clean":"2024-01-24T17:45:19.630068+0000","last_became_active":"2024-01-24T17:45:18.407022+0000","last_became_peered":"2024-01-24T17:45:18.407022+0000","last_unstale":"2024-01-24T17:45:19.630068+0000","last_undegraded":"2024-01-24T17:45:19.630068+0000","last_fullsized":"2024-01-24T17:45:19.630068+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:01.170973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629935+0000","last_change":"2024-01-24T17:45:18.414712+0000","last_active":"2024-01-24T17:45:19.629935+0000","last_peered":"2024-01-24T17:45:19.629935+0000","last_clean":"2024-01-24T17:45:19.629935+0000","last_became_active":"2024-01-24T17:45:18.414419+0000","last_became_peered":"2024-01-24T17:45:18.414419+0000","last_unstale":"2024-01-24T17:45:19.629935+0000","last_undegraded":"2024-01-24T17:45:19.629935+0000","last_fullsized":"2024-01-24T17:45:19.629935+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:58:02.429970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633006+0000","last_change":"2024-01-24T17:45:18.400760+0000","last_active":"2024-01-24T17:45:19.633006+0000","last_peered":"2024-01-24T17:45:19.633006+0000","last_clean":"2024-01-24T17:45:19.633006+0000","last_became_active":"2024-01-24T17:45:18.400461+0000","last_became_peered":"2024-01-24T17:45:18.400461+0000","last_unstale":"2024-01-24T17:45:19.633006+0000","last_undegraded":"2024-01-24T17:45:19.633006+0000","last_fullsized":"2024-01-24T17:45:19.633006+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:05:01.096215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396455+0000","last_change":"2024-01-24T17:45:18.429868+0000","last_active":"2024-01-24T17:45:19.396455+0000","last_peered":"2024-01-24T17:45:19.396455+0000","last_clean":"2024-01-24T17:45:19.396455+0000","last_became_active":"2024-01-24T17:45:18.429573+0000","last_became_peered":"2024-01-24T17:45:18.429573+0000","last_unstale":"2024-01-24T17:45:19.396455+0000","last_undegraded":"2024-01-24T17:45:19.396455+0000","last_fullsized":"2024-01-24T17:45:19.396455+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:04.335225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628022+0000","last_change":"2024-01-24T17:45:18.431055+0000","last_active":"2024-01-24T17:45:19.628022+0000","last_peered":"2024-01-24T17:45:19.628022+0000","last_clean":"2024-01-24T17:45:19.628022+0000","last_became_active":"2024-01-24T17:45:18.430791+0000","last_became_peered":"2024-01-24T17:45:18.430791+0000","last_unstale":"2024-01-24T17:45:19.628022+0000","last_undegraded":"2024-01-24T17:45:19.628022+0000","last_fullsized":"2024-01-24T17:45:19.628022+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:05:11.682782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397283+0000","last_change":"2024-01-24T17:45:18.414027+0000","last_active":"2024-01-24T17:45:19.397283+0000","last_peered":"2024-01-24T17:45:19.397283+0000","last_clean":"2024-01-24T17:45:19.397283+0000","last_became_active":"2024-01-24T17:45:18.413452+0000","last_became_peered":"2024-01-24T17:45:18.413452+0000","last_unstale":"2024-01-24T17:45:19.397283+0000","last_undegraded":"2024-01-24T17:45:19.397283+0000","last_fullsized":"2024-01-24T17:45:19.397283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:39:17.384315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"62'4","reported_seq":15,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.434749+0000","last_change":"2024-01-24T17:45:17.919800+0000","last_active":"2024-01-24T17:45:18.434749+0000","last_peered":"2024-01-24T17:45:18.434749+0000","last_clean":"2024-01-24T17:45:18.434749+0000","last_became_active":"2024-01-24T17:45:17.918994+0000","last_became_peered":"2024-01-24T17:45:17.918994+0000","last_unstale":"2024-01-24T17:45:18.434749+0000","last_undegraded":"2024-01-24T17:45:18.434749+0000","last_fullsized":"2024-01-24T17:45:18.434749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397389+0000","last_change":"2024-01-24T17:45:18.437398+0000","last_active":"2024-01-24T17:45:19.397389+0000","last_peered":"2024-01-24T17:45:19.397389+0000","last_clean":"2024-01-24T17:45:19.397389+0000","last_became_active":"2024-01-24T17:45:18.437095+0000","last_became_peered":"2024-01-24T17:45:18.437095+0000","last_unstale":"2024-01-24T17:45:19.397389+0000","last_undegraded":"2024-01-24T17:45:19.397389+0000","last_fullsized":"2024-01-24T17:45:19.397389+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:13.657421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.436986+0000","last_change":"2024-01-24T17:45:17.919211+0000","last_active":"2024-01-24T17:45:18.436986+0000","last_peered":"2024-01-24T17:45:18.436986+0000","last_clean":"2024-01-24T17:45:18.436986+0000","last_became_active":"2024-01-24T17:45:17.918942+0000","last_became_peered":"2024-01-24T17:45:17.918942+0000","last_unstale":"2024-01-24T17:45:18.436986+0000","last_undegraded":"2024-01-24T17:45:18.436986+0000","last_fullsized":"2024-01-24T17:45:18.436986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:49:04.288996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630291+0000","last_change":"2024-01-24T17:45:18.424280+0000","last_active":"2024-01-24T17:45:19.630291+0000","last_peered":"2024-01-24T17:45:19.630291+0000","last_clean":"2024-01-24T17:45:19.630291+0000","last_became_active":"2024-01-24T17:45:18.422789+0000","last_became_peered":"2024-01-24T17:45:18.422789+0000","last_unstale":"2024-01-24T17:45:19.630291+0000","last_undegraded":"2024-01-24T17:45:19.630291+0000","last_fullsized":"2024-01-24T17:45:19.630291+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:23.267943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.438612+0000","last_change":"2024-01-24T17:45:17.922512+0000","last_active":"2024-01-24T17:45:18.438612+0000","last_peered":"2024-01-24T17:45:18.438612+0000","last_clean":"2024-01-24T17:45:18.438612+0000","last_became_active":"2024-01-24T17:45:17.920772+0000","last_became_peered":"2024-01-24T17:45:17.920772+0000","last_unstale":"2024-01-24T17:45:18.438612+0000","last_undegraded":"2024-01-24T17:45:18.438612+0000","last_fullsized":"2024-01-24T17:45:18.438612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:05:02.184752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396198+0000","last_change":"2024-01-24T17:45:18.420475+0000","last_active":"2024-01-24T17:45:19.396198+0000","last_peered":"2024-01-24T17:45:19.396198+0000","last_clean":"2024-01-24T17:45:19.396198+0000","last_became_active":"2024-01-24T17:45:18.419965+0000","last_became_peered":"2024-01-24T17:45:18.419965+0000","last_unstale":"2024-01-24T17:45:19.396198+0000","last_undegraded":"2024-01-24T17:45:19.396198+0000","last_fullsized":"2024-01-24T17:45:19.396198+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:46:56.750634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.429670+0000","last_change":"2024-01-24T17:45:17.925679+0000","last_active":"2024-01-24T17:45:18.429670+0000","last_peered":"2024-01-24T17:45:18.429670+0000","last_clean":"2024-01-24T17:45:18.429670+0000","last_became_active":"2024-01-24T17:45:17.925384+0000","last_became_peered":"2024-01-24T17:45:17.925384+0000","last_unstale":"2024-01-24T17:45:18.429670+0000","last_undegraded":"2024-01-24T17:45:18.429670+0000","last_fullsized":"2024-01-24T17:45:18.429670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:41:31.769012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629796+0000","last_change":"2024-01-24T17:45:18.429943+0000","last_active":"2024-01-24T17:45:19.629796+0000","last_peered":"2024-01-24T17:45:19.629796+0000","last_clean":"2024-01-24T17:45:19.629796+0000","last_became_active":"2024-01-24T17:45:18.429477+0000","last_became_peered":"2024-01-24T17:45:18.429477+0000","last_unstale":"2024-01-24T17:45:19.629796+0000","last_undegraded":"2024-01-24T17:45:19.629796+0000","last_fullsized":"2024-01-24T17:45:19.629796+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:32:36.644421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1b","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.440256+0000","last_change":"2024-01-24T17:45:17.923006+0000","last_active":"2024-01-24T17:45:18.440256+0000","last_peered":"2024-01-24T17:45:18.440256+0000","last_clean":"2024-01-24T17:45:18.440256+0000","last_became_active":"2024-01-24T17:45:17.922289+0000","last_became_peered":"2024-01-24T17:45:17.922289+0000","last_unstale":"2024-01-24T17:45:18.440256+0000","last_undegraded":"2024-01-24T17:45:18.440256+0000","last_fullsized":"2024-01-24T17:45:18.440256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:16.899578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628295+0000","last_change":"2024-01-24T17:45:18.428697+0000","last_active":"2024-01-24T17:45:19.628295+0000","last_peered":"2024-01-24T17:45:19.628295+0000","last_clean":"2024-01-24T17:45:19.628295+0000","last_became_active":"2024-01-24T17:45:18.428426+0000","last_became_peered":"2024-01-24T17:45:18.428426+0000","last_unstale":"2024-01-24T17:45:19.628295+0000","last_undegraded":"2024-01-24T17:45:19.628295+0000","last_fullsized":"2024-01-24T17:45:19.628295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:49:33.451268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.435349+0000","last_change":"2024-01-24T17:45:17.923077+0000","last_active":"2024-01-24T17:45:18.435349+0000","last_peered":"2024-01-24T17:45:18.435349+0000","last_clean":"2024-01-24T17:45:18.435349+0000","last_became_active":"2024-01-24T17:45:17.922774+0000","last_became_peered":"2024-01-24T17:45:17.922774+0000","last_unstale":"2024-01-24T17:45:18.435349+0000","last_undegraded":"2024-01-24T17:45:18.435349+0000","last_fullsized":"2024-01-24T17:45:18.435349+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:26.416880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.628862+0000","last_change":"2024-01-24T17:45:18.418235+0000","last_active":"2024-01-24T17:45:19.628862+0000","last_peered":"2024-01-24T17:45:19.628862+0000","last_clean":"2024-01-24T17:45:19.628862+0000","last_became_active":"2024-01-24T17:45:18.417811+0000","last_became_peered":"2024-01-24T17:45:18.417811+0000","last_unstale":"2024-01-24T17:45:19.628862+0000","last_undegraded":"2024-01-24T17:45:19.628862+0000","last_fullsized":"2024-01-24T17:45:19.628862+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:19:01.175473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.408175+0000","last_change":"2024-01-24T17:45:17.912250+0000","last_active":"2024-01-24T17:45:18.408175+0000","last_peered":"2024-01-24T17:45:18.408175+0000","last_clean":"2024-01-24T17:45:18.408175+0000","last_became_active":"2024-01-24T17:45:17.911960+0000","last_became_peered":"2024-01-24T17:45:17.911960+0000","last_unstale":"2024-01-24T17:45:18.408175+0000","last_undegraded":"2024-01-24T17:45:18.408175+0000","last_fullsized":"2024-01-24T17:45:18.408175+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:34:38.987748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.395540+0000","last_change":"2024-01-24T17:45:18.410406+0000","last_active":"2024-01-24T17:45:19.395540+0000","last_peered":"2024-01-24T17:45:19.395540+0000","last_clean":"2024-01-24T17:45:19.395540+0000","last_became_active":"2024-01-24T17:45:18.409406+0000","last_became_peered":"2024-01-24T17:45:18.409406+0000","last_unstale":"2024-01-24T17:45:19.395540+0000","last_undegraded":"2024-01-24T17:45:19.395540+0000","last_fullsized":"2024-01-24T17:45:19.395540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:45.817758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.390230+0000","last_change":"2024-01-24T17:45:17.916659+0000","last_active":"2024-01-24T17:45:18.390230+0000","last_peered":"2024-01-24T17:45:18.390230+0000","last_clean":"2024-01-24T17:45:18.390230+0000","last_became_active":"2024-01-24T17:45:17.916394+0000","last_became_peered":"2024-01-24T17:45:17.916394+0000","last_unstale":"2024-01-24T17:45:18.390230+0000","last_undegraded":"2024-01-24T17:45:18.390230+0000","last_fullsized":"2024-01-24T17:45:18.390230+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631547+0000","last_change":"2024-01-24T17:45:18.431208+0000","last_active":"2024-01-24T17:45:19.631547+0000","last_peered":"2024-01-24T17:45:19.631547+0000","last_clean":"2024-01-24T17:45:19.631547+0000","last_became_active":"2024-01-24T17:45:18.430737+0000","last_became_peered":"2024-01-24T17:45:18.430737+0000","last_unstale":"2024-01-24T17:45:19.631547+0000","last_undegraded":"2024-01-24T17:45:19.631547+0000","last_fullsized":"2024-01-24T17:45:19.631547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:04.335225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.409308+0000","last_change":"2024-01-24T17:45:17.912678+0000","last_active":"2024-01-24T17:45:18.409308+0000","last_peered":"2024-01-24T17:45:18.409308+0000","last_clean":"2024-01-24T17:45:18.409308+0000","last_became_active":"2024-01-24T17:45:17.912008+0000","last_became_peered":"2024-01-24T17:45:17.912008+0000","last_unstale":"2024-01-24T17:45:18.409308+0000","last_undegraded":"2024-01-24T17:45:18.409308+0000","last_fullsized":"2024-01-24T17:45:18.409308+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:30:30.289664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633552+0000","last_change":"2024-01-24T17:45:18.423553+0000","last_active":"2024-01-24T17:45:19.633552+0000","last_peered":"2024-01-24T17:45:19.633552+0000","last_clean":"2024-01-24T17:45:19.633552+0000","last_became_active":"2024-01-24T17:45:18.420777+0000","last_became_peered":"2024-01-24T17:45:18.420777+0000","last_unstale":"2024-01-24T17:45:19.633552+0000","last_undegraded":"2024-01-24T17:45:19.633552+0000","last_fullsized":"2024-01-24T17:45:19.633552+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:23.692981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.398732+0000","last_change":"2024-01-24T17:45:17.922147+0000","last_active":"2024-01-24T17:45:18.398732+0000","last_peered":"2024-01-24T17:45:18.398732+0000","last_clean":"2024-01-24T17:45:18.398732+0000","last_became_active":"2024-01-24T17:45:17.921599+0000","last_became_peered":"2024-01-24T17:45:17.921599+0000","last_unstale":"2024-01-24T17:45:18.398732+0000","last_undegraded":"2024-01-24T17:45:18.398732+0000","last_fullsized":"2024-01-24T17:45:18.398732+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:12.547194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630533+0000","last_change":"2024-01-24T17:45:18.426483+0000","last_active":"2024-01-24T17:45:19.630533+0000","last_peered":"2024-01-24T17:45:19.630533+0000","last_clean":"2024-01-24T17:45:19.630533+0000","last_became_active":"2024-01-24T17:45:18.420858+0000","last_became_peered":"2024-01-24T17:45:18.420858+0000","last_unstale":"2024-01-24T17:45:19.630533+0000","last_undegraded":"2024-01-24T17:45:19.630533+0000","last_fullsized":"2024-01-24T17:45:19.630533+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:22:32.072368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"62'3","reported_seq":14,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.428360+0000","last_change":"2024-01-24T17:45:17.925319+0000","last_active":"2024-01-24T17:45:18.428360+0000","last_peered":"2024-01-24T17:45:18.428360+0000","last_clean":"2024-01-24T17:45:18.428360+0000","last_became_active":"2024-01-24T17:45:17.925026+0000","last_became_peered":"2024-01-24T17:45:17.925026+0000","last_unstale":"2024-01-24T17:45:18.428360+0000","last_undegraded":"2024-01-24T17:45:18.428360+0000","last_fullsized":"2024-01-24T17:45:18.428360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:22.676649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.632142+0000","last_change":"2024-01-24T17:45:18.409641+0000","last_active":"2024-01-24T17:45:19.632142+0000","last_peered":"2024-01-24T17:45:19.632142+0000","last_clean":"2024-01-24T17:45:19.632142+0000","last_became_active":"2024-01-24T17:45:18.409314+0000","last_became_peered":"2024-01-24T17:45:18.409314+0000","last_unstale":"2024-01-24T17:45:19.632142+0000","last_undegraded":"2024-01-24T17:45:19.632142+0000","last_fullsized":"2024-01-24T17:45:19.632142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:42:55.776505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"62'6","reported_seq":17,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.441892+0000","last_change":"2024-01-24T17:45:17.924919+0000","last_active":"2024-01-24T17:45:18.441892+0000","last_peered":"2024-01-24T17:45:18.441892+0000","last_clean":"2024-01-24T17:45:18.441892+0000","last_became_active":"2024-01-24T17:45:17.924653+0000","last_became_peered":"2024-01-24T17:45:17.924653+0000","last_unstale":"2024-01-24T17:45:18.441892+0000","last_undegraded":"2024-01-24T17:45:18.441892+0000","last_fullsized":"2024-01-24T17:45:18.441892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394970+0000","last_change":"2024-01-24T17:45:18.424098+0000","last_active":"2024-01-24T17:45:19.394970+0000","last_peered":"2024-01-24T17:45:19.394970+0000","last_clean":"2024-01-24T17:45:19.394970+0000","last_became_active":"2024-01-24T17:45:18.423052+0000","last_became_peered":"2024-01-24T17:45:18.423052+0000","last_unstale":"2024-01-24T17:45:19.394970+0000","last_undegraded":"2024-01-24T17:45:19.394970+0000","last_fullsized":"2024-01-24T17:45:19.394970+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:18.926779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.440287+0000","last_change":"2024-01-24T17:45:17.921223+0000","last_active":"2024-01-24T17:45:18.440287+0000","last_peered":"2024-01-24T17:45:18.440287+0000","last_clean":"2024-01-24T17:45:18.440287+0000","last_became_active":"2024-01-24T17:45:17.919530+0000","last_became_peered":"2024-01-24T17:45:17.919530+0000","last_unstale":"2024-01-24T17:45:18.440287+0000","last_undegraded":"2024-01-24T17:45:18.440287+0000","last_fullsized":"2024-01-24T17:45:18.440287+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:17:44.801426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629876+0000","last_change":"2024-01-24T17:45:18.414202+0000","last_active":"2024-01-24T17:45:19.629876+0000","last_peered":"2024-01-24T17:45:19.629876+0000","last_clean":"2024-01-24T17:45:19.629876+0000","last_became_active":"2024-01-24T17:45:18.413959+0000","last_became_peered":"2024-01-24T17:45:18.413959+0000","last_unstale":"2024-01-24T17:45:19.629876+0000","last_undegraded":"2024-01-24T17:45:19.629876+0000","last_fullsized":"2024-01-24T17:45:19.629876+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:29:15.124598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.396399+0000","last_change":"2024-01-24T17:45:17.916370+0000","last_active":"2024-01-24T17:45:18.396399+0000","last_peered":"2024-01-24T17:45:18.396399+0000","last_clean":"2024-01-24T17:45:18.396399+0000","last_became_active":"2024-01-24T17:45:17.916091+0000","last_became_peered":"2024-01-24T17:45:17.916091+0000","last_unstale":"2024-01-24T17:45:18.396399+0000","last_undegraded":"2024-01-24T17:45:18.396399+0000","last_fullsized":"2024-01-24T17:45:18.396399+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:58:00.154641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394743+0000","last_change":"2024-01-24T17:45:18.412289+0000","last_active":"2024-01-24T17:45:19.394743+0000","last_peered":"2024-01-24T17:45:19.394743+0000","last_clean":"2024-01-24T17:45:19.394743+0000","last_became_active":"2024-01-24T17:45:18.412031+0000","last_became_peered":"2024-01-24T17:45:18.412031+0000","last_unstale":"2024-01-24T17:45:19.394743+0000","last_undegraded":"2024-01-24T17:45:19.394743+0000","last_fullsized":"2024-01-24T17:45:19.394743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:39:31.796345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.396585+0000","last_change":"2024-01-24T17:45:17.914759+0000","last_active":"2024-01-24T17:45:18.396585+0000","last_peered":"2024-01-24T17:45:18.396585+0000","last_clean":"2024-01-24T17:45:18.396585+0000","last_became_active":"2024-01-24T17:45:17.914448+0000","last_became_peered":"2024-01-24T17:45:17.914448+0000","last_unstale":"2024-01-24T17:45:18.396585+0000","last_undegraded":"2024-01-24T17:45:18.396585+0000","last_fullsized":"2024-01-24T17:45:18.396585+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:32.202115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.631963+0000","last_change":"2024-01-24T17:45:18.426059+0000","last_active":"2024-01-24T17:45:19.631963+0000","last_peered":"2024-01-24T17:45:19.631963+0000","last_clean":"2024-01-24T17:45:19.631963+0000","last_became_active":"2024-01-24T17:45:18.425160+0000","last_became_peered":"2024-01-24T17:45:18.425160+0000","last_unstale":"2024-01-24T17:45:19.631963+0000","last_undegraded":"2024-01-24T17:45:19.631963+0000","last_fullsized":"2024-01-24T17:45:19.631963+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:54.914651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"62'8","reported_seq":19,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.431993+0000","last_change":"2024-01-24T17:45:17.919777+0000","last_active":"2024-01-24T17:45:18.431993+0000","last_peered":"2024-01-24T17:45:18.431993+0000","last_clean":"2024-01-24T17:45:18.431993+0000","last_became_active":"2024-01-24T17:45:17.919426+0000","last_became_peered":"2024-01-24T17:45:17.919426+0000","last_unstale":"2024-01-24T17:45:18.431993+0000","last_undegraded":"2024-01-24T17:45:18.431993+0000","last_fullsized":"2024-01-24T17:45:18.431993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:36.108099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633391+0000","last_change":"2024-01-24T17:45:18.431490+0000","last_active":"2024-01-24T17:45:19.633391+0000","last_peered":"2024-01-24T17:45:19.633391+0000","last_clean":"2024-01-24T17:45:19.633391+0000","last_became_active":"2024-01-24T17:45:18.429776+0000","last_became_peered":"2024-01-24T17:45:18.429776+0000","last_unstale":"2024-01-24T17:45:19.633391+0000","last_undegraded":"2024-01-24T17:45:19.633391+0000","last_fullsized":"2024-01-24T17:45:19.633391+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:22:30.250177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.439375+0000","last_change":"2024-01-24T17:45:17.922557+0000","last_active":"2024-01-24T17:45:18.439375+0000","last_peered":"2024-01-24T17:45:18.439375+0000","last_clean":"2024-01-24T17:45:18.439375+0000","last_became_active":"2024-01-24T17:45:17.921938+0000","last_became_peered":"2024-01-24T17:45:17.921938+0000","last_unstale":"2024-01-24T17:45:18.439375+0000","last_undegraded":"2024-01-24T17:45:18.439375+0000","last_fullsized":"2024-01-24T17:45:18.439375+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:28:50.106780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633829+0000","last_change":"2024-01-24T17:45:18.423319+0000","last_active":"2024-01-24T17:45:19.633829+0000","last_peered":"2024-01-24T17:45:19.633829+0000","last_clean":"2024-01-24T17:45:19.633829+0000","last_became_active":"2024-01-24T17:45:18.420403+0000","last_became_peered":"2024-01-24T17:45:18.420403+0000","last_unstale":"2024-01-24T17:45:19.633829+0000","last_undegraded":"2024-01-24T17:45:19.633829+0000","last_fullsized":"2024-01-24T17:45:19.633829+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:33.218446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.436502+0000","last_change":"2024-01-24T17:45:17.920941+0000","last_active":"2024-01-24T17:45:18.436502+0000","last_peered":"2024-01-24T17:45:18.436502+0000","last_clean":"2024-01-24T17:45:18.436502+0000","last_became_active":"2024-01-24T17:45:17.920695+0000","last_became_peered":"2024-01-24T17:45:17.920695+0000","last_unstale":"2024-01-24T17:45:18.436502+0000","last_undegraded":"2024-01-24T17:45:18.436502+0000","last_fullsized":"2024-01-24T17:45:18.436502+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:03.155346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397112+0000","last_change":"2024-01-24T17:45:18.431983+0000","last_active":"2024-01-24T17:45:19.397112+0000","last_peered":"2024-01-24T17:45:19.397112+0000","last_clean":"2024-01-24T17:45:19.397112+0000","last_became_active":"2024-01-24T17:45:18.431708+0000","last_became_peered":"2024-01-24T17:45:18.431708+0000","last_unstale":"2024-01-24T17:45:19.397112+0000","last_undegraded":"2024-01-24T17:45:19.397112+0000","last_fullsized":"2024-01-24T17:45:19.397112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:18:13.563526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.d","version":"62'1","reported_seq":12,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.438350+0000","last_change":"2024-01-24T17:45:17.922564+0000","last_active":"2024-01-24T17:45:18.438350+0000","last_peered":"2024-01-24T17:45:18.438350+0000","last_clean":"2024-01-24T17:45:18.438350+0000","last_became_active":"2024-01-24T17:45:17.921964+0000","last_became_peered":"2024-01-24T17:45:17.921964+0000","last_unstale":"2024-01-24T17:45:18.438350+0000","last_undegraded":"2024-01-24T17:45:18.438350+0000","last_fullsized":"2024-01-24T17:45:18.438350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:33:38.088746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.632387+0000","last_change":"2024-01-24T17:45:18.409031+0000","last_active":"2024-01-24T17:45:19.632387+0000","last_peered":"2024-01-24T17:45:19.632387+0000","last_clean":"2024-01-24T17:45:19.632387+0000","last_became_active":"2024-01-24T17:45:18.408776+0000","last_became_peered":"2024-01-24T17:45:18.408776+0000","last_unstale":"2024-01-24T17:45:19.632387+0000","last_undegraded":"2024-01-24T17:45:19.632387+0000","last_fullsized":"2024-01-24T17:45:19.632387+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:10:00.077325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.398988+0000","last_change":"2024-01-24T17:45:17.916140+0000","last_active":"2024-01-24T17:45:18.398988+0000","last_peered":"2024-01-24T17:45:18.398988+0000","last_clean":"2024-01-24T17:45:18.398988+0000","last_became_active":"2024-01-24T17:45:17.915075+0000","last_became_peered":"2024-01-24T17:45:17.915075+0000","last_unstale":"2024-01-24T17:45:18.398988+0000","last_undegraded":"2024-01-24T17:45:18.398988+0000","last_fullsized":"2024-01-24T17:45:18.398988+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:23:00.228501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394675+0000","last_change":"2024-01-24T17:45:18.414477+0000","last_active":"2024-01-24T17:45:19.394675+0000","last_peered":"2024-01-24T17:45:19.394675+0000","last_clean":"2024-01-24T17:45:19.394675+0000","last_became_active":"2024-01-24T17:45:18.414238+0000","last_became_peered":"2024-01-24T17:45:18.414238+0000","last_unstale":"2024-01-24T17:45:19.394675+0000","last_undegraded":"2024-01-24T17:45:19.394675+0000","last_fullsized":"2024-01-24T17:45:19.394675+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:27.433212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.435767+0000","last_change":"2024-01-24T17:45:17.921634+0000","last_active":"2024-01-24T17:45:18.435767+0000","last_peered":"2024-01-24T17:45:18.435767+0000","last_clean":"2024-01-24T17:45:18.435767+0000","last_became_active":"2024-01-24T17:45:17.920658+0000","last_became_peered":"2024-01-24T17:45:17.920658+0000","last_unstale":"2024-01-24T17:45:18.435767+0000","last_undegraded":"2024-01-24T17:45:18.435767+0000","last_fullsized":"2024-01-24T17:45:18.435767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629969+0000","last_change":"2024-01-24T17:45:18.424893+0000","last_active":"2024-01-24T17:45:19.629969+0000","last_peered":"2024-01-24T17:45:19.629969+0000","last_clean":"2024-01-24T17:45:19.629969+0000","last_became_active":"2024-01-24T17:45:18.424613+0000","last_became_peered":"2024-01-24T17:45:18.424613+0000","last_unstale":"2024-01-24T17:45:19.629969+0000","last_undegraded":"2024-01-24T17:45:19.629969+0000","last_fullsized":"2024-01-24T17:45:19.629969+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:25:02.027211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.395532+0000","last_change":"2024-01-24T17:45:17.920125+0000","last_active":"2024-01-24T17:45:18.395532+0000","last_peered":"2024-01-24T17:45:18.395532+0000","last_clean":"2024-01-24T17:45:18.395532+0000","last_became_active":"2024-01-24T17:45:17.919831+0000","last_became_peered":"2024-01-24T17:45:17.919831+0000","last_unstale":"2024-01-24T17:45:18.395532+0000","last_undegraded":"2024-01-24T17:45:18.395532+0000","last_fullsized":"2024-01-24T17:45:18.395532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:03.318893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629603+0000","last_change":"2024-01-24T17:45:18.406061+0000","last_active":"2024-01-24T17:45:19.629603+0000","last_peered":"2024-01-24T17:45:19.629603+0000","last_clean":"2024-01-24T17:45:19.629603+0000","last_became_active":"2024-01-24T17:45:18.404842+0000","last_became_peered":"2024-01-24T17:45:18.404842+0000","last_unstale":"2024-01-24T17:45:19.629603+0000","last_undegraded":"2024-01-24T17:45:19.629603+0000","last_fullsized":"2024-01-24T17:45:19.629603+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:38:37.124431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.407605+0000","last_change":"2024-01-24T17:45:17.908034+0000","last_active":"2024-01-24T17:45:18.407605+0000","last_peered":"2024-01-24T17:45:18.407605+0000","last_clean":"2024-01-24T17:45:18.407605+0000","last_became_active":"2024-01-24T17:45:17.907755+0000","last_became_peered":"2024-01-24T17:45:17.907755+0000","last_unstale":"2024-01-24T17:45:18.407605+0000","last_undegraded":"2024-01-24T17:45:18.407605+0000","last_fullsized":"2024-01-24T17:45:18.407605+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:21:59.679470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633058+0000","last_change":"2024-01-24T17:45:18.420321+0000","last_active":"2024-01-24T17:45:19.633058+0000","last_peered":"2024-01-24T17:45:19.633058+0000","last_clean":"2024-01-24T17:45:19.633058+0000","last_became_active":"2024-01-24T17:45:18.420048+0000","last_became_peered":"2024-01-24T17:45:18.420048+0000","last_unstale":"2024-01-24T17:45:19.633058+0000","last_undegraded":"2024-01-24T17:45:19.633058+0000","last_fullsized":"2024-01-24T17:45:19.633058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:13:50.434214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.439378+0000","last_change":"2024-01-24T17:45:17.922727+0000","last_active":"2024-01-24T17:45:18.439378+0000","last_peered":"2024-01-24T17:45:18.439378+0000","last_clean":"2024-01-24T17:45:18.439378+0000","last_became_active":"2024-01-24T17:45:17.922288+0000","last_became_peered":"2024-01-24T17:45:17.922288+0000","last_unstale":"2024-01-24T17:45:18.439378+0000","last_undegraded":"2024-01-24T17:45:18.439378+0000","last_fullsized":"2024-01-24T17:45:18.439378+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:53:47.067777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.633893+0000","last_change":"2024-01-24T17:45:18.424353+0000","last_active":"2024-01-24T17:45:19.633893+0000","last_peered":"2024-01-24T17:45:19.633893+0000","last_clean":"2024-01-24T17:45:19.633893+0000","last_became_active":"2024-01-24T17:45:18.424056+0000","last_became_peered":"2024-01-24T17:45:18.424056+0000","last_unstale":"2024-01-24T17:45:19.633893+0000","last_undegraded":"2024-01-24T17:45:19.633893+0000","last_fullsized":"2024-01-24T17:45:19.633893+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:03.006906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"63'1","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397509+0000","last_change":"2024-01-24T17:45:17.920500+0000","last_active":"2024-01-24T17:45:19.397509+0000","last_peered":"2024-01-24T17:45:19.397509+0000","last_clean":"2024-01-24T17:45:19.397509+0000","last_became_active":"2024-01-24T17:45:17.920207+0000","last_became_peered":"2024-01-24T17:45:17.920207+0000","last_unstale":"2024-01-24T17:45:19.397509+0000","last_undegraded":"2024-01-24T17:45:19.397509+0000","last_fullsized":"2024-01-24T17:45:19.397509+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:17.910448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.397363+0000","last_change":"2024-01-24T17:45:18.436099+0000","last_active":"2024-01-24T17:45:19.397363+0000","last_peered":"2024-01-24T17:45:19.397363+0000","last_clean":"2024-01-24T17:45:19.397363+0000","last_became_active":"2024-01-24T17:45:18.435809+0000","last_became_peered":"2024-01-24T17:45:18.435809+0000","last_unstale":"2024-01-24T17:45:19.397363+0000","last_undegraded":"2024-01-24T17:45:19.397363+0000","last_fullsized":"2024-01-24T17:45:19.397363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:31:04.171678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.408606+0000","last_change":"2024-01-24T17:45:17.923999+0000","last_active":"2024-01-24T17:45:18.408606+0000","last_peered":"2024-01-24T17:45:18.408606+0000","last_clean":"2024-01-24T17:45:18.408606+0000","last_became_active":"2024-01-24T17:45:17.923727+0000","last_became_peered":"2024-01-24T17:45:17.923727+0000","last_unstale":"2024-01-24T17:45:18.408606+0000","last_undegraded":"2024-01-24T17:45:18.408606+0000","last_fullsized":"2024-01-24T17:45:18.408606+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:35:45.998203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630604+0000","last_change":"2024-01-24T17:45:18.406857+0000","last_active":"2024-01-24T17:45:19.630604+0000","last_peered":"2024-01-24T17:45:19.630604+0000","last_clean":"2024-01-24T17:45:19.630604+0000","last_became_active":"2024-01-24T17:45:18.404804+0000","last_became_peered":"2024-01-24T17:45:18.404804+0000","last_unstale":"2024-01-24T17:45:19.630604+0000","last_undegraded":"2024-01-24T17:45:19.630604+0000","last_fullsized":"2024-01-24T17:45:19.630604+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:33.218446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.406718+0000","last_change":"2024-01-24T17:45:17.917767+0000","last_active":"2024-01-24T17:45:18.406718+0000","last_peered":"2024-01-24T17:45:18.406718+0000","last_clean":"2024-01-24T17:45:18.406718+0000","last_became_active":"2024-01-24T17:45:17.917485+0000","last_became_peered":"2024-01-24T17:45:17.917485+0000","last_unstale":"2024-01-24T17:45:18.406718+0000","last_undegraded":"2024-01-24T17:45:18.406718+0000","last_fullsized":"2024-01-24T17:45:18.406718+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:15:59.199419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396273+0000","last_change":"2024-01-24T17:45:18.437810+0000","last_active":"2024-01-24T17:45:19.396273+0000","last_peered":"2024-01-24T17:45:19.396273+0000","last_clean":"2024-01-24T17:45:19.396273+0000","last_became_active":"2024-01-24T17:45:18.437208+0000","last_became_peered":"2024-01-24T17:45:18.437208+0000","last_unstale":"2024-01-24T17:45:19.396273+0000","last_undegraded":"2024-01-24T17:45:19.396273+0000","last_fullsized":"2024-01-24T17:45:19.396273+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:17.915909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.422057+0000","last_change":"2024-01-24T17:45:17.925427+0000","last_active":"2024-01-24T17:45:18.422057+0000","last_peered":"2024-01-24T17:45:18.422057+0000","last_clean":"2024-01-24T17:45:18.422057+0000","last_became_active":"2024-01-24T17:45:17.925156+0000","last_became_peered":"2024-01-24T17:45:17.925156+0000","last_unstale":"2024-01-24T17:45:18.422057+0000","last_undegraded":"2024-01-24T17:45:18.422057+0000","last_fullsized":"2024-01-24T17:45:18.422057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630813+0000","last_change":"2024-01-24T17:45:18.409314+0000","last_active":"2024-01-24T17:45:19.630813+0000","last_peered":"2024-01-24T17:45:19.630813+0000","last_clean":"2024-01-24T17:45:19.630813+0000","last_became_active":"2024-01-24T17:45:18.409031+0000","last_became_peered":"2024-01-24T17:45:18.409031+0000","last_unstale":"2024-01-24T17:45:19.630813+0000","last_undegraded":"2024-01-24T17:45:19.630813+0000","last_fullsized":"2024-01-24T17:45:19.630813+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:00:17.915909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.406944+0000","last_change":"2024-01-24T17:45:17.921375+0000","last_active":"2024-01-24T17:45:18.406944+0000","last_peered":"2024-01-24T17:45:18.406944+0000","last_clean":"2024-01-24T17:45:18.406944+0000","last_became_active":"2024-01-24T17:45:17.921094+0000","last_became_peered":"2024-01-24T17:45:17.921094+0000","last_unstale":"2024-01-24T17:45:18.406944+0000","last_undegraded":"2024-01-24T17:45:18.406944+0000","last_fullsized":"2024-01-24T17:45:18.406944+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:09:47.749409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.629511+0000","last_change":"2024-01-24T17:45:18.416295+0000","last_active":"2024-01-24T17:45:19.629511+0000","last_peered":"2024-01-24T17:45:19.629511+0000","last_clean":"2024-01-24T17:45:19.629511+0000","last_became_active":"2024-01-24T17:45:18.415734+0000","last_became_peered":"2024-01-24T17:45:18.415734+0000","last_unstale":"2024-01-24T17:45:19.629511+0000","last_undegraded":"2024-01-24T17:45:19.629511+0000","last_fullsized":"2024-01-24T17:45:19.629511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:23.692981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.407138+0000","last_change":"2024-01-24T17:45:17.921933+0000","last_active":"2024-01-24T17:45:18.407138+0000","last_peered":"2024-01-24T17:45:18.407138+0000","last_clean":"2024-01-24T17:45:18.407138+0000","last_became_active":"2024-01-24T17:45:17.921674+0000","last_became_peered":"2024-01-24T17:45:17.921674+0000","last_unstale":"2024-01-24T17:45:18.407138+0000","last_undegraded":"2024-01-24T17:45:18.407138+0000","last_fullsized":"2024-01-24T17:45:18.407138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:30.797138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.396509+0000","last_change":"2024-01-24T17:45:18.431790+0000","last_active":"2024-01-24T17:45:19.396509+0000","last_peered":"2024-01-24T17:45:19.396509+0000","last_clean":"2024-01-24T17:45:19.396509+0000","last_became_active":"2024-01-24T17:45:18.431405+0000","last_became_peered":"2024-01-24T17:45:18.431405+0000","last_unstale":"2024-01-24T17:45:19.396509+0000","last_undegraded":"2024-01-24T17:45:19.396509+0000","last_fullsized":"2024-01-24T17:45:19.396509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:05:11.682782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.395382+0000","last_change":"2024-01-24T17:45:17.914736+0000","last_active":"2024-01-24T17:45:18.395382+0000","last_peered":"2024-01-24T17:45:18.395382+0000","last_clean":"2024-01-24T17:45:18.395382+0000","last_became_active":"2024-01-24T17:45:17.914309+0000","last_became_peered":"2024-01-24T17:45:17.914309+0000","last_unstale":"2024-01-24T17:45:18.395382+0000","last_undegraded":"2024-01-24T17:45:18.395382+0000","last_fullsized":"2024-01-24T17:45:18.395382+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:01:12.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.630025+0000","last_change":"2024-01-24T17:45:18.425885+0000","last_active":"2024-01-24T17:45:19.630025+0000","last_peered":"2024-01-24T17:45:19.630025+0000","last_clean":"2024-01-24T17:45:19.630025+0000","last_became_active":"2024-01-24T17:45:18.425596+0000","last_became_peered":"2024-01-24T17:45:18.425596+0000","last_unstale":"2024-01-24T17:45:19.630025+0000","last_undegraded":"2024-01-24T17:45:19.630025+0000","last_fullsized":"2024-01-24T17:45:19.630025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:17.864345+0000","last_clean_scrub_stamp":"2024-01-24T17:45:17.864345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:41:32.785344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-01-24T17:45:18.390567+0000","last_change":"2024-01-24T17:45:17.919042+0000","last_active":"2024-01-24T17:45:18.390567+0000","last_peered":"2024-01-24T17:45:18.390567+0000","last_clean":"2024-01-24T17:45:18.390567+0000","last_became_active":"2024-01-24T17:45:17.918689+0000","last_became_peered":"2024-01-24T17:45:17.918689+0000","last_unstale":"2024-01-24T17:45:18.390567+0000","last_undegraded":"2024-01-24T17:45:18.390567+0000","last_fullsized":"2024-01-24T17:45:18.390567+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:45:16.848013+0000","last_clean_scrub_stamp":"2024-01-24T17:45:16.848013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:22:47.636490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-01-24T17:45:19.394899+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:45:19.394899+0000","last_peered":"2024-01-24T17:45:19.394899+0000","last_clean":"2024-01-24T17:45:19.394899+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:45:19.394899+0000","last_undegraded":"2024-01-24T17:45:19.394899+0000","last_fullsized":"2024-01-24T17:45:19.394899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":765952,"data_stored":320907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.05,"15min":1.0249999999999999},"min":{"1min":0.80300000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.048,"15min":1.0329999999999999},"min":{"1min":0.68400000000000005,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.599,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.341}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0860000000000001,"15min":1.1000000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.04,"5min":2.04,"15min":2.04},"last":1.319},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.1859999999999999,"15min":1.1919999999999999},"min":{"1min":0.81499999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.649,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.494}]},{"osd":2,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.171,"15min":1.1579999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.764,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.181,"15min":1.2190000000000001},"min":{"1min":0.745,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.484,"5min":2.484,"15min":2.484},"last":1.2549999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.8779999999999999,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.282,"15min":1.3100000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.6269999999999998,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.3660000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.296,"15min":1.3200000000000001},"min":{"1min":0.84699999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.1190000000000002,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5449999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.468}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.319,"15min":1.3520000000000001},"min":{"1min":0.70799999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.133},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.3979999999999999,"15min":1.4350000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.966,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.1759999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.4990000000000001,"15min":1.5349999999999999},"min":{"1min":0.72599999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.5720000000000001,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.427},{"interface":"front","average":{"1min":1.399,"5min":1.472,"15min":1.488},"min":{"1min":0.86099999999999999,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.2450000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.22}]}]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":761856,"data_stored":320837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.034,"15min":1.024},"min":{"1min":0.70699999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.6919999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.141,"5min":1.044,"15min":1.0189999999999999},"min":{"1min":0.749,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.82699999999999996}]},{"osd":1,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.022,"15min":0.99199999999999999},"min":{"1min":0.70199999999999996,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4890000000000001,"5min":1.73,"15min":1.73},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1359999999999999,"15min":1.1180000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.722,"5min":1.972,"15min":1.972},"last":0.68200000000000005}]},{"osd":2,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.79300000000000004,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5369999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.169,"15min":1.167},"min":{"1min":0.90700000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.514,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.93899999999999995}]},{"osd":3,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.242,"15min":1.254},"min":{"1min":0.83799999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.542,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.046},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.2310000000000001,"15min":1.2529999999999999},"min":{"1min":0.84199999999999997,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.5980000000000001,"5min":2.46,"15min":2.46},"last":0.872}]},{"osd":4,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.2569999999999999,"15min":1.2370000000000001},"min":{"1min":0.76000000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.661,"5min":1.988,"15min":1.988},"last":1.095},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2989999999999999,"15min":1.3029999999999999},"min":{"1min":0.73899999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.8420000000000001,"5min":1.849,"15min":1.849},"last":1.228}]},{"osd":5,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3080000000000001,"15min":1.296},"min":{"1min":0.97999999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.645,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4259999999999999,"15min":1.4350000000000001},"min":{"1min":0.89800000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.9319999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.274}]},{"osd":7,"last update":"Wed Jan 24 17:44:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.355,"15min":1.367},"min":{"1min":0.85399999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.978,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.454,"15min":1.4630000000000001},"min":{"1min":1.008,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.899,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.202}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1355776,"data_stored":911217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0669999999999999,"15min":1.099},"min":{"1min":0.67500000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.47,"5min":1.508,"15min":1.508},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.998,"5min":1.0289999999999999,"15min":1.0309999999999999},"min":{"1min":0.67900000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5,"5min":1.889,"15min":1.889},"last":1.0920000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.077,"15min":1.0740000000000001},"min":{"1min":0.72899999999999998,"5min":0.622,"15min":0.622},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.22,"15min":1.264},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.552,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.0349999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.2230000000000001,"15min":1.2290000000000001},"min":{"1min":0.80600000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6539999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.23,"5min":1.28,"15min":1.3120000000000001},"min":{"1min":0.79800000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.9870000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.514}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2909999999999999,"15min":1.341},"min":{"1min":0.77800000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.6020000000000001,"5min":3.036,"15min":3.036},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.3080000000000001,"15min":1.3320000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.641,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.1339999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4339999999999999,"15min":1.4530000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.915,"5min":2.069,"15min":2.069},"last":1.224},{"interface":"front","average":{"1min":1.206,"5min":1.329,"15min":1.3600000000000001},"min":{"1min":0.77600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5780000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.383}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.339,"15min":1.3560000000000001},"min":{"1min":0.91100000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.724,"5min":2.504,"15min":2.504},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":0.88600000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.044,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.593}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.458,"15min":1.4930000000000001},"min":{"1min":0.63800000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.6719999999999999,"5min":3.161,"15min":3.161},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.5900000000000001,"15min":1.6339999999999999},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.1389999999999998,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.6519999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":765952,"data_stored":321431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.048,"15min":1.038},"min":{"1min":0.73499999999999999,"5min":0.621,"15min":0.621},"max":{"1min":1.673,"5min":1.698,"15min":1.698},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.093,"15min":1.0960000000000001},"min":{"1min":0.71199999999999997,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5529999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.0029999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.0800000000000001,"15min":1.052},"min":{"1min":0.78500000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5049999999999999,"5min":1.633,"15min":1.633},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1899999999999999,"15min":1.2170000000000001},"min":{"1min":0.67800000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.4139999999999999,"5min":1.849,"15min":1.849},"last":1.302}]},{"osd":2,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.206,"15min":1.212},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":1.054},{"interface":"front","average":{"1min":1.202,"5min":1.304,"15min":1.3340000000000001},"min":{"1min":0.71199999999999997,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.724,"5min":1.986,"15min":1.986},"last":0.81399999999999995}]},{"osd":3,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1850000000000001,"15min":1.1910000000000001},"min":{"1min":0.81000000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.456,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.105},{"interface":"front","average":{"1min":1.319,"5min":1.343,"15min":1.351},"min":{"1min":0.89400000000000002,"5min":0.752,"15min":0.752},"max":{"1min":1.8029999999999999,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.3320000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.3480000000000001,"15min":1.373},"min":{"1min":1.014,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.613,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.298,"15min":1.306},"min":{"1min":0.85799999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.5600000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":0.90100000000000002}]},{"osd":6,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4550000000000001,"15min":1.4730000000000001},"min":{"1min":0.93799999999999994,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.3159999999999998,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.385,"5min":1.4610000000000001,"15min":1.4790000000000001},"min":{"1min":0.86899999999999999,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.468}]},{"osd":7,"last update":"Wed Jan 24 17:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.4330000000000001,"15min":1.4259999999999999},"min":{"1min":0.92000000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0209999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.208},{"interface":"front","average":{"1min":1.512,"5min":1.6000000000000001,"15min":1.621},"min":{"1min":0.97299999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.2309999999999999,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":1.425}]}]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":761856,"data_stored":320905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.91600000000000004,"15min":0.872},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.792,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95499999999999996,"15min":0.89100000000000001},"min":{"1min":0.64200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.5680000000000001,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":1.347}]},{"osd":1,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.93899999999999995,"15min":0.95099999999999996},"min":{"1min":0.53100000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.819,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0720000000000001,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.663,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.157}]},{"osd":2,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.105,"15min":1.111},"min":{"1min":0.629,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.571,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.181,"15min":1.1639999999999999},"min":{"1min":0.70799999999999996,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.593,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":1.4239999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3200000000000001,"15min":1.292},"min":{"1min":0.90800000000000003,"5min":0.877,"15min":0.877},"max":{"1min":1.8240000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.9790000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.218}]},{"osd":5,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.4350000000000001,"15min":1.4370000000000001},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":2.2189999999999999,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.4159999999999999,"15min":1.4239999999999999},"min":{"1min":1.024,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.0579999999999998,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.2989999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.532,"15min":1.5029999999999999},"min":{"1min":1.0429999999999999,"5min":0.871,"15min":0.871},"max":{"1min":1.984,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4730000000000001,"15min":1.4630000000000001},"min":{"1min":0.90200000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.1120000000000001,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.6200000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:25 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.5389999999999999,"15min":1.5169999999999999},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.012,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.542,"15min":1.5249999999999999},"min":{"1min":1.0649999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.1160000000000001,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.4750000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":778240,"data_stored":322539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.439,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.24,"5min":1.3140000000000001,"15min":1.3140000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93600000000000005,"15min":0.93999999999999995},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.7989999999999999,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.78500000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0369999999999999,"15min":1.0549999999999999},"min":{"1min":0.51300000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.5509999999999999,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.173,"15min":1.2150000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.66900000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.98199999999999998,"15min":0.996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6910000000000001,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2050000000000001,"15min":1.218},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.7629999999999999,"5min":2.996,"15min":2.996},"last":0.72799999999999998}]},{"osd":4,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.306,"15min":1.329},"min":{"1min":0.72899999999999998,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.6950000000000001,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4099999999999999},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":1.7,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.1339999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3600000000000001,"15min":1.409},"min":{"1min":0.73999999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.762,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2969999999999999,"15min":1.331},"min":{"1min":0.86299999999999999,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.625,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.92500000000000004}]},{"osd":6,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.369,"15min":1.4019999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9350000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.4399999999999999,"15min":1.4830000000000001},"min":{"1min":0.80200000000000005,"5min":0.753,"15min":0.753},"max":{"1min":2.2530000000000001,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":0.85899999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:44:26 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.526,"15min":1.5860000000000001},"min":{"1min":0.82499999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.0209999999999999,"5min":3.343,"15min":3.343},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4630000000000001,"15min":1.512},"min":{"1min":0.755,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9510000000000001,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.0069999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1355776,"data_stored":911765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.874,"15min":0.85999999999999999},"min":{"1min":0.45500000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4419999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.115},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.93000000000000005,"15min":0.95299999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.155,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.36399999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.95399999999999996,"15min":0.98399999999999999},"min":{"1min":0.51800000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.4790000000000001,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5680000000000001,"5min":1.895,"15min":1.895},"last":1.5960000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.143,"15min":1.1639999999999999},"min":{"1min":0.61299999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.4930000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.194,"15min":1.244},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.4019999999999999,"5min":3.149,"15min":3.149},"last":1.246}]},{"osd":4,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.353,"15min":1.365},"min":{"1min":0.88700000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.77,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.296,"15min":1.3240000000000001},"min":{"1min":0.83599999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.698,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.3200000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.323,"15min":1.357},"min":{"1min":0.73499999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.806,"5min":2.577,"15min":2.577},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.871,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.9730000000000001,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":1.3999999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.4019999999999999,"15min":1.4450000000000001},"min":{"1min":0.80000000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.653,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.637},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.46,"15min":1.5309999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.921,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.675}]},{"osd":7,"last update":"Wed Jan 24 17:44:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4570000000000001,"15min":1.4750000000000001},"min":{"1min":0.91400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.0979999999999999,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.5049999999999999,"15min":1.5549999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.641,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.4339999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574900,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":770048,"data_stored":322379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80000000000000004,"15min":0.81799999999999995},"min":{"1min":0.35899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.242,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.875,"5min":0.93000000000000005,"15min":0.96999999999999997},"min":{"1min":0.52900000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"last":1.1299999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.84199999999999997,"15min":0.83399999999999996},"min":{"1min":0.45200000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2589999999999999,"5min":1.294,"15min":1.294},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.997,"5min":1.046,"15min":1.1000000000000001},"min":{"1min":0.46800000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4430000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":1.0249999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.1359999999999999,"15min":1.1890000000000001},"min":{"1min":0.51000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.639,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.165,"15min":1.1879999999999999},"min":{"1min":0.60699999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6040000000000001,"5min":2.407,"15min":2.407},"last":1.488}]},{"osd":4,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3540000000000001,"15min":1.4179999999999999},"min":{"1min":0.752,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.056,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.4139999999999999,"15min":1.488},"min":{"1min":0.89000000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":1.5840000000000001,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.431}]},{"osd":5,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.333,"15min":1.397},"min":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.7490000000000001,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.256,"15min":1.264},"min":{"1min":0.85599999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.694,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.375}]},{"osd":6,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.337,"15min":1.3600000000000001},"min":{"1min":0.93500000000000005,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.704,"5min":2.335,"15min":2.335},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.4139999999999999,"15min":1.46},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.877,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.5660000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:44:30 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.462,"15min":1.534},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.694,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.534},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.5369999999999999,"15min":1.605},"min":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":1.806,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.643}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:45:28.465 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-24T17:45:28.465 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-24T17:45:28.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.739+0000 7f8cb5c00640 1 Processor -- start 2024-01-24T17:45:28.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.740+0000 7f8cb5c00640 1 -- start start 2024-01-24T17:45:28.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.741+0000 7f8cb5c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8cb015a6e0 0x7f8cb015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.741+0000 7f8cb5c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb006fd80 con 0x7f8cb0071e70 2024-01-24T17:45:28.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.741+0000 7f8cb5c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb0070020 con 0x7f8cb00cb970 2024-01-24T17:45:28.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.741+0000 7f8cb5c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb015aac0 con 0x7f8cb015a6e0 2024-01-24T17:45:28.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.742+0000 7f8cafe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8cb015a6e0 0x7f8cb015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.742+0000 7f8ca7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8cb0071e70 0x7f8cb00cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:36770/0 (socket says 172.21.15.39:36770) 2024-01-24T17:45:28.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.742+0000 7f8ca7e00640 1 -- 172.21.15.39:0/1834552175 learned_addr learned my addr 172.21.15.39:0/1834552175 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:28.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.742+0000 7f8caf400640 1 -- 172.21.15.39:0/1834552175 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833705501 0 0) 0x7f8cb006fd80 con 0x7f8cb0071e70 2024-01-24T17:45:28.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.742+0000 7f8caf400640 1 -- 172.21.15.39:0/1834552175 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c90003520 con 0x7f8cb0071e70 2024-01-24T17:45:28.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.742+0000 7f8cafe00640 1 -- 172.21.15.39:0/1834552175 >> v1:172.21.15.100:6789/0 conn(0x7f8cb00cb970 legacy=0x7f8cb00c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.742+0000 7f8cafe00640 1 -- 172.21.15.39:0/1834552175 >> v1:172.21.15.39:6789/0 conn(0x7f8cb0071e70 legacy=0x7f8cb00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.742+0000 7f8cafe00640 1 -- 172.21.15.39:0/1834552175 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb0070300 con 0x7f8cb015a6e0 2024-01-24T17:45:28.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.744+0000 7f8cafe00640 1 --2- 172.21.15.39:0/1834552175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb015a6e0 0x7f8cb015eb50 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00097a0 tx=0x7f8ca002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d93ac2c1dd31f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.744+0000 7f8caf400640 1 -- 172.21.15.39:0/1834552175 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0030610 con 0x7f8cb015a6e0 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.744+0000 7f8caf400640 1 -- 172.21.15.39:0/1834552175 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca0030770 con 0x7f8cb015a6e0 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.744+0000 7f8caf400640 1 -- 172.21.15.39:0/1834552175 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0030b00 con 0x7f8cb015a6e0 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.745+0000 7f8cb5c00640 1 -- 172.21.15.39:0/1834552175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb015a6e0 msgr2=0x7f8cb015eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.745+0000 7f8cb5c00640 1 --2- 172.21.15.39:0/1834552175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb015a6e0 0x7f8cb015eb50 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00097a0 tx=0x7f8ca002b960 comp rx=0 tx=0).stop 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.745+0000 7f8cb5c00640 1 -- 172.21.15.39:0/1834552175 shutdown_connections 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.745+0000 7f8cb5c00640 1 --2- 172.21.15.39:0/1834552175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb015a6e0 0x7f8cb015eb50 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.745+0000 7f8cb5c00640 1 -- 172.21.15.39:0/1834552175 >> 172.21.15.39:0/1834552175 conn(0x7f8cb006a090 msgr2=0x7f8cb006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.745+0000 7f8cb5c00640 1 -- 172.21.15.39:0/1834552175 shutdown_connections 2024-01-24T17:45:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.745+0000 7f8cb5c00640 1 -- 172.21.15.39:0/1834552175 wait complete. 2024-01-24T17:45:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.747+0000 7f8cb5c00640 1 Processor -- start 2024-01-24T17:45:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.747+0000 7f8cb5c00640 1 -- start start 2024-01-24T17:45:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cb5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb00cb970 0x7f8cb01f2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cb5c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb015a6e0 0x7f8cb015d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cb5c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01f33f0 0x7f8cb00afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cb5c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb015d220 con 0x7f8cb00cb970 2024-01-24T17:45:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cb5c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01f28a0 con 0x7f8cb01f33f0 2024-01-24T17:45:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cb5c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb01f2b80 con 0x7f8cb015a6e0 2024-01-24T17:45:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cafe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb00cb970 0x7f8cb01f2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cafe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb00cb970 0x7f8cb01f2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47942/0 (socket says 172.21.15.39:47942) 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cafe00640 1 -- 172.21.15.39:0/790985398 learned_addr learned my addr 172.21.15.39:0/790985398 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cb5200640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01f33f0 0x7f8cb00afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8ca7e00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb015a6e0 0x7f8cb015d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.748+0000 7f8cafe00640 1 -- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb015a6e0 msgr2=0x7f8cb015d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cafe00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb015a6e0 0x7f8cb015d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cafe00640 1 -- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01f33f0 msgr2=0x7f8cb00afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cafe00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01f33f0 0x7f8cb00afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cafe00640 1 -- 172.21.15.39:0/790985398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca0009400 con 0x7f8cb00cb970 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cb5200640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01f33f0 0x7f8cb00afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8ca7e00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb015a6e0 0x7f8cb015d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cafe00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb00cb970 0x7f8cb01f2230 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8ca002bf70 tx=0x7f8ca00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cacc00640 1 -- 172.21.15.39:0/790985398 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca003e070 con 0x7f8cb00cb970 2024-01-24T17:45:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cacc00640 1 -- 172.21.15.39:0/790985398 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca0030e10 con 0x7f8cb00cb970 2024-01-24T17:45:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cacc00640 1 -- 172.21.15.39:0/790985398 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0039c30 con 0x7f8cb00cb970 2024-01-24T17:45:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb01f2e60 con 0x7f8cb00cb970 2024-01-24T17:45:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.749+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb00ada70 con 0x7f8cb00cb970 2024-01-24T17:45:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.751+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb004dff0 con 0x7f8cb00cb970 2024-01-24T17:45:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.751+0000 7f8cacc00640 1 -- 172.21.15.39:0/790985398 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8ca004b400 con 0x7f8cb00cb970 2024-01-24T17:45:28.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.752+0000 7f8cacc00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8806a430 0x7f8c8806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:28.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.752+0000 7f8cacc00640 1 -- 172.21.15.39:0/790985398 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ca00aac40 con 0x7f8cb00cb970 2024-01-24T17:45:28.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.753+0000 7f8ca7e00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8806a430 0x7f8c8806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.754+0000 7f8ca7e00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8806a430 0x7f8c8806c8f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00cd8a0 tx=0x7f8c98009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:28.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:28.756+0000 7f8cacc00640 1 -- 172.21.15.39:0/790985398 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8ca0076220 con 0x7f8cb00cb970 2024-01-24T17:45:29.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.024+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8cb015e490 con 0x7f8cb00cb970 2024-01-24T17:45:29.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.031+0000 7f8cacc00640 1 -- 172.21.15.39:0/790985398 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8ca0079840 con 0x7f8cb00cb970 2024-01-24T17:45:29.033 INFO:teuthology.orchestra.run.smithi039.stderr:updated caps for client.0 2024-01-24T17:45:29.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.034+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8806a430 msgr2=0x7f8c8806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.034+0000 7f8cb5c00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8806a430 0x7f8c8806c8f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00cd8a0 tx=0x7f8c98009040 comp rx=0 tx=0).stop 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.034+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb00cb970 msgr2=0x7f8cb01f2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.034+0000 7f8cb5c00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb00cb970 0x7f8cb01f2230 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8ca002bf70 tx=0x7f8ca00046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.035+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 shutdown_connections 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.035+0000 7f8cb5c00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb01f33f0 0x7f8cb00afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.035+0000 7f8cb5c00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8cb015a6e0 0x7f8cb015d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.035+0000 7f8cb5c00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c8806a430 0x7f8c8806c8f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.035+0000 7f8cb5c00640 1 --2- 172.21.15.39:0/790985398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8cb00cb970 0x7f8cb01f2230 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.035+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 >> 172.21.15.39:0/790985398 conn(0x7f8cb006a090 msgr2=0x7f8cb006a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.035+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 shutdown_connections 2024-01-24T17:45:29.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.035+0000 7f8cb5c00640 1 -- 172.21.15.39:0/790985398 wait complete. 2024-01-24T17:45:29.054 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:45:29.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.270+0000 7fc666800640 1 Processor -- start 2024-01-24T17:45:29.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.270+0000 7fc666800640 1 -- start start 2024-01-24T17:45:29.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.271+0000 7fc666800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc66015a720 0x7fc6600cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:29.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.271+0000 7fc666800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6600ce8e0 0x7fc6600ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:29.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.271+0000 7fc666800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6600cf260 con 0x7fc66015ab00 2024-01-24T17:45:29.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.271+0000 7fc666800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6600ce400 con 0x7fc6600ce8e0 2024-01-24T17:45:29.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.271+0000 7fc666800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6600ccef0 con 0x7fc66015a720 2024-01-24T17:45:29.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.272+0000 7fc665e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6600ce8e0 0x7fc6600ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:29.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.272+0000 7fc665e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6600ce8e0 0x7fc6600ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47954/0 (socket says 172.21.15.39:47954) 2024-01-24T17:45:29.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.272+0000 7fc665e00640 1 -- 172.21.15.39:0/2380148731 learned_addr learned my addr 172.21.15.39:0/2380148731 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:29.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.272+0000 7fc665400640 1 --2- 172.21.15.39:0/2380148731 >> v2:172.21.15.100:3300/0 conn(0x7fc66015a720 0x7fc6600cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:29.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.272+0000 7fc665e00640 1 -- 172.21.15.39:0/2380148731 >> v2:172.21.15.100:3300/0 conn(0x7fc66015a720 msgr2=0x7fc6600cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:29.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.272+0000 7fc665e00640 1 --2- 172.21.15.39:0/2380148731 >> v2:172.21.15.100:3300/0 conn(0x7fc66015a720 0x7fc6600cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.272+0000 7fc665e00640 1 -- 172.21.15.39:0/2380148731 >> v1:172.21.15.100:6789/0 conn(0x7fc66015ab00 legacy=0x7fc6600cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:29.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.272+0000 7fc665e00640 1 -- 172.21.15.39:0/2380148731 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6600ce110 con 0x7fc6600ce8e0 2024-01-24T17:45:29.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.273+0000 7fc665e00640 1 --2- 172.21.15.39:0/2380148731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6600ce8e0 0x7fc6600ced20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc66006c3e0 tx=0x7fc65802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf726f41e605c0a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:29.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.274+0000 7fc65fe00640 1 -- 172.21.15.39:0/2380148731 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65802c9f0 con 0x7fc6600ce8e0 2024-01-24T17:45:29.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.274+0000 7fc65fe00640 1 -- 172.21.15.39:0/2380148731 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65802cb50 con 0x7fc6600ce8e0 2024-01-24T17:45:29.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.274+0000 7fc65fe00640 1 -- 172.21.15.39:0/2380148731 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580343f0 con 0x7fc6600ce8e0 2024-01-24T17:45:29.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.274+0000 7fc666800640 1 -- 172.21.15.39:0/2380148731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6600ce8e0 msgr2=0x7fc6600ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:29.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.274+0000 7fc666800640 1 --2- 172.21.15.39:0/2380148731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6600ce8e0 0x7fc6600ced20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc66006c3e0 tx=0x7fc65802be50 comp rx=0 tx=0).stop 2024-01-24T17:45:29.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.275+0000 7fc666800640 1 -- 172.21.15.39:0/2380148731 shutdown_connections 2024-01-24T17:45:29.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.275+0000 7fc666800640 1 --2- 172.21.15.39:0/2380148731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6600ce8e0 0x7fc6600ced20 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.275+0000 7fc666800640 1 --2- 172.21.15.39:0/2380148731 >> v2:172.21.15.100:3300/0 conn(0x7fc66015a720 0x7fc6600cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.275+0000 7fc666800640 1 -- 172.21.15.39:0/2380148731 >> 172.21.15.39:0/2380148731 conn(0x7fc66006a090 msgr2=0x7fc66006a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:29.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.275+0000 7fc666800640 1 -- 172.21.15.39:0/2380148731 shutdown_connections 2024-01-24T17:45:29.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.275+0000 7fc666800640 1 -- 172.21.15.39:0/2380148731 wait complete. 2024-01-24T17:45:29.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.276+0000 7fc666800640 1 Processor -- start 2024-01-24T17:45:29.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.277+0000 7fc666800640 1 -- start start 2024-01-24T17:45:29.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.277+0000 7fc666800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6600ce8e0 0x7fc6600b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc666800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc66015a720 0x7fc6600c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc666800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc66015ab00 0x7fc6601efa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc666800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600c4e00 con 0x7fc66015ab00 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc666800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600b1d30 con 0x7fc6600ce8e0 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc666800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6600ae050 con 0x7fc66015a720 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc66015ab00 0x7fc6601efa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc66015ab00 0x7fc6601efa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47960/0 (socket says 172.21.15.39:47960) 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665e00640 1 -- 172.21.15.39:0/388436760 learned_addr learned my addr 172.21.15.39:0/388436760 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665400640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6600ce8e0 0x7fc6600b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665e00640 1 -- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc66015a720 msgr2=0x7fc6600c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665e00640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc66015a720 0x7fc6600c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665e00640 1 -- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6600ce8e0 msgr2=0x7fc6600b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:29.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665e00640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6600ce8e0 0x7fc6600b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665e00640 1 -- 172.21.15.39:0/388436760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc658009b30 con 0x7fc66015ab00 2024-01-24T17:45:29.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.278+0000 7fc665400640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6600ce8e0 0x7fc6600b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:29.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.279+0000 7fc665e00640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc66015ab00 0x7fc6601efa40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc658036040 tx=0x7fc658004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:29.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.279+0000 7fc65d600640 1 -- 172.21.15.39:0/388436760 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658009510 con 0x7fc66015ab00 2024-01-24T17:45:29.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.279+0000 7fc65d600640 1 -- 172.21.15.39:0/388436760 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65803e450 con 0x7fc66015ab00 2024-01-24T17:45:29.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.279+0000 7fc65d600640 1 -- 172.21.15.39:0/388436760 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65803e7e0 con 0x7fc66015ab00 2024-01-24T17:45:29.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.279+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6601eff80 con 0x7fc66015ab00 2024-01-24T17:45:29.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.279+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6600b1770 con 0x7fc66015ab00 2024-01-24T17:45:29.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.280+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6600ae200 con 0x7fc66015ab00 2024-01-24T17:45:29.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.281+0000 7fc65d600640 1 -- 172.21.15.39:0/388436760 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc65803e940 con 0x7fc66015ab00 2024-01-24T17:45:29.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.282+0000 7fc65d600640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc63806a4b0 0x7fc63806c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:29.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.282+0000 7fc65d600640 1 -- 172.21.15.39:0/388436760 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6580ab270 con 0x7fc66015ab00 2024-01-24T17:45:29.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.283+0000 7fc665400640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc63806a4b0 0x7fc63806c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:29.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.284+0000 7fc665400640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc63806a4b0 0x7fc63806c970 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc650006fd0 tx=0x7fc650008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:29.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.287+0000 7fc65d600640 1 -- 172.21.15.39:0/388436760 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc6600ae200 con 0x7fc66015ab00 2024-01-24T17:45:29.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.522+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6600ae1c0 con 0x7fc66015ab00 2024-01-24T17:45:29.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.523+0000 7fc65d600640 1 -- 172.21.15.39:0/388436760 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fc6580769d0 con 0x7fc66015ab00 2024-01-24T17:45:29.525 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:29.525 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3045325513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3045325513},{"type":"v1","addr":"172.21.15.100:6836","nonce":3045325513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5126,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/858868211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":858868211},{"type":"v1","addr":"172.21.15.100:6837","nonce":858868211}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3121826441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3121826441},{"type":"v1","addr":"172.21.15.39:6833","nonce":3121826441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:45:18.381640+0000","modified":"2024-01-24T17:45:21.392053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6261":{"gid":6261,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/527429915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":527429915},{"type":"v1","addr":"172.21.15.39:6835","nonce":527429915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-24T17:45:29.526 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-01-24T17:45:29.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc63806a4b0 msgr2=0x7fc63806c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:29.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc63806a4b0 0x7fc63806c970 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc650006fd0 tx=0x7fc650008040 comp rx=0 tx=0).stop 2024-01-24T17:45:29.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc66015ab00 msgr2=0x7fc6601efa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:29.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc66015ab00 0x7fc6601efa40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc658036040 tx=0x7fc658004a40 comp rx=0 tx=0).stop 2024-01-24T17:45:29.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 shutdown_connections 2024-01-24T17:45:29.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc63806a4b0 0x7fc63806c970 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc66015ab00 0x7fc6601efa40 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc66015a720 0x7fc6600c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 --2- 172.21.15.39:0/388436760 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6600ce8e0 0x7fc6600b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:29.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 >> 172.21.15.39:0/388436760 conn(0x7fc66006a090 msgr2=0x7fc66006a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:29.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 shutdown_connections 2024-01-24T17:45:29.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:29.526+0000 7fc666800640 1 -- 172.21.15.39:0/388436760 wait complete. 2024-01-24T17:45:29.950 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:45:18.381640+0000', 'modified': '2024-01-24T17:45:21.392053+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6261}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6261': {'gid': 6261, 'name': 'c', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6835/527429915', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6834', 'nonce': 527429915}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 527429915}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:45:29.950 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T17:45:29.950 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-24T17:45:30.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.183+0000 7fb5be400640 1 Processor -- start 2024-01-24T17:45:30.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.183+0000 7fb5be400640 1 -- start start 2024-01-24T17:45:30.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.184+0000 7fb5be400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb5b80cb910 0x7fb5b80cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.184+0000 7fb5be400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b815d310 con 0x7fb5b80c8a80 2024-01-24T17:45:30.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.184+0000 7fb5be400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b80cbcf0 con 0x7fb5b80cae70 2024-01-24T17:45:30.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.184+0000 7fb5be400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b815deb0 con 0x7fb5b80cb910 2024-01-24T17:45:30.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.185+0000 7fb5bd000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb5b80cb910 0x7fb5b80cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.185+0000 7fb5bda00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb5b80c8a80 0x7fb5b8160480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:36782/0 (socket says 172.21.15.39:36782) 2024-01-24T17:45:30.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.185+0000 7fb5bda00640 1 -- 172.21.15.39:0/2016045002 learned_addr learned my addr 172.21.15.39:0/2016045002 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:30.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.185+0000 7fb5b7e00640 1 -- 172.21.15.39:0/2016045002 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764357147 0 0) 0x7fb5b815d310 con 0x7fb5b80c8a80 2024-01-24T17:45:30.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.185+0000 7fb5b7e00640 1 -- 172.21.15.39:0/2016045002 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598003520 con 0x7fb5b80c8a80 2024-01-24T17:45:30.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.185+0000 7fb5bd000640 1 -- 172.21.15.39:0/2016045002 >> v1:172.21.15.100:6789/0 conn(0x7fb5b80cae70 legacy=0x7fb5b815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.185+0000 7fb5bd000640 1 -- 172.21.15.39:0/2016045002 >> v1:172.21.15.39:6789/0 conn(0x7fb5b80c8a80 legacy=0x7fb5b8160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.185+0000 7fb5bd000640 1 -- 172.21.15.39:0/2016045002 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b80ce730 con 0x7fb5b80cb910 2024-01-24T17:45:30.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.186+0000 7fb5bd000640 1 --2- 172.21.15.39:0/2016045002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80cb910 0x7fb5b80cde30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80097a0 tx=0x7fb5a802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4308a832cec7d02 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.187+0000 7fb5b7e00640 1 -- 172.21.15.39:0/2016045002 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8030610 con 0x7fb5b80cb910 2024-01-24T17:45:30.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.187+0000 7fb5b7e00640 1 -- 172.21.15.39:0/2016045002 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a8030770 con 0x7fb5b80cb910 2024-01-24T17:45:30.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.187+0000 7fb5b7e00640 1 -- 172.21.15.39:0/2016045002 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8030b00 con 0x7fb5b80cb910 2024-01-24T17:45:30.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.187+0000 7fb5be400640 1 -- 172.21.15.39:0/2016045002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80cb910 msgr2=0x7fb5b80cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.188+0000 7fb5be400640 1 --2- 172.21.15.39:0/2016045002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80cb910 0x7fb5b80cde30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80097a0 tx=0x7fb5a802b960 comp rx=0 tx=0).stop 2024-01-24T17:45:30.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.188+0000 7fb5be400640 1 -- 172.21.15.39:0/2016045002 shutdown_connections 2024-01-24T17:45:30.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.188+0000 7fb5be400640 1 --2- 172.21.15.39:0/2016045002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80cb910 0x7fb5b80cde30 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.188+0000 7fb5be400640 1 -- 172.21.15.39:0/2016045002 >> 172.21.15.39:0/2016045002 conn(0x7fb5b806c470 msgr2=0x7fb5b8069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:30.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.188+0000 7fb5be400640 1 -- 172.21.15.39:0/2016045002 shutdown_connections 2024-01-24T17:45:30.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.188+0000 7fb5be400640 1 -- 172.21.15.39:0/2016045002 wait complete. 2024-01-24T17:45:30.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.190+0000 7fb5be400640 1 Processor -- start 2024-01-24T17:45:30.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.190+0000 7fb5be400640 1 -- start start 2024-01-24T17:45:30.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5be400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80c8a80 0x7fb5b80c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5be400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5b80cae70 0x7fb5b80c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5be400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5b80cb910 0x7fb5b80c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5be400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b815dff0 con 0x7fb5b80c8a80 2024-01-24T17:45:30.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5be400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b80c8410 con 0x7fb5b80cae70 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5be400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b80c86f0 con 0x7fb5b80cb910 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80c8a80 0x7fb5b80c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80c8a80 0x7fb5b80c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47980/0 (socket says 172.21.15.39:47980) 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bd000640 1 -- 172.21.15.39:0/2907073447 learned_addr learned my addr 172.21.15.39:0/2907073447 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5afe00640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5b80cae70 0x7fb5b80c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bda00640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5b80cb910 0x7fb5b80c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bd000640 1 -- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5b80cb910 msgr2=0x7fb5b80c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bd000640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5b80cb910 0x7fb5b80c7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bd000640 1 -- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5b80cae70 msgr2=0x7fb5b80c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bd000640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5b80cae70 0x7fb5b80c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5bd000640 1 -- 172.21.15.39:0/2907073447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a8009400 con 0x7fb5b80c8a80 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.191+0000 7fb5afe00640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5b80cae70 0x7fb5b80c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.192+0000 7fb5bd000640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80c8a80 0x7fb5b80c5770 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb5a802bf70 tx=0x7fb5a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.192+0000 7fb5b5600640 1 -- 172.21.15.39:0/2907073447 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a803e070 con 0x7fb5b80c8a80 2024-01-24T17:45:30.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.192+0000 7fb5b5600640 1 -- 172.21.15.39:0/2907073447 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a8030e10 con 0x7fb5b80c8a80 2024-01-24T17:45:30.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.192+0000 7fb5b5600640 1 -- 172.21.15.39:0/2907073447 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8039c30 con 0x7fb5b80c8a80 2024-01-24T17:45:30.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.192+0000 7fb5be400640 1 -- 172.21.15.39:0/2907073447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b815e240 con 0x7fb5b80c8a80 2024-01-24T17:45:30.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.192+0000 7fb5be400640 1 -- 172.21.15.39:0/2907073447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b80c80f0 con 0x7fb5b80c8a80 2024-01-24T17:45:30.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.195+0000 7fb5b5600640 1 -- 172.21.15.39:0/2907073447 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb5a804b400 con 0x7fb5b80c8a80 2024-01-24T17:45:30.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.196+0000 7fb5b5600640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59006a430 0x7fb59006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.196+0000 7fb5b5600640 1 -- 172.21.15.39:0/2907073447 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5a80aad10 con 0x7fb5b80c8a80 2024-01-24T17:45:30.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.197+0000 7fb5afe00640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59006a430 0x7fb59006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.197+0000 7fb5c2817740 1 Processor -- start 2024-01-24T17:45:30.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.197+0000 7fb5c2817740 1 -- start start 2024-01-24T17:45:30.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5afe00640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59006a430 0x7fb59006c8f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8154430 tx=0x7fb5a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 0x55e92328a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92325a120 0x55e923287410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 0x55e923287950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5c2817740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9232356c0 con 0x55e92325a120 2024-01-24T17:45:30.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5c2817740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9231ca070 con 0x55e92325dd20 2024-01-24T17:45:30.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5c2817740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e9231c7800 con 0x55e92328a1a0 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92325a120 0x55e923287410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.198+0000 7fb5afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92325a120 0x55e923287410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47982/0 (socket says 172.21.15.39:47982) 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5afe00640 1 -- 172.21.15.39:0/2465980300 learned_addr learned my addr 172.21.15.39:0/2465980300 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5bda00640 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 0x55e923287950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5bd000640 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 0x55e92328a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5afe00640 1 -- 172.21.15.39:0/2465980300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 msgr2=0x55e92328a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5afe00640 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 0x55e92328a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5afe00640 1 -- 172.21.15.39:0/2465980300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 msgr2=0x55e923287950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5afe00640 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 0x55e923287950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5afe00640 1 -- 172.21.15.39:0/2465980300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e92318a780 con 0x55e92325a120 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.199+0000 7fb5afe00640 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92325a120 0x55e923287410 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000d7d0 tx=0x7fb5a0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e13182e56e72638 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5ad600640 1 -- 172.21.15.39:0/2465980300 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0014ba0 con 0x55e92325a120 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5ad600640 1 -- 172.21.15.39:0/2465980300 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a0014d00 con 0x55e92325a120 2024-01-24T17:45:30.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5ad600640 1 -- 172.21.15.39:0/2465980300 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0007230 con 0x55e92325a120 2024-01-24T17:45:30.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5c2817740 1 -- 172.21.15.39:0/2465980300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92325a120 msgr2=0x55e923287410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5c2817740 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92325a120 0x55e923287410 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000d7d0 tx=0x7fb5a0010000 comp rx=0 tx=0).stop 2024-01-24T17:45:30.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5c2817740 1 -- 172.21.15.39:0/2465980300 shutdown_connections 2024-01-24T17:45:30.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5c2817740 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 0x55e923287950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5c2817740 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92325a120 0x55e923287410 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5c2817740 1 --2- 172.21.15.39:0/2465980300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 0x55e92328a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.200+0000 7fb5c2817740 1 -- 172.21.15.39:0/2465980300 >> 172.21.15.39:0/2465980300 conn(0x55e92325c430 msgr2=0x55e9232a4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:30.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.201+0000 7fb5c2817740 1 -- 172.21.15.39:0/2465980300 shutdown_connections 2024-01-24T17:45:30.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.201+0000 7fb5c2817740 1 -- 172.21.15.39:0/2465980300 wait complete. 2024-01-24T17:45:30.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.201+0000 7fb5c2817740 1 -- start start 2024-01-24T17:45:30.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.201+0000 7fb5c2817740 1 Processor -- start 2024-01-24T17:45:30.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.202+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 0x55e92326e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.202+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328a1a0 0x55e92328d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.202+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328dc80 0x55e92328a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.202+0000 7fb5c2817740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9231c30f0 con 0x55e92328a1a0 2024-01-24T17:45:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.202+0000 7fb5c2817740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9231648c0 con 0x55e92328dc80 2024-01-24T17:45:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.202+0000 7fb5c2817740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e923164560 con 0x55e92325dd20 2024-01-24T17:45:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.202+0000 7fb5c2817740 10 client.0 _ll_register_callbacks cb 0x55e9231bca30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:45:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328a1a0 0x55e92328d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328a1a0 0x55e92328d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47998/0 (socket says 172.21.15.39:47998) 2024-01-24T17:45:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5afe00640 1 -- 172.21.15.39:0/1662205750 learned_addr learned my addr 172.21.15.39:0/1662205750 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5bda00640 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328dc80 0x55e92328a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5afe00640 1 -- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 msgr2=0x55e92326e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5bd000640 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 0x55e92326e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5afe00640 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 0x55e92326e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5afe00640 1 -- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328dc80 msgr2=0x55e92328a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5afe00640 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328dc80 0x55e92328a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5afe00640 1 -- 172.21.15.39:0/1662205750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b80cbcf0 con 0x55e92328a1a0 2024-01-24T17:45:30.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.203+0000 7fb5bda00640 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328dc80 0x55e92328a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:30.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5afe00640 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328a1a0 0x55e92328d740 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0022040 tx=0x7fb5a0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f8d11958ef43ab server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5a6000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:45:30.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5a6000640 1 -- 172.21.15.39:0/1662205750 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00074c0 con 0x55e92328a1a0 2024-01-24T17:45:30.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5a6000640 1 -- 172.21.15.39:0/1662205750 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5a0007620 con 0x55e92328a1a0 2024-01-24T17:45:30.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5c2817740 1 -- 172.21.15.39:0/1662205750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e9231ed640 con 0x55e92328a1a0 2024-01-24T17:45:30.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5a6000640 1 -- 172.21.15.39:0/1662205750 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5a00067d0 con 0x55e92328a1a0 2024-01-24T17:45:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5c2817740 10 client.6420 fetch_fsmap learned FSMap version 34 2024-01-24T17:45:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5c2817740 1 -- 172.21.15.39:0/1662205750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55e9231c1770 con 0x55e92328a1a0 2024-01-24T17:45:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.204+0000 7fb5a6000640 1 -- 172.21.15.39:0/1662205750 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e9231ed640 con 0x55e92328a1a0 2024-01-24T17:45:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5a6000640 1 -- 172.21.15.39:0/1662205750 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb5a0006930 con 0x55e92328a1a0 2024-01-24T17:45:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5c2817740 10 client.6420 fetch_fsmap finished waiting for FSMap version 34 2024-01-24T17:45:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5c2817740 10 client.6420 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:45:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5c2817740 20 client.6420 populate_metadata read hostname 'smithi039' 2024-01-24T17:45:30.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5c2817740 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] conn(0x55e923283d80 0x55e923284180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5c2817740 4 client.6420 mds_command: new command op to 6261 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:45:30.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5c2817740 1 -- 172.21.15.39:0/1662205750 --> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e9231b5940 con 0x55e923283d80 2024-01-24T17:45:30.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5bd000640 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] conn(0x55e923283d80 0x55e923284180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5bd000640 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] conn(0x55e923283d80 0x55e923284180 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=298a0581a2b789d7 server_cookie=445d3f85fa773b75 in_seq=0 out_seq=0 2024-01-24T17:45:30.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.205+0000 7fb5a6000640 10 client.6420 ms_handle_connect on v2:172.21.15.39:6834/527429915 2024-01-24T17:45:30.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.206+0000 7fb5a6000640 1 -- 172.21.15.39:0/1662205750 <== mds.0 v2:172.21.15.39:6834/527429915 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e9231b5940 con 0x55e923283d80 2024-01-24T17:45:30.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.206+0000 7fb5a6000640 10 client.6420 handle_command_reply: tid=0 2024-01-24T17:45:30.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.206+0000 7fb5c2817740 1 client.6420 shutdown 2024-01-24T17:45:30.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.206+0000 7fb5c2817740 1 -- 172.21.15.39:0/1662205750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328a1a0 msgr2=0x55e92328d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.206+0000 7fb5c2817740 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328a1a0 0x55e92328d740 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0022040 tx=0x7fb5a0000ed0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 1 -- 172.21.15.39:0/1662205750 shutdown_connections 2024-01-24T17:45:30.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328dc80 0x55e92328a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] conn(0x55e923283d80 0x55e923284180 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328a1a0 0x55e92328d740 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 1 --2- 172.21.15.39:0/1662205750 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 0x55e92326e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5a6000640 0 client.6420 ms_handle_reset on v2:172.21.15.39:6834/527429915 2024-01-24T17:45:30.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 1 -- 172.21.15.39:0/1662205750 >> 172.21.15.39:0/1662205750 conn(0x55e92325a120 msgr2=0x55e92325e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:30.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 1 -- 172.21.15.39:0/1662205750 shutdown_connections 2024-01-24T17:45:30.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 1 -- 172.21.15.39:0/1662205750 wait complete. 2024-01-24T17:45:30.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.207+0000 7fb5c2817740 20 client.6420 trim_cache size 0 max 16384 2024-01-24T17:45:30.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.223+0000 7fb5c2817740 1 Processor -- start 2024-01-24T17:45:30.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.224+0000 7fb5c2817740 1 -- start start 2024-01-24T17:45:30.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.224+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 0x55e92325a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.224+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328a1a0 0x55e92326db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.224+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e92328e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.224+0000 7fb5c2817740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e922f2af20 con 0x55e92328dc80 2024-01-24T17:45:30.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.224+0000 7fb5c2817740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e92328db20 con 0x55e92328a1a0 2024-01-24T17:45:30.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.224+0000 7fb5c2817740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e9232709e0 con 0x55e92325dd20 2024-01-24T17:45:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e92328e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e92328e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48014/0 (socket says 172.21.15.39:48014) 2024-01-24T17:45:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 -- 172.21.15.39:0/3660841890 learned_addr learned my addr 172.21.15.39:0/3660841890 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5afe00640 1 --2- 172.21.15.39:0/3660841890 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328a1a0 0x55e92326db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 -- 172.21.15.39:0/3660841890 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 msgr2=0x55e92325a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 --2- 172.21.15.39:0/3660841890 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 0x55e92325a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 -- 172.21.15.39:0/3660841890 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328a1a0 msgr2=0x55e92326db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 --2- 172.21.15.39:0/3660841890 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328a1a0 0x55e92326db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 -- 172.21.15.39:0/3660841890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9231f72f0 con 0x55e92328dc80 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.225+0000 7fb5bda00640 1 --2- 172.21.15.39:0/3660841890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e92328e060 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80c8a10 tx=0x7fb5b0001ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39671de1dc1a0782 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5ae000640 1 -- 172.21.15.39:0/3660841890 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b00020f0 con 0x55e92328dc80 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5ae000640 1 -- 172.21.15.39:0/3660841890 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b000d030 con 0x55e92328dc80 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5ae000640 1 -- 172.21.15.39:0/3660841890 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0007860 con 0x55e92328dc80 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5c2817740 1 -- 172.21.15.39:0/3660841890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 msgr2=0x55e92328e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5c2817740 1 --2- 172.21.15.39:0/3660841890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e92328e060 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80c8a10 tx=0x7fb5b0001ea0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5c2817740 1 -- 172.21.15.39:0/3660841890 shutdown_connections 2024-01-24T17:45:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5c2817740 1 --2- 172.21.15.39:0/3660841890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e92328e060 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5c2817740 1 --2- 172.21.15.39:0/3660841890 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92328a1a0 0x55e92326db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5c2817740 1 --2- 172.21.15.39:0/3660841890 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92325dd20 0x55e92325a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.226+0000 7fb5c2817740 1 -- 172.21.15.39:0/3660841890 >> 172.21.15.39:0/3660841890 conn(0x55e92325a120 msgr2=0x55e923284770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.227+0000 7fb5c2817740 1 -- 172.21.15.39:0/3660841890 shutdown_connections 2024-01-24T17:45:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.227+0000 7fb5c2817740 1 -- 172.21.15.39:0/3660841890 wait complete. 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.227+0000 7fb5c2817740 1 -- start start 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.227+0000 7fb5c2817740 1 Processor -- start 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.228+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 0x55e92330b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.228+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 0x55e92330be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.228+0000 7fb5c2817740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e923282960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.228+0000 7fb5c2817740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e92330c820 con 0x55e92328dc80 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.228+0000 7fb5c2817740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9232a8d10 con 0x55e92325dd20 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.228+0000 7fb5c2817740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e92330c380 con 0x55e92328a1a0 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.228+0000 7fb5c2817740 10 client.0 _ll_register_callbacks cb 0x7fb5a0022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e923282960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e923282960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48026/0 (socket says 172.21.15.39:48026) 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 -- 172.21.15.39:0/997387093 learned_addr learned my addr 172.21.15.39:0/997387093 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bd000640 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 0x55e92330b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 -- 172.21.15.39:0/997387093 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 msgr2=0x55e92330be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 0x55e92330be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 -- 172.21.15.39:0/997387093 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 msgr2=0x55e92330b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 0x55e92330b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 -- 172.21.15.39:0/997387093 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b000f040 con 0x55e92328dc80 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5bda00640 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e923282960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0001580 tx=0x7fb5b0001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3963a144125327d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.229+0000 7fb5a6000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5a6000640 1 -- 172.21.15.39:0/997387093 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b00149b0 con 0x55e92328dc80 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5a6000640 1 -- 172.21.15.39:0/997387093 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5b0014b10 con 0x55e92328dc80 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 1 -- 172.21.15.39:0/997387093 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5a0014d00 con 0x55e92328dc80 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5a6000640 1 -- 172.21.15.39:0/997387093 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5b00020f0 con 0x55e92328dc80 2024-01-24T17:45:30.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 10 client.6426 fetch_fsmap learned FSMap version 34 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 1 -- 172.21.15.39:0/997387093 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) v3 -- 0x55e923163ad0 con 0x55e92328dc80 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5a6000640 1 -- 172.21.15.39:0/997387093 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a0014d00 con 0x55e92328dc80 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5a6000640 1 -- 172.21.15.39:0/997387093 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb5b0014cd0 con 0x55e92328dc80 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 10 client.6426 fetch_fsmap finished waiting for FSMap version 34 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 10 client.6426 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 20 client.6426 populate_metadata read hostname 'smithi039' 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] conn(0x55e923282ef0 0x55e9232832d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 4 client.6426 mds_command: new command op to 6261 tid=0[{"prefix": "status"}] 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.230+0000 7fb5c2817740 1 -- 172.21.15.39:0/997387093 --> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e9231bc170 con 0x55e923282ef0 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.231+0000 7fb5bd000640 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] conn(0x55e923282ef0 0x55e9232832d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.231+0000 7fb5bd000640 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] conn(0x55e923282ef0 0x55e9232832d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f15ef924def5bd6b server_cookie=b79a438b032a70b3 in_seq=0 out_seq=0 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.231+0000 7fb5a6000640 10 client.6426 ms_handle_connect on v2:172.21.15.39:6834/527429915 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.231+0000 7fb5a6000640 1 -- 172.21.15.39:0/997387093 <== mds.0 v2:172.21.15.39:6834/527429915 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55e9231bc170 con 0x55e923282ef0 2024-01-24T17:45:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.231+0000 7fb5a6000640 10 client.6426 handle_command_reply: tid=0 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.231+0000 7fb5c2817740 1 client.6426 shutdown 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 -- 172.21.15.39:0/997387093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 msgr2=0x55e923282960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e923282960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0001580 tx=0x7fb5b0001250 comp rx=0 tx=0).stop 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 -- 172.21.15.39:0/997387093 shutdown_connections 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.39:6834/527429915,v1:172.21.15.39:6835/527429915] conn(0x55e923282ef0 0x55e9232832d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e92328dc80 0x55e923282960 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55e92328a1a0 0x55e92330be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 --2- 172.21.15.39:0/997387093 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55e92325dd20 0x55e92330b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5a6000640 0 client.6426 ms_handle_reset on v2:172.21.15.39:6834/527429915 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 -- 172.21.15.39:0/997387093 >> 172.21.15.39:0/997387093 conn(0x55e92325a120 msgr2=0x55e9232b73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 -- 172.21.15.39:0/997387093 shutdown_connections 2024-01-24T17:45:30.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 1 -- 172.21.15.39:0/997387093 wait complete. 2024-01-24T17:45:30.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.232+0000 7fb5c2817740 20 client.6426 trim_cache size 0 max 16384 2024-01-24T17:45:30.238 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:45:30.238 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 6261, 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 11.846879349, 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 34, 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 63, 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 63, 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 17.167892047999999 2024-01-24T17:45:30.239 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:45:30.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 -- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59006a430 msgr2=0x7fb59006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59006a430 0x7fb59006c8f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8154430 tx=0x7fb5a0009040 comp rx=0 tx=0).stop 2024-01-24T17:45:30.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 -- 172.21.15.39:0/2907073447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80c8a80 msgr2=0x7fb5b80c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80c8a80 0x7fb5b80c5770 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb5a802bf70 tx=0x7fb5a80046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 -- 172.21.15.39:0/2907073447 shutdown_connections 2024-01-24T17:45:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5b80cb910 0x7fb5b80c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5b80cae70 0x7fb5b80c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59006a430 0x7fb59006c8f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 --2- 172.21.15.39:0/2907073447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5b80c8a80 0x7fb5b80c5770 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 -- 172.21.15.39:0/2907073447 >> 172.21.15.39:0/2907073447 conn(0x7fb5b806c470 msgr2=0x7fb5b815d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 -- 172.21.15.39:0/2907073447 shutdown_connections 2024-01-24T17:45:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.233+0000 7fb5be400640 1 -- 172.21.15.39:0/2907073447 wait complete. 2024-01-24T17:45:30.252 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:45:30.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.455+0000 7f3ffca00640 1 Processor -- start 2024-01-24T17:45:30.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.456+0000 7f3ffca00640 1 -- start start 2024-01-24T17:45:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.457+0000 7f3ffca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3ff815cbf0 0x7f3ff80d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.457+0000 7f3ffca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff815e010 con 0x7f3ff80d0da0 2024-01-24T17:45:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.457+0000 7f3ffca00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff80d0a90 con 0x7f3ff815db60 2024-01-24T17:45:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.457+0000 7f3ffca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff80cd760 con 0x7f3ff815cbf0 2024-01-24T17:45:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.457+0000 7f3ff7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3ff815cbf0 0x7f3ff80d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.458+0000 7f3ff7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3ff80d0da0 0x7f3ff815e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:36794/0 (socket says 172.21.15.39:36794) 2024-01-24T17:45:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.458+0000 7f3ff7e00640 1 -- 172.21.15.39:0/2274511794 learned_addr learned my addr 172.21.15.39:0/2274511794 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:30.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.458+0000 7f3ff7400640 1 -- 172.21.15.39:0/2274511794 >> v1:172.21.15.100:6789/0 conn(0x7f3ff815db60 legacy=0x7f3ff80ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.458+0000 7f3ff7400640 1 -- 172.21.15.39:0/2274511794 >> v1:172.21.15.39:6789/0 conn(0x7f3ff80d0da0 legacy=0x7f3ff815e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.458+0000 7f3ff7400640 1 -- 172.21.15.39:0/2274511794 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff80cda40 con 0x7f3ff815cbf0 2024-01-24T17:45:30.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.458+0000 7f3ff6000640 1 -- 172.21.15.39:0/2274511794 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380114390 0 0) 0x7f3ff815e010 con 0x7f3ff80d0da0 2024-01-24T17:45:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.460+0000 7f3ff7400640 1 --2- 172.21.15.39:0/2274511794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815cbf0 0x7f3ff80d03b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009db0 tx=0x7f3fe802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aef8b2edd79b30e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.460+0000 7f3ff6000640 1 -- 172.21.15.39:0/2274511794 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe802ccd0 con 0x7f3ff815cbf0 2024-01-24T17:45:30.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.460+0000 7f3ff6000640 1 -- 172.21.15.39:0/2274511794 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe802ce30 con 0x7f3ff815cbf0 2024-01-24T17:45:30.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.460+0000 7f3ff6000640 1 -- 172.21.15.39:0/2274511794 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8034660 con 0x7f3ff815cbf0 2024-01-24T17:45:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.461+0000 7f3ffca00640 1 -- 172.21.15.39:0/2274511794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815cbf0 msgr2=0x7f3ff80d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.461+0000 7f3ffca00640 1 --2- 172.21.15.39:0/2274511794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815cbf0 0x7f3ff80d03b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009db0 tx=0x7f3fe802bf20 comp rx=0 tx=0).stop 2024-01-24T17:45:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.461+0000 7f3ffca00640 1 -- 172.21.15.39:0/2274511794 shutdown_connections 2024-01-24T17:45:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.461+0000 7f3ffca00640 1 --2- 172.21.15.39:0/2274511794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815cbf0 0x7f3ff80d03b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.461+0000 7f3ffca00640 1 -- 172.21.15.39:0/2274511794 >> 172.21.15.39:0/2274511794 conn(0x7f3ff809bb90 msgr2=0x7f3ff80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.461+0000 7f3ffca00640 1 -- 172.21.15.39:0/2274511794 shutdown_connections 2024-01-24T17:45:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.461+0000 7f3ffca00640 1 -- 172.21.15.39:0/2274511794 wait complete. 2024-01-24T17:45:30.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.463+0000 7f3ffca00640 1 Processor -- start 2024-01-24T17:45:30.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.463+0000 7f3ffca00640 1 -- start start 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ffca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ff80d0da0 0x7f3ff80c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ffca00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ff815cbf0 0x7f3ff8153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ffca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815db60 0x7f3ff8156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ffca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80c6730 con 0x7f3ff815db60 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ffca00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff81574b0 con 0x7f3ff815cbf0 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ffca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80c69d0 con 0x7f3ff80d0da0 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ff7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815db60 0x7f3ff8156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ff7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815db60 0x7f3ff8156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48052/0 (socket says 172.21.15.39:48052) 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.464+0000 7f3ff7e00640 1 -- 172.21.15.39:0/1449459299 learned_addr learned my addr 172.21.15.39:0/1449459299 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.465+0000 7f3ff6a00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ff815cbf0 0x7f3ff8153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.465+0000 7f3ff7400640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ff80d0da0 0x7f3ff80c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.465+0000 7f3ff7e00640 1 -- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ff80d0da0 msgr2=0x7f3ff80c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.465+0000 7f3ff7e00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ff80d0da0 0x7f3ff80c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.465+0000 7f3ff7e00640 1 -- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ff815cbf0 msgr2=0x7f3ff8153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.465+0000 7f3ff7e00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ff815cbf0 0x7f3ff8153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.465+0000 7f3ff7e00640 1 -- 172.21.15.39:0/1449459299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8009a10 con 0x7f3ff815db60 2024-01-24T17:45:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.465+0000 7f3ff7e00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815db60 0x7f3ff8156a00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000c480 tx=0x7f3ff000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.466+0000 7f3fef400640 1 -- 172.21.15.39:0/1449459299 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0005ce0 con 0x7f3ff815db60 2024-01-24T17:45:30.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.466+0000 7f3fef400640 1 -- 172.21.15.39:0/1449459299 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff0005e60 con 0x7f3ff815db60 2024-01-24T17:45:30.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.466+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff80c6cb0 con 0x7f3ff815db60 2024-01-24T17:45:30.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.466+0000 7f3fef400640 1 -- 172.21.15.39:0/1449459299 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff00072b0 con 0x7f3ff815db60 2024-01-24T17:45:30.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.466+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff8154900 con 0x7f3ff815db60 2024-01-24T17:45:30.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.467+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff8154af0 con 0x7f3ff815db60 2024-01-24T17:45:30.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.468+0000 7f3fef400640 1 -- 172.21.15.39:0/1449459299 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3ff0007410 con 0x7f3ff815db60 2024-01-24T17:45:30.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.469+0000 7f3fef400640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3fd006a430 0x7f3fd006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:30.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.469+0000 7f3fef400640 1 -- 172.21.15.39:0/1449459299 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ff0087e50 con 0x7f3ff815db60 2024-01-24T17:45:30.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.470+0000 7f3ff7400640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3fd006a430 0x7f3fd006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:30.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.471+0000 7f3ff7400640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3fd006a430 0x7f3fd006c8f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8036000 tx=0x7f3fe8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:30.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.474+0000 7f3fef400640 1 -- 172.21.15.39:0/1449459299 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3ff00534d0 con 0x7f3ff815db60 2024-01-24T17:45:30.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.711+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3ff80c6ea0 con 0x7f3ff815db60 2024-01-24T17:45:30.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.712+0000 7f3fef400640 1 -- 172.21.15.39:0/1449459299 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3ff0017070 con 0x7f3ff815db60 2024-01-24T17:45:30.713 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:45:30.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.714+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3fd006a430 msgr2=0x7f3fd006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.714+0000 7f3ffca00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3fd006a430 0x7f3fd006c8f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8036000 tx=0x7f3fe8030000 comp rx=0 tx=0).stop 2024-01-24T17:45:30.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.714+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815db60 msgr2=0x7f3ff8156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.714+0000 7f3ffca00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815db60 0x7f3ff8156a00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000c480 tx=0x7f3ff000c940 comp rx=0 tx=0).stop 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.715+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 shutdown_connections 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.715+0000 7f3ffca00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3fd006a430 0x7f3fd006c8f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.715+0000 7f3ffca00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ff815db60 0x7f3ff8156a00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.715+0000 7f3ffca00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ff815cbf0 0x7f3ff8153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.715+0000 7f3ffca00640 1 --2- 172.21.15.39:0/1449459299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3ff80d0da0 0x7f3ff80c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.715+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 >> 172.21.15.39:0/1449459299 conn(0x7f3ff809bb90 msgr2=0x7f3ff80d3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.715+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 shutdown_connections 2024-01-24T17:45:30.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:30.715+0000 7f3ffca00640 1 -- 172.21.15.39:0/1449459299 wait complete. 2024-01-24T17:45:30.737 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:45:30.737 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:45:30.737 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:45:30.738 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:45:30.738 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:45:30.738 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:45:30.738 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:45:30.738 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:45:30.738 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:45:30.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:30.738 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:45:30.795 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:45:30.795 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:45:30.795 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:45:30.795 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:30.795 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:45:30.795 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:30.795 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:45:30.795 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42401sec preferred_lft 42401sec 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:30.796 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:45:30.797 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:45:30.798 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:45:30.798 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:30.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:30.798 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:45:30.894 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:45:30.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:30.898 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:45:30.951 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:45:30.951 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:45:30.952 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:45:30.952 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:45:30.952 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:45:30.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:30.952 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:45:30.952 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:45:30.952 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:45:30.952 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:45:30.952 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:45:30.953 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:45:31.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:31.141 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:45:31.191 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:45:31.191 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:45:31.191 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:45:31.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:31.192 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:45:31.288 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:45:31.289 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:45:31.289 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:45:31.350 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:45:31.351 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:45:31.351 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:45:31.406 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:45:31.505 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:45:31.505 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:45:31.602 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:45:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:45:31.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:31.607 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:45:31.664 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:45:31.665 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:45:31.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:45:31.756 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:45:31.756 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:45:31.831 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:45:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:45:31.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:31.835 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:45:31.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:45:31.861+0000 7fcb9d449480 -1 init, newargv = 0x55557a3230c0 newargc=15 2024-01-24T17:45:31.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[59441]: starting ceph client 2024-01-24T17:45:31.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[59441]: starting fuse 2024-01-24T17:45:31.880 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:45:31.880 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:45:31.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:45:31.880 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:45:31.880 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:45:31.880 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:45:31.881 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:45:31.882 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:45:31.882 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:45:31.882 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:45:31.882 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:45:31.882 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:45:31.882 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:45:31.882 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:45:31.882 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:45:32.083 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.59441.asok 2024-01-24T17:45:32.264 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59441.asok 2024-01-24T17:45:32.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:32.264 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59441.asok status 2024-01-24T17:45:32.513 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "59441", 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:45:32.514 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 6444, 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 6444 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2677342968 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:45:32.515 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2677342968 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.6444 192.168.0.1:0/2677342968", 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/2677342968", 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 34, 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 63, 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:45:32.516 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:45:32.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:32.530 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:45:32.530 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:45:32.585 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:45:32.585 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:45:32.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:32.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:45:32.682 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:45:32.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.907+0000 7f444b200640 1 Processor -- start 2024-01-24T17:45:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.908+0000 7f444b200640 1 -- start start 2024-01-24T17:45:32.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.909+0000 7f444b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:32.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.909+0000 7f444b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444c15dfe0 con 0x7f444c15db60 2024-01-24T17:45:32.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.909+0000 7f444b200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444c0d0a90 con 0x7f444c0d0da0 2024-01-24T17:45:32.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.909+0000 7f444b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c0cd760 con 0x7f444c15cbf0 2024-01-24T17:45:32.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.910+0000 7f4449400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f444c15db60 0x7f444c0ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:36808/0 (socket says 172.21.15.39:36808) 2024-01-24T17:45:32.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.910+0000 7f4449400640 1 -- 172.21.15.39:0/3479688721 learned_addr learned my addr 172.21.15.39:0/3479688721 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:32.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.910+0000 7f4449e00640 1 --2- 172.21.15.39:0/3479688721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.910+0000 7f4448a00640 1 -- 172.21.15.39:0/3479688721 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801100970 0 0) 0x7f444c15dfe0 con 0x7f444c15db60 2024-01-24T17:45:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.910+0000 7f4448a00640 1 -- 172.21.15.39:0/3479688721 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f442c003520 con 0x7f444c15db60 2024-01-24T17:45:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.911+0000 7f4449e00640 1 -- 172.21.15.39:0/3479688721 >> v1:172.21.15.100:6789/0 conn(0x7f444c0d0da0 legacy=0x7f444c15e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.911+0000 7f4449e00640 1 -- 172.21.15.39:0/3479688721 >> v1:172.21.15.39:6789/0 conn(0x7f444c15db60 legacy=0x7f444c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.911+0000 7f4448a00640 1 -- 172.21.15.39:0/3479688721 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2407659802 0 0) 0x7f442c003520 con 0x7f444c15db60 2024-01-24T17:45:32.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.911+0000 7f4449e00640 1 -- 172.21.15.39:0/3479688721 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c0cda40 con 0x7f444c15cbf0 2024-01-24T17:45:32.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.912+0000 7f4449e00640 1 --2- 172.21.15.39:0/3479688721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c0d03b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f443c0097c0 tx=0x7f443c02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86219d56f93d5547 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.913+0000 7f4448a00640 1 -- 172.21.15.39:0/3479688721 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340019f0 con 0x7f444c15cbf0 2024-01-24T17:45:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.913+0000 7f4448a00640 1 -- 172.21.15.39:0/3479688721 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c0314d0 con 0x7f444c15cbf0 2024-01-24T17:45:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.913+0000 7f4448a00640 1 -- 172.21.15.39:0/3479688721 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c031840 con 0x7f444c15cbf0 2024-01-24T17:45:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.913+0000 7f444b200640 1 -- 172.21.15.39:0/3479688721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 msgr2=0x7f444c0d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.913+0000 7f444b200640 1 --2- 172.21.15.39:0/3479688721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c0d03b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f443c0097c0 tx=0x7f443c02b7f0 comp rx=0 tx=0).stop 2024-01-24T17:45:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.913+0000 7f444b200640 1 -- 172.21.15.39:0/3479688721 shutdown_connections 2024-01-24T17:45:32.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.913+0000 7f444b200640 1 --2- 172.21.15.39:0/3479688721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c0d03b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.914+0000 7f444b200640 1 -- 172.21.15.39:0/3479688721 >> 172.21.15.39:0/3479688721 conn(0x7f444c09bb90 msgr2=0x7f444c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.914+0000 7f444b200640 1 -- 172.21.15.39:0/3479688721 shutdown_connections 2024-01-24T17:45:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.914+0000 7f444b200640 1 -- 172.21.15.39:0/3479688721 wait complete. 2024-01-24T17:45:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.915+0000 7f444b200640 1 Processor -- start 2024-01-24T17:45:32.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.915+0000 7f444b200640 1 -- start start 2024-01-24T17:45:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.916+0000 7f444b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f444c0d0da0 0x7f444c0b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.916+0000 7f444b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c1f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.916+0000 7f444b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f444c15db60 0x7f444c1f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.916+0000 7f444b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c0c9f70 con 0x7f444c0d0da0 2024-01-24T17:45:32.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.916+0000 7f444b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c0b7fe0 con 0x7f444c15db60 2024-01-24T17:45:32.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.916+0000 7f444b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c0b82c0 con 0x7f444c15cbf0 2024-01-24T17:45:32.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f444c0d0da0 0x7f444c0b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:32.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f444c0d0da0 0x7f444c0b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48066/0 (socket says 172.21.15.39:48066) 2024-01-24T17:45:32.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449e00640 1 -- 172.21.15.39:0/1848523218 learned_addr learned my addr 172.21.15.39:0/1848523218 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:32.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449400640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c1f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449e00640 1 -- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 msgr2=0x7f444c1f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f444a800640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f444c15db60 0x7f444c1f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449e00640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c1f2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449e00640 1 -- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f444c15db60 msgr2=0x7f444c1f2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449e00640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f444c15db60 0x7f444c1f2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449e00640 1 -- 172.21.15.39:0/1848523218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0093a0 con 0x7f444c0d0da0 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.917+0000 7f4449400640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c1f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.918+0000 7f4449e00640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f444c0d0da0 0x7f444c0b77b0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f443c000c00 tx=0x7f443c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.918+0000 7f4442000640 1 -- 172.21.15.39:0/1848523218 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c03f890 con 0x7f444c0d0da0 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.918+0000 7f4442000640 1 -- 172.21.15.39:0/1848523218 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c03f9f0 con 0x7f444c0d0da0 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.918+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444c0b85a0 con 0x7f444c0d0da0 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.918+0000 7f4442000640 1 -- 172.21.15.39:0/1848523218 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c03fd60 con 0x7f444c0d0da0 2024-01-24T17:45:32.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.918+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444c0b7cf0 con 0x7f444c0d0da0 2024-01-24T17:45:32.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.919+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444c0ca400 con 0x7f444c0d0da0 2024-01-24T17:45:32.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.920+0000 7f4442000640 1 -- 172.21.15.39:0/1848523218 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f443c03b040 con 0x7f444c0d0da0 2024-01-24T17:45:32.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.921+0000 7f4442000640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f442406a430 0x7f442406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:32.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.921+0000 7f4442000640 1 -- 172.21.15.39:0/1848523218 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f443c0aad10 con 0x7f444c0d0da0 2024-01-24T17:45:32.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.922+0000 7f4449400640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f442406a430 0x7f442406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:32.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.923+0000 7f4449400640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f442406a430 0x7f442406c8f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4434001560 tx=0x7f4434009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:32.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:32.925+0000 7f4442000640 1 -- 172.21.15.39:0/1848523218 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f444c0ca400 con 0x7f444c0d0da0 2024-01-24T17:45:33.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.164+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f444c04dff0 con 0x7f444c0d0da0 2024-01-24T17:45:33.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.165+0000 7f4442000640 1 -- 172.21.15.39:0/1848523218 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f443c04f060 con 0x7f444c0d0da0 2024-01-24T17:45:33.167 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:45:33.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.167+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f442406a430 msgr2=0x7f442406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.167+0000 7f444b200640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f442406a430 0x7f442406c8f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4434001560 tx=0x7f4434009040 comp rx=0 tx=0).stop 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f444c0d0da0 msgr2=0x7f444c0b77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f444c0d0da0 0x7f444c0b77b0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f443c000c00 tx=0x7f443c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 shutdown_connections 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f444c15db60 0x7f444c1f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f444c15cbf0 0x7f444c1f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f442406a430 0x7f442406c8f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 --2- 172.21.15.39:0/1848523218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f444c0d0da0 0x7f444c0b77b0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 >> 172.21.15.39:0/1848523218 conn(0x7f444c09bb90 msgr2=0x7f444c0d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 shutdown_connections 2024-01-24T17:45:33.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:33.168+0000 7f444b200640 1 -- 172.21.15.39:0/1848523218 wait complete. 2024-01-24T17:45:33.190 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:45:33.190 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:45:33.190 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:45:33.191 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-24T17:45:33.191 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:45:33.191 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:45:33.191 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:45:33.191 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:45:33.191 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:45:33.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:33.191 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:45:33.250 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:45:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:45:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:45:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:45:33.250 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42396sec preferred_lft 42396sec 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:45:33.251 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:45:33.252 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:45:33.253 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:45:33.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:33.254 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:45:33.344 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:45:33.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:33.349 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:45:33.403 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:45:33.403 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:45:33.403 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:45:33.403 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:45:33.403 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:45:33.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:33.404 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:45:33.404 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:45:33.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:45:33.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:45:33.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:45:33.404 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:45:33.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:33.582 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:45:33.633 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:45:33.633 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:45:33.634 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:45:33.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:33.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:45:33.723 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-24T17:45:33.723 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:45:33.723 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:45:33.777 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:45:33.777 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:45:33.777 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:45:33.824 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:45:33.965 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:45:33.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:45:34.051 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:45:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:45:34.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:34.056 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:45:34.109 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:45:34.109 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-24T17:45:34.114 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:45:34.188 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:45:34.188 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:45:34.267 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:45:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:45:34.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:34.271 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:45:34.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:45:34.301+0000 7fc362c41480 -1 init, newargv = 0x55b74e093e90 newargc=15 2024-01-24T17:45:34.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[45373]: starting ceph client 2024-01-24T17:45:34.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[45373]: starting fuse 2024-01-24T17:45:35.414 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:45:35.516 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:45:35.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:45:35.610 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:45:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:45:35.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:35.614 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:45:35.667 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:45:35.668 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:45:35.668 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:45:35.668 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:45:35.668 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:45:35.668 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:45:35.669 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:45:35.670 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:45:35.670 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:45:35.670 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:45:35.670 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:45:35.670 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:45:35.670 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2024-01-24T17:45:35.670 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:45:35.836 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.45373.asok 2024-01-24T17:45:36.016 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45373.asok 2024-01-24T17:45:36.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:36.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45373.asok status 2024-01-24T17:45:36.249 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "45373", 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:45:36.250 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 5593, 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5593 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 999213264 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:45:36.251 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 999213264 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.5593 192.168.0.1:0/999213264", 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/999213264", 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 34, 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 63, 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:45:36.252 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:45:36.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:36.264 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:45:36.264 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:45:36.318 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:45:36.318 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:45:36.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:36.318 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:45:36.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:36.409 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir foo) 2024-01-24T17:45:36.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:36.483 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir foo/bar) 2024-01-24T17:45:36.549 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:45:36.549 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:45:36.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.822+0000 7f11c3a00640 1 Processor -- start 2024-01-24T17:45:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.823+0000 7f11c3a00640 1 -- start start 2024-01-24T17:45:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.824+0000 7f11c3a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f11bc15ca40 0x7f11bc0b8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.824+0000 7f11c3a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc0b7ce0 con 0x7f11bc0b4480 2024-01-24T17:45:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.824+0000 7f11c3a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc0b6e40 con 0x7f11bc0b69f0 2024-01-24T17:45:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.824+0000 7f11c3a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc0b70d0 con 0x7f11bc15ca40 2024-01-24T17:45:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.825+0000 7f11bbe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f11bc0b4480 0x7f11bc0d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59890/0 (socket says 172.21.15.39:59890) 2024-01-24T17:45:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.825+0000 7f11bbe00640 1 -- 172.21.15.39:0/2788095424 learned_addr learned my addr 172.21.15.39:0/2788095424 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.825+0000 7f11c1000640 1 --2- 172.21.15.39:0/2788095424 >> v2:172.21.15.100:3300/0 conn(0x7f11bc15ca40 0x7f11bc0b8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.825+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066846823 0 0) 0x7f11bc0b7ce0 con 0x7f11bc0b4480 2024-01-24T17:45:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.825+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f119c003520 con 0x7f11bc0b4480 2024-01-24T17:45:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2164070947 0 0) 0x7f119c003520 con 0x7f11bc0b4480 2024-01-24T17:45:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 >> v2:172.21.15.100:3300/0 conn(0x7f11bc15ca40 msgr2=0x7f11bc0b8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 --2- 172.21.15.39:0/2788095424 >> v2:172.21.15.100:3300/0 conn(0x7f11bc15ca40 0x7f11bc0b8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 >> v1:172.21.15.100:6789/0 conn(0x7f11bc0b69f0 legacy=0x7f11bc0d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0b73b0 con 0x7f11bc0b4480 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f11a4001670 con 0x7f11bc0b4480 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 >> v1:172.21.15.39:6789/0 conn(0x7f11bc0b4480 legacy=0x7f11bc0d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f119c0039c0 0x7f119c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f119c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f119c005180 0x7f119c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0b70d0 con 0x7f119c0047c0 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c00c080 con 0x7f119c005180 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119c00c240 con 0x7f119c0039c0 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bbe00640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f119c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bbe00640 1 -- 172.21.15.39:0/2788095424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f119c0039c0 msgr2=0x7f119c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bbe00640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f119c0039c0 0x7f119c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bbe00640 1 -- 172.21.15.39:0/2788095424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f119c005180 msgr2=0x7f119c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:36.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bbe00640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f119c005180 0x7f119c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:36.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11bbe00640 1 -- 172.21.15.39:0/2788095424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c003520 con 0x7f119c0047c0 2024-01-24T17:45:36.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.826+0000 7f11c1a00640 1 -- 172.21.15.39:0/2788095424 reap_dead start 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.828+0000 7f11bbe00640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f119c004c40 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f11a400c0e0 tx=0x7f119c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed5a7769e7669cf server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.828+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4005ad0 con 0x7f119c0047c0 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.828+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a4005c30 con 0x7f119c0047c0 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.828+0000 7f11bb400640 1 -- 172.21.15.39:0/2788095424 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a402ca50 con 0x7f119c0047c0 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.828+0000 7f11c3a00640 1 -- 172.21.15.39:0/2788095424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 msgr2=0x7f119c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.828+0000 7f11c3a00640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f119c004c40 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f11a400c0e0 tx=0x7f119c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.829+0000 7f11c3a00640 1 -- 172.21.15.39:0/2788095424 shutdown_connections 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.829+0000 7f11c3a00640 1 --2- 172.21.15.39:0/2788095424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f119c004c40 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.829+0000 7f11c3a00640 1 -- 172.21.15.39:0/2788095424 >> 172.21.15.39:0/2788095424 conn(0x7f11bc071790 msgr2=0x7f11bc071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.829+0000 7f11c3a00640 1 -- 172.21.15.39:0/2788095424 shutdown_connections 2024-01-24T17:45:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.829+0000 7f11c3a00640 1 -- 172.21.15.39:0/2788095424 wait complete. 2024-01-24T17:45:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.830+0000 7f11c3a00640 1 Processor -- start 2024-01-24T17:45:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.831+0000 7f11c3a00640 1 -- start start 2024-01-24T17:45:36.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.831+0000 7f11c3a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f11bc0d2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:36.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.831+0000 7f11c3a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11bc0d2a60 0x7f11bc0c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:36.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c3a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11bc0c8be0 0x7f11bc0c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c3a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc15eec0 con 0x7f119c0047c0 2024-01-24T17:45:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c3a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0d2ee0 con 0x7f11bc0c8be0 2024-01-24T17:45:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c3a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0c8040 con 0x7f11bc0d2a60 2024-01-24T17:45:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f11bc0d2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f11bc0d2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47368/0 (socket says 172.21.15.39:47368) 2024-01-24T17:45:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1000640 1 -- 172.21.15.39:0/1253149627 learned_addr learned my addr 172.21.15.39:0/1253149627 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:36.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11bbe00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11bc0d2a60 0x7f11bc0c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1000640 1 -- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11bc0d2a60 msgr2=0x7f11bc0c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1000640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11bc0d2a60 0x7f11bc0c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1a00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11bc0c8be0 0x7f11bc0c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1000640 1 -- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11bc0c8be0 msgr2=0x7f11bc0c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1000640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11bc0c8be0 0x7f11bc0c6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1000640 1 -- 172.21.15.39:0/1253149627 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0c9060 con 0x7f119c0047c0 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.832+0000 7f11c1a00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11bc0c8be0 0x7f11bc0c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.833+0000 7f11c1000640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f11bc0d2520 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00e7a0 tx=0x7f11ac00eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.833+0000 7f11b8c00640 1 -- 172.21.15.39:0/1253149627 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac00c730 con 0x7f119c0047c0 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.833+0000 7f11b8c00640 1 -- 172.21.15.39:0/1253149627 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac00c8b0 con 0x7f119c0047c0 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.833+0000 7f11b8c00640 1 -- 172.21.15.39:0/1253149627 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac00cc60 con 0x7f119c0047c0 2024-01-24T17:45:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.833+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc0c92f0 con 0x7f119c0047c0 2024-01-24T17:45:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.833+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc1f28c0 con 0x7f119c0047c0 2024-01-24T17:45:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.835+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc073ef0 con 0x7f119c0047c0 2024-01-24T17:45:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.835+0000 7f11b8c00640 1 -- 172.21.15.39:0/1253149627 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f11ac013470 con 0x7f119c0047c0 2024-01-24T17:45:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.836+0000 7f11b8c00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f119406a430 0x7f119406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.836+0000 7f11b8c00640 1 -- 172.21.15.39:0/1253149627 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11ac088aa0 con 0x7f119c0047c0 2024-01-24T17:45:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.837+0000 7f11bbe00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f119406a430 0x7f119406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:36.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.838+0000 7f11bbe00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f119406a430 0x7f119406c8f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f11a4034000 tx=0x7f11a4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:36.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:36.841+0000 7f11b8c00640 1 -- 172.21.15.39:0/1253149627 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f11ac053f60 con 0x7f119c0047c0 2024-01-24T17:45:37.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.040+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11bc06b5e0 con 0x7f119c0047c0 2024-01-24T17:45:37.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.042+0000 7f11b8c00640 1 -- 172.21.15.39:0/1253149627 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f11ac021090 con 0x7f119c0047c0 2024-01-24T17:45:37.044 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:37.044 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:37.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f119406a430 msgr2=0x7f119406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f119406a430 0x7f119406c8f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f11a4034000 tx=0x7f11a4009030 comp rx=0 tx=0).stop 2024-01-24T17:45:37.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 msgr2=0x7f11bc0d2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f11bc0d2520 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00e7a0 tx=0x7f11ac00eba0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 shutdown_connections 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f11bc0c8be0 0x7f11bc0c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f11bc0d2a60 0x7f11bc0c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f119406a430 0x7f119406c8f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 --2- 172.21.15.39:0/1253149627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f119c0047c0 0x7f11bc0d2520 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 >> 172.21.15.39:0/1253149627 conn(0x7f11bc071790 msgr2=0x7f11bc0b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 shutdown_connections 2024-01-24T17:45:37.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.045+0000 7f11c3a00640 1 -- 172.21.15.39:0/1253149627 wait complete. 2024-01-24T17:45:37.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:45:16.503215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:45:17.386565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:45:37.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:45:37.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.291+0000 7ff94a000640 1 Processor -- start 2024-01-24T17:45:37.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.291+0000 7ff94a000640 1 -- start start 2024-01-24T17:45:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.293+0000 7ff94a000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff9440ce820 0x7ff9440c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.293+0000 7ff94a000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94415d090 con 0x7ff94415ccb0 2024-01-24T17:45:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.293+0000 7ff94a000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9440ce1a0 con 0x7ff9440ce390 2024-01-24T17:45:37.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.293+0000 7ff94a000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9440cac20 con 0x7ff9440ce820 2024-01-24T17:45:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.293+0000 7ff948c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff94415ccb0 0x7ff9440cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59892/0 (socket says 172.21.15.39:59892) 2024-01-24T17:45:37.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.293+0000 7ff948c00640 1 -- 172.21.15.39:0/1025169240 learned_addr learned my addr 172.21.15.39:0/1025169240 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.294+0000 7ff949600640 1 --2- 172.21.15.39:0/1025169240 >> v2:172.21.15.100:3300/0 conn(0x7ff9440ce820 0x7ff9440c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.294+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971059668 0 0) 0x7ff94415d090 con 0x7ff94415ccb0 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.294+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff924003520 con 0x7ff94415ccb0 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3005328609 0 0) 0x7ff924003520 con 0x7ff94415ccb0 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 >> v2:172.21.15.100:3300/0 conn(0x7ff9440ce820 msgr2=0x7ff9440c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 --2- 172.21.15.39:0/1025169240 >> v2:172.21.15.100:3300/0 conn(0x7ff9440ce820 0x7ff9440c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 >> v1:172.21.15.100:6789/0 conn(0x7ff9440ce390 legacy=0x7ff9440c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94406c540 con 0x7ff94415ccb0 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff934005090 con 0x7ff94415ccb0 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 >> v1:172.21.15.39:6789/0 conn(0x7ff94415ccb0 legacy=0x7ff9440cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff924003970 0x7ff924003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9240047c0 0x7ff924004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff924005180 0x7ff92400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440cac20 con 0x7ff924003970 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92400e230 con 0x7ff924005180 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92400e3f0 con 0x7ff9240047c0 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff948c00640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff924003970 0x7ff924003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff949600640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff924005180 0x7ff92400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.295+0000 7ff943e00640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9240047c0 0x7ff924004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.296+0000 7ff948c00640 1 -- 172.21.15.39:0/1025169240 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9240047c0 msgr2=0x7ff924004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.296+0000 7ff948c00640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9240047c0 0x7ff924004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.296+0000 7ff948c00640 1 -- 172.21.15.39:0/1025169240 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff924005180 msgr2=0x7ff92400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.296+0000 7ff948c00640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff924005180 0x7ff92400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.296+0000 7ff948c00640 1 -- 172.21.15.39:0/1025169240 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff924003520 con 0x7ff924003970 2024-01-24T17:45:37.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.296+0000 7ff949600640 1 -- 172.21.15.39:0/1025169240 reap_dead start 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.297+0000 7ff948c00640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff924003970 0x7ff924003d90 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff93400c190 tx=0x7ff9240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b55ca1bcd6160a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.297+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff934005ad0 con 0x7ff924003970 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.297+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff934005c30 con 0x7ff924003970 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.297+0000 7ff943400640 1 -- 172.21.15.39:0/1025169240 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93402cab0 con 0x7ff924003970 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.298+0000 7ff94a000640 1 -- 172.21.15.39:0/1025169240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff924003970 msgr2=0x7ff924003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.298+0000 7ff94a000640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff924003970 0x7ff924003d90 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff93400c190 tx=0x7ff9240057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.298+0000 7ff94a000640 1 -- 172.21.15.39:0/1025169240 shutdown_connections 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.298+0000 7ff94a000640 1 --2- 172.21.15.39:0/1025169240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff924003970 0x7ff924003d90 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.298+0000 7ff94a000640 1 -- 172.21.15.39:0/1025169240 >> 172.21.15.39:0/1025169240 conn(0x7ff944069e50 msgr2=0x7ff94406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:37.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.298+0000 7ff94a000640 1 -- 172.21.15.39:0/1025169240 shutdown_connections 2024-01-24T17:45:37.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.298+0000 7ff94a000640 1 -- 172.21.15.39:0/1025169240 wait complete. 2024-01-24T17:45:37.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.300+0000 7ff94a000640 1 Processor -- start 2024-01-24T17:45:37.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.300+0000 7ff94a000640 1 -- start start 2024-01-24T17:45:37.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff94a000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff924003970 0x7ff9441ed080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff94a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9440c54f0 0x7ff9441ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff94a000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9440ac650 0x7ff9441f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff94a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9441edb00 con 0x7ff9440c54f0 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff94a000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440c5c90 con 0x7ff9440ac650 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff94a000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9440ac1c0 con 0x7ff924003970 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff943e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9440c54f0 0x7ff9441ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff943e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9440c54f0 0x7ff9441ed5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47388/0 (socket says 172.21.15.39:47388) 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.301+0000 7ff943e00640 1 -- 172.21.15.39:0/1896710227 learned_addr learned my addr 172.21.15.39:0/1896710227 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff943e00640 1 -- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff924003970 msgr2=0x7ff9441ed080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff943e00640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff924003970 0x7ff9441ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff943e00640 1 -- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9440ac650 msgr2=0x7ff9441f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff943e00640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9440ac650 0x7ff9441f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff943e00640 1 -- 172.21.15.39:0/1896710227 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440c59b0 con 0x7ff9440c54f0 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff943e00640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9440c54f0 0x7ff9441ed5c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00e470 tx=0x7ff92c00e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff940c00640 1 -- 172.21.15.39:0/1896710227 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c0052e0 con 0x7ff9440c54f0 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff940c00640 1 -- 172.21.15.39:0/1896710227 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c005440 con 0x7ff9440c54f0 2024-01-24T17:45:37.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff940c00640 1 -- 172.21.15.39:0/1896710227 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c003830 con 0x7ff9440c54f0 2024-01-24T17:45:37.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.302+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9440abb70 con 0x7ff9440c54f0 2024-01-24T17:45:37.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.303+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94414f9d0 con 0x7ff9440c54f0 2024-01-24T17:45:37.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.304+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94404dff0 con 0x7ff9440c54f0 2024-01-24T17:45:37.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.305+0000 7ff940c00640 1 -- 172.21.15.39:0/1896710227 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff92c003990 con 0x7ff9440c54f0 2024-01-24T17:45:37.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.305+0000 7ff940c00640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff91c06a460 0x7ff91c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.306+0000 7ff940c00640 1 -- 172.21.15.39:0/1896710227 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff92c087f20 con 0x7ff9440c54f0 2024-01-24T17:45:37.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.306+0000 7ff948c00640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff91c06a460 0x7ff91c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.307+0000 7ff948c00640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff91c06a460 0x7ff91c06c920 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff934034000 tx=0x7ff934039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:37.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.310+0000 7ff940c00640 1 -- 172.21.15.39:0/1896710227 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff92c0535b0 con 0x7ff9440c54f0 2024-01-24T17:45:37.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.504+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9440ac350 con 0x7ff9440c54f0 2024-01-24T17:45:37.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.505+0000 7ff940c00640 1 -- 172.21.15.39:0/1896710227 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff92c0190a0 con 0x7ff9440c54f0 2024-01-24T17:45:37.507 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff91c06a460 msgr2=0x7ff91c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff91c06a460 0x7ff91c06c920 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff934034000 tx=0x7ff934039040 comp rx=0 tx=0).stop 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9440c54f0 msgr2=0x7ff9441ed5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9440c54f0 0x7ff9441ed5c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00e470 tx=0x7ff92c00e870 comp rx=0 tx=0).stop 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 shutdown_connections 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9440ac650 0x7ff9441f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff91c06a460 0x7ff91c06c920 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9440c54f0 0x7ff9441ed5c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 --2- 172.21.15.39:0/1896710227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff924003970 0x7ff9441ed080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 >> 172.21.15.39:0/1896710227 conn(0x7ff944069e50 msgr2=0x7ff9440cc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:37.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 shutdown_connections 2024-01-24T17:45:37.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.508+0000 7ff94a000640 1 -- 172.21.15.39:0/1896710227 wait complete. 2024-01-24T17:45:37.529 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:45:37.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.754+0000 7f25d1000640 1 Processor -- start 2024-01-24T17:45:37.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.754+0000 7f25d1000640 1 -- start start 2024-01-24T17:45:37.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25d1000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc15c8d0 0x7f25cc0d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25d1000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f25cc0b2710 0x7f25cc0d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25d1000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f25cc0d2cb0 0x7f25cc0cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25d1000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25cc0d3140 con 0x7f25cc0b2710 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25d1000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25cc0b38a0 con 0x7f25cc0d2cb0 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25d1000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc0d13e0 con 0x7f25cc15c8d0 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25caa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f25cc0b2710 0x7f25cc0d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25caa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f25cc0b2710 0x7f25cc0d2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47390/0 (socket says 172.21.15.39:47390) 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.755+0000 7f25caa00640 1 -- 172.21.15.39:0/522499534 learned_addr learned my addr 172.21.15.39:0/522499534 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.756+0000 7f25cb400640 1 --2- 172.21.15.39:0/522499534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc15c8d0 0x7f25cc0d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.756+0000 7f25cbe00640 1 --2- 172.21.15.39:0/522499534 >> v2:172.21.15.100:3300/0 conn(0x7f25cc0d2cb0 0x7f25cc0cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.756+0000 7f25caa00640 1 -- 172.21.15.39:0/522499534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc15c8d0 msgr2=0x7f25cc0d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.756+0000 7f25caa00640 1 --2- 172.21.15.39:0/522499534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc15c8d0 0x7f25cc0d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.756+0000 7f25caa00640 1 -- 172.21.15.39:0/522499534 >> v2:172.21.15.100:3300/0 conn(0x7f25cc0d2cb0 msgr2=0x7f25cc0cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:37.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.756+0000 7f25caa00640 1 --2- 172.21.15.39:0/522499534 >> v2:172.21.15.100:3300/0 conn(0x7f25cc0d2cb0 0x7f25cc0cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.756+0000 7f25caa00640 1 -- 172.21.15.39:0/522499534 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc0d0ab0 con 0x7f25cc0b2710 2024-01-24T17:45:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.757+0000 7f25caa00640 1 --2- 172.21.15.39:0/522499534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0b2710 0x7f25cc0d2770 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009e80 tx=0x7f25b402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a52d663ffa88827 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.757+0000 7f25ca000640 1 -- 172.21.15.39:0/522499534 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b402ca50 con 0x7f25cc0b2710 2024-01-24T17:45:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25ca000640 1 -- 172.21.15.39:0/522499534 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b402cbb0 con 0x7f25cc0b2710 2024-01-24T17:45:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25ca000640 1 -- 172.21.15.39:0/522499534 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b40343f0 con 0x7f25cc0b2710 2024-01-24T17:45:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 -- 172.21.15.39:0/522499534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0b2710 msgr2=0x7f25cc0d2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 --2- 172.21.15.39:0/522499534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0b2710 0x7f25cc0d2770 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009e80 tx=0x7f25b402beb0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 -- 172.21.15.39:0/522499534 shutdown_connections 2024-01-24T17:45:37.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 --2- 172.21.15.39:0/522499534 >> v2:172.21.15.100:3300/0 conn(0x7f25cc0d2cb0 0x7f25cc0cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 --2- 172.21.15.39:0/522499534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc0b2710 0x7f25cc0d2770 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 --2- 172.21.15.39:0/522499534 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc15c8d0 0x7f25cc0d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 -- 172.21.15.39:0/522499534 >> 172.21.15.39:0/522499534 conn(0x7f25cc071dd0 msgr2=0x7f25cc09bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:37.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 -- 172.21.15.39:0/522499534 shutdown_connections 2024-01-24T17:45:37.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.758+0000 7f25d1000640 1 -- 172.21.15.39:0/522499534 wait complete. 2024-01-24T17:45:37.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.760+0000 7f25d1000640 1 Processor -- start 2024-01-24T17:45:37.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.760+0000 7f25d1000640 1 -- start start 2024-01-24T17:45:37.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.761+0000 7f25d1000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc0b2710 0x7f25cc15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.761+0000 7f25d1000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f25cc0d2cb0 0x7f25cc0cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.761+0000 7f25d1000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc15c8d0 0x7f25cc151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.761+0000 7f25d1000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc15fe00 con 0x7f25cc15c8d0 2024-01-24T17:45:37.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.761+0000 7f25d1000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc0cca00 con 0x7f25cc0d2cb0 2024-01-24T17:45:37.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.761+0000 7f25d1000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc0c79d0 con 0x7f25cc0b2710 2024-01-24T17:45:37.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc15c8d0 0x7f25cc151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc15c8d0 0x7f25cc151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47392/0 (socket says 172.21.15.39:47392) 2024-01-24T17:45:37.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cbe00640 1 -- 172.21.15.39:0/432058218 learned_addr learned my addr 172.21.15.39:0/432058218 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:37.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cb400640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc0b2710 0x7f25cc15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25caa00640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f25cc0d2cb0 0x7f25cc0cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cbe00640 1 -- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc0b2710 msgr2=0x7f25cc15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cbe00640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc0b2710 0x7f25cc15f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cbe00640 1 -- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f25cc0d2cb0 msgr2=0x7f25cc0cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cbe00640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f25cc0d2cb0 0x7f25cc0cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cbe00640 1 -- 172.21.15.39:0/432058218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b4009b30 con 0x7f25cc15c8d0 2024-01-24T17:45:37.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.762+0000 7f25cb400640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc0b2710 0x7f25cc15f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:45:37.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.763+0000 7f25cbe00640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc15c8d0 0x7f25cc151380 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f25c400d830 tx=0x7f25c400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:37.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.763+0000 7f25c3400640 1 -- 172.21.15.39:0/432058218 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c40096c0 con 0x7f25cc15c8d0 2024-01-24T17:45:37.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.763+0000 7f25c3400640 1 -- 172.21.15.39:0/432058218 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c4009840 con 0x7f25cc15c8d0 2024-01-24T17:45:37.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.763+0000 7f25c3400640 1 -- 172.21.15.39:0/432058218 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4009bf0 con 0x7f25cc15c8d0 2024-01-24T17:45:37.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.763+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25cc0c6dc0 con 0x7f25cc15c8d0 2024-01-24T17:45:37.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.763+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25cc0c7380 con 0x7f25cc15c8d0 2024-01-24T17:45:37.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.764+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25cc0c7b40 con 0x7f25cc15c8d0 2024-01-24T17:45:37.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.765+0000 7f25c3400640 1 -- 172.21.15.39:0/432058218 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f25c4014410 con 0x7f25cc15c8d0 2024-01-24T17:45:37.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.766+0000 7f25c3400640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f25a406a460 0x7f25a406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:37.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.766+0000 7f25c3400640 1 -- 172.21.15.39:0/432058218 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25c4088f30 con 0x7f25cc15c8d0 2024-01-24T17:45:37.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.767+0000 7f25cb400640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f25a406a460 0x7f25a406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:37.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.768+0000 7f25cb400640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f25a406a460 0x7f25a406c920 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f25bc006fd0 tx=0x7f25bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:37.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.771+0000 7f25c3400640 1 -- 172.21.15.39:0/432058218 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f25c4054480 con 0x7f25cc15c8d0 2024-01-24T17:45:37.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.964+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f25cc04dff0 con 0x7f25cc15c8d0 2024-01-24T17:45:37.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.965+0000 7f25c3400640 1 -- 172.21.15.39:0/432058218 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v63) v1 ==== 104+0+11 (secure 0 0 0) 0x7f25c4057aa0 con 0x7f25cc15c8d0 2024-01-24T17:45:37.967 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f25a406a460 msgr2=0x7f25a406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f25a406a460 0x7f25a406c920 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f25bc006fd0 tx=0x7f25bc008040 comp rx=0 tx=0).stop 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc15c8d0 msgr2=0x7f25cc151380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc15c8d0 0x7f25cc151380 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f25c400d830 tx=0x7f25c400dcf0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 shutdown_connections 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f25a406a460 0x7f25a406c920 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25cc15c8d0 0x7f25cc151380 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f25cc0d2cb0 0x7f25cc0cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 --2- 172.21.15.39:0/432058218 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f25cc0b2710 0x7f25cc15f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 >> 172.21.15.39:0/432058218 conn(0x7f25cc071dd0 msgr2=0x7f25cc0b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 shutdown_connections 2024-01-24T17:45:37.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:37.968+0000 7f25d1000640 1 -- 172.21.15.39:0/432058218 wait complete. 2024-01-24T17:45:37.989 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:45:38.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.215+0000 7f3609a00640 1 Processor -- start 2024-01-24T17:45:38.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.216+0000 7f3609a00640 1 -- start start 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.217+0000 7f3609a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f36040ad530 0x7f360406c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.217+0000 7f3609a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36040c13f0 0x7f36040c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.217+0000 7f3609a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360406c430 con 0x7f3604161040 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.217+0000 7f3609a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3604161420 con 0x7f36040c13f0 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.217+0000 7f3609a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36040c1da0 con 0x7f36040ad530 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.217+0000 7f3609000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36040c13f0 0x7f36040c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.218+0000 7f3609000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36040c13f0 0x7f36040c1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47408/0 (socket says 172.21.15.39:47408) 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.218+0000 7f3609000640 1 -- 172.21.15.39:0/611127475 learned_addr learned my addr 172.21.15.39:0/611127475 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.218+0000 7f35fbe00640 1 --2- 172.21.15.39:0/611127475 >> v2:172.21.15.100:3300/0 conn(0x7f36040ad530 0x7f360406c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.218+0000 7f3609000640 1 -- 172.21.15.39:0/611127475 >> v2:172.21.15.100:3300/0 conn(0x7f36040ad530 msgr2=0x7f360406c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.218+0000 7f3609000640 1 --2- 172.21.15.39:0/611127475 >> v2:172.21.15.100:3300/0 conn(0x7f36040ad530 0x7f360406c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.218+0000 7f3609000640 1 -- 172.21.15.39:0/611127475 >> v1:172.21.15.39:6789/0 conn(0x7f3604161040 legacy=0x7f36040b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.218+0000 7f3609000640 1 -- 172.21.15.39:0/611127475 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36040cf320 con 0x7f36040c13f0 2024-01-24T17:45:38.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.219+0000 7f3609000640 1 --2- 172.21.15.39:0/611127475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040c13f0 0x7f36040c1860 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f36041649f0 tx=0x7f35fc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd938a51ec88697 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:38.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.219+0000 7f3603400640 1 -- 172.21.15.39:0/611127475 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc02c9f0 con 0x7f36040c13f0 2024-01-24T17:45:38.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.220+0000 7f3603400640 1 -- 172.21.15.39:0/611127475 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35fc02cb50 con 0x7f36040c13f0 2024-01-24T17:45:38.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.220+0000 7f3603400640 1 -- 172.21.15.39:0/611127475 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc0343f0 con 0x7f36040c13f0 2024-01-24T17:45:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.220+0000 7f3609a00640 1 -- 172.21.15.39:0/611127475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040c13f0 msgr2=0x7f36040c1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.220+0000 7f3609a00640 1 --2- 172.21.15.39:0/611127475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040c13f0 0x7f36040c1860 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f36041649f0 tx=0x7f35fc02be50 comp rx=0 tx=0).stop 2024-01-24T17:45:38.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.220+0000 7f3609a00640 1 -- 172.21.15.39:0/611127475 shutdown_connections 2024-01-24T17:45:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.220+0000 7f3609a00640 1 --2- 172.21.15.39:0/611127475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040c13f0 0x7f36040c1860 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.220+0000 7f3609a00640 1 --2- 172.21.15.39:0/611127475 >> v2:172.21.15.100:3300/0 conn(0x7f36040ad530 0x7f360406c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.220+0000 7f3609a00640 1 -- 172.21.15.39:0/611127475 >> 172.21.15.39:0/611127475 conn(0x7f3604069e50 msgr2=0x7f360406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.221+0000 7f3609a00640 1 -- 172.21.15.39:0/611127475 shutdown_connections 2024-01-24T17:45:38.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.221+0000 7f3609a00640 1 -- 172.21.15.39:0/611127475 wait complete. 2024-01-24T17:45:38.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.222+0000 7f3609a00640 1 Processor -- start 2024-01-24T17:45:38.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.222+0000 7f3609a00640 1 -- start start 2024-01-24T17:45:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.223+0000 7f3609a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040ad530 0x7f36040c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.223+0000 7f3609a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36040c13f0 0x7f36040ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.223+0000 7f3609a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3604161040 0x7f360415d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.223+0000 7f3609a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36040c3d70 con 0x7f36040ad530 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.223+0000 7f3609a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36040c4050 con 0x7f36040c13f0 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.223+0000 7f3609a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36040c4330 con 0x7f3604161040 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040ad530 0x7f36040c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040ad530 0x7f36040c2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47424/0 (socket says 172.21.15.39:47424) 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 -- 172.21.15.39:0/1765475203 learned_addr learned my addr 172.21.15.39:0/1765475203 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f35fbe00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36040c13f0 0x7f36040ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3609000640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3604161040 0x7f360415d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 -- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3604161040 msgr2=0x7f360415d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3604161040 0x7f360415d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 -- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36040c13f0 msgr2=0x7f36040ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36040c13f0 0x7f36040ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 -- 172.21.15.39:0/1765475203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc009b30 con 0x7f36040ad530 2024-01-24T17:45:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.224+0000 7f3603e00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040ad530 0x7f36040c2b10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f35f400c4d0 tx=0x7f35f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.225+0000 7f3600c00640 1 -- 172.21.15.39:0/1765475203 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4005480 con 0x7f36040ad530 2024-01-24T17:45:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.225+0000 7f3600c00640 1 -- 172.21.15.39:0/1765475203 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f40035f0 con 0x7f36040ad530 2024-01-24T17:45:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.225+0000 7f3600c00640 1 -- 172.21.15.39:0/1765475203 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4003960 con 0x7f36040ad530 2024-01-24T17:45:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.225+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3604162750 con 0x7f36040ad530 2024-01-24T17:45:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.225+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36040c3050 con 0x7f36040ad530 2024-01-24T17:45:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.226+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36041625b0 con 0x7f36040ad530 2024-01-24T17:45:38.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.227+0000 7f3600c00640 1 -- 172.21.15.39:0/1765475203 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f35f4003ac0 con 0x7f36040ad530 2024-01-24T17:45:38.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.228+0000 7f3600c00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f35dc06a430 0x7f35dc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.228+0000 7f3600c00640 1 -- 172.21.15.39:0/1765475203 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35f4087ce0 con 0x7f36040ad530 2024-01-24T17:45:38.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.229+0000 7f35fbe00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f35dc06a430 0x7f35dc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.230+0000 7f35fbe00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f35dc06a430 0x7f35dc06c8f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f35ec006fd0 tx=0x7f35ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.232+0000 7f3600c00640 1 -- 172.21.15.39:0/1765475203 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f36041625b0 con 0x7f36040ad530 2024-01-24T17:45:38.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.428+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3604162c70 con 0x7f36040ad530 2024-01-24T17:45:38.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.429+0000 7f3600c00640 1 -- 172.21.15.39:0/1765475203 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v63) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3604162c70 con 0x7f36040ad530 2024-01-24T17:45:38.431 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f35dc06a430 msgr2=0x7f35dc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f35dc06a430 0x7f35dc06c8f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f35ec006fd0 tx=0x7f35ec008040 comp rx=0 tx=0).stop 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040ad530 msgr2=0x7f36040c2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040ad530 0x7f36040c2b10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f35f400c4d0 tx=0x7f35f400c990 comp rx=0 tx=0).stop 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 shutdown_connections 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3604161040 0x7f360415d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36040c13f0 0x7f36040ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f35dc06a430 0x7f35dc06c8f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 --2- 172.21.15.39:0/1765475203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36040ad530 0x7f36040c2b10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 >> 172.21.15.39:0/1765475203 conn(0x7f3604069e50 msgr2=0x7f36040ab030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 shutdown_connections 2024-01-24T17:45:38.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.432+0000 7f3609a00640 1 -- 172.21.15.39:0/1765475203 wait complete. 2024-01-24T17:45:38.454 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:45:38.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.675+0000 7f20aec00640 1 Processor -- start 2024-01-24T17:45:38.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.675+0000 7f20aec00640 1 -- start start 2024-01-24T17:45:38.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.676+0000 7f20aec00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f20a80d2d50 0x7f20a80d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.676+0000 7f20aec00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a80d03e0 con 0x7f20a806c2e0 2024-01-24T17:45:38.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.676+0000 7f20aec00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a80d0630 con 0x7f20a815c900 2024-01-24T17:45:38.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.676+0000 7f20aec00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a80d0910 con 0x7f20a80d2d50 2024-01-24T17:45:38.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.676+0000 7f20ae200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f20a80d2d50 0x7f20a80d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.676+0000 7f20ace00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f20a806c2e0 0x7f20a80d2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59918/0 (socket says 172.21.15.39:59918) 2024-01-24T17:45:38.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.676+0000 7f20ace00640 1 -- 172.21.15.39:0/781831810 learned_addr learned my addr 172.21.15.39:0/781831810 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.677+0000 7f20ae200640 1 -- 172.21.15.39:0/781831810 >> v1:172.21.15.100:6789/0 conn(0x7f20a815c900 legacy=0x7f20a80cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.677+0000 7f20ae200640 1 -- 172.21.15.39:0/781831810 >> v1:172.21.15.39:6789/0 conn(0x7f20a806c2e0 legacy=0x7f20a80d2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.677+0000 7f20ae200640 1 -- 172.21.15.39:0/781831810 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a80d0bf0 con 0x7f20a80d2d50 2024-01-24T17:45:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.677+0000 7f20a7e00640 1 -- 172.21.15.39:0/781831810 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531483143 0 0) 0x7f20a80d03e0 con 0x7f20a806c2e0 2024-01-24T17:45:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.678+0000 7f20ae200640 1 --2- 172.21.15.39:0/781831810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a80d13f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009d90 tx=0x7f20a002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ac56b176313bfc server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.678+0000 7f20a7e00640 1 -- 172.21.15.39:0/781831810 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a002cd00 con 0x7f20a80d2d50 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.678+0000 7f20a7e00640 1 -- 172.21.15.39:0/781831810 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a002ce60 con 0x7f20a80d2d50 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.678+0000 7f20a7e00640 1 -- 172.21.15.39:0/781831810 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0034660 con 0x7f20a80d2d50 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.679+0000 7f20aec00640 1 -- 172.21.15.39:0/781831810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 msgr2=0x7f20a80d13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.679+0000 7f20aec00640 1 --2- 172.21.15.39:0/781831810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a80d13f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009d90 tx=0x7f20a002bf50 comp rx=0 tx=0).stop 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.679+0000 7f20aec00640 1 -- 172.21.15.39:0/781831810 shutdown_connections 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.679+0000 7f20aec00640 1 --2- 172.21.15.39:0/781831810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a80d13f0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.679+0000 7f20aec00640 1 -- 172.21.15.39:0/781831810 >> 172.21.15.39:0/781831810 conn(0x7f20a809bb30 msgr2=0x7f20a8089fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.679+0000 7f20aec00640 1 -- 172.21.15.39:0/781831810 shutdown_connections 2024-01-24T17:45:38.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.679+0000 7f20aec00640 1 -- 172.21.15.39:0/781831810 wait complete. 2024-01-24T17:45:38.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.681+0000 7f20aec00640 1 Processor -- start 2024-01-24T17:45:38.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.681+0000 7f20aec00640 1 -- start start 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20aec00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a806c2e0 0x7f20a81f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20aec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a8157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20aec00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20a815c900 0x7f20a8145920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20aec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a8157c70 con 0x7f20a80d2d50 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20aec00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80cbb00 con 0x7f20a806c2e0 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20aec00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80caef0 con 0x7f20a815c900 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20ace00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a8157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20ace00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a8157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47434/0 (socket says 172.21.15.39:47434) 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.682+0000 7f20ace00640 1 -- 172.21.15.39:0/4025472737 learned_addr learned my addr 172.21.15.39:0/4025472737 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20ad800640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a806c2e0 0x7f20a81f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20ace00640 1 -- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20a815c900 msgr2=0x7f20a8145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20ace00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20a815c900 0x7f20a8145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20ace00640 1 -- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a806c2e0 msgr2=0x7f20a81f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20ace00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a806c2e0 0x7f20a81f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20ace00640 1 -- 172.21.15.39:0/4025472737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a0009a70 con 0x7f20a80d2d50 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20ad800640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a806c2e0 0x7f20a81f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20ace00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a8157730 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f209000c4a0 tx=0x7f209000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20a5600640 1 -- 172.21.15.39:0/4025472737 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900074e0 con 0x7f20a80d2d50 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20a5600640 1 -- 172.21.15.39:0/4025472737 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090007640 con 0x7f20a80d2d50 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20a5600640 1 -- 172.21.15.39:0/4025472737 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090004a00 con 0x7f20a80d2d50 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a80cb1d0 con 0x7f20a80d2d50 2024-01-24T17:45:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.683+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a80cb790 con 0x7f20a80d2d50 2024-01-24T17:45:38.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.685+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a804dff0 con 0x7f20a80d2d50 2024-01-24T17:45:38.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.686+0000 7f20a5600640 1 -- 172.21.15.39:0/4025472737 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2090004b60 con 0x7f20a80d2d50 2024-01-24T17:45:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.686+0000 7f20a5600640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f208006a430 0x7f208006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.687+0000 7f20a5600640 1 -- 172.21.15.39:0/4025472737 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2090087ab0 con 0x7f20a80d2d50 2024-01-24T17:45:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.687+0000 7f20ad800640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f208006a430 0x7f208006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:38.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.688+0000 7f20ad800640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f208006a430 0x7f208006c8f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2098000fb0 tx=0x7f2098009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.691+0000 7f20a5600640 1 -- 172.21.15.39:0/4025472737 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f20900530a0 con 0x7f20a80d2d50 2024-01-24T17:45:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.897+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f20a80ccba0 con 0x7f20a80d2d50 2024-01-24T17:45:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.898+0000 7f20a5600640 1 -- 172.21.15.39:0/4025472737 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v63) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2090018020 con 0x7f20a80d2d50 2024-01-24T17:45:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:45:38.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.900+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f208006a430 msgr2=0x7f208006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.900+0000 7f20aec00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f208006a430 0x7f208006c8f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2098000fb0 tx=0x7f2098009040 comp rx=0 tx=0).stop 2024-01-24T17:45:38.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.900+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 msgr2=0x7f20a8157730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.900+0000 7f20aec00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a8157730 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f209000c4a0 tx=0x7f209000c960 comp rx=0 tx=0).stop 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.900+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 shutdown_connections 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.900+0000 7f20aec00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f20a815c900 0x7f20a8145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.901+0000 7f20aec00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f208006a430 0x7f208006c8f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.901+0000 7f20aec00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a80d2d50 0x7f20a8157730 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.901+0000 7f20aec00640 1 --2- 172.21.15.39:0/4025472737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f20a806c2e0 0x7f20a81f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.901+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 >> 172.21.15.39:0/4025472737 conn(0x7f20a809bb30 msgr2=0x7f20a80d2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.901+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 shutdown_connections 2024-01-24T17:45:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:38.901+0000 7f20aec00640 1 -- 172.21.15.39:0/4025472737 wait complete. 2024-01-24T17:45:38.922 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:45:38.922 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:45:38.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:39.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.189+0000 7f4290a00640 1 Processor -- start 2024-01-24T17:45:39.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.190+0000 7f4290a00640 1 -- start start 2024-01-24T17:45:39.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.191+0000 7f4290a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f428c15ca50 0x7f428c0d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.191+0000 7f4290a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c0d1680 0x7f428c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.191+0000 7f4290a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f428c0cf770 0x7f428c0cd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.191+0000 7f4290a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428c0d0240 con 0x7f428c15ca50 2024-01-24T17:45:39.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.191+0000 7f4290a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f428c0cff60 con 0x7f428c0cf770 2024-01-24T17:45:39.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.191+0000 7f4290a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428c0d0660 con 0x7f428c0d1680 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f428c15ca50 0x7f428c0d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f428c15ca50 0x7f428c0d1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47440/0 (socket says 172.21.15.39:47440) 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428b400640 1 -- 172.21.15.39:0/2807102458 learned_addr learned my addr 172.21.15.39:0/2807102458 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428be00640 1 --2- 172.21.15.39:0/2807102458 >> v2:172.21.15.100:3300/0 conn(0x7f428c0cf770 0x7f428c0cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428aa00640 1 --2- 172.21.15.39:0/2807102458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c0d1680 0x7f428c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428b400640 1 -- 172.21.15.39:0/2807102458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c0d1680 msgr2=0x7f428c0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428b400640 1 --2- 172.21.15.39:0/2807102458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c0d1680 0x7f428c0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428b400640 1 -- 172.21.15.39:0/2807102458 >> v2:172.21.15.100:3300/0 conn(0x7f428c0cf770 msgr2=0x7f428c0cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:39.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428b400640 1 --2- 172.21.15.39:0/2807102458 >> v2:172.21.15.100:3300/0 conn(0x7f428c0cf770 0x7f428c0cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.192+0000 7f428b400640 1 -- 172.21.15.39:0/2807102458 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c0d0940 con 0x7f428c15ca50 2024-01-24T17:45:39.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.193+0000 7f428b400640 1 --2- 172.21.15.39:0/2807102458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c15ca50 0x7f428c0d1140 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f428c08bda0 tx=0x7f427c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416d7b90efe1f2fa server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:39.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.194+0000 7f428a000640 1 -- 172.21.15.39:0/2807102458 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c0317d0 con 0x7f428c15ca50 2024-01-24T17:45:39.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.194+0000 7f428a000640 1 -- 172.21.15.39:0/2807102458 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c031930 con 0x7f428c15ca50 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.194+0000 7f428a000640 1 -- 172.21.15.39:0/2807102458 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c031cc0 con 0x7f428c15ca50 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.194+0000 7f4290a00640 1 -- 172.21.15.39:0/2807102458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c15ca50 msgr2=0x7f428c0d1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.194+0000 7f4290a00640 1 --2- 172.21.15.39:0/2807102458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c15ca50 0x7f428c0d1140 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f428c08bda0 tx=0x7f427c02b870 comp rx=0 tx=0).stop 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.195+0000 7f4290a00640 1 -- 172.21.15.39:0/2807102458 shutdown_connections 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.195+0000 7f4290a00640 1 --2- 172.21.15.39:0/2807102458 >> v2:172.21.15.100:3300/0 conn(0x7f428c0cf770 0x7f428c0cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.195+0000 7f4290a00640 1 --2- 172.21.15.39:0/2807102458 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c0d1680 0x7f428c0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.195+0000 7f4290a00640 1 --2- 172.21.15.39:0/2807102458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c15ca50 0x7f428c0d1140 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.195+0000 7f4290a00640 1 -- 172.21.15.39:0/2807102458 >> 172.21.15.39:0/2807102458 conn(0x7f428c071dd0 msgr2=0x7f428c09bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.195+0000 7f4290a00640 1 -- 172.21.15.39:0/2807102458 shutdown_connections 2024-01-24T17:45:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.195+0000 7f4290a00640 1 -- 172.21.15.39:0/2807102458 wait complete. 2024-01-24T17:45:39.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.196+0000 7f4290a00640 1 Processor -- start 2024-01-24T17:45:39.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.197+0000 7f4290a00640 1 -- start start 2024-01-24T17:45:39.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f4290a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c0cf770 0x7f428c157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f4290a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f428c0d1680 0x7f428c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f4290a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c15ca50 0x7f428c15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f4290a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c0b3290 con 0x7f428c0cf770 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f4290a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c0b2ca0 con 0x7f428c0d1680 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f4290a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428c15fc70 con 0x7f428c15ca50 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c0cf770 0x7f428c157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c0cf770 0x7f428c157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47452/0 (socket says 172.21.15.39:47452) 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428b400640 1 -- 172.21.15.39:0/3555173155 learned_addr learned my addr 172.21.15.39:0/3555173155 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428aa00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f428c0d1680 0x7f428c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428be00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c15ca50 0x7f428c15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428b400640 1 -- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c15ca50 msgr2=0x7f428c15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428b400640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c15ca50 0x7f428c15be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428b400640 1 -- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f428c0d1680 msgr2=0x7f428c15b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428b400640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f428c0d1680 0x7f428c15b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.198+0000 7f428b400640 1 -- 172.21.15.39:0/3555173155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c0094d0 con 0x7f428c0cf770 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.199+0000 7f428b400640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c0cf770 0x7f428c157dc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f427c000c00 tx=0x7f427c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.199+0000 7f4283400640 1 -- 172.21.15.39:0/3555173155 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c032870 con 0x7f428c0cf770 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.199+0000 7f4283400640 1 -- 172.21.15.39:0/3555173155 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c032a10 con 0x7f428c0cf770 2024-01-24T17:45:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.199+0000 7f4283400640 1 -- 172.21.15.39:0/3555173155 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c032dc0 con 0x7f428c0cf770 2024-01-24T17:45:39.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.199+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c15ff50 con 0x7f428c0cf770 2024-01-24T17:45:39.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.199+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c158300 con 0x7f428c0cf770 2024-01-24T17:45:39.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.200+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c0677b0 con 0x7f428c0cf770 2024-01-24T17:45:39.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.201+0000 7f4283400640 1 -- 172.21.15.39:0/3555173155 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f427c030070 con 0x7f428c0cf770 2024-01-24T17:45:39.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.202+0000 7f4283400640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f426406a460 0x7f426406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.202+0000 7f4283400640 1 -- 172.21.15.39:0/3555173155 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f427c0aa780 con 0x7f428c0cf770 2024-01-24T17:45:39.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.203+0000 7f428aa00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f426406a460 0x7f426406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.204+0000 7f428aa00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f426406a460 0x7f426406c920 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f42740070b0 tx=0x7f4274007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:39.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.206+0000 7f4283400640 1 -- 172.21.15.39:0/3555173155 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f427c075d60 con 0x7f428c0cf770 2024-01-24T17:45:39.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.404+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f428c0cd910 con 0x7f428c0cf770 2024-01-24T17:45:39.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.405+0000 7f4283400640 1 -- 172.21.15.39:0/3555173155 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f427c079380 con 0x7f428c0cf770 2024-01-24T17:45:39.407 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:39.407 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:39.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f426406a460 msgr2=0x7f426406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f426406a460 0x7f426406c920 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f42740070b0 tx=0x7f4274007040 comp rx=0 tx=0).stop 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c0cf770 msgr2=0x7f428c157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c0cf770 0x7f428c157dc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f427c000c00 tx=0x7f427c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 shutdown_connections 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f428c15ca50 0x7f428c15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f428c0d1680 0x7f428c15b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f426406a460 0x7f426406c920 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 --2- 172.21.15.39:0/3555173155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f428c0cf770 0x7f428c157dc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 >> 172.21.15.39:0/3555173155 conn(0x7f428c071dd0 msgr2=0x7f428c159e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 shutdown_connections 2024-01-24T17:45:39.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.408+0000 7f4290a00640 1 -- 172.21.15.39:0/3555173155 wait complete. 2024-01-24T17:45:39.429 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:45:16.503215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:45:17.386565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:45:39.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:45:39.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.650+0000 7fcf23200640 1 Processor -- start 2024-01-24T17:45:39.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.651+0000 7fcf23200640 1 -- start start 2024-01-24T17:45:39.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.652+0000 7fcf23200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf2415c9e0 0x7fcf240b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.652+0000 7fcf23200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcf240b8250 0x7fcf240cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.652+0000 7fcf23200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf240b7e30 con 0x7fcf240cdb90 2024-01-24T17:45:39.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.652+0000 7fcf23200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf240ce020 con 0x7fcf240b8250 2024-01-24T17:45:39.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.652+0000 7fcf23200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240b6d90 con 0x7fcf2415c9e0 2024-01-24T17:45:39.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcf240b8250 0x7fcf240cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcf240b8250 0x7fcf240cd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47456/0 (socket says 172.21.15.39:47456) 2024-01-24T17:45:39.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21400640 1 -- 172.21.15.39:0/2362010081 learned_addr learned my addr 172.21.15.39:0/2362010081 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:39.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21e00640 1 --2- 172.21.15.39:0/2362010081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf2415c9e0 0x7fcf240b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21400640 1 -- 172.21.15.39:0/2362010081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf2415c9e0 msgr2=0x7fcf240b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:39.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21400640 1 --2- 172.21.15.39:0/2362010081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf2415c9e0 0x7fcf240b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21400640 1 -- 172.21.15.39:0/2362010081 >> v1:172.21.15.100:6789/0 conn(0x7fcf240cdb90 legacy=0x7fcf240b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:39.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21400640 1 -- 172.21.15.39:0/2362010081 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240b7070 con 0x7fcf240b8250 2024-01-24T17:45:39.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.653+0000 7fcf21e00640 1 --2- 172.21.15.39:0/2362010081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf2415c9e0 0x7fcf240b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:39.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.654+0000 7fcf21400640 1 --2- 172.21.15.39:0/2362010081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf240b8250 0x7fcf240cd650 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009e50 tx=0x7fcf0c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67be712570f49800 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:39.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.655+0000 7fcf20a00640 1 -- 172.21.15.39:0/2362010081 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c02cb00 con 0x7fcf240b8250 2024-01-24T17:45:39.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.655+0000 7fcf20a00640 1 -- 172.21.15.39:0/2362010081 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c02cc60 con 0x7fcf240b8250 2024-01-24T17:45:39.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.655+0000 7fcf20a00640 1 -- 172.21.15.39:0/2362010081 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0343f0 con 0x7fcf240b8250 2024-01-24T17:45:39.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.655+0000 7fcf23200640 1 -- 172.21.15.39:0/2362010081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf240b8250 msgr2=0x7fcf240cd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.655+0000 7fcf23200640 1 --2- 172.21.15.39:0/2362010081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf240b8250 0x7fcf240cd650 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009e50 tx=0x7fcf0c02bf60 comp rx=0 tx=0).stop 2024-01-24T17:45:39.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.656+0000 7fcf23200640 1 -- 172.21.15.39:0/2362010081 shutdown_connections 2024-01-24T17:45:39.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.656+0000 7fcf23200640 1 --2- 172.21.15.39:0/2362010081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf240b8250 0x7fcf240cd650 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.656+0000 7fcf23200640 1 --2- 172.21.15.39:0/2362010081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf2415c9e0 0x7fcf240b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.656+0000 7fcf23200640 1 -- 172.21.15.39:0/2362010081 >> 172.21.15.39:0/2362010081 conn(0x7fcf24071790 msgr2=0x7fcf24071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:39.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.656+0000 7fcf23200640 1 -- 172.21.15.39:0/2362010081 shutdown_connections 2024-01-24T17:45:39.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.656+0000 7fcf23200640 1 -- 172.21.15.39:0/2362010081 wait complete. 2024-01-24T17:45:39.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.657+0000 7fcf23200640 1 Processor -- start 2024-01-24T17:45:39.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.657+0000 7fcf23200640 1 -- start start 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.658+0000 7fcf23200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf240b8250 0x7fcf241f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.658+0000 7fcf23200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf240cdb90 0x7fcf2414cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.658+0000 7fcf23200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf2415c9e0 0x7fcf241505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.658+0000 7fcf23200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2415ee20 con 0x7fcf2415c9e0 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.658+0000 7fcf23200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2415e210 con 0x7fcf240b8250 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.658+0000 7fcf23200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2415e4f0 con 0x7fcf240cdb90 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf22800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf2415c9e0 0x7fcf241505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf22800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf2415c9e0 0x7fcf241505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47470/0 (socket says 172.21.15.39:47470) 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf22800640 1 -- 172.21.15.39:0/2438087002 learned_addr learned my addr 172.21.15.39:0/2438087002 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:39.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf21e00640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf240b8250 0x7fcf241f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf21400640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf240cdb90 0x7fcf2414cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf22800640 1 -- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf240cdb90 msgr2=0x7fcf2414cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf22800640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf240cdb90 0x7fcf2414cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf22800640 1 -- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf240b8250 msgr2=0x7fcf241f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf22800640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf240b8250 0x7fcf241f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.659+0000 7fcf22800640 1 -- 172.21.15.39:0/2438087002 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c009b30 con 0x7fcf2415c9e0 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.660+0000 7fcf22800640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf2415c9e0 0x7fcf241505d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00c410 tx=0x7fcf1c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.660+0000 7fcf1a000640 1 -- 172.21.15.39:0/2438087002 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c005230 con 0x7fcf2415c9e0 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.660+0000 7fcf1a000640 1 -- 172.21.15.39:0/2438087002 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c005390 con 0x7fcf2415c9e0 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.660+0000 7fcf1a000640 1 -- 172.21.15.39:0/2438087002 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c0117d0 con 0x7fcf2415c9e0 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.660+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2415e7d0 con 0x7fcf2415c9e0 2024-01-24T17:45:39.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.660+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2414d1d0 con 0x7fcf2415c9e0 2024-01-24T17:45:39.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.661+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2415ef90 con 0x7fcf2415c9e0 2024-01-24T17:45:39.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.662+0000 7fcf1a000640 1 -- 172.21.15.39:0/2438087002 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcf1c011930 con 0x7fcf2415c9e0 2024-01-24T17:45:39.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.663+0000 7fcf1a000640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcefc06a430 0x7fcefc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:39.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.663+0000 7fcf1a000640 1 -- 172.21.15.39:0/2438087002 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf1c087de0 con 0x7fcf2415c9e0 2024-01-24T17:45:39.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.664+0000 7fcf21e00640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcefc06a430 0x7fcefc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:39.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.665+0000 7fcf21e00640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcefc06a430 0x7fcefc06c8f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400a950 tx=0x7fcf14008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:39.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.668+0000 7fcf1a000640 1 -- 172.21.15.39:0/2438087002 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcf1c053330 con 0x7fcf2415c9e0 2024-01-24T17:45:39.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.860+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf24067040 con 0x7fcf2415c9e0 2024-01-24T17:45:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.861+0000 7fcf1a000640 1 -- 172.21.15.39:0/2438087002 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf1c056950 con 0x7fcf2415c9e0 2024-01-24T17:45:39.863 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:45:39.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcefc06a430 msgr2=0x7fcefc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcefc06a430 0x7fcefc06c8f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400a950 tx=0x7fcf14008040 comp rx=0 tx=0).stop 2024-01-24T17:45:39.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf2415c9e0 msgr2=0x7fcf241505d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:39.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf2415c9e0 0x7fcf241505d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00c410 tx=0x7fcf1c00c8d0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 shutdown_connections 2024-01-24T17:45:39.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcefc06a430 0x7fcefc06c8f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf2415c9e0 0x7fcf241505d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcf240cdb90 0x7fcf2414cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 --2- 172.21.15.39:0/2438087002 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcf240b8250 0x7fcf241f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:39.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 >> 172.21.15.39:0/2438087002 conn(0x7fcf24071790 msgr2=0x7fcf2415afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:39.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 shutdown_connections 2024-01-24T17:45:39.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:39.864+0000 7fcf23200640 1 -- 172.21.15.39:0/2438087002 wait complete. 2024-01-24T17:45:39.886 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:45:40.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.104+0000 7efc87e00640 1 Processor -- start 2024-01-24T17:45:40.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.104+0000 7efc87e00640 1 -- start start 2024-01-24T17:45:40.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc87e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efc8015fff0 0x7efc800b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc87e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc800cda90 con 0x7efc800d14c0 2024-01-24T17:45:40.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc87e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc800d32c0 con 0x7efc800b2a10 2024-01-24T17:45:40.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc87e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc800b2e90 con 0x7efc8015fff0 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc85e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efc800d14c0 0x7efc800cdf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59926/0 (socket says 172.21.15.39:59926) 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc85e00640 1 -- 172.21.15.39:0/3933787146 learned_addr learned my addr 172.21.15.39:0/3933787146 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc85400640 1 --2- 172.21.15.39:0/3933787146 >> v2:172.21.15.100:3300/0 conn(0x7efc8015fff0 0x7efc800b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604467215 0 0) 0x7efc800cda90 con 0x7efc800d14c0 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60003520 con 0x7efc800d14c0 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554646766 0 0) 0x7efc60003520 con 0x7efc800d14c0 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 >> v2:172.21.15.100:3300/0 conn(0x7efc8015fff0 msgr2=0x7efc800b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 --2- 172.21.15.39:0/3933787146 >> v2:172.21.15.100:3300/0 conn(0x7efc8015fff0 0x7efc800b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 >> v1:172.21.15.100:6789/0 conn(0x7efc800b2a10 legacy=0x7efc800d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc800b30a0 con 0x7efc800d14c0 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7efc780019f0 con 0x7efc800d14c0 2024-01-24T17:45:40.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 >> v1:172.21.15.39:6789/0 conn(0x7efc800d14c0 legacy=0x7efc800cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc60003970 0x7efc60003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc60005180 0x7efc6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc800b2e90 con 0x7efc600047c0 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6000bfd0 con 0x7efc60005180 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.107+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6000c190 con 0x7efc60003970 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc84a00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc85e00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc60005180 0x7efc6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc85400640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc60003970 0x7efc60003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc84a00640 1 -- 172.21.15.39:0/3933787146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc60003970 msgr2=0x7efc60003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc84a00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc60003970 0x7efc60003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc84a00640 1 -- 172.21.15.39:0/3933787146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc60005180 msgr2=0x7efc6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:40.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc84a00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc60005180 0x7efc6000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc84a00640 1 -- 172.21.15.39:0/3933787146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60003520 con 0x7efc600047c0 2024-01-24T17:45:40.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.108+0000 7efc85e00640 1 -- 172.21.15.39:0/3933787146 reap_dead start 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.109+0000 7efc84a00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc60004c40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efc68001830 tx=0x7efc600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae231c93cdc8be98 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68032600 con 0x7efc600047c0 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6802e050 con 0x7efc600047c0 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc7fe00640 1 -- 172.21.15.39:0/3933787146 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc680329c0 con 0x7efc600047c0 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc87e00640 1 -- 172.21.15.39:0/3933787146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 msgr2=0x7efc60004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc87e00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc60004c40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efc68001830 tx=0x7efc600057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc87e00640 1 -- 172.21.15.39:0/3933787146 shutdown_connections 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc87e00640 1 --2- 172.21.15.39:0/3933787146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc60004c40 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc87e00640 1 -- 172.21.15.39:0/3933787146 >> 172.21.15.39:0/3933787146 conn(0x7efc8009bcc0 msgr2=0x7efc800d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:40.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc87e00640 1 -- 172.21.15.39:0/3933787146 shutdown_connections 2024-01-24T17:45:40.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.110+0000 7efc87e00640 1 -- 172.21.15.39:0/3933787146 wait complete. 2024-01-24T17:45:40.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.112+0000 7efc87e00640 1 Processor -- start 2024-01-24T17:45:40.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.112+0000 7efc87e00640 1 -- start start 2024-01-24T17:45:40.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.113+0000 7efc87e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc80150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.113+0000 7efc87e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc801e52f0 0x7efc801527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.113+0000 7efc87e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc800b0dd0 0x7efc800b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.113+0000 7efc87e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc800b1790 con 0x7efc600047c0 2024-01-24T17:45:40.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.113+0000 7efc87e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc800caf60 con 0x7efc800b0dd0 2024-01-24T17:45:40.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.113+0000 7efc87e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc80153850 con 0x7efc801e52f0 2024-01-24T17:45:40.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc80150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc80150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47494/0 (socket says 172.21.15.39:47494) 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85400640 1 -- 172.21.15.39:0/690492667 learned_addr learned my addr 172.21.15.39:0/690492667 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc84a00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc801e52f0 0x7efc801527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85e00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc800b0dd0 0x7efc800b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85400640 1 -- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc801e52f0 msgr2=0x7efc801527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85400640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc801e52f0 0x7efc801527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85400640 1 -- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc800b0dd0 msgr2=0x7efc800b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85400640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc800b0dd0 0x7efc800b1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.114+0000 7efc85400640 1 -- 172.21.15.39:0/690492667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc800cb220 con 0x7efc600047c0 2024-01-24T17:45:40.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.115+0000 7efc85400640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc80150f00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efc70011510 tx=0x7efc700119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:40.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.115+0000 7efc7d600640 1 -- 172.21.15.39:0/690492667 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc70015870 con 0x7efc600047c0 2024-01-24T17:45:40.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.115+0000 7efc7d600640 1 -- 172.21.15.39:0/690492667 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc700159d0 con 0x7efc600047c0 2024-01-24T17:45:40.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.115+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc80152d10 con 0x7efc600047c0 2024-01-24T17:45:40.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.115+0000 7efc7d600640 1 -- 172.21.15.39:0/690492667 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc70015d40 con 0x7efc600047c0 2024-01-24T17:45:40.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.115+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc80153290 con 0x7efc600047c0 2024-01-24T17:45:40.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.116+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc80073ef0 con 0x7efc600047c0 2024-01-24T17:45:40.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.117+0000 7efc7d600640 1 -- 172.21.15.39:0/690492667 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efc70013400 con 0x7efc600047c0 2024-01-24T17:45:40.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.118+0000 7efc7d600640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5806a460 0x7efc5806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.119+0000 7efc7d600640 1 -- 172.21.15.39:0/690492667 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc70089160 con 0x7efc600047c0 2024-01-24T17:45:40.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.119+0000 7efc84a00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5806a460 0x7efc5806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.120+0000 7efc84a00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5806a460 0x7efc5806c920 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efc68035000 tx=0x7efc68030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:40.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.124+0000 7efc7d600640 1 -- 172.21.15.39:0/690492667 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efc700546b0 con 0x7efc600047c0 2024-01-24T17:45:40.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.319+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc801e6d40 con 0x7efc600047c0 2024-01-24T17:45:40.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.319+0000 7efc7d600640 1 -- 172.21.15.39:0/690492667 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v63) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc70057cd0 con 0x7efc600047c0 2024-01-24T17:45:40.321 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:45:40.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5806a460 msgr2=0x7efc5806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5806a460 0x7efc5806c920 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efc68035000 tx=0x7efc68030000 comp rx=0 tx=0).stop 2024-01-24T17:45:40.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 msgr2=0x7efc80150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc80150f00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7efc70011510 tx=0x7efc700119d0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 shutdown_connections 2024-01-24T17:45:40.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc800b0dd0 0x7efc800b1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc801e52f0 0x7efc801527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc5806a460 0x7efc5806c920 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 --2- 172.21.15.39:0/690492667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600047c0 0x7efc80150f00 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 >> 172.21.15.39:0/690492667 conn(0x7efc8009bcc0 msgr2=0x7efc80155080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:40.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 shutdown_connections 2024-01-24T17:45:40.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.322+0000 7efc87e00640 1 -- 172.21.15.39:0/690492667 wait complete. 2024-01-24T17:45:40.343 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:45:40.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.573+0000 7efeca200640 1 Processor -- start 2024-01-24T17:45:40.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.573+0000 7efeca200640 1 -- start start 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.573+0000 7efeca200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efec41600d0 0x7efec4070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.573+0000 7efeca200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efec41607d0 0x7efec4160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.573+0000 7efeca200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec4070660 con 0x7efec40b00e0 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.573+0000 7efeca200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec41612d0 con 0x7efec41607d0 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.573+0000 7efeca200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec41615b0 con 0x7efec41600d0 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.574+0000 7efec2a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efec41607d0 0x7efec4160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.574+0000 7efec2a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efec41607d0 0x7efec4160c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47500/0 (socket says 172.21.15.39:47500) 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.574+0000 7efec2a00640 1 -- 172.21.15.39:0/464548507 learned_addr learned my addr 172.21.15.39:0/464548507 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.574+0000 7efec2a00640 1 -- 172.21.15.39:0/464548507 >> v2:172.21.15.100:3300/0 conn(0x7efec41600d0 msgr2=0x7efec4070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:40.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.574+0000 7efec2a00640 1 --2- 172.21.15.39:0/464548507 >> v2:172.21.15.100:3300/0 conn(0x7efec41600d0 0x7efec4070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.574+0000 7efec2a00640 1 -- 172.21.15.39:0/464548507 >> v1:172.21.15.100:6789/0 conn(0x7efec40b00e0 legacy=0x7efec40ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:40.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.574+0000 7efec2a00640 1 -- 172.21.15.39:0/464548507 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec4070010 con 0x7efec41607d0 2024-01-24T17:45:40.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.575+0000 7efec2a00640 1 --2- 172.21.15.39:0/464548507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec41607d0 0x7efec4160c60 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efec4066f40 tx=0x7efeac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45e8e0706066fd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:40.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.576+0000 7efec2000640 1 -- 172.21.15.39:0/464548507 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac031870 con 0x7efec41607d0 2024-01-24T17:45:40.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.576+0000 7efec2000640 1 -- 172.21.15.39:0/464548507 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeac0319d0 con 0x7efec41607d0 2024-01-24T17:45:40.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.576+0000 7efec2000640 1 -- 172.21.15.39:0/464548507 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeac031d60 con 0x7efec41607d0 2024-01-24T17:45:40.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.576+0000 7efeca200640 1 -- 172.21.15.39:0/464548507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec41607d0 msgr2=0x7efec4160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.576+0000 7efeca200640 1 --2- 172.21.15.39:0/464548507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec41607d0 0x7efec4160c60 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7efec4066f40 tx=0x7efeac02b910 comp rx=0 tx=0).stop 2024-01-24T17:45:40.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.577+0000 7efeca200640 1 -- 172.21.15.39:0/464548507 shutdown_connections 2024-01-24T17:45:40.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.577+0000 7efeca200640 1 --2- 172.21.15.39:0/464548507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec41607d0 0x7efec4160c60 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.577+0000 7efeca200640 1 --2- 172.21.15.39:0/464548507 >> v2:172.21.15.100:3300/0 conn(0x7efec41600d0 0x7efec4070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.577+0000 7efeca200640 1 -- 172.21.15.39:0/464548507 >> 172.21.15.39:0/464548507 conn(0x7efec40cb0e0 msgr2=0x7efec40cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:40.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.577+0000 7efeca200640 1 -- 172.21.15.39:0/464548507 shutdown_connections 2024-01-24T17:45:40.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.577+0000 7efeca200640 1 -- 172.21.15.39:0/464548507 wait complete. 2024-01-24T17:45:40.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.579+0000 7efeca200640 1 Processor -- start 2024-01-24T17:45:40.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.579+0000 7efeca200640 1 -- start start 2024-01-24T17:45:40.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efeca200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec40b00e0 0x7efec40c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efeca200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efec41600d0 0x7efec4150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efeca200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efec41607d0 0x7efec41522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efeca200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec40c9b70 con 0x7efec40b00e0 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efeca200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec40c9e50 con 0x7efec41607d0 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efeca200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec40ca130 con 0x7efec41600d0 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efec3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec40b00e0 0x7efec40c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efec3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec40b00e0 0x7efec40c9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47512/0 (socket says 172.21.15.39:47512) 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.580+0000 7efec3400640 1 -- 172.21.15.39:0/1660702187 learned_addr learned my addr 172.21.15.39:0/1660702187 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efec2a00640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efec41600d0 0x7efec4150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efec3400640 1 -- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efec41600d0 msgr2=0x7efec4150a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efec3e00640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efec41607d0 0x7efec41522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efec3400640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efec41600d0 0x7efec4150a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efec3400640 1 -- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efec41607d0 msgr2=0x7efec41522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efec3400640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efec41607d0 0x7efec41522f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efec3400640 1 -- 172.21.15.39:0/1660702187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeac0094d0 con 0x7efec40b00e0 2024-01-24T17:45:40.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efec3400640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec40b00e0 0x7efec40c9290 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7efeb400b410 tx=0x7efeb400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:40.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efebb400640 1 -- 172.21.15.39:0/1660702187 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4011830 con 0x7efec40b00e0 2024-01-24T17:45:40.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efebb400640 1 -- 172.21.15.39:0/1660702187 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb40119b0 con 0x7efec40b00e0 2024-01-24T17:45:40.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.581+0000 7efebb400640 1 -- 172.21.15.39:0/1660702187 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4011d60 con 0x7efec40b00e0 2024-01-24T17:45:40.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.582+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec40ca410 con 0x7efec40b00e0 2024-01-24T17:45:40.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.582+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec40cac10 con 0x7efec40b00e0 2024-01-24T17:45:40.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.583+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec4073ef0 con 0x7efec40b00e0 2024-01-24T17:45:40.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.584+0000 7efebb400640 1 -- 172.21.15.39:0/1660702187 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efeb401a400 con 0x7efec40b00e0 2024-01-24T17:45:40.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.584+0000 7efebb400640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9c06a480 0x7efe9c06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:40.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.585+0000 7efebb400640 1 -- 172.21.15.39:0/1660702187 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efeb40889a0 con 0x7efec40b00e0 2024-01-24T17:45:40.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.585+0000 7efec2a00640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9c06a480 0x7efe9c06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:40.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.586+0000 7efec2a00640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9c06a480 0x7efe9c06c940 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efeac036000 tx=0x7efeac008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:40.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.590+0000 7efebb400640 1 -- 172.21.15.39:0/1660702187 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efeb4054070 con 0x7efec40b00e0 2024-01-24T17:45:40.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.782+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efec40cbe80 con 0x7efec40b00e0 2024-01-24T17:45:40.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.784+0000 7efebb400640 1 -- 172.21.15.39:0/1660702187 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v63) v1 ==== 100+0+11 (secure 0 0 0) 0x7efeb4015020 con 0x7efec40b00e0 2024-01-24T17:45:40.785 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:45:40.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9c06a480 msgr2=0x7efe9c06c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9c06a480 0x7efe9c06c940 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efeac036000 tx=0x7efeac008030 comp rx=0 tx=0).stop 2024-01-24T17:45:40.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec40b00e0 msgr2=0x7efec40c9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:40.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec40b00e0 0x7efec40c9290 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7efeb400b410 tx=0x7efeb400b8d0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 shutdown_connections 2024-01-24T17:45:40.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efec41607d0 0x7efec41522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efec41600d0 0x7efec4150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9c06a480 0x7efe9c06c940 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 --2- 172.21.15.39:0/1660702187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efec40b00e0 0x7efec40c9290 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:40.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 >> 172.21.15.39:0/1660702187 conn(0x7efec40cb0e0 msgr2=0x7efec406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:40.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.787+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 shutdown_connections 2024-01-24T17:45:40.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:40.788+0000 7efeca200640 1 -- 172.21.15.39:0/1660702187 wait complete. 2024-01-24T17:45:40.809 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:45:41.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.029+0000 7fada1800640 1 Processor -- start 2024-01-24T17:45:41.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.029+0000 7fada1800640 1 -- start start 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.030+0000 7fada1800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c0cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.030+0000 7fada1800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fad9c0b1390 0x7fad9c0b1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.030+0000 7fada1800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c0cb8f0 con 0x7fad9c15a950 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.030+0000 7fada1800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9c0b43d0 con 0x7fad9c0b1390 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.030+0000 7fada1800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c0c8f70 con 0x7fad9c0caf50 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.031+0000 7fad9aa00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fad9c15a950 0x7fad9c0ca560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59934/0 (socket says 172.21.15.39:59934) 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.031+0000 7fad9aa00640 1 -- 172.21.15.39:0/471055780 learned_addr learned my addr 172.21.15.39:0/471055780 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.031+0000 7fad9b400640 1 --2- 172.21.15.39:0/471055780 >> v2:172.21.15.39:3300/0 conn(0x7fad9c0b1390 0x7fad9c0b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.031+0000 7fad93e00640 1 --2- 172.21.15.39:0/471055780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c0cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.031+0000 7fad9b400640 1 -- 172.21.15.39:0/471055780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 msgr2=0x7fad9c0cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.031+0000 7fad9b400640 1 --2- 172.21.15.39:0/471055780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c0cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.031+0000 7fad9b400640 1 -- 172.21.15.39:0/471055780 >> v1:172.21.15.39:6789/0 conn(0x7fad9c15a950 legacy=0x7fad9c0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:41.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.031+0000 7fad9b400640 1 -- 172.21.15.39:0/471055780 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c0c9180 con 0x7fad9c0b1390 2024-01-24T17:45:41.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.032+0000 7fad9b400640 1 --2- 172.21.15.39:0/471055780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c0b1390 0x7fad9c0b1820 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fad94009d90 tx=0x7fad9402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2a6da090604331 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:41.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.033+0000 7fad9a000640 1 -- 172.21.15.39:0/471055780 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9402cc60 con 0x7fad9c0b1390 2024-01-24T17:45:41.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.033+0000 7fad9a000640 1 -- 172.21.15.39:0/471055780 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9402cdc0 con 0x7fad9c0b1390 2024-01-24T17:45:41.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.033+0000 7fad9a000640 1 -- 172.21.15.39:0/471055780 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94034660 con 0x7fad9c0b1390 2024-01-24T17:45:41.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.033+0000 7fada1800640 1 -- 172.21.15.39:0/471055780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c0b1390 msgr2=0x7fad9c0b1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:41.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.033+0000 7fada1800640 1 --2- 172.21.15.39:0/471055780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c0b1390 0x7fad9c0b1820 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fad94009d90 tx=0x7fad9402beb0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.033+0000 7fada1800640 1 -- 172.21.15.39:0/471055780 shutdown_connections 2024-01-24T17:45:41.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.034+0000 7fada1800640 1 --2- 172.21.15.39:0/471055780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c0b1390 0x7fad9c0b1820 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.034+0000 7fada1800640 1 --2- 172.21.15.39:0/471055780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c0cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.034+0000 7fada1800640 1 -- 172.21.15.39:0/471055780 >> 172.21.15.39:0/471055780 conn(0x7fad9c06a3e0 msgr2=0x7fad9c06bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:41.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.034+0000 7fada1800640 1 -- 172.21.15.39:0/471055780 shutdown_connections 2024-01-24T17:45:41.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.034+0000 7fada1800640 1 -- 172.21.15.39:0/471055780 wait complete. 2024-01-24T17:45:41.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.035+0000 7fada1800640 1 Processor -- start 2024-01-24T17:45:41.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.035+0000 7fada1800640 1 -- start start 2024-01-24T17:45:41.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.036+0000 7fada1800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9c0b1390 0x7fad9c0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.036+0000 7fada1800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c14d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.036+0000 7fada1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c15a950 0x7fad9c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.036+0000 7fada1800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c15c330 con 0x7fad9c15a950 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.036+0000 7fada1800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c15b2f0 con 0x7fad9c0b1390 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.036+0000 7fada1800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c15b5d0 con 0x7fad9c0caf50 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c15a950 0x7fad9c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c15a950 0x7fad9c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47520/0 (socket says 172.21.15.39:47520) 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9b400640 1 -- 172.21.15.39:0/1900739414 learned_addr learned my addr 172.21.15.39:0/1900739414 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9aa00640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9c0b1390 0x7fad9c0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad93e00640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c14d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9b400640 1 -- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 msgr2=0x7fad9c14d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9b400640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c14d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9b400640 1 -- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9c0b1390 msgr2=0x7fad9c0afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9b400640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9c0b1390 0x7fad9c0afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad9b400640 1 -- 172.21.15.39:0/1900739414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94009a70 con 0x7fad9c15a950 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.037+0000 7fad93e00640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c14d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.038+0000 7fad9b400640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c15a950 0x7fad9c0c5610 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fad94009d90 tx=0x7fad94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.038+0000 7fad92a00640 1 -- 172.21.15.39:0/1900739414 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94040650 con 0x7fad9c15a950 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.038+0000 7fad92a00640 1 -- 172.21.15.39:0/1900739414 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad940407d0 con 0x7fad9c15a950 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.038+0000 7fad92a00640 1 -- 172.21.15.39:0/1900739414 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94040b80 con 0x7fad9c15a950 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.038+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9c15b8b0 con 0x7fad9c15a950 2024-01-24T17:45:41.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.038+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9c0b00a0 con 0x7fad9c15a950 2024-01-24T17:45:41.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.039+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9c073ef0 con 0x7fad9c15a950 2024-01-24T17:45:41.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.040+0000 7fad92a00640 1 -- 172.21.15.39:0/1900739414 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fad94040ce0 con 0x7fad9c15a950 2024-01-24T17:45:41.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.041+0000 7fad92a00640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7406a430 0x7fad7406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:41.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.041+0000 7fad92a00640 1 -- 172.21.15.39:0/1900739414 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad940ab790 con 0x7fad9c15a950 2024-01-24T17:45:41.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.042+0000 7fad9aa00640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7406a430 0x7fad7406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:41.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.043+0000 7fad9aa00640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7406a430 0x7fad7406c8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fad8c005880 tx=0x7fad8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:41.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.046+0000 7fad92a00640 1 -- 172.21.15.39:0/1900739414 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fad94076d80 con 0x7fad9c15a950 2024-01-24T17:45:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.250+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fad9c0c8b40 con 0x7fad9c15a950 2024-01-24T17:45:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.251+0000 7fad92a00640 1 -- 172.21.15.39:0/1900739414 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v63) v1 ==== 80+0+0 (secure 0 0 0) 0x7fad9407a3a0 con 0x7fad9c15a950 2024-01-24T17:45:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7406a430 msgr2=0x7fad7406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7406a430 0x7fad7406c8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fad8c005880 tx=0x7fad8c009040 comp rx=0 tx=0).stop 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c15a950 msgr2=0x7fad9c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c15a950 0x7fad9c0c5610 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fad94009d90 tx=0x7fad94004a40 comp rx=0 tx=0).stop 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 shutdown_connections 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fad7406a430 0x7fad7406c8f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad9c15a950 0x7fad9c0c5610 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fad9c0caf50 0x7fad9c14d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 --2- 172.21.15.39:0/1900739414 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fad9c0b1390 0x7fad9c0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 >> 172.21.15.39:0/1900739414 conn(0x7fad9c06a3e0 msgr2=0x7fad9c0acde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 shutdown_connections 2024-01-24T17:45:41.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:41.254+0000 7fada1800640 1 -- 172.21.15.39:0/1900739414 wait complete. 2024-01-24T17:45:41.273 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:45:41.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:41.273 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:45:41.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[59441]: fuse finished with error 0 and tester_r 0 2024-01-24T17:45:41.364 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:45:47.466 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:45:47.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:47.466 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:45:47.527 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:45:47.528 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:45:47.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:47.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.813+0000 7ff753000640 1 Processor -- start 2024-01-24T17:45:47.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.814+0000 7ff753000640 1 -- start start 2024-01-24T17:45:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.815+0000 7ff753000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0cf990 0x7ff74c0cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.815+0000 7ff753000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff74c15fb90 0x7ff74c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.815+0000 7ff753000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74c0d02b0 con 0x7ff74c15d020 2024-01-24T17:45:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.815+0000 7ff753000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff74c15e080 con 0x7ff74c15fb90 2024-01-24T17:45:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.815+0000 7ff753000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c15dac0 con 0x7ff74c0cf990 2024-01-24T17:45:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff743200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff74c15d020 0x7ff74c160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59114/0 (socket says 172.21.15.39:59114) 2024-01-24T17:45:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff743200640 1 -- 172.21.15.39:0/2286670864 learned_addr learned my addr 172.21.15.39:0/2286670864 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:47.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff74be00640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0cf990 0x7ff74c0cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:47.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff751000640 1 --2- 172.21.15.39:0/2286670864 >> v2:172.21.15.100:3300/0 conn(0x7ff74c15fb90 0x7ff74c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:47.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575031722 0 0) 0x7ff74c0d02b0 con 0x7ff74c15d020 2024-01-24T17:45:47.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff72c003520 con 0x7ff74c15d020 2024-01-24T17:45:47.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031084862 0 0) 0x7ff72c003520 con 0x7ff74c15d020 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0cf990 msgr2=0x7ff74c0cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff74b400640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0cf990 0x7ff74c0cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.816+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 >> v2:172.21.15.100:3300/0 conn(0x7ff74c15fb90 msgr2=0x7ff74c15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 --2- 172.21.15.39:0/2286670864 >> v2:172.21.15.100:3300/0 conn(0x7ff74c15fb90 0x7ff74c15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c15dda0 con 0x7ff74c15d020 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff734001670 con 0x7ff74c15d020 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 >> v1:172.21.15.39:6789/0 conn(0x7ff74c15d020 legacy=0x7ff74c160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff72c003970 0x7ff72c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0cf990 0x7ff72c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff72c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c15e080 con 0x7ff72c004d00 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c15dac0 con 0x7ff72c003970 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c00b980 con 0x7ff74c0cf990 2024-01-24T17:45:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff751000640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff72c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff743200640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0cf990 0x7ff72c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.817+0000 7ff74be00640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff72c003970 0x7ff72c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.818+0000 7ff751000640 1 -- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0cf990 msgr2=0x7ff72c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.818+0000 7ff751000640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0cf990 0x7ff72c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.818+0000 7ff751000640 1 -- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff72c003970 msgr2=0x7ff72c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.818+0000 7ff751000640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff72c003970 0x7ff72c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.818+0000 7ff751000640 1 -- 172.21.15.39:0/2286670864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c003520 con 0x7ff72c004d00 2024-01-24T17:45:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.819+0000 7ff751000640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff72c005180 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7ff744009f90 tx=0x7ff72c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e325e8e60f97d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.819+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74402cc20 con 0x7ff72c004d00 2024-01-24T17:45:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.819+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74400f040 con 0x7ff72c004d00 2024-01-24T17:45:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.819+0000 7ff74b400640 1 -- 172.21.15.39:0/2286670864 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff744034440 con 0x7ff72c004d00 2024-01-24T17:45:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.820+0000 7ff753000640 1 -- 172.21.15.39:0/2286670864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 msgr2=0x7ff72c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.820+0000 7ff753000640 1 --2- 172.21.15.39:0/2286670864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff72c005180 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7ff744009f90 tx=0x7ff72c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.820+0000 7ff751000640 1 -- 172.21.15.39:0/2286670864 reap_dead start 2024-01-24T17:45:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.820+0000 7ff753000640 1 -- 172.21.15.39:0/2286670864 shutdown_connections 2024-01-24T17:45:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.820+0000 7ff753000640 1 -- 172.21.15.39:0/2286670864 >> 172.21.15.39:0/2286670864 conn(0x7ff74c071790 msgr2=0x7ff74c071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.820+0000 7ff753000640 1 -- 172.21.15.39:0/2286670864 shutdown_connections 2024-01-24T17:45:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.820+0000 7ff753000640 1 -- 172.21.15.39:0/2286670864 wait complete. 2024-01-24T17:45:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.822+0000 7ff753000640 1 Processor -- start 2024-01-24T17:45:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.822+0000 7ff753000640 1 -- start start 2024-01-24T17:45:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff753000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff74c156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff753000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c0d29a0 0x7ff74c0c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff753000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0c7580 0x7ff74c158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff753000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c0d3440 con 0x7ff72c004d00 2024-01-24T17:45:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff753000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c0c7ca0 con 0x7ff74c0d29a0 2024-01-24T17:45:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff753000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c0d2e80 con 0x7ff74c0c7580 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff74be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff74c156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff74be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff74c156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38968/0 (socket says 172.21.15.39:38968) 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff74be00640 1 -- 172.21.15.39:0/1411922721 learned_addr learned my addr 172.21.15.39:0/1411922721 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff74be00640 1 -- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0c7580 msgr2=0x7ff74c158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.823+0000 7ff751000640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0c7580 0x7ff74c158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff74be00640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0c7580 0x7ff74c158c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff74be00640 1 -- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c0d29a0 msgr2=0x7ff74c0c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff74be00640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c0d29a0 0x7ff74c0c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff74be00640 1 -- 172.21.15.39:0/1411922721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c0d3160 con 0x7ff72c004d00 2024-01-24T17:45:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff74be00640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff74c156cb0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff73c011600 tx=0x7ff73c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:47.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff748c00640 1 -- 172.21.15.39:0/1411922721 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c00f920 con 0x7ff72c004d00 2024-01-24T17:45:47.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff748c00640 1 -- 172.21.15.39:0/1411922721 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73c00fa80 con 0x7ff72c004d00 2024-01-24T17:45:47.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff748c00640 1 -- 172.21.15.39:0/1411922721 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c00fdf0 con 0x7ff72c004d00 2024-01-24T17:45:47.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74c0c7f40 con 0x7ff72c004d00 2024-01-24T17:45:47.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.824+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74c1591c0 con 0x7ff72c004d00 2024-01-24T17:45:47.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.826+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74c073ef0 con 0x7ff72c004d00 2024-01-24T17:45:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.826+0000 7ff748c00640 1 -- 172.21.15.39:0/1411922721 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff73c015460 con 0x7ff72c004d00 2024-01-24T17:45:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.827+0000 7ff748c00640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72406a460 0x7ff72406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.827+0000 7ff748c00640 1 -- 172.21.15.39:0/1411922721 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff73c088dd0 con 0x7ff72c004d00 2024-01-24T17:45:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.828+0000 7ff743200640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72406a460 0x7ff72406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:47.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.829+0000 7ff743200640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72406a460 0x7ff72406c920 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff734001eb0 tx=0x7ff734001dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:47.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:47.832+0000 7ff748c00640 1 -- 172.21.15.39:0/1411922721 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff73c0543c0 con 0x7ff72c004d00 2024-01-24T17:45:48.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.030+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff74c0b2140 con 0x7ff72c004d00 2024-01-24T17:45:48.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.033+0000 7ff748c00640 1 -- 172.21.15.39:0/1411922721 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7ff73c0579e0 con 0x7ff72c004d00 2024-01-24T17:45:48.033 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:48.034 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:48.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72406a460 msgr2=0x7ff72406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72406a460 0x7ff72406c920 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff734001eb0 tx=0x7ff734001dc0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 msgr2=0x7ff74c156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff74c156cb0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff73c011600 tx=0x7ff73c011ac0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 shutdown_connections 2024-01-24T17:45:48.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff74c0c7580 0x7ff74c158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c0d29a0 0x7ff74c0c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72406a460 0x7ff72406c920 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 --2- 172.21.15.39:0/1411922721 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff72c004d00 0x7ff74c156cb0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.035+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 >> 172.21.15.39:0/1411922721 conn(0x7ff74c071790 msgr2=0x7ff74c0b81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:48.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.036+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 shutdown_connections 2024-01-24T17:45:48.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.036+0000 7ff753000640 1 -- 172.21.15.39:0/1411922721 wait complete. 2024-01-24T17:45:48.057 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:45:16.503215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:45:17.386565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:45:48.057 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:45:48.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.284+0000 7f0e07400640 1 Processor -- start 2024-01-24T17:45:48.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.284+0000 7f0e07400640 1 -- start start 2024-01-24T17:45:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.285+0000 7f0e07400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 0x7f0e000cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.285+0000 7f0e07400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0e000ceb20 0x7f0e000cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.285+0000 7f0e07400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e000cb5d0 con 0x7f0e00160a00 2024-01-24T17:45:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.285+0000 7f0e07400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e001619d0 con 0x7f0e000ceb20 2024-01-24T17:45:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.285+0000 7f0e07400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0006b5f0 con 0x7f0e0015ce70 2024-01-24T17:45:48.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e05600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0e000ceb20 0x7f0e000cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e05600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0e000ceb20 0x7f0e000cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38982/0 (socket says 172.21.15.39:38982) 2024-01-24T17:45:48.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e05600640 1 -- 172.21.15.39:0/4207671619 learned_addr learned my addr 172.21.15.39:0/4207671619 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:48.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e06a00640 1 --1- 172.21.15.39:0/4207671619 >> v1:172.21.15.39:6789/0 conn(0x7f0e00160a00 0x7f0e00160e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59130/0 (socket says 172.21.15.39:59130) 2024-01-24T17:45:48.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e06000640 1 --2- 172.21.15.39:0/4207671619 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 0x7f0e000cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e05600640 1 -- 172.21.15.39:0/4207671619 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 msgr2=0x7f0e000cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:48.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e05600640 1 --2- 172.21.15.39:0/4207671619 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 0x7f0e000cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e05600640 1 -- 172.21.15.39:0/4207671619 >> v1:172.21.15.39:6789/0 conn(0x7f0e00160a00 legacy=0x7f0e00160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e04c00640 1 -- 172.21.15.39:0/4207671619 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886213942 0 0) 0x7f0e000cb5d0 con 0x7f0e00160a00 2024-01-24T17:45:48.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.286+0000 7f0e05600640 1 -- 172.21.15.39:0/4207671619 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e000cf4c0 con 0x7f0e000ceb20 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.287+0000 7f0e05600640 1 --2- 172.21.15.39:0/4207671619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000cef80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009870 tx=0x7f0de802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368e709e28ce6494 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e04c00640 1 -- 172.21.15.39:0/4207671619 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8030840 con 0x7f0e000ceb20 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e04c00640 1 -- 172.21.15.39:0/4207671619 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de80309a0 con 0x7f0e000ceb20 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e04c00640 1 -- 172.21.15.39:0/4207671619 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8030d30 con 0x7f0e000ceb20 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e07400640 1 -- 172.21.15.39:0/4207671619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 msgr2=0x7f0e000cef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e07400640 1 --2- 172.21.15.39:0/4207671619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000cef80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009870 tx=0x7f0de802b980 comp rx=0 tx=0).stop 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e07400640 1 -- 172.21.15.39:0/4207671619 shutdown_connections 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e07400640 1 --2- 172.21.15.39:0/4207671619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000cef80 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e07400640 1 --2- 172.21.15.39:0/4207671619 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 0x7f0e000cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.288+0000 7f0e07400640 1 -- 172.21.15.39:0/4207671619 >> 172.21.15.39:0/4207671619 conn(0x7f0e00069c70 msgr2=0x7f0e0006a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:48.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.289+0000 7f0e07400640 1 -- 172.21.15.39:0/4207671619 shutdown_connections 2024-01-24T17:45:48.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.289+0000 7f0e07400640 1 -- 172.21.15.39:0/4207671619 wait complete. 2024-01-24T17:45:48.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.290+0000 7f0e07400640 1 Processor -- start 2024-01-24T17:45:48.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.290+0000 7f0e07400640 1 -- start start 2024-01-24T17:45:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.291+0000 7f0e07400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000c4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.291+0000 7f0e07400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 0x7f0e00157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.291+0000 7f0e07400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e00160a00 0x7f0e001581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.291+0000 7f0e07400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e001f27e0 con 0x7f0e000ceb20 2024-01-24T17:45:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.291+0000 7f0e07400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e001f2aa0 con 0x7f0e00160a00 2024-01-24T17:45:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.291+0000 7f0e07400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e001f2d80 con 0x7f0e0015ce70 2024-01-24T17:45:48.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000c4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38988/0 (socket says 172.21.15.39:38988) 2024-01-24T17:45:48.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06000640 1 -- 172.21.15.39:0/2311024922 learned_addr learned my addr 172.21.15.39:0/2311024922 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:48.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e05600640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 0x7f0e00157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06000640 1 -- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 msgr2=0x7f0e00157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06000640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 0x7f0e00157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06a00640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e00160a00 0x7f0e001581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06000640 1 -- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e00160a00 msgr2=0x7f0e001581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06000640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e00160a00 0x7f0e001581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06000640 1 -- 172.21.15.39:0/2311024922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de80094d0 con 0x7f0e000ceb20 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.292+0000 7f0e06a00640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e00160a00 0x7f0e001581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.293+0000 7f0e06000640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000c4190 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0df000d7d0 tx=0x7f0df000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.293+0000 7f0dfe000640 1 -- 172.21.15.39:0/2311024922 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df00096c0 con 0x7f0e000ceb20 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.293+0000 7f0dfe000640 1 -- 172.21.15.39:0/2311024922 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df0009820 con 0x7f0e000ceb20 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.293+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e001f3060 con 0x7f0e000ceb20 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.293+0000 7f0dfe000640 1 -- 172.21.15.39:0/2311024922 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0009bd0 con 0x7f0e000ceb20 2024-01-24T17:45:48.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.293+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e001f3620 con 0x7f0e000ceb20 2024-01-24T17:45:48.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.294+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0004dff0 con 0x7f0e000ceb20 2024-01-24T17:45:48.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.295+0000 7f0dfe000640 1 -- 172.21.15.39:0/2311024922 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0df0009d30 con 0x7f0e000ceb20 2024-01-24T17:45:48.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.296+0000 7f0dfe000640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0dd806a430 0x7f0dd806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.296+0000 7f0dfe000640 1 -- 172.21.15.39:0/2311024922 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0df0089140 con 0x7f0e000ceb20 2024-01-24T17:45:48.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.297+0000 7f0e05600640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0dd806a430 0x7f0dd806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.298+0000 7f0e05600640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0dd806a430 0x7f0dd806c8f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0de8036000 tx=0x7f0de803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.301+0000 7f0dfe000640 1 -- 172.21.15.39:0/2311024922 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0df0054720 con 0x7f0e000ceb20 2024-01-24T17:45:48.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.496+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e00160720 con 0x7f0e000ceb20 2024-01-24T17:45:48.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.496+0000 7f0dfe000640 1 -- 172.21.15.39:0/2311024922 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0df0057d40 con 0x7f0e000ceb20 2024-01-24T17:45:48.497 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:45:48.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0dd806a430 msgr2=0x7f0dd806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0dd806a430 0x7f0dd806c8f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0de8036000 tx=0x7f0de803b040 comp rx=0 tx=0).stop 2024-01-24T17:45:48.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 msgr2=0x7f0e000c4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000c4190 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0df000d7d0 tx=0x7f0df000dc90 comp rx=0 tx=0).stop 2024-01-24T17:45:48.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 shutdown_connections 2024-01-24T17:45:48.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e00160a00 0x7f0e001581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0e0015ce70 0x7f0e00157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0dd806a430 0x7f0dd806c8f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 --2- 172.21.15.39:0/2311024922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e000ceb20 0x7f0e000c4190 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 >> 172.21.15.39:0/2311024922 conn(0x7f0e00069c70 msgr2=0x7f0e000cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:48.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.499+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 shutdown_connections 2024-01-24T17:45:48.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.500+0000 7f0e07400640 1 -- 172.21.15.39:0/2311024922 wait complete. 2024-01-24T17:45:48.520 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:45:48.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.744+0000 7fe9b5e00640 1 Processor -- start 2024-01-24T17:45:48.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.745+0000 7fe9b5e00640 1 -- start start 2024-01-24T17:45:48.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.746+0000 7fe9b5e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b00d14c0 0x7fe9b00b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.746+0000 7fe9b5e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe9b00b2a10 0x7fe9b00cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.746+0000 7fe9b5e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b00cda90 con 0x7fe9b015fff0 2024-01-24T17:45:48.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.746+0000 7fe9b5e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b00b2ec0 con 0x7fe9b00b2a10 2024-01-24T17:45:48.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.746+0000 7fe9b5e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00d32c0 con 0x7fe9b00d14c0 2024-01-24T17:45:48.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.746+0000 7fe9aea00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe9b015fff0 0x7fe9b00d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59146/0 (socket says 172.21.15.39:59146) 2024-01-24T17:45:48.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.746+0000 7fe9aea00640 1 -- 172.21.15.39:0/3870764249 learned_addr learned my addr 172.21.15.39:0/3870764249 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:48.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9a5e00640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b00d14c0 0x7fe9b00b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9af400640 1 --2- 172.21.15.39:0/3870764249 >> v2:172.21.15.100:3300/0 conn(0x7fe9b00b2a10 0x7fe9b00cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479721960 0 0) 0x7fe9b00cda90 con 0x7fe9b015fff0 2024-01-24T17:45:48.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe990003520 con 0x7fe9b015fff0 2024-01-24T17:45:48.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631444759 0 0) 0x7fe990003520 con 0x7fe9b015fff0 2024-01-24T17:45:48.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b00d14c0 msgr2=0x7fe9b00b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:48.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b00d14c0 0x7fe9b00b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 >> v2:172.21.15.100:3300/0 conn(0x7fe9b00b2a10 msgr2=0x7fe9b00cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 --2- 172.21.15.39:0/3870764249 >> v2:172.21.15.100:3300/0 conn(0x7fe9b00b2a10 0x7fe9b00cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b00b30a0 con 0x7fe9b015fff0 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fe9a0001670 con 0x7fe9b015fff0 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 >> v1:172.21.15.39:6789/0 conn(0x7fe9b015fff0 legacy=0x7fe9b00d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe990003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9900047c0 0x7fe990004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe990005180 0x7fe99000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00b2ec0 con 0x7fe990003970 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.747+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00d32c0 con 0x7fe9900047c0 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.748+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99000bef0 con 0x7fe990005180 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.748+0000 7fe9a5e00640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b00d14c0 0x7fe9b00b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.748+0000 7fe9aea00640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe990003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.748+0000 7fe9aea00640 1 -- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe990005180 msgr2=0x7fe99000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:48.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.748+0000 7fe9aea00640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe990005180 0x7fe99000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.748+0000 7fe9aea00640 1 -- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9900047c0 msgr2=0x7fe990004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:48.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.748+0000 7fe9aea00640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9900047c0 0x7fe990004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.748+0000 7fe9aea00640 1 -- 172.21.15.39:0/3870764249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe990003520 con 0x7fe990003970 2024-01-24T17:45:48.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.749+0000 7fe9aea00640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe990003d90 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c210 tx=0x7fe9900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654e851876cfb2be server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:48.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.750+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0005ad0 con 0x7fe990003970 2024-01-24T17:45:48.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.750+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a0005c30 con 0x7fe990003970 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.750+0000 7fe9ae000640 1 -- 172.21.15.39:0/3870764249 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a002c8d0 con 0x7fe990003970 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.750+0000 7fe9b5e00640 1 -- 172.21.15.39:0/3870764249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 msgr2=0x7fe990003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.750+0000 7fe9b5e00640 1 --2- 172.21.15.39:0/3870764249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe990003d90 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c210 tx=0x7fe9900057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.750+0000 7fe9af400640 1 -- 172.21.15.39:0/3870764249 reap_dead start 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.750+0000 7fe9b5e00640 1 -- 172.21.15.39:0/3870764249 shutdown_connections 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.750+0000 7fe9b5e00640 1 -- 172.21.15.39:0/3870764249 >> 172.21.15.39:0/3870764249 conn(0x7fe9b009bcc0 msgr2=0x7fe9b00d2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.751+0000 7fe9b5e00640 1 -- 172.21.15.39:0/3870764249 shutdown_connections 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.751+0000 7fe9b5e00640 1 -- 172.21.15.39:0/3870764249 wait complete. 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.752+0000 7fe9b5e00640 1 Processor -- start 2024-01-24T17:45:48.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.752+0000 7fe9b5e00640 1 -- start start 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.753+0000 7fe9b5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe9b00cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.753+0000 7fe9b5e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b015d820 0x7fe9b0144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.753+0000 7fe9b5e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9b0145100 0x7fe9b0142b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.753+0000 7fe9b5e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b0145580 con 0x7fe990003970 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.753+0000 7fe9b5e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00ccd40 con 0x7fe9b0145100 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.753+0000 7fe9b5e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b015d390 con 0x7fe9b015d820 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe9b00cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe9b00cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39000/0 (socket says 172.21.15.39:39000) 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 -- 172.21.15.39:0/2518098503 learned_addr learned my addr 172.21.15.39:0/2518098503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:48.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9af400640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9b0145100 0x7fe9b0142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9a5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b015d820 0x7fe9b0144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 -- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b015d820 msgr2=0x7fe9b0144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b015d820 0x7fe9b0144bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 -- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9b0145100 msgr2=0x7fe9b0142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9b0145100 0x7fe9b0142b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 -- 172.21.15.39:0/2518098503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b0156250 con 0x7fe990003970 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9af400640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9b0145100 0x7fe9b0142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.754+0000 7fe9aea00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe9b00cc800 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c210 tx=0x7fe9b00b3fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.755+0000 7fe9a7e00640 1 -- 172.21.15.39:0/2518098503 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0039070 con 0x7fe990003970 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.755+0000 7fe9a7e00640 1 -- 172.21.15.39:0/2518098503 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a002cce0 con 0x7fe990003970 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.755+0000 7fe9a7e00640 1 -- 172.21.15.39:0/2518098503 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a003c040 con 0x7fe990003970 2024-01-24T17:45:48.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.755+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b015d040 con 0x7fe990003970 2024-01-24T17:45:48.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.755+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b0155c90 con 0x7fe990003970 2024-01-24T17:45:48.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.756+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b0073ef0 con 0x7fe990003970 2024-01-24T17:45:48.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.757+0000 7fe9a7e00640 1 -- 172.21.15.39:0/2518098503 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe9a0032070 con 0x7fe990003970 2024-01-24T17:45:48.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.758+0000 7fe9a7e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe98806a460 0x7fe98806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:48.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.758+0000 7fe9a7e00640 1 -- 172.21.15.39:0/2518098503 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9a00a9e00 con 0x7fe990003970 2024-01-24T17:45:48.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.758+0000 7fe9a5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe98806a460 0x7fe98806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:48.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.760+0000 7fe9a5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe98806a460 0x7fe98806c920 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe998006a90 tx=0x7fe998008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:48.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.763+0000 7fe9a7e00640 1 -- 172.21.15.39:0/2518098503 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe9a0078fa0 con 0x7fe990003970 2024-01-24T17:45:48.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.957+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe9b006bd90 con 0x7fe990003970 2024-01-24T17:45:48.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.958+0000 7fe9a7e00640 1 -- 172.21.15.39:0/2518098503 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v63) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe9a0078b30 con 0x7fe990003970 2024-01-24T17:45:48.958 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe98806a460 msgr2=0x7fe98806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe98806a460 0x7fe98806c920 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe998006a90 tx=0x7fe998008040 comp rx=0 tx=0).stop 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 msgr2=0x7fe9b00cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe9b00cc800 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000c210 tx=0x7fe9b00b3fb0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 shutdown_connections 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe9b0145100 0x7fe9b0142b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe9b015d820 0x7fe9b0144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe98806a460 0x7fe98806c920 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 --2- 172.21.15.39:0/2518098503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe990003970 0x7fe9b00cc800 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:48.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 >> 172.21.15.39:0/2518098503 conn(0x7fe9b009bcc0 msgr2=0x7fe9b00c8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:48.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 shutdown_connections 2024-01-24T17:45:48.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:48.961+0000 7fe9b5e00640 1 -- 172.21.15.39:0/2518098503 wait complete. 2024-01-24T17:45:48.981 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:45:49.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.207+0000 7fc6b7200640 1 Processor -- start 2024-01-24T17:45:49.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.207+0000 7fc6b7200640 1 -- start start 2024-01-24T17:45:49.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6b815ccb0 0x7fc6b80cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b80c9d60 0x7fc6b80ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b7200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b815d090 con 0x7fc6b80ce390 2024-01-24T17:45:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b80cac20 con 0x7fc6b815ccb0 2024-01-24T17:45:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80ced60 con 0x7fc6b80c9d60 2024-01-24T17:45:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6b815ccb0 0x7fc6b80cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6b815ccb0 0x7fc6b80cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39004/0 (socket says 172.21.15.39:39004) 2024-01-24T17:45:49.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b5e00640 1 -- 172.21.15.39:0/2629307415 learned_addr learned my addr 172.21.15.39:0/2629307415 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:49.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.209+0000 7fc6b6800640 1 --2- 172.21.15.39:0/2629307415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b80c9d60 0x7fc6b80ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.210+0000 7fc6b5e00640 1 -- 172.21.15.39:0/2629307415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b80c9d60 msgr2=0x7fc6b80ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:49.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.210+0000 7fc6b5e00640 1 --2- 172.21.15.39:0/2629307415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b80c9d60 0x7fc6b80ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.210+0000 7fc6b5e00640 1 -- 172.21.15.39:0/2629307415 >> v1:172.21.15.100:6789/0 conn(0x7fc6b80ce390 legacy=0x7fc6b80cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:49.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.210+0000 7fc6b5e00640 1 -- 172.21.15.39:0/2629307415 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b806c540 con 0x7fc6b815ccb0 2024-01-24T17:45:49.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.211+0000 7fc6b5e00640 1 --2- 172.21.15.39:0/2629307415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b815ccb0 0x7fc6b80cda90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009850 tx=0x7fc6a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d0d99d4dbcf5a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:49.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.211+0000 7fc6b5400640 1 -- 172.21.15.39:0/2629307415 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8004230 con 0x7fc6b815ccb0 2024-01-24T17:45:49.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.211+0000 7fc6b5400640 1 -- 172.21.15.39:0/2629307415 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a8004390 con 0x7fc6b815ccb0 2024-01-24T17:45:49.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.211+0000 7fc6b5400640 1 -- 172.21.15.39:0/2629307415 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8031830 con 0x7fc6b815ccb0 2024-01-24T17:45:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.212+0000 7fc6b7200640 1 -- 172.21.15.39:0/2629307415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b815ccb0 msgr2=0x7fc6b80cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.212+0000 7fc6b7200640 1 --2- 172.21.15.39:0/2629307415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b815ccb0 0x7fc6b80cda90 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009850 tx=0x7fc6a802b8d0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.212+0000 7fc6b7200640 1 -- 172.21.15.39:0/2629307415 shutdown_connections 2024-01-24T17:45:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.212+0000 7fc6b7200640 1 --2- 172.21.15.39:0/2629307415 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b80c9d60 0x7fc6b80ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.212+0000 7fc6b7200640 1 --2- 172.21.15.39:0/2629307415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b815ccb0 0x7fc6b80cda90 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.212+0000 7fc6b7200640 1 -- 172.21.15.39:0/2629307415 >> 172.21.15.39:0/2629307415 conn(0x7fc6b8069e50 msgr2=0x7fc6b806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.212+0000 7fc6b7200640 1 -- 172.21.15.39:0/2629307415 shutdown_connections 2024-01-24T17:45:49.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.212+0000 7fc6b7200640 1 -- 172.21.15.39:0/2629307415 wait complete. 2024-01-24T17:45:49.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.214+0000 7fc6b7200640 1 Processor -- start 2024-01-24T17:45:49.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.214+0000 7fc6b7200640 1 -- start start 2024-01-24T17:45:49.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b7200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6b80c9d60 0x7fc6b80ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b80ce390 0x7fc6b815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b815ccb0 0x7fc6b80c2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80acb40 con 0x7fc6b80ce390 2024-01-24T17:45:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b7200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80ab840 con 0x7fc6b80c9d60 2024-01-24T17:45:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b80abb20 con 0x7fc6b815ccb0 2024-01-24T17:45:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b80ce390 0x7fc6b815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b80ce390 0x7fc6b815a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39016/0 (socket says 172.21.15.39:39016) 2024-01-24T17:45:49.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6afe00640 1 -- 172.21.15.39:0/4138701375 learned_addr learned my addr 172.21.15.39:0/4138701375 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b5e00640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6b80c9d60 0x7fc6b80ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b6800640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b815ccb0 0x7fc6b80c2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6afe00640 1 -- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b815ccb0 msgr2=0x7fc6b80c2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6afe00640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b815ccb0 0x7fc6b80c2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6afe00640 1 -- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6b80c9d60 msgr2=0x7fc6b80ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6afe00640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6b80c9d60 0x7fc6b80ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6afe00640 1 -- 172.21.15.39:0/4138701375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8009460 con 0x7fc6b80ce390 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.215+0000 7fc6b5e00640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6b80c9d60 0x7fc6b80ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.216+0000 7fc6b6800640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b815ccb0 0x7fc6b80c2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.216+0000 7fc6afe00640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b80ce390 0x7fc6b815a720 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c430 tx=0x7fc6a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.216+0000 7fc6ae000640 1 -- 172.21.15.39:0/4138701375 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0011790 con 0x7fc6b80ce390 2024-01-24T17:45:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.216+0000 7fc6ae000640 1 -- 172.21.15.39:0/4138701375 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a00118f0 con 0x7fc6b80ce390 2024-01-24T17:45:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.216+0000 7fc6ae000640 1 -- 172.21.15.39:0/4138701375 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0011ca0 con 0x7fc6b80ce390 2024-01-24T17:45:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.216+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b80abe00 con 0x7fc6b80ce390 2024-01-24T17:45:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.216+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b80ab520 con 0x7fc6b80ce390 2024-01-24T17:45:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.217+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b80ac320 con 0x7fc6b80ce390 2024-01-24T17:45:49.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.218+0000 7fc6ae000640 1 -- 172.21.15.39:0/4138701375 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc6a001c460 con 0x7fc6b80ce390 2024-01-24T17:45:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.220+0000 7fc6ae000640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69006a430 0x7fc69006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.220+0000 7fc6ae000640 1 -- 172.21.15.39:0/4138701375 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6a0088920 con 0x7fc6b80ce390 2024-01-24T17:45:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.220+0000 7fc6b5e00640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69006a430 0x7fc69006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.221+0000 7fc6b5e00640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69006a430 0x7fc69006c8f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8035000 tx=0x7fc6a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:49.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.224+0000 7fc6ae000640 1 -- 172.21.15.39:0/4138701375 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc6b80ac320 con 0x7fc6b80ce390 2024-01-24T17:45:49.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.418+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6b80ac320 con 0x7fc6b80ce390 2024-01-24T17:45:49.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.419+0000 7fc6ae000640 1 -- 172.21.15.39:0/4138701375 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v63) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc6b80ac320 con 0x7fc6b80ce390 2024-01-24T17:45:49.420 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:45:49.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69006a430 msgr2=0x7fc69006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69006a430 0x7fc69006c8f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8035000 tx=0x7fc6a8008030 comp rx=0 tx=0).stop 2024-01-24T17:45:49.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b80ce390 msgr2=0x7fc6b815a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b80ce390 0x7fc6b815a720 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c430 tx=0x7fc6a000c8f0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 shutdown_connections 2024-01-24T17:45:49.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6b815ccb0 0x7fc6b80c2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69006a430 0x7fc69006c8f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b80ce390 0x7fc6b815a720 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 --2- 172.21.15.39:0/4138701375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6b80c9d60 0x7fc6b80ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 >> 172.21.15.39:0/4138701375 conn(0x7fc6b8069e50 msgr2=0x7fc6b80ccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:49.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 shutdown_connections 2024-01-24T17:45:49.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.422+0000 7fc6b7200640 1 -- 172.21.15.39:0/4138701375 wait complete. 2024-01-24T17:45:49.443 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:45:49.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.665+0000 7f3761c00640 1 Processor -- start 2024-01-24T17:45:49.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.665+0000 7f3761c00640 1 -- start start 2024-01-24T17:45:49.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.666+0000 7f3761c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c06c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.666+0000 7f3761c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f375c0cd830 0x7f375c0cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.667+0000 7f3761c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375c15e010 con 0x7f375c0cbf70 2024-01-24T17:45:49.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.667+0000 7f3761c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375c06c6d0 con 0x7f375c0cd830 2024-01-24T17:45:49.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.667+0000 7f3761c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c159d50 con 0x7f375c06bdb0 2024-01-24T17:45:49.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.667+0000 7f375be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c06c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.668+0000 7f3761200640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f375c0cbf70 0x7f375c0cc400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:36792/0 (socket says 172.21.15.39:36792) 2024-01-24T17:45:49.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.668+0000 7f3761200640 1 -- 172.21.15.39:0/421190926 learned_addr learned my addr 172.21.15.39:0/421190926 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:49.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.668+0000 7f375b400640 1 --2- 172.21.15.39:0/421190926 >> v2:172.21.15.100:3300/0 conn(0x7f375c0cd830 0x7f375c0cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.668+0000 7f375b400640 1 -- 172.21.15.39:0/421190926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 msgr2=0x7f375c06c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:49.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.668+0000 7f375b400640 1 --2- 172.21.15.39:0/421190926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c06c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.668+0000 7f375b400640 1 -- 172.21.15.39:0/421190926 >> v1:172.21.15.100:6789/0 conn(0x7f375c0cbf70 legacy=0x7f375c0cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.668+0000 7f375b400640 1 -- 172.21.15.39:0/421190926 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375c15a030 con 0x7f375c0cd830 2024-01-24T17:45:49.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.668+0000 7f375be00640 1 --2- 172.21.15.39:0/421190926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c06c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:45:49.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.670+0000 7f375b400640 1 --2- 172.21.15.39:0/421190926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 0x7f375c0cdc90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f374400bd90 tx=0x7f374402dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c0af87d7855d442 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:49.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.670+0000 7f375aa00640 1 -- 172.21.15.39:0/421190926 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374402f8a0 con 0x7f375c0cd830 2024-01-24T17:45:49.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.670+0000 7f375aa00640 1 -- 172.21.15.39:0/421190926 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374402fa00 con 0x7f375c0cd830 2024-01-24T17:45:49.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.670+0000 7f375aa00640 1 -- 172.21.15.39:0/421190926 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374402fd90 con 0x7f375c0cd830 2024-01-24T17:45:49.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.671+0000 7f3761c00640 1 -- 172.21.15.39:0/421190926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 msgr2=0x7f375c0cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.671+0000 7f3761c00640 1 --2- 172.21.15.39:0/421190926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 0x7f375c0cdc90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f374400bd90 tx=0x7f374402dae0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.671+0000 7f3761c00640 1 -- 172.21.15.39:0/421190926 shutdown_connections 2024-01-24T17:45:49.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.671+0000 7f3761c00640 1 --2- 172.21.15.39:0/421190926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 0x7f375c0cdc90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.671+0000 7f3761c00640 1 --2- 172.21.15.39:0/421190926 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c06c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.671+0000 7f3761c00640 1 -- 172.21.15.39:0/421190926 >> 172.21.15.39:0/421190926 conn(0x7f375c06a2a0 msgr2=0x7f375c0afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:49.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.671+0000 7f3761c00640 1 -- 172.21.15.39:0/421190926 shutdown_connections 2024-01-24T17:45:49.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.671+0000 7f3761c00640 1 -- 172.21.15.39:0/421190926 wait complete. 2024-01-24T17:45:49.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.673+0000 7f3761c00640 1 Processor -- start 2024-01-24T17:45:49.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.674+0000 7f3761c00640 1 -- start start 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f3761c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c14b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f3761c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c0cbf70 0x7f375c0aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f3761c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 0x7f375c0af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f3761c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0b1160 con 0x7f375c0cbf70 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f3761c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0b1400 con 0x7f375c0cd830 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f3761c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0b16e0 con 0x7f375c06bdb0 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f375b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c0cbf70 0x7f375c0aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f375b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c0cbf70 0x7f375c0aef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39018/0 (socket says 172.21.15.39:39018) 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.675+0000 7f375b400640 1 -- 172.21.15.39:0/3891718122 learned_addr learned my addr 172.21.15.39:0/3891718122 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f3761200640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 0x7f375c0af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f375be00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c14b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f375b400640 1 -- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 msgr2=0x7f375c14b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f375b400640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c14b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f375b400640 1 -- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 msgr2=0x7f375c0af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f375b400640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 0x7f375c0af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f375b400640 1 -- 172.21.15.39:0/3891718122 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c009400 con 0x7f375c0cbf70 2024-01-24T17:45:49.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f375b400640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c0cbf70 0x7f375c0aef50 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f374402e0f0 tx=0x7f37440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:49.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f3753e00640 1 -- 172.21.15.39:0/3891718122 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374403d040 con 0x7f375c0cbf70 2024-01-24T17:45:49.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.676+0000 7f3753e00640 1 -- 172.21.15.39:0/3891718122 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37440082c0 con 0x7f375c0cbf70 2024-01-24T17:45:49.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.677+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374400ba70 con 0x7f375c0cbf70 2024-01-24T17:45:49.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.677+0000 7f3753e00640 1 -- 172.21.15.39:0/3891718122 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3744008670 con 0x7f375c0cbf70 2024-01-24T17:45:49.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.677+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c0b1ca0 con 0x7f375c0cbf70 2024-01-24T17:45:49.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.678+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c0b0bd0 con 0x7f375c0cbf70 2024-01-24T17:45:49.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.679+0000 7f3753e00640 1 -- 172.21.15.39:0/3891718122 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f37440087d0 con 0x7f375c0cbf70 2024-01-24T17:45:49.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.680+0000 7f3753e00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f373406a430 0x7f373406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:49.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.680+0000 7f3753e00640 1 -- 172.21.15.39:0/3891718122 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37440aa8a0 con 0x7f375c0cbf70 2024-01-24T17:45:49.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.680+0000 7f375be00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f373406a430 0x7f373406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:49.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.681+0000 7f375be00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f373406a430 0x7f373406c8f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f374c009b80 tx=0x7f374c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:49.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.686+0000 7f3753e00640 1 -- 172.21.15.39:0/3891718122 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3744075df0 con 0x7f375c0cbf70 2024-01-24T17:45:49.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.893+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f375c0b4320 con 0x7f375c0cbf70 2024-01-24T17:45:49.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.893+0000 7f3753e00640 1 -- 172.21.15.39:0/3891718122 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v63) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3744048080 con 0x7f375c0cbf70 2024-01-24T17:45:49.894 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f373406a430 msgr2=0x7f373406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f373406a430 0x7f373406c8f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f374c009b80 tx=0x7f374c008040 comp rx=0 tx=0).stop 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c0cbf70 msgr2=0x7f375c0aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c0cbf70 0x7f375c0aef50 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f374402e0f0 tx=0x7f37440046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 shutdown_connections 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f375c0cd830 0x7f375c0af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f373406a430 0x7f373406c8f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c0cbf70 0x7f375c0aef50 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 --2- 172.21.15.39:0/3891718122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f375c06bdb0 0x7f375c14b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:49.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 >> 172.21.15.39:0/3891718122 conn(0x7f375c06a2a0 msgr2=0x7f375c0cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:49.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.896+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 shutdown_connections 2024-01-24T17:45:49.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:49.897+0000 7f3761c00640 1 -- 172.21.15.39:0/3891718122 wait complete. 2024-01-24T17:45:49.917 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:45:49.918 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:45:49.975 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:45:50.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.200+0000 7faead000640 1 Processor -- start 2024-01-24T17:45:50.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.200+0000 7faead000640 1 -- start start 2024-01-24T17:45:50.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.202+0000 7faead000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faea815cbf0 0x7faea80d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.202+0000 7faead000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7faea80d0da0 0x7faea80d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.202+0000 7faead000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea815dfe0 con 0x7faea815db60 2024-01-24T17:45:50.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.202+0000 7faead000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea80d1770 con 0x7faea80d0da0 2024-01-24T17:45:50.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.202+0000 7faead000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea80d0a90 con 0x7faea815cbf0 2024-01-24T17:45:50.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.202+0000 7fae9fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7faea815db60 0x7faea80ce240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59162/0 (socket says 172.21.15.39:59162) 2024-01-24T17:45:50.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.202+0000 7fae9fe00640 1 -- 172.21.15.39:0/2583126465 learned_addr learned my addr 172.21.15.39:0/2583126465 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:50.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea7e00640 1 --2- 172.21.15.39:0/2583126465 >> v2:172.21.15.100:3300/0 conn(0x7faea80d0da0 0x7faea80d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea7400640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faea815cbf0 0x7faea80d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057946046 0 0) 0x7faea815dfe0 con 0x7faea815db60 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae88003520 con 0x7faea815db60 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1065804695 0 0) 0x7fae88003520 con 0x7faea815db60 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faea815cbf0 msgr2=0x7faea80d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faea815cbf0 0x7faea80d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 >> v2:172.21.15.100:3300/0 conn(0x7faea80d0da0 msgr2=0x7faea80d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 --2- 172.21.15.39:0/2583126465 >> v2:172.21.15.100:3300/0 conn(0x7faea80d0da0 0x7faea80d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea80cd760 con 0x7faea815db60 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fae90001670 con 0x7faea815db60 2024-01-24T17:45:50.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 >> v1:172.21.15.39:6789/0 conn(0x7faea815db60 legacy=0x7faea80ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae88003970 0x7fae88003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faea815cbf0 0x7fae880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae88004d00 0x7fae88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea80d1770 con 0x7fae88003970 2024-01-24T17:45:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea80d0a90 con 0x7fae88004d00 2024-01-24T17:45:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.203+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8800b980 con 0x7faea815cbf0 2024-01-24T17:45:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.204+0000 7faea7400640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae88003970 0x7fae88003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.204+0000 7faea7400640 1 -- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faea815cbf0 msgr2=0x7fae880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:50.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.204+0000 7faea7400640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faea815cbf0 0x7fae880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.204+0000 7faea7400640 1 -- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae88004d00 msgr2=0x7fae88005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:50.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.204+0000 7faea7400640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae88004d00 0x7fae88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.204+0000 7faea7400640 1 -- 172.21.15.39:0/2583126465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae88003520 con 0x7fae88003970 2024-01-24T17:45:50.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.205+0000 7faea7400640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae88003970 0x7fae88003d90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fae98009ab0 tx=0x7fae880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a98891b1e42cc83 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.206+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98005990 con 0x7fae88003970 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.206+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae98005af0 con 0x7fae88003970 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.206+0000 7faea6a00640 1 -- 172.21.15.39:0/2583126465 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98005ea0 con 0x7fae88003970 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.206+0000 7faead000640 1 -- 172.21.15.39:0/2583126465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae88003970 msgr2=0x7fae88003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.206+0000 7faead000640 1 --2- 172.21.15.39:0/2583126465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae88003970 0x7fae88003d90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fae98009ab0 tx=0x7fae880057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.206+0000 7faea7e00640 1 -- 172.21.15.39:0/2583126465 reap_dead start 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.207+0000 7faead000640 1 -- 172.21.15.39:0/2583126465 shutdown_connections 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.207+0000 7faead000640 1 -- 172.21.15.39:0/2583126465 >> 172.21.15.39:0/2583126465 conn(0x7faea809bb90 msgr2=0x7faea80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.207+0000 7faead000640 1 -- 172.21.15.39:0/2583126465 shutdown_connections 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.207+0000 7faead000640 1 -- 172.21.15.39:0/2583126465 wait complete. 2024-01-24T17:45:50.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.208+0000 7faead000640 1 Processor -- start 2024-01-24T17:45:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.209+0000 7faead000640 1 -- start start 2024-01-24T17:45:50.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.209+0000 7faead000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae88003970 0x7faea814fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7faead000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea8150ea0 0x7faea81502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7faead000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea81508e0 0x7faea8156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7faead000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea8157210 con 0x7faea8150ea0 2024-01-24T17:45:50.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7faead000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea81574f0 con 0x7faea81508e0 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7faead000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea80c60e0 con 0x7fae88003970 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7fae9fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea8150ea0 0x7faea81502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7fae9fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea8150ea0 0x7faea81502d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39032/0 (socket says 172.21.15.39:39032) 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7fae9fe00640 1 -- 172.21.15.39:0/3249665636 learned_addr learned my addr 172.21.15.39:0/3249665636 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7faea7e00640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea81508e0 0x7faea8156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7faea7400640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae88003970 0x7faea814fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7fae9fe00640 1 -- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae88003970 msgr2=0x7faea814fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7fae9fe00640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae88003970 0x7faea814fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7fae9fe00640 1 -- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea81508e0 msgr2=0x7faea8156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7fae9fe00640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea81508e0 0x7faea8156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.210+0000 7fae9fe00640 1 -- 172.21.15.39:0/3249665636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea80c63c0 con 0x7faea8150ea0 2024-01-24T17:45:50.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.211+0000 7fae9fe00640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea8150ea0 0x7faea81502d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c7a0 tx=0x7fae9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:50.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.211+0000 7fae9f400640 1 -- 172.21.15.39:0/3249665636 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae900074e0 con 0x7faea8150ea0 2024-01-24T17:45:50.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.211+0000 7fae9f400640 1 -- 172.21.15.39:0/3249665636 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae90007660 con 0x7faea8150ea0 2024-01-24T17:45:50.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.211+0000 7fae9f400640 1 -- 172.21.15.39:0/3249665636 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90004a20 con 0x7faea8150ea0 2024-01-24T17:45:50.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.211+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea80c66a0 con 0x7faea8150ea0 2024-01-24T17:45:50.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.211+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea8156c90 con 0x7faea8150ea0 2024-01-24T17:45:50.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.213+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea804dff0 con 0x7faea8150ea0 2024-01-24T17:45:50.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.213+0000 7fae9f400640 1 -- 172.21.15.39:0/3249665636 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fae90004b80 con 0x7faea8150ea0 2024-01-24T17:45:50.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.215+0000 7fae9f400640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8006a460 0x7fae8006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.215+0000 7fae9f400640 1 -- 172.21.15.39:0/3249665636 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae90087df0 con 0x7faea8150ea0 2024-01-24T17:45:50.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.216+0000 7faea7400640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8006a460 0x7fae8006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.217+0000 7faea7400640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8006a460 0x7fae8006c920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fae98035000 tx=0x7fae98030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:50.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.220+0000 7fae9f400640 1 -- 172.21.15.39:0/3249665636 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fae90053340 con 0x7faea8150ea0 2024-01-24T17:45:50.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.423+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faea80d3680 con 0x7faea8150ea0 2024-01-24T17:45:50.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.425+0000 7fae9f400640 1 -- 172.21.15.39:0/3249665636 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7fae90018090 con 0x7faea8150ea0 2024-01-24T17:45:50.425 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:50.425 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8006a460 msgr2=0x7fae8006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8006a460 0x7fae8006c920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fae98035000 tx=0x7fae98030000 comp rx=0 tx=0).stop 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea8150ea0 msgr2=0x7faea81502d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea8150ea0 0x7faea81502d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c7a0 tx=0x7fae9000cc60 comp rx=0 tx=0).stop 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 shutdown_connections 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea81508e0 0x7faea8156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8006a460 0x7fae8006c920 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea8150ea0 0x7faea81502d0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 --2- 172.21.15.39:0/3249665636 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae88003970 0x7faea814fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 >> 172.21.15.39:0/3249665636 conn(0x7faea809bb90 msgr2=0x7faea815e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:50.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 shutdown_connections 2024-01-24T17:45:50.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.428+0000 7faead000640 1 -- 172.21.15.39:0/3249665636 wait complete. 2024-01-24T17:45:50.449 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:45:16.503215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:45:17.386565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:45:50.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:45:50.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.668+0000 7f68e6800640 1 Processor -- start 2024-01-24T17:45:50.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.668+0000 7f68e6800640 1 -- start start 2024-01-24T17:45:50.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.669+0000 7f68e6800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e00cb540 0x7f68e006d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.669+0000 7f68e6800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e00ad0b0 con 0x7f68e00accd0 2024-01-24T17:45:50.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.669+0000 7f68e6800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e00b2370 con 0x7f68e006c3a0 2024-01-24T17:45:50.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.669+0000 7f68e6800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e006d840 con 0x7f68e00cb540 2024-01-24T17:45:50.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.670+0000 7f68e5400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f68e00accd0 0x7f68e00cae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59172/0 (socket says 172.21.15.39:59172) 2024-01-24T17:45:50.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.670+0000 7f68e5400640 1 -- 172.21.15.39:0/2565024327 learned_addr learned my addr 172.21.15.39:0/2565024327 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.670+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618023280 0 0) 0x7f68e00ad0b0 con 0x7f68e00accd0 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.670+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c0003520 con 0x7f68e00accd0 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.670+0000 7f68e4a00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e00cb540 0x7f68e006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943503253 0 0) 0x7f68c0003520 con 0x7f68e00accd0 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e00cb540 msgr2=0x7f68e006d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e00cb540 0x7f68e006d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 >> v1:172.21.15.100:6789/0 conn(0x7f68e006c3a0 legacy=0x7f68e00ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e00b2650 con 0x7f68e00accd0 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f68d00017f0 con 0x7f68e00accd0 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 >> v1:172.21.15.39:6789/0 conn(0x7f68e00accd0 legacy=0x7f68e00cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e00cb540 0x7f68c00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004020 0x7f68c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68c0004d00 0x7f68c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e006d840 con 0x7f68c0004d00 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c000bbc0 con 0x7f68c0004020 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c000bd80 con 0x7f68e00cb540 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68e5e00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68c0004d00 0x7f68c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68e4a00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004020 0x7f68c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.671+0000 7f68e5400640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e00cb540 0x7f68c00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.672+0000 7f68e5e00640 1 -- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e00cb540 msgr2=0x7f68c00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.672+0000 7f68e5e00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e00cb540 0x7f68c00039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.672+0000 7f68e5e00640 1 -- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004020 msgr2=0x7f68c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.672+0000 7f68e5e00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004020 0x7f68c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.672+0000 7f68e5e00640 1 -- 172.21.15.39:0/2565024327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c0003520 con 0x7f68c0004d00 2024-01-24T17:45:50.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.672+0000 7f68e4a00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004020 0x7f68c00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.673+0000 7f68e5e00640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68c0004d00 0x7f68c0005180 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f68d8001e50 tx=0x7f68c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea79376bce7fd80 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.673+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8003b70 con 0x7f68c0004d00 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.673+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d8003cd0 con 0x7f68c0004d00 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.673+0000 7f68dfe00640 1 -- 172.21.15.39:0/2565024327 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d8005310 con 0x7f68c0004d00 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.674+0000 7f68e6800640 1 -- 172.21.15.39:0/2565024327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68c0004d00 msgr2=0x7f68c0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.674+0000 7f68e6800640 1 --2- 172.21.15.39:0/2565024327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68c0004d00 0x7f68c0005180 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f68d8001e50 tx=0x7f68c00057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.674+0000 7f68e5e00640 1 -- 172.21.15.39:0/2565024327 reap_dead start 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.674+0000 7f68e6800640 1 -- 172.21.15.39:0/2565024327 shutdown_connections 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.674+0000 7f68e6800640 1 -- 172.21.15.39:0/2565024327 >> 172.21.15.39:0/2565024327 conn(0x7f68e0069e50 msgr2=0x7f68e006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.674+0000 7f68e6800640 1 -- 172.21.15.39:0/2565024327 shutdown_connections 2024-01-24T17:45:50.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.674+0000 7f68e6800640 1 -- 172.21.15.39:0/2565024327 wait complete. 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.676+0000 7f68e6800640 1 Processor -- start 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.676+0000 7f68e6800640 1 -- start start 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e6800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004d00 0x7f68e01f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68e01f2ca0 0x7f68e00714e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e6800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e01f3120 0x7f68e0071a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e6800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e01f55d0 con 0x7f68e01f2ca0 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e6800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e00ae020 con 0x7f68c0004d00 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e6800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e00ae2c0 con 0x7f68e01f3120 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68e01f2ca0 0x7f68e00714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68e01f2ca0 0x7f68e00714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39062/0 (socket says 172.21.15.39:39062) 2024-01-24T17:45:50.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e4a00640 1 -- 172.21.15.39:0/2994011297 learned_addr learned my addr 172.21.15.39:0/2994011297 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e4a00640 1 -- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e01f3120 msgr2=0x7f68e0071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e4a00640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e01f3120 0x7f68e0071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.677+0000 7f68e4a00640 1 -- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004d00 msgr2=0x7f68e01f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.678+0000 7f68e4a00640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004d00 0x7f68e01f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.678+0000 7f68e4a00640 1 -- 172.21.15.39:0/2994011297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e00ae5a0 con 0x7f68e01f2ca0 2024-01-24T17:45:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.678+0000 7f68e4a00640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68e01f2ca0 0x7f68e00714e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f68c800e7e0 tx=0x7f68c800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.678+0000 7f68dd600640 1 -- 172.21.15.39:0/2994011297 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c800c5f0 con 0x7f68e01f2ca0 2024-01-24T17:45:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.678+0000 7f68dd600640 1 -- 172.21.15.39:0/2994011297 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c800c750 con 0x7f68e01f2ca0 2024-01-24T17:45:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.678+0000 7f68dd600640 1 -- 172.21.15.39:0/2994011297 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c800cb00 con 0x7f68e01f2ca0 2024-01-24T17:45:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.678+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e00ae880 con 0x7f68e01f2ca0 2024-01-24T17:45:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.678+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e00af080 con 0x7f68e01f2ca0 2024-01-24T17:45:50.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.680+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e006a260 con 0x7f68e01f2ca0 2024-01-24T17:45:50.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.680+0000 7f68dd600640 1 -- 172.21.15.39:0/2994011297 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f68c800cc60 con 0x7f68e01f2ca0 2024-01-24T17:45:50.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.681+0000 7f68dd600640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68b806a460 0x7f68b806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:50.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.681+0000 7f68dd600640 1 -- 172.21.15.39:0/2994011297 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68c8088660 con 0x7f68e01f2ca0 2024-01-24T17:45:50.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.682+0000 7f68e5400640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68b806a460 0x7f68b806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:50.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.683+0000 7f68e5400640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68b806a460 0x7f68b806c920 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f68d0001920 tx=0x7f68d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:50.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.686+0000 7f68dd600640 1 -- 172.21.15.39:0/2994011297 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f68c8053bb0 con 0x7f68e01f2ca0 2024-01-24T17:45:50.822 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:45:50.829 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:45:50.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.882+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f68e00b2be0 con 0x7f68e01f2ca0 2024-01-24T17:45:50.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.883+0000 7f68dd600640 1 -- 172.21.15.39:0/2994011297 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7f68c80571d0 con 0x7f68e01f2ca0 2024-01-24T17:45:50.884 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68b806a460 msgr2=0x7f68b806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68b806a460 0x7f68b806c920 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f68d0001920 tx=0x7f68d0009040 comp rx=0 tx=0).stop 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68e01f2ca0 msgr2=0x7f68e00714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68e01f2ca0 0x7f68e00714e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f68c800e7e0 tx=0x7f68c800eca0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 shutdown_connections 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68e01f3120 0x7f68e0071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f68b806a460 0x7f68b806c920 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68e01f2ca0 0x7f68e00714e0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 --2- 172.21.15.39:0/2994011297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68c0004d00 0x7f68e01f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 >> 172.21.15.39:0/2994011297 conn(0x7f68e0069e50 msgr2=0x7f68e00ad4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 shutdown_connections 2024-01-24T17:45:50.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:50.886+0000 7f68e6800640 1 -- 172.21.15.39:0/2994011297 wait complete. 2024-01-24T17:45:50.907 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:45:51.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.103+0000 7f9018e00640 1 Processor -- start 2024-01-24T17:45:51.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.103+0000 7f9018e00640 1 -- start start 2024-01-24T17:45:51.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.104+0000 7f9018e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9014160480 0x7f90140ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.104+0000 7f9018e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f901406f140 0x7f90140cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.104+0000 7f9018e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90140cea20 con 0x7f90141600a0 2024-01-24T17:45:51.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.104+0000 7f9018e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f901406ec60 con 0x7f9014160480 2024-01-24T17:45:51.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.104+0000 7f9018e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140ced00 con 0x7f901406f140 2024-01-24T17:45:51.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9012a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9014160480 0x7f90140ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9013400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f90141600a0 0x7f901406f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:59178/0 (socket says 172.21.15.39:59178) 2024-01-24T17:45:51.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9012a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9014160480 0x7f90140ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39076/0 (socket says 172.21.15.39:39076) 2024-01-24T17:45:51.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9013400640 1 -- 172.21.15.39:0/3771605365 learned_addr learned my addr 172.21.15.39:0/3771605365 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:51.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9012a00640 1 -- 172.21.15.39:0/3771605365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f901406f140 msgr2=0x7f90140cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:51.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9012a00640 1 --2- 172.21.15.39:0/3771605365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f901406f140 0x7f90140cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9012a00640 1 -- 172.21.15.39:0/3771605365 >> v1:172.21.15.39:6789/0 conn(0x7f90141600a0 legacy=0x7f901406f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9012a00640 1 -- 172.21.15.39:0/3771605365 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140cefe0 con 0x7f9014160480 2024-01-24T17:45:51.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.105+0000 7f9012000640 1 -- 172.21.15.39:0/3771605365 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937633026 0 0) 0x7f90140cea20 con 0x7f90141600a0 2024-01-24T17:45:51.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.106+0000 7f9012a00640 1 --2- 172.21.15.39:0/3771605365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9014160480 0x7f90140ccf70 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0097a0 tx=0x7f8ffc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddff9e6079f29d99 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:51.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.107+0000 7f9012000640 1 -- 172.21.15.39:0/3771605365 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc02e070 con 0x7f9014160480 2024-01-24T17:45:51.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.107+0000 7f9012000640 1 -- 172.21.15.39:0/3771605365 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ffc0315a0 con 0x7f9014160480 2024-01-24T17:45:51.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.107+0000 7f9012000640 1 -- 172.21.15.39:0/3771605365 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc031930 con 0x7f9014160480 2024-01-24T17:45:51.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.107+0000 7f9018e00640 1 -- 172.21.15.39:0/3771605365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9014160480 msgr2=0x7f90140ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.107+0000 7f9018e00640 1 --2- 172.21.15.39:0/3771605365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9014160480 0x7f90140ccf70 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc0097a0 tx=0x7f8ffc02b840 comp rx=0 tx=0).stop 2024-01-24T17:45:51.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.108+0000 7f9018e00640 1 -- 172.21.15.39:0/3771605365 shutdown_connections 2024-01-24T17:45:51.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.108+0000 7f9018e00640 1 --2- 172.21.15.39:0/3771605365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9014160480 0x7f90140ccf70 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.108+0000 7f9018e00640 1 --2- 172.21.15.39:0/3771605365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f901406f140 0x7f90140cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.108+0000 7f9018e00640 1 -- 172.21.15.39:0/3771605365 >> 172.21.15.39:0/3771605365 conn(0x7f9014089fd0 msgr2=0x7f90140d06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:51.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.108+0000 7f9018e00640 1 -- 172.21.15.39:0/3771605365 shutdown_connections 2024-01-24T17:45:51.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.108+0000 7f9018e00640 1 -- 172.21.15.39:0/3771605365 wait complete. 2024-01-24T17:45:51.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.109+0000 7f9018e00640 1 Processor -- start 2024-01-24T17:45:51.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.110+0000 7f9018e00640 1 -- start start 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.110+0000 7f9018e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f901406f140 0x7f90141f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.110+0000 7f9018e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90141600a0 0x7f9014148370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.110+0000 7f9018e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9014160480 0x7f90140c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.110+0000 7f9018e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90141f2f50 con 0x7f901406f140 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.110+0000 7f9018e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90141488b0 con 0x7f90141600a0 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.110+0000 7f9018e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901414b210 con 0x7f9014160480 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.111+0000 7f9013400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f901406f140 0x7f90141f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.111+0000 7f9013400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f901406f140 0x7f90141f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39082/0 (socket says 172.21.15.39:39082) 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.111+0000 7f9013400640 1 -- 172.21.15.39:0/1555000742 learned_addr learned my addr 172.21.15.39:0/1555000742 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:51.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.111+0000 7f9013400640 1 -- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9014160480 msgr2=0x7f90140c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.111+0000 7f9013400640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9014160480 0x7f90140c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.111+0000 7f9013400640 1 -- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90141600a0 msgr2=0x7f9014148370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.111+0000 7f9013400640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90141600a0 0x7f9014148370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.111+0000 7f9013400640 1 -- 172.21.15.39:0/1555000742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc009400 con 0x7f901406f140 2024-01-24T17:45:51.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.112+0000 7f9013400640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f901406f140 0x7f90141f2a10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f900400c7a0 tx=0x7f900400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:51.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.112+0000 7f900b400640 1 -- 172.21.15.39:0/1555000742 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900400a640 con 0x7f901406f140 2024-01-24T17:45:51.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.112+0000 7f900b400640 1 -- 172.21.15.39:0/1555000742 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900400a7a0 con 0x7f901406f140 2024-01-24T17:45:51.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.112+0000 7f900b400640 1 -- 172.21.15.39:0/1555000742 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900400abd0 con 0x7f901406f140 2024-01-24T17:45:51.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.112+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f901414b4f0 con 0x7f901406f140 2024-01-24T17:45:51.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.112+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90141f5620 con 0x7f901406f140 2024-01-24T17:45:51.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.113+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90141f5810 con 0x7f901406f140 2024-01-24T17:45:51.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.114+0000 7f900b400640 1 -- 172.21.15.39:0/1555000742 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f900400ad30 con 0x7f901406f140 2024-01-24T17:45:51.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.115+0000 7f900b400640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8fec06a430 0x7f8fec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.115+0000 7f900b400640 1 -- 172.21.15.39:0/1555000742 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9004088260 con 0x7f901406f140 2024-01-24T17:45:51.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.116+0000 7f9012a00640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8fec06a430 0x7f8fec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.117+0000 7f9012a00640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8fec06a430 0x7f8fec06c8f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc036000 tx=0x7f8ffc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:51.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.119+0000 7f900b400640 1 -- 172.21.15.39:0/1555000742 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9004053840 con 0x7f901406f140 2024-01-24T17:45:51.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.314+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90140cf2c0 con 0x7f901406f140 2024-01-24T17:45:51.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.315+0000 7f900b400640 1 -- 172.21.15.39:0/1555000742 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v63) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9004017020 con 0x7f901406f140 2024-01-24T17:45:51.316 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:45:51.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8fec06a430 msgr2=0x7f8fec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8fec06a430 0x7f8fec06c8f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc036000 tx=0x7f8ffc030000 comp rx=0 tx=0).stop 2024-01-24T17:45:51.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f901406f140 msgr2=0x7f90141f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f901406f140 0x7f90141f2a10 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f900400c7a0 tx=0x7f900400cc60 comp rx=0 tx=0).stop 2024-01-24T17:45:51.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 shutdown_connections 2024-01-24T17:45:51.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9014160480 0x7f90140c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f90141600a0 0x7f9014148370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8fec06a430 0x7f8fec06c8f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 --2- 172.21.15.39:0/1555000742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f901406f140 0x7f90141f2a10 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.318+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 >> 172.21.15.39:0/1555000742 conn(0x7f9014089fd0 msgr2=0x7f901415cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:51.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.319+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 shutdown_connections 2024-01-24T17:45:51.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.319+0000 7f9018e00640 1 -- 172.21.15.39:0/1555000742 wait complete. 2024-01-24T17:45:51.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:45:51.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.562+0000 7ffbf7200640 1 Processor -- start 2024-01-24T17:45:51.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.563+0000 7ffbf7200640 1 -- start start 2024-01-24T17:45:51.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.564+0000 7ffbf7200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffbf015c8d0 0x7ffbf00d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.564+0000 7ffbf7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffbf00b2710 0x7ffbf00d2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.564+0000 7ffbf7200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf00b38a0 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.564+0000 7ffbf7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf00d3100 con 0x7ffbf00b2710 2024-01-24T17:45:51.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.564+0000 7ffbf7200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf00d13e0 con 0x7ffbf015c8d0 2024-01-24T17:45:51.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.565+0000 7ffbf5400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffbf00b2710 0x7ffbf00d2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.565+0000 7ffbf5400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffbf00b2710 0x7ffbf00d2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39096/0 (socket says 172.21.15.39:39096) 2024-01-24T17:45:51.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.565+0000 7ffbf5400640 1 -- 172.21.15.39:0/1242808182 learned_addr learned my addr 172.21.15.39:0/1242808182 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:51.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.565+0000 7ffbf5e00640 1 --2- 172.21.15.39:0/1242808182 >> v2:172.21.15.100:3300/0 conn(0x7ffbf015c8d0 0x7ffbf00d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.566+0000 7ffbf5400640 1 -- 172.21.15.39:0/1242808182 >> v2:172.21.15.100:3300/0 conn(0x7ffbf015c8d0 msgr2=0x7ffbf00d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:51.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.566+0000 7ffbf5400640 1 --2- 172.21.15.39:0/1242808182 >> v2:172.21.15.100:3300/0 conn(0x7ffbf015c8d0 0x7ffbf00d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.566+0000 7ffbf5400640 1 -- 172.21.15.39:0/1242808182 >> v1:172.21.15.100:6789/0 conn(0x7ffbf00d2cb0 legacy=0x7ffbf00cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.566+0000 7ffbf5400640 1 -- 172.21.15.39:0/1242808182 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf00d0ab0 con 0x7ffbf00b2710 2024-01-24T17:45:51.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.567+0000 7ffbf5400640 1 --2- 172.21.15.39:0/1242808182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00b2710 0x7ffbf00d2770 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ffbf006a5d0 tx=0x7ffbd802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca235028805624f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:51.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.567+0000 7ffbf4a00640 1 -- 172.21.15.39:0/1242808182 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8030860 con 0x7ffbf00b2710 2024-01-24T17:45:51.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.567+0000 7ffbf4a00640 1 -- 172.21.15.39:0/1242808182 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd80309c0 con 0x7ffbf00b2710 2024-01-24T17:45:51.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.567+0000 7ffbf4a00640 1 -- 172.21.15.39:0/1242808182 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8030d50 con 0x7ffbf00b2710 2024-01-24T17:45:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.568+0000 7ffbf7200640 1 -- 172.21.15.39:0/1242808182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00b2710 msgr2=0x7ffbf00d2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.568+0000 7ffbf7200640 1 --2- 172.21.15.39:0/1242808182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00b2710 0x7ffbf00d2770 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ffbf006a5d0 tx=0x7ffbd802b9a0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.568+0000 7ffbf7200640 1 -- 172.21.15.39:0/1242808182 shutdown_connections 2024-01-24T17:45:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.568+0000 7ffbf7200640 1 --2- 172.21.15.39:0/1242808182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00b2710 0x7ffbf00d2770 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.568+0000 7ffbf7200640 1 --2- 172.21.15.39:0/1242808182 >> v2:172.21.15.100:3300/0 conn(0x7ffbf015c8d0 0x7ffbf00d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.568+0000 7ffbf7200640 1 -- 172.21.15.39:0/1242808182 >> 172.21.15.39:0/1242808182 conn(0x7ffbf0071dd0 msgr2=0x7ffbf009bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.568+0000 7ffbf7200640 1 -- 172.21.15.39:0/1242808182 shutdown_connections 2024-01-24T17:45:51.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.568+0000 7ffbf7200640 1 -- 172.21.15.39:0/1242808182 wait complete. 2024-01-24T17:45:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.570+0000 7ffbf7200640 1 Processor -- start 2024-01-24T17:45:51.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.570+0000 7ffbf7200640 1 -- start start 2024-01-24T17:45:51.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf7200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbf00b2710 0x7ffbf00cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00d2cb0 0x7ffbf00b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbf015c8d0 0x7ffbf015ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf00b3f70 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf7200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf00b4210 con 0x7ffbf00b2710 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf00b44f0 con 0x7ffbf015c8d0 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00d2cb0 0x7ffbf00b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00d2cb0 0x7ffbf00b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39110/0 (socket says 172.21.15.39:39110) 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.571+0000 7ffbf5400640 1 -- 172.21.15.39:0/2739164472 learned_addr learned my addr 172.21.15.39:0/2739164472 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.572+0000 7ffbf6800640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbf015c8d0 0x7ffbf015ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.572+0000 7ffbf5e00640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbf00b2710 0x7ffbf00cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.572+0000 7ffbf5400640 1 -- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbf015c8d0 msgr2=0x7ffbf015ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.572+0000 7ffbf5400640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbf015c8d0 0x7ffbf015ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.572+0000 7ffbf5400640 1 -- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbf00b2710 msgr2=0x7ffbf00cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.572+0000 7ffbf5400640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbf00b2710 0x7ffbf00cb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.572+0000 7ffbf5400640 1 -- 172.21.15.39:0/2739164472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd80094d0 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.572+0000 7ffbf5400640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00d2cb0 0x7ffbf00b3a30 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802b9e0 tx=0x7ffbd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:51.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.573+0000 7ffbee000640 1 -- 172.21.15.39:0/2739164472 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8039a90 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.573+0000 7ffbee000640 1 -- 172.21.15.39:0/2739164472 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd8039c10 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.573+0000 7ffbee000640 1 -- 172.21.15.39:0/2739164472 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8043410 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.573+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf00b47d0 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.573+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf015b240 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.574+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf004dff0 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.575+0000 7ffbee000640 1 -- 172.21.15.39:0/2739164472 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ffbd802f3b0 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.576+0000 7ffbee000640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffbc806a430 0x7ffbc806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.577+0000 7ffbee000640 1 -- 172.21.15.39:0/2739164472 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbd80aabc0 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.577+0000 7ffbf5e00640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffbc806a430 0x7ffbc806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:51.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.578+0000 7ffbf5e00640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffbc806a430 0x7ffbc806c8f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009a10 tx=0x7ffbe0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:51.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.581+0000 7ffbee000640 1 -- 172.21.15.39:0/2739164472 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ffbd8076230 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.770+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffbf00cfa30 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.771+0000 7ffbee000640 1 -- 172.21.15.39:0/2739164472 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v63) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffbd80340a0 con 0x7ffbf00d2cb0 2024-01-24T17:45:51.772 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:45:51.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffbc806a430 msgr2=0x7ffbc806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffbc806a430 0x7ffbc806c8f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009a10 tx=0x7ffbe0008040 comp rx=0 tx=0).stop 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00d2cb0 msgr2=0x7ffbf00b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00d2cb0 0x7ffbf00b3a30 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802b9e0 tx=0x7ffbd8004730 comp rx=0 tx=0).stop 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 shutdown_connections 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffbf015c8d0 0x7ffbf015ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffbc806a430 0x7ffbc806c8f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbf00d2cb0 0x7ffbf00b3a30 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 --2- 172.21.15.39:0/2739164472 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffbf00b2710 0x7ffbf00cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.773+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 >> 172.21.15.39:0/2739164472 conn(0x7ffbf0071dd0 msgr2=0x7ffbf015eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.774+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 shutdown_connections 2024-01-24T17:45:51.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:51.774+0000 7ffbf7200640 1 -- 172.21.15.39:0/2739164472 wait complete. 2024-01-24T17:45:51.793 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:45:52.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.008+0000 7f2307e00640 1 Processor -- start 2024-01-24T17:45:52.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.008+0000 7f2307e00640 1 -- start start 2024-01-24T17:45:52.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.010+0000 7f2307e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23080ae650 0x7f23080aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:52.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.010+0000 7f2307e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f23080cef30 0x7f23080cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:52.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.010+0000 7f2307e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23080ca1a0 con 0x7f230815cb40 2024-01-24T17:45:52.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.010+0000 7f2307e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23080cd450 con 0x7f23080ae650 2024-01-24T17:45:52.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.010+0000 7f2307e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23080c9c20 con 0x7f23080cef30 2024-01-24T17:45:52.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.010+0000 7f2306000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23080ae650 0x7f23080aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:52.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.010+0000 7f2306000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23080ae650 0x7f23080aef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39124/0 (socket says 172.21.15.39:39124) 2024-01-24T17:45:52.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.011+0000 7f2306000640 1 -- 172.21.15.39:0/985208011 learned_addr learned my addr 172.21.15.39:0/985208011 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:52.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.011+0000 7f2307400640 1 --2- 172.21.15.39:0/985208011 >> v2:172.21.15.100:3300/0 conn(0x7f23080cef30 0x7f23080cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:52.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.011+0000 7f2306000640 1 -- 172.21.15.39:0/985208011 >> v2:172.21.15.100:3300/0 conn(0x7f23080cef30 msgr2=0x7f23080cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:52.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.011+0000 7f2306000640 1 --2- 172.21.15.39:0/985208011 >> v2:172.21.15.100:3300/0 conn(0x7f23080cef30 0x7f23080cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.011+0000 7f2306000640 1 -- 172.21.15.39:0/985208011 >> v1:172.21.15.100:6789/0 conn(0x7f230815cb40 legacy=0x7f23080ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:52.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.011+0000 7f2306000640 1 -- 172.21.15.39:0/985208011 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080cc9f0 con 0x7f23080ae650 2024-01-24T17:45:52.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.011+0000 7f2307400640 1 --2- 172.21.15.39:0/985208011 >> v2:172.21.15.100:3300/0 conn(0x7f23080cef30 0x7f23080cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:45:52.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.013+0000 7f2306000640 1 --2- 172.21.15.39:0/985208011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080ae650 0x7f23080aef30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f23080c9bb0 tx=0x7f22f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a96966f7e8450b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:52.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.013+0000 7f2305600640 1 -- 172.21.15.39:0/985208011 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0031870 con 0x7f23080ae650 2024-01-24T17:45:52.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.013+0000 7f2305600640 1 -- 172.21.15.39:0/985208011 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f00319d0 con 0x7f23080ae650 2024-01-24T17:45:52.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.013+0000 7f2305600640 1 -- 172.21.15.39:0/985208011 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0031d60 con 0x7f23080ae650 2024-01-24T17:45:52.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.014+0000 7f2307e00640 1 -- 172.21.15.39:0/985208011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080ae650 msgr2=0x7f23080aef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:52.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.014+0000 7f2307e00640 1 --2- 172.21.15.39:0/985208011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080ae650 0x7f23080aef30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f23080c9bb0 tx=0x7f22f002b910 comp rx=0 tx=0).stop 2024-01-24T17:45:52.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.014+0000 7f2307e00640 1 -- 172.21.15.39:0/985208011 shutdown_connections 2024-01-24T17:45:52.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.014+0000 7f2307e00640 1 --2- 172.21.15.39:0/985208011 >> v2:172.21.15.100:3300/0 conn(0x7f23080cef30 0x7f23080cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.014+0000 7f2307e00640 1 --2- 172.21.15.39:0/985208011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080ae650 0x7f23080aef30 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.014+0000 7f2307e00640 1 -- 172.21.15.39:0/985208011 >> 172.21.15.39:0/985208011 conn(0x7f2308098a40 msgr2=0x7f23080cf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:52.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.014+0000 7f2307e00640 1 -- 172.21.15.39:0/985208011 shutdown_connections 2024-01-24T17:45:52.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.014+0000 7f2307e00640 1 -- 172.21.15.39:0/985208011 wait complete. 2024-01-24T17:45:52.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.016+0000 7f2307e00640 1 Processor -- start 2024-01-24T17:45:52.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.016+0000 7f2307e00640 1 -- start start 2024-01-24T17:45:52.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2307e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23080ae650 0x7f2308148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2307e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080cef30 0x7f23080c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2307e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f230815cb40 0x7f23080b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2307e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080c66a0 con 0x7f23080cef30 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2307e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080c6980 con 0x7f230815cb40 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2307e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23080c6c60 con 0x7f23080ae650 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2306000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080cef30 0x7f23080c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2306000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080cef30 0x7f23080c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39138/0 (socket says 172.21.15.39:39138) 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.017+0000 7f2306000640 1 -- 172.21.15.39:0/2296888802 learned_addr learned my addr 172.21.15.39:0/2296888802 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:52.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.018+0000 7f2306a00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23080ae650 0x7f2308148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:52.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.018+0000 7f2307400640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f230815cb40 0x7f23080b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:52.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.018+0000 7f2306000640 1 -- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23080ae650 msgr2=0x7f2308148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:52.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.018+0000 7f2306000640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23080ae650 0x7f2308148a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.018+0000 7f2306000640 1 -- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f230815cb40 msgr2=0x7f23080b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:52.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.018+0000 7f2306000640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f230815cb40 0x7f23080b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.018+0000 7f2306000640 1 -- 172.21.15.39:0/2296888802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f00094d0 con 0x7f23080cef30 2024-01-24T17:45:52.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.018+0000 7f2306000640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080cef30 0x7f23080c3cf0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f22f002b950 tx=0x7f22f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:52.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.019+0000 7f22fea00640 1 -- 172.21.15.39:0/2296888802 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f003ad30 con 0x7f23080cef30 2024-01-24T17:45:52.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.019+0000 7f22fea00640 1 -- 172.21.15.39:0/2296888802 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f00443f0 con 0x7f23080cef30 2024-01-24T17:45:52.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.019+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23080c6f40 con 0x7f23080cef30 2024-01-24T17:45:52.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.019+0000 7f22fea00640 1 -- 172.21.15.39:0/2296888802 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f00447a0 con 0x7f23080cef30 2024-01-24T17:45:52.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.019+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23080c7500 con 0x7f23080cef30 2024-01-24T17:45:52.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.020+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23080c76f0 con 0x7f23080cef30 2024-01-24T17:45:52.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.021+0000 7f22fea00640 1 -- 172.21.15.39:0/2296888802 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f22f004d450 con 0x7f23080cef30 2024-01-24T17:45:52.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.022+0000 7f22fea00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22e006a430 0x7f22e006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:52.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.022+0000 7f22fea00640 1 -- 172.21.15.39:0/2296888802 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22f002d040 con 0x7f23080cef30 2024-01-24T17:45:52.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.022+0000 7f2306a00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22e006a430 0x7f22e006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:52.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.023+0000 7f2306a00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22e006a430 0x7f22e006c8f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f22f8000f60 tx=0x7f22f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:52.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.026+0000 7f22fea00640 1 -- 172.21.15.39:0/2296888802 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f22f00764c0 con 0x7f23080cef30 2024-01-24T17:45:52.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.237+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2308067040 con 0x7f23080cef30 2024-01-24T17:45:52.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.237+0000 7f22fea00640 1 -- 172.21.15.39:0/2296888802 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v63) v1 ==== 80+0+0 (secure 0 0 0) 0x7f22f0035020 con 0x7f23080cef30 2024-01-24T17:45:52.238 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22e006a430 msgr2=0x7f22e006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22e006a430 0x7f22e006c8f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f22f8000f60 tx=0x7f22f8009040 comp rx=0 tx=0).stop 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080cef30 msgr2=0x7f23080c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080cef30 0x7f23080c3cf0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f22f002b950 tx=0x7f22f00046c0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 shutdown_connections 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f230815cb40 0x7f23080b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22e006a430 0x7f22e006c8f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23080cef30 0x7f23080c3cf0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 --2- 172.21.15.39:0/2296888802 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23080ae650 0x7f2308148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 >> 172.21.15.39:0/2296888802 conn(0x7f2308098a40 msgr2=0x7f23080ca5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:52.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 shutdown_connections 2024-01-24T17:45:52.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:52.240+0000 7f2307e00640 1 -- 172.21.15.39:0/2296888802 wait complete. 2024-01-24T17:45:52.261 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:45:52.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:52.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:45:52.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[45373]: fuse finished with error 0 and tester_r 0 2024-01-24T17:45:52.365 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:45:58.466 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:45:58.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:58.467 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:45:58.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth get-or-create client.badguy mds 'allow rw path=/foo/bar' osd 'allow rw' mon 'allow r' 2024-01-24T17:45:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.766+0000 7f56c9000640 1 Processor -- start 2024-01-24T17:45:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.766+0000 7f56c9000640 1 -- start start 2024-01-24T17:45:58.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.767+0000 7f56c9000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f56c4071e70 0x7f56c40cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.767+0000 7f56c9000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c406fd80 con 0x7f56c40b1210 2024-01-24T17:45:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.767+0000 7f56c9000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c4070020 con 0x7f56c415a6e0 2024-01-24T17:45:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.767+0000 7f56c9000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c415aac0 con 0x7f56c4071e70 2024-01-24T17:45:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.767+0000 7f56c2a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f56c4071e70 0x7f56c40cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.768+0000 7f56c3e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f56c40b1210 0x7f56c40c9140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:45464/0 (socket says 172.21.15.39:45464) 2024-01-24T17:45:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.768+0000 7f56c2a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f56c4071e70 0x7f56c40cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45322/0 (socket says 172.21.15.39:45322) 2024-01-24T17:45:58.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.768+0000 7f56c3e00640 1 -- 172.21.15.39:0/441996753 learned_addr learned my addr 172.21.15.39:0/441996753 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:58.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.768+0000 7f56c2a00640 1 -- 172.21.15.39:0/441996753 >> v1:172.21.15.100:6789/0 conn(0x7f56c415a6e0 legacy=0x7f56c415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:58.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.768+0000 7f56c2a00640 1 -- 172.21.15.39:0/441996753 >> v1:172.21.15.39:6789/0 conn(0x7f56c40b1210 legacy=0x7f56c40c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:58.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.768+0000 7f56c2a00640 1 -- 172.21.15.39:0/441996753 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4070300 con 0x7f56c4071e70 2024-01-24T17:45:58.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.768+0000 7f56c2000640 1 -- 172.21.15.39:0/441996753 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658862815 0 0) 0x7f56c406fd80 con 0x7f56c40b1210 2024-01-24T17:45:58.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.769+0000 7f56c2a00640 1 --2- 172.21.15.39:0/441996753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c4071e70 0x7f56c40cb260 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0097a0 tx=0x7f56ac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ec50c6f7a7e7f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:58.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c2000640 1 -- 172.21.15.39:0/441996753 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac030610 con 0x7f56c4071e70 2024-01-24T17:45:58.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c2000640 1 -- 172.21.15.39:0/441996753 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac030770 con 0x7f56c4071e70 2024-01-24T17:45:58.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c2000640 1 -- 172.21.15.39:0/441996753 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac030b00 con 0x7f56c4071e70 2024-01-24T17:45:58.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c9000640 1 -- 172.21.15.39:0/441996753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c4071e70 msgr2=0x7f56c40cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:58.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c9000640 1 --2- 172.21.15.39:0/441996753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c4071e70 0x7f56c40cb260 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0097a0 tx=0x7f56ac02b960 comp rx=0 tx=0).stop 2024-01-24T17:45:58.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c9000640 1 -- 172.21.15.39:0/441996753 shutdown_connections 2024-01-24T17:45:58.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c9000640 1 --2- 172.21.15.39:0/441996753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c4071e70 0x7f56c40cb260 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:58.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c9000640 1 -- 172.21.15.39:0/441996753 >> 172.21.15.39:0/441996753 conn(0x7f56c406a090 msgr2=0x7f56c406a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:58.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c9000640 1 -- 172.21.15.39:0/441996753 shutdown_connections 2024-01-24T17:45:58.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.770+0000 7f56c9000640 1 -- 172.21.15.39:0/441996753 wait complete. 2024-01-24T17:45:58.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.772+0000 7f56c9000640 1 Processor -- start 2024-01-24T17:45:58.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.772+0000 7f56c9000640 1 -- start start 2024-01-24T17:45:58.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.773+0000 7f56c9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c40b1210 0x7f56c41531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:58.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.773+0000 7f56c9000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c415a6e0 0x7f56c40af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:58.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.773+0000 7f56c9000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c41efd20 0x7f56c40c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:58.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.773+0000 7f56c9000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40afe60 con 0x7f56c40b1210 2024-01-24T17:45:58.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.773+0000 7f56c9000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40b0140 con 0x7f56c415a6e0 2024-01-24T17:45:58.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.773+0000 7f56c9000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40c5610 con 0x7f56c41efd20 2024-01-24T17:45:58.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.773+0000 7f56c3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c40b1210 0x7f56c41531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:58.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c40b1210 0x7f56c41531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45328/0 (socket says 172.21.15.39:45328) 2024-01-24T17:45:58.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3400640 1 -- 172.21.15.39:0/2537674797 learned_addr learned my addr 172.21.15.39:0/2537674797 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:58.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c2a00640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c415a6e0 0x7f56c40af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:58.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3e00640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c41efd20 0x7f56c40c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:58.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3400640 1 -- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c41efd20 msgr2=0x7f56c40c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:58.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3400640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c41efd20 0x7f56c40c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:58.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3400640 1 -- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c415a6e0 msgr2=0x7f56c40af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:58.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3400640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c415a6e0 0x7f56c40af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:58.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3400640 1 -- 172.21.15.39:0/2537674797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac009400 con 0x7f56c40b1210 2024-01-24T17:45:58.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c2a00640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c415a6e0 0x7f56c40af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:58.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3e00640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c41efd20 0x7f56c40c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:45:58.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.774+0000 7f56c3400640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c40b1210 0x7f56c41531d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c400 tx=0x7f56b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:58.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.775+0000 7f56bb400640 1 -- 172.21.15.39:0/2537674797 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b40052d0 con 0x7f56c40b1210 2024-01-24T17:45:58.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.775+0000 7f56bb400640 1 -- 172.21.15.39:0/2537674797 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b4005430 con 0x7f56c40b1210 2024-01-24T17:45:58.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.775+0000 7f56bb400640 1 -- 172.21.15.39:0/2537674797 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b4003830 con 0x7f56c40b1210 2024-01-24T17:45:58.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.775+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c40c58f0 con 0x7f56c40b1210 2024-01-24T17:45:58.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.775+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c4157120 con 0x7f56c40b1210 2024-01-24T17:45:58.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.776+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c4157310 con 0x7f56c40b1210 2024-01-24T17:45:58.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.777+0000 7f56bb400640 1 -- 172.21.15.39:0/2537674797 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f56b4003990 con 0x7f56c40b1210 2024-01-24T17:45:58.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.778+0000 7f56bb400640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f569c06a460 0x7f569c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:58.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.778+0000 7f56bb400640 1 -- 172.21.15.39:0/2537674797 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56b4087ec0 con 0x7f56c40b1210 2024-01-24T17:45:58.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.779+0000 7f56c2a00640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f569c06a460 0x7f569c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:58.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.780+0000 7f56c2a00640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f569c06a460 0x7f569c06c920 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f56ac036000 tx=0x7f56ac03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:58.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:58.783+0000 7f56bb400640 1 -- 172.21.15.39:0/2537674797 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f56b40534a0 con 0x7f56c40b1210 2024-01-24T17:45:59.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.052+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.badguy", "caps": ["mds", "allow rw path=/foo/bar", "osd", "allow rw", "mon", "allow r"]} v 0) v1 -- 0x7f56c40c5470 con 0x7f56c40b1210 2024-01-24T17:45:59.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.062+0000 7f56bb400640 1 -- 172.21.15.39:0/2537674797 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.badguy", "caps": ["mds", "allow rw path=/foo/bar", "osd", "allow rw", "mon", "allow r"]}]=0 v14) v1 ==== 173+0+64 (secure 0 0 0) 0x7f56c40c5470 con 0x7f56c40b1210 2024-01-24T17:45:59.063 INFO:teuthology.orchestra.run.smithi039.stdout:[client.badguy] 2024-01-24T17:45:59.063 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQDXTLFlLXBAAxAAmFCWwaJ4A7NtzfQcEI/TdA== 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f569c06a460 msgr2=0x7f569c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f569c06a460 0x7f569c06c920 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f56ac036000 tx=0x7f56ac03b040 comp rx=0 tx=0).stop 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c40b1210 msgr2=0x7f56c41531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c40b1210 0x7f56c41531d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c400 tx=0x7f56b400c8c0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 shutdown_connections 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f56c41efd20 0x7f56c40c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56c415a6e0 0x7f56c40af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f569c06a460 0x7f569c06c920 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 --2- 172.21.15.39:0/2537674797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56c40b1210 0x7f56c41531d0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 >> 172.21.15.39:0/2537674797 conn(0x7f56c406a090 msgr2=0x7f56c40c98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:59.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 shutdown_connections 2024-01-24T17:45:59.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.065+0000 7f56c9000640 1 -- 172.21.15.39:0/2537674797 wait complete. 2024-01-24T17:45:59.087 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:45:59.087 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.client.badguy.keyring 2024-01-24T17:45:59.190 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:45:59.198 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.375 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.376 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.377 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.378 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-01-24T17:45:59.379 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-01-24T17:45:59.380 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-01-24T17:45:59.381 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-01-24T17:45:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:45:59.383 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:45:59.384 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-01-24T17:45:59.385 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:[client.badguy] 2024-01-24T17:45:59.386 INFO:teuthology.orchestra.run.smithi039.stdout: keyring = /etc/ceph/ceph.client.badguy.keyring 2024-01-24T17:45:59.397 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-01-24T17:45:59.397 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-01-24T17:45:59.397 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-01-24T17:45:59.398 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:45:59.399 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.400 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:45:59.401 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-01-24T17:45:59.402 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-01-24T17:45:59.403 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = bitmap 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.404 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-01-24T17:45:59.405 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-01-24T17:45:59.406 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: mon op complaint time = 120 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: client mount timeout = 600 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = "" 2024-01-24T17:45:59.407 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout:[mds] 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds = 20 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds balancer = 20 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug frag = True 2024-01-24T17:45:59.408 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug scatterstat = True 2024-01-24T17:45:59.409 INFO:teuthology.orchestra.run.smithi100.stdout: mds op complaint time = 180 2024-01-24T17:45:59.409 INFO:teuthology.orchestra.run.smithi100.stdout: mds verify scatter = True 2024-01-24T17:45:59.409 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:45:59.409 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:45:59.409 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:45:59.409 INFO:teuthology.orchestra.run.smithi100.stdout:[client.badguy] 2024-01-24T17:45:59.409 INFO:teuthology.orchestra.run.smithi100.stdout: keyring = /etc/ceph/ceph.client.badguy.keyring 2024-01-24T17:45:59.444 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:45:59.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.670+0000 7f4b87e00640 1 Processor -- start 2024-01-24T17:45:59.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.670+0000 7f4b87e00640 1 -- start start 2024-01-24T17:45:59.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.671+0000 7f4b87e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4b8815c9c0 0x7f4b880ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:59.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.671+0000 7f4b87e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b880d1470 con 0x7f4b8815cda0 2024-01-24T17:45:59.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.671+0000 7f4b87e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b880d1900 con 0x7f4b880cf9d0 2024-01-24T17:45:59.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.671+0000 7f4b87e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b880d0860 con 0x7f4b8815c9c0 2024-01-24T17:45:59.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.672+0000 7f4b86000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4b8815cda0 0x7f4b880cf2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:45478/0 (socket says 172.21.15.39:45478) 2024-01-24T17:45:59.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.672+0000 7f4b86000640 1 -- 172.21.15.39:0/2738777068 learned_addr learned my addr 172.21.15.39:0/2738777068 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:45:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.672+0000 7f4b86a00640 1 --2- 172.21.15.39:0/2738777068 >> v2:172.21.15.100:3300/0 conn(0x7f4b8815c9c0 0x7f4b880ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.672+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766672145 0 0) 0x7f4b880d1470 con 0x7f4b8815cda0 2024-01-24T17:45:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.672+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b68003520 con 0x7f4b8815cda0 2024-01-24T17:45:59.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 375437961 0 0) 0x7f4b68003520 con 0x7f4b8815cda0 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 >> v2:172.21.15.100:3300/0 conn(0x7f4b8815c9c0 msgr2=0x7f4b880ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 --2- 172.21.15.39:0/2738777068 >> v2:172.21.15.100:3300/0 conn(0x7f4b8815c9c0 0x7f4b880ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 >> v1:172.21.15.100:6789/0 conn(0x7f4b880cf9d0 legacy=0x7f4b880b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b880d0b40 con 0x7f4b8815cda0 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f4b70001670 con 0x7f4b8815cda0 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 >> v1:172.21.15.39:6789/0 conn(0x7f4b8815cda0 legacy=0x7f4b880cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b680039c0 0x7f4b68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b680047c0 0x7f4b68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880d0860 con 0x7f4b68005180 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6800c080 con 0x7f4b680039c0 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6800c240 con 0x7f4b680047c0 2024-01-24T17:45:59.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.673+0000 7f4b87400640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.674+0000 7f4b86a00640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b680039c0 0x7f4b68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.674+0000 7f4b86000640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b680047c0 0x7f4b68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.674+0000 7f4b87400640 1 -- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b680047c0 msgr2=0x7f4b68004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.674+0000 7f4b87400640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b680047c0 0x7f4b68004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.674+0000 7f4b87400640 1 -- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b680039c0 msgr2=0x7f4b68003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.674+0000 7f4b87400640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b680039c0 0x7f4b68003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.674+0000 7f4b87400640 1 -- 172.21.15.39:0/2738777068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68003520 con 0x7f4b68005180 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.674+0000 7f4b86a00640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b680039c0 0x7f4b68003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.675+0000 7f4b87400640 1 -- 172.21.15.39:0/2738777068 reap_dead start 2024-01-24T17:45:59.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.675+0000 7f4b87400640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b6800b980 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f4b80001e20 tx=0x7f4b680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beab93946a207732 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.675+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80003b70 con 0x7f4b68005180 2024-01-24T17:45:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.675+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b80003cd0 con 0x7f4b68005180 2024-01-24T17:45:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.675+0000 7f4b85600640 1 -- 172.21.15.39:0/2738777068 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80005310 con 0x7f4b68005180 2024-01-24T17:45:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.676+0000 7f4b87e00640 1 -- 172.21.15.39:0/2738777068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 msgr2=0x7f4b6800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.676+0000 7f4b87e00640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b6800b980 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f4b80001e20 tx=0x7f4b680057f0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.676+0000 7f4b87e00640 1 -- 172.21.15.39:0/2738777068 shutdown_connections 2024-01-24T17:45:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.676+0000 7f4b87e00640 1 --2- 172.21.15.39:0/2738777068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b6800b980 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.676+0000 7f4b87e00640 1 -- 172.21.15.39:0/2738777068 >> 172.21.15.39:0/2738777068 conn(0x7f4b8808a000 msgr2=0x7f4b880d28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.676+0000 7f4b87e00640 1 -- 172.21.15.39:0/2738777068 shutdown_connections 2024-01-24T17:45:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.676+0000 7f4b87e00640 1 -- 172.21.15.39:0/2738777068 wait complete. 2024-01-24T17:45:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.678+0000 7f4b87e00640 1 Processor -- start 2024-01-24T17:45:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.678+0000 7f4b87e00640 1 -- start start 2024-01-24T17:45:59.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b880ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:59.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b88159720 0x7f4b880cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:59.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b880cc400 0x7f4b8815f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b87e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880ca940 con 0x7f4b68005180 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b87e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880ca680 con 0x7f4b88159720 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b87e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8815e590 con 0x7f4b880cc400 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b86a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b880ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b86a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b880ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45346/0 (socket says 172.21.15.39:45346) 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.679+0000 7f4b86a00640 1 -- 172.21.15.39:0/1893833461 learned_addr learned my addr 172.21.15.39:0/1893833461 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b86a00640 1 -- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b880cc400 msgr2=0x7f4b8815f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b86a00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b880cc400 0x7f4b8815f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b86a00640 1 -- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b88159720 msgr2=0x7f4b880cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b86a00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b88159720 0x7f4b880cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b86a00640 1 -- 172.21.15.39:0/1893833461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8815e870 con 0x7f4b68005180 2024-01-24T17:45:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b86a00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b880ca140 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4b780090e0 tx=0x7f4b78011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:59.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b7ea00640 1 -- 172.21.15.39:0/1893833461 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78005e00 con 0x7f4b68005180 2024-01-24T17:45:59.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b7ea00640 1 -- 172.21.15.39:0/1893833461 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b78004020 con 0x7f4b68005180 2024-01-24T17:45:59.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b7ea00640 1 -- 172.21.15.39:0/1893833461 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b780043d0 con 0x7f4b68005180 2024-01-24T17:45:59.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.680+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8815eb50 con 0x7f4b68005180 2024-01-24T17:45:59.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.681+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8815f110 con 0x7f4b68005180 2024-01-24T17:45:59.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.682+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8815f2d0 con 0x7f4b68005180 2024-01-24T17:45:59.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.683+0000 7f4b7ea00640 1 -- 172.21.15.39:0/1893833461 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4b78003610 con 0x7f4b68005180 2024-01-24T17:45:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.684+0000 7f4b7ea00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b6006a430 0x7f4b6006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:45:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.684+0000 7f4b7ea00640 1 -- 172.21.15.39:0/1893833461 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b78088b80 con 0x7f4b68005180 2024-01-24T17:45:59.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.684+0000 7f4b86000640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b6006a430 0x7f4b6006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:45:59.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.685+0000 7f4b86000640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b6006a430 0x7f4b6006c8f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4b70001050 tx=0x7f4b70000f60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:45:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.689+0000 7f4b7ea00640 1 -- 172.21.15.39:0/1893833461 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4b78054170 con 0x7f4b68005180 2024-01-24T17:45:59.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.922+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4b8804e840 con 0x7f4b68005180 2024-01-24T17:45:59.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.923+0000 7f4b7ea00640 1 -- 172.21.15.39:0/1893833461 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4b78019070 con 0x7f4b68005180 2024-01-24T17:45:59.924 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b6006a430 msgr2=0x7f4b6006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b6006a430 0x7f4b6006c8f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4b70001050 tx=0x7f4b70000f60 comp rx=0 tx=0).stop 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 msgr2=0x7f4b880ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b880ca140 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4b780090e0 tx=0x7f4b78011a60 comp rx=0 tx=0).stop 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 shutdown_connections 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b880cc400 0x7f4b8815f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b88159720 0x7f4b880cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b6006a430 0x7f4b6006c8f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 --2- 172.21.15.39:0/1893833461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b68005180 0x7f4b880ca140 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:45:59.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 >> 172.21.15.39:0/1893833461 conn(0x7f4b8808a000 msgr2=0x7f4b8806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:45:59.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 shutdown_connections 2024-01-24T17:45:59.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:45:59.926+0000 7f4b87e00640 1 -- 172.21.15.39:0/1893833461 wait complete. 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:self.client.name = client.badguy 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /foo/bar 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:45:59.947 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:45:59.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:45:59.948 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:00.004 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42369sec preferred_lft 42369sec 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:46:00.005 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:00.006 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:46:00.006 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:46:00.006 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:46:00.006 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:00.006 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:46:00.006 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:46:00.006 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:46:00.006 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:00.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:00.007 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:46:00.094 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:46:00.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:00.098 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:46:00.151 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:46:00.151 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:46:00.151 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:46:00.151 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:46:00.151 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:46:00.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:00.151 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:46:00.152 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:46:00.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:46:00.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:46:00.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:46:00.152 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:46:00.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:00.328 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:46:00.377 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:46:00.377 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:46:00.378 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:46:00.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:00.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:46:00.458 INFO:tasks.cephfs.fuse_mount:Client client.badguy config is {} 2024-01-24T17:46:00.458 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:46:00.458 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:00.513 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:46:00.513 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:46:00.513 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:00.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:46:00.655 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:00.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:00.747 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:00.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:00.752 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:46:00.806 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:46:00.807 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id badguy --client_mountpoint=/foo/bar) 2024-01-24T17:46:00.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:46:00.893 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:00.894 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:00.961 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:00.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:00.964 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:46:00.998 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:46:00.997+0000 7f7b1100c480 -1 init, newargv = 0x5591e56e29a0 newargc=15 2024-01-24T17:46:00.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[45932]: starting ceph client 2024-01-24T17:46:01.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[45932]: starting fuse 2024-01-24T17:46:02.107 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:46:02.208 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:02.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:02.300 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:02.304 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:02.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:02.305 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:46:02.356 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:46:02.357 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.badguy.*.asok" 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:46:02.357 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:46:02.358 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:46:02.359 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:46:02.359 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:46:02.359 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.badguy")) 2024-01-24T17:46:02.359 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:46:02.530 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.badguy.45932.asok 2024-01-24T17:46:02.712 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.badguy.45932.asok 2024-01-24T17:46:02.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:02.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.badguy.45932.asok status 2024-01-24T17:46:02.952 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "badguy", 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "45932", 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/foo/bar" 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:46:02.953 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 5683, 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5683 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:46:02.954 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2020976969 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 2020976969 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.5683 192.168.0.1:0/2020976969", 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/2020976969", 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 34, 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 63, 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:46:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:46:02.956 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:46:02.956 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:46:02.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:02.967 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:46:02.967 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:03.020 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:46:03.020 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:03.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:03.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:03.115 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:03.115 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:03.168 DEBUG:tasks.cephfs.mount:Creating test file 2024-01-24 17:46:03.168788 badguy 2024-01-24T17:46:03.169 DEBUG:teuthology.orchestra.run.smithi100:> touch '/home/ubuntu/cephtest/mnt.1/2024-01-24 17:46:03.168788 badguy' 2024-01-24T17:46:03.228 DEBUG:tasks.cephfs.mount:Deleting test file 2024-01-24 17:46:03.168788 badguy 2024-01-24T17:46:03.228 DEBUG:teuthology.orchestra.run.smithi100:> rm -f '/home/ubuntu/cephtest/mnt.1/2024-01-24 17:46:03.168788 badguy' 2024-01-24T17:46:03.284 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:03.284 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:03.338 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:03.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.568+0000 7f7a96000640 1 Processor -- start 2024-01-24T17:46:03.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.568+0000 7f7a96000640 1 -- start start 2024-01-24T17:46:03.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.570+0000 7f7a96000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7a9015cd00 0x7f7a900c9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:03.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.570+0000 7f7a96000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7a900b0100 0x7f7a900b0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:03.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.570+0000 7f7a96000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a900ac000 con 0x7f7a900cbc10 2024-01-24T17:46:03.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.570+0000 7f7a96000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a900b0ad0 con 0x7f7a9015cd00 2024-01-24T17:46:03.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.570+0000 7f7a96000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a9015d0e0 con 0x7f7a900b0100 2024-01-24T17:46:03.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.570+0000 7f7a8f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7a9015cd00 0x7f7a900c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:03.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.570+0000 7f7a8f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7a9015cd00 0x7f7a900c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35966/0 (socket says 172.21.15.39:35966) 2024-01-24T17:46:03.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.570+0000 7f7a8f400640 1 -- 172.21.15.39:0/641194915 learned_addr learned my addr 172.21.15.39:0/641194915 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:03.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.571+0000 7f7a8fe00640 1 --2- 172.21.15.39:0/641194915 >> v2:172.21.15.100:3300/0 conn(0x7f7a900b0100 0x7f7a900b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:03.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.571+0000 7f7a8f400640 1 -- 172.21.15.39:0/641194915 >> v2:172.21.15.100:3300/0 conn(0x7f7a900b0100 msgr2=0x7f7a900b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:03.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.571+0000 7f7a8f400640 1 --2- 172.21.15.39:0/641194915 >> v2:172.21.15.100:3300/0 conn(0x7f7a900b0100 0x7f7a900b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.571+0000 7f7a8f400640 1 -- 172.21.15.39:0/641194915 >> v1:172.21.15.39:6789/0 conn(0x7f7a900cbc10 legacy=0x7f7a900ab8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:03.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.571+0000 7f7a8f400640 1 -- 172.21.15.39:0/641194915 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a900cc920 con 0x7f7a9015cd00 2024-01-24T17:46:03.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.573+0000 7f7a8f400640 1 --2- 172.21.15.39:0/641194915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a900c9650 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f7a900c5700 tx=0x7f7a8002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5801be5a56b3f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:03.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.573+0000 7f7a8ea00640 1 -- 172.21.15.39:0/641194915 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80030860 con 0x7f7a9015cd00 2024-01-24T17:46:03.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.573+0000 7f7a8ea00640 1 -- 172.21.15.39:0/641194915 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a800309c0 con 0x7f7a9015cd00 2024-01-24T17:46:03.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.573+0000 7f7a8ea00640 1 -- 172.21.15.39:0/641194915 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80030d50 con 0x7f7a9015cd00 2024-01-24T17:46:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.574+0000 7f7a96000640 1 -- 172.21.15.39:0/641194915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 msgr2=0x7f7a900c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.574+0000 7f7a96000640 1 --2- 172.21.15.39:0/641194915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a900c9650 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f7a900c5700 tx=0x7f7a8002b9a0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.574+0000 7f7a96000640 1 -- 172.21.15.39:0/641194915 shutdown_connections 2024-01-24T17:46:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.574+0000 7f7a96000640 1 --2- 172.21.15.39:0/641194915 >> v2:172.21.15.100:3300/0 conn(0x7f7a900b0100 0x7f7a900b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.574+0000 7f7a96000640 1 --2- 172.21.15.39:0/641194915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a900c9650 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.574+0000 7f7a96000640 1 -- 172.21.15.39:0/641194915 >> 172.21.15.39:0/641194915 conn(0x7f7a90069c70 msgr2=0x7f7a9006a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.574+0000 7f7a96000640 1 -- 172.21.15.39:0/641194915 shutdown_connections 2024-01-24T17:46:03.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.574+0000 7f7a96000640 1 -- 172.21.15.39:0/641194915 wait complete. 2024-01-24T17:46:03.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.576+0000 7f7a96000640 1 Processor -- start 2024-01-24T17:46:03.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.576+0000 7f7a96000640 1 -- start start 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a96000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a900b0100 0x7f7a90150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a96000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a900cbc10 0x7f7a900711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a96000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a90071730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a96000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900c21b0 con 0x7f7a9015cd00 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a96000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900c2490 con 0x7f7a900b0100 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a96000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900c2770 con 0x7f7a900cbc10 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a8fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a90071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a8fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a90071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35970/0 (socket says 172.21.15.39:35970) 2024-01-24T17:46:03.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.577+0000 7f7a8fe00640 1 -- 172.21.15.39:0/153708246 learned_addr learned my addr 172.21.15.39:0/153708246 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a8f400640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a900b0100 0x7f7a90150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a86800640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a900cbc10 0x7f7a900711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a8fe00640 1 -- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a900cbc10 msgr2=0x7f7a900711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a8fe00640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a900cbc10 0x7f7a900711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a8fe00640 1 -- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a900b0100 msgr2=0x7f7a90150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a8fe00640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a900b0100 0x7f7a90150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a8fe00640 1 -- 172.21.15.39:0/153708246 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a800094d0 con 0x7f7a9015cd00 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a86800640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a900cbc10 0x7f7a900711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:03.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a8f400640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a900b0100 0x7f7a90150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:03.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a8fe00640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a90071730 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7a88007e60 tx=0x7f7a8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:03.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a87e00640 1 -- 172.21.15.39:0/153708246 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88014420 con 0x7f7a9015cd00 2024-01-24T17:46:03.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a87e00640 1 -- 172.21.15.39:0/153708246 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a880145a0 con 0x7f7a9015cd00 2024-01-24T17:46:03.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a87e00640 1 -- 172.21.15.39:0/153708246 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88014950 con 0x7f7a9015cd00 2024-01-24T17:46:03.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.578+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a900c2a50 con 0x7f7a9015cd00 2024-01-24T17:46:03.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.579+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a900c6470 con 0x7f7a9015cd00 2024-01-24T17:46:03.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.580+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a900c2f70 con 0x7f7a9015cd00 2024-01-24T17:46:03.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.581+0000 7f7a87e00640 1 -- 172.21.15.39:0/153708246 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7a88014ab0 con 0x7f7a9015cd00 2024-01-24T17:46:03.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.581+0000 7f7a87e00640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a6806a430 0x7f7a6806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:03.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.582+0000 7f7a87e00640 1 -- 172.21.15.39:0/153708246 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a88089240 con 0x7f7a9015cd00 2024-01-24T17:46:03.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.582+0000 7f7a8f400640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a6806a430 0x7f7a6806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:03.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.584+0000 7f7a8f400640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a6806a430 0x7f7a6806c8f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7a80036000 tx=0x7f7a8003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:03.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.587+0000 7f7a87e00640 1 -- 172.21.15.39:0/153708246 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7a900c2f70 con 0x7f7a9015cd00 2024-01-24T17:46:03.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.781+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a900c2f70 con 0x7f7a9015cd00 2024-01-24T17:46:03.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.784+0000 7f7a87e00640 1 -- 172.21.15.39:0/153708246 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f7a900c2f70 con 0x7f7a9015cd00 2024-01-24T17:46:03.785 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:03.786 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:03.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a6806a430 msgr2=0x7f7a6806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:03.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a6806a430 0x7f7a6806c8f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7a80036000 tx=0x7f7a8003b040 comp rx=0 tx=0).stop 2024-01-24T17:46:03.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 msgr2=0x7f7a90071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:03.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a90071730 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7a88007e60 tx=0x7f7a8800e4b0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 shutdown_connections 2024-01-24T17:46:03.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a6806a430 0x7f7a6806c8f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7a9015cd00 0x7f7a90071730 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7a900cbc10 0x7f7a900711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 --2- 172.21.15.39:0/153708246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7a900b0100 0x7f7a90150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:03.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 >> 172.21.15.39:0/153708246 conn(0x7f7a90069c70 msgr2=0x7f7a9015b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:03.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 shutdown_connections 2024-01-24T17:46:03.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:03.787+0000 7f7a96000640 1 -- 172.21.15.39:0/153708246 wait complete. 2024-01-24T17:46:03.807 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:45:16.503215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:45:17.386565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:46:03.807 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:46:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.013+0000 7fca47200640 1 Processor -- start 2024-01-24T17:46:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.014+0000 7fca47200640 1 -- start start 2024-01-24T17:46:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.015+0000 7fca47200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fca480b4340 0x7fca480b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.015+0000 7fca47200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca480cd4f0 0x7fca480cd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.015+0000 7fca47200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4806ad30 con 0x7fca4815cf00 2024-01-24T17:46:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.015+0000 7fca47200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca480b4c60 con 0x7fca480b4340 2024-01-24T17:46:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.015+0000 7fca47200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4806afa0 con 0x7fca480cd4f0 2024-01-24T17:46:04.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.016+0000 7fca45400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fca4815cf00 0x7fca480d0610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:38844/0 (socket says 172.21.15.39:38844) 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.016+0000 7fca45400640 1 -- 172.21.15.39:0/426716843 learned_addr learned my addr 172.21.15.39:0/426716843 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.016+0000 7fca46800640 1 --2- 172.21.15.39:0/426716843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca480cd4f0 0x7fca480cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.016+0000 7fca45e00640 1 --2- 172.21.15.39:0/426716843 >> v2:172.21.15.100:3300/0 conn(0x7fca480b4340 0x7fca480b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.016+0000 7fca44a00640 1 -- 172.21.15.39:0/426716843 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36670281 0 0) 0x7fca4806ad30 con 0x7fca4815cf00 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.016+0000 7fca44a00640 1 -- 172.21.15.39:0/426716843 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca28003520 con 0x7fca4815cf00 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.017+0000 7fca44a00640 1 -- 172.21.15.39:0/426716843 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147624894 0 0) 0x7fca28003520 con 0x7fca4815cf00 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.017+0000 7fca46800640 1 -- 172.21.15.39:0/426716843 >> v2:172.21.15.100:3300/0 conn(0x7fca480b4340 msgr2=0x7fca480b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.017+0000 7fca46800640 1 --2- 172.21.15.39:0/426716843 >> v2:172.21.15.100:3300/0 conn(0x7fca480b4340 0x7fca480b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.017+0000 7fca46800640 1 -- 172.21.15.39:0/426716843 >> v1:172.21.15.39:6789/0 conn(0x7fca4815cf00 legacy=0x7fca480d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.017+0000 7fca46800640 1 -- 172.21.15.39:0/426716843 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4806b280 con 0x7fca480cd4f0 2024-01-24T17:46:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.018+0000 7fca46800640 1 --2- 172.21.15.39:0/426716843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca480cd4f0 0x7fca480cd940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca400097a0 tx=0x7fca4002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b6680ff7fa4b82c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.019+0000 7fca44a00640 1 -- 172.21.15.39:0/426716843 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30001690 con 0x7fca480cd4f0 2024-01-24T17:46:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.019+0000 7fca44a00640 1 -- 172.21.15.39:0/426716843 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4002e050 con 0x7fca480cd4f0 2024-01-24T17:46:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.019+0000 7fca44a00640 1 -- 172.21.15.39:0/426716843 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40030730 con 0x7fca480cd4f0 2024-01-24T17:46:04.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.019+0000 7fca47200640 1 -- 172.21.15.39:0/426716843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca480cd4f0 msgr2=0x7fca480cd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.019+0000 7fca47200640 1 --2- 172.21.15.39:0/426716843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca480cd4f0 0x7fca480cd940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca400097a0 tx=0x7fca4002b790 comp rx=0 tx=0).stop 2024-01-24T17:46:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.020+0000 7fca47200640 1 -- 172.21.15.39:0/426716843 shutdown_connections 2024-01-24T17:46:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.020+0000 7fca47200640 1 --2- 172.21.15.39:0/426716843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca480cd4f0 0x7fca480cd940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.020+0000 7fca47200640 1 --2- 172.21.15.39:0/426716843 >> v2:172.21.15.100:3300/0 conn(0x7fca480b4340 0x7fca480b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.020+0000 7fca47200640 1 -- 172.21.15.39:0/426716843 >> 172.21.15.39:0/426716843 conn(0x7fca480d28c0 msgr2=0x7fca480d2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.020+0000 7fca47200640 1 -- 172.21.15.39:0/426716843 shutdown_connections 2024-01-24T17:46:04.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.020+0000 7fca47200640 1 -- 172.21.15.39:0/426716843 wait complete. 2024-01-24T17:46:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.021+0000 7fca47200640 1 Processor -- start 2024-01-24T17:46:04.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.022+0000 7fca47200640 1 -- start start 2024-01-24T17:46:04.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca47200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca480b4340 0x7fca4815fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca47200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca480cd4f0 0x7fca480ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca47200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4815cf00 0x7fca4815eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca47200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480cb240 con 0x7fca480b4340 2024-01-24T17:46:04.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca47200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca48160260 con 0x7fca480cd4f0 2024-01-24T17:46:04.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca47200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca480ca9a0 con 0x7fca4815cf00 2024-01-24T17:46:04.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca45e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca480b4340 0x7fca4815fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca45e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca480b4340 0x7fca4815fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35978/0 (socket says 172.21.15.39:35978) 2024-01-24T17:46:04.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca45e00640 1 -- 172.21.15.39:0/3904540216 learned_addr learned my addr 172.21.15.39:0/3904540216 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:04.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca45e00640 1 -- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4815cf00 msgr2=0x7fca4815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca45e00640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4815cf00 0x7fca4815eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca45e00640 1 -- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca480cd4f0 msgr2=0x7fca480ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca45e00640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca480cd4f0 0x7fca480ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.023+0000 7fca45e00640 1 -- 172.21.15.39:0/3904540216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca40009400 con 0x7fca480b4340 2024-01-24T17:46:04.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.024+0000 7fca45e00640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca480b4340 0x7fca4815fd20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fca3800efc0 tx=0x7fca3800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.024+0000 7fca3e000640 1 -- 172.21.15.39:0/3904540216 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38014480 con 0x7fca480b4340 2024-01-24T17:46:04.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.024+0000 7fca3e000640 1 -- 172.21.15.39:0/3904540216 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3800f040 con 0x7fca480b4340 2024-01-24T17:46:04.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.024+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca480cac80 con 0x7fca480b4340 2024-01-24T17:46:04.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.024+0000 7fca3e000640 1 -- 172.21.15.39:0/3904540216 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38014800 con 0x7fca480b4340 2024-01-24T17:46:04.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.024+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca481f3a30 con 0x7fca480b4340 2024-01-24T17:46:04.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.025+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca481f3bc0 con 0x7fca480b4340 2024-01-24T17:46:04.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.026+0000 7fca3e000640 1 -- 172.21.15.39:0/3904540216 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fca38014960 con 0x7fca480b4340 2024-01-24T17:46:04.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.027+0000 7fca3e000640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2006a430 0x7fca2006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.027+0000 7fca3e000640 1 -- 172.21.15.39:0/3904540216 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca38089600 con 0x7fca480b4340 2024-01-24T17:46:04.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.028+0000 7fca45400640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2006a430 0x7fca2006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.029+0000 7fca45400640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2006a430 0x7fca2006c8f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fca30001200 tx=0x7fca30001170 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.031+0000 7fca3e000640 1 -- 172.21.15.39:0/3904540216 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fca38054be0 con 0x7fca480b4340 2024-01-24T17:46:04.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.226+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca48073860 con 0x7fca480b4340 2024-01-24T17:46:04.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.227+0000 7fca3e000640 1 -- 172.21.15.39:0/3904540216 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7fca38058200 con 0x7fca480b4340 2024-01-24T17:46:04.228 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2006a430 msgr2=0x7fca2006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2006a430 0x7fca2006c8f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fca30001200 tx=0x7fca30001170 comp rx=0 tx=0).stop 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca480b4340 msgr2=0x7fca4815fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca480b4340 0x7fca4815fd20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fca3800efc0 tx=0x7fca3800c490 comp rx=0 tx=0).stop 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 shutdown_connections 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fca4815cf00 0x7fca4815eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fca480cd4f0 0x7fca480ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fca2006a430 0x7fca2006c8f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 --2- 172.21.15.39:0/3904540216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca480b4340 0x7fca4815fd20 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 >> 172.21.15.39:0/3904540216 conn(0x7fca480d28c0 msgr2=0x7fca480b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:04.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 shutdown_connections 2024-01-24T17:46:04.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.230+0000 7fca47200640 1 -- 172.21.15.39:0/3904540216 wait complete. 2024-01-24T17:46:04.249 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:46:04.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.467+0000 7fed26e00640 1 Processor -- start 2024-01-24T17:46:04.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.467+0000 7fed26e00640 1 -- start start 2024-01-24T17:46:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.469+0000 7fed26e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed200ceb00 0x7fed200708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.469+0000 7fed26e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fed2015ad40 0x7fed2015c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.469+0000 7fed26e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2015c030 con 0x7fed200704d0 2024-01-24T17:46:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.469+0000 7fed26e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2015b1f0 con 0x7fed2015ad40 2024-01-24T17:46:04.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.469+0000 7fed26e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed20155210 con 0x7fed200ceb00 2024-01-24T17:46:04.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.469+0000 7fed25000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed200ceb00 0x7fed200708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.470+0000 7fed26400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fed2015ad40 0x7fed2015c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.470+0000 7fed25a00640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fed200704d0 0x7fed200ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:39304/0 (socket says 172.21.15.39:39304) 2024-01-24T17:46:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.470+0000 7fed25000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed200ceb00 0x7fed200708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:36734/0 (socket says 172.21.15.39:36734) 2024-01-24T17:46:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.470+0000 7fed25a00640 1 -- 172.21.15.39:0/691656490 learned_addr learned my addr 172.21.15.39:0/691656490 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.470+0000 7fed25000640 1 -- 172.21.15.39:0/691656490 >> v2:172.21.15.100:3300/0 conn(0x7fed2015ad40 msgr2=0x7fed2015c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.471+0000 7fed25000640 1 --2- 172.21.15.39:0/691656490 >> v2:172.21.15.100:3300/0 conn(0x7fed2015ad40 0x7fed2015c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.471+0000 7fed25000640 1 -- 172.21.15.39:0/691656490 >> v1:172.21.15.100:6789/0 conn(0x7fed200704d0 legacy=0x7fed200ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.471+0000 7fed25000640 1 -- 172.21.15.39:0/691656490 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed200b4110 con 0x7fed200ceb00 2024-01-24T17:46:04.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.472+0000 7fed25000640 1 --2- 172.21.15.39:0/691656490 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed200ceb00 0x7fed200708b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed08009d90 tx=0x7fed0802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46a55e3a8b067b79 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.472+0000 7fed1fe00640 1 -- 172.21.15.39:0/691656490 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0802cc60 con 0x7fed200ceb00 2024-01-24T17:46:04.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.473+0000 7fed1fe00640 1 -- 172.21.15.39:0/691656490 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0802cdc0 con 0x7fed200ceb00 2024-01-24T17:46:04.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.473+0000 7fed1fe00640 1 -- 172.21.15.39:0/691656490 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08034660 con 0x7fed200ceb00 2024-01-24T17:46:04.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.473+0000 7fed26e00640 1 -- 172.21.15.39:0/691656490 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed200ceb00 msgr2=0x7fed200708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.473+0000 7fed26e00640 1 --2- 172.21.15.39:0/691656490 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed200ceb00 0x7fed200708b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fed08009d90 tx=0x7fed0802beb0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.473+0000 7fed26e00640 1 -- 172.21.15.39:0/691656490 shutdown_connections 2024-01-24T17:46:04.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.473+0000 7fed26e00640 1 --2- 172.21.15.39:0/691656490 >> v2:172.21.15.100:3300/0 conn(0x7fed2015ad40 0x7fed2015c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.473+0000 7fed26e00640 1 --2- 172.21.15.39:0/691656490 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed200ceb00 0x7fed200708b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.473+0000 7fed26e00640 1 -- 172.21.15.39:0/691656490 >> 172.21.15.39:0/691656490 conn(0x7fed2006a2a0 msgr2=0x7fed2006beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:04.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.474+0000 7fed26e00640 1 -- 172.21.15.39:0/691656490 shutdown_connections 2024-01-24T17:46:04.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.474+0000 7fed26e00640 1 -- 172.21.15.39:0/691656490 wait complete. 2024-01-24T17:46:04.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.475+0000 7fed26e00640 1 Processor -- start 2024-01-24T17:46:04.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.475+0000 7fed26e00640 1 -- start start 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.476+0000 7fed26e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed200704d0 0x7fed201591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.476+0000 7fed26e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed200ceb00 0x7fed2015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.476+0000 7fed26e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed2015ad40 0x7fed2015d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.476+0000 7fed26e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed201596e0 con 0x7fed200ceb00 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.476+0000 7fed26e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed20151be0 con 0x7fed200704d0 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.476+0000 7fed26e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed20157620 con 0x7fed2015ad40 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed200ceb00 0x7fed2015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed200ceb00 0x7fed2015d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35986/0 (socket says 172.21.15.39:35986) 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25000640 1 -- 172.21.15.39:0/3046597591 learned_addr learned my addr 172.21.15.39:0/3046597591 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25a00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed200704d0 0x7fed201591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed26400640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed2015ad40 0x7fed2015d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25000640 1 -- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed2015ad40 msgr2=0x7fed2015d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25000640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed2015ad40 0x7fed2015d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25000640 1 -- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed200704d0 msgr2=0x7fed201591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25000640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed200704d0 0x7fed201591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.477+0000 7fed25000640 1 -- 172.21.15.39:0/3046597591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08009a70 con 0x7fed200ceb00 2024-01-24T17:46:04.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.478+0000 7fed25000640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed200ceb00 0x7fed2015d410 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fed08009d90 tx=0x7fed08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.478+0000 7fed1d600640 1 -- 172.21.15.39:0/3046597591 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08040650 con 0x7fed200ceb00 2024-01-24T17:46:04.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.478+0000 7fed1d600640 1 -- 172.21.15.39:0/3046597591 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed080407b0 con 0x7fed200ceb00 2024-01-24T17:46:04.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.478+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed20155480 con 0x7fed200ceb00 2024-01-24T17:46:04.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.478+0000 7fed1d600640 1 -- 172.21.15.39:0/3046597591 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08040b60 con 0x7fed200ceb00 2024-01-24T17:46:04.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.478+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed20155a40 con 0x7fed200ceb00 2024-01-24T17:46:04.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.479+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2004e840 con 0x7fed200ceb00 2024-01-24T17:46:04.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.480+0000 7fed1d600640 1 -- 172.21.15.39:0/3046597591 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fed08040cc0 con 0x7fed200ceb00 2024-01-24T17:46:04.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.481+0000 7fed1d600640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fecf806a430 0x7fecf806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.481+0000 7fed1d600640 1 -- 172.21.15.39:0/3046597591 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed080ab7b0 con 0x7fed200ceb00 2024-01-24T17:46:04.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.482+0000 7fed25a00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fecf806a430 0x7fecf806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.483+0000 7fed25a00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fecf806a430 0x7fecf806c8f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fed10005880 tx=0x7fed10009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.486+0000 7fed1d600640 1 -- 172.21.15.39:0/3046597591 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fed08076d90 con 0x7fed200ceb00 2024-01-24T17:46:04.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.678+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed2006cca0 con 0x7fed200ceb00 2024-01-24T17:46:04.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.679+0000 7fed1d600640 1 -- 172.21.15.39:0/3046597591 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v63) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed0807a3b0 con 0x7fed200ceb00 2024-01-24T17:46:04.679 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:46:04.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fecf806a430 msgr2=0x7fecf806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fecf806a430 0x7fecf806c8f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fed10005880 tx=0x7fed10009040 comp rx=0 tx=0).stop 2024-01-24T17:46:04.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed200ceb00 msgr2=0x7fed2015d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed200ceb00 0x7fed2015d410 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fed08009d90 tx=0x7fed08004a40 comp rx=0 tx=0).stop 2024-01-24T17:46:04.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 shutdown_connections 2024-01-24T17:46:04.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fed2015ad40 0x7fed2015d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fecf806a430 0x7fecf806c8f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed200ceb00 0x7fed2015d410 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 --2- 172.21.15.39:0/3046597591 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed200704d0 0x7fed201591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 >> 172.21.15.39:0/3046597591 conn(0x7fed2006a2a0 msgr2=0x7fed200af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:04.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 shutdown_connections 2024-01-24T17:46:04.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.681+0000 7fed26e00640 1 -- 172.21.15.39:0/3046597591 wait complete. 2024-01-24T17:46:04.702 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:46:04.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.925+0000 7fa29d200640 1 Processor -- start 2024-01-24T17:46:04.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.925+0000 7fa29d200640 1 -- start start 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.926+0000 7fa29d200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa2981505f0 0x7fa2980719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.926+0000 7fa29d200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa298071ef0 0x7fa298159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa29d200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa29815a0e0 0x7fa2980cc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa29d200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa298070f30 con 0x7fa298071ef0 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa29d200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29815a660 con 0x7fa2981505f0 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa29d200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29815a940 con 0x7fa29815a0e0 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa28fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa298071ef0 0x7fa298159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa28fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa298071ef0 0x7fa298159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36002/0 (socket says 172.21.15.39:36002) 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa297400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa2981505f0 0x7fa2980719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa28fe00640 1 -- 172.21.15.39:0/442151209 learned_addr learned my addr 172.21.15.39:0/442151209 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:04.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.927+0000 7fa297e00640 1 --2- 172.21.15.39:0/442151209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa29815a0e0 0x7fa2980cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.928+0000 7fa28fe00640 1 -- 172.21.15.39:0/442151209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa29815a0e0 msgr2=0x7fa2980cc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:04.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.928+0000 7fa28fe00640 1 --2- 172.21.15.39:0/442151209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa29815a0e0 0x7fa2980cc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.928+0000 7fa28fe00640 1 -- 172.21.15.39:0/442151209 >> v2:172.21.15.100:3300/0 conn(0x7fa2981505f0 msgr2=0x7fa2980719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:04.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.928+0000 7fa28fe00640 1 --2- 172.21.15.39:0/442151209 >> v2:172.21.15.100:3300/0 conn(0x7fa2981505f0 0x7fa2980719b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.928+0000 7fa28fe00640 1 -- 172.21.15.39:0/442151209 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2980711b0 con 0x7fa298071ef0 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.929+0000 7fa28fe00640 1 --2- 172.21.15.39:0/442151209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa298159ba0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa2980cbe70 tx=0x7fa28002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c46b90ebd61c39 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa296a00640 1 -- 172.21.15.39:0/442151209 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28002ee90 con 0x7fa298071ef0 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa296a00640 1 -- 172.21.15.39:0/442151209 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa280036440 con 0x7fa298071ef0 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa296a00640 1 -- 172.21.15.39:0/442151209 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2800367d0 con 0x7fa298071ef0 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa29d200640 1 -- 172.21.15.39:0/442151209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 msgr2=0x7fa298159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa29d200640 1 --2- 172.21.15.39:0/442151209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa298159ba0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa2980cbe70 tx=0x7fa28002e0e0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa29d200640 1 -- 172.21.15.39:0/442151209 shutdown_connections 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa29d200640 1 --2- 172.21.15.39:0/442151209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa29815a0e0 0x7fa2980cc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa29d200640 1 --2- 172.21.15.39:0/442151209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa298159ba0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa29d200640 1 --2- 172.21.15.39:0/442151209 >> v2:172.21.15.100:3300/0 conn(0x7fa2981505f0 0x7fa2980719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.930+0000 7fa29d200640 1 -- 172.21.15.39:0/442151209 >> 172.21.15.39:0/442151209 conn(0x7fa29806a2a0 msgr2=0x7fa29806beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:04.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.931+0000 7fa29d200640 1 -- 172.21.15.39:0/442151209 shutdown_connections 2024-01-24T17:46:04.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.931+0000 7fa29d200640 1 -- 172.21.15.39:0/442151209 wait complete. 2024-01-24T17:46:04.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.932+0000 7fa29d200640 1 Processor -- start 2024-01-24T17:46:04.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.933+0000 7fa29d200640 1 -- start start 2024-01-24T17:46:04.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.933+0000 7fa29d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa2981f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.933+0000 7fa29d200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2981505f0 0x7fa2980caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.933+0000 7fa29d200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa29815a0e0 0x7fa29814c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.933+0000 7fa29d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2980ca5f0 con 0x7fa298071ef0 2024-01-24T17:46:04.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa29d200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa298157190 con 0x7fa29815a0e0 2024-01-24T17:46:04.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa29d200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2981f6870 con 0x7fa2981505f0 2024-01-24T17:46:04.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa2981f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa2981f6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36004/0 (socket says 172.21.15.39:36004) 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297400640 1 -- 172.21.15.39:0/385076584 learned_addr learned my addr 172.21.15.39:0/385076584 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa28fe00640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2981505f0 0x7fa2980caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297e00640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa29815a0e0 0x7fa29814c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297400640 1 -- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2981505f0 msgr2=0x7fa2980caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297400640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2981505f0 0x7fa2980caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297400640 1 -- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa29815a0e0 msgr2=0x7fa29814c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297400640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa29815a0e0 0x7fa29814c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.934+0000 7fa297400640 1 -- 172.21.15.39:0/385076584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28000b770 con 0x7fa298071ef0 2024-01-24T17:46:04.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.935+0000 7fa297400640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa2981f6330 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa28800e890 tx=0x7fa28800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.935+0000 7fa28f400640 1 -- 172.21.15.39:0/385076584 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800c940 con 0x7fa298071ef0 2024-01-24T17:46:04.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.935+0000 7fa28f400640 1 -- 172.21.15.39:0/385076584 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28800caa0 con 0x7fa298071ef0 2024-01-24T17:46:04.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.935+0000 7fa28f400640 1 -- 172.21.15.39:0/385076584 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800ce30 con 0x7fa298071ef0 2024-01-24T17:46:04.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.935+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2980c99e0 con 0x7fa298071ef0 2024-01-24T17:46:04.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.935+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2980c9fa0 con 0x7fa298071ef0 2024-01-24T17:46:04.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.936+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29804e840 con 0x7fa298071ef0 2024-01-24T17:46:04.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.937+0000 7fa28f400640 1 -- 172.21.15.39:0/385076584 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa288013470 con 0x7fa298071ef0 2024-01-24T17:46:04.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.938+0000 7fa28f400640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa27006a460 0x7fa27006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:04.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.938+0000 7fa28f400640 1 -- 172.21.15.39:0/385076584 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa288088bf0 con 0x7fa298071ef0 2024-01-24T17:46:04.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.939+0000 7fa28fe00640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa27006a460 0x7fa27006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:04.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.940+0000 7fa28fe00640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa27006a460 0x7fa27006c920 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa280038000 tx=0x7fa280008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:04.943+0000 7fa28f400640 1 -- 172.21.15.39:0/385076584 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa288054140 con 0x7fa298071ef0 2024-01-24T17:46:05.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.136+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa2980cd060 con 0x7fa298071ef0 2024-01-24T17:46:05.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.137+0000 7fa28f400640 1 -- 172.21.15.39:0/385076584 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v63) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa2880170a0 con 0x7fa298071ef0 2024-01-24T17:46:05.137 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:46:05.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa27006a460 msgr2=0x7fa27006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa27006a460 0x7fa27006c920 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa280038000 tx=0x7fa280008040 comp rx=0 tx=0).stop 2024-01-24T17:46:05.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 msgr2=0x7fa2981f6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa2981f6330 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa28800e890 tx=0x7fa28800ed50 comp rx=0 tx=0).stop 2024-01-24T17:46:05.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 shutdown_connections 2024-01-24T17:46:05.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa29815a0e0 0x7fa29814c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa2981505f0 0x7fa2980caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa27006a460 0x7fa27006c920 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 --2- 172.21.15.39:0/385076584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa298071ef0 0x7fa2981f6330 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.139+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 >> 172.21.15.39:0/385076584 conn(0x7fa29806a2a0 msgr2=0x7fa2980c4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:05.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.140+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 shutdown_connections 2024-01-24T17:46:05.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.140+0000 7fa29d200640 1 -- 172.21.15.39:0/385076584 wait complete. 2024-01-24T17:46:05.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:46:05.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.379+0000 7f7464a00640 1 Processor -- start 2024-01-24T17:46:05.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.379+0000 7f7464a00640 1 -- start start 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.380+0000 7f7464a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f745c0b43b0 0x7f745c0b0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7464a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 0x7f745c0700a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7464a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c06df20 con 0x7f745c15a9a0 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7464a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745c0b0b50 con 0x7f745c0b43b0 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7464a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c15e610 con 0x7f745c0b06a0 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7461600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f745c0b43b0 0x7f745c0b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7461600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f745c0b43b0 0x7f745c0b0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36006/0 (socket says 172.21.15.39:36006) 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7461600640 1 -- 172.21.15.39:0/3083282759 learned_addr learned my addr 172.21.15.39:0/3083282759 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:05.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7462a00640 1 --2- 172.21.15.39:0/3083282759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 0x7f745c0700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7461600640 1 -- 172.21.15.39:0/3083282759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 msgr2=0x7f745c0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:05.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7461600640 1 --2- 172.21.15.39:0/3083282759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 0x7f745c0700a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7461600640 1 -- 172.21.15.39:0/3083282759 >> v1:172.21.15.100:6789/0 conn(0x7f745c15a9a0 legacy=0x7f745c06d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:05.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.381+0000 7f7461600640 1 -- 172.21.15.39:0/3083282759 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c15e8f0 con 0x7f745c0b43b0 2024-01-24T17:46:05.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.383+0000 7f7461600640 1 --2- 172.21.15.39:0/3083282759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c0b0160 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7444009870 tx=0x7f744402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35664448c4ace72 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:05.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.383+0000 7f7460c00640 1 -- 172.21.15.39:0/3083282759 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7444031640 con 0x7f745c0b43b0 2024-01-24T17:46:05.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.383+0000 7f7460c00640 1 -- 172.21.15.39:0/3083282759 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74440317a0 con 0x7f745c0b43b0 2024-01-24T17:46:05.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.383+0000 7f7460c00640 1 -- 172.21.15.39:0/3083282759 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7444031b30 con 0x7f745c0b43b0 2024-01-24T17:46:05.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.384+0000 7f7464a00640 1 -- 172.21.15.39:0/3083282759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 msgr2=0x7f745c0b0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.384+0000 7f7464a00640 1 --2- 172.21.15.39:0/3083282759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c0b0160 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7444009870 tx=0x7f744402b8f0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.384+0000 7f7464a00640 1 -- 172.21.15.39:0/3083282759 shutdown_connections 2024-01-24T17:46:05.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.384+0000 7f7464a00640 1 --2- 172.21.15.39:0/3083282759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c0b0160 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.384+0000 7f7464a00640 1 --2- 172.21.15.39:0/3083282759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 0x7f745c0700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.384+0000 7f7464a00640 1 -- 172.21.15.39:0/3083282759 >> 172.21.15.39:0/3083282759 conn(0x7f745c0cb160 msgr2=0x7f745c0cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:05.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.384+0000 7f7464a00640 1 -- 172.21.15.39:0/3083282759 shutdown_connections 2024-01-24T17:46:05.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.384+0000 7f7464a00640 1 -- 172.21.15.39:0/3083282759 wait complete. 2024-01-24T17:46:05.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.386+0000 7f7464a00640 1 Processor -- start 2024-01-24T17:46:05.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.386+0000 7f7464a00640 1 -- start start 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.387+0000 7f7464a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 0x7f745c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.387+0000 7f7464a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c15d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.387+0000 7f7464a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c15a9a0 0x7f745c151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.387+0000 7f7464a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c15cc70 con 0x7f745c0b43b0 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.387+0000 7f7464a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c15c640 con 0x7f745c15a9a0 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.387+0000 7f7464a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c1521e0 con 0x7f745c0b06a0 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7461600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c15d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7461600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c15d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36010/0 (socket says 172.21.15.39:36010) 2024-01-24T17:46:05.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7461600640 1 -- 172.21.15.39:0/951122303 learned_addr learned my addr 172.21.15.39:0/951122303 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7462a00640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c15a9a0 0x7f745c151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7462000640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 0x7f745c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7461600640 1 -- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 msgr2=0x7f745c0ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7461600640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 0x7f745c0ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7461600640 1 -- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c15a9a0 msgr2=0x7f745c151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7461600640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c15a9a0 0x7f745c151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.388+0000 7f7461600640 1 -- 172.21.15.39:0/951122303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74440094d0 con 0x7f745c0b43b0 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.389+0000 7f7461600640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c15d090 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f7444000c00 tx=0x7f74440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.389+0000 7f745a000640 1 -- 172.21.15.39:0/951122303 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744403ac60 con 0x7f745c0b43b0 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.389+0000 7f745a000640 1 -- 172.21.15.39:0/951122303 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744403adc0 con 0x7f745c0b43b0 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.389+0000 7f745a000640 1 -- 172.21.15.39:0/951122303 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7444044630 con 0x7f745c0b43b0 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.389+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c1524c0 con 0x7f745c0b43b0 2024-01-24T17:46:05.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.389+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c150a90 con 0x7f745c0b43b0 2024-01-24T17:46:05.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.390+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745c150c80 con 0x7f745c0b43b0 2024-01-24T17:46:05.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.391+0000 7f745a000640 1 -- 172.21.15.39:0/951122303 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f744402f050 con 0x7f745c0b43b0 2024-01-24T17:46:05.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.392+0000 7f745a000640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f743406a430 0x7f743406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.392+0000 7f745a000640 1 -- 172.21.15.39:0/951122303 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74440aa840 con 0x7f745c0b43b0 2024-01-24T17:46:05.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.393+0000 7f7462000640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f743406a430 0x7f743406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.394+0000 7f7462000640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f743406a430 0x7f743406c8f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f744c006fd0 tx=0x7f744c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:05.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.397+0000 7f745a000640 1 -- 172.21.15.39:0/951122303 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7444075e20 con 0x7f745c0b43b0 2024-01-24T17:46:05.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.611+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f745c152650 con 0x7f745c0b43b0 2024-01-24T17:46:05.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.612+0000 7f745a000640 1 -- 172.21.15.39:0/951122303 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v63) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7444079440 con 0x7f745c0b43b0 2024-01-24T17:46:05.612 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:46:05.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f743406a430 msgr2=0x7f743406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f743406a430 0x7f743406c8f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f744c006fd0 tx=0x7f744c008040 comp rx=0 tx=0).stop 2024-01-24T17:46:05.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 msgr2=0x7f745c15d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c15d090 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f7444000c00 tx=0x7f74440046c0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 shutdown_connections 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f745c15a9a0 0x7f745c151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f743406a430 0x7f743406c8f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f745c0b43b0 0x7f745c15d090 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 --2- 172.21.15.39:0/951122303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f745c0b06a0 0x7f745c0ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.614+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 >> 172.21.15.39:0/951122303 conn(0x7f745c0cb160 msgr2=0x7f745c158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.615+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 shutdown_connections 2024-01-24T17:46:05.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.615+0000 7f7464a00640 1 -- 172.21.15.39:0/951122303 wait complete. 2024-01-24T17:46:05.635 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:05.635 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:05.694 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:05.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.925+0000 7efc7ca00640 1 Processor -- start 2024-01-24T17:46:05.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.925+0000 7efc7ca00640 1 -- start start 2024-01-24T17:46:05.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc7ca00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efc7415ceb0 0x7efc740cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc7ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efc740cdda0 0x7efc740d1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc7ca00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc741617e0 con 0x7efc7415cad0 2024-01-24T17:46:05.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc7ca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc740b5cc0 con 0x7efc740cdda0 2024-01-24T17:46:05.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc7ca00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc740ce1f0 con 0x7efc7415ceb0 2024-01-24T17:46:05.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc7aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efc740cdda0 0x7efc740d1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc7aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efc740cdda0 0x7efc740d1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36016/0 (socket says 172.21.15.39:36016) 2024-01-24T17:46:05.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc7aa00640 1 -- 172.21.15.39:0/3200743108 learned_addr learned my addr 172.21.15.39:0/3200743108 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:05.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.927+0000 7efc79600640 1 --2- 172.21.15.39:0/3200743108 >> v2:172.21.15.100:3300/0 conn(0x7efc7415ceb0 0x7efc740cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.928+0000 7efc7aa00640 1 -- 172.21.15.39:0/3200743108 >> v2:172.21.15.100:3300/0 conn(0x7efc7415ceb0 msgr2=0x7efc740cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:05.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.928+0000 7efc7aa00640 1 --2- 172.21.15.39:0/3200743108 >> v2:172.21.15.100:3300/0 conn(0x7efc7415ceb0 0x7efc740cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.928+0000 7efc7aa00640 1 -- 172.21.15.39:0/3200743108 >> v1:172.21.15.100:6789/0 conn(0x7efc7415cad0 legacy=0x7efc74160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:05.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.928+0000 7efc7aa00640 1 -- 172.21.15.39:0/3200743108 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74161220 con 0x7efc740cdda0 2024-01-24T17:46:05.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.929+0000 7efc7aa00640 1 --2- 172.21.15.39:0/3200743108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740cdda0 0x7efc740d1620 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7efc740d3c90 tx=0x7efc6c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78fa593d1440662 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:05.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.929+0000 7efc78c00640 1 -- 172.21.15.39:0/3200743108 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c02ca80 con 0x7efc740cdda0 2024-01-24T17:46:05.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.929+0000 7efc78c00640 1 -- 172.21.15.39:0/3200743108 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c02cbe0 con 0x7efc740cdda0 2024-01-24T17:46:05.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.929+0000 7efc78c00640 1 -- 172.21.15.39:0/3200743108 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0343f0 con 0x7efc740cdda0 2024-01-24T17:46:05.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.930+0000 7efc7ca00640 1 -- 172.21.15.39:0/3200743108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740cdda0 msgr2=0x7efc740d1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.930+0000 7efc7ca00640 1 --2- 172.21.15.39:0/3200743108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740cdda0 0x7efc740d1620 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7efc740d3c90 tx=0x7efc6c02bee0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.930+0000 7efc7ca00640 1 -- 172.21.15.39:0/3200743108 shutdown_connections 2024-01-24T17:46:05.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.930+0000 7efc7ca00640 1 --2- 172.21.15.39:0/3200743108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc740cdda0 0x7efc740d1620 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.930+0000 7efc7ca00640 1 --2- 172.21.15.39:0/3200743108 >> v2:172.21.15.100:3300/0 conn(0x7efc7415ceb0 0x7efc740cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.930+0000 7efc7ca00640 1 -- 172.21.15.39:0/3200743108 >> 172.21.15.39:0/3200743108 conn(0x7efc7406af20 msgr2=0x7efc7406b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:05.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.930+0000 7efc7ca00640 1 -- 172.21.15.39:0/3200743108 shutdown_connections 2024-01-24T17:46:05.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.930+0000 7efc7ca00640 1 -- 172.21.15.39:0/3200743108 wait complete. 2024-01-24T17:46:05.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.932+0000 7efc7ca00640 1 Processor -- start 2024-01-24T17:46:05.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.932+0000 7efc7ca00640 1 -- start start 2024-01-24T17:46:05.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7ca00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc740cdda0 0x7efc74151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7ca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc7415cad0 0x7efc740c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7415ceb0 0x7efc740c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74151550 con 0x7efc7415ceb0 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7ca00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740c70d0 con 0x7efc740cdda0 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7ca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc740c7f30 con 0x7efc7415cad0 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7415ceb0 0x7efc740c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7415ceb0 0x7efc740c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36022/0 (socket says 172.21.15.39:36022) 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.933+0000 7efc7aa00640 1 -- 172.21.15.39:0/2923693219 learned_addr learned my addr 172.21.15.39:0/2923693219 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.934+0000 7efc7a000640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc740cdda0 0x7efc74151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.934+0000 7efc79600640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc7415cad0 0x7efc740c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.934+0000 7efc7aa00640 1 -- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc7415cad0 msgr2=0x7efc740c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.934+0000 7efc7aa00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc7415cad0 0x7efc740c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.934+0000 7efc7aa00640 1 -- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc740cdda0 msgr2=0x7efc74151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.934+0000 7efc7aa00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc740cdda0 0x7efc74151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:05.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.934+0000 7efc7aa00640 1 -- 172.21.15.39:0/2923693219 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c009b30 con 0x7efc7415ceb0 2024-01-24T17:46:05.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.934+0000 7efc7aa00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7415ceb0 0x7efc740c8ca0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7efc6c036040 tx=0x7efc6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:05.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.935+0000 7efc72000640 1 -- 172.21.15.39:0/2923693219 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c009510 con 0x7efc7415ceb0 2024-01-24T17:46:05.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.935+0000 7efc72000640 1 -- 172.21.15.39:0/2923693219 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c03e450 con 0x7efc7415ceb0 2024-01-24T17:46:05.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.935+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc740c7320 con 0x7efc7415ceb0 2024-01-24T17:46:05.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.935+0000 7efc72000640 1 -- 172.21.15.39:0/2923693219 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c03e7e0 con 0x7efc7415ceb0 2024-01-24T17:46:05.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.935+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc740c78e0 con 0x7efc7415ceb0 2024-01-24T17:46:05.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.936+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc740c80e0 con 0x7efc7415ceb0 2024-01-24T17:46:05.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.937+0000 7efc72000640 1 -- 172.21.15.39:0/2923693219 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efc6c02cd50 con 0x7efc7415ceb0 2024-01-24T17:46:05.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.938+0000 7efc72000640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a460 0x7efc4c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:05.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.938+0000 7efc72000640 1 -- 172.21.15.39:0/2923693219 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc6c0ab5a0 con 0x7efc7415ceb0 2024-01-24T17:46:05.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.939+0000 7efc7a000640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a460 0x7efc4c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:05.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.940+0000 7efc7a000640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a460 0x7efc4c06c920 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efc64004ea0 tx=0x7efc64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:05.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:05.943+0000 7efc72000640 1 -- 172.21.15.39:0/2923693219 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efc740c80e0 con 0x7efc7415ceb0 2024-01-24T17:46:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.140+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc741617e0 con 0x7efc7415ceb0 2024-01-24T17:46:06.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.143+0000 7efc72000640 1 -- 172.21.15.39:0/2923693219 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7efc741617e0 con 0x7efc7415ceb0 2024-01-24T17:46:06.144 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:06.144 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a460 msgr2=0x7efc4c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a460 0x7efc4c06c920 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efc64004ea0 tx=0x7efc64009040 comp rx=0 tx=0).stop 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7415ceb0 msgr2=0x7efc740c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7415ceb0 0x7efc740c8ca0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7efc6c036040 tx=0x7efc6c004a40 comp rx=0 tx=0).stop 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 shutdown_connections 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc4c06a460 0x7efc4c06c920 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc7415ceb0 0x7efc740c8ca0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efc7415cad0 0x7efc740c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 --2- 172.21.15.39:0/2923693219 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efc740cdda0 0x7efc74151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 >> 172.21.15.39:0/2923693219 conn(0x7efc7406af20 msgr2=0x7efc740b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:06.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 shutdown_connections 2024-01-24T17:46:06.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.146+0000 7efc7ca00640 1 -- 172.21.15.39:0/2923693219 wait complete. 2024-01-24T17:46:06.165 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:45:16.503215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:45:17.386565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:46:06.165 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:46:06.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.384+0000 7f5149800640 1 Processor -- start 2024-01-24T17:46:06.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.385+0000 7f5149800640 1 -- start start 2024-01-24T17:46:06.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.386+0000 7f5149800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51440cdf70 0x7f5144167bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.386+0000 7f5149800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51441617e0 con 0x7f514415ceb0 2024-01-24T17:46:06.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.387+0000 7f5149800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5144161220 con 0x7f514415cad0 2024-01-24T17:46:06.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.387+0000 7f5149800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51440b5cc0 con 0x7f51440cdf70 2024-01-24T17:46:06.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.387+0000 7f5142000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f514415ceb0 0x7f51440cd860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:38852/0 (socket says 172.21.15.39:38852) 2024-01-24T17:46:06.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.387+0000 7f5142000640 1 -- 172.21.15.39:0/2047505338 learned_addr learned my addr 172.21.15.39:0/2047505338 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:06.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.387+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461709627 0 0) 0x7f5134001490 con 0x7f514415ceb0 2024-01-24T17:46:06.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.387+0000 7f5143400640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51440cdf70 0x7f5144167bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.387+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5124003520 con 0x7f514415ceb0 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019116639 0 0) 0x7f5124003520 con 0x7f514415ceb0 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51440cdf70 msgr2=0x7f5144167bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51440cdf70 0x7f5144167bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 >> v1:172.21.15.100:6789/0 conn(0x7f514415cad0 legacy=0x7f5144160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5144161500 con 0x7f514415ceb0 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f5134001b80 con 0x7f514415ceb0 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 >> v1:172.21.15.39:6789/0 conn(0x7f514415ceb0 legacy=0x7f51440cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5124003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51240047c0 0x7f5124004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51440cdf70 0x7f5124005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51440b5cc0 con 0x7f51240039a0 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512400ba60 con 0x7f51240047c0 2024-01-24T17:46:06.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512400bc20 con 0x7f51440cdf70 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5142a00640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5124003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.388+0000 7f5143400640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51440cdf70 0x7f5124005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.389+0000 7f5142000640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51240047c0 0x7f5124004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.389+0000 7f5142a00640 1 -- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51440cdf70 msgr2=0x7f5124005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.389+0000 7f5142a00640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51440cdf70 0x7f5124005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.389+0000 7f5142a00640 1 -- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51240047c0 msgr2=0x7f5124004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.389+0000 7f5142a00640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51240047c0 0x7f5124004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.389+0000 7f5142a00640 1 -- 172.21.15.39:0/2047505338 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5124003520 con 0x7f51240039a0 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.389+0000 7f5142000640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51240047c0 0x7f5124004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.390+0000 7f5142a00640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5124003dc0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f512c001720 tx=0x7f51240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad039b115b678638 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.390+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c0052d0 con 0x7f51240039a0 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.390+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c02e050 con 0x7f51240039a0 2024-01-24T17:46:06.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.390+0000 7f5141600640 1 -- 172.21.15.39:0/2047505338 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c0035f0 con 0x7f51240039a0 2024-01-24T17:46:06.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.391+0000 7f5149800640 1 -- 172.21.15.39:0/2047505338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 msgr2=0x7f5124003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.391+0000 7f5149800640 1 --2- 172.21.15.39:0/2047505338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5124003dc0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f512c001720 tx=0x7f51240057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.391+0000 7f5143400640 1 -- 172.21.15.39:0/2047505338 reap_dead start 2024-01-24T17:46:06.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.391+0000 7f5149800640 1 -- 172.21.15.39:0/2047505338 shutdown_connections 2024-01-24T17:46:06.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.391+0000 7f5149800640 1 -- 172.21.15.39:0/2047505338 >> 172.21.15.39:0/2047505338 conn(0x7f514406af20 msgr2=0x7f514406b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:06.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.391+0000 7f5149800640 1 -- 172.21.15.39:0/2047505338 shutdown_connections 2024-01-24T17:46:06.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.391+0000 7f5149800640 1 -- 172.21.15.39:0/2047505338 wait complete. 2024-01-24T17:46:06.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.393+0000 7f5149800640 1 Processor -- start 2024-01-24T17:46:06.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.393+0000 7f5149800640 1 -- start start 2024-01-24T17:46:06.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5149800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5144154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5149800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5144152000 0x7f51440b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5149800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51440b73c0 0x7f51441528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5149800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51440b7840 con 0x7f51240039a0 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5149800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5144152480 con 0x7f51440b73c0 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5149800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51440c8470 con 0x7f5144152000 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5144154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5144154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36026/0 (socket says 172.21.15.39:36026) 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142a00640 1 -- 172.21.15.39:0/116368836 learned_addr learned my addr 172.21.15.39:0/116368836 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5143400640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51440b73c0 0x7f51441528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142a00640 1 -- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5144152000 msgr2=0x7f51440b6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142000640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5144152000 0x7f51440b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142a00640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5144152000 0x7f51440b6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142a00640 1 -- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51440b73c0 msgr2=0x7f51441528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142a00640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51440b73c0 0x7f51441528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5142a00640 1 -- 172.21.15.39:0/116368836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51440c8750 con 0x7f51240039a0 2024-01-24T17:46:06.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.394+0000 7f5143400640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51440b73c0 0x7f51441528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:06.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.395+0000 7f5142a00640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5144154610 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f512c008fd0 tx=0x7f512c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:06.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.395+0000 7f513aa00640 1 -- 172.21.15.39:0/116368836 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c0076c0 con 0x7f51240039a0 2024-01-24T17:46:06.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.395+0000 7f513aa00640 1 -- 172.21.15.39:0/116368836 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c007a30 con 0x7f51240039a0 2024-01-24T17:46:06.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.395+0000 7f513aa00640 1 -- 172.21.15.39:0/116368836 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c006be0 con 0x7f51240039a0 2024-01-24T17:46:06.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.395+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51440c8a30 con 0x7f51240039a0 2024-01-24T17:46:06.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.395+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51440c8090 con 0x7f51240039a0 2024-01-24T17:46:06.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.397+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51440c8220 con 0x7f51240039a0 2024-01-24T17:46:06.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.397+0000 7f513aa00640 1 -- 172.21.15.39:0/116368836 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f512c006d40 con 0x7f51240039a0 2024-01-24T17:46:06.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.398+0000 7f513aa00640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511c06a430 0x7f511c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.398+0000 7f513aa00640 1 -- 172.21.15.39:0/116368836 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f512c0aaab0 con 0x7f51240039a0 2024-01-24T17:46:06.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.399+0000 7f5142000640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511c06a430 0x7f511c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.400+0000 7f5142000640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511c06a430 0x7f511c06c8f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5134007c40 tx=0x7f5134009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:06.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.403+0000 7f513aa00640 1 -- 172.21.15.39:0/116368836 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f512c0761b0 con 0x7f51240039a0 2024-01-24T17:46:06.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.606+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f514404dff0 con 0x7f51240039a0 2024-01-24T17:46:06.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.607+0000 7f513aa00640 1 -- 172.21.15.39:0/116368836 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7f512c04b0a0 con 0x7f51240039a0 2024-01-24T17:46:06.608 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511c06a430 msgr2=0x7f511c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511c06a430 0x7f511c06c8f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5134007c40 tx=0x7f5134009040 comp rx=0 tx=0).stop 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 msgr2=0x7f5144154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5144154610 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f512c008fd0 tx=0x7f512c001120 comp rx=0 tx=0).stop 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 shutdown_connections 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51440b73c0 0x7f51441528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5144152000 0x7f51440b6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f511c06a430 0x7f511c06c8f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 --2- 172.21.15.39:0/116368836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51240039a0 0x7f5144154610 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 >> 172.21.15.39:0/116368836 conn(0x7f514406af20 msgr2=0x7f51441563c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:06.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 shutdown_connections 2024-01-24T17:46:06.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.610+0000 7f5149800640 1 -- 172.21.15.39:0/116368836 wait complete. 2024-01-24T17:46:06.630 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:46:06.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.841+0000 7f23d7200640 1 Processor -- start 2024-01-24T17:46:06.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.842+0000 7f23d7200640 1 -- start start 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.843+0000 7f23d7200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f23d80b4340 0x7f23d80b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.843+0000 7f23d7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d815cf00 0x7f23d80d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.843+0000 7f23d7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23d806bd60 0x7f23d806c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.843+0000 7f23d7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d806ad30 con 0x7f23d806bd60 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.843+0000 7f23d7200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d80b4c60 con 0x7f23d80b4340 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.843+0000 7f23d7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d806afa0 con 0x7f23d815cf00 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d6800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23d806bd60 0x7f23d806c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d6800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23d806bd60 0x7f23d806c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36036/0 (socket says 172.21.15.39:36036) 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d6800640 1 -- 172.21.15.39:0/1641203404 learned_addr learned my addr 172.21.15.39:0/1641203404 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:06.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d5e00640 1 --2- 172.21.15.39:0/1641203404 >> v2:172.21.15.100:3300/0 conn(0x7f23d80b4340 0x7f23d80b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d5400640 1 --2- 172.21.15.39:0/1641203404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d815cf00 0x7f23d80d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d6800640 1 -- 172.21.15.39:0/1641203404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d815cf00 msgr2=0x7f23d80d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:06.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d6800640 1 --2- 172.21.15.39:0/1641203404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d815cf00 0x7f23d80d0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d6800640 1 -- 172.21.15.39:0/1641203404 >> v2:172.21.15.100:3300/0 conn(0x7f23d80b4340 msgr2=0x7f23d80b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:06.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d6800640 1 --2- 172.21.15.39:0/1641203404 >> v2:172.21.15.100:3300/0 conn(0x7f23d80b4340 0x7f23d80b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d6800640 1 -- 172.21.15.39:0/1641203404 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d806b280 con 0x7f23d806bd60 2024-01-24T17:46:06.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.844+0000 7f23d5e00640 1 --2- 172.21.15.39:0/1641203404 >> v2:172.21.15.100:3300/0 conn(0x7f23d80b4340 0x7f23d80b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:06.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.846+0000 7f23d6800640 1 --2- 172.21.15.39:0/1641203404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d806bd60 0x7f23d806c210 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f23d8066dc0 tx=0x7f23d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d946e17e13e417c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.846+0000 7f23d4a00640 1 -- 172.21.15.39:0/1641203404 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0031870 con 0x7f23d806bd60 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.846+0000 7f23d4a00640 1 -- 172.21.15.39:0/1641203404 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d00319d0 con 0x7f23d806bd60 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.846+0000 7f23d4a00640 1 -- 172.21.15.39:0/1641203404 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0031d60 con 0x7f23d806bd60 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.846+0000 7f23d7200640 1 -- 172.21.15.39:0/1641203404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d806bd60 msgr2=0x7f23d806c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.846+0000 7f23d7200640 1 --2- 172.21.15.39:0/1641203404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d806bd60 0x7f23d806c210 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f23d8066dc0 tx=0x7f23d002b910 comp rx=0 tx=0).stop 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.847+0000 7f23d7200640 1 -- 172.21.15.39:0/1641203404 shutdown_connections 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.847+0000 7f23d7200640 1 --2- 172.21.15.39:0/1641203404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d806bd60 0x7f23d806c210 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.847+0000 7f23d7200640 1 --2- 172.21.15.39:0/1641203404 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d815cf00 0x7f23d80d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.847+0000 7f23d7200640 1 --2- 172.21.15.39:0/1641203404 >> v2:172.21.15.100:3300/0 conn(0x7f23d80b4340 0x7f23d80b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.847+0000 7f23d7200640 1 -- 172.21.15.39:0/1641203404 >> 172.21.15.39:0/1641203404 conn(0x7f23d80d28c0 msgr2=0x7f23d80d2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.847+0000 7f23d7200640 1 -- 172.21.15.39:0/1641203404 shutdown_connections 2024-01-24T17:46:06.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.847+0000 7f23d7200640 1 -- 172.21.15.39:0/1641203404 wait complete. 2024-01-24T17:46:06.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.848+0000 7f23d7200640 1 Processor -- start 2024-01-24T17:46:06.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.849+0000 7f23d7200640 1 -- start start 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.849+0000 7f23d7200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23d806bd60 0x7f23d815bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d80b4340 0x7f23d815e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d815cf00 0x7f23d80c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d815c2a0 con 0x7f23d815cf00 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d7200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80c6400 con 0x7f23d806bd60 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80c66e0 con 0x7f23d80b4340 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d815cf00 0x7f23d80c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d815cf00 0x7f23d80c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36040/0 (socket says 172.21.15.39:36040) 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d6800640 1 -- 172.21.15.39:0/2242265202 learned_addr learned my addr 172.21.15.39:0/2242265202 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:06.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d5400640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d80b4340 0x7f23d815e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d5e00640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23d806bd60 0x7f23d815bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d6800640 1 -- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d80b4340 msgr2=0x7f23d815e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d6800640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d80b4340 0x7f23d815e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d6800640 1 -- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23d806bd60 msgr2=0x7f23d815bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d6800640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23d806bd60 0x7f23d815bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d6800640 1 -- 172.21.15.39:0/2242265202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d00094d0 con 0x7f23d815cf00 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.850+0000 7f23d5400640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d80b4340 0x7f23d815e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.851+0000 7f23d6800640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d815cf00 0x7f23d80c74a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f23d00090b0 tx=0x7f23d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.851+0000 7f23ce000640 1 -- 172.21.15.39:0/2242265202 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d003ac60 con 0x7f23d815cf00 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.851+0000 7f23ce000640 1 -- 172.21.15.39:0/2242265202 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d003adc0 con 0x7f23d815cf00 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.851+0000 7f23ce000640 1 -- 172.21.15.39:0/2242265202 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0044630 con 0x7f23d815cf00 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.851+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d80c69c0 con 0x7f23d815cf00 2024-01-24T17:46:06.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.851+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d80c6f80 con 0x7f23d815cf00 2024-01-24T17:46:06.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.853+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d80c7170 con 0x7f23d815cf00 2024-01-24T17:46:06.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.853+0000 7f23ce000640 1 -- 172.21.15.39:0/2242265202 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f23d004d450 con 0x7f23d815cf00 2024-01-24T17:46:06.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.854+0000 7f23ce000640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23b006a430 0x7f23b006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:06.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.854+0000 7f23ce000640 1 -- 172.21.15.39:0/2242265202 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23d002d040 con 0x7f23d815cf00 2024-01-24T17:46:06.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.855+0000 7f23d5e00640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23b006a430 0x7f23b006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:06.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.856+0000 7f23d5e00640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23b006a430 0x7f23b006c8f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f23d80b4060 tx=0x7f23c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:06.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:06.858+0000 7f23ce000640 1 -- 172.21.15.39:0/2242265202 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f23d0076530 con 0x7f23d815cf00 2024-01-24T17:46:07.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.053+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f23d804e840 con 0x7f23d815cf00 2024-01-24T17:46:07.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.054+0000 7f23ce000640 1 -- 172.21.15.39:0/2242265202 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v63) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23d00350a0 con 0x7f23d815cf00 2024-01-24T17:46:07.055 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23b006a430 msgr2=0x7f23b006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23b006a430 0x7f23b006c8f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f23d80b4060 tx=0x7f23c0008040 comp rx=0 tx=0).stop 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d815cf00 msgr2=0x7f23d80c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d815cf00 0x7f23d80c74a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f23d00090b0 tx=0x7f23d00046c0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 shutdown_connections 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23b006a430 0x7f23b006c8f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23d815cf00 0x7f23d80c74a0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23d80b4340 0x7f23d815e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 --2- 172.21.15.39:0/2242265202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23d806bd60 0x7f23d815bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.057+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 >> 172.21.15.39:0/2242265202 conn(0x7f23d80d28c0 msgr2=0x7f23d80cb910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:07.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.058+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 shutdown_connections 2024-01-24T17:46:07.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.058+0000 7f23d7200640 1 -- 172.21.15.39:0/2242265202 wait complete. 2024-01-24T17:46:07.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:46:07.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.303+0000 7f431d200640 1 Processor -- start 2024-01-24T17:46:07.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.303+0000 7f431d200640 1 -- start start 2024-01-24T17:46:07.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.305+0000 7f431d200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f43180b4a70 0x7f43180d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.305+0000 7f431d200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180d3670 con 0x7f4318160170 2024-01-24T17:46:07.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.305+0000 7f431d200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180ce150 con 0x7f43180b6980 2024-01-24T17:46:07.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.305+0000 7f431d200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43180ce430 con 0x7f43180b4a70 2024-01-24T17:46:07.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.305+0000 7f4317400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4318160170 0x7f43180b4360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:38860/0 (socket says 172.21.15.39:38860) 2024-01-24T17:46:07.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.305+0000 7f4317400640 1 -- 172.21.15.39:0/3366900717 learned_addr learned my addr 172.21.15.39:0/3366900717 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:07.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.305+0000 7f4316a00640 1 --2- 172.21.15.39:0/3366900717 >> v2:172.21.15.39:3300/0 conn(0x7f43180b4a70 0x7f43180d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:07.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.306+0000 7f4316000640 1 -- 172.21.15.39:0/3366900717 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008276523 0 0) 0x7f43180d3670 con 0x7f4318160170 2024-01-24T17:46:07.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.306+0000 7f4316000640 1 -- 172.21.15.39:0/3366900717 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f8003520 con 0x7f4318160170 2024-01-24T17:46:07.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.306+0000 7f4316a00640 1 -- 172.21.15.39:0/3366900717 >> v1:172.21.15.100:6789/0 conn(0x7f43180b6980 legacy=0x7f43180d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:07.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.306+0000 7f4316a00640 1 -- 172.21.15.39:0/3366900717 >> v1:172.21.15.39:6789/0 conn(0x7f4318160170 legacy=0x7f43180b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.306+0000 7f4316a00640 1 -- 172.21.15.39:0/3366900717 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43180ce710 con 0x7f43180b4a70 2024-01-24T17:46:07.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.307+0000 7f4316a00640 1 --2- 172.21.15.39:0/3366900717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180b4a70 0x7f43180d0420 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f43000097a0 tx=0x7f430002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d4da14c7234342 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:07.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.308+0000 7f4316000640 1 -- 172.21.15.39:0/3366900717 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300030610 con 0x7f43180b4a70 2024-01-24T17:46:07.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.308+0000 7f4316000640 1 -- 172.21.15.39:0/3366900717 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4300030770 con 0x7f43180b4a70 2024-01-24T17:46:07.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.308+0000 7f4316000640 1 -- 172.21.15.39:0/3366900717 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300030b00 con 0x7f43180b4a70 2024-01-24T17:46:07.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.308+0000 7f431d200640 1 -- 172.21.15.39:0/3366900717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180b4a70 msgr2=0x7f43180d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.308+0000 7f431d200640 1 --2- 172.21.15.39:0/3366900717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180b4a70 0x7f43180d0420 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f43000097a0 tx=0x7f430002b960 comp rx=0 tx=0).stop 2024-01-24T17:46:07.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.309+0000 7f431d200640 1 -- 172.21.15.39:0/3366900717 shutdown_connections 2024-01-24T17:46:07.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.309+0000 7f431d200640 1 --2- 172.21.15.39:0/3366900717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43180b4a70 0x7f43180d0420 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.309+0000 7f431d200640 1 -- 172.21.15.39:0/3366900717 >> 172.21.15.39:0/3366900717 conn(0x7f431809bbf0 msgr2=0x7f43180d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:07.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.309+0000 7f431d200640 1 -- 172.21.15.39:0/3366900717 shutdown_connections 2024-01-24T17:46:07.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.309+0000 7f431d200640 1 -- 172.21.15.39:0/3366900717 wait complete. 2024-01-24T17:46:07.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.310+0000 7f431d200640 1 Processor -- start 2024-01-24T17:46:07.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.311+0000 7f431d200640 1 -- start start 2024-01-24T17:46:07.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.311+0000 7f431d200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180b4a70 0x7f431815ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.311+0000 7f431d200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180b6980 0x7f431815d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f431d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4318160170 0x7f43181e6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f431d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431815c960 con 0x7f4318160170 2024-01-24T17:46:07.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f431d200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431815bd50 con 0x7f43180b6980 2024-01-24T17:46:07.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f431d200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431815c030 con 0x7f43180b4a70 2024-01-24T17:46:07.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4318160170 0x7f43181e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:07.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4318160170 0x7f43181e6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36056/0 (socket says 172.21.15.39:36056) 2024-01-24T17:46:07.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317e00640 1 -- 172.21.15.39:0/1098266451 learned_addr learned my addr 172.21.15.39:0/1098266451 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:07.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317400640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180b4a70 0x7f431815ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:07.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317e00640 1 -- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180b4a70 msgr2=0x7f431815ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4316a00640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180b6980 0x7f431815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:07.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317e00640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180b4a70 0x7f431815ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317e00640 1 -- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180b6980 msgr2=0x7f431815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317e00640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180b6980 0x7f431815d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.312+0000 7f4317e00640 1 -- 172.21.15.39:0/1098266451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4300009400 con 0x7f4318160170 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.313+0000 7f4317e00640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4318160170 0x7f43181e6560 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f431000c7a0 tx=0x7f431000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.313+0000 7f430f400640 1 -- 172.21.15.39:0/1098266451 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310003b70 con 0x7f4318160170 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.313+0000 7f430f400640 1 -- 172.21.15.39:0/1098266451 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4310003cf0 con 0x7f4318160170 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.313+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431815c310 con 0x7f4318160170 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.313+0000 7f430f400640 1 -- 172.21.15.39:0/1098266451 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310003ea0 con 0x7f4318160170 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.313+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431815b1a0 con 0x7f4318160170 2024-01-24T17:46:07.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.314+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43180b4890 con 0x7f4318160170 2024-01-24T17:46:07.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.315+0000 7f430f400640 1 -- 172.21.15.39:0/1098266451 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f431000aa90 con 0x7f4318160170 2024-01-24T17:46:07.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.316+0000 7f430f400640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42f006a430 0x7f42f006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.317+0000 7f430f400640 1 -- 172.21.15.39:0/1098266451 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4310088770 con 0x7f4318160170 2024-01-24T17:46:07.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.317+0000 7f4317400640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42f006a430 0x7f42f006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:07.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.318+0000 7f4317400640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42f006a430 0x7f42f006c8f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f43080010e0 tx=0x7f4308009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:07.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.321+0000 7f430f400640 1 -- 172.21.15.39:0/1098266451 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4310053d50 con 0x7f4318160170 2024-01-24T17:46:07.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.515+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f431804e840 con 0x7f4318160170 2024-01-24T17:46:07.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.516+0000 7f430f400640 1 -- 172.21.15.39:0/1098266451 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v63) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4310057370 con 0x7f4318160170 2024-01-24T17:46:07.517 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:46:07.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42f006a430 msgr2=0x7f42f006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42f006a430 0x7f42f006c8f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f43080010e0 tx=0x7f4308009040 comp rx=0 tx=0).stop 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4318160170 msgr2=0x7f43181e6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4318160170 0x7f43181e6560 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f431000c7a0 tx=0x7f431000cc60 comp rx=0 tx=0).stop 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 shutdown_connections 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42f006a430 0x7f42f006c8f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4318160170 0x7f43181e6560 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43180b6980 0x7f431815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 --2- 172.21.15.39:0/1098266451 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43180b4a70 0x7f431815ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.519+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 >> 172.21.15.39:0/1098266451 conn(0x7f431809bbf0 msgr2=0x7f43181e7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.520+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 shutdown_connections 2024-01-24T17:46:07.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.520+0000 7f431d200640 1 -- 172.21.15.39:0/1098266451 wait complete. 2024-01-24T17:46:07.539 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:46:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.760+0000 7f1d4a800640 1 Processor -- start 2024-01-24T17:46:07.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.760+0000 7f1d4a800640 1 -- start start 2024-01-24T17:46:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.762+0000 7f1d4a800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1d440d06a0 0x7f1d440cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.762+0000 7f1d4a800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d440cd8b0 con 0x7f1d440b1810 2024-01-24T17:46:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.762+0000 7f1d4a800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d440b1d60 con 0x7f1d440d0ea0 2024-01-24T17:46:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.762+0000 7f1d4a800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4406b520 con 0x7f1d440d06a0 2024-01-24T17:46:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.762+0000 7f1d48a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1d440b1810 0x7f1d4406ae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:38874/0 (socket says 172.21.15.39:38874) 2024-01-24T17:46:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.762+0000 7f1d48a00640 1 -- 172.21.15.39:0/28231473 learned_addr learned my addr 172.21.15.39:0/28231473 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:07.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.763+0000 7f1d49e00640 1 --2- 172.21.15.39:0/28231473 >> v2:172.21.15.100:3300/0 conn(0x7f1d440d06a0 0x7f1d440cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.763+0000 7f1d43e00640 1 -- 172.21.15.39:0/28231473 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394046366 0 0) 0x7f1d440cd8b0 con 0x7f1d440b1810 2024-01-24T17:46:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.763+0000 7f1d43e00640 1 -- 172.21.15.39:0/28231473 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d24003520 con 0x7f1d440b1810 2024-01-24T17:46:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.763+0000 7f1d49e00640 1 -- 172.21.15.39:0/28231473 >> v1:172.21.15.100:6789/0 conn(0x7f1d440d0ea0 legacy=0x7f1d440d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.763+0000 7f1d43e00640 1 -- 172.21.15.39:0/28231473 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 455417136 0 0) 0x7f1d24003520 con 0x7f1d440b1810 2024-01-24T17:46:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.763+0000 7f1d49e00640 1 -- 172.21.15.39:0/28231473 >> v1:172.21.15.39:6789/0 conn(0x7f1d440b1810 legacy=0x7f1d4406ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.763+0000 7f1d49e00640 1 -- 172.21.15.39:0/28231473 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d440b2040 con 0x7f1d440d06a0 2024-01-24T17:46:07.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.765+0000 7f1d49e00640 1 --2- 172.21.15.39:0/28231473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d440d06a0 0x7f1d440cdd40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009d90 tx=0x7f1d3c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ee5e007d0179e21 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:07.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.765+0000 7f1d43e00640 1 -- 172.21.15.39:0/28231473 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c001670 con 0x7f1d440d06a0 2024-01-24T17:46:07.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.765+0000 7f1d43e00640 1 -- 172.21.15.39:0/28231473 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c02c940 con 0x7f1d440d06a0 2024-01-24T17:46:07.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.765+0000 7f1d43e00640 1 -- 172.21.15.39:0/28231473 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c02ccb0 con 0x7f1d440d06a0 2024-01-24T17:46:07.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.766+0000 7f1d4a800640 1 -- 172.21.15.39:0/28231473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d440d06a0 msgr2=0x7f1d440cdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.766+0000 7f1d4a800640 1 --2- 172.21.15.39:0/28231473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d440d06a0 0x7f1d440cdd40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009d90 tx=0x7f1d3c02be10 comp rx=0 tx=0).stop 2024-01-24T17:46:07.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.766+0000 7f1d4a800640 1 -- 172.21.15.39:0/28231473 shutdown_connections 2024-01-24T17:46:07.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.766+0000 7f1d4a800640 1 --2- 172.21.15.39:0/28231473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d440d06a0 0x7f1d440cdd40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.766+0000 7f1d4a800640 1 -- 172.21.15.39:0/28231473 >> 172.21.15.39:0/28231473 conn(0x7f1d440d28c0 msgr2=0x7f1d440d2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:07.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.766+0000 7f1d4a800640 1 -- 172.21.15.39:0/28231473 shutdown_connections 2024-01-24T17:46:07.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.766+0000 7f1d4a800640 1 -- 172.21.15.39:0/28231473 wait complete. 2024-01-24T17:46:07.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.768+0000 7f1d4a800640 1 Processor -- start 2024-01-24T17:46:07.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.768+0000 7f1d4a800640 1 -- start start 2024-01-24T17:46:07.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d4a800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d440b1810 0x7f1d440cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d4a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d440d06a0 0x7f1d440b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d4a800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d440d0ea0 0x7f1d440b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d4a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440ca9e0 con 0x7f1d440d06a0 2024-01-24T17:46:07.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d4a800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440cacc0 con 0x7f1d440d0ea0 2024-01-24T17:46:07.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d4a800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440cafa0 con 0x7f1d440b1810 2024-01-24T17:46:07.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d48a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d440d06a0 0x7f1d440b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:07.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d48a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d440d06a0 0x7f1d440b4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36072/0 (socket says 172.21.15.39:36072) 2024-01-24T17:46:07.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d48a00640 1 -- 172.21.15.39:0/3165774816 learned_addr learned my addr 172.21.15.39:0/3165774816 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:07.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d48a00640 1 -- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d440b1810 msgr2=0x7f1d440cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d48a00640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d440b1810 0x7f1d440cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d48a00640 1 -- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d440d0ea0 msgr2=0x7f1d440b7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d48a00640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d440d0ea0 0x7f1d440b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.769+0000 7f1d48a00640 1 -- 172.21.15.39:0/3165774816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c009a70 con 0x7f1d440d06a0 2024-01-24T17:46:07.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.770+0000 7f1d48a00640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d440d06a0 0x7f1d440b4990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0011e0 tx=0x7f1d2c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:07.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.770+0000 7f1d41600640 1 -- 172.21.15.39:0/3165774816 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c005ad0 con 0x7f1d440d06a0 2024-01-24T17:46:07.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.770+0000 7f1d41600640 1 -- 172.21.15.39:0/3165774816 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c005c50 con 0x7f1d440d06a0 2024-01-24T17:46:07.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.770+0000 7f1d41600640 1 -- 172.21.15.39:0/3165774816 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0072b0 con 0x7f1d440d06a0 2024-01-24T17:46:07.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.770+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d440cb280 con 0x7f1d440d06a0 2024-01-24T17:46:07.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.770+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d440cb840 con 0x7f1d440d06a0 2024-01-24T17:46:07.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.771+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d440cba30 con 0x7f1d440d06a0 2024-01-24T17:46:07.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.772+0000 7f1d41600640 1 -- 172.21.15.39:0/3165774816 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1d2c007410 con 0x7f1d440d06a0 2024-01-24T17:46:07.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.773+0000 7f1d41600640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d1c06a460 0x7f1d1c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:07.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.773+0000 7f1d41600640 1 -- 172.21.15.39:0/3165774816 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d2c087fc0 con 0x7f1d440d06a0 2024-01-24T17:46:07.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.774+0000 7f1d49400640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d1c06a460 0x7f1d1c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:07.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.775+0000 7f1d49400640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d1c06a460 0x7f1d1c06c920 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1d34007fd0 tx=0x7f1d340079a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.778+0000 7f1d41600640 1 -- 172.21.15.39:0/3165774816 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1d2c0535a0 con 0x7f1d440d06a0 2024-01-24T17:46:07.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.994+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1d4404e840 con 0x7f1d440d06a0 2024-01-24T17:46:07.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.995+0000 7f1d41600640 1 -- 172.21.15.39:0/3165774816 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v63) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1d2c017080 con 0x7f1d440d06a0 2024-01-24T17:46:07.996 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:46:07.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d1c06a460 msgr2=0x7f1d1c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d1c06a460 0x7f1d1c06c920 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1d34007fd0 tx=0x7f1d340079a0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d440d06a0 msgr2=0x7f1d440b4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:07.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d440d06a0 0x7f1d440b4990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0011e0 tx=0x7f1d2c00cb90 comp rx=0 tx=0).stop 2024-01-24T17:46:07.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 shutdown_connections 2024-01-24T17:46:07.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d440d0ea0 0x7f1d440b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:07.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d1c06a460 0x7f1d1c06c920 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d440d06a0 0x7f1d440b4990 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 --2- 172.21.15.39:0/3165774816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d440b1810 0x7f1d440cbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 >> 172.21.15.39:0/3165774816 conn(0x7f1d440d28c0 msgr2=0x7f1d4415d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 shutdown_connections 2024-01-24T17:46:08.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:07.998+0000 7f1d4a800640 1 -- 172.21.15.39:0/3165774816 wait complete. 2024-01-24T17:46:08.019 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:46:08.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:08.019 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:08.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[45932]: fuse finished with error 0 and tester_r 0 2024-01-24T17:46:08.119 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:46:08.202 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:46:08.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:08.202 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:46:08.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:46:08.264 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:46:08.428 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-01-24T17:46:08.428 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-01-24T17:46:08.428 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:46:08.428 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-01-24T17:46:08.428 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:46:08.429 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:46:08.430 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.430 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-01-24T17:46:08.430 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.430 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-01-24T17:46:08.430 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-01-24T17:46:08.430 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:46:08.430 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:46:08.430 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-01-24T17:46:08.431 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:46:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.433 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = bitmap 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:08.434 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:46:08.435 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-01-24T17:46:08.436 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:46:08.437 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: mon op complaint time = 120 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: client mount timeout = 600 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = "" 2024-01-24T17:46:08.438 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout:[mds] 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds = 20 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds balancer = 20 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug frag = True 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug scatterstat = True 2024-01-24T17:46:08.439 INFO:teuthology.orchestra.run.smithi100.stdout: mds op complaint time = 180 2024-01-24T17:46:08.440 INFO:teuthology.orchestra.run.smithi100.stdout: mds verify scatter = True 2024-01-24T17:46:08.440 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:46:08.440 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:46:08.440 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:46:08.440 INFO:teuthology.orchestra.run.smithi100.stdout:[client.badguy] 2024-01-24T17:46:08.440 INFO:teuthology.orchestra.run.smithi100.stdout: keyring = /etc/ceph/ceph.client.badguy.keyring 2024-01-24T17:46:08.440 INFO:teuthology.orchestra.run.smithi100.stdout: client_metadata = root=/baz 2024-01-24T17:46:08.440 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-01-24T17:46:08.441 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:46:08.442 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.443 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:46:08.444 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.445 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:08.446 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:46:08.447 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:08.448 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:08.449 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-01-24T17:46:08.450 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:46:08.451 INFO:teuthology.orchestra.run.smithi039.stdout:[client.badguy] 2024-01-24T17:46:08.452 INFO:teuthology.orchestra.run.smithi039.stdout: keyring = /etc/ceph/ceph.client.badguy.keyring 2024-01-24T17:46:08.452 INFO:teuthology.orchestra.run.smithi039.stdout: client_metadata = root=/baz 2024-01-24T17:46:08.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo daemon-helper kill ceph --cluster ceph -w 2024-01-24T17:46:08.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:46:08.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.680+0000 7fb4d4200640 1 Processor -- start 2024-01-24T17:46:08.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.680+0000 7fb4d4200640 1 -- start start 2024-01-24T17:46:08.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.681+0000 7fb4d4200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0cf500 0x7fb4cc0d3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.682+0000 7fb4d4200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb4cc0d08d0 0x7fb4cc0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.682+0000 7fb4d4200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc0cd950 con 0x7fb4cc0ce9b0 2024-01-24T17:46:08.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.682+0000 7fb4d4200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc0cfe20 con 0x7fb4cc0d08d0 2024-01-24T17:46:08.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.682+0000 7fb4d4200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0cdbf0 con 0x7fb4cc0cf500 2024-01-24T17:46:08.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.682+0000 7fb4d0e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb4cc0d08d0 0x7fb4cc0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.682+0000 7fb4d0e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb4cc0d08d0 0x7fb4cc0cf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:53284/0 (socket says 172.21.15.39:53284) 2024-01-24T17:46:08.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.682+0000 7fb4d0e00640 1 -- 172.21.15.39:0/1688277215 learned_addr learned my addr 172.21.15.39:0/1688277215 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:08.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.682+0000 7fb4d1800640 1 --2- 172.21.15.39:0/1688277215 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0cf500 0x7fb4cc0d3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.683+0000 7fb4d0e00640 1 -- 172.21.15.39:0/1688277215 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0cf500 msgr2=0x7fb4cc0d3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.683+0000 7fb4d0e00640 1 --2- 172.21.15.39:0/1688277215 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0cf500 0x7fb4cc0d3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.683+0000 7fb4d0e00640 1 -- 172.21.15.39:0/1688277215 >> v1:172.21.15.100:6789/0 conn(0x7fb4cc0ce9b0 legacy=0x7fb4cc160a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.683+0000 7fb4d0e00640 1 -- 172.21.15.39:0/1688277215 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0cded0 con 0x7fb4cc0d08d0 2024-01-24T17:46:08.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.685+0000 7fb4d0e00640 1 --2- 172.21.15.39:0/1688277215 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0d08d0 0x7fb4cc0cf8e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009870 tx=0x7fb4b402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5abf7128b2401489 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:08.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.685+0000 7fb4cbe00640 1 -- 172.21.15.39:0/1688277215 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4030840 con 0x7fb4cc0d08d0 2024-01-24T17:46:08.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.685+0000 7fb4cbe00640 1 -- 172.21.15.39:0/1688277215 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b40309a0 con 0x7fb4cc0d08d0 2024-01-24T17:46:08.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.685+0000 7fb4cbe00640 1 -- 172.21.15.39:0/1688277215 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4030d30 con 0x7fb4cc0d08d0 2024-01-24T17:46:08.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.686+0000 7fb4d4200640 1 -- 172.21.15.39:0/1688277215 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0d08d0 msgr2=0x7fb4cc0cf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.686+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1688277215 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0d08d0 0x7fb4cc0cf8e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009870 tx=0x7fb4b402b980 comp rx=0 tx=0).stop 2024-01-24T17:46:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.686+0000 7fb4d4200640 1 -- 172.21.15.39:0/1688277215 shutdown_connections 2024-01-24T17:46:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.686+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1688277215 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0d08d0 0x7fb4cc0cf8e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.686+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1688277215 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0cf500 0x7fb4cc0d3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.686+0000 7fb4d4200640 1 -- 172.21.15.39:0/1688277215 >> 172.21.15.39:0/1688277215 conn(0x7fb4cc0d2af0 msgr2=0x7fb4cc0d2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.686+0000 7fb4d4200640 1 -- 172.21.15.39:0/1688277215 shutdown_connections 2024-01-24T17:46:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.686+0000 7fb4d4200640 1 -- 172.21.15.39:0/1688277215 wait complete. 2024-01-24T17:46:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.688+0000 7fb4d4200640 1 Processor -- start 2024-01-24T17:46:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.688+0000 7fb4d4200640 1 -- start start 2024-01-24T17:46:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d4200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0ce9b0 0x7fb4cc1f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d4200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4cc0cf500 0x7fb4cc0b6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d4200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0d08d0 0x7fb4cc0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d4200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc1f5790 con 0x7fb4cc0cf500 2024-01-24T17:46:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d4200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0b6d80 con 0x7fb4cc0ce9b0 2024-01-24T17:46:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d4200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0c7ae0 con 0x7fb4cc0d08d0 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4cc0cf500 0x7fb4cc0b6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4cc0cf500 0x7fb4cc0b6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36080/0 (socket says 172.21.15.39:36080) 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d0e00640 1 -- 172.21.15.39:0/1514772973 learned_addr learned my addr 172.21.15.39:0/1514772973 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d1800640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0ce9b0 0x7fb4cc1f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.689+0000 7fb4d2200640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0d08d0 0x7fb4cc0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4d0e00640 1 -- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0d08d0 msgr2=0x7fb4cc0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4d0e00640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0d08d0 0x7fb4cc0c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4d0e00640 1 -- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0ce9b0 msgr2=0x7fb4cc1f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4d0e00640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0ce9b0 0x7fb4cc1f5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4d0e00640 1 -- 172.21.15.39:0/1514772973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40094d0 con 0x7fb4cc0cf500 2024-01-24T17:46:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4d1800640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0ce9b0 0x7fb4cc1f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:08.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4d0e00640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4cc0cf500 0x7fb4cc0b6840 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4000c00 tx=0x7fb4b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:08.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4c9600640 1 -- 172.21.15.39:0/1514772973 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4039a90 con 0x7fb4cc0cf500 2024-01-24T17:46:08.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4c9600640 1 -- 172.21.15.39:0/1514772973 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b4039bf0 con 0x7fb4cc0cf500 2024-01-24T17:46:08.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4c9600640 1 -- 172.21.15.39:0/1514772973 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4043410 con 0x7fb4cc0cf500 2024-01-24T17:46:08.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.690+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc0b6f70 con 0x7fb4cc0cf500 2024-01-24T17:46:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.691+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc0b7530 con 0x7fb4cc0cf500 2024-01-24T17:46:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.692+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc067360 con 0x7fb4cc0cf500 2024-01-24T17:46:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.693+0000 7fb4c9600640 1 -- 172.21.15.39:0/1514772973 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb4b402f390 con 0x7fb4cc0cf500 2024-01-24T17:46:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.693+0000 7fb4c9600640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a406a430 0x7fb4a406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.694+0000 7fb4c9600640 1 -- 172.21.15.39:0/1514772973 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4b40aaa20 con 0x7fb4cc0cf500 2024-01-24T17:46:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.694+0000 7fb4d1800640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a406a430 0x7fb4a406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.695+0000 7fb4d1800640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a406a430 0x7fb4a406c8f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009b90 tx=0x7fb4bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:08.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.698+0000 7fb4c9600640 1 -- 172.21.15.39:0/1514772973 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb4b4076000 con 0x7fb4cc0cf500 2024-01-24T17:46:08.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.713+0000 7f8c9a600640 1 Processor -- start 2024-01-24T17:46:08.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.713+0000 7f8c9a600640 1 -- start start 2024-01-24T17:46:08.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.715+0000 7f8c9a600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940cb100 0x7f8c9415efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.715+0000 7f8c9a600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8c940ce020 0x7f8c940cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.715+0000 7f8c9a600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9415eb50 con 0x7f8c940adb20 2024-01-24T17:46:08.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.715+0000 7f8c9a600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c9415e540 con 0x7f8c940ce020 2024-01-24T17:46:08.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.715+0000 7f8c9a600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940caa80 con 0x7f8c940cb100 2024-01-24T17:46:08.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.715+0000 7f8c93e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8c940ce020 0x7f8c940cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.715+0000 7f8c93400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940cb100 0x7f8c9415efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.716+0000 7f8c93e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8c940ce020 0x7f8c940cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:53308/0 (socket says 172.21.15.39:53308) 2024-01-24T17:46:08.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.716+0000 7f8c93e00640 1 -- 172.21.15.39:0/2290579504 learned_addr learned my addr 172.21.15.39:0/2290579504 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:08.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.716+0000 7f8c93e00640 1 -- 172.21.15.39:0/2290579504 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940cb100 msgr2=0x7f8c9415efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:08.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.716+0000 7f8c93e00640 1 --2- 172.21.15.39:0/2290579504 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940cb100 0x7f8c9415efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.716+0000 7f8c93e00640 1 -- 172.21.15.39:0/2290579504 >> v1:172.21.15.100:6789/0 conn(0x7f8c940adb20 legacy=0x7f8c940cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.716+0000 7f8c93e00640 1 -- 172.21.15.39:0/2290579504 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c940cad60 con 0x7f8c940ce020 2024-01-24T17:46:08.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.718+0000 7f8c93e00640 1 --2- 172.21.15.39:0/2290579504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c940ce020 0x7f8c940cc2c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009e80 tx=0x7f8c8c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc7b20fdfd7d588a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:08.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c92a00640 1 -- 172.21.15.39:0/2290579504 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c02ca50 con 0x7f8c940ce020 2024-01-24T17:46:08.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c92a00640 1 -- 172.21.15.39:0/2290579504 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c02cbb0 con 0x7f8c940ce020 2024-01-24T17:46:08.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c92a00640 1 -- 172.21.15.39:0/2290579504 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0343f0 con 0x7f8c940ce020 2024-01-24T17:46:08.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c9a600640 1 -- 172.21.15.39:0/2290579504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c940ce020 msgr2=0x7f8c940cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c9a600640 1 --2- 172.21.15.39:0/2290579504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c940ce020 0x7f8c940cc2c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009e80 tx=0x7f8c8c02beb0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c9a600640 1 -- 172.21.15.39:0/2290579504 shutdown_connections 2024-01-24T17:46:08.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c9a600640 1 --2- 172.21.15.39:0/2290579504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c940ce020 0x7f8c940cc2c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c9a600640 1 --2- 172.21.15.39:0/2290579504 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940cb100 0x7f8c9415efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.719+0000 7f8c9a600640 1 -- 172.21.15.39:0/2290579504 >> 172.21.15.39:0/2290579504 conn(0x7f8c9406a3e0 msgr2=0x7f8c940b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.720+0000 7f8c9a600640 1 -- 172.21.15.39:0/2290579504 shutdown_connections 2024-01-24T17:46:08.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.720+0000 7f8c9a600640 1 -- 172.21.15.39:0/2290579504 wait complete. 2024-01-24T17:46:08.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.721+0000 7f8c9a600640 1 Processor -- start 2024-01-24T17:46:08.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.721+0000 7f8c9a600640 1 -- start start 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.722+0000 7f8c9a600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c940adb20 0x7f8c94157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.722+0000 7f8c9a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c940cb100 0x7f8c940c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.722+0000 7f8c9a600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940ce020 0x7f8c9415c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.722+0000 7f8c9a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940c6d40 con 0x7f8c940cb100 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.722+0000 7f8c9a600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940c60c0 con 0x7f8c940adb20 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.722+0000 7f8c9a600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940c63a0 con 0x7f8c940ce020 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c8be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c940cb100 0x7f8c940c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c8be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c940cb100 0x7f8c940c57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36086/0 (socket says 172.21.15.39:36086) 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c8be00640 1 -- 172.21.15.39:0/3338875132 learned_addr learned my addr 172.21.15.39:0/3338875132 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c93400640 1 --2- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c940adb20 0x7f8c94157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c93e00640 1 --2- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940ce020 0x7f8c9415c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c8be00640 1 -- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940ce020 msgr2=0x7f8c9415c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c8be00640 1 --2- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8c940ce020 0x7f8c9415c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c8be00640 1 -- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c940adb20 msgr2=0x7f8c94157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c8be00640 1 --2- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8c940adb20 0x7f8c94157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.723+0000 7f8c8be00640 1 -- 172.21.15.39:0/3338875132 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c009b30 con 0x7f8c940cb100 2024-01-24T17:46:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.724+0000 7f8c8be00640 1 --2- 172.21.15.39:0/3338875132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c940cb100 0x7f8c940c57c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00c450 tx=0x7f8c7c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.724+0000 7f8c8b400640 1 -- 172.21.15.39:0/3338875132 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0052d0 con 0x7f8c940cb100 2024-01-24T17:46:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.724+0000 7f8c8b400640 1 -- 172.21.15.39:0/3338875132 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c005430 con 0x7f8c940cb100 2024-01-24T17:46:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.724+0000 7f8c8b400640 1 -- 172.21.15.39:0/3338875132 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c012870 con 0x7f8c940cb100 2024-01-24T17:46:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.724+0000 7f8c9a600640 1 -- 172.21.15.39:0/3338875132 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c940c6680 con 0x7f8c940cb100 2024-01-24T17:46:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.724+0000 7f8c9a600640 1 -- 172.21.15.39:0/3338875132 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c94157dc0 con 0x7f8c940cb100 2024-01-24T17:46:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.725+0000 7f8c9a600640 1 -- 172.21.15.39:0/3338875132 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f8c940c6e80 con 0x7f8c940cb100 2024-01-24T17:46:08.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.726+0000 7f8c8b400640 1 -- 172.21.15.39:0/3338875132 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8c7c0129d0 con 0x7f8c940cb100 2024-01-24T17:46:08.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.727+0000 7f8c8b400640 1 --2- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c6c06a430 0x7f8c6c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:08.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.727+0000 7f8c8b400640 1 -- 172.21.15.39:0/3338875132 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c7c088c90 con 0x7f8c940cb100 2024-01-24T17:46:08.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.727+0000 7f8c8b400640 1 -- 172.21.15.39:0/3338875132 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+499 (secure 0 0 0) 0x7f8c7c089100 con 0x7f8c940cb100 2024-01-24T17:46:08.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.728+0000 7f8c9a600640 1 -- 172.21.15.39:0/3338875132 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f8c940c6e80 con 0x7f8c940cb100 2024-01-24T17:46:08.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.728+0000 7f8c93400640 1 --2- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c6c06a430 0x7f8c6c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:08.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.729+0000 7f8c93400640 1 --2- 172.21.15.39:0/3338875132 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8c6c06a430 0x7f8c6c06c8f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009ad0 tx=0x7f8c84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:08.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.933+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb4cc0d0eb0 con 0x7fb4cc0cf500 2024-01-24T17:46:08.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.933+0000 7fb4c9600640 1 -- 172.21.15.39:0/1514772973 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb4b4048060 con 0x7fb4cc0cf500 2024-01-24T17:46:08.934 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:46:08.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a406a430 msgr2=0x7fb4a406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a406a430 0x7fb4a406c8f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009b90 tx=0x7fb4bc008040 comp rx=0 tx=0).stop 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4cc0cf500 msgr2=0x7fb4cc0b6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4cc0cf500 0x7fb4cc0b6840 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4000c00 tx=0x7fb4b4004730 comp rx=0 tx=0).stop 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 shutdown_connections 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4cc0d08d0 0x7fb4cc0c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a406a430 0x7fb4a406c8f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4cc0cf500 0x7fb4cc0b6840 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 --2- 172.21.15.39:0/1514772973 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4cc0ce9b0 0x7fb4cc1f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.936+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 >> 172.21.15.39:0/1514772973 conn(0x7fb4cc0d2af0 msgr2=0x7fb4cc0afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.937+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 shutdown_connections 2024-01-24T17:46:08.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:08.937+0000 7fb4d4200640 1 -- 172.21.15.39:0/1514772973 wait complete. 2024-01-24T17:46:08.955 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:46:08.955 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:46:08.955 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:46:08.955 INFO:tasks.cephfs.mount:self.client.name = client.badguy 2024-01-24T17:46:08.955 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:08.956 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:46:08.956 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /foo/bar 2024-01-24T17:46:08.956 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:46:08.956 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:46:08.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:08.956 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:46:09.011 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:46:09.011 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:46:09.011 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:46:09.012 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42360sec preferred_lft 42360sec 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:46:09.013 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:09.014 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:46:09.014 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:46:09.014 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:46:09.014 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:09.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:09.015 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:46:09.102 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:46:09.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:09.107 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:46:09.159 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:46:09.159 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:46:09.159 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:46:09.159 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:46:09.159 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:46:09.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:09.160 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:46:09.160 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:46:09.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:46:09.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:46:09.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:46:09.160 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:46:09.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:09.339 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:46:09.390 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:46:09.390 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:46:09.391 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:46:09.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:09.391 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:46:09.479 INFO:tasks.cephfs.fuse_mount:Client client.badguy config is {} 2024-01-24T17:46:09.479 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:46:09.479 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:09.531 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:46:09.531 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:46:09.531 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:09.581 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:46:09.673 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:09.674 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:09.766 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:09.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:09.771 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:46:09.824 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:46:09.824 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id badguy --client_mountpoint=/foo/bar) 2024-01-24T17:46:09.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:46:09.911 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:09.911 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:09.987 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:09.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:09.991 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:46:10.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:46:10.010+0000 7f270cf53480 -1 init, newargv = 0x55b5ee6b15d0 newargc=15 2024-01-24T17:46:10.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[46502]: starting ceph client 2024-01-24T17:46:10.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:46:10.015+0000 7f26f3e00640 -1 client.5743 mds.0 rejected us (non-allowable root '/baz') 2024-01-24T17:46:10.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[46502]: ceph mount failed with (1) Operation not permitted 2024-01-24T17:46:10.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:daemon-helper: command failed with exit status 1 2024-01-24T17:46:10.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:10.887+0000 7f8c8b400640 1 -- 172.21.15.39:0/3338875132 <== mon.0 v2:172.21.15.39:3300/0 7 ==== log(1 entries from seq 1 at 2024-01-24T17:46:10.016455+0000) v1 ==== 261+0+0 (secure 0 0 0) 0x7f8c7c0586d0 con 0x7f8c940cb100 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: cluster: 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: id: b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: health: HEALTH_OK 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: services: 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: mon: 3 daemons, quorum a,b,c (age 4m) 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: mgr: x(active, since 4m), standbys: y 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: mds: 1/1 daemons up, 3 standby 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: osd: 8 osds: 8 up (since 4m), 8 in (since 4m) 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: data: 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: volumes: 1/1 healthy 2024-01-24T17:46:10.889 INFO:teuthology.orchestra.run.smithi039.stdout: pools: 3 pools, 97 pgs 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout: objects: 24 objects, 587 KiB 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout: usage: 246 MiB used, 720 GiB / 720 GiB avail 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout: pgs: 97 active+clean 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout: io: 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout: client: 255 B/s wr, 0 op/s rd, 0 op/s wr 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:10.890 INFO:teuthology.orchestra.run.smithi039.stdout:2024-01-24T17:46:10.016455+0000 mds.c [WRN] client session with non-allowable root '/baz' denied (client.5743 192.168.0.1:0/2735963411) 2024-01-24T17:46:11.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:46:11.235 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:11.235 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:11.328 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:11.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:11.332 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:46:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:46:11.386 INFO:tasks.cephfs.fuse_mount:mount command failed. 2024-01-24T17:46:11.487 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:46:11.487 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:46:11.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:46:11.723 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:46:11.724 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:46:11.724 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:11.784 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:46:11.784 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:46:11.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:11.785 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:46:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:46:11.845 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-24T17:46:11.845 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:11.846 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:11.904 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:46:11.904 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:46:11.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:11.904 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:46:11.957 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:46:11.957 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:12.019 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:46:12.019 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:46:12.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:12.020 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-24T17:46:12.075 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-24T17:46:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:46:12.076 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:12.076 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:12.129 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:46:12.129 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:46:12.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:12.129 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-24T17:46:12.182 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-24T17:46:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:46:12.183 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:46:12.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.412+0000 7f985d800640 1 Processor -- start 2024-01-24T17:46:12.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.412+0000 7f985d800640 1 -- start start 2024-01-24T17:46:12.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.414+0000 7f985d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9858159930 0x7f985815d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.414+0000 7f985d800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f98580b3470 0x7f985815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.414+0000 7f985d800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98580c8ad0 con 0x7f9858159d10 2024-01-24T17:46:12.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.414+0000 7f985d800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98580b3930 con 0x7f98580b3470 2024-01-24T17:46:12.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.414+0000 7f985d800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98580b3c10 con 0x7f9858159930 2024-01-24T17:46:12.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.414+0000 7f985ce00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9858159d10 0x7f98580cc050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:38882/0 (socket says 172.21.15.39:38882) 2024-01-24T17:46:12.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.414+0000 7f985ce00640 1 -- 172.21.15.39:0/2919406078 learned_addr learned my addr 172.21.15.39:0/2919406078 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:12.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857e00640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9858159930 0x7f985815d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f984f200640 1 --2- 172.21.15.39:0/2919406078 >> v2:172.21.15.100:3300/0 conn(0x7f98580b3470 0x7f985815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571932366 0 0) 0x7f98580c8ad0 con 0x7f9858159d10 2024-01-24T17:46:12.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9838003520 con 0x7f9858159d10 2024-01-24T17:46:12.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882199680 0 0) 0x7f9838003520 con 0x7f9858159d10 2024-01-24T17:46:12.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9858159930 msgr2=0x7f985815d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:12.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9858159930 0x7f985815d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 >> v2:172.21.15.100:3300/0 conn(0x7f98580b3470 msgr2=0x7f985815dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 --2- 172.21.15.39:0/2919406078 >> v2:172.21.15.100:3300/0 conn(0x7f98580b3470 0x7f985815dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98580b3ef0 con 0x7f9858159d10 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f98500019f0 con 0x7f9858159d10 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 >> v1:172.21.15.39:6789/0 conn(0x7f9858159d10 legacy=0x7f98580cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9858159930 0x7f98380039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9838003f60 0x7f98380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838004d00 0x7f9838005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98580b3930 con 0x7f9838003f60 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98580b3c10 con 0x7f9838004d00 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.415+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983800dbc0 con 0x7f9858159930 2024-01-24T17:46:12.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.416+0000 7f984f200640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9838003f60 0x7f98380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.416+0000 7f9857e00640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9858159930 0x7f98380039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.416+0000 7f985ce00640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838004d00 0x7f9838005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.416+0000 7f984f200640 1 -- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9858159930 msgr2=0x7f98380039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:12.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.416+0000 7f984f200640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9858159930 0x7f98380039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.416+0000 7f984f200640 1 -- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838004d00 msgr2=0x7f9838005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:12.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.416+0000 7f984f200640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838004d00 0x7f9838005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.416+0000 7f984f200640 1 -- 172.21.15.39:0/2919406078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838003520 con 0x7f9838003f60 2024-01-24T17:46:12.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.417+0000 7f984f200640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9838003f60 0x7f98380047c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f9840009ac0 tx=0x7f98380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73327af42184e9f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:12.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.417+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984002c710 con 0x7f9838003f60 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.417+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984002c870 con 0x7f9838003f60 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.417+0000 7f9857400640 1 -- 172.21.15.39:0/2919406078 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984002cc00 con 0x7f9838003f60 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.418+0000 7f985d800640 1 -- 172.21.15.39:0/2919406078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9838003f60 msgr2=0x7f98380047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.418+0000 7f985d800640 1 --2- 172.21.15.39:0/2919406078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9838003f60 0x7f98380047c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f9840009ac0 tx=0x7f98380057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.418+0000 7f985ce00640 1 -- 172.21.15.39:0/2919406078 reap_dead start 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.418+0000 7f985d800640 1 -- 172.21.15.39:0/2919406078 shutdown_connections 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.418+0000 7f985d800640 1 -- 172.21.15.39:0/2919406078 >> 172.21.15.39:0/2919406078 conn(0x7f985806a550 msgr2=0x7f985806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.418+0000 7f985d800640 1 -- 172.21.15.39:0/2919406078 shutdown_connections 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.418+0000 7f985d800640 1 -- 172.21.15.39:0/2919406078 wait complete. 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.420+0000 7f985d800640 1 Processor -- start 2024-01-24T17:46:12.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.420+0000 7f985d800640 1 -- start start 2024-01-24T17:46:12.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985d800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838003f60 0x7f98581515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98581f6650 0x7f98581f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98581f1900 0x7f98581f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9858070aa0 con 0x7f98581f1900 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985d800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98581f22a0 con 0x7f9838003f60 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985d800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985806fbb0 con 0x7f98581f6650 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98581f1900 0x7f98581f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98581f1900 0x7f98581f1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36104/0 (socket says 172.21.15.39:36104) 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.421+0000 7f985ce00640 1 -- 172.21.15.39:0/1399507308 learned_addr learned my addr 172.21.15.39:0/1399507308 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.422+0000 7f9857e00640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838003f60 0x7f98581515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.422+0000 7f984f200640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98581f6650 0x7f98581f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.422+0000 7f985ce00640 1 -- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98581f6650 msgr2=0x7f98581f5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.422+0000 7f985ce00640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98581f6650 0x7f98581f5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.422+0000 7f985ce00640 1 -- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838003f60 msgr2=0x7f98581515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.422+0000 7f985ce00640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838003f60 0x7f98581515a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.422+0000 7f985ce00640 1 -- 172.21.15.39:0/1399507308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985806fe90 con 0x7f98581f1900 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.422+0000 7f985ce00640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98581f1900 0x7f98581f1d60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9850000f80 tx=0x7f985000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.423+0000 7f9854c00640 1 -- 172.21.15.39:0/1399507308 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850004d60 con 0x7f98581f1900 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.423+0000 7f9854c00640 1 -- 172.21.15.39:0/1399507308 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985000bb80 con 0x7f98581f1900 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.423+0000 7f9854c00640 1 -- 172.21.15.39:0/1399507308 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850007110 con 0x7f98581f1900 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.423+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9858070170 con 0x7f98581f1900 2024-01-24T17:46:12.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.423+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9858070730 con 0x7f98581f1900 2024-01-24T17:46:12.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.424+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9858061880 con 0x7f98581f1900 2024-01-24T17:46:12.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.425+0000 7f9854c00640 1 -- 172.21.15.39:0/1399507308 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9850007270 con 0x7f98581f1900 2024-01-24T17:46:12.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.426+0000 7f9854c00640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f983006a460 0x7f983006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.426+0000 7f9854c00640 1 -- 172.21.15.39:0/1399507308 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9850088b80 con 0x7f98581f1900 2024-01-24T17:46:12.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.426+0000 7f9857e00640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f983006a460 0x7f983006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.428+0000 7f9857e00640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f983006a460 0x7f983006c920 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9848009930 tx=0x7f9848008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:12.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.431+0000 7f9854c00640 1 -- 172.21.15.39:0/1399507308 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9850054160 con 0x7f98581f1900 2024-01-24T17:46:12.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.664+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f985806b740 con 0x7f98581f1900 2024-01-24T17:46:12.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.665+0000 7f9854c00640 1 -- 172.21.15.39:0/1399507308 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f9850088e60 con 0x7f98581f1900 2024-01-24T17:46:12.666 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:12.666 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3045325513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3045325513},{"type":"v1","addr":"172.21.15.100:6836","nonce":3045325513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5126,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/858868211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":858868211},{"type":"v1","addr":"172.21.15.100:6837","nonce":858868211}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3121826441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3121826441},{"type":"v1","addr":"172.21.15.39:6833","nonce":3121826441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:45:18.381640+0000","modified":"2024-01-24T17:45:21.392053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6261":{"gid":6261,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/527429915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":527429915},{"type":"v1","addr":"172.21.15.39:6835","nonce":527429915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-24T17:46:12.666 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-01-24T17:46:12.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.667+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f983006a460 msgr2=0x7f983006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.667+0000 7f985d800640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f983006a460 0x7f983006c920 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9848009930 tx=0x7f9848008040 comp rx=0 tx=0).stop 2024-01-24T17:46:12.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98581f1900 msgr2=0x7f98581f1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98581f1900 0x7f98581f1d60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9850000f80 tx=0x7f985000b4a0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 shutdown_connections 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f983006a460 0x7f983006c920 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98581f1900 0x7f98581f1d60 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98581f6650 0x7f98581f5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 --2- 172.21.15.39:0/1399507308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9838003f60 0x7f98581515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 >> 172.21.15.39:0/1399507308 conn(0x7f985806a550 msgr2=0x7f98580ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 shutdown_connections 2024-01-24T17:46:12.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.668+0000 7f985d800640 1 -- 172.21.15.39:0/1399507308 wait complete. 2024-01-24T17:46:12.688 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.909+0000 7f5ed7200640 1 Processor -- start 2024-01-24T17:46:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.909+0000 7f5ed7200640 1 -- start start 2024-01-24T17:46:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.911+0000 7f5ed7200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5ed015ccd0 0x7f5ed00d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.911+0000 7f5ed7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed00af840 0x7f5ed006b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.911+0000 7f5ed7200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed006af90 con 0x7f5ed006bbd0 2024-01-24T17:46:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.911+0000 7f5ed7200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed006b530 con 0x7f5ed015ccd0 2024-01-24T17:46:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.911+0000 7f5ed7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed006c050 con 0x7f5ed00af840 2024-01-24T17:46:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.912+0000 7f5ed5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed00af840 0x7f5ed006b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.912+0000 7f5ed5e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5ed015ccd0 0x7f5ed00d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.912+0000 7f5ed5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed00af840 0x7f5ed006b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:36920/0 (socket says 172.21.15.39:36920) 2024-01-24T17:46:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.912+0000 7f5ed6800640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f5ed006bbd0 0x7f5ed00ccff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:39364/0 (socket says 172.21.15.39:39364) 2024-01-24T17:46:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.912+0000 7f5ed5e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5ed015ccd0 0x7f5ed00d1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:53342/0 (socket says 172.21.15.39:53342) 2024-01-24T17:46:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.912+0000 7f5ed5400640 1 -- 172.21.15.39:0/3905196383 learned_addr learned my addr 172.21.15.39:0/3905196383 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.912+0000 7f5ed5400640 1 -- 172.21.15.39:0/3905196383 >> v2:172.21.15.100:3300/0 conn(0x7f5ed015ccd0 msgr2=0x7f5ed00d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.912+0000 7f5ed5400640 1 --2- 172.21.15.39:0/3905196383 >> v2:172.21.15.100:3300/0 conn(0x7f5ed015ccd0 0x7f5ed00d1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.913+0000 7f5ed5400640 1 -- 172.21.15.39:0/3905196383 >> v1:172.21.15.100:6789/0 conn(0x7f5ed006bbd0 legacy=0x7f5ed00ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.913+0000 7f5ed5400640 1 -- 172.21.15.39:0/3905196383 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed006b250 con 0x7f5ed00af840 2024-01-24T17:46:12.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.913+0000 7f5ed5e00640 1 --2- 172.21.15.39:0/3905196383 >> v2:172.21.15.100:3300/0 conn(0x7f5ed015ccd0 0x7f5ed00d1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:46:12.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.913+0000 7f5ed4a00640 1 -- 172.21.15.39:0/3905196383 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588199236 0 0) 0x7f5ed006af90 con 0x7f5ed006bbd0 2024-01-24T17:46:12.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.914+0000 7f5ed5400640 1 --2- 172.21.15.39:0/3905196383 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed00af840 0x7f5ed006b690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009870 tx=0x7f5eb802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1fbb146be7b5748 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.915+0000 7f5ed4a00640 1 -- 172.21.15.39:0/3905196383 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8030840 con 0x7f5ed00af840 2024-01-24T17:46:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.915+0000 7f5ed4a00640 1 -- 172.21.15.39:0/3905196383 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb80309a0 con 0x7f5ed00af840 2024-01-24T17:46:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.915+0000 7f5ed4a00640 1 -- 172.21.15.39:0/3905196383 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8030d30 con 0x7f5ed00af840 2024-01-24T17:46:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.915+0000 7f5ed7200640 1 -- 172.21.15.39:0/3905196383 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed00af840 msgr2=0x7f5ed006b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.915+0000 7f5ed7200640 1 --2- 172.21.15.39:0/3905196383 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed00af840 0x7f5ed006b690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009870 tx=0x7f5eb802b980 comp rx=0 tx=0).stop 2024-01-24T17:46:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.916+0000 7f5ed7200640 1 -- 172.21.15.39:0/3905196383 shutdown_connections 2024-01-24T17:46:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.916+0000 7f5ed7200640 1 --2- 172.21.15.39:0/3905196383 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed00af840 0x7f5ed006b690 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.916+0000 7f5ed7200640 1 --2- 172.21.15.39:0/3905196383 >> v2:172.21.15.100:3300/0 conn(0x7f5ed015ccd0 0x7f5ed00d1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.916+0000 7f5ed7200640 1 -- 172.21.15.39:0/3905196383 >> 172.21.15.39:0/3905196383 conn(0x7f5ed00d0770 msgr2=0x7f5ed00d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.916+0000 7f5ed7200640 1 -- 172.21.15.39:0/3905196383 shutdown_connections 2024-01-24T17:46:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.916+0000 7f5ed7200640 1 -- 172.21.15.39:0/3905196383 wait complete. 2024-01-24T17:46:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.917+0000 7f5ed7200640 1 Processor -- start 2024-01-24T17:46:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.918+0000 7f5ed7200640 1 -- start start 2024-01-24T17:46:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.918+0000 7f5ed7200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ed006bbd0 0x7f5ed00b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed00af840 0x7f5ed0151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed015ccd0 0x7f5ed006e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00b2010 con 0x7f5ed00af840 2024-01-24T17:46:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed7200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00b22f0 con 0x7f5ed006bbd0 2024-01-24T17:46:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00b25d0 con 0x7f5ed015ccd0 2024-01-24T17:46:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed00af840 0x7f5ed0151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed00af840 0x7f5ed0151330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36118/0 (socket says 172.21.15.39:36118) 2024-01-24T17:46:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5400640 1 -- 172.21.15.39:0/358094635 learned_addr learned my addr 172.21.15.39:0/358094635 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5e00640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ed006bbd0 0x7f5ed00b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed6800640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed015ccd0 0x7f5ed006e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5400640 1 -- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed015ccd0 msgr2=0x7f5ed006e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5400640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed015ccd0 0x7f5ed006e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5400640 1 -- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ed006bbd0 msgr2=0x7f5ed00b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5400640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ed006bbd0 0x7f5ed00b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5400640 1 -- 172.21.15.39:0/358094635 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec00093a0 con 0x7f5ed00af840 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed6800640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed015ccd0 0x7f5ed006e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.919+0000 7f5ed5e00640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ed006bbd0 0x7f5ed00b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.920+0000 7f5ed5400640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed00af840 0x7f5ed0151330 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009870 tx=0x7f5eb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.920+0000 7f5ece000640 1 -- 172.21.15.39:0/358094635 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8039a90 con 0x7f5ed00af840 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.920+0000 7f5ece000640 1 -- 172.21.15.39:0/358094635 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb8039bf0 con 0x7f5ed00af840 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.920+0000 7f5ece000640 1 -- 172.21.15.39:0/358094635 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb80433f0 con 0x7f5ed00af840 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.920+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb80094d0 con 0x7f5ed00af840 2024-01-24T17:46:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.920+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed00b2b90 con 0x7f5ed00af840 2024-01-24T17:46:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.921+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed0061880 con 0x7f5ed00af840 2024-01-24T17:46:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.922+0000 7f5ece000640 1 -- 172.21.15.39:0/358094635 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5eb8042390 con 0x7f5ed00af840 2024-01-24T17:46:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.923+0000 7f5ece000640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ea806a430 0x7f5ea806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.923+0000 7f5ece000640 1 -- 172.21.15.39:0/358094635 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5eb80aaae0 con 0x7f5ed00af840 2024-01-24T17:46:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.924+0000 7f5ed5e00640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ea806a430 0x7f5ea806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:12.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.925+0000 7f5ed5e00640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ea806a430 0x7f5ea806c8f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0009ba0 tx=0x7f5ec0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:12.928+0000 7f5ece000640 1 -- 172.21.15.39:0/358094635 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5eb8076040 con 0x7f5ed00af840 2024-01-24T17:46:13.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.124+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ed015c2f0 con 0x7f5ed00af840 2024-01-24T17:46:13.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.126+0000 7f5ece000640 1 -- 172.21.15.39:0/358094635 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f5eb8079660 con 0x7f5ed00af840 2024-01-24T17:46:13.127 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:13.127 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:13.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ea806a430 msgr2=0x7f5ea806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ea806a430 0x7f5ea806c8f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0009ba0 tx=0x7f5ec0008040 comp rx=0 tx=0).stop 2024-01-24T17:46:13.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed00af840 msgr2=0x7f5ed0151330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed00af840 0x7f5ed0151330 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009870 tx=0x7f5eb8004730 comp rx=0 tx=0).stop 2024-01-24T17:46:13.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 shutdown_connections 2024-01-24T17:46:13.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5ed015ccd0 0x7f5ed006e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5ea806a430 0x7f5ea806c8f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ed00af840 0x7f5ed0151330 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 --2- 172.21.15.39:0/358094635 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5ed006bbd0 0x7f5ed00b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 >> 172.21.15.39:0/358094635 conn(0x7f5ed00d0770 msgr2=0x7f5ed01594b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:13.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 shutdown_connections 2024-01-24T17:46:13.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.129+0000 7f5ed7200640 1 -- 172.21.15.39:0/358094635 wait complete. 2024-01-24T17:46:13.148 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:45:16.503215+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:45:17.386565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:46:13.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:46:13.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.373+0000 7ff429c00640 1 Processor -- start 2024-01-24T17:46:13.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.374+0000 7ff429c00640 1 -- start start 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.375+0000 7ff429c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff42415ce60 0x7ff4240d3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.375+0000 7ff429c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4241609c0 0x7ff424160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.375+0000 7ff429c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4240d10f0 con 0x7ff4240d1580 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.375+0000 7ff429c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff424161370 con 0x7ff42415ce60 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.375+0000 7ff429c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240ce9c0 con 0x7ff4241609c0 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.376+0000 7ff422a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff42415ce60 0x7ff4240d3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.376+0000 7ff422a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff42415ce60 0x7ff4240d3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37970/0 (socket says 172.21.15.39:37970) 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.376+0000 7ff422a00640 1 -- 172.21.15.39:0/2222672679 learned_addr learned my addr 172.21.15.39:0/2222672679 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.376+0000 7ff423400640 1 --2- 172.21.15.39:0/2222672679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4241609c0 0x7ff424160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.376+0000 7ff422a00640 1 -- 172.21.15.39:0/2222672679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4241609c0 msgr2=0x7ff424160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:13.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.376+0000 7ff422a00640 1 --2- 172.21.15.39:0/2222672679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4241609c0 0x7ff424160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.376+0000 7ff422a00640 1 -- 172.21.15.39:0/2222672679 >> v1:172.21.15.100:6789/0 conn(0x7ff4240d1580 legacy=0x7ff4240ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.376+0000 7ff422a00640 1 -- 172.21.15.39:0/2222672679 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240d07c0 con 0x7ff42415ce60 2024-01-24T17:46:13.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.378+0000 7ff422a00640 1 --2- 172.21.15.39:0/2222672679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42415ce60 0x7ff4240d3700 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff4140098a0 tx=0x7ff41402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be2af264f28155b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:13.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.378+0000 7ff422000640 1 -- 172.21.15.39:0/2222672679 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414030840 con 0x7ff42415ce60 2024-01-24T17:46:13.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.378+0000 7ff422000640 1 -- 172.21.15.39:0/2222672679 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4140309a0 con 0x7ff42415ce60 2024-01-24T17:46:13.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.378+0000 7ff422000640 1 -- 172.21.15.39:0/2222672679 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414030d30 con 0x7ff42415ce60 2024-01-24T17:46:13.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.379+0000 7ff429c00640 1 -- 172.21.15.39:0/2222672679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42415ce60 msgr2=0x7ff4240d3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.379+0000 7ff429c00640 1 --2- 172.21.15.39:0/2222672679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42415ce60 0x7ff4240d3700 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff4140098a0 tx=0x7ff41402b980 comp rx=0 tx=0).stop 2024-01-24T17:46:13.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.379+0000 7ff429c00640 1 -- 172.21.15.39:0/2222672679 shutdown_connections 2024-01-24T17:46:13.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.379+0000 7ff429c00640 1 --2- 172.21.15.39:0/2222672679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff4241609c0 0x7ff424160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.379+0000 7ff429c00640 1 --2- 172.21.15.39:0/2222672679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42415ce60 0x7ff4240d3700 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.379+0000 7ff429c00640 1 -- 172.21.15.39:0/2222672679 >> 172.21.15.39:0/2222672679 conn(0x7ff4240d2b50 msgr2=0x7ff4240d2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:13.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.379+0000 7ff429c00640 1 -- 172.21.15.39:0/2222672679 shutdown_connections 2024-01-24T17:46:13.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.379+0000 7ff429c00640 1 -- 172.21.15.39:0/2222672679 wait complete. 2024-01-24T17:46:13.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.381+0000 7ff429c00640 1 Processor -- start 2024-01-24T17:46:13.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.382+0000 7ff429c00640 1 -- start start 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff429c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4240d1580 0x7ff4240b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff429c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff42415ce60 0x7ff42415fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff429c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4241609c0 0x7ff4240b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff429c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4241587e0 con 0x7ff4240d1580 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff429c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff424160160 con 0x7ff4241609c0 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff429c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240cae80 con 0x7ff42415ce60 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff422a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4240d1580 0x7ff4240b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff422a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4240d1580 0x7ff4240b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37976/0 (socket says 172.21.15.39:37976) 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.383+0000 7ff422a00640 1 -- 172.21.15.39:0/3040434903 learned_addr learned my addr 172.21.15.39:0/3040434903 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.384+0000 7ff41be00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff42415ce60 0x7ff42415fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.384+0000 7ff423400640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4241609c0 0x7ff4240b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.384+0000 7ff422a00640 1 -- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff42415ce60 msgr2=0x7ff42415fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.384+0000 7ff422a00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff42415ce60 0x7ff42415fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.384+0000 7ff422a00640 1 -- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4241609c0 msgr2=0x7ff4240b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.384+0000 7ff422a00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4241609c0 0x7ff4240b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.384+0000 7ff422a00640 1 -- 172.21.15.39:0/3040434903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4140094d0 con 0x7ff4240d1580 2024-01-24T17:46:13.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.384+0000 7ff422a00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4240d1580 0x7ff4240b25b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff414000c00 tx=0x7ff414004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:13.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.385+0000 7ff41aa00640 1 -- 172.21.15.39:0/3040434903 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414047070 con 0x7ff4240d1580 2024-01-24T17:46:13.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.385+0000 7ff41aa00640 1 -- 172.21.15.39:0/3040434903 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff414039a90 con 0x7ff4240d1580 2024-01-24T17:46:13.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.385+0000 7ff41aa00640 1 -- 172.21.15.39:0/3040434903 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414039e40 con 0x7ff4240d1580 2024-01-24T17:46:13.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.385+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4240cb160 con 0x7ff4240d1580 2024-01-24T17:46:13.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.385+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4240cb720 con 0x7ff4240d1580 2024-01-24T17:46:13.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.386+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff424067360 con 0x7ff4240d1580 2024-01-24T17:46:13.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.387+0000 7ff41aa00640 1 -- 172.21.15.39:0/3040434903 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff41404c410 con 0x7ff4240d1580 2024-01-24T17:46:13.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.388+0000 7ff41aa00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3fc06a430 0x7ff3fc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.388+0000 7ff41aa00640 1 -- 172.21.15.39:0/3040434903 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff41403e070 con 0x7ff4240d1580 2024-01-24T17:46:13.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.389+0000 7ff41be00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3fc06a430 0x7ff3fc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.390+0000 7ff41be00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3fc06a430 0x7ff3fc06c8f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff4240d0400 tx=0x7ff40c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:13.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.393+0000 7ff41aa00640 1 -- 172.21.15.39:0/3040434903 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff414076570 con 0x7ff4240d1580 2024-01-24T17:46:13.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.584+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff42415c610 con 0x7ff4240d1580 2024-01-24T17:46:13.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.585+0000 7ff41aa00640 1 -- 172.21.15.39:0/3040434903 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff414079b90 con 0x7ff4240d1580 2024-01-24T17:46:13.586 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:46:13.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.588+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3fc06a430 msgr2=0x7ff3fc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.588+0000 7ff429c00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3fc06a430 0x7ff3fc06c8f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff4240d0400 tx=0x7ff40c009040 comp rx=0 tx=0).stop 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.588+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4240d1580 msgr2=0x7ff4240b25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.588+0000 7ff429c00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4240d1580 0x7ff4240b25b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7ff414000c00 tx=0x7ff414004730 comp rx=0 tx=0).stop 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.589+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 shutdown_connections 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.589+0000 7ff429c00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4241609c0 0x7ff4240b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.589+0000 7ff429c00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff42415ce60 0x7ff42415fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.589+0000 7ff429c00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3fc06a430 0x7ff3fc06c8f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.589+0000 7ff429c00640 1 --2- 172.21.15.39:0/3040434903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4240d1580 0x7ff4240b25b0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.589+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 >> 172.21.15.39:0/3040434903 conn(0x7ff4240d2b50 msgr2=0x7ff42415af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.589+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 shutdown_connections 2024-01-24T17:46:13.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.589+0000 7ff429c00640 1 -- 172.21.15.39:0/3040434903 wait complete. 2024-01-24T17:46:13.609 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:46:13.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.826+0000 7fdecf200640 1 Processor -- start 2024-01-24T17:46:13.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.826+0000 7fdecf200640 1 -- start start 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecf200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdec815ae10 0x7fdec80ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecf200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec80c9d80 0x7fdec80c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecf200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdec80c2720 0x7fdec80ca470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecf200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec806aeb0 con 0x7fdec80c2720 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecf200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec806b190 con 0x7fdec815ae10 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecf200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec806b470 con 0x7fdec80c9d80 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecd200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdec80c2720 0x7fdec80ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecd200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdec80c2720 0x7fdec80ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37990/0 (socket says 172.21.15.39:37990) 2024-01-24T17:46:13.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.828+0000 7fdecd200640 1 -- 172.21.15.39:0/3268086974 learned_addr learned my addr 172.21.15.39:0/3268086974 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.829+0000 7fdec7e00640 1 --2- 172.21.15.39:0/3268086974 >> v2:172.21.15.100:3300/0 conn(0x7fdec815ae10 0x7fdec80ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.829+0000 7fdebf200640 1 --2- 172.21.15.39:0/3268086974 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec80c9d80 0x7fdec80c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.829+0000 7fdecd200640 1 -- 172.21.15.39:0/3268086974 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec80c9d80 msgr2=0x7fdec80c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.829+0000 7fdecd200640 1 --2- 172.21.15.39:0/3268086974 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec80c9d80 0x7fdec80c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.829+0000 7fdecd200640 1 -- 172.21.15.39:0/3268086974 >> v2:172.21.15.100:3300/0 conn(0x7fdec815ae10 msgr2=0x7fdec80ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.829+0000 7fdecd200640 1 --2- 172.21.15.39:0/3268086974 >> v2:172.21.15.100:3300/0 conn(0x7fdec815ae10 0x7fdec80ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.829+0000 7fdecd200640 1 -- 172.21.15.39:0/3268086974 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec806b750 con 0x7fdec80c2720 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.830+0000 7fdecd200640 1 --2- 172.21.15.39:0/3268086974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80c2720 0x7fdec80ca470 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fdec80719a0 tx=0x7fdec002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1d6e0185ce470d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.830+0000 7fdec7400640 1 -- 172.21.15.39:0/3268086974 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0031860 con 0x7fdec80c2720 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.830+0000 7fdec7400640 1 -- 172.21.15.39:0/3268086974 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec00319c0 con 0x7fdec80c2720 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.830+0000 7fdec7400640 1 -- 172.21.15.39:0/3268086974 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0031d50 con 0x7fdec80c2720 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 -- 172.21.15.39:0/3268086974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80c2720 msgr2=0x7fdec80ca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 --2- 172.21.15.39:0/3268086974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80c2720 0x7fdec80ca470 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fdec80719a0 tx=0x7fdec002b900 comp rx=0 tx=0).stop 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 -- 172.21.15.39:0/3268086974 shutdown_connections 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 --2- 172.21.15.39:0/3268086974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80c2720 0x7fdec80ca470 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 --2- 172.21.15.39:0/3268086974 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec80c9d80 0x7fdec80c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 --2- 172.21.15.39:0/3268086974 >> v2:172.21.15.100:3300/0 conn(0x7fdec815ae10 0x7fdec80ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 -- 172.21.15.39:0/3268086974 >> 172.21.15.39:0/3268086974 conn(0x7fdec80ce690 msgr2=0x7fdec80ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:13.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 -- 172.21.15.39:0/3268086974 shutdown_connections 2024-01-24T17:46:13.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.831+0000 7fdecf200640 1 -- 172.21.15.39:0/3268086974 wait complete. 2024-01-24T17:46:13.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.833+0000 7fdecf200640 1 Processor -- start 2024-01-24T17:46:13.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.833+0000 7fdecf200640 1 -- start start 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecf200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdec80c9d80 0x7fdec814d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecf200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec815ae10 0x7fdec81563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecf200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80b3f70 0x7fdec815bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecf200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec814dc30 con 0x7fdec80b3f70 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecf200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec8156ca0 con 0x7fdec80c9d80 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecf200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec8156f80 con 0x7fdec815ae10 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80b3f70 0x7fdec815bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80b3f70 0x7fdec815bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37992/0 (socket says 172.21.15.39:37992) 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.834+0000 7fdecd200640 1 -- 172.21.15.39:0/3616030058 learned_addr learned my addr 172.21.15.39:0/3616030058 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.835+0000 7fdec7e00640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdec80c9d80 0x7fdec814d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.835+0000 7fdebf200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec815ae10 0x7fdec81563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.835+0000 7fdecd200640 1 -- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec815ae10 msgr2=0x7fdec81563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.835+0000 7fdecd200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec815ae10 0x7fdec81563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.835+0000 7fdecd200640 1 -- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdec80c9d80 msgr2=0x7fdec814d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.835+0000 7fdecd200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdec80c9d80 0x7fdec814d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.835+0000 7fdecd200640 1 -- 172.21.15.39:0/3616030058 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec00094d0 con 0x7fdec80b3f70 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.835+0000 7fdecd200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80b3f70 0x7fdec815bb60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fdec00090b0 tx=0x7fdec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.836+0000 7fdec4c00640 1 -- 172.21.15.39:0/3616030058 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec003ad20 con 0x7fdec80b3f70 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.836+0000 7fdec4c00640 1 -- 172.21.15.39:0/3616030058 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec00443f0 con 0x7fdec80b3f70 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.836+0000 7fdec4c00640 1 -- 172.21.15.39:0/3616030058 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec003aea0 con 0x7fdec80b3f70 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.836+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec80b1680 con 0x7fdec80b3f70 2024-01-24T17:46:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.836+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec8156920 con 0x7fdec80b3f70 2024-01-24T17:46:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.837+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec80b14e0 con 0x7fdec80b3f70 2024-01-24T17:46:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.838+0000 7fdec4c00640 1 -- 172.21.15.39:0/3616030058 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdec004d450 con 0x7fdec80b3f70 2024-01-24T17:46:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.839+0000 7fdec4c00640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdea006a430 0x7fdea006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.839+0000 7fdec4c00640 1 -- 172.21.15.39:0/3616030058 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdec00aae70 con 0x7fdec80b3f70 2024-01-24T17:46:13.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.839+0000 7fdec7e00640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdea006a430 0x7fdea006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:13.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.840+0000 7fdec7e00640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdea006a430 0x7fdea006c8f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80099c0 tx=0x7fdeb8003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:13.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:13.843+0000 7fdec4c00640 1 -- 172.21.15.39:0/3616030058 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdec80b14e0 con 0x7fdec80b3f70 2024-01-24T17:46:14.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.040+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdec80b1ba0 con 0x7fdec80b3f70 2024-01-24T17:46:14.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.041+0000 7fdec4c00640 1 -- 172.21.15.39:0/3616030058 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v63) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdec80b1ba0 con 0x7fdec80b3f70 2024-01-24T17:46:14.042 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:46:14.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdea006a430 msgr2=0x7fdea006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdea006a430 0x7fdea006c8f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80099c0 tx=0x7fdeb8003990 comp rx=0 tx=0).stop 2024-01-24T17:46:14.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80b3f70 msgr2=0x7fdec815bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80b3f70 0x7fdec815bb60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fdec00090b0 tx=0x7fdec00046c0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 shutdown_connections 2024-01-24T17:46:14.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdea006a430 0x7fdea006c8f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdec80b3f70 0x7fdec815bb60 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdec815ae10 0x7fdec81563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 --2- 172.21.15.39:0/3616030058 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdec80c9d80 0x7fdec814d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 >> 172.21.15.39:0/3616030058 conn(0x7fdec80ce690 msgr2=0x7fdec80c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:14.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.044+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 shutdown_connections 2024-01-24T17:46:14.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.045+0000 7fdecf200640 1 -- 172.21.15.39:0/3616030058 wait complete. 2024-01-24T17:46:14.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:46:14.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.287+0000 7ff956a00640 1 Processor -- start 2024-01-24T17:46:14.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.287+0000 7ff956a00640 1 -- start start 2024-01-24T17:46:14.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.288+0000 7ff956a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9500cf640 0x7ff9500d1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.288+0000 7ff956a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9500cbf90 con 0x7ff95015ac60 2024-01-24T17:46:14.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.288+0000 7ff956a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9500cebd0 con 0x7ff9500ce730 2024-01-24T17:46:14.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.288+0000 7ff956a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500cb940 con 0x7ff9500cf640 2024-01-24T17:46:14.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff95015ac60 0x7ff9500cc420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58772/0 (socket says 172.21.15.39:58772) 2024-01-24T17:46:14.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94fe00640 1 -- 172.21.15.39:0/2472667335 learned_addr learned my addr 172.21.15.39:0/2472667335 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:14.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff954a00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9500cf640 0x7ff9500d1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746869435 0 0) 0x7ff9500cbf90 con 0x7ff95015ac60 2024-01-24T17:46:14.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff930003520 con 0x7ff95015ac60 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825336009 0 0) 0x7ff930003520 con 0x7ff95015ac60 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9500cf640 msgr2=0x7ff9500d1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94ea00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9500cf640 0x7ff9500d1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 >> v1:172.21.15.100:6789/0 conn(0x7ff9500ce730 legacy=0x7ff9500cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.289+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9500cbc20 con 0x7ff95015ac60 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff9400019f0 con 0x7ff95015ac60 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 >> v1:172.21.15.39:6789/0 conn(0x7ff95015ac60 legacy=0x7ff9500cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94ea00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9300039a0 0x7ff930003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94ea00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300047c0 0x7ff930004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94ea00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9500cf640 0x7ff930005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500cb940 con 0x7ff9300039a0 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93000bb40 con 0x7ff9300047c0 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93000bd00 con 0x7ff9500cf640 2024-01-24T17:46:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94fe00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9300039a0 0x7ff930003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff94f400640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300047c0 0x7ff930004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.290+0000 7ff954a00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9500cf640 0x7ff930005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.291+0000 7ff94fe00640 1 -- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9500cf640 msgr2=0x7ff930005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:14.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.291+0000 7ff94fe00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9500cf640 0x7ff930005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.291+0000 7ff94fe00640 1 -- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300047c0 msgr2=0x7ff930004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:14.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.291+0000 7ff94fe00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300047c0 0x7ff930004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.291+0000 7ff94fe00640 1 -- 172.21.15.39:0/2472667335 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff930003520 con 0x7ff9300039a0 2024-01-24T17:46:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.292+0000 7ff94fe00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9300039a0 0x7ff930003dc0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff94000a760 tx=0x7ff9300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad10527d610ab60 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.292+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff940034630 con 0x7ff9300039a0 2024-01-24T17:46:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.292+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94002f070 con 0x7ff9300039a0 2024-01-24T17:46:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.292+0000 7ff94ea00640 1 -- 172.21.15.39:0/2472667335 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9400349d0 con 0x7ff9300039a0 2024-01-24T17:46:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.293+0000 7ff956a00640 1 -- 172.21.15.39:0/2472667335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9300039a0 msgr2=0x7ff930003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.293+0000 7ff956a00640 1 --2- 172.21.15.39:0/2472667335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9300039a0 0x7ff930003dc0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff94000a760 tx=0x7ff9300057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.293+0000 7ff954a00640 1 -- 172.21.15.39:0/2472667335 reap_dead start 2024-01-24T17:46:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.293+0000 7ff956a00640 1 -- 172.21.15.39:0/2472667335 shutdown_connections 2024-01-24T17:46:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.293+0000 7ff956a00640 1 -- 172.21.15.39:0/2472667335 >> 172.21.15.39:0/2472667335 conn(0x7ff9500d0950 msgr2=0x7ff9500d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.293+0000 7ff956a00640 1 -- 172.21.15.39:0/2472667335 shutdown_connections 2024-01-24T17:46:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.293+0000 7ff956a00640 1 -- 172.21.15.39:0/2472667335 wait complete. 2024-01-24T17:46:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.295+0000 7ff956a00640 1 Processor -- start 2024-01-24T17:46:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.295+0000 7ff956a00640 1 -- start start 2024-01-24T17:46:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff956a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300039a0 0x7ff9500c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff956a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9501468a0 0x7ff9500c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff956a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950146d00 0x7ff9500c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff956a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500b0690 con 0x7ff950146d00 2024-01-24T17:46:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff956a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500c9f50 con 0x7ff9300039a0 2024-01-24T17:46:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff956a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500b4bf0 con 0x7ff9501468a0 2024-01-24T17:46:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff954a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950146d00 0x7ff9500c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff954a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950146d00 0x7ff9500c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38000/0 (socket says 172.21.15.39:38000) 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.296+0000 7ff954a00640 1 -- 172.21.15.39:0/2693210799 learned_addr learned my addr 172.21.15.39:0/2693210799 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff94fe00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300039a0 0x7ff9500c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff94f400640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9501468a0 0x7ff9500c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff954a00640 1 -- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9501468a0 msgr2=0x7ff9500c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff954a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9501468a0 0x7ff9500c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff954a00640 1 -- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300039a0 msgr2=0x7ff9500c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff954a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300039a0 0x7ff9500c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff954a00640 1 -- 172.21.15.39:0/2693210799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9500b4ed0 con 0x7ff950146d00 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff94fe00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300039a0 0x7ff9500c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff954a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950146d00 0x7ff9500c9a10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7ff9500caf50 tx=0x7ff94800f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:14.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff947e00640 1 -- 172.21.15.39:0/2693210799 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948004440 con 0x7ff950146d00 2024-01-24T17:46:14.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff947e00640 1 -- 172.21.15.39:0/2693210799 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff948003610 con 0x7ff950146d00 2024-01-24T17:46:14.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.297+0000 7ff947e00640 1 -- 172.21.15.39:0/2693210799 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9480039c0 con 0x7ff950146d00 2024-01-24T17:46:14.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.298+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9500b51b0 con 0x7ff950146d00 2024-01-24T17:46:14.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.298+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9500b2b30 con 0x7ff950146d00 2024-01-24T17:46:14.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.299+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9500b0840 con 0x7ff950146d00 2024-01-24T17:46:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.300+0000 7ff947e00640 1 -- 172.21.15.39:0/2693210799 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff948003b20 con 0x7ff950146d00 2024-01-24T17:46:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.301+0000 7ff947e00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff92806a430 0x7ff92806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.301+0000 7ff947e00640 1 -- 172.21.15.39:0/2693210799 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9480885d0 con 0x7ff950146d00 2024-01-24T17:46:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.301+0000 7ff94fe00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff92806a430 0x7ff92806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.303+0000 7ff94fe00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff92806a430 0x7ff92806c8f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff940034000 tx=0x7ff94002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:14.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.306+0000 7ff947e00640 1 -- 172.21.15.39:0/2693210799 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff9500b0840 con 0x7ff950146d00 2024-01-24T17:46:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.508+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff9500b0480 con 0x7ff950146d00 2024-01-24T17:46:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.509+0000 7ff947e00640 1 -- 172.21.15.39:0/2693210799 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v63) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff948000e60 con 0x7ff950146d00 2024-01-24T17:46:14.510 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff92806a430 msgr2=0x7ff92806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff92806a430 0x7ff92806c8f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff940034000 tx=0x7ff94002f000 comp rx=0 tx=0).stop 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950146d00 msgr2=0x7ff9500c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950146d00 0x7ff9500c9a10 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7ff9500caf50 tx=0x7ff94800f920 comp rx=0 tx=0).stop 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 shutdown_connections 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff92806a430 0x7ff92806c8f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff950146d00 0x7ff9500c9a10 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9501468a0 0x7ff9500c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 --2- 172.21.15.39:0/2693210799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9300039a0 0x7ff9500c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 >> 172.21.15.39:0/2693210799 conn(0x7ff9500d0950 msgr2=0x7ff95006e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.511+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 shutdown_connections 2024-01-24T17:46:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.512+0000 7ff956a00640 1 -- 172.21.15.39:0/2693210799 wait complete. 2024-01-24T17:46:14.530 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:46:14.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.735+0000 7f51e1600640 1 Processor -- start 2024-01-24T17:46:14.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.735+0000 7f51e1600640 1 -- start start 2024-01-24T17:46:14.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.736+0000 7f51e1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f51dc15ccd0 0x7f51dc0b8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.736+0000 7f51e1600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f51dc0cdea0 0x7f51dc0d1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.736+0000 7f51e1600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51dc0d0d90 con 0x7f51dc1607a0 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.736+0000 7f51e1600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51dc0b5b60 con 0x7f51dc15ccd0 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.736+0000 7f51e1600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51dc15d0b0 con 0x7f51dc0cdea0 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.736+0000 7f51dbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f51dc15ccd0 0x7f51dc0b8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.736+0000 7f51dbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f51dc15ccd0 0x7f51dc0b8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38010/0 (socket says 172.21.15.39:38010) 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.736+0000 7f51dbe00640 1 -- 172.21.15.39:0/1747200202 learned_addr learned my addr 172.21.15.39:0/1747200202 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.737+0000 7f51e0c00640 1 --2- 172.21.15.39:0/1747200202 >> v2:172.21.15.100:3300/0 conn(0x7f51dc0cdea0 0x7f51dc0d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.737+0000 7f51dbe00640 1 -- 172.21.15.39:0/1747200202 >> v2:172.21.15.100:3300/0 conn(0x7f51dc0cdea0 msgr2=0x7f51dc0d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.737+0000 7f51dbe00640 1 --2- 172.21.15.39:0/1747200202 >> v2:172.21.15.100:3300/0 conn(0x7f51dc0cdea0 0x7f51dc0d1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.737+0000 7f51dbe00640 1 -- 172.21.15.39:0/1747200202 >> v1:172.21.15.100:6789/0 conn(0x7f51dc1607a0 legacy=0x7f51dc0cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.737+0000 7f51dbe00640 1 -- 172.21.15.39:0/1747200202 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc160f80 con 0x7f51dc15ccd0 2024-01-24T17:46:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.737+0000 7f51e0c00640 1 --2- 172.21.15.39:0/1747200202 >> v2:172.21.15.100:3300/0 conn(0x7f51dc0cdea0 0x7f51dc0d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:14.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.738+0000 7f51dbe00640 1 --2- 172.21.15.39:0/1747200202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc15ccd0 0x7f51dc0b8570 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0b5ab0 tx=0x7f51cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e6e2b28ec24d0c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.738+0000 7f51daa00640 1 -- 172.21.15.39:0/1747200202 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc030860 con 0x7f51dc15ccd0 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.738+0000 7f51daa00640 1 -- 172.21.15.39:0/1747200202 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc0309c0 con 0x7f51dc15ccd0 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.738+0000 7f51daa00640 1 -- 172.21.15.39:0/1747200202 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc030d50 con 0x7f51dc15ccd0 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.739+0000 7f51e1600640 1 -- 172.21.15.39:0/1747200202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc15ccd0 msgr2=0x7f51dc0b8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.739+0000 7f51e1600640 1 --2- 172.21.15.39:0/1747200202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc15ccd0 0x7f51dc0b8570 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0b5ab0 tx=0x7f51cc02b9a0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.739+0000 7f51e1600640 1 -- 172.21.15.39:0/1747200202 shutdown_connections 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.739+0000 7f51e1600640 1 --2- 172.21.15.39:0/1747200202 >> v2:172.21.15.100:3300/0 conn(0x7f51dc0cdea0 0x7f51dc0d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.739+0000 7f51e1600640 1 --2- 172.21.15.39:0/1747200202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc15ccd0 0x7f51dc0b8570 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.739+0000 7f51e1600640 1 -- 172.21.15.39:0/1747200202 >> 172.21.15.39:0/1747200202 conn(0x7f51dc0cf440 msgr2=0x7f51dc0cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.739+0000 7f51e1600640 1 -- 172.21.15.39:0/1747200202 shutdown_connections 2024-01-24T17:46:14.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.739+0000 7f51e1600640 1 -- 172.21.15.39:0/1747200202 wait complete. 2024-01-24T17:46:14.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.741+0000 7f51e1600640 1 Processor -- start 2024-01-24T17:46:14.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.741+0000 7f51e1600640 1 -- start start 2024-01-24T17:46:14.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e1600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51dc0cdea0 0x7f51dc150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e1600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51dc15ccd0 0x7f51dc15fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc1607a0 0x7f51dc0b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e1600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc151250 con 0x7f51dc1607a0 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e1600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc1600b0 con 0x7f51dc0cdea0 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e1600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc15ead0 con 0x7f51dc15ccd0 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc1607a0 0x7f51dc0b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc1607a0 0x7f51dc0b3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38012/0 (socket says 172.21.15.39:38012) 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.742+0000 7f51e0c00640 1 -- 172.21.15.39:0/2657204585 learned_addr learned my addr 172.21.15.39:0/2657204585 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51dbe00640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51dc0cdea0 0x7f51dc150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51db400640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51dc15ccd0 0x7f51dc15fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51e0c00640 1 -- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51dc15ccd0 msgr2=0x7f51dc15fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51e0c00640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51dc15ccd0 0x7f51dc15fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51e0c00640 1 -- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51dc0cdea0 msgr2=0x7f51dc150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51e0c00640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51dc0cdea0 0x7f51dc150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51e0c00640 1 -- 172.21.15.39:0/2657204585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc0094d0 con 0x7f51dc1607a0 2024-01-24T17:46:14.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51dbe00640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51dc0cdea0 0x7f51dc150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:14.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51e0c00640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc1607a0 0x7f51dc0b3570 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f51d400cd80 tx=0x7f51d400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:14.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.743+0000 7f51d3e00640 1 -- 172.21.15.39:0/2657204585 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4003b90 con 0x7f51dc1607a0 2024-01-24T17:46:14.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.744+0000 7f51d3e00640 1 -- 172.21.15.39:0/2657204585 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d4003d10 con 0x7f51dc1607a0 2024-01-24T17:46:14.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.744+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51dc15edb0 con 0x7f51dc1607a0 2024-01-24T17:46:14.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.744+0000 7f51d3e00640 1 -- 172.21.15.39:0/2657204585 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4004260 con 0x7f51dc1607a0 2024-01-24T17:46:14.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.744+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc15f370 con 0x7f51dc1607a0 2024-01-24T17:46:14.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.745+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51dc0c6270 con 0x7f51dc1607a0 2024-01-24T17:46:14.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.746+0000 7f51d3e00640 1 -- 172.21.15.39:0/2657204585 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f51d400b040 con 0x7f51dc1607a0 2024-01-24T17:46:14.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.747+0000 7f51d3e00640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f51b406a430 0x7f51b406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:14.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.747+0000 7f51d3e00640 1 -- 172.21.15.39:0/2657204585 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51d4089170 con 0x7f51dc1607a0 2024-01-24T17:46:14.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.748+0000 7f51dbe00640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f51b406a430 0x7f51b406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:14.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.749+0000 7f51dbe00640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f51b406a430 0x7f51b406c8f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f51cc036000 tx=0x7f51cc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:14.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.752+0000 7f51d3e00640 1 -- 172.21.15.39:0/2657204585 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f51d4054630 con 0x7f51dc1607a0 2024-01-24T17:46:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.974+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51dc061880 con 0x7f51dc1607a0 2024-01-24T17:46:14.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.975+0000 7f51d3e00640 1 -- 172.21.15.39:0/2657204585 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f51d401d090 con 0x7f51dc1607a0 2024-01-24T17:46:14.976 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:14.976 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3045325513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3045325513},{"type":"v1","addr":"172.21.15.100:6836","nonce":3045325513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5126,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/858868211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":858868211},{"type":"v1","addr":"172.21.15.100:6837","nonce":858868211}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3121826441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3121826441},{"type":"v1","addr":"172.21.15.39:6833","nonce":3121826441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:45:18.381640+0000","modified":"2024-01-24T17:45:21.392053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6261":{"gid":6261,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/527429915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":527429915},{"type":"v1","addr":"172.21.15.39:6835","nonce":527429915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-24T17:46:14.977 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-01-24T17:46:14.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f51b406a430 msgr2=0x7f51b406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f51b406a430 0x7f51b406c8f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f51cc036000 tx=0x7f51cc03b040 comp rx=0 tx=0).stop 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc1607a0 msgr2=0x7f51dc0b3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc1607a0 0x7f51dc0b3570 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f51d400cd80 tx=0x7f51d400a650 comp rx=0 tx=0).stop 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 shutdown_connections 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f51b406a430 0x7f51b406c8f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51dc1607a0 0x7f51dc0b3570 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f51dc15ccd0 0x7f51dc15fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 --2- 172.21.15.39:0/2657204585 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f51dc0cdea0 0x7f51dc150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 >> 172.21.15.39:0/2657204585 conn(0x7f51dc0cf440 msgr2=0x7f51dc15b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 shutdown_connections 2024-01-24T17:46:14.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:14.978+0000 7f51e1600640 1 -- 172.21.15.39:0/2657204585 wait complete. 2024-01-24T17:46:14.997 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:15.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.212+0000 7f78a2a00640 1 Processor -- start 2024-01-24T17:46:15.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.212+0000 7f78a2a00640 1 -- start start 2024-01-24T17:46:15.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.213+0000 7f78a2a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142e30 0x7f789c0b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.213+0000 7f78a2a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f789c071290 0x7f789c06f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.213+0000 7f78a2a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c142b20 con 0x7f789c142740 2024-01-24T17:46:15.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f78a2a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c15d4c0 con 0x7f789c071290 2024-01-24T17:46:15.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f78a2a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0b2610 con 0x7f789c142e30 2024-01-24T17:46:15.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f78a0a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f789c071290 0x7f789c06f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f789be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f789c142740 0x7f789c070b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58776/0 (socket says 172.21.15.39:58776) 2024-01-24T17:46:15.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f789be00640 1 -- 172.21.15.39:0/750820165 learned_addr learned my addr 172.21.15.39:0/750820165 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:15.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f7893e00640 1 --2- 172.21.15.39:0/750820165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142e30 0x7f789c0b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f789b400640 1 -- 172.21.15.39:0/750820165 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171881400 0 0) 0x7f788c001110 con 0x7f789c142740 2024-01-24T17:46:15.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f78a0a00640 1 -- 172.21.15.39:0/750820165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142e30 msgr2=0x7f789c0b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:15.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f78a0a00640 1 --2- 172.21.15.39:0/750820165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142e30 0x7f789c0b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f78a0a00640 1 -- 172.21.15.39:0/750820165 >> v1:172.21.15.39:6789/0 conn(0x7f789c142740 legacy=0x7f789c070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.214+0000 7f78a0a00640 1 -- 172.21.15.39:0/750820165 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c15d710 con 0x7f789c071290 2024-01-24T17:46:15.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.216+0000 7f78a0a00640 1 --2- 172.21.15.39:0/750820165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c06f5b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f7894009d90 tx=0x7f789402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e01ff3e72328d10 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:15.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.216+0000 7f789b400640 1 -- 172.21.15.39:0/750820165 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789402cbc0 con 0x7f789c071290 2024-01-24T17:46:15.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.216+0000 7f789b400640 1 -- 172.21.15.39:0/750820165 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789402cd20 con 0x7f789c071290 2024-01-24T17:46:15.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.216+0000 7f789b400640 1 -- 172.21.15.39:0/750820165 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894034660 con 0x7f789c071290 2024-01-24T17:46:15.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.216+0000 7f78a2a00640 1 -- 172.21.15.39:0/750820165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 msgr2=0x7f789c06f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.216+0000 7f78a2a00640 1 --2- 172.21.15.39:0/750820165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c06f5b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f7894009d90 tx=0x7f789402be10 comp rx=0 tx=0).stop 2024-01-24T17:46:15.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.217+0000 7f78a2a00640 1 -- 172.21.15.39:0/750820165 shutdown_connections 2024-01-24T17:46:15.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.217+0000 7f78a2a00640 1 --2- 172.21.15.39:0/750820165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c06f5b0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.217+0000 7f78a2a00640 1 --2- 172.21.15.39:0/750820165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142e30 0x7f789c0b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.217+0000 7f78a2a00640 1 -- 172.21.15.39:0/750820165 >> 172.21.15.39:0/750820165 conn(0x7f789c069e50 msgr2=0x7f789c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:15.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.217+0000 7f78a2a00640 1 -- 172.21.15.39:0/750820165 shutdown_connections 2024-01-24T17:46:15.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.217+0000 7f78a2a00640 1 -- 172.21.15.39:0/750820165 wait complete. 2024-01-24T17:46:15.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.218+0000 7f78a2a00640 1 Processor -- start 2024-01-24T17:46:15.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.219+0000 7f78a2a00640 1 -- start start 2024-01-24T17:46:15.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.219+0000 7f78a2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c13e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.219+0000 7f78a2a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142740 0x7f789c141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.219+0000 7f78a2a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f789c142e30 0x7f789c06b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f78a2a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c1402c0 con 0x7f789c071290 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f78a2a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c140560 con 0x7f789c142e30 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f78a2a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c140840 con 0x7f789c142740 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f789be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c13e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f789be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c13e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38038/0 (socket says 172.21.15.39:38038) 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f789be00640 1 -- 172.21.15.39:0/4047802907 learned_addr learned my addr 172.21.15.39:0/4047802907 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f789be00640 1 -- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142740 msgr2=0x7f789c141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f789be00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142740 0x7f789c141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f789be00640 1 -- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f789c142e30 msgr2=0x7f789c06b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f78a0a00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f789c142e30 0x7f789c06b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f789be00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f789c142e30 0x7f789c06b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.220+0000 7f789be00640 1 -- 172.21.15.39:0/4047802907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7894009a70 con 0x7f789c071290 2024-01-24T17:46:15.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.221+0000 7f789be00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c13e000 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c780 tx=0x7f788c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:15.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.221+0000 7f7898c00640 1 -- 172.21.15.39:0/4047802907 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c0072d0 con 0x7f789c071290 2024-01-24T17:46:15.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.221+0000 7f7898c00640 1 -- 172.21.15.39:0/4047802907 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c007450 con 0x7f789c071290 2024-01-24T17:46:15.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.221+0000 7f7898c00640 1 -- 172.21.15.39:0/4047802907 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c0047e0 con 0x7f789c071290 2024-01-24T17:46:15.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.221+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c140b20 con 0x7f789c071290 2024-01-24T17:46:15.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.221+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c13e540 con 0x7f789c071290 2024-01-24T17:46:15.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.222+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c15df90 con 0x7f789c071290 2024-01-24T17:46:15.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.223+0000 7f7898c00640 1 -- 172.21.15.39:0/4047802907 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f788c004960 con 0x7f789c071290 2024-01-24T17:46:15.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.224+0000 7f7898c00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787406a430 0x7f787406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.224+0000 7f7898c00640 1 -- 172.21.15.39:0/4047802907 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f788c087ad0 con 0x7f789c071290 2024-01-24T17:46:15.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.225+0000 7f7893e00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787406a430 0x7f787406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.226+0000 7f7893e00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787406a430 0x7f787406c8f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f789c15bdb0 tx=0x7f7884008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:15.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.228+0000 7f7898c00640 1 -- 172.21.15.39:0/4047802907 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f789c15df90 con 0x7f789c071290 2024-01-24T17:46:15.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.423+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f789c141040 con 0x7f789c071290 2024-01-24T17:46:15.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.424+0000 7f7898c00640 1 -- 172.21.15.39:0/4047802907 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16097 (secure 0 0 0) 0x7f789c141040 con 0x7f789c071290 2024-01-24T17:46:15.426 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:15.427 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":63,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:45:19.385107+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:15.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787406a430 msgr2=0x7f787406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787406a430 0x7f787406c8f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f789c15bdb0 tx=0x7f7884008040 comp rx=0 tx=0).stop 2024-01-24T17:46:15.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 msgr2=0x7f789c13e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c13e000 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c780 tx=0x7f788c00cc40 comp rx=0 tx=0).stop 2024-01-24T17:46:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 shutdown_connections 2024-01-24T17:46:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f789c142e30 0x7f789c06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f789c142740 0x7f789c141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f787406a430 0x7f787406c8f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 --2- 172.21.15.39:0/4047802907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f789c071290 0x7f789c13e000 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 >> 172.21.15.39:0/4047802907 conn(0x7f789c069e50 msgr2=0x7f789c06a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 shutdown_connections 2024-01-24T17:46:15.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.427+0000 7f78a2a00640 1 -- 172.21.15.39:0/4047802907 wait complete. 2024-01-24T17:46:15.446 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-24T17:46:15.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:46:15.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.663+0000 7fe119200640 1 Processor -- start 2024-01-24T17:46:15.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.663+0000 7fe119200640 1 -- start start 2024-01-24T17:46:15.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.664+0000 7fe119200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe11406a380 0x7fe1140cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.664+0000 7fe119200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11415d080 con 0x7fe1140b1b40 2024-01-24T17:46:15.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.664+0000 7fe119200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe114160930 con 0x7fe11415cca0 2024-01-24T17:46:15.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.664+0000 7fe119200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1140ccf40 con 0x7fe11406a380 2024-01-24T17:46:15.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.664+0000 7fe112a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe11406a380 0x7fe1140cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.664+0000 7fe112a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe11406a380 0x7fe1140cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38042/0 (socket says 172.21.15.39:38042) 2024-01-24T17:46:15.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.664+0000 7fe112a00640 1 -- 172.21.15.39:0/1495456873 learned_addr learned my addr 172.21.15.39:0/1495456873 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:15.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.665+0000 7fe112a00640 1 -- 172.21.15.39:0/1495456873 >> v1:172.21.15.100:6789/0 conn(0x7fe11415cca0 legacy=0x7fe114069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:15.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.665+0000 7fe112a00640 1 -- 172.21.15.39:0/1495456873 >> v1:172.21.15.39:6789/0 conn(0x7fe1140b1b40 legacy=0x7fe11406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.665+0000 7fe112a00640 1 -- 172.21.15.39:0/1495456873 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114160c10 con 0x7fe11406a380 2024-01-24T17:46:15.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.665+0000 7fe112000640 1 -- 172.21.15.39:0/1495456873 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122607060 0 0) 0x7fe11415d080 con 0x7fe1140b1b40 2024-01-24T17:46:15.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.666+0000 7fe112a00640 1 --2- 172.21.15.39:0/1495456873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11406a380 0x7fe1140cca00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009870 tx=0x7fe0fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b980b4e87c7f5e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:15.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.666+0000 7fe112000640 1 -- 172.21.15.39:0/1495456873 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc031870 con 0x7fe11406a380 2024-01-24T17:46:15.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.666+0000 7fe112000640 1 -- 172.21.15.39:0/1495456873 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc0319d0 con 0x7fe11406a380 2024-01-24T17:46:15.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.666+0000 7fe112000640 1 -- 172.21.15.39:0/1495456873 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc031d60 con 0x7fe11406a380 2024-01-24T17:46:15.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.667+0000 7fe119200640 1 -- 172.21.15.39:0/1495456873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11406a380 msgr2=0x7fe1140cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.667+0000 7fe119200640 1 --2- 172.21.15.39:0/1495456873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11406a380 0x7fe1140cca00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009870 tx=0x7fe0fc02b910 comp rx=0 tx=0).stop 2024-01-24T17:46:15.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.667+0000 7fe119200640 1 -- 172.21.15.39:0/1495456873 shutdown_connections 2024-01-24T17:46:15.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.667+0000 7fe119200640 1 --2- 172.21.15.39:0/1495456873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11406a380 0x7fe1140cca00 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.667+0000 7fe119200640 1 -- 172.21.15.39:0/1495456873 >> 172.21.15.39:0/1495456873 conn(0x7fe11406ad80 msgr2=0x7fe11406b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:15.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.667+0000 7fe119200640 1 -- 172.21.15.39:0/1495456873 shutdown_connections 2024-01-24T17:46:15.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.667+0000 7fe119200640 1 -- 172.21.15.39:0/1495456873 wait complete. 2024-01-24T17:46:15.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.669+0000 7fe119200640 1 Processor -- start 2024-01-24T17:46:15.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.669+0000 7fe119200640 1 -- start start 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe119200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe11406a380 0x7fe1140b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe119200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1140b1b40 0x7fe1140b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe119200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11415cca0 0x7fe1140c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe119200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1140c8350 con 0x7fe11415cca0 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe119200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1140b1490 con 0x7fe1140b1b40 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe119200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1140b2840 con 0x7fe11406a380 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe113e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11415cca0 0x7fe1140c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe113e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11415cca0 0x7fe1140c6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38044/0 (socket says 172.21.15.39:38044) 2024-01-24T17:46:15.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe113e00640 1 -- 172.21.15.39:0/1757307245 learned_addr learned my addr 172.21.15.39:0/1757307245 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:15.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.670+0000 7fe112a00640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1140b1b40 0x7fe1140b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe113400640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe11406a380 0x7fe1140b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe113e00640 1 -- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe11406a380 msgr2=0x7fe1140b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe113e00640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe11406a380 0x7fe1140b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe113e00640 1 -- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1140b1b40 msgr2=0x7fe1140b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe113e00640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1140b1b40 0x7fe1140b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe113e00640 1 -- 172.21.15.39:0/1757307245 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc0094d0 con 0x7fe11415cca0 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe112a00640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1140b1b40 0x7fe1140b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe113e00640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11415cca0 0x7fe1140c6700 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c740 tx=0x7fe10c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe10b400640 1 -- 172.21.15.39:0/1757307245 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c003da0 con 0x7fe11415cca0 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe10b400640 1 -- 172.21.15.39:0/1757307245 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c00ce50 con 0x7fe11415cca0 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe10b400640 1 -- 172.21.15.39:0/1757307245 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c005290 con 0x7fe11415cca0 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.671+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1140b2ae0 con 0x7fe11415cca0 2024-01-24T17:46:15.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.672+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1140b30a0 con 0x7fe11415cca0 2024-01-24T17:46:15.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.673+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe114061880 con 0x7fe11415cca0 2024-01-24T17:46:15.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.673+0000 7fe10b400640 1 -- 172.21.15.39:0/1757307245 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe10c0053f0 con 0x7fe11415cca0 2024-01-24T17:46:15.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.674+0000 7fe10b400640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0ec06a430 0x7fe0ec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:15.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.674+0000 7fe10b400640 1 -- 172.21.15.39:0/1757307245 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe10c088120 con 0x7fe11415cca0 2024-01-24T17:46:15.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.675+0000 7fe113400640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0ec06a430 0x7fe0ec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:15.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.676+0000 7fe113400640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0ec06a430 0x7fe0ec06c8f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe104006fd0 tx=0x7fe104008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:15.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.679+0000 7fe10b400640 1 -- 172.21.15.39:0/1757307245 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe10c053710 con 0x7fe11415cca0 2024-01-24T17:46:15.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.914+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe11406ab60 con 0x7fe11415cca0 2024-01-24T17:46:15.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.915+0000 7fe10b400640 1 -- 172.21.15.39:0/1757307245 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fe10c056d30 con 0x7fe11415cca0 2024-01-24T17:46:15.917 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:15.917 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3045325513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3045325513},{"type":"v1","addr":"172.21.15.100:6836","nonce":3045325513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5126,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/858868211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":858868211},{"type":"v1","addr":"172.21.15.100:6837","nonce":858868211}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3121826441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3121826441},{"type":"v1","addr":"172.21.15.39:6833","nonce":3121826441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:45:18.381640+0000","modified":"2024-01-24T17:45:21.392053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6261":{"gid":6261,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/527429915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":527429915},{"type":"v1","addr":"172.21.15.39:6835","nonce":527429915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-24T17:46:15.917 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-01-24T17:46:15.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.917+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0ec06a430 msgr2=0x7fe0ec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.917+0000 7fe119200640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0ec06a430 0x7fe0ec06c8f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe104006fd0 tx=0x7fe104008040 comp rx=0 tx=0).stop 2024-01-24T17:46:15.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.917+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11415cca0 msgr2=0x7fe1140c6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:15.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.917+0000 7fe119200640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11415cca0 0x7fe1140c6700 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c740 tx=0x7fe10c00cc00 comp rx=0 tx=0).stop 2024-01-24T17:46:15.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.918+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 shutdown_connections 2024-01-24T17:46:15.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.918+0000 7fe119200640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe0ec06a430 0x7fe0ec06c8f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.918+0000 7fe119200640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11415cca0 0x7fe1140c6700 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.918+0000 7fe119200640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1140b1b40 0x7fe1140b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.918+0000 7fe119200640 1 --2- 172.21.15.39:0/1757307245 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe11406a380 0x7fe1140b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:15.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.918+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 >> 172.21.15.39:0/1757307245 conn(0x7fe11406ad80 msgr2=0x7fe114159d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:15.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.918+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 shutdown_connections 2024-01-24T17:46:15.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:15.918+0000 7fe119200640 1 -- 172.21.15.39:0/1757307245 wait complete. 2024-01-24T17:46:15.940 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:46:16.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.163+0000 7f508ce00640 1 Processor -- start 2024-01-24T17:46:16.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.163+0000 7f508ce00640 1 -- start start 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.164+0000 7f508ce00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5088142740 0x7f50880b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.164+0000 7f508ce00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5088071290 0x7f508806f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.164+0000 7f508ce00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5088142b20 con 0x7f5088142e30 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f508ce00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f508815d4c0 con 0x7f5088142740 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f508ce00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50880b2610 con 0x7f5088071290 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f5085e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5088142740 0x7f50880b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f5085e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5088142740 0x7f50880b20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38050/0 (socket says 172.21.15.39:38050) 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f5085e00640 1 -- 172.21.15.39:0/114717391 learned_addr learned my addr 172.21.15.39:0/114717391 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:16.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f5086800640 1 --2- 172.21.15.39:0/114717391 >> v2:172.21.15.100:3300/0 conn(0x7f5088071290 0x7f508806f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f5085e00640 1 -- 172.21.15.39:0/114717391 >> v2:172.21.15.100:3300/0 conn(0x7f5088071290 msgr2=0x7f508806f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:16.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f5085e00640 1 --2- 172.21.15.39:0/114717391 >> v2:172.21.15.100:3300/0 conn(0x7f5088071290 0x7f508806f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f5085e00640 1 -- 172.21.15.39:0/114717391 >> v1:172.21.15.100:6789/0 conn(0x7f5088142e30 legacy=0x7f5088070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:16.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.165+0000 7f5085e00640 1 -- 172.21.15.39:0/114717391 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508815d710 con 0x7f5088142740 2024-01-24T17:46:16.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.167+0000 7f5085e00640 1 --2- 172.21.15.39:0/114717391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088142740 0x7f50880b20d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f508806cc50 tx=0x7f507802da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def7d9a1c8064407 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:16.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.167+0000 7f5085400640 1 -- 172.21.15.39:0/114717391 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507802f850 con 0x7f5088142740 2024-01-24T17:46:16.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.167+0000 7f5085400640 1 -- 172.21.15.39:0/114717391 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507802f9b0 con 0x7f5088142740 2024-01-24T17:46:16.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.167+0000 7f5085400640 1 -- 172.21.15.39:0/114717391 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507802fd40 con 0x7f5088142740 2024-01-24T17:46:16.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.167+0000 7f508ce00640 1 -- 172.21.15.39:0/114717391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088142740 msgr2=0x7f50880b20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.167+0000 7f508ce00640 1 --2- 172.21.15.39:0/114717391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088142740 0x7f50880b20d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f508806cc50 tx=0x7f507802da90 comp rx=0 tx=0).stop 2024-01-24T17:46:16.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.167+0000 7f508ce00640 1 -- 172.21.15.39:0/114717391 shutdown_connections 2024-01-24T17:46:16.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.167+0000 7f508ce00640 1 --2- 172.21.15.39:0/114717391 >> v2:172.21.15.100:3300/0 conn(0x7f5088071290 0x7f508806f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.168+0000 7f508ce00640 1 --2- 172.21.15.39:0/114717391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088142740 0x7f50880b20d0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.168+0000 7f508ce00640 1 -- 172.21.15.39:0/114717391 >> 172.21.15.39:0/114717391 conn(0x7f5088069e50 msgr2=0x7f508806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:16.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.168+0000 7f508ce00640 1 -- 172.21.15.39:0/114717391 shutdown_connections 2024-01-24T17:46:16.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.168+0000 7f508ce00640 1 -- 172.21.15.39:0/114717391 wait complete. 2024-01-24T17:46:16.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.169+0000 7f508ce00640 1 Processor -- start 2024-01-24T17:46:16.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.169+0000 7f508ce00640 1 -- start start 2024-01-24T17:46:16.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.170+0000 7f508ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088071290 0x7f508813e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.170+0000 7f508ce00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5088142740 0x7f5088140640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.170+0000 7f508ce00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5088142e30 0x7f5088140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.170+0000 7f508ce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50880cb690 con 0x7f5088071290 2024-01-24T17:46:16.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.170+0000 7f508ce00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508813e6f0 con 0x7f5088142e30 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.170+0000 7f508ce00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50880caa10 con 0x7f5088142740 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088071290 0x7f508813e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088071290 0x7f508813e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38066/0 (socket says 172.21.15.39:38066) 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 -- 172.21.15.39:0/783959133 learned_addr learned my addr 172.21.15.39:0/783959133 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 -- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5088142740 msgr2=0x7f5088140640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5088142740 0x7f5088140640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 -- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5088142e30 msgr2=0x7f5088140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5088142e30 0x7f5088140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 -- 172.21.15.39:0/783959133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50780094d0 con 0x7f5088071290 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.171+0000 7f5085e00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088071290 0x7f508813e1b0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f507802dad0 tx=0x7f50780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.172+0000 7f507e000640 1 -- 172.21.15.39:0/783959133 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507803d070 con 0x7f5088071290 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.172+0000 7f507e000640 1 -- 172.21.15.39:0/783959133 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5078039480 con 0x7f5088071290 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.172+0000 7f507e000640 1 -- 172.21.15.39:0/783959133 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5078039830 con 0x7f5088071290 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.172+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50880cacf0 con 0x7f5088071290 2024-01-24T17:46:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.172+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50880cb2b0 con 0x7f5088071290 2024-01-24T17:46:16.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.173+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5088066480 con 0x7f5088071290 2024-01-24T17:46:16.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.174+0000 7f507e000640 1 -- 172.21.15.39:0/783959133 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5078039990 con 0x7f5088071290 2024-01-24T17:46:16.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.175+0000 7f507e000640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f506006a430 0x7f506006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.175+0000 7f507e000640 1 -- 172.21.15.39:0/783959133 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5078034070 con 0x7f5088071290 2024-01-24T17:46:16.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.175+0000 7f507fe00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f506006a430 0x7f506006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.177+0000 7f507fe00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f506006a430 0x7f506006c8f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f50881405e0 tx=0x7f50700079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:16.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.179+0000 7f507e000640 1 -- 172.21.15.39:0/783959133 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5078075e70 con 0x7f5088071290 2024-01-24T17:46:16.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.415+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5088142b20 con 0x7f5088071290 2024-01-24T17:46:16.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.417+0000 7f507e000640 1 -- 172.21.15.39:0/783959133 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f5088142b20 con 0x7f5088071290 2024-01-24T17:46:16.419 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:16.419 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3045325513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3045325513},{"type":"v1","addr":"172.21.15.100:6836","nonce":3045325513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5126,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/858868211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":858868211},{"type":"v1","addr":"172.21.15.100:6837","nonce":858868211}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3121826441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3121826441},{"type":"v1","addr":"172.21.15.39:6833","nonce":3121826441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:45:18.381640+0000","modified":"2024-01-24T17:45:21.392053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6261":{"gid":6261,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/527429915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":527429915},{"type":"v1","addr":"172.21.15.39:6835","nonce":527429915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-24T17:46:16.420 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 34 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f506006a430 msgr2=0x7f506006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f506006a430 0x7f506006c8f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f50881405e0 tx=0x7f50700079b0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088071290 msgr2=0x7f508813e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088071290 0x7f508813e1b0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f507802dad0 tx=0x7f50780046c0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 shutdown_connections 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5088142e30 0x7f5088140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5088142740 0x7f5088140640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f506006a430 0x7f506006c8f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 --2- 172.21.15.39:0/783959133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5088071290 0x7f508813e1b0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 >> 172.21.15.39:0/783959133 conn(0x7f5088069e50 msgr2=0x7f508806f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 shutdown_connections 2024-01-24T17:46:16.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.421+0000 7f508ce00640 1 -- 172.21.15.39:0/783959133 wait complete. 2024-01-24T17:46:16.441 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:16.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.651+0000 7fb55d200640 1 Processor -- start 2024-01-24T17:46:16.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.652+0000 7fb55d200640 1 -- start start 2024-01-24T17:46:16.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.653+0000 7fb55d200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cfa40 0x7fb55806be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.653+0000 7fb55d200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb558071560 0x7fb5580cad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.653+0000 7fb55d200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806cdf0 con 0x7fb5580cb270 2024-01-24T17:46:16.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.653+0000 7fb55d200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580cd320 con 0x7fb558071560 2024-01-24T17:46:16.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.653+0000 7fb55d200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580c75a0 con 0x7fb5580cfa40 2024-01-24T17:46:16.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.654+0000 7fb556a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb558071560 0x7fb5580cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.654+0000 7fb557400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cfa40 0x7fb55806be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.654+0000 7fb556a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb558071560 0x7fb5580cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:51844/0 (socket says 172.21.15.39:51844) 2024-01-24T17:46:16.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.654+0000 7fb556a00640 1 -- 172.21.15.39:0/3367266376 learned_addr learned my addr 172.21.15.39:0/3367266376 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:16.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.655+0000 7fb556a00640 1 -- 172.21.15.39:0/3367266376 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cfa40 msgr2=0x7fb55806be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:16.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.655+0000 7fb556a00640 1 --2- 172.21.15.39:0/3367266376 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cfa40 0x7fb55806be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.655+0000 7fb556a00640 1 -- 172.21.15.39:0/3367266376 >> v1:172.21.15.100:6789/0 conn(0x7fb5580cb270 legacy=0x7fb5580cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.655+0000 7fb556a00640 1 -- 172.21.15.39:0/3367266376 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55806c550 con 0x7fb558071560 2024-01-24T17:46:16.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.656+0000 7fb556a00640 1 --2- 172.21.15.39:0/3367266376 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 0x7fb5580cad30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb548009e50 tx=0x7fb54802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=332d04c254739f84 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:16.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.657+0000 7fb556000640 1 -- 172.21.15.39:0/3367266376 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54802ca80 con 0x7fb558071560 2024-01-24T17:46:16.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.657+0000 7fb556000640 1 -- 172.21.15.39:0/3367266376 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54802cbe0 con 0x7fb558071560 2024-01-24T17:46:16.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.657+0000 7fb556000640 1 -- 172.21.15.39:0/3367266376 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480343f0 con 0x7fb558071560 2024-01-24T17:46:16.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.657+0000 7fb55d200640 1 -- 172.21.15.39:0/3367266376 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 msgr2=0x7fb5580cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.657+0000 7fb55d200640 1 --2- 172.21.15.39:0/3367266376 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 0x7fb5580cad30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb548009e50 tx=0x7fb54802bee0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.658+0000 7fb55d200640 1 -- 172.21.15.39:0/3367266376 shutdown_connections 2024-01-24T17:46:16.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.658+0000 7fb55d200640 1 --2- 172.21.15.39:0/3367266376 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 0x7fb5580cad30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.658+0000 7fb55d200640 1 --2- 172.21.15.39:0/3367266376 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cfa40 0x7fb55806be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.658+0000 7fb55d200640 1 -- 172.21.15.39:0/3367266376 >> 172.21.15.39:0/3367266376 conn(0x7fb5580b21f0 msgr2=0x7fb5580b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:16.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.658+0000 7fb55d200640 1 -- 172.21.15.39:0/3367266376 shutdown_connections 2024-01-24T17:46:16.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.658+0000 7fb55d200640 1 -- 172.21.15.39:0/3367266376 wait complete. 2024-01-24T17:46:16.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.659+0000 7fb55d200640 1 Processor -- start 2024-01-24T17:46:16.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.660+0000 7fb55d200640 1 -- start start 2024-01-24T17:46:16.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.660+0000 7fb55d200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 0x7fb5580af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb55d200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cb270 0x7fb55815c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb55d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5580cfa40 0x7fb5580b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb55d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580af8b0 con 0x7fb5580cfa40 2024-01-24T17:46:16.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb55d200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580c63b0 con 0x7fb558071560 2024-01-24T17:46:16.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb55d200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580c57a0 con 0x7fb5580cb270 2024-01-24T17:46:16.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5580cfa40 0x7fb5580b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5580cfa40 0x7fb5580b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38076/0 (socket says 172.21.15.39:38076) 2024-01-24T17:46:16.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557e00640 1 -- 172.21.15.39:0/1952361830 learned_addr learned my addr 172.21.15.39:0/1952361830 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:16.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557400640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 0x7fb5580af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb556a00640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cb270 0x7fb55815c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557e00640 1 -- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cb270 msgr2=0x7fb55815c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557e00640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cb270 0x7fb55815c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557e00640 1 -- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 msgr2=0x7fb5580af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557e00640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 0x7fb5580af370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb557e00640 1 -- 172.21.15.39:0/1952361830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548009b30 con 0x7fb5580cfa40 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.661+0000 7fb556a00640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cb270 0x7fb55815c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.662+0000 7fb557e00640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5580cfa40 0x7fb5580b3300 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb55000c3c0 tx=0x7fb55000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.662+0000 7fb54f400640 1 -- 172.21.15.39:0/1952361830 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500052d0 con 0x7fb5580cfa40 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.662+0000 7fb54f400640 1 -- 172.21.15.39:0/1952361830 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb550005430 con 0x7fb5580cfa40 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.662+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5580c5a80 con 0x7fb5580cfa40 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.662+0000 7fb54f400640 1 -- 172.21.15.39:0/1952361830 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550003810 con 0x7fb5580cfa40 2024-01-24T17:46:16.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.662+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5580c6040 con 0x7fb5580cfa40 2024-01-24T17:46:16.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.664+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb558061880 con 0x7fb5580cfa40 2024-01-24T17:46:16.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.664+0000 7fb54f400640 1 -- 172.21.15.39:0/1952361830 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb550003970 con 0x7fb5580cfa40 2024-01-24T17:46:16.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.665+0000 7fb54f400640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb53006a430 0x7fb53006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:16.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.665+0000 7fb54f400640 1 -- 172.21.15.39:0/1952361830 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb550087c90 con 0x7fb5580cfa40 2024-01-24T17:46:16.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.666+0000 7fb557400640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb53006a430 0x7fb53006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:16.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.667+0000 7fb557400640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb53006a430 0x7fb53006c8f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb540009ad0 tx=0x7fb540008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:16.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.670+0000 7fb54f400640 1 -- 172.21.15.39:0/1952361830 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb550053240 con 0x7fb5580cfa40 2024-01-24T17:46:16.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.868+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb55806cdf0 con 0x7fb5580cfa40 2024-01-24T17:46:16.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.869+0000 7fb54f400640 1 -- 172.21.15.39:0/1952361830 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+16193 (secure 0 0 0) 0x7fb55806cdf0 con 0x7fb5580cfa40 2024-01-24T17:46:16.871 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:16.871 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":64,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:16.379205+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:45:16.503215+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-24T17:45:17.386565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.28","mappings":[{"from":6,"to":4}]},{"pgid":"9.30","mappings":[{"from":6,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb53006a430 msgr2=0x7fb53006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb53006a430 0x7fb53006c8f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb540009ad0 tx=0x7fb540008040 comp rx=0 tx=0).stop 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5580cfa40 msgr2=0x7fb5580b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5580cfa40 0x7fb5580b3300 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb55000c3c0 tx=0x7fb55000c880 comp rx=0 tx=0).stop 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 shutdown_connections 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb53006a430 0x7fb53006c8f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5580cfa40 0x7fb5580b3300 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5580cb270 0x7fb55815c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 --2- 172.21.15.39:0/1952361830 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb558071560 0x7fb5580af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:16.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 >> 172.21.15.39:0/1952361830 conn(0x7fb5580b21f0 msgr2=0x7fb5580b12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:16.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 shutdown_connections 2024-01-24T17:46:16.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:16.872+0000 7fb55d200640 1 -- 172.21.15.39:0/1952361830 wait complete. 2024-01-24T17:46:16.893 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-24T17:46:17.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.119+0000 7f0674600640 1 Processor -- start 2024-01-24T17:46:17.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.119+0000 7f0674600640 1 -- start start 2024-01-24T17:46:17.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.120+0000 7f0674600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c0b85f0 0x7f066c160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.121+0000 7f0674600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c160dd0 con 0x7f066c0cfbd0 2024-01-24T17:46:17.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.121+0000 7f0674600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c0d0e10 con 0x7f066c15cdc0 2024-01-24T17:46:17.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.121+0000 7f0674600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0b8a60 con 0x7f066c0b85f0 2024-01-24T17:46:17.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.121+0000 7f0671200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f066c0cfbd0 0x7f066c0d1580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58804/0 (socket says 172.21.15.39:58804) 2024-01-24T17:46:17.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.121+0000 7f0671200640 1 -- 172.21.15.39:0/3016990269 learned_addr learned my addr 172.21.15.39:0/3016990269 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.121+0000 7f0672600640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c0b85f0 0x7f066c160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.121+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435255560 0 0) 0x7f066c160dd0 con 0x7f066c0cfbd0 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.121+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f064c003520 con 0x7f066c0cfbd0 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1256573142 0 0) 0x7f064c003520 con 0x7f066c0cfbd0 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c0b85f0 msgr2=0x7f066c160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c0b85f0 0x7f066c160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 >> v1:172.21.15.100:6789/0 conn(0x7f066c15cdc0 legacy=0x7f066c0cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c15d1a0 con 0x7f066c0cfbd0 2024-01-24T17:46:17.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f0654001670 con 0x7f066c0cfbd0 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 >> v1:172.21.15.39:6789/0 conn(0x7f066c0cfbd0 legacy=0x7f066c0d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c0b85f0 0x7f064c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f064c003f20 0x7f064c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f064c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0b8a60 con 0x7f064c004d00 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c00ba60 con 0x7f064c003f20 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064c00bc20 con 0x7f066c0b85f0 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.122+0000 7f0672600640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f064c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.123+0000 7f0671200640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f064c003f20 0x7f064c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.123+0000 7f0672600640 1 -- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c0b85f0 msgr2=0x7f064c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.123+0000 7f0672600640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c0b85f0 0x7f064c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.123+0000 7f0672600640 1 -- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f064c003f20 msgr2=0x7f064c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:17.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.123+0000 7f0672600640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f064c003f20 0x7f064c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.123+0000 7f0672600640 1 -- 172.21.15.39:0/3016990269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c003520 con 0x7f064c004d00 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.124+0000 7f0672600640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f064c005180 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f0664009fc0 tx=0x7f064c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2969963c806710c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.124+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066402cc60 con 0x7f064c004d00 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.124+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066400f040 con 0x7f064c004d00 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.124+0000 7f066be00640 1 -- 172.21.15.39:0/3016990269 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664034440 con 0x7f064c004d00 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.125+0000 7f0674600640 1 -- 172.21.15.39:0/3016990269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 msgr2=0x7f064c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.125+0000 7f0674600640 1 --2- 172.21.15.39:0/3016990269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f064c005180 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f0664009fc0 tx=0x7f064c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.125+0000 7f0672600640 1 -- 172.21.15.39:0/3016990269 reap_dead start 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.125+0000 7f0674600640 1 -- 172.21.15.39:0/3016990269 shutdown_connections 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.125+0000 7f0674600640 1 -- 172.21.15.39:0/3016990269 >> 172.21.15.39:0/3016990269 conn(0x7f066c06a100 msgr2=0x7f066c06a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.125+0000 7f0674600640 1 -- 172.21.15.39:0/3016990269 shutdown_connections 2024-01-24T17:46:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.125+0000 7f0674600640 1 -- 172.21.15.39:0/3016990269 wait complete. 2024-01-24T17:46:17.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.127+0000 7f0674600640 1 Processor -- start 2024-01-24T17:46:17.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.127+0000 7f0674600640 1 -- start start 2024-01-24T17:46:17.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0674600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f066c0cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0674600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f066c0cc8b0 0x7f066c0d2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0674600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c15dd10 0x7f066c0d2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0674600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c157680 con 0x7f064c004d00 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0674600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0cb810 con 0x7f066c0cc8b0 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0674600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0cbaf0 con 0x7f066c15dd10 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0671c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f066c0cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0671c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f066c0cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38096/0 (socket says 172.21.15.39:38096) 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.128+0000 7f0671c00640 1 -- 172.21.15.39:0/2545104464 learned_addr learned my addr 172.21.15.39:0/2545104464 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0671200640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f066c0cc8b0 0x7f066c0d2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0671c00640 1 -- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c15dd10 msgr2=0x7f066c0d2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0671c00640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c15dd10 0x7f066c0d2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0671c00640 1 -- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f066c0cc8b0 msgr2=0x7f066c0d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0671c00640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f066c0cc8b0 0x7f066c0d2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0671c00640 1 -- 172.21.15.39:0/2545104464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c0cbdd0 con 0x7f064c004d00 2024-01-24T17:46:17.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0671c00640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f066c0cb110 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f065c00c450 tx=0x7f065c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:17.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0669600640 1 -- 172.21.15.39:0/2545104464 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c0052d0 con 0x7f064c004d00 2024-01-24T17:46:17.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0669600640 1 -- 172.21.15.39:0/2545104464 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065c005430 con 0x7f064c004d00 2024-01-24T17:46:17.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0669600640 1 -- 172.21.15.39:0/2545104464 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c012870 con 0x7f064c004d00 2024-01-24T17:46:17.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.129+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066c0cc0b0 con 0x7f064c004d00 2024-01-24T17:46:17.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.130+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066c0c9b80 con 0x7f064c004d00 2024-01-24T17:46:17.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.131+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066c067360 con 0x7f064c004d00 2024-01-24T17:46:17.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.132+0000 7f0669600640 1 -- 172.21.15.39:0/2545104464 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f065c0129d0 con 0x7f064c004d00 2024-01-24T17:46:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.133+0000 7f0669600640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f064406a430 0x7f064406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.133+0000 7f0669600640 1 -- 172.21.15.39:0/2545104464 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f065c088510 con 0x7f064c004d00 2024-01-24T17:46:17.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.134+0000 7f0671200640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f064406a430 0x7f064406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.135+0000 7f0671200640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f064406a430 0x7f064406c8f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0654001eb0 tx=0x7f0654009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:17.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.137+0000 7f0669600640 1 -- 172.21.15.39:0/2545104464 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f065c053a20 con 0x7f064c004d00 2024-01-24T17:46:17.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.371+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f066c0c7890 con 0x7f064c004d00 2024-01-24T17:46:17.375 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:46:17.373+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:46:17.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.393+0000 7f0669600640 1 -- 172.21.15.39:0/2545104464 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7f065c057040 con 0x7f064c004d00 2024-01-24T17:46:17.394 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-24T17:46:17.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.395+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f064406a430 msgr2=0x7f064406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.395+0000 7f0674600640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f064406a430 0x7f064406c8f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0654001eb0 tx=0x7f0654009040 comp rx=0 tx=0).stop 2024-01-24T17:46:17.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.395+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 msgr2=0x7f066c0cb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.395+0000 7f0674600640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f066c0cb110 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f065c00c450 tx=0x7f065c00c910 comp rx=0 tx=0).stop 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.396+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 shutdown_connections 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.396+0000 7f0674600640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f066c15dd10 0x7f066c0d2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.396+0000 7f0674600640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f066c0cc8b0 0x7f066c0d2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.396+0000 7f0674600640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f064406a430 0x7f064406c8f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.396+0000 7f0674600640 1 --2- 172.21.15.39:0/2545104464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f064c004d00 0x7f066c0cb110 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.396+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 >> 172.21.15.39:0/2545104464 conn(0x7f066c06a100 msgr2=0x7f066c15a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.396+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 shutdown_connections 2024-01-24T17:46:17.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.396+0000 7f0674600640 1 -- 172.21.15.39:0/2545104464 wait complete. 2024-01-24T17:46:17.403 INFO:tasks.ceph.mds.c.smithi039.stderr: -4643> 2024-01-24T17:45:23.849+0000 7f1927400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:46:17.417 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-24T17:46:17.456 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:46:17.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.604+0000 7ff9dfa00640 1 Processor -- start 2024-01-24T17:46:17.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.605+0000 7ff9dfa00640 1 -- start start 2024-01-24T17:46:17.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.606+0000 7ff9dfa00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff9d80b0750 0x7ff9d80b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.606+0000 7ff9dfa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80b19d0 0x7ff9d80b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.606+0000 7ff9dfa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d806b630 con 0x7ff9d80b3ce0 2024-01-24T17:46:17.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.606+0000 7ff9dfa00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d806b8d0 con 0x7ff9d80b0750 2024-01-24T17:46:17.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.606+0000 7ff9dfa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d806bbb0 con 0x7ff9d80b19d0 2024-01-24T17:46:17.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.606+0000 7ff9dd000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff9d80b3ce0 0x7ff9d80b40c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58822/0 (socket says 172.21.15.39:58822) 2024-01-24T17:46:17.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.606+0000 7ff9dd000640 1 -- 172.21.15.39:0/4151378485 learned_addr learned my addr 172.21.15.39:0/4151378485 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9dda00640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80b19d0 0x7ff9d80b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7e00640 1 --2- 172.21.15.39:0/4151378485 >> v2:172.21.15.100:3300/0 conn(0x7ff9d80b0750 0x7ff9d80b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172318749 0 0) 0x7ff9d806b630 con 0x7ff9d80b3ce0 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b8003520 con 0x7ff9d80b3ce0 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1164496262 0 0) 0x7ff9b8003520 con 0x7ff9d80b3ce0 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80b19d0 msgr2=0x7ff9d80b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80b19d0 0x7ff9d80b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 >> v2:172.21.15.100:3300/0 conn(0x7ff9d80b0750 msgr2=0x7ff9d80b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:17.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 --2- 172.21.15.39:0/4151378485 >> v2:172.21.15.100:3300/0 conn(0x7ff9d80b0750 0x7ff9d80b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d806be90 con 0x7ff9d80b3ce0 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff9c8001670 con 0x7ff9d80b3ce0 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 >> v1:172.21.15.39:6789/0 conn(0x7ff9d80b3ce0 legacy=0x7ff9d80b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9b8003970 0x7ff9b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80b19d0 0x7ff9b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d806b8d0 con 0x7ff9b8004d00 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d806bbb0 con 0x7ff9b8003970 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.607+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b800b980 con 0x7ff9d80b19d0 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.608+0000 7ff9dda00640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.608+0000 7ff9dd000640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9b8003970 0x7ff9b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.608+0000 7ff9dda00640 1 -- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80b19d0 msgr2=0x7ff9b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:17.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.608+0000 7ff9dda00640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80b19d0 0x7ff9b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.608+0000 7ff9dda00640 1 -- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9b8003970 msgr2=0x7ff9b8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:17.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.608+0000 7ff9dda00640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9b8003970 0x7ff9b8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.608+0000 7ff9dda00640 1 -- 172.21.15.39:0/4151378485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b8003520 con 0x7ff9b8004d00 2024-01-24T17:46:17.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.608+0000 7ff9dd000640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9b8003970 0x7ff9b8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:17.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.609+0000 7ff9dda00640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9b8005180 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009fc0 tx=0x7ff9b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa148ba94d6bee27 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:17.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.609+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d002cc60 con 0x7ff9b8004d00 2024-01-24T17:46:17.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.609+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d000f040 con 0x7ff9b8004d00 2024-01-24T17:46:17.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.609+0000 7ff9d7400640 1 -- 172.21.15.39:0/4151378485 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0034440 con 0x7ff9b8004d00 2024-01-24T17:46:17.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.610+0000 7ff9dfa00640 1 -- 172.21.15.39:0/4151378485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 msgr2=0x7ff9b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.610+0000 7ff9dfa00640 1 --2- 172.21.15.39:0/4151378485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9b8005180 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009fc0 tx=0x7ff9b80057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.610+0000 7ff9dda00640 1 -- 172.21.15.39:0/4151378485 reap_dead start 2024-01-24T17:46:17.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.610+0000 7ff9dfa00640 1 -- 172.21.15.39:0/4151378485 shutdown_connections 2024-01-24T17:46:17.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.610+0000 7ff9dfa00640 1 -- 172.21.15.39:0/4151378485 >> 172.21.15.39:0/4151378485 conn(0x7ff9d8069e50 msgr2=0x7ff9d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:17.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.610+0000 7ff9dfa00640 1 -- 172.21.15.39:0/4151378485 shutdown_connections 2024-01-24T17:46:17.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.610+0000 7ff9dfa00640 1 -- 172.21.15.39:0/4151378485 wait complete. 2024-01-24T17:46:17.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.612+0000 7ff9dfa00640 1 Processor -- start 2024-01-24T17:46:17.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.612+0000 7ff9dfa00640 1 -- start start 2024-01-24T17:46:17.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dfa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9d81f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dfa00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9d81f0bb0 0x7ff9d80c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dfa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80c87f0 0x7ff9d814df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dfa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d81f10a0 con 0x7ff9b8004d00 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dfa00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d81f1380 con 0x7ff9d81f0bb0 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dfa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d81f1660 con 0x7ff9d80c87f0 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9d81f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9d81f0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38112/0 (socket says 172.21.15.39:38112) 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.613+0000 7ff9dd000640 1 -- 172.21.15.39:0/859627768 learned_addr learned my addr 172.21.15.39:0/859627768 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9dda00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80c87f0 0x7ff9d814df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9d7e00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9d81f0bb0 0x7ff9d80c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9dd000640 1 -- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80c87f0 msgr2=0x7ff9d814df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9dd000640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80c87f0 0x7ff9d814df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9dd000640 1 -- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9d81f0bb0 msgr2=0x7ff9d80c82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9dd000640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9d81f0bb0 0x7ff9d80c82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9dd000640 1 -- 172.21.15.39:0/859627768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d81f1940 con 0x7ff9b8004d00 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9d7e00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9d81f0bb0 0x7ff9d80c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:17.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9dd000640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9d81f0670 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800d890 tx=0x7ff9c800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:17.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.614+0000 7ff9d4c00640 1 -- 172.21.15.39:0/859627768 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80074e0 con 0x7ff9b8004d00 2024-01-24T17:46:17.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.615+0000 7ff9d4c00640 1 -- 172.21.15.39:0/859627768 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8007640 con 0x7ff9b8004d00 2024-01-24T17:46:17.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.615+0000 7ff9d4c00640 1 -- 172.21.15.39:0/859627768 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8004a20 con 0x7ff9b8004d00 2024-01-24T17:46:17.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.615+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d80c8d80 con 0x7ff9b8004d00 2024-01-24T17:46:17.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.615+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d814e650 con 0x7ff9b8004d00 2024-01-24T17:46:17.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.616+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d80b3b30 con 0x7ff9b8004d00 2024-01-24T17:46:17.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.617+0000 7ff9d4c00640 1 -- 172.21.15.39:0/859627768 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff9c8004b80 con 0x7ff9b8004d00 2024-01-24T17:46:17.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.618+0000 7ff9d4c00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff9b006a430 0x7ff9b006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:17.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.618+0000 7ff9d4c00640 1 -- 172.21.15.39:0/859627768 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff9c8088540 con 0x7ff9b8004d00 2024-01-24T17:46:17.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.618+0000 7ff9d7e00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff9b006a430 0x7ff9b006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:17.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.619+0000 7ff9d7e00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff9b006a430 0x7ff9b006c8f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff9d81f0fb0 tx=0x7ff9c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:17.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.622+0000 7ff9d4c00640 1 -- 172.21.15.39:0/859627768 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff9c8053a00 con 0x7ff9b8004d00 2024-01-24T17:46:17.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:17.850+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff9d81f11e0 con 0x7ff9b8004d00 2024-01-24T17:46:18.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.422+0000 7ff9d4c00640 1 -- 172.21.15.39:0/859627768 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff9d81f11e0 con 0x7ff9b8004d00 2024-01-24T17:46:18.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff9b006a430 msgr2=0x7ff9b006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:18.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff9b006a430 0x7ff9b006c8f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff9d81f0fb0 tx=0x7ff9c0008040 comp rx=0 tx=0).stop 2024-01-24T17:46:18.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 msgr2=0x7ff9d81f0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:18.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9d81f0670 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800d890 tx=0x7ff9c800dd50 comp rx=0 tx=0).stop 2024-01-24T17:46:18.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 shutdown_connections 2024-01-24T17:46:18.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff9d80c87f0 0x7ff9d814df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff9d81f0bb0 0x7ff9d80c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff9b006a430 0x7ff9b006c8f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 --2- 172.21.15.39:0/859627768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9b8004d00 0x7ff9d81f0670 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 >> 172.21.15.39:0/859627768 conn(0x7ff9d8069e50 msgr2=0x7ff9d815cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:18.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 shutdown_connections 2024-01-24T17:46:18.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.425+0000 7ff9dfa00640 1 -- 172.21.15.39:0/859627768 wait complete. 2024-01-24T17:46:18.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-24T17:46:18.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.715+0000 7f4400200640 1 Processor -- start 2024-01-24T17:46:18.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.715+0000 7f4400200640 1 -- start start 2024-01-24T17:46:18.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.717+0000 7f4400200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80afcc0 0x7f43f80b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:18.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.717+0000 7f4400200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f43f80b2a00 0x7f43f806fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:18.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.717+0000 7f4400200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f815d720 con 0x7f43f8150910 2024-01-24T17:46:18.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.717+0000 7f4400200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f80b3210 con 0x7f43f80b2a00 2024-01-24T17:46:18.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.717+0000 7f4400200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f815d510 con 0x7f43f80afcc0 2024-01-24T17:46:18.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fe200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f43f80b2a00 0x7f43f806fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fe200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f43f80b2a00 0x7f43f806fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:51926/0 (socket says 172.21.15.39:51926) 2024-01-24T17:46:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fe200640 1 -- 172.21.15.39:0/401106911 learned_addr learned my addr 172.21.15.39:0/401106911 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fd800640 1 --1- 172.21.15.39:0/401106911 >> v1:172.21.15.100:6789/0 conn(0x7f43f8150910 0x7f43f80af5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48616/0 (socket says 172.21.15.39:48616) 2024-01-24T17:46:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fce00640 1 --2- 172.21.15.39:0/401106911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80afcc0 0x7f43f80b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fe200640 1 -- 172.21.15.39:0/401106911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80afcc0 msgr2=0x7f43f80b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:18.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fe200640 1 --2- 172.21.15.39:0/401106911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80afcc0 0x7f43f80b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fe200640 1 -- 172.21.15.39:0/401106911 >> v1:172.21.15.100:6789/0 conn(0x7f43f8150910 legacy=0x7f43f80af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.718+0000 7f43fe200640 1 -- 172.21.15.39:0/401106911 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f815da00 con 0x7f43f80b2a00 2024-01-24T17:46:18.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.720+0000 7f43fe200640 1 --2- 172.21.15.39:0/401106911 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80b2a00 0x7f43f806fbd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009e50 tx=0x7f43f002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ae7a1cd0ec27292 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:18.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.720+0000 7f43f7e00640 1 -- 172.21.15.39:0/401106911 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f002cb00 con 0x7f43f80b2a00 2024-01-24T17:46:18.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.720+0000 7f43f7e00640 1 -- 172.21.15.39:0/401106911 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f002cc60 con 0x7f43f80b2a00 2024-01-24T17:46:18.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.720+0000 7f43f7e00640 1 -- 172.21.15.39:0/401106911 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f00343f0 con 0x7f43f80b2a00 2024-01-24T17:46:18.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.721+0000 7f4400200640 1 -- 172.21.15.39:0/401106911 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80b2a00 msgr2=0x7f43f806fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:18.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.721+0000 7f4400200640 1 --2- 172.21.15.39:0/401106911 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80b2a00 0x7f43f806fbd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009e50 tx=0x7f43f002bf60 comp rx=0 tx=0).stop 2024-01-24T17:46:18.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.721+0000 7f4400200640 1 -- 172.21.15.39:0/401106911 shutdown_connections 2024-01-24T17:46:18.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.721+0000 7f4400200640 1 --2- 172.21.15.39:0/401106911 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80b2a00 0x7f43f806fbd0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.721+0000 7f4400200640 1 --2- 172.21.15.39:0/401106911 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80afcc0 0x7f43f80b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.721+0000 7f4400200640 1 -- 172.21.15.39:0/401106911 >> 172.21.15.39:0/401106911 conn(0x7f43f8069c70 msgr2=0x7f43f806a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:18.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.721+0000 7f4400200640 1 -- 172.21.15.39:0/401106911 shutdown_connections 2024-01-24T17:46:18.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.721+0000 7f4400200640 1 -- 172.21.15.39:0/401106911 wait complete. 2024-01-24T17:46:18.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.724+0000 7f4400200640 1 Processor -- start 2024-01-24T17:46:18.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.724+0000 7f4400200640 1 -- start start 2024-01-24T17:46:18.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.725+0000 7f4400200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80afcc0 0x7f43f8147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:18.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.725+0000 7f4400200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80b2a00 0x7f43f80cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:18.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.725+0000 7f4400200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43f8150910 0x7f43f814e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:18.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.725+0000 7f4400200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80cd100 con 0x7f43f8150910 2024-01-24T17:46:18.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.725+0000 7f4400200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f8147990 con 0x7f43f80afcc0 2024-01-24T17:46:18.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.725+0000 7f4400200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80cbd20 con 0x7f43f80b2a00 2024-01-24T17:46:18.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43f8150910 0x7f43f814e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:18.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43f8150910 0x7f43f814e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38114/0 (socket says 172.21.15.39:38114) 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 -- 172.21.15.39:0/305662100 learned_addr learned my addr 172.21.15.39:0/305662100 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fd800640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80afcc0 0x7f43f8147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fce00640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80b2a00 0x7f43f80cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 -- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80b2a00 msgr2=0x7f43f80cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80b2a00 0x7f43f80cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 -- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80afcc0 msgr2=0x7f43f8147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80afcc0 0x7f43f8147450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 -- 172.21.15.39:0/305662100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0009b30 con 0x7f43f8150910 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fd800640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80afcc0 0x7f43f8147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.726+0000 7f43fe200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43f8150910 0x7f43f814e5e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009e50 tx=0x7f43f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.727+0000 7f43f5600640 1 -- 172.21.15.39:0/305662100 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0009510 con 0x7f43f8150910 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.727+0000 7f43f5600640 1 -- 172.21.15.39:0/305662100 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f003e450 con 0x7f43f8150910 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.727+0000 7f43f5600640 1 -- 172.21.15.39:0/305662100 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f003e800 con 0x7f43f8150910 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.727+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f80cc000 con 0x7f43f8150910 2024-01-24T17:46:18.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.727+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f80cc5c0 con 0x7f43f8150910 2024-01-24T17:46:18.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.728+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f804e840 con 0x7f43f8150910 2024-01-24T17:46:18.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.730+0000 7f43f5600640 1 -- 172.21.15.39:0/305662100 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f43f003e960 con 0x7f43f8150910 2024-01-24T17:46:18.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.730+0000 7f43f5600640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f43d006a460 0x7f43d006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:18.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.731+0000 7f43f5600640 1 -- 172.21.15.39:0/305662100 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f43f00ab340 con 0x7f43f8150910 2024-01-24T17:46:18.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.731+0000 7f43fd800640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f43d006a460 0x7f43d006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:18.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.732+0000 7f43fd800640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f43d006a460 0x7f43d006c920 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f43e8007fd0 tx=0x7f43e8007d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:18.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.735+0000 7f43f5600640 1 -- 172.21.15.39:0/305662100 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f43f0076930 con 0x7f43f8150910 2024-01-24T17:46:18.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:18.930+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f43f806a420 con 0x7f43f8150910 2024-01-24T17:46:19.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.429+0000 7f43f5600640 1 -- 172.21.15.39:0/305662100 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v66) v1 ==== 181+0+0 (secure 0 0 0) 0x7f43f0079f50 con 0x7f43f8150910 2024-01-24T17:46:19.431 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-01-24T17:46:19.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.433+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f43d006a460 msgr2=0x7f43d006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:19.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.433+0000 7f4400200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f43d006a460 0x7f43d006c920 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f43e8007fd0 tx=0x7f43e8007d40 comp rx=0 tx=0).stop 2024-01-24T17:46:19.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.433+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43f8150910 msgr2=0x7f43f814e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:19.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.433+0000 7f4400200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43f8150910 0x7f43f814e5e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009e50 tx=0x7f43f0004a40 comp rx=0 tx=0).stop 2024-01-24T17:46:19.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.434+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 shutdown_connections 2024-01-24T17:46:19.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.434+0000 7f4400200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f43d006a460 0x7f43d006c920 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.434+0000 7f4400200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43f8150910 0x7f43f814e5e0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.434+0000 7f4400200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43f80b2a00 0x7f43f80cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.434+0000 7f4400200640 1 --2- 172.21.15.39:0/305662100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f43f80afcc0 0x7f43f8147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.434+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 >> 172.21.15.39:0/305662100 conn(0x7f43f8069c70 msgr2=0x7f43f80b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:19.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.434+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 shutdown_connections 2024-01-24T17:46:19.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.434+0000 7f4400200640 1 -- 172.21.15.39:0/305662100 wait complete. 2024-01-24T17:46:19.461 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-24T17:46:19.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.695+0000 7f291ea00640 1 Processor -- start 2024-01-24T17:46:19.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.695+0000 7f291ea00640 1 -- start start 2024-01-24T17:46:19.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.696+0000 7f291ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f29181586b0 0x7f29180b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:19.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.696+0000 7f291ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b1f40 0x7f29180adf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:19.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.696+0000 7f291ea00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29180b2520 con 0x7f29180b3920 2024-01-24T17:46:19.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f291ea00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29180b3db0 con 0x7f29181586b0 2024-01-24T17:46:19.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f291ea00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29180b2800 con 0x7f29180b1f40 2024-01-24T17:46:19.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f2917e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f29181586b0 0x7f29180b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:19.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f2917e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f29181586b0 0x7f29180b33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38128/0 (socket says 172.21.15.39:38128) 2024-01-24T17:46:19.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f2917e00640 1 -- 172.21.15.39:0/1897505091 learned_addr learned my addr 172.21.15.39:0/1897505091 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:19.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f291ca00640 1 --2- 172.21.15.39:0/1897505091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b1f40 0x7f29180adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:19.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f2917e00640 1 -- 172.21.15.39:0/1897505091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b1f40 msgr2=0x7f29180adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:19.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f2917e00640 1 --2- 172.21.15.39:0/1897505091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b1f40 0x7f29180adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f2917e00640 1 -- 172.21.15.39:0/1897505091 >> v1:172.21.15.100:6789/0 conn(0x7f29180b3920 legacy=0x7f29180b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:19.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.697+0000 7f2917e00640 1 -- 172.21.15.39:0/1897505091 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29180b2ae0 con 0x7f29181586b0 2024-01-24T17:46:19.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.699+0000 7f2917e00640 1 --2- 172.21.15.39:0/1897505091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 0x7f29180b33e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f2908009870 tx=0x7f290802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb28415453143f28 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:19.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.699+0000 7f2916a00640 1 -- 172.21.15.39:0/1897505091 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908031740 con 0x7f29181586b0 2024-01-24T17:46:19.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.699+0000 7f2916a00640 1 -- 172.21.15.39:0/1897505091 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29080318a0 con 0x7f29181586b0 2024-01-24T17:46:19.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.699+0000 7f2916a00640 1 -- 172.21.15.39:0/1897505091 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908031c30 con 0x7f29181586b0 2024-01-24T17:46:19.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.699+0000 7f291ea00640 1 -- 172.21.15.39:0/1897505091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 msgr2=0x7f29180b33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:19.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.699+0000 7f291ea00640 1 --2- 172.21.15.39:0/1897505091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 0x7f29180b33e0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f2908009870 tx=0x7f290802b7e0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.700+0000 7f291ea00640 1 -- 172.21.15.39:0/1897505091 shutdown_connections 2024-01-24T17:46:19.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.700+0000 7f291ea00640 1 --2- 172.21.15.39:0/1897505091 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b1f40 0x7f29180adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.700+0000 7f291ea00640 1 --2- 172.21.15.39:0/1897505091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 0x7f29180b33e0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.700+0000 7f291ea00640 1 -- 172.21.15.39:0/1897505091 >> 172.21.15.39:0/1897505091 conn(0x7f2918069c70 msgr2=0x7f291806a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:19.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.700+0000 7f291ea00640 1 -- 172.21.15.39:0/1897505091 shutdown_connections 2024-01-24T17:46:19.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.700+0000 7f291ea00640 1 -- 172.21.15.39:0/1897505091 wait complete. 2024-01-24T17:46:19.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.701+0000 7f291ea00640 1 Processor -- start 2024-01-24T17:46:19.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.702+0000 7f291ea00640 1 -- start start 2024-01-24T17:46:19.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f291ea00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29180b1f40 0x7f291814dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:19.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f291ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b3920 0x7f29180cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:19.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f291ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 0x7f29180cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:19.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f291ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29180ace80 con 0x7f29181586b0 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f291ea00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291814e520 con 0x7f29180b1f40 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f291ea00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29180a98c0 con 0x7f29180b3920 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f2917e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29180b1f40 0x7f291814dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f291ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 0x7f29180cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f2917e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29180b1f40 0x7f291814dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:51942/0 (socket says 172.21.15.39:51942) 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.703+0000 7f2917e00640 1 -- 172.21.15.39:0/826684305 learned_addr learned my addr 172.21.15.39:0/826684305 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.704+0000 7f2917400640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b3920 0x7f29180cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.704+0000 7f291ca00640 1 -- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b3920 msgr2=0x7f29180cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.704+0000 7f291ca00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b3920 0x7f29180cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.704+0000 7f291ca00640 1 -- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29180b1f40 msgr2=0x7f291814dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:19.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.704+0000 7f291ca00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29180b1f40 0x7f291814dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:19.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.704+0000 7f291ca00640 1 -- 172.21.15.39:0/826684305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29080094d0 con 0x7f29181586b0 2024-01-24T17:46:19.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.704+0000 7f291ca00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 0x7f29180cdfb0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2910007dc0 tx=0x7f291000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:19.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.705+0000 7f290fe00640 1 -- 172.21.15.39:0/826684305 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910018070 con 0x7f29181586b0 2024-01-24T17:46:19.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.705+0000 7f290fe00640 1 -- 172.21.15.39:0/826684305 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291000edd0 con 0x7f29181586b0 2024-01-24T17:46:19.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.705+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29180ac5e0 con 0x7f29181586b0 2024-01-24T17:46:19.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.705+0000 7f290fe00640 1 -- 172.21.15.39:0/826684305 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910003bb0 con 0x7f29181586b0 2024-01-24T17:46:19.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.705+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29180acba0 con 0x7f29181586b0 2024-01-24T17:46:19.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.706+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291804e840 con 0x7f29181586b0 2024-01-24T17:46:19.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.707+0000 7f290fe00640 1 -- 172.21.15.39:0/826684305 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2910003d10 con 0x7f29181586b0 2024-01-24T17:46:19.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.708+0000 7f290fe00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f28f006a430 0x7f28f006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:19.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.708+0000 7f290fe00640 1 -- 172.21.15.39:0/826684305 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f2910088970 con 0x7f29181586b0 2024-01-24T17:46:19.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.709+0000 7f2917e00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f28f006a430 0x7f28f006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:19.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.710+0000 7f2917e00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f28f006a430 0x7f28f006c8f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f29080090b0 tx=0x7f2908008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:19.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.713+0000 7f290fe00640 1 -- 172.21.15.39:0/826684305 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2910054070 con 0x7f29181586b0 2024-01-24T17:46:19.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:19.915+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f29180af9b0 con 0x7f29181586b0 2024-01-24T17:46:20.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.442+0000 7f290fe00640 1 -- 172.21.15.39:0/826684305 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v67) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2910057690 con 0x7f29181586b0 2024-01-24T17:46:20.444 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-01-24T17:46:20.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.447+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f28f006a430 msgr2=0x7f28f006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:20.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.447+0000 7f291ea00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f28f006a430 0x7f28f006c8f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f29080090b0 tx=0x7f2908008030 comp rx=0 tx=0).stop 2024-01-24T17:46:20.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.447+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 msgr2=0x7f29180cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:20.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.447+0000 7f291ea00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 0x7f29180cdfb0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2910007dc0 tx=0x7f291000e4b0 comp rx=0 tx=0).stop 2024-01-24T17:46:20.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.448+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 shutdown_connections 2024-01-24T17:46:20.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.448+0000 7f291ea00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f28f006a430 0x7f28f006c8f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:20.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.448+0000 7f291ea00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29181586b0 0x7f29180cdfb0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:20.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.448+0000 7f291ea00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29180b3920 0x7f29180cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:20.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.448+0000 7f291ea00640 1 --2- 172.21.15.39:0/826684305 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29180b1f40 0x7f291814dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:20.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.448+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 >> 172.21.15.39:0/826684305 conn(0x7f2918069c70 msgr2=0x7f29180b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:20.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.448+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 shutdown_connections 2024-01-24T17:46:20.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:20.448+0000 7f291ea00640 1 -- 172.21.15.39:0/826684305 wait complete. 2024-01-24T17:46:20.478 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:46:20.485 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:46:20.652 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-01-24T17:46:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:46:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-01-24T17:46:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:46:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-01-24T17:46:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-01-24T17:46:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:46:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-01-24T17:46:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-01-24T17:46:20.663 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:46:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:46:20.665 INFO:teuthology.orchestra.run.smithi039.stdout:[client.badguy] 2024-01-24T17:46:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: client_metadata = root=/baz 2024-01-24T17:46:20.666 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-01-24T17:46:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-01-24T17:46:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:46:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-01-24T17:46:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:46:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-01-24T17:46:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:46:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-01-24T17:46:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = bitmap 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-01-24T17:46:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:46:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: mon op complaint time = 120 2024-01-24T17:46:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.676 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-01-24T17:46:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-01-24T17:46:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-01-24T17:46:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-01-24T17:46:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:46:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:46:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: client mount timeout = 600 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = "" 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout:[mds] 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds = 20 2024-01-24T17:46:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds balancer = 20 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug frag = True 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug scatterstat = True 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: mds op complaint time = 180 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: mds verify scatter = True 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout:[client.badguy] 2024-01-24T17:46:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: client_metadata = root=/baz 2024-01-24T17:46:20.698 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:46:20.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:46:20.854 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-01-24T17:46:20.854 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-01-24T17:46:20.854 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:46:20.854 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-01-24T17:46:20.854 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.854 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-01-24T17:46:20.854 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.854 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.855 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.856 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.857 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:46:20.858 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-01-24T17:46:20.859 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = bitmap 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:46:20.860 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:46:20.861 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.862 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: mon op complaint time = 120 2024-01-24T17:46:20.863 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:46:20.864 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-01-24T17:46:20.864 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-01-24T17:46:20.864 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-01-24T17:46:20.864 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-01-24T17:46:20.864 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:46:20.864 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:46:20.864 INFO:teuthology.orchestra.run.smithi100.stdout: client mount timeout = 600 2024-01-24T17:46:20.864 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2024-01-24T17:46:20.865 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.865 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = "" 2024-01-24T17:46:20.865 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:46:20.865 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:46:20.865 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-01-24T17:46:20.865 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-01-24T17:46:20.865 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-01-24T17:46:20.865 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout:[mds] 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds = 20 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds balancer = 20 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug frag = True 2024-01-24T17:46:20.866 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug scatterstat = True 2024-01-24T17:46:20.867 INFO:teuthology.orchestra.run.smithi100.stdout: mds op complaint time = 180 2024-01-24T17:46:20.867 INFO:teuthology.orchestra.run.smithi100.stdout: mds verify scatter = True 2024-01-24T17:46:20.867 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:46:20.867 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:46:20.867 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:46:20.867 INFO:teuthology.orchestra.run.smithi100.stdout:[client.badguy] 2024-01-24T17:46:20.874 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-01-24T17:46:20.874 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-01-24T17:46:20.874 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:46:20.874 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-01-24T17:46:20.875 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:46:20.876 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-01-24T17:46:20.877 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-01-24T17:46:20.878 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.879 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-01-24T17:46:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-01-24T17:46:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:46:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-01-24T17:46:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-01-24T17:46:20.884 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:46:20.885 INFO:teuthology.orchestra.run.smithi039.stdout:[client.badguy] 2024-01-24T17:46:20.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-24T17:46:20.963 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:46:20.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:46:21.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.124+0000 7ff63b200640 1 Processor -- start 2024-01-24T17:46:21.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.124+0000 7ff63b200640 1 -- start start 2024-01-24T17:46:21.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.126+0000 7ff63b200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff63c1516c0 0x7ff63c0b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.126+0000 7ff63b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff63c070430 0x7ff63c0ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.126+0000 7ff63b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c0b12b0 con 0x7ff63c15ac70 2024-01-24T17:46:21.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.126+0000 7ff63b200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63c15b170 con 0x7ff63c1516c0 2024-01-24T17:46:21.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.126+0000 7ff63b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c15b450 con 0x7ff63c070430 2024-01-24T17:46:21.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.126+0000 7ff633e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff63c15ac70 0x7ff63c06fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58832/0 (socket says 172.21.15.39:58832) 2024-01-24T17:46:21.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.126+0000 7ff633e00640 1 -- 172.21.15.39:0/2095129008 learned_addr learned my addr 172.21.15.39:0/2095129008 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff63a800640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff63c070430 0x7ff63c0ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639e00640 1 --2- 172.21.15.39:0/2095129008 >> v2:172.21.15.100:3300/0 conn(0x7ff63c1516c0 0x7ff63c0b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904486283 0 0) 0x7ff63c0b12b0 con 0x7ff63c15ac70 2024-01-24T17:46:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff61c003520 con 0x7ff63c15ac70 2024-01-24T17:46:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578923077 0 0) 0x7ff61c003520 con 0x7ff63c15ac70 2024-01-24T17:46:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff63c070430 msgr2=0x7ff63c0ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff63c070430 0x7ff63c0ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 >> v2:172.21.15.100:3300/0 conn(0x7ff63c1516c0 msgr2=0x7ff63c0b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 --2- 172.21.15.39:0/2095129008 >> v2:172.21.15.100:3300/0 conn(0x7ff63c1516c0 0x7ff63c0b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c15b730 con 0x7ff63c15ac70 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff624001670 con 0x7ff63c15ac70 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 >> v1:172.21.15.39:6789/0 conn(0x7ff63c15ac70 legacy=0x7ff63c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff61c003970 0x7ff61c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff63c070430 0x7ff61c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff61c004d00 0x7ff61c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c15b170 con 0x7ff61c004d00 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c15b450 con 0x7ff61c003970 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.127+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c00b980 con 0x7ff63c070430 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.128+0000 7ff63a800640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff61c004d00 0x7ff61c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.128+0000 7ff639e00640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff61c003970 0x7ff61c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.128+0000 7ff633e00640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff63c070430 0x7ff61c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.128+0000 7ff63a800640 1 -- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff63c070430 msgr2=0x7ff61c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.128+0000 7ff63a800640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff63c070430 0x7ff61c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.128+0000 7ff63a800640 1 -- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff61c003970 msgr2=0x7ff61c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.128+0000 7ff63a800640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff61c003970 0x7ff61c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.128+0000 7ff63a800640 1 -- 172.21.15.39:0/2095129008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c003520 con 0x7ff61c004d00 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.129+0000 7ff63a800640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff61c004d00 0x7ff61c005180 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0b09a0 tx=0x7ff61c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f42a77d24fc11af server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63402cb50 con 0x7ff61c004d00 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63400f040 con 0x7ff61c004d00 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff639400640 1 -- 172.21.15.39:0/2095129008 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634034440 con 0x7ff61c004d00 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff63b200640 1 -- 172.21.15.39:0/2095129008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff61c004d00 msgr2=0x7ff61c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff63b200640 1 --2- 172.21.15.39:0/2095129008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff61c004d00 0x7ff61c005180 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0b09a0 tx=0x7ff61c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff63a800640 1 -- 172.21.15.39:0/2095129008 reap_dead start 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff63b200640 1 -- 172.21.15.39:0/2095129008 shutdown_connections 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff63b200640 1 -- 172.21.15.39:0/2095129008 >> 172.21.15.39:0/2095129008 conn(0x7ff63c06a200 msgr2=0x7ff63c0affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff63b200640 1 -- 172.21.15.39:0/2095129008 shutdown_connections 2024-01-24T17:46:21.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.130+0000 7ff63b200640 1 -- 172.21.15.39:0/2095129008 wait complete. 2024-01-24T17:46:21.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.132+0000 7ff63b200640 1 Processor -- start 2024-01-24T17:46:21.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.132+0000 7ff63b200640 1 -- start start 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.133+0000 7ff63b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff61c004d00 0x7ff63c147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.133+0000 7ff63b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff63c0aca80 0x7ff63c1f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.133+0000 7ff63b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff63c0a77a0 0x7ff63c0a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.133+0000 7ff63b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c147b10 con 0x7ff63c0aca80 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.133+0000 7ff63b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c1481f0 con 0x7ff63c0a77a0 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.133+0000 7ff63b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0cc650 con 0x7ff61c004d00 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff63c0aca80 0x7ff63c1f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff63c0aca80 0x7ff63c1f4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38154/0 (socket says 172.21.15.39:38154) 2024-01-24T17:46:21.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 -- 172.21.15.39:0/3331876365 learned_addr learned my addr 172.21.15.39:0/3331876365 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff63a800640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff63c0a77a0 0x7ff63c0a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff639e00640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff61c004d00 0x7ff63c147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 -- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff61c004d00 msgr2=0x7ff63c147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff61c004d00 0x7ff63c147cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 -- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff63c0a77a0 msgr2=0x7ff63c0a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff63c0a77a0 0x7ff63c0a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 -- 172.21.15.39:0/3331876365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0cc070 con 0x7ff63c0aca80 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff639e00640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff61c004d00 0x7ff63c147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff63a800640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff63c0a77a0 0x7ff63c0a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.134+0000 7ff633e00640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff63c0aca80 0x7ff63c1f4fc0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff62400e760 tx=0x7ff62400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.135+0000 7ff632000640 1 -- 172.21.15.39:0/3331876365 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62400ee70 con 0x7ff63c0aca80 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.135+0000 7ff632000640 1 -- 172.21.15.39:0/3331876365 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62400c870 con 0x7ff63c0aca80 2024-01-24T17:46:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.135+0000 7ff632000640 1 -- 172.21.15.39:0/3331876365 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62400cc00 con 0x7ff63c0aca80 2024-01-24T17:46:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.135+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c0acff0 con 0x7ff63c0aca80 2024-01-24T17:46:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.135+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c1f5500 con 0x7ff63c0aca80 2024-01-24T17:46:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.136+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c04dff0 con 0x7ff63c0aca80 2024-01-24T17:46:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.137+0000 7ff632000640 1 -- 172.21.15.39:0/3331876365 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff624013420 con 0x7ff63c0aca80 2024-01-24T17:46:21.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.138+0000 7ff632000640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff61406a460 0x7ff61406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.138+0000 7ff632000640 1 -- 172.21.15.39:0/3331876365 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff6240882a0 con 0x7ff63c0aca80 2024-01-24T17:46:21.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.138+0000 7ff639e00640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff61406a460 0x7ff61406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.139+0000 7ff639e00640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff61406a460 0x7ff61406c920 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009a80 tx=0x7ff62c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:21.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.142+0000 7ff632000640 1 -- 172.21.15.39:0/3331876365 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff624053b80 con 0x7ff63c0aca80 2024-01-24T17:46:21.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.338+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff63c0cc790 con 0x7ff63c0aca80 2024-01-24T17:46:21.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.344+0000 7ff632000640 1 -- 172.21.15.39:0/3331876365 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff6240571a0 con 0x7ff63c0aca80 2024-01-24T17:46:21.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff61406a460 msgr2=0x7ff61406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff61406a460 0x7ff61406c920 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009a80 tx=0x7ff62c008040 comp rx=0 tx=0).stop 2024-01-24T17:46:21.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff63c0aca80 msgr2=0x7ff63c1f4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff63c0aca80 0x7ff63c1f4fc0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff62400e760 tx=0x7ff62400ec20 comp rx=0 tx=0).stop 2024-01-24T17:46:21.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 shutdown_connections 2024-01-24T17:46:21.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff63c0a77a0 0x7ff63c0a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff61406a460 0x7ff61406c920 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff63c0aca80 0x7ff63c1f4fc0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 --2- 172.21.15.39:0/3331876365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff61c004d00 0x7ff63c147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.346+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 >> 172.21.15.39:0/3331876365 conn(0x7ff63c06a200 msgr2=0x7ff63c15cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:21.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.347+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 shutdown_connections 2024-01-24T17:46:21.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.347+0000 7ff63b200640 1 -- 172.21.15.39:0/3331876365 wait complete. 2024-01-24T17:46:21.366 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_reject' 2024-01-24T17:46:21.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.582+0000 7ff62b200640 1 Processor -- start 2024-01-24T17:46:21.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.582+0000 7ff62b200640 1 -- start start 2024-01-24T17:46:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.584+0000 7ff62b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff62415db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.584+0000 7ff62b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6240ccbe0 0x7ff6240c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.584+0000 7ff62b200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62415e0d0 con 0x7ff62406bd40 2024-01-24T17:46:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.584+0000 7ff62b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62415d140 con 0x7ff6240ccbe0 2024-01-24T17:46:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.584+0000 7ff62b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62415d700 con 0x7ff624159740 2024-01-24T17:46:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.584+0000 7ff629400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6240ccbe0 0x7ff6240c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.585+0000 7ff629400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6240ccbe0 0x7ff6240c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38164/0 (socket says 172.21.15.39:38164) 2024-01-24T17:46:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.585+0000 7ff629400640 1 -- 172.21.15.39:0/1857383758 learned_addr learned my addr 172.21.15.39:0/1857383758 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.585+0000 7ff629e00640 1 --2- 172.21.15.39:0/1857383758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff62415db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.585+0000 7ff629400640 1 -- 172.21.15.39:0/1857383758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 msgr2=0x7ff62415db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.585+0000 7ff629400640 1 --2- 172.21.15.39:0/1857383758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff62415db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.585+0000 7ff629400640 1 -- 172.21.15.39:0/1857383758 >> v1:172.21.15.100:6789/0 conn(0x7ff62406bd40 legacy=0x7ff624070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.585+0000 7ff629400640 1 -- 172.21.15.39:0/1857383758 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240ca500 con 0x7ff6240ccbe0 2024-01-24T17:46:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.586+0000 7ff629400640 1 --2- 172.21.15.39:0/1857383758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240c9c20 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009870 tx=0x7ff60c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a41294f6f36c2c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.587+0000 7ff628a00640 1 -- 172.21.15.39:0/1857383758 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c031640 con 0x7ff6240ccbe0 2024-01-24T17:46:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.587+0000 7ff628a00640 1 -- 172.21.15.39:0/1857383758 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c0317a0 con 0x7ff6240ccbe0 2024-01-24T17:46:21.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.587+0000 7ff628a00640 1 -- 172.21.15.39:0/1857383758 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c031b30 con 0x7ff6240ccbe0 2024-01-24T17:46:21.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.587+0000 7ff62b200640 1 -- 172.21.15.39:0/1857383758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 msgr2=0x7ff6240c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.587+0000 7ff62b200640 1 --2- 172.21.15.39:0/1857383758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240c9c20 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009870 tx=0x7ff60c02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.588+0000 7ff62b200640 1 -- 172.21.15.39:0/1857383758 shutdown_connections 2024-01-24T17:46:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.588+0000 7ff62b200640 1 --2- 172.21.15.39:0/1857383758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240c9c20 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.588+0000 7ff62b200640 1 --2- 172.21.15.39:0/1857383758 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff62415db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.588+0000 7ff62b200640 1 -- 172.21.15.39:0/1857383758 >> 172.21.15.39:0/1857383758 conn(0x7ff62406a200 msgr2=0x7ff6240aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.588+0000 7ff62b200640 1 -- 172.21.15.39:0/1857383758 shutdown_connections 2024-01-24T17:46:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.588+0000 7ff62b200640 1 -- 172.21.15.39:0/1857383758 wait complete. 2024-01-24T17:46:21.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.590+0000 7ff62b200640 1 Processor -- start 2024-01-24T17:46:21.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.590+0000 7ff62b200640 1 -- start start 2024-01-24T17:46:21.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff62b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff62406bd40 0x7ff6240af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff62b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff62b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff6240abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff62b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240c39b0 con 0x7ff6240ccbe0 2024-01-24T17:46:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff62b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240c3c90 con 0x7ff62406bd40 2024-01-24T17:46:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff62b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240c3040 con 0x7ff624159740 2024-01-24T17:46:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff629400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff629400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240ab8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38180/0 (socket says 172.21.15.39:38180) 2024-01-24T17:46:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff629400640 1 -- 172.21.15.39:0/2845280801 learned_addr learned my addr 172.21.15.39:0/2845280801 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:21.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff629e00640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff62406bd40 0x7ff6240af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff629400640 1 -- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 msgr2=0x7ff6240abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.591+0000 7ff62a800640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff6240abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff629400640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff6240abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff629400640 1 -- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff62406bd40 msgr2=0x7ff6240af8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff629400640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff62406bd40 0x7ff6240af8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff629400640 1 -- 172.21.15.39:0/2845280801 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0094d0 con 0x7ff6240ccbe0 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff629e00640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff62406bd40 0x7ff6240af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff62a800640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff6240abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff629400640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240ab8c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009870 tx=0x7ff60c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff622000640 1 -- 172.21.15.39:0/2845280801 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c03ac60 con 0x7ff6240ccbe0 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff622000640 1 -- 172.21.15.39:0/2845280801 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c03adc0 con 0x7ff6240ccbe0 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff622000640 1 -- 172.21.15.39:0/2845280801 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c044630 con 0x7ff6240ccbe0 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.592+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6240c3320 con 0x7ff6240ccbe0 2024-01-24T17:46:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.593+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6240afe20 con 0x7ff6240ccbe0 2024-01-24T17:46:21.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.594+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62404dff0 con 0x7ff6240ccbe0 2024-01-24T17:46:21.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.595+0000 7ff622000640 1 -- 172.21.15.39:0/2845280801 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff60c02f050 con 0x7ff6240ccbe0 2024-01-24T17:46:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.595+0000 7ff622000640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff5fc06a430 0x7ff5fc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.596+0000 7ff622000640 1 -- 172.21.15.39:0/2845280801 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff60c0aa490 con 0x7ff6240ccbe0 2024-01-24T17:46:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.596+0000 7ff629e00640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff5fc06a430 0x7ff5fc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:21.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.597+0000 7ff629e00640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff5fc06a430 0x7ff5fc06c8f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff614007380 tx=0x7ff61400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:21.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.600+0000 7ff622000640 1 -- 172.21.15.39:0/2845280801 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff60c075cf0 con 0x7ff6240ccbe0 2024-01-24T17:46:21.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:21.862+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_reject"]} v 0) v1 -- 0x7ff62415e0d0 con 0x7ff6240ccbe0 2024-01-24T17:46:22.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.455+0000 7ff622000640 1 -- 172.21.15.39:0/2845280801 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_reject"]}]=0 v258) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff62415e0d0 con 0x7ff6240ccbe0 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.458+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff5fc06a430 msgr2=0x7ff5fc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.458+0000 7ff62b200640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff5fc06a430 0x7ff5fc06c8f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff614007380 tx=0x7ff61400b040 comp rx=0 tx=0).stop 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.458+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 msgr2=0x7ff6240ab8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.458+0000 7ff62b200640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240ab8c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009870 tx=0x7ff60c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.458+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 shutdown_connections 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.458+0000 7ff62b200640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff624159740 0x7ff6240abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.459+0000 7ff62b200640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff5fc06a430 0x7ff5fc06c8f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.459+0000 7ff62b200640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6240ccbe0 0x7ff6240ab8c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.459+0000 7ff62b200640 1 --2- 172.21.15.39:0/2845280801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff62406bd40 0x7ff6240af8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.459+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 >> 172.21.15.39:0/2845280801 conn(0x7ff62406a200 msgr2=0x7ff62415c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.459+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 shutdown_connections 2024-01-24T17:46:22.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.459+0000 7ff62b200640 1 -- 172.21.15.39:0/2845280801 wait complete. 2024-01-24T17:46:22.481 INFO:tasks.cephfs_test_runner:test_session_reject (tasks.cephfs.test_sessionmap.TestSessionMap) ... ok 2024-01-24T17:46:22.481 INFO:tasks.cephfs_test_runner:Starting test: test_tell_conn_close (tasks.cephfs.test_sessionmap.TestSessionMap) That when a `tell` command is sent using the python CLI, 2024-01-24T17:46:22.482 INFO:tasks.cephfs_test_runner:test_tell_conn_close (tasks.cephfs.test_sessionmap.TestSessionMap) 2024-01-24T17:46:22.482 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_conn_close' 2024-01-24T17:46:22.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.695+0000 7ff425a00640 1 Processor -- start 2024-01-24T17:46:22.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.695+0000 7ff425a00640 1 -- start start 2024-01-24T17:46:22.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.696+0000 7ff425a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff42015f390 0x7ff420165ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:22.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.696+0000 7ff425a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff420070470 con 0x7ff420070920 2024-01-24T17:46:22.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.696+0000 7ff425a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42015ef00 con 0x7ff4200b0010 2024-01-24T17:46:22.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.696+0000 7ff425a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4200710a0 con 0x7ff42015f390 2024-01-24T17:46:22.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.697+0000 7ff415e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff420070920 0x7ff4200b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58842/0 (socket says 172.21.15.39:58842) 2024-01-24T17:46:22.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.697+0000 7ff415e00640 1 -- 172.21.15.39:0/3704262281 learned_addr learned my addr 172.21.15.39:0/3704262281 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:22.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.697+0000 7ff41f400640 1 --2- 172.21.15.39:0/3704262281 >> v2:172.21.15.39:3300/0 conn(0x7ff42015f390 0x7ff420165ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:22.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.697+0000 7ff41e000640 1 -- 172.21.15.39:0/3704262281 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354464736 0 0) 0x7ff420070470 con 0x7ff420070920 2024-01-24T17:46:22.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.697+0000 7ff41e000640 1 -- 172.21.15.39:0/3704262281 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff400003520 con 0x7ff420070920 2024-01-24T17:46:22.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.698+0000 7ff41f400640 1 -- 172.21.15.39:0/3704262281 >> v1:172.21.15.100:6789/0 conn(0x7ff4200b0010 legacy=0x7ff42015bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:22.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.698+0000 7ff41f400640 1 -- 172.21.15.39:0/3704262281 >> v1:172.21.15.39:6789/0 conn(0x7ff420070920 legacy=0x7ff4200b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:22.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.698+0000 7ff41f400640 1 -- 172.21.15.39:0/3704262281 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420070dc0 con 0x7ff42015f390 2024-01-24T17:46:22.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.699+0000 7ff41f400640 1 --2- 172.21.15.39:0/3704262281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff420165ac0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff418009d90 tx=0x7ff41802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50cb75cb7851352 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:22.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.699+0000 7ff41e000640 1 -- 172.21.15.39:0/3704262281 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41802cd80 con 0x7ff42015f390 2024-01-24T17:46:22.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.699+0000 7ff41e000640 1 -- 172.21.15.39:0/3704262281 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff418034440 con 0x7ff42015f390 2024-01-24T17:46:22.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.699+0000 7ff41e000640 1 -- 172.21.15.39:0/3704262281 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4180347d0 con 0x7ff42015f390 2024-01-24T17:46:22.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.700+0000 7ff425a00640 1 -- 172.21.15.39:0/3704262281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 msgr2=0x7ff420165ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:22.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.700+0000 7ff425a00640 1 --2- 172.21.15.39:0/3704262281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff420165ac0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff418009d90 tx=0x7ff41802bfd0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.700+0000 7ff425a00640 1 -- 172.21.15.39:0/3704262281 shutdown_connections 2024-01-24T17:46:22.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.700+0000 7ff425a00640 1 --2- 172.21.15.39:0/3704262281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff420165ac0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.700+0000 7ff425a00640 1 -- 172.21.15.39:0/3704262281 >> 172.21.15.39:0/3704262281 conn(0x7ff42006bf20 msgr2=0x7ff42006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:22.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.700+0000 7ff425a00640 1 -- 172.21.15.39:0/3704262281 shutdown_connections 2024-01-24T17:46:22.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.700+0000 7ff425a00640 1 -- 172.21.15.39:0/3704262281 wait complete. 2024-01-24T17:46:22.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.702+0000 7ff425a00640 1 Processor -- start 2024-01-24T17:46:22.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.702+0000 7ff425a00640 1 -- start start 2024-01-24T17:46:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff425a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff420070920 0x7ff420156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff425a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4200b0010 0x7ff4200a8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff425a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff4200a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff425a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200c3550 con 0x7ff42015f390 2024-01-24T17:46:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff425a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200c2f60 con 0x7ff4200b0010 2024-01-24T17:46:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff425a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200a8a70 con 0x7ff420070920 2024-01-24T17:46:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff41f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff4200a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff41f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff4200a9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38192/0 (socket says 172.21.15.39:38192) 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff41f400640 1 -- 172.21.15.39:0/515886904 learned_addr learned my addr 172.21.15.39:0/515886904 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff415e00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4200b0010 0x7ff4200a8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff41ea00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff420070920 0x7ff420156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.703+0000 7ff41f400640 1 -- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff420070920 msgr2=0x7ff420156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff41f400640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff420070920 0x7ff420156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff41f400640 1 -- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4200b0010 msgr2=0x7ff4200a8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff41f400640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4200b0010 0x7ff4200a8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff41f400640 1 -- 172.21.15.39:0/515886904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff418009a70 con 0x7ff42015f390 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff415e00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4200b0010 0x7ff4200a8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff41ea00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff420070920 0x7ff420156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff41f400640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff4200a9af0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff418009d90 tx=0x7ff418004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff417400640 1 -- 172.21.15.39:0/515886904 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff418040650 con 0x7ff42015f390 2024-01-24T17:46:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff417400640 1 -- 172.21.15.39:0/515886904 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4180407b0 con 0x7ff42015f390 2024-01-24T17:46:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff417400640 1 -- 172.21.15.39:0/515886904 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff418040b60 con 0x7ff42015f390 2024-01-24T17:46:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.704+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4200a8d50 con 0x7ff42015f390 2024-01-24T17:46:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.705+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff420157210 con 0x7ff42015f390 2024-01-24T17:46:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.706+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4200c3690 con 0x7ff42015f390 2024-01-24T17:46:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.707+0000 7ff417400640 1 -- 172.21.15.39:0/515886904 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff418040cc0 con 0x7ff42015f390 2024-01-24T17:46:22.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.707+0000 7ff417400640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f806a430 0x7ff3f806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:22.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.708+0000 7ff417400640 1 -- 172.21.15.39:0/515886904 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff4180ab410 con 0x7ff42015f390 2024-01-24T17:46:22.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.708+0000 7ff41ea00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f806a430 0x7ff3f806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:22.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.710+0000 7ff41ea00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f806a430 0x7ff3f806c8f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff410000f60 tx=0x7ff410009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:22.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.712+0000 7ff417400640 1 -- 172.21.15.39:0/515886904 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff4200c3690 con 0x7ff42015f390 2024-01-24T17:46:22.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:22.980+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_conn_close"]} v 0) v1 -- 0x7ff4200c3690 con 0x7ff42015f390 2024-01-24T17:46:23.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.462+0000 7ff417400640 1 -- 172.21.15.39:0/515886904 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_conn_close"]}]=0 v259) v1 ==== 146+0+0 (secure 0 0 0) 0x7ff4200c3690 con 0x7ff42015f390 2024-01-24T17:46:23.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f806a430 msgr2=0x7ff3f806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:23.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f806a430 0x7ff3f806c8f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff410000f60 tx=0x7ff410009040 comp rx=0 tx=0).stop 2024-01-24T17:46:23.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 msgr2=0x7ff4200a9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:23.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff4200a9af0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ff418009d90 tx=0x7ff418004a40 comp rx=0 tx=0).stop 2024-01-24T17:46:23.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 shutdown_connections 2024-01-24T17:46:23.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f806a430 0x7ff3f806c8f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff42015f390 0x7ff4200a9af0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff4200b0010 0x7ff4200a8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 --2- 172.21.15.39:0/515886904 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff420070920 0x7ff420156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 >> 172.21.15.39:0/515886904 conn(0x7ff42006bf20 msgr2=0x7ff42015d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:23.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.465+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 shutdown_connections 2024-01-24T17:46:23.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.466+0000 7ff425a00640 1 -- 172.21.15.39:0/515886904 wait complete. 2024-01-24T17:46:23.488 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-24T17:46:23.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.711+0000 7f048f200640 1 Processor -- start 2024-01-24T17:46:23.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.711+0000 7f048f200640 1 -- start start 2024-01-24T17:46:23.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.713+0000 7f048f200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f048815d450 0x7f04880c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:23.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.713+0000 7f048f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04880cbf90 0x7f048815d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:23.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.713+0000 7f048f200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04880c9290 con 0x7f04880ce350 2024-01-24T17:46:23.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.713+0000 7f048f200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04880c9570 con 0x7f048815d450 2024-01-24T17:46:23.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.713+0000 7f048f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048806fed0 con 0x7f04880cbf90 2024-01-24T17:46:23.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.714+0000 7f048d400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04880cbf90 0x7f048815d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:23.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.714+0000 7f048d400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04880cbf90 0x7f048815d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:46594/0 (socket says 172.21.15.39:46594) 2024-01-24T17:46:23.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.714+0000 7f048d400640 1 -- 172.21.15.39:0/844064413 learned_addr learned my addr 172.21.15.39:0/844064413 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:23.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.714+0000 7f048de00640 1 --2- 172.21.15.39:0/844064413 >> v2:172.21.15.100:3300/0 conn(0x7f048815d450 0x7f04880c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:23.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.715+0000 7f048d400640 1 -- 172.21.15.39:0/844064413 >> v2:172.21.15.100:3300/0 conn(0x7f048815d450 msgr2=0x7f04880c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:23.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.715+0000 7f048d400640 1 --2- 172.21.15.39:0/844064413 >> v2:172.21.15.100:3300/0 conn(0x7f048815d450 0x7f04880c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.715+0000 7f048d400640 1 -- 172.21.15.39:0/844064413 >> v1:172.21.15.100:6789/0 conn(0x7f04880ce350 legacy=0x7f04880706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:23.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.715+0000 7f048d400640 1 -- 172.21.15.39:0/844064413 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04880701b0 con 0x7f04880cbf90 2024-01-24T17:46:23.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.716+0000 7f048d400640 1 --2- 172.21.15.39:0/844064413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04880cbf90 0x7f048815d830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0470009870 tx=0x7f047002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac7b1a4dda0e42a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:23.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048ca00640 1 -- 172.21.15.39:0/844064413 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470031640 con 0x7f04880cbf90 2024-01-24T17:46:23.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048ca00640 1 -- 172.21.15.39:0/844064413 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04700317a0 con 0x7f04880cbf90 2024-01-24T17:46:23.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048ca00640 1 -- 172.21.15.39:0/844064413 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470031b30 con 0x7f04880cbf90 2024-01-24T17:46:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048f200640 1 -- 172.21.15.39:0/844064413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04880cbf90 msgr2=0x7f048815d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048f200640 1 --2- 172.21.15.39:0/844064413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04880cbf90 0x7f048815d830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0470009870 tx=0x7f047002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048f200640 1 -- 172.21.15.39:0/844064413 shutdown_connections 2024-01-24T17:46:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048f200640 1 --2- 172.21.15.39:0/844064413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04880cbf90 0x7f048815d830 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048f200640 1 --2- 172.21.15.39:0/844064413 >> v2:172.21.15.100:3300/0 conn(0x7f048815d450 0x7f04880c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048f200640 1 -- 172.21.15.39:0/844064413 >> 172.21.15.39:0/844064413 conn(0x7f048806a200 msgr2=0x7f04880affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048f200640 1 -- 172.21.15.39:0/844064413 shutdown_connections 2024-01-24T17:46:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.717+0000 7f048f200640 1 -- 172.21.15.39:0/844064413 wait complete. 2024-01-24T17:46:23.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.719+0000 7f048f200640 1 Processor -- start 2024-01-24T17:46:23.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.719+0000 7f048f200640 1 -- start start 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.720+0000 7f048f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04880cbf90 0x7f048815a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.720+0000 7f048f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04880ce350 0x7f04881584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.720+0000 7f048f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f048815d450 0x7f04881f5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.720+0000 7f048f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04880c5790 con 0x7f04880cbf90 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.720+0000 7f048f200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04880aef90 con 0x7f04880ce350 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.720+0000 7f048f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04880af270 con 0x7f048815d450 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04880cbf90 0x7f048815a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04880cbf90 0x7f048815a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46950/0 (socket says 172.21.15.39:46950) 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048de00640 1 -- 172.21.15.39:0/114230953 learned_addr learned my addr 172.21.15.39:0/114230953 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048d400640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04880ce350 0x7f04881584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048e800640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f048815d450 0x7f04881f5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048de00640 1 -- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f048815d450 msgr2=0x7f04881f5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048de00640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f048815d450 0x7f04881f5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048de00640 1 -- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04880ce350 msgr2=0x7f04881584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:23.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048de00640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04880ce350 0x7f04881584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048de00640 1 -- 172.21.15.39:0/114230953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04700094d0 con 0x7f04880cbf90 2024-01-24T17:46:23.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.721+0000 7f048d400640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04880ce350 0x7f04881584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:23.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.722+0000 7f048de00640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04880cbf90 0x7f048815a440 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f047800e870 tx=0x7f047800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:23.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.722+0000 7f0486000640 1 -- 172.21.15.39:0/114230953 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04780094f0 con 0x7f04880cbf90 2024-01-24T17:46:23.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.722+0000 7f0486000640 1 -- 172.21.15.39:0/114230953 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047800c940 con 0x7f04880cbf90 2024-01-24T17:46:23.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.722+0000 7f0486000640 1 -- 172.21.15.39:0/114230953 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047800ccf0 con 0x7f04880cbf90 2024-01-24T17:46:23.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.722+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04880af550 con 0x7f04880cbf90 2024-01-24T17:46:23.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.722+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0488155d40 con 0x7f04880cbf90 2024-01-24T17:46:23.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.723+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04880738c0 con 0x7f04880cbf90 2024-01-24T17:46:23.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.724+0000 7f0486000640 1 -- 172.21.15.39:0/114230953 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0478013470 con 0x7f04880cbf90 2024-01-24T17:46:23.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.725+0000 7f0486000640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046006a430 0x7f046006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:23.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.726+0000 7f0486000640 1 -- 172.21.15.39:0/114230953 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f04780887d0 con 0x7f04880cbf90 2024-01-24T17:46:23.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.726+0000 7f048d400640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046006a430 0x7f046006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:23.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.727+0000 7f048d400640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046006a430 0x7f046006c8f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0470036000 tx=0x7f047003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:23.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.730+0000 7f0486000640 1 -- 172.21.15.39:0/114230953 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0478054030 con 0x7f04880cbf90 2024-01-24T17:46:23.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.928+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f04880c58d0 con 0x7f04880cbf90 2024-01-24T17:46:23.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.933+0000 7f0486000640 1 -- 172.21.15.39:0/114230953 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0478021020 con 0x7f04880cbf90 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046006a430 msgr2=0x7f046006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046006a430 0x7f046006c8f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0470036000 tx=0x7f047003c040 comp rx=0 tx=0).stop 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04880cbf90 msgr2=0x7f048815a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04880cbf90 0x7f048815a440 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f047800e870 tx=0x7f047800ed30 comp rx=0 tx=0).stop 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 shutdown_connections 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f048815d450 0x7f04881f5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04880ce350 0x7f04881584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046006a430 0x7f046006c8f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 --2- 172.21.15.39:0/114230953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04880cbf90 0x7f048815a440 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:23.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 >> 172.21.15.39:0/114230953 conn(0x7f048806a200 msgr2=0x7f04880cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:23.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 shutdown_connections 2024-01-24T17:46:23.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:23.936+0000 7f048f200640 1 -- 172.21.15.39:0/114230953 wait complete. 2024-01-24T17:46:23.958 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:46:23.959 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:46:24.072 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:46:24.189 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:46:24.190 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:46:24.190 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:24.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:46:24.252 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:24.252 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:24.309 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:46:24.309 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:46:24.309 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:24.369 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:46:24.369 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:24.369 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:24.426 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:46:24.426 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-24T17:46:24.426 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:46:24.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.652+0000 7f0f5b200640 1 Processor -- start 2024-01-24T17:46:24.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.652+0000 7f0f5b200640 1 -- start start 2024-01-24T17:46:24.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.653+0000 7f0f5b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c070540 0x7f0f5c15ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.653+0000 7f0f5b200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0f5c15e910 0x7f0f5c15eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.653+0000 7f0f5b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c1551b0 con 0x7f0f5c0b3e00 2024-01-24T17:46:24.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.653+0000 7f0f5b200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c15e410 con 0x7f0f5c15e910 2024-01-24T17:46:24.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.653+0000 7f0f5b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c15b1b0 con 0x7f0f5c070540 2024-01-24T17:46:24.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.654+0000 7f0f51200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0f5c0b3e00 0x7f0f5c0c8a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50616/0 (socket says 172.21.15.39:50616) 2024-01-24T17:46:24.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.654+0000 7f0f51200640 1 -- 172.21.15.39:0/2460338701 learned_addr learned my addr 172.21.15.39:0/2460338701 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:24.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.654+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855428446 0 0) 0x7f0f5c1551b0 con 0x7f0f5c0b3e00 2024-01-24T17:46:24.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.654+0000 7f0f5a800640 1 --2- 172.21.15.39:0/2460338701 >> v2:172.21.15.100:3300/0 conn(0x7f0f5c15e910 0x7f0f5c15eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.654+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3c003520 con 0x7f0f5c0b3e00 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.654+0000 7f0f59e00640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c070540 0x7f0f5c15ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1210252327 0 0) 0x7f0f3c003520 con 0x7f0f5c0b3e00 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c070540 msgr2=0x7f0f5c15ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c070540 0x7f0f5c15ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 >> v2:172.21.15.100:3300/0 conn(0x7f0f5c15e910 msgr2=0x7f0f5c15eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 --2- 172.21.15.39:0/2460338701 >> v2:172.21.15.100:3300/0 conn(0x7f0f5c15e910 0x7f0f5c15eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c15b480 con 0x7f0f5c0b3e00 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f0f44001670 con 0x7f0f5c0b3e00 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f5a800640 1 --2- 172.21.15.39:0/2460338701 >> v2:172.21.15.100:3300/0 conn(0x7f0f5c15e910 0x7f0f5c15eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 >> v1:172.21.15.39:6789/0 conn(0x7f0f5c0b3e00 legacy=0x7f0f5c0c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f3c0047c0 0x7f0f3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f3c005180 0x7f0f3c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c15e410 con 0x7f0f3c003970 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c15b1b0 con 0x7f0f3c0047c0 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c00bfc0 con 0x7f0f3c005180 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59e00640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c070540 0x7f0f5c15ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f59e00640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.655+0000 7f0f51200640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f3c0047c0 0x7f0f3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.656+0000 7f0f5a800640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f3c005180 0x7f0f3c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.656+0000 7f0f59e00640 1 -- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f3c005180 msgr2=0x7f0f3c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.656+0000 7f0f59e00640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f3c005180 0x7f0f3c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.656+0000 7f0f59e00640 1 -- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f3c0047c0 msgr2=0x7f0f3c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.656+0000 7f0f59e00640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f3c0047c0 0x7f0f3c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.656+0000 7f0f59e00640 1 -- 172.21.15.39:0/2460338701 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c003520 con 0x7f0f3c003970 2024-01-24T17:46:24.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.657+0000 7f0f59e00640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f3c003d90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009f50 tx=0x7f0f3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0ecd36c34ce492 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.657+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c02cca0 con 0x7f0f3c003970 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.657+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4c02ce00 con 0x7f0f3c003970 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.657+0000 7f0f59400640 1 -- 172.21.15.39:0/2460338701 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c003bb0 con 0x7f0f3c003970 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.658+0000 7f0f5b200640 1 -- 172.21.15.39:0/2460338701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 msgr2=0x7f0f3c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.658+0000 7f0f5b200640 1 --2- 172.21.15.39:0/2460338701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f3c003d90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009f50 tx=0x7f0f3c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.658+0000 7f0f5a800640 1 -- 172.21.15.39:0/2460338701 reap_dead start 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.658+0000 7f0f5b200640 1 -- 172.21.15.39:0/2460338701 shutdown_connections 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.658+0000 7f0f5b200640 1 -- 172.21.15.39:0/2460338701 >> 172.21.15.39:0/2460338701 conn(0x7f0f5c06a340 msgr2=0x7f0f5c0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.658+0000 7f0f5b200640 1 -- 172.21.15.39:0/2460338701 shutdown_connections 2024-01-24T17:46:24.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.658+0000 7f0f5b200640 1 -- 172.21.15.39:0/2460338701 wait complete. 2024-01-24T17:46:24.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.660+0000 7f0f5b200640 1 Processor -- start 2024-01-24T17:46:24.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.660+0000 7f0f5b200640 1 -- start start 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.661+0000 7f0f5b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f5c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.661+0000 7f0f5b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5c1ec8d0 0x7f0f5c1f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.661+0000 7f0f5b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c0c3d00 0x7f0f5c0c4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.661+0000 7f0f5b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1ecd50 con 0x7f0f3c003970 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.661+0000 7f0f5b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1f3810 con 0x7f0f5c1ec8d0 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.661+0000 7f0f5b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1e4250 con 0x7f0f5c0c3d00 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.661+0000 7f0f59e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f5c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f59e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f5c0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46964/0 (socket says 172.21.15.39:46964) 2024-01-24T17:46:24.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f59e00640 1 -- 172.21.15.39:0/1168090857 learned_addr learned my addr 172.21.15.39:0/1168090857 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f51200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5c1ec8d0 0x7f0f5c1f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f5a800640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c0c3d00 0x7f0f5c0c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f59e00640 1 -- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c0c3d00 msgr2=0x7f0f5c0c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f59e00640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c0c3d00 0x7f0f5c0c4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f59e00640 1 -- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5c1ec8d0 msgr2=0x7f0f5c1f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f59e00640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5c1ec8d0 0x7f0f5c1f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f59e00640 1 -- 172.21.15.39:0/1168090857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c1e4530 con 0x7f0f3c003970 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.662+0000 7f0f59e00640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f5c0cdba0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c007420 tx=0x7f0f4c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.663+0000 7f0f52a00640 1 -- 172.21.15.39:0/1168090857 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c03c070 con 0x7f0f3c003970 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.663+0000 7f0f52a00640 1 -- 172.21.15.39:0/1168090857 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4c0042c0 con 0x7f0f3c003970 2024-01-24T17:46:24.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.663+0000 7f0f52a00640 1 -- 172.21.15.39:0/1168090857 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c004650 con 0x7f0f3c003970 2024-01-24T17:46:24.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.663+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c1e4810 con 0x7f0f3c003970 2024-01-24T17:46:24.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.663+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c0c37e0 con 0x7f0f3c003970 2024-01-24T17:46:24.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.664+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c04dff0 con 0x7f0f3c003970 2024-01-24T17:46:24.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.665+0000 7f0f52a00640 1 -- 172.21.15.39:0/1168090857 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0f4c037460 con 0x7f0f3c003970 2024-01-24T17:46:24.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.666+0000 7f0f52a00640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3406a430 0x7f0f3406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:24.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.666+0000 7f0f52a00640 1 -- 172.21.15.39:0/1168090857 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0f4c0a9f90 con 0x7f0f3c003970 2024-01-24T17:46:24.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.666+0000 7f0f51200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3406a430 0x7f0f3406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:24.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.668+0000 7f0f51200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3406a430 0x7f0f3406c8f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0f44001470 tx=0x7f0f44009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:24.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.671+0000 7f0f52a00640 1 -- 172.21.15.39:0/1168090857 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0f4c0dccb0 con 0x7f0f3c003970 2024-01-24T17:46:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.906+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f5c0cd8f0 con 0x7f0f3c003970 2024-01-24T17:46:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.907+0000 7f0f52a00640 1 -- 172.21.15.39:0/1168090857 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f0f4c045020 con 0x7f0f3c003970 2024-01-24T17:46:24.909 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:24.909 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3045325513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3045325513},{"type":"v1","addr":"172.21.15.100:6836","nonce":3045325513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5126,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/858868211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":858868211},{"type":"v1","addr":"172.21.15.100:6837","nonce":858868211}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6252,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3121826441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3121826441},{"type":"v1","addr":"172.21.15.39:6833","nonce":3121826441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6741,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/895363672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":895363672},{"type":"v1","addr":"172.21.15.39:6835","nonce":895363672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[]} 2024-01-24T17:46:24.909 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 36 2024-01-24T17:46:24.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3406a430 msgr2=0x7f0f3406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:24.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3406a430 0x7f0f3406c8f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0f44001470 tx=0x7f0f44009040 comp rx=0 tx=0).stop 2024-01-24T17:46:24.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 msgr2=0x7f0f5c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:24.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f5c0cdba0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c007420 tx=0x7f0f4c004ab0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 shutdown_connections 2024-01-24T17:46:24.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f5c0c3d00 0x7f0f5c0c4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f5c1ec8d0 0x7f0f5c1f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f3406a430 0x7f0f3406c8f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 --2- 172.21.15.39:0/1168090857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f3c003970 0x7f0f5c0cdba0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:24.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.909+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 >> 172.21.15.39:0/1168090857 conn(0x7f0f5c06a340 msgr2=0x7f0f5c0ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:24.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.910+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 shutdown_connections 2024-01-24T17:46:24.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:24.910+0000 7f0f5b200640 1 -- 172.21.15.39:0/1168090857 wait complete. 2024-01-24T17:46:24.932 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-24T17:46:24.932 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-24T17:46:24.932 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-24T17:46:24.932 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:46:24.932 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-24T17:46:24.932 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-24T17:46:24.933 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-24T17:46:24.933 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:46:24.933 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-24T17:46:24.933 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-24T17:46:24.933 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-24T17:46:24.933 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:46:24.933 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-24T17:46:24.933 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-24T17:46:24.934 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-24T17:46:24.934 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:46:24.950 INFO:tasks.ceph.mds.c:Stopped 2024-01-24T17:46:24.950 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-24T17:46:24.954 INFO:tasks.ceph.mds.a:Stopped 2024-01-24T17:46:24.955 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-24T17:46:24.958 INFO:tasks.ceph.mds.b:Stopped 2024-01-24T17:46:24.958 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-24T17:46:24.960 INFO:tasks.ceph.mds.d:Stopped 2024-01-24T17:46:24.960 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-24T17:46:25.002 INFO:tasks.ceph.mds.c:Started 2024-01-24T17:46:25.003 DEBUG:teuthology.parallel:result is None 2024-01-24T17:46:25.005 INFO:tasks.ceph.mds.b:Started 2024-01-24T17:46:25.006 DEBUG:teuthology.parallel:result is None 2024-01-24T17:46:25.008 INFO:tasks.ceph.mds.a:Started 2024-01-24T17:46:25.009 DEBUG:teuthology.parallel:result is None 2024-01-24T17:46:25.012 INFO:tasks.ceph.mds.d:Started 2024-01-24T17:46:25.012 DEBUG:teuthology.parallel:result is None 2024-01-24T17:46:25.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-24T17:46:25.164 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:46:25.167 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:46:25.169 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:46:25.169 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:46:25.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.205+0000 7fbb58a00640 1 Processor -- start 2024-01-24T17:46:25.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.205+0000 7fbb58a00640 1 -- start start 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb58a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb58a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbb500d15a0 0x7fbb50166b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb58a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb500b8820 con 0x7fbb500ce510 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb58a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb500b5790 con 0x7fbb500d15a0 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb58a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb500b5a50 con 0x7fbb5015cbe0 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb56a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbb500d15a0 0x7fbb50166b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb55600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fbb500ce510 0x7fbb500d0e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50644/0 (socket says 172.21.15.39:50644) 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb55600640 1 -- 172.21.15.39:0/3839788528 learned_addr learned my addr 172.21.15.39:0/3839788528 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.206+0000 7fbb56000640 1 --2- 172.21.15.39:0/3839788528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.207+0000 7fbb54c00640 1 -- 172.21.15.39:0/3839788528 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536824018 0 0) 0x7fbb500b8820 con 0x7fbb500ce510 2024-01-24T17:46:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.207+0000 7fbb54c00640 1 -- 172.21.15.39:0/3839788528 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb30003520 con 0x7fbb500ce510 2024-01-24T17:46:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.207+0000 7fbb56a00640 1 -- 172.21.15.39:0/3839788528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 msgr2=0x7fbb500cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.207+0000 7fbb56a00640 1 --2- 172.21.15.39:0/3839788528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.207+0000 7fbb56a00640 1 -- 172.21.15.39:0/3839788528 >> v1:172.21.15.39:6789/0 conn(0x7fbb500ce510 legacy=0x7fbb500d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.207+0000 7fbb56a00640 1 -- 172.21.15.39:0/3839788528 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb500b5d30 con 0x7fbb500d15a0 2024-01-24T17:46:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.207+0000 7fbb56000640 1 --2- 172.21.15.39:0/3839788528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.208+0000 7fbb56a00640 1 --2- 172.21.15.39:0/3839788528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500d15a0 0x7fbb50166b50 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009d90 tx=0x7fbb4802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d448a18907ab36d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.208+0000 7fbb54c00640 1 -- 172.21.15.39:0/3839788528 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4802cc50 con 0x7fbb500d15a0 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.208+0000 7fbb54c00640 1 -- 172.21.15.39:0/3839788528 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4802cdb0 con 0x7fbb500d15a0 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.208+0000 7fbb54c00640 1 -- 172.21.15.39:0/3839788528 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48034660 con 0x7fbb500d15a0 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.209+0000 7fbb58a00640 1 -- 172.21.15.39:0/3839788528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500d15a0 msgr2=0x7fbb50166b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.209+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3839788528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500d15a0 0x7fbb50166b50 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009d90 tx=0x7fbb4802bea0 comp rx=0 tx=0).stop 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.209+0000 7fbb58a00640 1 -- 172.21.15.39:0/3839788528 shutdown_connections 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.209+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3839788528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500d15a0 0x7fbb50166b50 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.209+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3839788528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.209+0000 7fbb58a00640 1 -- 172.21.15.39:0/3839788528 >> 172.21.15.39:0/3839788528 conn(0x7fbb5009bc80 msgr2=0x7fbb500d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.209+0000 7fbb58a00640 1 -- 172.21.15.39:0/3839788528 shutdown_connections 2024-01-24T17:46:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.209+0000 7fbb58a00640 1 -- 172.21.15.39:0/3839788528 wait complete. 2024-01-24T17:46:25.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.211+0000 7fbb58a00640 1 Processor -- start 2024-01-24T17:46:25.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.211+0000 7fbb58a00640 1 -- start start 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb58a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500ce510 0x7fbb5014a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb58a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbb500d15a0 0x7fbb5014afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb58a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb58a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb50150ed0 con 0x7fbb500ce510 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb58a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5014b8c0 con 0x7fbb500d15a0 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb58a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5014bba0 con 0x7fbb5015cbe0 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb56000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500ce510 0x7fbb5014a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb56000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500ce510 0x7fbb5014a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47004/0 (socket says 172.21.15.39:47004) 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb56000640 1 -- 172.21.15.39:0/3373686318 learned_addr learned my addr 172.21.15.39:0/3373686318 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb55600640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbb500d15a0 0x7fbb5014afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.212+0000 7fbb56a00640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb56000640 1 -- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 msgr2=0x7fbb500b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb56000640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500b72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb56000640 1 -- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbb500d15a0 msgr2=0x7fbb5014afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb56000640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbb500d15a0 0x7fbb5014afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb56000640 1 -- 172.21.15.39:0/3373686318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb400093a0 con 0x7fbb500ce510 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb56000640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500ce510 0x7fbb5014a6e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000e9b0 tx=0x7fbb4000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb4e000640 1 -- 172.21.15.39:0/3373686318 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40005990 con 0x7fbb500ce510 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb4e000640 1 -- 172.21.15.39:0/3373686318 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb40005af0 con 0x7fbb500ce510 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.213+0000 7fbb4e000640 1 -- 172.21.15.39:0/3373686318 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40005e80 con 0x7fbb500ce510 2024-01-24T17:46:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.214+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb48009a70 con 0x7fbb500ce510 2024-01-24T17:46:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.214+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5014c160 con 0x7fbb500ce510 2024-01-24T17:46:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.215+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5004dff0 con 0x7fbb500ce510 2024-01-24T17:46:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.216+0000 7fbb4e000640 1 -- 172.21.15.39:0/3373686318 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbb4000c9a0 con 0x7fbb500ce510 2024-01-24T17:46:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.216+0000 7fbb4e000640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbb2806a430 0x7fbb2806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.217+0000 7fbb4e000640 1 -- 172.21.15.39:0/3373686318 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbb40088a50 con 0x7fbb500ce510 2024-01-24T17:46:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.217+0000 7fbb55600640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbb2806a430 0x7fbb2806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:25.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.218+0000 7fbb55600640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbb2806a430 0x7fbb2806c8f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbb501f56e0 tx=0x7fbb38007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:25.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.221+0000 7fbb4e000640 1 -- 172.21.15.39:0/3373686318 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbb40054350 con 0x7fbb500ce510 2024-01-24T17:46:25.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:25.427+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbb500d0c70 con 0x7fbb500ce510 2024-01-24T17:46:26.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.289+0000 7fbb4e000640 1 -- 172.21.15.39:0/3373686318 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v72) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbb40020020 con 0x7fbb500ce510 2024-01-24T17:46:26.291 INFO:teuthology.orchestra.run.smithi039.stderr: removed all blocklist entries 2024-01-24T17:46:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.292+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbb2806a430 msgr2=0x7fbb2806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.292+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbb2806a430 0x7fbb2806c8f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbb501f56e0 tx=0x7fbb38007d30 comp rx=0 tx=0).stop 2024-01-24T17:46:26.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.292+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500ce510 msgr2=0x7fbb5014a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.292+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500ce510 0x7fbb5014a6e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000e9b0 tx=0x7fbb4000ee70 comp rx=0 tx=0).stop 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.293+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 shutdown_connections 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.293+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbb5015cbe0 0x7fbb500b72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.293+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbb500d15a0 0x7fbb5014afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.293+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbb2806a430 0x7fbb2806c8f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.293+0000 7fbb58a00640 1 --2- 172.21.15.39:0/3373686318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500ce510 0x7fbb5014a6e0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.293+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 >> 172.21.15.39:0/3373686318 conn(0x7fbb5009bc80 msgr2=0x7fbb500cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.293+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 shutdown_connections 2024-01-24T17:46:26.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.293+0000 7fbb58a00640 1 -- 172.21.15.39:0/3373686318 wait complete. 2024-01-24T17:46:26.314 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-24T17:46:26.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.533+0000 7f4b5d000640 1 Processor -- start 2024-01-24T17:46:26.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.533+0000 7f4b5d000640 1 -- start start 2024-01-24T17:46:26.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b5d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4b5806bd40 0x7f4b580ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:26.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b5d000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b580cac10 con 0x7f4b5815a9a0 2024-01-24T17:46:26.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b5d000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5815b040 con 0x7f4b580cadc0 2024-01-24T17:46:26.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b5d000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b5806c1d0 con 0x7f4b5806bd40 2024-01-24T17:46:26.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b56a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4b5815a9a0 0x7f4b5815b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50648/0 (socket says 172.21.15.39:50648) 2024-01-24T17:46:26.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b56a00640 1 -- 172.21.15.39:0/3707639131 learned_addr learned my addr 172.21.15.39:0/3707639131 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:26.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b57e00640 1 --2- 172.21.15.39:0/3707639131 >> v2:172.21.15.39:3300/0 conn(0x7f4b5806bd40 0x7f4b580ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:26.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b56000640 1 -- 172.21.15.39:0/3707639131 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273051823 0 0) 0x7f4b580cac10 con 0x7f4b5815a9a0 2024-01-24T17:46:26.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.535+0000 7f4b56000640 1 -- 172.21.15.39:0/3707639131 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b38003520 con 0x7f4b5815a9a0 2024-01-24T17:46:26.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.536+0000 7f4b57e00640 1 -- 172.21.15.39:0/3707639131 >> v1:172.21.15.100:6789/0 conn(0x7f4b580cadc0 legacy=0x7f4b580cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:26.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.536+0000 7f4b57e00640 1 -- 172.21.15.39:0/3707639131 >> v1:172.21.15.39:6789/0 conn(0x7f4b5815a9a0 legacy=0x7f4b5815b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:26.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.536+0000 7f4b57e00640 1 -- 172.21.15.39:0/3707639131 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5815bbe0 con 0x7f4b5806bd40 2024-01-24T17:46:26.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.537+0000 7f4b57e00640 1 --2- 172.21.15.39:0/3707639131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b5806bd40 0x7f4b580ca6d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009d90 tx=0x7f4b5002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eea2eaaa5951aec server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:26.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.537+0000 7f4b56000640 1 -- 172.21.15.39:0/3707639131 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5002cbd0 con 0x7f4b5806bd40 2024-01-24T17:46:26.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.537+0000 7f4b56000640 1 -- 172.21.15.39:0/3707639131 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5002cd30 con 0x7f4b5806bd40 2024-01-24T17:46:26.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.537+0000 7f4b56000640 1 -- 172.21.15.39:0/3707639131 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5002cea0 con 0x7f4b5806bd40 2024-01-24T17:46:26.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.538+0000 7f4b5d000640 1 -- 172.21.15.39:0/3707639131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b5806bd40 msgr2=0x7f4b580ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:26.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.538+0000 7f4b5d000640 1 --2- 172.21.15.39:0/3707639131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b5806bd40 0x7f4b580ca6d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009d90 tx=0x7f4b5002be20 comp rx=0 tx=0).stop 2024-01-24T17:46:26.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.538+0000 7f4b5d000640 1 -- 172.21.15.39:0/3707639131 shutdown_connections 2024-01-24T17:46:26.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.538+0000 7f4b5d000640 1 --2- 172.21.15.39:0/3707639131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b5806bd40 0x7f4b580ca6d0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.538+0000 7f4b5d000640 1 -- 172.21.15.39:0/3707639131 >> 172.21.15.39:0/3707639131 conn(0x7f4b5806a200 msgr2=0x7f4b580aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:26.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.538+0000 7f4b5d000640 1 -- 172.21.15.39:0/3707639131 shutdown_connections 2024-01-24T17:46:26.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.538+0000 7f4b5d000640 1 -- 172.21.15.39:0/3707639131 wait complete. 2024-01-24T17:46:26.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.540+0000 7f4b5d000640 1 Processor -- start 2024-01-24T17:46:26.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.540+0000 7f4b5d000640 1 -- start start 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b5d000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b5806bd40 0x7f4b580c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b5d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b580cadc0 0x7f4b58155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b5d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b5815a9a0 0x7f4b580c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b5d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b580c26b0 con 0x7f4b580cadc0 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b5d000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b58155d70 con 0x7f4b5806bd40 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b5d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b580b3fe0 con 0x7f4b5815a9a0 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b56a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b580cadc0 0x7f4b58155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b56a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b580cadc0 0x7f4b58155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47018/0 (socket says 172.21.15.39:47018) 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b56a00640 1 -- 172.21.15.39:0/300103793 learned_addr learned my addr 172.21.15.39:0/300103793 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b56a00640 1 -- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b5815a9a0 msgr2=0x7f4b580c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b56a00640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b5815a9a0 0x7f4b580c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b56a00640 1 -- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b5806bd40 msgr2=0x7f4b580c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.541+0000 7f4b57400640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b5806bd40 0x7f4b580c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.542+0000 7f4b56a00640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b5806bd40 0x7f4b580c2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.542+0000 7f4b56a00640 1 -- 172.21.15.39:0/300103793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b50009a70 con 0x7f4b580cadc0 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.542+0000 7f4b56a00640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b580cadc0 0x7f4b58155830 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000c500 tx=0x7f4b4000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.542+0000 7f4b4f400640 1 -- 172.21.15.39:0/300103793 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40005ce0 con 0x7f4b580cadc0 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.542+0000 7f4b4f400640 1 -- 172.21.15.39:0/300103793 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b40005e40 con 0x7f4b580cadc0 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.542+0000 7f4b4f400640 1 -- 172.21.15.39:0/300103793 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40007330 con 0x7f4b580cadc0 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.542+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b580c8440 con 0x7f4b580cadc0 2024-01-24T17:46:26.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.542+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b580c2340 con 0x7f4b580cadc0 2024-01-24T17:46:26.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.543+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b580c2820 con 0x7f4b580cadc0 2024-01-24T17:46:26.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.544+0000 7f4b4f400640 1 -- 172.21.15.39:0/300103793 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4b40007550 con 0x7f4b580cadc0 2024-01-24T17:46:26.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.545+0000 7f4b4f400640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b3006a430 0x7f4b3006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:26.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.545+0000 7f4b4f400640 1 -- 172.21.15.39:0/300103793 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4b40087c80 con 0x7f4b580cadc0 2024-01-24T17:46:26.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.546+0000 7f4b57400640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b3006a430 0x7f4b3006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:26.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.547+0000 7f4b57400640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b3006a430 0x7f4b3006c8f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4b580b43f0 tx=0x7f4b48009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:26.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.549+0000 7f4b4f400640 1 -- 172.21.15.39:0/300103793 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4b40053670 con 0x7f4b580cadc0 2024-01-24T17:46:26.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.803+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4b5804dff0 con 0x7f4b580cadc0 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.804+0000 7f4b4f400640 1 -- 172.21.15.39:0/300103793 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5580 (secure 0 0 0) 0x7f4b40017090 con 0x7f4b580cadc0 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_dump": [ 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.a", 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlZZEtLBAAUxf4chL3tgy3zHlZoraXlg==", 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.806 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.b", 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Fln4utEhAA52ngJFqrdfa9bx56mVhRVA==", 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.c", 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flo8iGARAAf+29RXWr4RHkILIz7em3mA==", 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.d", 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flh+08IRAARhZxy//j8c9b3Hg9uV9ajg==", 2024-01-24T17:46:26.808 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.0", 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCQS7FlOj/pLBAAzi89tAacOfDKdlU0BUJvvA==", 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.1", 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCUS7FlbRAKDRAAAuAy47MDTtvmh8qNA3ZLTg==", 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.2", 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCXS7Fl7QmRGRAAVECH7oYTdvDPSHBy9WmHwA==", 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:46:26.810 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.3", 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCaS7FlCQHIMxAA9OchsMzsKFxs110SeJ3Atw==", 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.4", 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCiS7FlDsBKBRAAXF2HafraT5nRbB5tBRUgWQ==", 2024-01-24T17:46:26.811 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.5", 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQClS7FlfrXBHhAAy8GP3TLX/iok3OYPGTP5Mw==", 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.812 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.6", 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCoS7Fla11hNhAA/3oMP7cFtPSTdJl7eSnPTA==", 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.7", 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCsS7FlKFMtEhAAPmaFzr5XHVQaKZYkJiIarA==", 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:46:26.813 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.0", 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlTi/3LhAAc8FwXtLH2XjlFMO7wDMcSQ==", 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow r", 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.1", 2024-01-24T17:46:26.814 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlH5s7ChAAlFfFDziTgfv4ibWO08N+4Q==", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.2", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlIISlOhAAoVxxez7UjXDxyHQNNPnn2g==", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:46:26.815 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.3", 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlO/U/FBAA2pI9aGSS2cZ3AMnn+W1fQA==", 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.admin", 2024-01-24T17:46:26.816 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCGS7FlxKo7HhAAlu037DIc1WmWkz81BG3j7Q==", 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.badguy", 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQDXTLFlLXBAAxAAmFCWwaJ4A7NtzfQcEI/TdA==", 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow rw path=/foo/bar", 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow r", 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rw" 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.x", 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlRbF6HxAAoFjlyWb5oqIkC3L1iPRmHw==", 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.y", 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7Fl512kFRAAdCpU+QjRmGJmMD5uTdFmHQ==", 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:46:26.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:46:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:26.819 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:46:26.819 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:46:26.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b3006a430 msgr2=0x7f4b3006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b3006a430 0x7f4b3006c8f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4b580b43f0 tx=0x7f4b48009040 comp rx=0 tx=0).stop 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b580cadc0 msgr2=0x7f4b58155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b580cadc0 0x7f4b58155830 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000c500 tx=0x7f4b4000c9c0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 shutdown_connections 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4b5815a9a0 0x7f4b580c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4b3006a430 0x7f4b3006c8f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b580cadc0 0x7f4b58155830 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 --2- 172.21.15.39:0/300103793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b5806bd40 0x7f4b580c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 >> 172.21.15.39:0/300103793 conn(0x7f4b5806a200 msgr2=0x7f4b5815d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 shutdown_connections 2024-01-24T17:46:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:26.808+0000 7f4b5d000640 1 -- 172.21.15.39:0/300103793 wait complete. 2024-01-24T17:46:26.828 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.badguy 2024-01-24T17:46:27.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.048+0000 7f4cc3a00640 1 Processor -- start 2024-01-24T17:46:27.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.048+0000 7f4cc3a00640 1 -- start start 2024-01-24T17:46:27.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.050+0000 7f4cc3a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4cbc15cf10 0x7f4cbc070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.050+0000 7f4cc3a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc0ccf90 con 0x7f4cbc0b0720 2024-01-24T17:46:27.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.050+0000 7f4cc3a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc0c9bd0 con 0x7f4cbc070980 2024-01-24T17:46:27.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.050+0000 7f4cc3a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0c9eb0 con 0x7f4cbc15cf10 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cc1a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f4cbc0b0720 0x7f4cbc0cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50656/0 (socket says 172.21.15.39:50656) 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cc1a00640 1 -- 172.21.15.39:0/53268696 learned_addr learned my addr 172.21.15.39:0/53268696 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cc1000640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4cbc15cf10 0x7f4cbc070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384541856 0 0) 0x7f4cbc0ccf90 con 0x7f4cbc0b0720 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c9c003520 con 0x7f4cbc0b0720 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2743050740 0 0) 0x7f4c9c003520 con 0x7f4cbc0b0720 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4cbc15cf10 msgr2=0x7f4cbc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4cbc15cf10 0x7f4cbc070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 >> v1:172.21.15.100:6789/0 conn(0x7f4cbc070980 legacy=0x7f4cbc0b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc0cc940 con 0x7f4cbc0b0720 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f4cb40019f0 con 0x7f4cbc0b0720 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.051+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 >> v1:172.21.15.39:6789/0 conn(0x7f4cbc0b0720 legacy=0x7f4cbc0cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cbbe00640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9c0039a0 0x7f4c9c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cbbe00640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c9c0047c0 0x7f4c9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cbbe00640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4cbc15cf10 0x7f4c9c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0c9eb0 con 0x7f4c9c0039a0 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c00bb10 con 0x7f4c9c0047c0 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c00bcd0 con 0x7f4cbc15cf10 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cc1000640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9c0039a0 0x7f4c9c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cc1000640 1 -- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4cbc15cf10 msgr2=0x7f4c9c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cc1000640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4cbc15cf10 0x7f4c9c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cc1000640 1 -- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c9c0047c0 msgr2=0x7f4c9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cb3e00640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c9c0047c0 0x7f4c9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cc1000640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c9c0047c0 0x7f4c9c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.052+0000 7f4cc1000640 1 -- 172.21.15.39:0/53268696 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c003520 con 0x7f4c9c0039a0 2024-01-24T17:46:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.054+0000 7f4cc1000640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9c0039a0 0x7f4c9c003dc0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009ac0 tx=0x7f4c9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f390d784062aa421 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.054+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac02c710 con 0x7f4c9c0039a0 2024-01-24T17:46:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.054+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac02c870 con 0x7f4c9c0039a0 2024-01-24T17:46:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.054+0000 7f4cbbe00640 1 -- 172.21.15.39:0/53268696 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac02cc00 con 0x7f4c9c0039a0 2024-01-24T17:46:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.055+0000 7f4cc3a00640 1 -- 172.21.15.39:0/53268696 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9c0039a0 msgr2=0x7f4c9c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.055+0000 7f4cc3a00640 1 --2- 172.21.15.39:0/53268696 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c9c0039a0 0x7f4c9c003dc0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009ac0 tx=0x7f4c9c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.055+0000 7f4cc1a00640 1 -- 172.21.15.39:0/53268696 reap_dead start 2024-01-24T17:46:27.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.055+0000 7f4cc3a00640 1 -- 172.21.15.39:0/53268696 shutdown_connections 2024-01-24T17:46:27.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.055+0000 7f4cc3a00640 1 -- 172.21.15.39:0/53268696 >> 172.21.15.39:0/53268696 conn(0x7f4cbc06bf20 msgr2=0x7f4cbc06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:27.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.055+0000 7f4cc3a00640 1 -- 172.21.15.39:0/53268696 shutdown_connections 2024-01-24T17:46:27.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.055+0000 7f4cc3a00640 1 -- 172.21.15.39:0/53268696 wait complete. 2024-01-24T17:46:27.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.057+0000 7f4cc3a00640 1 Processor -- start 2024-01-24T17:46:27.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.057+0000 7f4cc3a00640 1 -- start start 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc3a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c9c0039a0 0x7f4cbc154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc3a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4cbc0cdf50 0x7f4cbc0ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc3a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4cbc0b1550 0x7f4cbc0b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc3a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0b2cb0 con 0x7f4cbc0b1550 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc3a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0ce910 con 0x7f4cbc0cdf50 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc3a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0b1f10 con 0x7f4c9c0039a0 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4cbc0b1550 0x7f4cbc0b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4cbc0b1550 0x7f4cbc0b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47026/0 (socket says 172.21.15.39:47026) 2024-01-24T17:46:27.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc1a00640 1 -- 172.21.15.39:0/818449172 learned_addr learned my addr 172.21.15.39:0/818449172 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:27.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cb3e00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4cbc0cdf50 0x7f4cbc0ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.058+0000 7f4cc1000640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c9c0039a0 0x7f4cbc154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cc1a00640 1 -- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c9c0039a0 msgr2=0x7f4cbc154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cc1a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c9c0039a0 0x7f4cbc154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cc1a00640 1 -- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4cbc0cdf50 msgr2=0x7f4cbc0ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cc1a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4cbc0cdf50 0x7f4cbc0ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cc1a00640 1 -- 172.21.15.39:0/818449172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc0c7550 con 0x7f4cbc0b1550 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cc1a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4cbc0b1550 0x7f4cbc0b19d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc06cdd0 tx=0x7f4cb400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cb9600640 1 -- 172.21.15.39:0/818449172 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400cec0 con 0x7f4cbc0b1550 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cb9600640 1 -- 172.21.15.39:0/818449172 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb400a5b0 con 0x7f4cbc0b1550 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cb9600640 1 -- 172.21.15.39:0/818449172 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400a960 con 0x7f4cbc0b1550 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc15d710 con 0x7f4cbc0b1550 2024-01-24T17:46:27.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.059+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc0b2410 con 0x7f4cbc0b1550 2024-01-24T17:46:27.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.060+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc15d960 con 0x7f4cbc0b1550 2024-01-24T17:46:27.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.061+0000 7f4cb9600640 1 -- 172.21.15.39:0/818449172 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4cb400aac0 con 0x7f4cbc0b1550 2024-01-24T17:46:27.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.062+0000 7f4cb9600640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c9406a430 0x7f4c9406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.062+0000 7f4cb9600640 1 -- 172.21.15.39:0/818449172 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4cb4087e00 con 0x7f4cbc0b1550 2024-01-24T17:46:27.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.063+0000 7f4cc1000640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c9406a430 0x7f4c9406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.064+0000 7f4cc1000640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c9406a430 0x7f4c9406c8f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009ac0 tx=0x7f4cac030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:27.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.067+0000 7f4cb9600640 1 -- 172.21.15.39:0/818449172 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4cbc15d960 con 0x7f4cbc0b1550 2024-01-24T17:46:27.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.333+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.badguy"} v 0) v1 -- 0x7f4cbc15d960 con 0x7f4cbc0b1550 2024-01-24T17:46:27.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.341+0000 7f4cb9600640 1 -- 172.21.15.39:0/818449172 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.badguy"}]=0 v15) v1 ==== 83+0+0 (secure 0 0 0) 0x7f4cbc15d960 con 0x7f4cbc0b1550 2024-01-24T17:46:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c9406a430 msgr2=0x7f4c9406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c9406a430 0x7f4c9406c8f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009ac0 tx=0x7f4cac030000 comp rx=0 tx=0).stop 2024-01-24T17:46:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4cbc0b1550 msgr2=0x7f4cbc0b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4cbc0b1550 0x7f4cbc0b19d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc06cdd0 tx=0x7f4cb400ccd0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 shutdown_connections 2024-01-24T17:46:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c9406a430 0x7f4c9406c8f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4cbc0b1550 0x7f4cbc0b19d0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4cbc0cdf50 0x7f4cbc0ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 --2- 172.21.15.39:0/818449172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c9c0039a0 0x7f4cbc154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 >> 172.21.15.39:0/818449172 conn(0x7f4cbc06bf20 msgr2=0x7f4cbc0c20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:27.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 shutdown_connections 2024-01-24T17:46:27.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.344+0000 7f4cc3a00640 1 -- 172.21.15.39:0/818449172 wait complete. 2024-01-24T17:46:27.364 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:27.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.583+0000 7f1d7a200640 1 Processor -- start 2024-01-24T17:46:27.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.583+0000 7f1d7a200640 1 -- start start 2024-01-24T17:46:27.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.585+0000 7f1d7a200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d74070540 0x7f1d7415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.585+0000 7f1d7a200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d741551b0 con 0x7f1d7415e910 2024-01-24T17:46:27.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.585+0000 7f1d7a200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7415e410 con 0x7f1d740b3e00 2024-01-24T17:46:27.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.585+0000 7f1d7a200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7415b1b0 con 0x7f1d74070540 2024-01-24T17:46:27.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.585+0000 7f1d79800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1d7415e910 0x7f1d740afe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50664/0 (socket says 172.21.15.39:50664) 2024-01-24T17:46:27.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.585+0000 7f1d79800640 1 -- 172.21.15.39:0/2399097650 learned_addr learned my addr 172.21.15.39:0/2399097650 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:27.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.586+0000 7f1d78e00640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d74070540 0x7f1d7415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.586+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745004946 0 0) 0x7f1d741551b0 con 0x7f1d7415e910 2024-01-24T17:46:27.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.586+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d54003520 con 0x7f1d7415e910 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.586+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095778792 0 0) 0x7f1d54003520 con 0x7f1d7415e910 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d74070540 msgr2=0x7f1d7415ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d74070540 0x7f1d7415ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 >> v1:172.21.15.100:6789/0 conn(0x7f1d740b3e00 legacy=0x7f1d740c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7415b480 con 0x7f1d7415e910 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1d6c0019f0 con 0x7f1d7415e910 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 >> v1:172.21.15.39:6789/0 conn(0x7f1d7415e910 legacy=0x7f1d740afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d540039a0 0x7f1d54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d74070540 0x7f1d540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d54004d00 0x7f1d54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7415b1b0 con 0x7f1d54004d00 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5400ba30 con 0x7f1d540039a0 2024-01-24T17:46:27.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5400bbf0 con 0x7f1d74070540 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d79800640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d54004d00 0x7f1d54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d78e00640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d540039a0 0x7f1d54003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d73e00640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d74070540 0x7f1d540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.587+0000 7f1d79800640 1 -- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d74070540 msgr2=0x7f1d540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.588+0000 7f1d79800640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d74070540 0x7f1d540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.588+0000 7f1d79800640 1 -- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d540039a0 msgr2=0x7f1d54003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.588+0000 7f1d79800640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d540039a0 0x7f1d54003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.588+0000 7f1d79800640 1 -- 172.21.15.39:0/2399097650 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d54003520 con 0x7f1d54004d00 2024-01-24T17:46:27.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.588+0000 7f1d78e00640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d540039a0 0x7f1d54003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:27.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.589+0000 7f1d79800640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d54004d00 0x7f1d54005180 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00a800 tx=0x7f1d540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02d8a495af0a9b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:27.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.589+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c0072d0 con 0x7f1d54004d00 2024-01-24T17:46:27.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.589+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c02f070 con 0x7f1d54004d00 2024-01-24T17:46:27.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.589+0000 7f1d73400640 1 -- 172.21.15.39:0/2399097650 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c0076b0 con 0x7f1d54004d00 2024-01-24T17:46:27.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.589+0000 7f1d7a200640 1 -- 172.21.15.39:0/2399097650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d54004d00 msgr2=0x7f1d54005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.589+0000 7f1d7a200640 1 --2- 172.21.15.39:0/2399097650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d54004d00 0x7f1d54005180 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00a800 tx=0x7f1d540057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.589+0000 7f1d79800640 1 -- 172.21.15.39:0/2399097650 reap_dead start 2024-01-24T17:46:27.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.590+0000 7f1d7a200640 1 -- 172.21.15.39:0/2399097650 shutdown_connections 2024-01-24T17:46:27.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.590+0000 7f1d7a200640 1 -- 172.21.15.39:0/2399097650 >> 172.21.15.39:0/2399097650 conn(0x7f1d7406a340 msgr2=0x7f1d740cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:27.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.590+0000 7f1d7a200640 1 -- 172.21.15.39:0/2399097650 shutdown_connections 2024-01-24T17:46:27.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.590+0000 7f1d7a200640 1 -- 172.21.15.39:0/2399097650 wait complete. 2024-01-24T17:46:27.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.592+0000 7f1d7a200640 1 Processor -- start 2024-01-24T17:46:27.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.592+0000 7f1d7a200640 1 -- start start 2024-01-24T17:46:27.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d7a200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d54004d00 0x7f1d740c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d7a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740c5d30 0x7f1d740c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d7a200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740cd5a0 0x7f1d740cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d7a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740cdf40 con 0x7f1d740c5d30 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d7a200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74154370 con 0x7f1d54004d00 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d7a200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74154e40 con 0x7f1d740cd5a0 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d73e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740c5d30 0x7f1d740c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d73e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740c5d30 0x7f1d740c6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47030/0 (socket says 172.21.15.39:47030) 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d73e00640 1 -- 172.21.15.39:0/2779673656 learned_addr learned my addr 172.21.15.39:0/2779673656 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d78e00640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d54004d00 0x7f1d740c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d79800640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740cd5a0 0x7f1d740cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d73e00640 1 -- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740cd5a0 msgr2=0x7f1d740cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d73e00640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740cd5a0 0x7f1d740cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d73e00640 1 -- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d54004d00 msgr2=0x7f1d740c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d73e00640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d54004d00 0x7f1d740c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.593+0000 7f1d73e00640 1 -- 172.21.15.39:0/2779673656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d74153760 con 0x7f1d740c5d30 2024-01-24T17:46:27.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.594+0000 7f1d73e00640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740c5d30 0x7f1d740c6190 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00e550 tx=0x7f1d5c00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:27.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.594+0000 7f1d70c00640 1 -- 172.21.15.39:0/2779673656 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c0052e0 con 0x7f1d740c5d30 2024-01-24T17:46:27.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.594+0000 7f1d70c00640 1 -- 172.21.15.39:0/2779673656 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c005440 con 0x7f1d740c5d30 2024-01-24T17:46:27.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.594+0000 7f1d70c00640 1 -- 172.21.15.39:0/2779673656 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c003810 con 0x7f1d740c5d30 2024-01-24T17:46:27.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.594+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d74153a40 con 0x7f1d740c5d30 2024-01-24T17:46:27.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.594+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d74154000 con 0x7f1d740c5d30 2024-01-24T17:46:27.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.596+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7404dff0 con 0x7f1d740c5d30 2024-01-24T17:46:27.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.596+0000 7f1d70c00640 1 -- 172.21.15.39:0/2779673656 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1d5c003970 con 0x7f1d740c5d30 2024-01-24T17:46:27.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.597+0000 7f1d70c00640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d4c06a430 0x7f1d4c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:27.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.597+0000 7f1d70c00640 1 -- 172.21.15.39:0/2779673656 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1d5c0882f0 con 0x7f1d740c5d30 2024-01-24T17:46:27.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.598+0000 7f1d78e00640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d4c06a430 0x7f1d4c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:27.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.599+0000 7f1d78e00640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d4c06a430 0x7f1d4c06c8f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009a00 tx=0x7f1d64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:27.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.602+0000 7f1d70c00640 1 -- 172.21.15.39:0/2779673656 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1d5c053c30 con 0x7f1d740c5d30 2024-01-24T17:46:27.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.797+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d7415a690 con 0x7f1d740c5d30 2024-01-24T17:46:27.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.799+0000 7f1d70c00640 1 -- 172.21.15.39:0/2779673656 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+12389 (secure 0 0 0) 0x7f1d5c057250 con 0x7f1d740c5d30 2024-01-24T17:46:27.801 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:27.801 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":72,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:26.277924+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:27.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.801+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d4c06a430 msgr2=0x7f1d4c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.801+0000 7f1d7a200640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d4c06a430 0x7f1d4c06c8f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009a00 tx=0x7f1d64008040 comp rx=0 tx=0).stop 2024-01-24T17:46:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.801+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740c5d30 msgr2=0x7f1d740c6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.801+0000 7f1d7a200640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740c5d30 0x7f1d740c6190 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00e550 tx=0x7f1d5c00ea10 comp rx=0 tx=0).stop 2024-01-24T17:46:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.802+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 shutdown_connections 2024-01-24T17:46:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.802+0000 7f1d7a200640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d740cd5a0 0x7f1d740cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.802+0000 7f1d7a200640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d4c06a430 0x7f1d4c06c8f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.802+0000 7f1d7a200640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d740c5d30 0x7f1d740c6190 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.802+0000 7f1d7a200640 1 --2- 172.21.15.39:0/2779673656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d54004d00 0x7f1d740c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.802+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 >> 172.21.15.39:0/2779673656 conn(0x7f1d7406a340 msgr2=0x7f1d740b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.802+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 shutdown_connections 2024-01-24T17:46:27.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:27.802+0000 7f1d7a200640 1 -- 172.21.15.39:0/2779673656 wait complete. 2024-01-24T17:46:27.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:46:27.821 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:46:28.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.040+0000 7f7e56a00640 1 Processor -- start 2024-01-24T17:46:28.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.040+0000 7f7e56a00640 1 -- start start 2024-01-24T17:46:28.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.041+0000 7f7e56a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e5015ca30 0x7f7e500d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.041+0000 7f7e56a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7e500d0e90 0x7f7e501606b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.041+0000 7f7e56a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e50161040 con 0x7f7e50160bf0 2024-01-24T17:46:28.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.041+0000 7f7e56a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e500d37a0 con 0x7f7e500d0e90 2024-01-24T17:46:28.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.041+0000 7f7e56a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e500d13c0 con 0x7f7e5015ca30 2024-01-24T17:46:28.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e54a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7e50160bf0 0x7f7e500cd570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50668/0 (socket says 172.21.15.39:50668) 2024-01-24T17:46:28.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e54a00640 1 -- 172.21.15.39:0/3664362048 learned_addr learned my addr 172.21.15.39:0/3664362048 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:28.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e4f400640 1 --2- 172.21.15.39:0/3664362048 >> v2:172.21.15.39:3300/0 conn(0x7f7e500d0e90 0x7f7e501606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e4fe00640 1 --2- 172.21.15.39:0/3664362048 >> v2:172.21.15.100:3300/0 conn(0x7f7e5015ca30 0x7f7e500d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e4ea00640 1 -- 172.21.15.39:0/3664362048 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660110429 0 0) 0x7f7e48001110 con 0x7f7e50160bf0 2024-01-24T17:46:28.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e4ea00640 1 -- 172.21.15.39:0/3664362048 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e30003520 con 0x7f7e50160bf0 2024-01-24T17:46:28.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e4f400640 1 -- 172.21.15.39:0/3664362048 >> v2:172.21.15.100:3300/0 conn(0x7f7e5015ca30 msgr2=0x7f7e500d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:28.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e4f400640 1 --2- 172.21.15.39:0/3664362048 >> v2:172.21.15.100:3300/0 conn(0x7f7e5015ca30 0x7f7e500d3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e4f400640 1 -- 172.21.15.39:0/3664362048 >> v1:172.21.15.39:6789/0 conn(0x7f7e50160bf0 legacy=0x7f7e500cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.042+0000 7f7e4f400640 1 -- 172.21.15.39:0/3664362048 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e500d16a0 con 0x7f7e500d0e90 2024-01-24T17:46:28.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.044+0000 7f7e4f400640 1 --2- 172.21.15.39:0/3664362048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e501606b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f7e5008ba00 tx=0x7f7e3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc86fb05317b80ba server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.044+0000 7f7e4ea00640 1 -- 172.21.15.39:0/3664362048 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3802cc80 con 0x7f7e500d0e90 2024-01-24T17:46:28.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.044+0000 7f7e4ea00640 1 -- 172.21.15.39:0/3664362048 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3802cde0 con 0x7f7e500d0e90 2024-01-24T17:46:28.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.044+0000 7f7e4ea00640 1 -- 172.21.15.39:0/3664362048 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38034640 con 0x7f7e500d0e90 2024-01-24T17:46:28.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.044+0000 7f7e56a00640 1 -- 172.21.15.39:0/3664362048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 msgr2=0x7f7e501606b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.044+0000 7f7e56a00640 1 --2- 172.21.15.39:0/3664362048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e501606b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f7e5008ba00 tx=0x7f7e3802bed0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.045+0000 7f7e56a00640 1 -- 172.21.15.39:0/3664362048 shutdown_connections 2024-01-24T17:46:28.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.045+0000 7f7e56a00640 1 --2- 172.21.15.39:0/3664362048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e501606b0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.045+0000 7f7e56a00640 1 --2- 172.21.15.39:0/3664362048 >> v2:172.21.15.100:3300/0 conn(0x7f7e5015ca30 0x7f7e500d3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.045+0000 7f7e56a00640 1 -- 172.21.15.39:0/3664362048 >> 172.21.15.39:0/3664362048 conn(0x7f7e5009bc80 msgr2=0x7f7e500d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:28.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.045+0000 7f7e56a00640 1 -- 172.21.15.39:0/3664362048 shutdown_connections 2024-01-24T17:46:28.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.045+0000 7f7e56a00640 1 -- 172.21.15.39:0/3664362048 wait complete. 2024-01-24T17:46:28.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.047+0000 7f7e56a00640 1 Processor -- start 2024-01-24T17:46:28.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.047+0000 7f7e56a00640 1 -- start start 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e56a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e50156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e56a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e5015ca30 0x7f7e5015dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e56a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e50160bf0 0x7f7e50157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e56a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5015f3b0 con 0x7f7e500d0e90 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e56a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5015e3c0 con 0x7f7e50160bf0 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e56a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500cb510 con 0x7f7e5015ca30 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e4fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e50156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e4fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e50156af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47046/0 (socket says 172.21.15.39:47046) 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e4fe00640 1 -- 172.21.15.39:0/1827568603 learned_addr learned my addr 172.21.15.39:0/1827568603 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e4fe00640 1 -- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e5015ca30 msgr2=0x7f7e5015dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e4fe00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e5015ca30 0x7f7e5015dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e4fe00640 1 -- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e50160bf0 msgr2=0x7f7e50157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e4fe00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e50160bf0 0x7f7e50157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.048+0000 7f7e4fe00640 1 -- 172.21.15.39:0/1827568603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e38009a70 con 0x7f7e500d0e90 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.049+0000 7f7e4fe00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e50156af0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000e9e0 tx=0x7f7e4000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.049+0000 7f7e47e00640 1 -- 172.21.15.39:0/1827568603 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40009670 con 0x7f7e500d0e90 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.049+0000 7f7e47e00640 1 -- 172.21.15.39:0/1827568603 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4000c9d0 con 0x7f7e500d0e90 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.049+0000 7f7e47e00640 1 -- 172.21.15.39:0/1827568603 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4000cd80 con 0x7f7e500d0e90 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.049+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e500cabe0 con 0x7f7e500d0e90 2024-01-24T17:46:28.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.049+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e500cb1a0 con 0x7f7e500d0e90 2024-01-24T17:46:28.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.050+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5004dff0 con 0x7f7e500d0e90 2024-01-24T17:46:28.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.051+0000 7f7e47e00640 1 -- 172.21.15.39:0/1827568603 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7e40013470 con 0x7f7e500d0e90 2024-01-24T17:46:28.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.052+0000 7f7e47e00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2806a430 0x7f7e2806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.052+0000 7f7e47e00640 1 -- 172.21.15.39:0/1827568603 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e40088880 con 0x7f7e500d0e90 2024-01-24T17:46:28.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.053+0000 7f7e4f400640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2806a430 0x7f7e2806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.054+0000 7f7e4f400640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2806a430 0x7f7e2806c8f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7e38036000 tx=0x7f7e3802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.056+0000 7f7e47e00640 1 -- 172.21.15.39:0/1827568603 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7e400541b0 con 0x7f7e500d0e90 2024-01-24T17:46:28.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.256+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e5015b3e0 con 0x7f7e500d0e90 2024-01-24T17:46:28.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.257+0000 7f7e47e00640 1 -- 172.21.15.39:0/1827568603 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e400577d0 con 0x7f7e500d0e90 2024-01-24T17:46:28.259 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2806a430 msgr2=0x7f7e2806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2806a430 0x7f7e2806c8f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7e38036000 tx=0x7f7e3802e000 comp rx=0 tx=0).stop 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 msgr2=0x7f7e50156af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e50156af0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000e9e0 tx=0x7f7e4000eea0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 shutdown_connections 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e50160bf0 0x7f7e50157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e5015ca30 0x7f7e5015dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2806a430 0x7f7e2806c8f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 --2- 172.21.15.39:0/1827568603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e500d0e90 0x7f7e50156af0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 >> 172.21.15.39:0/1827568603 conn(0x7f7e5009bc80 msgr2=0x7f7e5015a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:28.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 shutdown_connections 2024-01-24T17:46:28.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.260+0000 7f7e56a00640 1 -- 172.21.15.39:0/1827568603 wait complete. 2024-01-24T17:46:28.282 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T17:46:28.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.500+0000 7f7f95000640 1 Processor -- start 2024-01-24T17:46:28.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.500+0000 7f7f95000640 1 -- start start 2024-01-24T17:46:28.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.502+0000 7f7f95000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7f900cf8a0 0x7f7f9015e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.502+0000 7f7f95000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9015e280 con 0x7f7f9015dea0 2024-01-24T17:46:28.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.502+0000 7f7f95000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f900cdee0 con 0x7f7f900b3780 2024-01-24T17:46:28.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.502+0000 7f7f95000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f900d0700 con 0x7f7f900cf8a0 2024-01-24T17:46:28.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.502+0000 7f7f8f400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7f9015dea0 0x7f7f900d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50674/0 (socket says 172.21.15.39:50674) 2024-01-24T17:46:28.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.502+0000 7f7f8f400640 1 -- 172.21.15.39:0/336899478 learned_addr learned my addr 172.21.15.39:0/336899478 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:28.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8fe00640 1 --2- 172.21.15.39:0/336899478 >> v2:172.21.15.100:3300/0 conn(0x7f7f900cf8a0 0x7f7f9015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088007193 0 0) 0x7f7f9015e280 con 0x7f7f9015dea0 2024-01-24T17:46:28.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f70003520 con 0x7f7f9015dea0 2024-01-24T17:46:28.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257440089 0 0) 0x7f7f70003520 con 0x7f7f9015dea0 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 >> v2:172.21.15.100:3300/0 conn(0x7f7f900cf8a0 msgr2=0x7f7f9015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 --2- 172.21.15.39:0/336899478 >> v2:172.21.15.100:3300/0 conn(0x7f7f900cf8a0 0x7f7f9015e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 >> v1:172.21.15.100:6789/0 conn(0x7f7f900b3780 legacy=0x7f7f900cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f900ce130 con 0x7f7f9015dea0 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7f800017f0 con 0x7f7f9015dea0 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 >> v1:172.21.15.39:6789/0 conn(0x7f7f9015dea0 legacy=0x7f7f900d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f700047c0 0x7f7f70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f70005180 0x7f7f7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900d0700 con 0x7f7f70003970 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7000c0d0 con 0x7f7f70005180 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.503+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7000c290 con 0x7f7f700047c0 2024-01-24T17:46:28.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8f400640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8ea00640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f700047c0 0x7f7f70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8fe00640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f70005180 0x7f7f7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8f400640 1 -- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f700047c0 msgr2=0x7f7f70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8f400640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f700047c0 0x7f7f70004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8f400640 1 -- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f70005180 msgr2=0x7f7f7000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8f400640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f70005180 0x7f7f7000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8f400640 1 -- 172.21.15.39:0/336899478 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f70003520 con 0x7f7f70003970 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8ea00640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f700047c0 0x7f7f70004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8fe00640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f70005180 0x7f7f7000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:28.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.504+0000 7f7f8fe00640 1 -- 172.21.15.39:0/336899478 reap_dead start 2024-01-24T17:46:28.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.505+0000 7f7f8f400640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f70003d90 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c190 tx=0x7f7f700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1fd7fdcabcd198 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.505+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80007270 con 0x7f7f70003970 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.505+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f800073f0 con 0x7f7f70003970 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.505+0000 7f7f8e000640 1 -- 172.21.15.39:0/336899478 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8002c8a0 con 0x7f7f70003970 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.506+0000 7f7f95000640 1 -- 172.21.15.39:0/336899478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 msgr2=0x7f7f70003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.506+0000 7f7f95000640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f70003d90 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000c190 tx=0x7f7f700057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.506+0000 7f7f95000640 1 -- 172.21.15.39:0/336899478 shutdown_connections 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.506+0000 7f7f95000640 1 --2- 172.21.15.39:0/336899478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f70003d90 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.506+0000 7f7f95000640 1 -- 172.21.15.39:0/336899478 >> 172.21.15.39:0/336899478 conn(0x7f7f9009bb30 msgr2=0x7f7f900d27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.506+0000 7f7f95000640 1 -- 172.21.15.39:0/336899478 shutdown_connections 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.506+0000 7f7f95000640 1 -- 172.21.15.39:0/336899478 wait complete. 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.508+0000 7f7f95000640 1 Processor -- start 2024-01-24T17:46:28.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.508+0000 7f7f95000640 1 -- start start 2024-01-24T17:46:28.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.509+0000 7f7f95000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f900c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.509+0000 7f7f95000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f90159730 0x7f7f900b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.509+0000 7f7f95000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f901f1990 0x7f7f901f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.509+0000 7f7f95000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900c9890 con 0x7f7f70003970 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.509+0000 7f7f95000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900c7600 con 0x7f7f901f1990 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.509+0000 7f7f95000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900c78e0 con 0x7f7f90159730 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f900c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f900c6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47054/0 (socket says 172.21.15.39:47054) 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 -- 172.21.15.39:0/3910740835 learned_addr learned my addr 172.21.15.39:0/3910740835 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8fe00640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f901f1990 0x7f7f901f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8ea00640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f90159730 0x7f7f900b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 -- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f90159730 msgr2=0x7f7f900b7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f90159730 0x7f7f900b7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 -- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f901f1990 msgr2=0x7f7f901f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f901f1990 0x7f7f901f1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 -- 172.21.15.39:0/3910740835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f900c7bc0 con 0x7f7f70003970 2024-01-24T17:46:28.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8fe00640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f901f1990 0x7f7f901f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:28.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.510+0000 7f7f8f400640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f900c6d00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7f8002c440 tx=0x7f7f8002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.511+0000 7f7f87400640 1 -- 172.21.15.39:0/3910740835 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8003c040 con 0x7f7f70003970 2024-01-24T17:46:28.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.511+0000 7f7f87400640 1 -- 172.21.15.39:0/3910740835 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f800017b0 con 0x7f7f70003970 2024-01-24T17:46:28.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.511+0000 7f7f87400640 1 -- 172.21.15.39:0/3910740835 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f800059a0 con 0x7f7f70003970 2024-01-24T17:46:28.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.511+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f900c7ea0 con 0x7f7f70003970 2024-01-24T17:46:28.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.511+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f901f2360 con 0x7f7f70003970 2024-01-24T17:46:28.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.512+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f90067050 con 0x7f7f70003970 2024-01-24T17:46:28.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.513+0000 7f7f87400640 1 -- 172.21.15.39:0/3910740835 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7f80032070 con 0x7f7f70003970 2024-01-24T17:46:28.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.514+0000 7f7f87400640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7f6806a460 0x7f7f6806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.514+0000 7f7f87400640 1 -- 172.21.15.39:0/3910740835 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f800a9950 con 0x7f7f70003970 2024-01-24T17:46:28.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.514+0000 7f7f8ea00640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7f6806a460 0x7f7f6806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.516+0000 7f7f8ea00640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7f6806a460 0x7f7f6806c920 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7f900d2b20 tx=0x7f7f78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.518+0000 7f7f87400640 1 -- 172.21.15.39:0/3910740835 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7f80075290 con 0x7f7f70003970 2024-01-24T17:46:28.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.714+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7f900cffb0 con 0x7f7f70003970 2024-01-24T17:46:28.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.715+0000 7f7f87400640 1 -- 172.21.15.39:0/3910740835 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7f800788b0 con 0x7f7f70003970 2024-01-24T17:46:28.717 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:28.717 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-01-24T17:46:28.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:46:28.717 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-01-24T17:46:28.717 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-01-24T17:46:28.718 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:28.718 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-01-24T17:46:28.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7f6806a460 msgr2=0x7f7f6806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7f6806a460 0x7f7f6806c920 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7f900d2b20 tx=0x7f7f78009040 comp rx=0 tx=0).stop 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 msgr2=0x7f7f900c6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f900c6d00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7f8002c440 tx=0x7f7f8002e070 comp rx=0 tx=0).stop 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 shutdown_connections 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7f901f1990 0x7f7f901f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7f90159730 0x7f7f900b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7f6806a460 0x7f7f6806c920 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 --2- 172.21.15.39:0/3910740835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f70003970 0x7f7f900c6d00 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 >> 172.21.15.39:0/3910740835 conn(0x7f7f9009bb30 msgr2=0x7f7f900b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.718+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 shutdown_connections 2024-01-24T17:46:28.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.719+0000 7f7f95000640 1 -- 172.21.15.39:0/3910740835 wait complete. 2024-01-24T17:46:28.738 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T17:46:28.738 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T17:46:28.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.960+0000 7fdd5b200640 1 Processor -- start 2024-01-24T17:46:28.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.960+0000 7fdd5b200640 1 -- start start 2024-01-24T17:46:28.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.961+0000 7fdd5b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdd540cda00 0x7fdd540d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.961+0000 7fdd5b200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdd540d1f30 0x7fdd540b6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.961+0000 7fdd5b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd540b5dd0 con 0x7fdd5406add0 2024-01-24T17:46:28.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.961+0000 7fdd5b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd540cdf00 con 0x7fdd540cda00 2024-01-24T17:46:28.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.961+0000 7fdd5b200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd540ce1e0 con 0x7fdd540d1f30 2024-01-24T17:46:28.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd53400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdd540cda00 0x7fdd540d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd53400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdd540cda00 0x7fdd540d19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47066/0 (socket says 172.21.15.39:47066) 2024-01-24T17:46:28.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd53400640 1 -- 172.21.15.39:0/303946364 learned_addr learned my addr 172.21.15.39:0/303946364 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:28.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd53e00640 1 --1- 172.21.15.39:0/303946364 >> v1:172.21.15.39:6789/0 conn(0x7fdd5406add0 0x7fdd5406b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50682/0 (socket says 172.21.15.39:50682) 2024-01-24T17:46:28.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd59200640 1 --2- 172.21.15.39:0/303946364 >> v2:172.21.15.100:3300/0 conn(0x7fdd540d1f30 0x7fdd540b6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd53400640 1 -- 172.21.15.39:0/303946364 >> v2:172.21.15.100:3300/0 conn(0x7fdd540d1f30 msgr2=0x7fdd540b6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:28.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd53400640 1 --2- 172.21.15.39:0/303946364 >> v2:172.21.15.100:3300/0 conn(0x7fdd540d1f30 0x7fdd540b6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd53400640 1 -- 172.21.15.39:0/303946364 >> v1:172.21.15.39:6789/0 conn(0x7fdd5406add0 legacy=0x7fdd5406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.962+0000 7fdd53400640 1 -- 172.21.15.39:0/303946364 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd540b6070 con 0x7fdd540cda00 2024-01-24T17:46:28.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.963+0000 7fdd53400640 1 --2- 172.21.15.39:0/303946364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd540cda00 0x7fdd540d19f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fdd540722e0 tx=0x7fdd3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29592cba95e5c4a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.964+0000 7fdd52a00640 1 -- 172.21.15.39:0/303946364 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c030860 con 0x7fdd540cda00 2024-01-24T17:46:28.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.964+0000 7fdd52a00640 1 -- 172.21.15.39:0/303946364 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c0309c0 con 0x7fdd540cda00 2024-01-24T17:46:28.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.964+0000 7fdd52a00640 1 -- 172.21.15.39:0/303946364 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c030d50 con 0x7fdd540cda00 2024-01-24T17:46:28.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.964+0000 7fdd5b200640 1 -- 172.21.15.39:0/303946364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd540cda00 msgr2=0x7fdd540d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.964+0000 7fdd5b200640 1 --2- 172.21.15.39:0/303946364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd540cda00 0x7fdd540d19f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fdd540722e0 tx=0x7fdd3c02b9a0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.965+0000 7fdd5b200640 1 -- 172.21.15.39:0/303946364 shutdown_connections 2024-01-24T17:46:28.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.965+0000 7fdd5b200640 1 --2- 172.21.15.39:0/303946364 >> v2:172.21.15.100:3300/0 conn(0x7fdd540d1f30 0x7fdd540b6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.965+0000 7fdd5b200640 1 --2- 172.21.15.39:0/303946364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd540cda00 0x7fdd540d19f0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.965+0000 7fdd5b200640 1 -- 172.21.15.39:0/303946364 >> 172.21.15.39:0/303946364 conn(0x7fdd540d2a30 msgr2=0x7fdd540d2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:28.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.965+0000 7fdd5b200640 1 -- 172.21.15.39:0/303946364 shutdown_connections 2024-01-24T17:46:28.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.965+0000 7fdd5b200640 1 -- 172.21.15.39:0/303946364 wait complete. 2024-01-24T17:46:28.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.967+0000 7fdd5b200640 1 Processor -- start 2024-01-24T17:46:28.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.967+0000 7fdd5b200640 1 -- start start 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd5b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5406add0 0x7fdd540cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd5b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd540cda00 0x7fdd54157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd5b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd540d1f30 0x7fdd541f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd5b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54158a70 con 0x7fdd5406add0 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd5b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54157df0 con 0x7fdd540d1f30 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd5b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd541580d0 con 0x7fdd540cda00 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5406add0 0x7fdd540cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5406add0 0x7fdd540cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47068/0 (socket says 172.21.15.39:47068) 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53e00640 1 -- 172.21.15.39:0/4288790897 learned_addr learned my addr 172.21.15.39:0/4288790897 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd59200640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd540d1f30 0x7fdd541f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53400640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd540cda00 0x7fdd54157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53e00640 1 -- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd540cda00 msgr2=0x7fdd54157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53e00640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd540cda00 0x7fdd54157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53e00640 1 -- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd540d1f30 msgr2=0x7fdd541f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53e00640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd540d1f30 0x7fdd541f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd53e00640 1 -- 172.21.15.39:0/4288790897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0094d0 con 0x7fdd5406add0 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.968+0000 7fdd59200640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd540d1f30 0x7fdd541f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.969+0000 7fdd53e00640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5406add0 0x7fdd540cc160 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c4d0 tx=0x7fdd4400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.969+0000 7fdd4be00640 1 -- 172.21.15.39:0/4288790897 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44007480 con 0x7fdd5406add0 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.969+0000 7fdd4be00640 1 -- 172.21.15.39:0/4288790897 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd440075e0 con 0x7fdd5406add0 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.969+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd541583b0 con 0x7fdd5406add0 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.969+0000 7fdd4be00640 1 -- 172.21.15.39:0/4288790897 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd440049e0 con 0x7fdd5406add0 2024-01-24T17:46:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.969+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd54149e00 con 0x7fdd5406add0 2024-01-24T17:46:28.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.970+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd540c9a70 con 0x7fdd5406add0 2024-01-24T17:46:28.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.971+0000 7fdd4be00640 1 -- 172.21.15.39:0/4288790897 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdd44004b40 con 0x7fdd5406add0 2024-01-24T17:46:28.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.973+0000 7fdd4be00640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd2c06a430 0x7fdd2c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:28.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.973+0000 7fdd4be00640 1 -- 172.21.15.39:0/4288790897 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdd44087b80 con 0x7fdd5406add0 2024-01-24T17:46:28.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.973+0000 7fdd53400640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd2c06a430 0x7fdd2c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:28.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.974+0000 7fdd53400640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd2c06a430 0x7fdd2c06c8f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c036000 tx=0x7fdd3c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:28.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:28.977+0000 7fdd4be00640 1 -- 172.21.15.39:0/4288790897 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdd540c9a70 con 0x7fdd5406add0 2024-01-24T17:46:29.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.174+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdd540c9a70 con 0x7fdd5406add0 2024-01-24T17:46:29.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.516+0000 7fdd4be00640 1 -- 172.21.15.39:0/4288790897 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdd540c9a70 con 0x7fdd5406add0 2024-01-24T17:46:29.519 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-01-24T17:46:29.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd2c06a430 msgr2=0x7fdd2c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:29.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd2c06a430 0x7fdd2c06c8f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c036000 tx=0x7fdd3c03b040 comp rx=0 tx=0).stop 2024-01-24T17:46:29.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5406add0 msgr2=0x7fdd540cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:29.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5406add0 0x7fdd540cc160 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c4d0 tx=0x7fdd4400c990 comp rx=0 tx=0).stop 2024-01-24T17:46:29.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 shutdown_connections 2024-01-24T17:46:29.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdd540d1f30 0x7fdd541f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdd540cda00 0x7fdd54157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd2c06a430 0x7fdd2c06c8f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 --2- 172.21.15.39:0/4288790897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd5406add0 0x7fdd540cc160 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.522+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 >> 172.21.15.39:0/4288790897 conn(0x7fdd540d2a30 msgr2=0x7fdd540b7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:29.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.523+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 shutdown_connections 2024-01-24T17:46:29.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.523+0000 7fdd5b200640 1 -- 172.21.15.39:0/4288790897 wait complete. 2024-01-24T17:46:29.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T17:46:29.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.762+0000 7f934e800640 1 Processor -- start 2024-01-24T17:46:29.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.762+0000 7f934e800640 1 -- start start 2024-01-24T17:46:29.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.763+0000 7f934e800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934815aa60 0x7f93480ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:29.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.763+0000 7f934e800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f93480cb280 0x7f93480cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:29.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.764+0000 7f934e800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480cbc20 con 0x7f934806cec0 2024-01-24T17:46:29.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.764+0000 7f934e800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93480cae90 con 0x7f93480cb280 2024-01-24T17:46:29.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.764+0000 7f934e800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93480ca4c0 con 0x7f934815aa60 2024-01-24T17:46:29.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.764+0000 7f9347400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934815aa60 0x7f93480ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:29.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.764+0000 7f9347400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934815aa60 0x7f93480ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:46792/0 (socket says 172.21.15.39:46792) 2024-01-24T17:46:29.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.764+0000 7f9347400640 1 -- 172.21.15.39:0/850901610 learned_addr learned my addr 172.21.15.39:0/850901610 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:29.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.764+0000 7f933fe00640 1 --2- 172.21.15.39:0/850901610 >> v2:172.21.15.100:3300/0 conn(0x7f93480cb280 0x7f93480cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:29.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.765+0000 7f9347400640 1 -- 172.21.15.39:0/850901610 >> v2:172.21.15.100:3300/0 conn(0x7f93480cb280 msgr2=0x7f93480cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:29.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.765+0000 7f9347400640 1 --2- 172.21.15.39:0/850901610 >> v2:172.21.15.100:3300/0 conn(0x7f93480cb280 0x7f93480cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.765+0000 7f9347400640 1 -- 172.21.15.39:0/850901610 >> v1:172.21.15.100:6789/0 conn(0x7f934806cec0 legacy=0x7f934806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:29.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.765+0000 7f9347400640 1 -- 172.21.15.39:0/850901610 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934806da40 con 0x7f934815aa60 2024-01-24T17:46:29.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.766+0000 7f9347400640 1 --2- 172.21.15.39:0/850901610 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934815aa60 0x7f93480ca950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9338009870 tx=0x7f933802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aabfacfd70f9f32 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:29.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.767+0000 7f9346a00640 1 -- 172.21.15.39:0/850901610 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338030840 con 0x7f934815aa60 2024-01-24T17:46:29.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.767+0000 7f9346a00640 1 -- 172.21.15.39:0/850901610 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93380309a0 con 0x7f934815aa60 2024-01-24T17:46:29.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.767+0000 7f9346a00640 1 -- 172.21.15.39:0/850901610 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9338030d30 con 0x7f934815aa60 2024-01-24T17:46:29.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.768+0000 7f934e800640 1 -- 172.21.15.39:0/850901610 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934815aa60 msgr2=0x7f93480ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:29.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.768+0000 7f934e800640 1 --2- 172.21.15.39:0/850901610 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934815aa60 0x7f93480ca950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9338009870 tx=0x7f933802b980 comp rx=0 tx=0).stop 2024-01-24T17:46:29.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.768+0000 7f934e800640 1 -- 172.21.15.39:0/850901610 shutdown_connections 2024-01-24T17:46:29.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.768+0000 7f934e800640 1 --2- 172.21.15.39:0/850901610 >> v2:172.21.15.100:3300/0 conn(0x7f93480cb280 0x7f93480cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.768+0000 7f934e800640 1 --2- 172.21.15.39:0/850901610 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934815aa60 0x7f93480ca950 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.768+0000 7f934e800640 1 -- 172.21.15.39:0/850901610 >> 172.21.15.39:0/850901610 conn(0x7f93480ce9e0 msgr2=0x7f93480cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:29.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.768+0000 7f934e800640 1 -- 172.21.15.39:0/850901610 shutdown_connections 2024-01-24T17:46:29.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.768+0000 7f934e800640 1 -- 172.21.15.39:0/850901610 wait complete. 2024-01-24T17:46:29.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.771+0000 7f934e800640 1 Processor -- start 2024-01-24T17:46:29.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.771+0000 7f934e800640 1 -- start start 2024-01-24T17:46:29.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.772+0000 7f934e800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934806cec0 0x7f934815d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:29.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.772+0000 7f934e800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93480cb280 0x7f93480c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:29.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.772+0000 7f934e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934815aa60 0x7f9348155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:29.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.772+0000 7f934e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93480c3970 con 0x7f934815aa60 2024-01-24T17:46:29.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.772+0000 7f934e800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93480c32f0 con 0x7f93480cb280 2024-01-24T17:46:29.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.772+0000 7f934e800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934815af70 con 0x7f934806cec0 2024-01-24T17:46:29.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934815aa60 0x7f9348155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:29.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934806cec0 0x7f934815d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:29.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934815aa60 0x7f9348155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47078/0 (socket says 172.21.15.39:47078) 2024-01-24T17:46:29.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347e00640 1 -- 172.21.15.39:0/3299608174 learned_addr learned my addr 172.21.15.39:0/3299608174 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:29.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f933fe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93480cb280 0x7f93480c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:29.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347e00640 1 -- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934806cec0 msgr2=0x7f934815d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:29.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347e00640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934806cec0 0x7f934815d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347e00640 1 -- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93480cb280 msgr2=0x7f93480c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347e00640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93480cb280 0x7f93480c3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.773+0000 7f9347e00640 1 -- 172.21.15.39:0/3299608174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93380094d0 con 0x7f934815aa60 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.774+0000 7f9347e00640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934815aa60 0x7f9348155a30 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f934000c6b0 tx=0x7f934000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.774+0000 7f933f400640 1 -- 172.21.15.39:0/3299608174 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340003b70 con 0x7f934815aa60 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.774+0000 7f933f400640 1 -- 172.21.15.39:0/3299608174 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9340003cf0 con 0x7f934815aa60 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.774+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93480c3590 con 0x7f934815aa60 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.774+0000 7f933f400640 1 -- 172.21.15.39:0/3299608174 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9340003ea0 con 0x7f934815aa60 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.774+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93480c42d0 con 0x7f934815aa60 2024-01-24T17:46:29.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.775+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93480c3ab0 con 0x7f934815aa60 2024-01-24T17:46:29.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.777+0000 7f933f400640 1 -- 172.21.15.39:0/3299608174 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9340005310 con 0x7f934815aa60 2024-01-24T17:46:29.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.778+0000 7f933f400640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f932006a460 0x7f932006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:29.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.778+0000 7f933f400640 1 -- 172.21.15.39:0/3299608174 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f9340088cc0 con 0x7f934815aa60 2024-01-24T17:46:29.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.779+0000 7f9347400640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f932006a460 0x7f932006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:29.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.780+0000 7f9347400640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f932006a460 0x7f932006c920 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9338036000 tx=0x7f9338008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:29.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.782+0000 7f933f400640 1 -- 172.21.15.39:0/3299608174 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9340054520 con 0x7f934815aa60 2024-01-24T17:46:29.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:29.982+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f93480c3ab0 con 0x7f934815aa60 2024-01-24T17:46:30.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.534+0000 7f933f400640 1 -- 172.21.15.39:0/3299608174 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v74) v1 ==== 170+0+0 (secure 0 0 0) 0x7f93480c3ab0 con 0x7f934815aa60 2024-01-24T17:46:30.536 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-01-24T17:46:30.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f932006a460 msgr2=0x7f932006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:30.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f932006a460 0x7f932006c920 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9338036000 tx=0x7f9338008030 comp rx=0 tx=0).stop 2024-01-24T17:46:30.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934815aa60 msgr2=0x7f9348155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:30.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934815aa60 0x7f9348155a30 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f934000c6b0 tx=0x7f934000cb70 comp rx=0 tx=0).stop 2024-01-24T17:46:30.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 shutdown_connections 2024-01-24T17:46:30.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f932006a460 0x7f932006c920 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f934815aa60 0x7f9348155a30 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93480cb280 0x7f93480c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 --2- 172.21.15.39:0/3299608174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f934806cec0 0x7f934815d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.539+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 >> 172.21.15.39:0/3299608174 conn(0x7f93480ce9e0 msgr2=0x7f93480b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:30.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.542+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 shutdown_connections 2024-01-24T17:46:30.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.542+0000 7f934e800640 1 -- 172.21.15.39:0/3299608174 wait complete. 2024-01-24T17:46:30.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T17:46:30.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.761+0000 7f16b4600640 1 Processor -- start 2024-01-24T17:46:30.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.761+0000 7f16b4600640 1 -- start start 2024-01-24T17:46:30.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b4600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f16ac15aad0 0x7f16ac0c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:30.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b4600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f16ac0b0700 0x7f16ac0c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:30.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b4600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0c8df0 con 0x7f16ac0ca810 2024-01-24T17:46:30.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b4600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ac0ca0f0 con 0x7f16ac0b0700 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b4600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ac070e10 con 0x7f16ac15aad0 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b2600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f16ac0b0700 0x7f16ac0c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b2600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f16ac0b0700 0x7f16ac0c5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47094/0 (socket says 172.21.15.39:47094) 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b2600640 1 -- 172.21.15.39:0/3257272462 learned_addr learned my addr 172.21.15.39:0/3257272462 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.763+0000 7f16b1c00640 1 --2- 172.21.15.39:0/3257272462 >> v2:172.21.15.100:3300/0 conn(0x7f16ac15aad0 0x7f16ac0c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.764+0000 7f16b2600640 1 -- 172.21.15.39:0/3257272462 >> v2:172.21.15.100:3300/0 conn(0x7f16ac15aad0 msgr2=0x7f16ac0c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.764+0000 7f16b2600640 1 --2- 172.21.15.39:0/3257272462 >> v2:172.21.15.100:3300/0 conn(0x7f16ac15aad0 0x7f16ac0c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.764+0000 7f16b2600640 1 -- 172.21.15.39:0/3257272462 >> v1:172.21.15.100:6789/0 conn(0x7f16ac0ca810 legacy=0x7f16ac0afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:30.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.764+0000 7f16b2600640 1 -- 172.21.15.39:0/3257272462 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac070290 con 0x7f16ac0b0700 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.765+0000 7f16b2600640 1 --2- 172.21.15.39:0/3257272462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0b0700 0x7f16ac0c5a00 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0caed0 tx=0x7f16a402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd70fbad5375fc49 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.765+0000 7f16abe00640 1 -- 172.21.15.39:0/3257272462 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a402c960 con 0x7f16ac0b0700 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.765+0000 7f16abe00640 1 -- 172.21.15.39:0/3257272462 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a402cac0 con 0x7f16ac0b0700 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.765+0000 7f16abe00640 1 -- 172.21.15.39:0/3257272462 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a402ce50 con 0x7f16ac0b0700 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.766+0000 7f16b4600640 1 -- 172.21.15.39:0/3257272462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0b0700 msgr2=0x7f16ac0c5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.766+0000 7f16b4600640 1 --2- 172.21.15.39:0/3257272462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0b0700 0x7f16ac0c5a00 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0caed0 tx=0x7f16a402bdc0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.766+0000 7f16b4600640 1 -- 172.21.15.39:0/3257272462 shutdown_connections 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.766+0000 7f16b4600640 1 --2- 172.21.15.39:0/3257272462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0b0700 0x7f16ac0c5a00 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.766+0000 7f16b4600640 1 --2- 172.21.15.39:0/3257272462 >> v2:172.21.15.100:3300/0 conn(0x7f16ac15aad0 0x7f16ac0c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.766+0000 7f16b4600640 1 -- 172.21.15.39:0/3257272462 >> 172.21.15.39:0/3257272462 conn(0x7f16ac06bf90 msgr2=0x7f16ac06c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:30.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.766+0000 7f16b4600640 1 -- 172.21.15.39:0/3257272462 shutdown_connections 2024-01-24T17:46:30.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.766+0000 7f16b4600640 1 -- 172.21.15.39:0/3257272462 wait complete. 2024-01-24T17:46:30.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.768+0000 7f16b4600640 1 Processor -- start 2024-01-24T17:46:30.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.768+0000 7f16b4600640 1 -- start start 2024-01-24T17:46:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b4600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16ac0b0700 0x7f16ac158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b4600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0ca810 0x7f16ac0ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b4600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16ac15aad0 0x7f16ac0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b4600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1f6210 con 0x7f16ac0ca810 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b4600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0c42c0 con 0x7f16ac0b0700 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b4600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0c45a0 con 0x7f16ac15aad0 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0ca810 0x7f16ac0ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0ca810 0x7f16ac0ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47108/0 (socket says 172.21.15.39:47108) 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.769+0000 7f16b1200640 1 -- 172.21.15.39:0/749695755 learned_addr learned my addr 172.21.15.39:0/749695755 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b1c00640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16ac0b0700 0x7f16ac158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b1200640 1 -- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16ac15aad0 msgr2=0x7f16ac0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b2600640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16ac15aad0 0x7f16ac0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b1200640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16ac15aad0 0x7f16ac0cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b1200640 1 -- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16ac0b0700 msgr2=0x7f16ac158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b1200640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16ac0b0700 0x7f16ac158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b1200640 1 -- 172.21.15.39:0/749695755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4009b30 con 0x7f16ac0ca810 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b1c00640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16ac0b0700 0x7f16ac158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b1200640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0ca810 0x7f16ac0ccf20 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f169400c430 tx=0x7f169400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16a9600640 1 -- 172.21.15.39:0/749695755 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694011790 con 0x7f16ac0ca810 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16a9600640 1 -- 172.21.15.39:0/749695755 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16940118f0 con 0x7f16ac0ca810 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16a9600640 1 -- 172.21.15.39:0/749695755 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694011ce0 con 0x7f16ac0ca810 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.770+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac0c4880 con 0x7f16ac0ca810 2024-01-24T17:46:30.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.771+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac0c5080 con 0x7f16ac0ca810 2024-01-24T17:46:30.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.772+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac0c4da0 con 0x7f16ac0ca810 2024-01-24T17:46:30.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.773+0000 7f16a9600640 1 -- 172.21.15.39:0/749695755 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f169401c460 con 0x7f16ac0ca810 2024-01-24T17:46:30.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.774+0000 7f16a9600640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f168406a430 0x7f168406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:30.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.774+0000 7f16a9600640 1 -- 172.21.15.39:0/749695755 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1694088820 con 0x7f16ac0ca810 2024-01-24T17:46:30.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.774+0000 7f16b1c00640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f168406a430 0x7f168406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:30.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.775+0000 7f16b1c00640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f168406a430 0x7f168406c8f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f169c006fd0 tx=0x7f169c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:30.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:30.779+0000 7f16a9600640 1 -- 172.21.15.39:0/749695755 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f16ac0c4da0 con 0x7f16ac0ca810 2024-01-24T17:46:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.012+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f16ac0c4da0 con 0x7f16ac0ca810 2024-01-24T17:46:31.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.020+0000 7f16a9600640 1 -- 172.21.15.39:0/749695755 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f16ac0c4da0 con 0x7f16ac0ca810 2024-01-24T17:46:31.021 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 10 and data pool 11 2024-01-24T17:46:31.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.024+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f168406a430 msgr2=0x7f168406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.024+0000 7f16b4600640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f168406a430 0x7f168406c8f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f169c006fd0 tx=0x7f169c008040 comp rx=0 tx=0).stop 2024-01-24T17:46:31.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.024+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0ca810 msgr2=0x7f16ac0ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.024+0000 7f16b4600640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0ca810 0x7f16ac0ccf20 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f169400c430 tx=0x7f169400c8f0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.024+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 shutdown_connections 2024-01-24T17:46:31.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.024+0000 7f16b4600640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16ac15aad0 0x7f16ac0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.024+0000 7f16b4600640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f168406a430 0x7f168406c8f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.024+0000 7f16b4600640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16ac0ca810 0x7f16ac0ccf20 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.025+0000 7f16b4600640 1 --2- 172.21.15.39:0/749695755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16ac0b0700 0x7f16ac158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.025+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 >> 172.21.15.39:0/749695755 conn(0x7f16ac06bf90 msgr2=0x7f16ac0ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:31.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.026+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 shutdown_connections 2024-01-24T17:46:31.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.026+0000 7f16b4600640 1 -- 172.21.15.39:0/749695755 wait complete. 2024-01-24T17:46:31.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:31.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.241+0000 7f24dca00640 1 Processor -- start 2024-01-24T17:46:31.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.242+0000 7f24dca00640 1 -- start start 2024-01-24T17:46:31.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.243+0000 7f24dca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d4070920 0x7f24d40b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.243+0000 7f24dca00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f24d415f390 0x7f24d40c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.243+0000 7f24dca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d4070470 con 0x7f24d40b0010 2024-01-24T17:46:31.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.243+0000 7f24dca00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d415ef00 con 0x7f24d415f390 2024-01-24T17:46:31.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.243+0000 7f24dca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40710a0 con 0x7f24d4070920 2024-01-24T17:46:31.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.243+0000 7f24da000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f24d40b0010 0x7f24d415bb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:50694/0 (socket says 172.21.15.39:50694) 2024-01-24T17:46:31.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.243+0000 7f24da000640 1 -- 172.21.15.39:0/3818779751 learned_addr learned my addr 172.21.15.39:0/3818779751 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:31.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d9600640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d4070920 0x7f24d40b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916330618 0 0) 0x7f24d4070470 con 0x7f24d40b0010 2024-01-24T17:46:31.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24daa00640 1 --2- 172.21.15.39:0/3818779751 >> v2:172.21.15.100:3300/0 conn(0x7f24d415f390 0x7f24d40c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b4003520 con 0x7f24d40b0010 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 949022979 0 0) 0x7f24b4003520 con 0x7f24d40b0010 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d4070920 msgr2=0x7f24d40b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d4070920 0x7f24d40b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 >> v2:172.21.15.100:3300/0 conn(0x7f24d415f390 msgr2=0x7f24d40c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 --2- 172.21.15.39:0/3818779751 >> v2:172.21.15.100:3300/0 conn(0x7f24d415f390 0x7f24d40c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d40b0930 con 0x7f24d40b0010 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f24c40019f0 con 0x7f24d40b0010 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 >> v1:172.21.15.39:6789/0 conn(0x7f24d40b0010 legacy=0x7f24d415bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24b4003970 0x7f24b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d4070920 0x7f24b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d415ef00 con 0x7f24b4004d00 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40710a0 con 0x7f24b4003970 2024-01-24T17:46:31.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.244+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b400b980 con 0x7f24d4070920 2024-01-24T17:46:31.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.245+0000 7f24daa00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.245+0000 7f24d9600640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d4070920 0x7f24b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.245+0000 7f24da000640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24b4003970 0x7f24b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.245+0000 7f24daa00640 1 -- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d4070920 msgr2=0x7f24b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:31.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.245+0000 7f24daa00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d4070920 0x7f24b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.245+0000 7f24daa00640 1 -- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24b4003970 msgr2=0x7f24b4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:31.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.245+0000 7f24daa00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24b4003970 0x7f24b4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.245+0000 7f24daa00640 1 -- 172.21.15.39:0/3818779751 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b4003520 con 0x7f24b4004d00 2024-01-24T17:46:31.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.246+0000 7f24daa00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24b4005180 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009ac0 tx=0x7f24b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5803335a143546e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc02c710 con 0x7f24b4004d00 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24cc02c870 con 0x7f24b4004d00 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24d8c00640 1 -- 172.21.15.39:0/3818779751 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc02cc00 con 0x7f24b4004d00 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24dca00640 1 -- 172.21.15.39:0/3818779751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 msgr2=0x7f24b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24dca00640 1 --2- 172.21.15.39:0/3818779751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24b4005180 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009ac0 tx=0x7f24b40057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24daa00640 1 -- 172.21.15.39:0/3818779751 reap_dead start 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24dca00640 1 -- 172.21.15.39:0/3818779751 shutdown_connections 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24dca00640 1 -- 172.21.15.39:0/3818779751 >> 172.21.15.39:0/3818779751 conn(0x7f24d406bf20 msgr2=0x7f24d406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24dca00640 1 -- 172.21.15.39:0/3818779751 shutdown_connections 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.247+0000 7f24dca00640 1 -- 172.21.15.39:0/3818779751 wait complete. 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.249+0000 7f24dca00640 1 Processor -- start 2024-01-24T17:46:31.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.249+0000 7f24dca00640 1 -- start start 2024-01-24T17:46:31.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24dca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24d414fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24dca00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d415a5d0 0x7f24d415aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24dca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d40c7a50 0x7f24d40c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24dca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d415af90 con 0x7f24b4004d00 2024-01-24T17:46:31.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24dca00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40c8410 con 0x7f24d415a5d0 2024-01-24T17:46:31.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24dca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d415c890 con 0x7f24d40c7a50 2024-01-24T17:46:31.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24da000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24d414fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24da000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24d414fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47122/0 (socket says 172.21.15.39:47122) 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.250+0000 7f24da000640 1 -- 172.21.15.39:0/2650623820 learned_addr learned my addr 172.21.15.39:0/2650623820 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24daa00640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d40c7a50 0x7f24d40c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24d9600640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d415a5d0 0x7f24d415aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24da000640 1 -- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d40c7a50 msgr2=0x7f24d40c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24da000640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d40c7a50 0x7f24d40c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24da000640 1 -- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d415a5d0 msgr2=0x7f24d415aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24da000640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d415a5d0 0x7f24d415aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24da000640 1 -- 172.21.15.39:0/2650623820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d415d6b0 con 0x7f24b4004d00 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24da000640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24d414fbd0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f24c40013d0 tx=0x7f24c400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24d2000640 1 -- 172.21.15.39:0/2650623820 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c400ee20 con 0x7f24b4004d00 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24d2000640 1 -- 172.21.15.39:0/2650623820 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c4013420 con 0x7f24b4004d00 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24d2000640 1 -- 172.21.15.39:0/2650623820 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c40137d0 con 0x7f24b4004d00 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.251+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d415cb30 con 0x7f24b4004d00 2024-01-24T17:46:31.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.252+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d415d0f0 con 0x7f24b4004d00 2024-01-24T17:46:31.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.253+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d404dff0 con 0x7f24b4004d00 2024-01-24T17:46:31.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.254+0000 7f24d2000640 1 -- 172.21.15.39:0/2650623820 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f24c4013930 con 0x7f24b4004d00 2024-01-24T17:46:31.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.255+0000 7f24d2000640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24ac06a430 0x7f24ac06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.255+0000 7f24d2000640 1 -- 172.21.15.39:0/2650623820 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24c40890a0 con 0x7f24b4004d00 2024-01-24T17:46:31.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.255+0000 7f24d9600640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24ac06a430 0x7f24ac06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.257+0000 7f24d9600640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24ac06a430 0x7f24ac06c8f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f24d41f3790 tx=0x7f24bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:31.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.259+0000 7f24d2000640 1 -- 172.21.15.39:0/2650623820 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f24c4054720 con 0x7f24b4004d00 2024-01-24T17:46:31.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.448+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24d4070700 con 0x7f24b4004d00 2024-01-24T17:46:31.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.451+0000 7f24d2000640 1 -- 172.21.15.39:0/2650623820 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16053 (secure 0 0 0) 0x7f24c4057d40 con 0x7f24b4004d00 2024-01-24T17:46:31.452 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:31.452 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":75,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:31.014172+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.453+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24ac06a430 msgr2=0x7f24ac06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24ac06a430 0x7f24ac06c8f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f24d41f3790 tx=0x7f24bc008040 comp rx=0 tx=0).stop 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 msgr2=0x7f24d414fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24d414fbd0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f24c40013d0 tx=0x7f24c400e530 comp rx=0 tx=0).stop 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 shutdown_connections 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f24d40c7a50 0x7f24d40c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24d415a5d0 0x7f24d415aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f24ac06a430 0x7f24ac06c8f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 --2- 172.21.15.39:0/2650623820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4004d00 0x7f24d414fbd0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 >> 172.21.15.39:0/2650623820 conn(0x7f24d406bf20 msgr2=0x7f24d406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:31.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 shutdown_connections 2024-01-24T17:46:31.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.454+0000 7f24dca00640 1 -- 172.21.15.39:0/2650623820 wait complete. 2024-01-24T17:46:31.473 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:31.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.681+0000 7f8ea2800640 1 Processor -- start 2024-01-24T17:46:31.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.681+0000 7f8ea2800640 1 -- start start 2024-01-24T17:46:31.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.683+0000 7f8ea2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8e9c0cf200 0x7f8e9c0d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.683+0000 7f8ea2800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8e9c0b1bf0 0x7f8e9c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.683+0000 7f8ea2800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0ce2f0 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.683+0000 7f8ea2800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9c0cd440 con 0x7f8e9c0cf200 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.683+0000 7f8ea2800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e9c0ce780 con 0x7f8e9c0b1bf0 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.683+0000 7f8ea1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8e9c0cf200 0x7f8e9c0d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.683+0000 7f8ea1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8e9c0cf200 0x7f8e9c0d3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47126/0 (socket says 172.21.15.39:47126) 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.684+0000 7f8ea1400640 1 -- 172.21.15.39:0/3225753417 learned_addr learned my addr 172.21.15.39:0/3225753417 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.684+0000 7f8ea1e00640 1 --2- 172.21.15.39:0/3225753417 >> v2:172.21.15.100:3300/0 conn(0x7f8e9c0b1bf0 0x7f8e9c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.684+0000 7f8ea1400640 1 -- 172.21.15.39:0/3225753417 >> v2:172.21.15.100:3300/0 conn(0x7f8e9c0b1bf0 msgr2=0x7f8e9c0b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.684+0000 7f8ea1400640 1 --2- 172.21.15.39:0/3225753417 >> v2:172.21.15.100:3300/0 conn(0x7f8e9c0b1bf0 0x7f8e9c0b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.684+0000 7f8ea1400640 1 -- 172.21.15.39:0/3225753417 >> v1:172.21.15.100:6789/0 conn(0x7f8e9c0d05d0 legacy=0x7f8e9c0cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.684+0000 7f8ea1400640 1 -- 172.21.15.39:0/3225753417 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c0cd6e0 con 0x7f8e9c0cf200 2024-01-24T17:46:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.686+0000 7f8ea1400640 1 --2- 172.21.15.39:0/3225753417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0cf200 0x7f8e9c0d3310 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0b5380 tx=0x7f8e8c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a126039b88d84ff0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:31.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.686+0000 7f8e9be00640 1 -- 172.21.15.39:0/3225753417 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c030860 con 0x7f8e9c0cf200 2024-01-24T17:46:31.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.686+0000 7f8e9be00640 1 -- 172.21.15.39:0/3225753417 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c0309c0 con 0x7f8e9c0cf200 2024-01-24T17:46:31.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.686+0000 7f8e9be00640 1 -- 172.21.15.39:0/3225753417 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c030d50 con 0x7f8e9c0cf200 2024-01-24T17:46:31.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.687+0000 7f8ea2800640 1 -- 172.21.15.39:0/3225753417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0cf200 msgr2=0x7f8e9c0d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.687+0000 7f8ea2800640 1 --2- 172.21.15.39:0/3225753417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0cf200 0x7f8e9c0d3310 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0b5380 tx=0x7f8e8c02b9a0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.687+0000 7f8ea2800640 1 -- 172.21.15.39:0/3225753417 shutdown_connections 2024-01-24T17:46:31.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.687+0000 7f8ea2800640 1 --2- 172.21.15.39:0/3225753417 >> v2:172.21.15.100:3300/0 conn(0x7f8e9c0b1bf0 0x7f8e9c0b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.687+0000 7f8ea2800640 1 --2- 172.21.15.39:0/3225753417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0cf200 0x7f8e9c0d3310 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.687+0000 7f8ea2800640 1 -- 172.21.15.39:0/3225753417 >> 172.21.15.39:0/3225753417 conn(0x7f8e9c0d27f0 msgr2=0x7f8e9c0d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:31.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.687+0000 7f8ea2800640 1 -- 172.21.15.39:0/3225753417 shutdown_connections 2024-01-24T17:46:31.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.687+0000 7f8ea2800640 1 -- 172.21.15.39:0/3225753417 wait complete. 2024-01-24T17:46:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.689+0000 7f8ea2800640 1 Processor -- start 2024-01-24T17:46:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.689+0000 7f8ea2800640 1 -- start start 2024-01-24T17:46:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea2800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0b1bf0 0x7f8e9c1f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea2800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c0cf200 0x7f8e9c0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0d05d0 0x7f8e9c0c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea2800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c0cb210 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea2800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c0ca600 con 0x7f8e9c0b1bf0 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea2800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c0ca8e0 con 0x7f8e9c0cf200 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0d05d0 0x7f8e9c0c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0d05d0 0x7f8e9c0c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47130/0 (socket says 172.21.15.39:47130) 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.690+0000 7f8ea1e00640 1 -- 172.21.15.39:0/2996899044 learned_addr learned my addr 172.21.15.39:0/2996899044 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea1400640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0b1bf0 0x7f8e9c1f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea0a00640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c0cf200 0x7f8e9c0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea1e00640 1 -- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c0cf200 msgr2=0x7f8e9c0c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea1e00640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c0cf200 0x7f8e9c0c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea1e00640 1 -- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0b1bf0 msgr2=0x7f8e9c1f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea1e00640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0b1bf0 0x7f8e9c1f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea1e00640 1 -- 172.21.15.39:0/2996899044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c009470 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea1e00640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0d05d0 0x7f8e9c0c8390 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8e94007e40 tx=0x7f8e9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:31.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8e99600640 1 -- 172.21.15.39:0/2996899044 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94014620 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8e99600640 1 -- 172.21.15.39:0/2996899044 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9400ee90 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9c0cabc0 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.691+0000 7f8e99600640 1 -- 172.21.15.39:0/2996899044 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e940149e0 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.692+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9c0ca2d0 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.693+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9c0cb3c0 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.694+0000 7f8e99600640 1 -- 172.21.15.39:0/2996899044 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8e94014b40 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.694+0000 7f8e99600640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e7406a430 0x7f8e7406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:31.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.694+0000 7f8e99600640 1 -- 172.21.15.39:0/2996899044 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e940891f0 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.695+0000 7f8ea1400640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e7406a430 0x7f8e7406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:31.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.696+0000 7f8ea1400640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e7406a430 0x7f8e7406c8f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c036000 tx=0x7f8e8c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:31.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.699+0000 7f8e99600640 1 -- 172.21.15.39:0/2996899044 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8e9c0cb3c0 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.884+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e9c04dff0 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.885+0000 7f8e99600640 1 -- 172.21.15.39:0/2996899044 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16053 (secure 0 0 0) 0x7f8e94054870 con 0x7f8e9c0d05d0 2024-01-24T17:46:31.886 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:31.886 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":75,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:31.014172+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e7406a430 msgr2=0x7f8e7406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e7406a430 0x7f8e7406c8f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c036000 tx=0x7f8e8c008030 comp rx=0 tx=0).stop 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0d05d0 msgr2=0x7f8e9c0c8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0d05d0 0x7f8e9c0c8390 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f8e94007e40 tx=0x7f8e9400e4b0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 shutdown_connections 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8e7406a430 0x7f8e7406c8f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e9c0d05d0 0x7f8e9c0c8390 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8e9c0cf200 0x7f8e9c0c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 --2- 172.21.15.39:0/2996899044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8e9c0b1bf0 0x7f8e9c1f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:31.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 >> 172.21.15.39:0/2996899044 conn(0x7f8e9c0d27f0 msgr2=0x7f8e9c0d2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:31.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 shutdown_connections 2024-01-24T17:46:31.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:31.888+0000 7f8ea2800640 1 -- 172.21.15.39:0/2996899044 wait complete. 2024-01-24T17:46:31.907 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T17:46:32.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.106+0000 7f9a8cc00640 1 Processor -- start 2024-01-24T17:46:32.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.106+0000 7f9a8cc00640 1 -- start start 2024-01-24T17:46:32.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a8cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9a880b0010 0x7f9a880b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a8cc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880ccf90 con 0x7f9a880cd420 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a8cc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880c9bd0 con 0x7f9a8815cf10 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a8cc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a880b09d0 con 0x7f9a880b0010 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a85400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9a880b0010 0x7f9a880b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a85400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9a880b0010 0x7f9a880b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47140/0 (socket says 172.21.15.39:47140) 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a85400640 1 -- 172.21.15.39:0/3493038398 learned_addr learned my addr 172.21.15.39:0/3493038398 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a85400640 1 -- 172.21.15.39:0/3493038398 >> v1:172.21.15.100:6789/0 conn(0x7f9a8815cf10 legacy=0x7f9a88070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a85400640 1 -- 172.21.15.39:0/3493038398 >> v1:172.21.15.39:6789/0 conn(0x7f9a880cd420 legacy=0x7f9a880c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.108+0000 7f9a85400640 1 -- 172.21.15.39:0/3493038398 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880c9eb0 con 0x7f9a880b0010 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.110+0000 7f9a85400640 1 --2- 172.21.15.39:0/3493038398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a880b0010 0x7f9a880b0490 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009870 tx=0x7f9a7002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af985551b5bb8f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.110+0000 7f9a84a00640 1 -- 172.21.15.39:0/3493038398 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70030860 con 0x7f9a880b0010 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.110+0000 7f9a84a00640 1 -- 172.21.15.39:0/3493038398 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a700309c0 con 0x7f9a880b0010 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.110+0000 7f9a84a00640 1 -- 172.21.15.39:0/3493038398 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70030d50 con 0x7f9a880b0010 2024-01-24T17:46:32.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.110+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3493038398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a880b0010 msgr2=0x7f9a880b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:32.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.110+0000 7f9a8cc00640 1 --2- 172.21.15.39:0/3493038398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a880b0010 0x7f9a880b0490 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009870 tx=0x7f9a7002b9a0 comp rx=0 tx=0).stop 2024-01-24T17:46:32.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.110+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3493038398 shutdown_connections 2024-01-24T17:46:32.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.110+0000 7f9a8cc00640 1 --2- 172.21.15.39:0/3493038398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a880b0010 0x7f9a880b0490 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:32.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.111+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3493038398 >> 172.21.15.39:0/3493038398 conn(0x7f9a8806bf20 msgr2=0x7f9a8806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:32.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.111+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3493038398 shutdown_connections 2024-01-24T17:46:32.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.111+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3493038398 wait complete. 2024-01-24T17:46:32.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.112+0000 7f9a8cc00640 1 Processor -- start 2024-01-24T17:46:32.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.112+0000 7f9a8cc00640 1 -- start start 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.113+0000 7f9a8cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a880b0010 0x7f9a880b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.113+0000 7f9a8cc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a880cd420 0x7f9a88147400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.113+0000 7f9a8cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8815cf10 0x7f9a880ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.113+0000 7f9a8cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880c6630 con 0x7f9a8815cf10 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.113+0000 7f9a8cc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880c7730 con 0x7f9a880cd420 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.113+0000 7f9a8cc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a880c6e00 con 0x7f9a880b0010 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8815cf10 0x7f9a880ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8815cf10 0x7f9a880ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47150/0 (socket says 172.21.15.39:47150) 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 -- 172.21.15.39:0/3970169341 learned_addr learned my addr 172.21.15.39:0/3970169341 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:32.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a85e00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a880b0010 0x7f9a880b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a85400640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a880cd420 0x7f9a88147400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 -- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a880b0010 msgr2=0x7f9a880b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a880b0010 0x7f9a880b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 -- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a880cd420 msgr2=0x7f9a88147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a880cd420 0x7f9a88147400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 -- 172.21.15.39:0/3970169341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a700094d0 con 0x7f9a8815cf10 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.114+0000 7f9a86800640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8815cf10 0x7f9a880ce8e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000c6b0 tx=0x7f9a8000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.115+0000 7f9a7e000640 1 -- 172.21.15.39:0/3970169341 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80003b70 con 0x7f9a8815cf10 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.115+0000 7f9a7e000640 1 -- 172.21.15.39:0/3970169341 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a80003cf0 con 0x7f9a8815cf10 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.115+0000 7f9a7e000640 1 -- 172.21.15.39:0/3970169341 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80005310 con 0x7f9a8815cf10 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.115+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a880c70e0 con 0x7f9a8815cf10 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.115+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a880b3c40 con 0x7f9a8815cf10 2024-01-24T17:46:32.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.116+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8804dff0 con 0x7f9a8815cf10 2024-01-24T17:46:32.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.117+0000 7f9a7e000640 1 -- 172.21.15.39:0/3970169341 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9a80005470 con 0x7f9a8815cf10 2024-01-24T17:46:32.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.117+0000 7f9a7e000640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a6006a480 0x7f9a6006c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:32.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.118+0000 7f9a7e000640 1 -- 172.21.15.39:0/3970169341 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a80088670 con 0x7f9a8815cf10 2024-01-24T17:46:32.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.118+0000 7f9a85e00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a6006a480 0x7f9a6006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:32.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.119+0000 7f9a85e00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a6006a480 0x7f9a6006c940 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9a78003d90 tx=0x7f9a78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:32.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.122+0000 7f9a7e000640 1 -- 172.21.15.39:0/3970169341 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9a80053cb0 con 0x7f9a8815cf10 2024-01-24T17:46:32.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:32.356+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9a880c7870 con 0x7f9a8815cf10 2024-01-24T17:46:33.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.037+0000 7f9a7e000640 1 -- 172.21.15.39:0/3970169341 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9a80003e60 con 0x7f9a8815cf10 2024-01-24T17:46:33.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.041+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a6006a480 msgr2=0x7f9a6006c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.041+0000 7f9a8cc00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a6006a480 0x7f9a6006c940 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9a78003d90 tx=0x7f9a78009040 comp rx=0 tx=0).stop 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.041+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8815cf10 msgr2=0x7f9a880ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.041+0000 7f9a8cc00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8815cf10 0x7f9a880ce8e0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000c6b0 tx=0x7f9a8000cb70 comp rx=0 tx=0).stop 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.042+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 shutdown_connections 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.042+0000 7f9a8cc00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9a6006a480 0x7f9a6006c940 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.042+0000 7f9a8cc00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8815cf10 0x7f9a880ce8e0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.042+0000 7f9a8cc00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a880cd420 0x7f9a88147400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.042+0000 7f9a8cc00640 1 --2- 172.21.15.39:0/3970169341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9a880b0010 0x7f9a880b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.042+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 >> 172.21.15.39:0/3970169341 conn(0x7f9a8806bf20 msgr2=0x7f9a88071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.042+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 shutdown_connections 2024-01-24T17:46:33.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.042+0000 7f9a8cc00640 1 -- 172.21.15.39:0/3970169341 wait complete. 2024-01-24T17:46:33.063 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-24T17:46:33.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-24T17:46:33.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.290+0000 7fd674800640 1 Processor -- start 2024-01-24T17:46:33.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.290+0000 7fd674800640 1 -- start start 2024-01-24T17:46:33.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.292+0000 7fd674800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd66c0d06b0 0x7fd66c15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:33.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.292+0000 7fd674800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0d0ae0 con 0x7fd66c0d0f20 2024-01-24T17:46:33.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.292+0000 7fd674800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0cd820 con 0x7fd66c0ce8c0 2024-01-24T17:46:33.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.292+0000 7fd674800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c0cdb00 con 0x7fd66c0d06b0 2024-01-24T17:46:33.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.292+0000 7fd671e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd66c0d0f20 0x7fd66c0d1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60162/0 (socket says 172.21.15.39:60162) 2024-01-24T17:46:33.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.292+0000 7fd671e00640 1 -- 172.21.15.39:0/3709118449 learned_addr learned my addr 172.21.15.39:0/3709118449 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:33.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd672800640 1 --2- 172.21.15.39:0/3709118449 >> v2:172.21.15.100:3300/0 conn(0x7fd66c0d06b0 0x7fd66c15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:33.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871338932 0 0) 0x7fd66c0d0ae0 con 0x7fd66c0d0f20 2024-01-24T17:46:33.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd64c003520 con 0x7fd66c0d0f20 2024-01-24T17:46:33.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196416796 0 0) 0x7fd64c003520 con 0x7fd66c0d0f20 2024-01-24T17:46:33.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 >> v2:172.21.15.100:3300/0 conn(0x7fd66c0d06b0 msgr2=0x7fd66c15dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 --2- 172.21.15.39:0/3709118449 >> v2:172.21.15.100:3300/0 conn(0x7fd66c0d06b0 0x7fd66c15dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 >> v1:172.21.15.100:6789/0 conn(0x7fd66c0ce8c0 legacy=0x7fd66c15e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0cdde0 con 0x7fd66c0d0f20 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fd65c0017f0 con 0x7fd66c0d0f20 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 >> v1:172.21.15.39:6789/0 conn(0x7fd66c0d0f20 legacy=0x7fd66c0d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd64c0039c0 0x7fd64c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c0047c0 0x7fd64c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c005180 0x7fd64c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.293+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0cdb00 con 0x7fd64c005180 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c00c080 con 0x7fd64c0039c0 2024-01-24T17:46:33.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c00c240 con 0x7fd64c0047c0 2024-01-24T17:46:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd672800640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c005180 0x7fd64c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd671e00640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd64c0039c0 0x7fd64c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd671400640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c0047c0 0x7fd64c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd672800640 1 -- 172.21.15.39:0/3709118449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c0047c0 msgr2=0x7fd64c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd672800640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c0047c0 0x7fd64c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd672800640 1 -- 172.21.15.39:0/3709118449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd64c0039c0 msgr2=0x7fd64c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd672800640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd64c0039c0 0x7fd64c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.294+0000 7fd672800640 1 -- 172.21.15.39:0/3709118449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c003520 con 0x7fd64c005180 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.295+0000 7fd672800640 1 -- 172.21.15.39:0/3709118449 reap_dead start 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.295+0000 7fd672800640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c005180 0x7fd64c00b980 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd664009fc0 tx=0x7fd64c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6daa49db570664af server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66402cc60 con 0x7fd64c005180 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66400f040 con 0x7fd64c005180 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd670a00640 1 -- 172.21.15.39:0/3709118449 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664003990 con 0x7fd64c005180 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd674800640 1 -- 172.21.15.39:0/3709118449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c005180 msgr2=0x7fd64c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd674800640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c005180 0x7fd64c00b980 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd664009fc0 tx=0x7fd64c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd674800640 1 -- 172.21.15.39:0/3709118449 shutdown_connections 2024-01-24T17:46:33.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd674800640 1 --2- 172.21.15.39:0/3709118449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c005180 0x7fd64c00b980 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd674800640 1 -- 172.21.15.39:0/3709118449 >> 172.21.15.39:0/3709118449 conn(0x7fd66c09bca0 msgr2=0x7fd66c0d2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd674800640 1 -- 172.21.15.39:0/3709118449 shutdown_connections 2024-01-24T17:46:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.296+0000 7fd674800640 1 -- 172.21.15.39:0/3709118449 wait complete. 2024-01-24T17:46:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.298+0000 7fd674800640 1 Processor -- start 2024-01-24T17:46:33.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.298+0000 7fd674800640 1 -- start start 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.299+0000 7fd674800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c005180 0x7fd66c0ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.299+0000 7fd674800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd66c0b5420 0x7fd66c1519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.299+0000 7fd674800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd66c0b6970 0x7fd66c0d2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.299+0000 7fd674800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0b6560 con 0x7fd66c0b6970 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.299+0000 7fd674800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0b6280 con 0x7fd66c0b5420 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.299+0000 7fd674800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0b5950 con 0x7fd64c005180 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd672800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd66c0b6970 0x7fd66c0d2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd672800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd66c0b6970 0x7fd66c0d2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47228/0 (socket says 172.21.15.39:47228) 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd672800640 1 -- 172.21.15.39:0/3589184649 learned_addr learned my addr 172.21.15.39:0/3589184649 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd671e00640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c005180 0x7fd66c0ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd671400640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd66c0b5420 0x7fd66c1519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd672800640 1 -- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c005180 msgr2=0x7fd66c0ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:33.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd672800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c005180 0x7fd66c0ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd672800640 1 -- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd66c0b5420 msgr2=0x7fd66c1519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd672800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd66c0b5420 0x7fd66c1519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.300+0000 7fd672800640 1 -- 172.21.15.39:0/3589184649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0b5c30 con 0x7fd66c0b6970 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.301+0000 7fd672800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd66c0b6970 0x7fd66c0d2940 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fd66402c6d0 tx=0x7fd664009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.301+0000 7fd66a000640 1 -- 172.21.15.39:0/3589184649 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664035dc0 con 0x7fd66c0b6970 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.301+0000 7fd66a000640 1 -- 172.21.15.39:0/3589184649 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664003990 con 0x7fd66c0b6970 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.301+0000 7fd66a000640 1 -- 172.21.15.39:0/3589184649 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664003d40 con 0x7fd66c0b6970 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.301+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c0b5f10 con 0x7fd66c0b6970 2024-01-24T17:46:33.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.301+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c151ee0 con 0x7fd66c0b6970 2024-01-24T17:46:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.302+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c1f2450 con 0x7fd66c0b6970 2024-01-24T17:46:33.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.303+0000 7fd66a000640 1 -- 172.21.15.39:0/3589184649 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd664004030 con 0x7fd66c0b6970 2024-01-24T17:46:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.305+0000 7fd66a000640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd64406a430 0x7fd64406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:33.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.305+0000 7fd66a000640 1 -- 172.21.15.39:0/3589184649 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6640ab5d0 con 0x7fd66c0b6970 2024-01-24T17:46:33.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.305+0000 7fd671e00640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd64406a430 0x7fd64406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:33.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.306+0000 7fd671e00640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd64406a430 0x7fd64406c8f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd65c001eb0 tx=0x7fd65c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:33.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.309+0000 7fd66a000640 1 -- 172.21.15.39:0/3589184649 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd66c1f2450 con 0x7fd66c0b6970 2024-01-24T17:46:33.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:33.539+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd66c1f2450 con 0x7fd66c0b6970 2024-01-24T17:46:34.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.557+0000 7fd66a000640 1 -- 172.21.15.39:0/3589184649 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd664076bc0 con 0x7fd66c0b6970 2024-01-24T17:46:34.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd64406a430 msgr2=0x7fd64406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:34.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd64406a430 0x7fd64406c8f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd65c001eb0 tx=0x7fd65c009040 comp rx=0 tx=0).stop 2024-01-24T17:46:34.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd66c0b6970 msgr2=0x7fd66c0d2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:34.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd66c0b6970 0x7fd66c0d2940 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fd66402c6d0 tx=0x7fd664009750 comp rx=0 tx=0).stop 2024-01-24T17:46:34.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 shutdown_connections 2024-01-24T17:46:34.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd64406a430 0x7fd64406c8f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd66c0b6970 0x7fd66c0d2940 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd66c0b5420 0x7fd66c1519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 --2- 172.21.15.39:0/3589184649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd64c005180 0x7fd66c0ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.560+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 >> 172.21.15.39:0/3589184649 conn(0x7fd66c09bca0 msgr2=0x7fd66c15f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:34.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.561+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 shutdown_connections 2024-01-24T17:46:34.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.561+0000 7fd674800640 1 -- 172.21.15.39:0/3589184649 wait complete. 2024-01-24T17:46:34.582 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-24T17:46:34.583 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:46:34.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.806+0000 7f08a0c00640 1 Processor -- start 2024-01-24T17:46:34.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.806+0000 7f08a0c00640 1 -- start start 2024-01-24T17:46:34.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.807+0000 7f08a0c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f089c0b0010 0x7f089c15bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:34.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f08a0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f089c15f390 0x7f089c0c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:34.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f08a0c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f089c070470 con 0x7f089c070920 2024-01-24T17:46:34.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f08a0c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f089c15ef00 con 0x7f089c0b0010 2024-01-24T17:46:34.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f08a0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c0710a0 con 0x7f089c15f390 2024-01-24T17:46:34.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f0899e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f089c0b0010 0x7f089c15bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:34.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f0899e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f089c0b0010 0x7f089c15bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47238/0 (socket says 172.21.15.39:47238) 2024-01-24T17:46:34.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f0899e00640 1 -- 172.21.15.39:0/2813924611 learned_addr learned my addr 172.21.15.39:0/2813924611 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:34.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f0899e00640 1 -- 172.21.15.39:0/2813924611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f089c15f390 msgr2=0x7f089c0c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:34.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f0899e00640 1 --2- 172.21.15.39:0/2813924611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f089c15f390 0x7f089c0c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f0899e00640 1 -- 172.21.15.39:0/2813924611 >> v1:172.21.15.100:6789/0 conn(0x7f089c070920 legacy=0x7f089c0b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:34.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.808+0000 7f0899e00640 1 -- 172.21.15.39:0/2813924611 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c15c0d0 con 0x7f089c0b0010 2024-01-24T17:46:34.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.809+0000 7f0899e00640 1 --2- 172.21.15.39:0/2813924611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c15bb90 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f088c009e50 tx=0x7f088c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780883bb8de0ca11 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:34.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.810+0000 7f0899400640 1 -- 172.21.15.39:0/2813924611 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c02cb00 con 0x7f089c0b0010 2024-01-24T17:46:34.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.810+0000 7f0899400640 1 -- 172.21.15.39:0/2813924611 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c02cc60 con 0x7f089c0b0010 2024-01-24T17:46:34.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.810+0000 7f0899400640 1 -- 172.21.15.39:0/2813924611 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c0343f0 con 0x7f089c0b0010 2024-01-24T17:46:34.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.811+0000 7f08a0c00640 1 -- 172.21.15.39:0/2813924611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 msgr2=0x7f089c15bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:34.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.811+0000 7f08a0c00640 1 --2- 172.21.15.39:0/2813924611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c15bb90 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f088c009e50 tx=0x7f088c02bf60 comp rx=0 tx=0).stop 2024-01-24T17:46:34.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.811+0000 7f08a0c00640 1 -- 172.21.15.39:0/2813924611 shutdown_connections 2024-01-24T17:46:34.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.811+0000 7f08a0c00640 1 --2- 172.21.15.39:0/2813924611 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f089c15f390 0x7f089c0c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.811+0000 7f08a0c00640 1 --2- 172.21.15.39:0/2813924611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c15bb90 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.811+0000 7f08a0c00640 1 -- 172.21.15.39:0/2813924611 >> 172.21.15.39:0/2813924611 conn(0x7f089c06bf20 msgr2=0x7f089c06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:34.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.811+0000 7f08a0c00640 1 -- 172.21.15.39:0/2813924611 shutdown_connections 2024-01-24T17:46:34.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.811+0000 7f08a0c00640 1 -- 172.21.15.39:0/2813924611 wait complete. 2024-01-24T17:46:34.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.813+0000 7f08a0c00640 1 Processor -- start 2024-01-24T17:46:34.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.813+0000 7f08a0c00640 1 -- start start 2024-01-24T17:46:34.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.814+0000 7f08a0c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f089c070920 0x7f089c1515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:34.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.814+0000 7f08a0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c1509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:34.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.814+0000 7f08a0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f089c15f390 0x7f089c152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:34.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.814+0000 7f08a0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c1536f0 con 0x7f089c0b0010 2024-01-24T17:46:34.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.814+0000 7f08a0c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c1539d0 con 0x7f089c070920 2024-01-24T17:46:34.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.814+0000 7f08a0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089c153cb0 con 0x7f089c15f390 2024-01-24T17:46:34.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c1509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:34.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c1509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47240/0 (socket says 172.21.15.39:47240) 2024-01-24T17:46:34.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 -- 172.21.15.39:0/3755334792 learned_addr learned my addr 172.21.15.39:0/3755334792 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 -- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f089c15f390 msgr2=0x7f089c152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f089c15f390 0x7f089c152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 -- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f089c070920 msgr2=0x7f089c1515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f089c070920 0x7f089c1515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 -- 172.21.15.39:0/3755334792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c009b30 con 0x7f089c0b0010 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.815+0000 7f0891200640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c1509e0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f088400c410 tx=0x7f088400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.816+0000 7f0892a00640 1 -- 172.21.15.39:0/3755334792 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08840035f0 con 0x7f089c0b0010 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.816+0000 7f0892a00640 1 -- 172.21.15.39:0/3755334792 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0884003750 con 0x7f089c0b0010 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.816+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089c153f90 con 0x7f089c0b0010 2024-01-24T17:46:34.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.816+0000 7f0892a00640 1 -- 172.21.15.39:0/3755334792 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0884003b40 con 0x7f089c0b0010 2024-01-24T17:46:34.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.816+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089c150f20 con 0x7f089c0b0010 2024-01-24T17:46:34.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.817+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089c04dff0 con 0x7f089c0b0010 2024-01-24T17:46:34.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.818+0000 7f0892a00640 1 -- 172.21.15.39:0/3755334792 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0884003ca0 con 0x7f089c0b0010 2024-01-24T17:46:34.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.819+0000 7f0892a00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f087406a430 0x7f087406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:34.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.819+0000 7f0892a00640 1 -- 172.21.15.39:0/3755334792 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0884087f10 con 0x7f089c0b0010 2024-01-24T17:46:34.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.819+0000 7f0899e00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f087406a430 0x7f087406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:34.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.820+0000 7f0899e00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f087406a430 0x7f087406c8f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f088c035000 tx=0x7f088c02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:34.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:34.824+0000 7f0892a00640 1 -- 172.21.15.39:0/3755334792 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0884053590 con 0x7f089c0b0010 2024-01-24T17:46:35.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.054+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f089c070700 con 0x7f089c0b0010 2024-01-24T17:46:35.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.055+0000 7f0892a00640 1 -- 172.21.15.39:0/3755334792 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f0884017090 con 0x7f089c0b0010 2024-01-24T17:46:35.055 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:35.056 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1886779141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1886779141},{"type":"v1","addr":"172.21.15.39:6833","nonce":1886779141}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5854,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2624477336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":2624477336},{"type":"v1","addr":"172.21.15.100:6837","nonce":2624477336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2465319794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2465319794},{"type":"v1","addr":"172.21.15.39:6835","nonce":2465319794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:46:31.015014+0000","modified":"2024-01-24T17:46:34.537399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6836/2467066037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2467066037},{"type":"v1","addr":"172.21.15.100:6836","nonce":2467066037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-24T17:46:35.056 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 42 2024-01-24T17:46:35.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f087406a430 msgr2=0x7f087406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f087406a430 0x7f087406c8f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f088c035000 tx=0x7f088c02e000 comp rx=0 tx=0).stop 2024-01-24T17:46:35.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 msgr2=0x7f089c1509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c1509e0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f088400c410 tx=0x7f088400c8d0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 shutdown_connections 2024-01-24T17:46:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f089c15f390 0x7f089c152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f087406a430 0x7f087406c8f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f089c0b0010 0x7f089c1509e0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 --2- 172.21.15.39:0/3755334792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f089c070920 0x7f089c1515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.057+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 >> 172.21.15.39:0/3755334792 conn(0x7f089c06bf20 msgr2=0x7f089c0c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.058+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 shutdown_connections 2024-01-24T17:46:35.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.058+0000 7f08a0c00640 1 -- 172.21.15.39:0/3755334792 wait complete. 2024-01-24T17:46:35.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.305+0000 7f6cb5a00640 1 Processor -- start 2024-01-24T17:46:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.305+0000 7f6cb5a00640 1 -- start start 2024-01-24T17:46:35.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.307+0000 7f6cb5a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6cb015d500 0x7f6cb0070510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.307+0000 7f6cb5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb00afe60 0x7f6cb00b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.307+0000 7f6cb5a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00cc410 con 0x7f6cb0158bb0 2024-01-24T17:46:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.307+0000 7f6cb5a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb00b0850 con 0x7f6cb015d500 2024-01-24T17:46:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.307+0000 7f6cb5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb00c9500 con 0x7f6cb00afe60 2024-01-24T17:46:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.308+0000 7f6ca7200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6cb015d500 0x7f6cb0070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.308+0000 7f6cb5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb00afe60 0x7f6cb00b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.308+0000 7f6ca7200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6cb015d500 0x7f6cb0070510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49556/0 (socket says 172.21.15.39:49556) 2024-01-24T17:46:35.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.308+0000 7f6ca7200640 1 -- 172.21.15.39:0/1205924447 learned_addr learned my addr 172.21.15.39:0/1205924447 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:35.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.308+0000 7f6ca7200640 1 -- 172.21.15.39:0/1205924447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb00afe60 msgr2=0x7f6cb00b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:35.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.308+0000 7f6ca7200640 1 --2- 172.21.15.39:0/1205924447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb00afe60 0x7f6cb00b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.308+0000 7f6ca7200640 1 -- 172.21.15.39:0/1205924447 >> v1:172.21.15.100:6789/0 conn(0x7f6cb0158bb0 legacy=0x7f6cb00c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.308+0000 7f6ca7200640 1 -- 172.21.15.39:0/1205924447 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb00c96f0 con 0x7f6cb015d500 2024-01-24T17:46:35.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.310+0000 7f6ca7200640 1 --2- 172.21.15.39:0/1205924447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb015d500 0x7f6cb0070510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009870 tx=0x7f6c9802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df69cd8238c2a4e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.310+0000 7f6caf400640 1 -- 172.21.15.39:0/1205924447 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98030840 con 0x7f6cb015d500 2024-01-24T17:46:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.310+0000 7f6caf400640 1 -- 172.21.15.39:0/1205924447 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c980309a0 con 0x7f6cb015d500 2024-01-24T17:46:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.310+0000 7f6caf400640 1 -- 172.21.15.39:0/1205924447 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98030d30 con 0x7f6cb015d500 2024-01-24T17:46:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.311+0000 7f6cb5a00640 1 -- 172.21.15.39:0/1205924447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb015d500 msgr2=0x7f6cb0070510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.311+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/1205924447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb015d500 0x7f6cb0070510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009870 tx=0x7f6c9802b980 comp rx=0 tx=0).stop 2024-01-24T17:46:35.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.311+0000 7f6cb5a00640 1 -- 172.21.15.39:0/1205924447 shutdown_connections 2024-01-24T17:46:35.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.311+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/1205924447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb015d500 0x7f6cb0070510 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.311+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/1205924447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb00afe60 0x7f6cb00b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.311+0000 7f6cb5a00640 1 -- 172.21.15.39:0/1205924447 >> 172.21.15.39:0/1205924447 conn(0x7f6cb006a340 msgr2=0x7f6cb00cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:35.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.311+0000 7f6cb5a00640 1 -- 172.21.15.39:0/1205924447 shutdown_connections 2024-01-24T17:46:35.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.311+0000 7f6cb5a00640 1 -- 172.21.15.39:0/1205924447 wait complete. 2024-01-24T17:46:35.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.313+0000 7f6cb5a00640 1 Processor -- start 2024-01-24T17:46:35.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.314+0000 7f6cb5a00640 1 -- start start 2024-01-24T17:46:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb00afe60 0x7f6cb0149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb0158bb0 0x7f6cb0146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb015d500 0x7f6cb00b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb0146920 con 0x7f6cb015d500 2024-01-24T17:46:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb0147ea0 con 0x7f6cb00afe60 2024-01-24T17:46:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb00b1fd0 con 0x7f6cb0158bb0 2024-01-24T17:46:35.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb015d500 0x7f6cb00b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb015d500 0x7f6cb00b2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47244/0 (socket says 172.21.15.39:47244) 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cb5000640 1 -- 172.21.15.39:0/4207746084 learned_addr learned my addr 172.21.15.39:0/4207746084 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6cafe00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb00afe60 0x7f6cb0149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.315+0000 7f6ca7200640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb0158bb0 0x7f6cb0146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cb5000640 1 -- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb0158bb0 msgr2=0x7f6cb0146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cb5000640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb0158bb0 0x7f6cb0146db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cb5000640 1 -- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb00afe60 msgr2=0x7f6cb0149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cb5000640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb00afe60 0x7f6cb0149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cb5000640 1 -- 172.21.15.39:0/4207746084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c980094d0 con 0x7f6cb015d500 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cafe00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb00afe60 0x7f6cb0149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cb5000640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb015d500 0x7f6cb00b2830 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800e930 tx=0x7f6ca800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cacc00640 1 -- 172.21.15.39:0/4207746084 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8009690 con 0x7f6cb015d500 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cacc00640 1 -- 172.21.15.39:0/4207746084 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca800c980 con 0x7f6cb015d500 2024-01-24T17:46:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cacc00640 1 -- 172.21.15.39:0/4207746084 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca800cd30 con 0x7f6cb015d500 2024-01-24T17:46:35.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.316+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb00b22b0 con 0x7f6cb015d500 2024-01-24T17:46:35.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.317+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb01465b0 con 0x7f6cb015d500 2024-01-24T17:46:35.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.318+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb0146a90 con 0x7f6cb015d500 2024-01-24T17:46:35.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.319+0000 7f6cacc00640 1 -- 172.21.15.39:0/4207746084 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6ca8013470 con 0x7f6cb015d500 2024-01-24T17:46:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.319+0000 7f6cacc00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6c8806a460 0x7f6c8806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.320+0000 7f6cacc00640 1 -- 172.21.15.39:0/4207746084 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ca8088a90 con 0x7f6cb015d500 2024-01-24T17:46:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.320+0000 7f6cafe00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6c8806a460 0x7f6c8806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.321+0000 7f6cafe00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6c8806a460 0x7f6c8806c920 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0001330 tx=0x7f6ca0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:35.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.324+0000 7f6cacc00640 1 -- 172.21.15.39:0/4207746084 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6ca80541d0 con 0x7f6cb015d500 2024-01-24T17:46:35.490 INFO:tasks.ceph.mds.d.smithi100.stderr:2024-01-24T17:46:35.488+0000 7faf32c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:46:35.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.518+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cb004dff0 con 0x7f6cb015d500 2024-01-24T17:46:35.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.521+0000 7f6cacc00640 1 -- 172.21.15.39:0/4207746084 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16040 (secure 0 0 0) 0x7f6ca8017090 con 0x7f6cb015d500 2024-01-24T17:46:35.522 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:35.522 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":76,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:32.019249+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:35.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.523+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6c8806a460 msgr2=0x7f6c8806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.523+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6c8806a460 0x7f6c8806c920 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0001330 tx=0x7f6ca0009040 comp rx=0 tx=0).stop 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.523+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb015d500 msgr2=0x7f6cb00b2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.523+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb015d500 0x7f6cb00b2830 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800e930 tx=0x7f6ca800edf0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.524+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 shutdown_connections 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.524+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6c8806a460 0x7f6c8806c920 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.524+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cb015d500 0x7f6cb00b2830 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.524+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6cb0158bb0 0x7f6cb0146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.524+0000 7f6cb5a00640 1 --2- 172.21.15.39:0/4207746084 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6cb00afe60 0x7f6cb0149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.524+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 >> 172.21.15.39:0/4207746084 conn(0x7f6cb006a340 msgr2=0x7f6cb00ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.524+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 shutdown_connections 2024-01-24T17:46:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.524+0000 7f6cb5a00640 1 -- 172.21.15.39:0/4207746084 wait complete. 2024-01-24T17:46:35.545 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-24T17:46:35.545 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:46:35.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.769+0000 7f2f6b000640 1 Processor -- start 2024-01-24T17:46:35.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.769+0000 7f2f6b000640 1 -- start start 2024-01-24T17:46:35.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.770+0000 7f2f6b000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2f64070510 0x7f2f640c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.770+0000 7f2f6b000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f640cc410 con 0x7f2f6415d500 2024-01-24T17:46:35.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.771+0000 7f2f6b000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f640c96f0 con 0x7f2f640cb570 2024-01-24T17:46:35.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.771+0000 7f2f6b000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f640709a0 con 0x7f2f64070510 2024-01-24T17:46:35.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.771+0000 7f2f69c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2f6415d500 0x7f2f64158bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60166/0 (socket says 172.21.15.39:60166) 2024-01-24T17:46:35.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.771+0000 7f2f69c00640 1 -- 172.21.15.39:0/3325611372 learned_addr learned my addr 172.21.15.39:0/3325611372 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.771+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247509980 0 0) 0x7f2f640cc410 con 0x7f2f6415d500 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.771+0000 7f2f69200640 1 --2- 172.21.15.39:0/3325611372 >> v2:172.21.15.100:3300/0 conn(0x7f2f64070510 0x7f2f640c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.771+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f44005730 con 0x7f2f6415d500 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261786938 0 0) 0x7f2f44005730 con 0x7f2f6415d500 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 >> v2:172.21.15.100:3300/0 conn(0x7f2f64070510 msgr2=0x7f2f640c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 --2- 172.21.15.39:0/3325611372 >> v2:172.21.15.100:3300/0 conn(0x7f2f64070510 0x7f2f640c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 >> v1:172.21.15.100:6789/0 conn(0x7f2f640cb570 legacy=0x7f2f640afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f640c99d0 con 0x7f2f6415d500 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2f540017f0 con 0x7f2f6415d500 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 >> v1:172.21.15.39:6789/0 conn(0x7f2f6415d500 legacy=0x7f2f64158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44005b80 0x7f2f44005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f440069d0 0x7f2f44006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4400e200 con 0x7f2f44007390 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4400e370 con 0x7f2f440069d0 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4400e530 con 0x7f2f44005b80 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.772+0000 7f2f6a600640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.773+0000 7f2f69c00640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44005b80 0x7f2f44005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.773+0000 7f2f69200640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f440069d0 0x7f2f44006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.773+0000 7f2f6a600640 1 -- 172.21.15.39:0/3325611372 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44005b80 msgr2=0x7f2f44005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.773+0000 7f2f6a600640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44005b80 0x7f2f44005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.773+0000 7f2f6a600640 1 -- 172.21.15.39:0/3325611372 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f440069d0 msgr2=0x7f2f44006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.773+0000 7f2f6a600640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f440069d0 0x7f2f44006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.773+0000 7f2f6a600640 1 -- 172.21.15.39:0/3325611372 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f44005730 con 0x7f2f44007390 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.774+0000 7f2f6a600640 1 -- 172.21.15.39:0/3325611372 reap_dead start 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.774+0000 7f2f6a600640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c001e50 tx=0x7f2f44007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c55718069c9fc8c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.774+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c003b70 con 0x7f2f44007390 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.774+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c003d10 con 0x7f2f44007390 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.774+0000 7f2f63e00640 1 -- 172.21.15.39:0/3325611372 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c005310 con 0x7f2f44007390 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.775+0000 7f2f6b000640 1 -- 172.21.15.39:0/3325611372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f44007390 msgr2=0x7f2f4400db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.775+0000 7f2f6b000640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c001e50 tx=0x7f2f44007a00 comp rx=0 tx=0).stop 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.775+0000 7f2f6b000640 1 -- 172.21.15.39:0/3325611372 shutdown_connections 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.775+0000 7f2f6b000640 1 --2- 172.21.15.39:0/3325611372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.775+0000 7f2f6b000640 1 -- 172.21.15.39:0/3325611372 >> 172.21.15.39:0/3325611372 conn(0x7f2f6406a340 msgr2=0x7f2f640cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:35.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.775+0000 7f2f6b000640 1 -- 172.21.15.39:0/3325611372 shutdown_connections 2024-01-24T17:46:35.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.775+0000 7f2f6b000640 1 -- 172.21.15.39:0/3325611372 wait complete. 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.777+0000 7f2f6b000640 1 Processor -- start 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.777+0000 7f2f6b000640 1 -- start start 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f6b000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44007390 0x7f2f641483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f6b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f641561a0 0x7f2f64156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f6b000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f6415b2a0 0x7f2f6415b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f6b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f640c1f10 con 0x7f2f641561a0 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f6b000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64159e50 con 0x7f2f6415b2a0 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f6b000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64156b60 con 0x7f2f44007390 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f69200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f641561a0 0x7f2f64156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f69200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f641561a0 0x7f2f64156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47256/0 (socket says 172.21.15.39:47256) 2024-01-24T17:46:35.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f69200640 1 -- 172.21.15.39:0/4131474994 learned_addr learned my addr 172.21.15.39:0/4131474994 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.778+0000 7f2f69c00640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44007390 0x7f2f641483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f6a600640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f6415b2a0 0x7f2f6415b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f69200640 1 -- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44007390 msgr2=0x7f2f641483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f69200640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44007390 0x7f2f641483d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f69200640 1 -- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f6415b2a0 msgr2=0x7f2f6415b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f69200640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f6415b2a0 0x7f2f6415b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f69200640 1 -- 172.21.15.39:0/4131474994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6415c090 con 0x7f2f641561a0 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f69200640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f641561a0 0x7f2f64156620 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0115a0 tx=0x7f2f4c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f61600640 1 -- 172.21.15.39:0/4131474994 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c015930 con 0x7f2f641561a0 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f61600640 1 -- 172.21.15.39:0/4131474994 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c015a90 con 0x7f2f641561a0 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f61600640 1 -- 172.21.15.39:0/4131474994 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c015e20 con 0x7f2f641561a0 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f640c2120 con 0x7f2f641561a0 2024-01-24T17:46:35.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.779+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f640c26e0 con 0x7f2f641561a0 2024-01-24T17:46:35.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.781+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f640c2870 con 0x7f2f641561a0 2024-01-24T17:46:35.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.781+0000 7f2f61600640 1 -- 172.21.15.39:0/4131474994 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2f4c00f480 con 0x7f2f641561a0 2024-01-24T17:46:35.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.782+0000 7f2f61600640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f3c06a430 0x7f2f3c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:35.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.782+0000 7f2f61600640 1 -- 172.21.15.39:0/4131474994 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f4c088e40 con 0x7f2f641561a0 2024-01-24T17:46:35.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.783+0000 7f2f69c00640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f3c06a430 0x7f2f3c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:35.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.784+0000 7f2f69c00640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f3c06a430 0x7f2f3c06c8f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2f540011d0 tx=0x7f2f54009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:35.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.786+0000 7f2f61600640 1 -- 172.21.15.39:0/4131474994 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2f4c054430 con 0x7f2f641561a0 2024-01-24T17:46:35.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.975+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f640c9f90 con 0x7f2f3c06a430 2024-01-24T17:46:36.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:35.999+0000 7f2f61600640 1 -- 172.21.15.39:0/4131474994 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287187 (secure 0 0 0) 0x7f2f54006d60 con 0x7f2f3c06a430 2024-01-24T17:46:36.004 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:36.006 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f3c06a430 msgr2=0x7f2f3c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f3c06a430 0x7f2f3c06c8f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2f540011d0 tx=0x7f2f54009040 comp rx=0 tx=0).stop 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f641561a0 msgr2=0x7f2f64156620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f641561a0 0x7f2f64156620 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0115a0 tx=0x7f2f4c011a60 comp rx=0 tx=0).stop 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 shutdown_connections 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f6415b2a0 0x7f2f6415b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f3c06a430 0x7f2f3c06c8f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f641561a0 0x7f2f64156620 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 --2- 172.21.15.39:0/4131474994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f44007390 0x7f2f641483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 >> 172.21.15.39:0/4131474994 conn(0x7f2f6406a340 msgr2=0x7f2f64152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:36.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 shutdown_connections 2024-01-24T17:46:36.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.005+0000 7f2f6b000640 1 -- 172.21.15.39:0/4131474994 wait complete. 2024-01-24T17:46:36.025 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-01-24T17:46:34.319112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257364,"kb_used_data":8384,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754717356,"statfs":{"total":773094113280,"available":772830572544,"internally_reserved":0,"allocated":8585216,"data_stored":4311700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.802959"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045082+0000","last_change":"2024-01-24T17:46:31.054672+0000","last_active":"2024-01-24T17:46:32.045082+0000","last_peered":"2024-01-24T17:46:32.045082+0000","last_clean":"2024-01-24T17:46:32.045082+0000","last_became_active":"2024-01-24T17:46:31.054438+0000","last_became_peered":"2024-01-24T17:46:31.054438+0000","last_unstale":"2024-01-24T17:46:32.045082+0000","last_undegraded":"2024-01-24T17:46:32.045082+0000","last_fullsized":"2024-01-24T17:46:32.045082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:24:27.292422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042806+0000","last_change":"2024-01-24T17:46:31.061225+0000","last_active":"2024-01-24T17:46:32.042806+0000","last_peered":"2024-01-24T17:46:32.042806+0000","last_clean":"2024-01-24T17:46:32.042806+0000","last_became_active":"2024-01-24T17:46:31.060468+0000","last_became_peered":"2024-01-24T17:46:31.060468+0000","last_unstale":"2024-01-24T17:46:32.042806+0000","last_undegraded":"2024-01-24T17:46:32.042806+0000","last_fullsized":"2024-01-24T17:46:32.042806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:00:52.884562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042381+0000","last_change":"2024-01-24T17:46:31.058896+0000","last_active":"2024-01-24T17:46:32.042381+0000","last_peered":"2024-01-24T17:46:32.042381+0000","last_clean":"2024-01-24T17:46:32.042381+0000","last_became_active":"2024-01-24T17:46:31.058603+0000","last_became_peered":"2024-01-24T17:46:31.058603+0000","last_unstale":"2024-01-24T17:46:32.042381+0000","last_undegraded":"2024-01-24T17:46:32.042381+0000","last_fullsized":"2024-01-24T17:46:32.042381+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:58:33.404255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042742+0000","last_change":"2024-01-24T17:46:31.055283+0000","last_active":"2024-01-24T17:46:32.042742+0000","last_peered":"2024-01-24T17:46:32.042742+0000","last_clean":"2024-01-24T17:46:32.042742+0000","last_became_active":"2024-01-24T17:46:31.054935+0000","last_became_peered":"2024-01-24T17:46:31.054935+0000","last_unstale":"2024-01-24T17:46:32.042742+0000","last_undegraded":"2024-01-24T17:46:32.042742+0000","last_fullsized":"2024-01-24T17:46:32.042742+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:54:20.923764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036460+0000","last_change":"2024-01-24T17:46:31.050700+0000","last_active":"2024-01-24T17:46:32.036460+0000","last_peered":"2024-01-24T17:46:32.036460+0000","last_clean":"2024-01-24T17:46:32.036460+0000","last_became_active":"2024-01-24T17:46:31.050144+0000","last_became_peered":"2024-01-24T17:46:31.050144+0000","last_unstale":"2024-01-24T17:46:32.036460+0000","last_undegraded":"2024-01-24T17:46:32.036460+0000","last_fullsized":"2024-01-24T17:46:32.036460+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:35:11.783864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044887+0000","last_change":"2024-01-24T17:46:31.056008+0000","last_active":"2024-01-24T17:46:32.044887+0000","last_peered":"2024-01-24T17:46:32.044887+0000","last_clean":"2024-01-24T17:46:32.044887+0000","last_became_active":"2024-01-24T17:46:31.055681+0000","last_became_peered":"2024-01-24T17:46:31.055681+0000","last_unstale":"2024-01-24T17:46:32.044887+0000","last_undegraded":"2024-01-24T17:46:32.044887+0000","last_fullsized":"2024-01-24T17:46:32.044887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:03:20.262992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034983+0000","last_change":"2024-01-24T17:46:31.051289+0000","last_active":"2024-01-24T17:46:32.034983+0000","last_peered":"2024-01-24T17:46:32.034983+0000","last_clean":"2024-01-24T17:46:32.034983+0000","last_became_active":"2024-01-24T17:46:31.051035+0000","last_became_peered":"2024-01-24T17:46:31.051035+0000","last_unstale":"2024-01-24T17:46:32.034983+0000","last_undegraded":"2024-01-24T17:46:32.034983+0000","last_fullsized":"2024-01-24T17:46:32.034983+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:36:13.230601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041352+0000","last_change":"2024-01-24T17:46:31.056745+0000","last_active":"2024-01-24T17:46:32.041352+0000","last_peered":"2024-01-24T17:46:32.041352+0000","last_clean":"2024-01-24T17:46:32.041352+0000","last_became_active":"2024-01-24T17:46:31.056492+0000","last_became_peered":"2024-01-24T17:46:31.056492+0000","last_unstale":"2024-01-24T17:46:32.041352+0000","last_undegraded":"2024-01-24T17:46:32.041352+0000","last_fullsized":"2024-01-24T17:46:32.041352+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:01:00.817919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043032+0000","last_change":"2024-01-24T17:46:31.041542+0000","last_active":"2024-01-24T17:46:32.043032+0000","last_peered":"2024-01-24T17:46:32.043032+0000","last_clean":"2024-01-24T17:46:32.043032+0000","last_became_active":"2024-01-24T17:46:31.040890+0000","last_became_peered":"2024-01-24T17:46:31.040890+0000","last_unstale":"2024-01-24T17:46:32.043032+0000","last_undegraded":"2024-01-24T17:46:32.043032+0000","last_fullsized":"2024-01-24T17:46:32.043032+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:27:39.874755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035731+0000","last_change":"2024-01-24T17:46:31.057978+0000","last_active":"2024-01-24T17:46:32.035731+0000","last_peered":"2024-01-24T17:46:32.035731+0000","last_clean":"2024-01-24T17:46:32.035731+0000","last_became_active":"2024-01-24T17:46:31.057652+0000","last_became_peered":"2024-01-24T17:46:31.057652+0000","last_unstale":"2024-01-24T17:46:32.035731+0000","last_undegraded":"2024-01-24T17:46:32.035731+0000","last_fullsized":"2024-01-24T17:46:32.035731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:09:00.744685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044222+0000","last_change":"2024-01-24T17:46:31.058608+0000","last_active":"2024-01-24T17:46:32.044222+0000","last_peered":"2024-01-24T17:46:32.044222+0000","last_clean":"2024-01-24T17:46:32.044222+0000","last_became_active":"2024-01-24T17:46:31.058060+0000","last_became_peered":"2024-01-24T17:46:31.058060+0000","last_unstale":"2024-01-24T17:46:32.044222+0000","last_undegraded":"2024-01-24T17:46:32.044222+0000","last_fullsized":"2024-01-24T17:46:32.044222+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:20:20.136491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044806+0000","last_change":"2024-01-24T17:46:31.046406+0000","last_active":"2024-01-24T17:46:32.044806+0000","last_peered":"2024-01-24T17:46:32.044806+0000","last_clean":"2024-01-24T17:46:32.044806+0000","last_became_active":"2024-01-24T17:46:31.041915+0000","last_became_peered":"2024-01-24T17:46:31.041915+0000","last_unstale":"2024-01-24T17:46:32.044806+0000","last_undegraded":"2024-01-24T17:46:32.044806+0000","last_fullsized":"2024-01-24T17:46:32.044806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:00:16.571792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042629+0000","last_change":"2024-01-24T17:46:31.059008+0000","last_active":"2024-01-24T17:46:32.042629+0000","last_peered":"2024-01-24T17:46:32.042629+0000","last_clean":"2024-01-24T17:46:32.042629+0000","last_became_active":"2024-01-24T17:46:31.058686+0000","last_became_peered":"2024-01-24T17:46:31.058686+0000","last_unstale":"2024-01-24T17:46:32.042629+0000","last_undegraded":"2024-01-24T17:46:32.042629+0000","last_fullsized":"2024-01-24T17:46:32.042629+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:06:14.187190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044594+0000","last_change":"2024-01-24T17:46:31.065860+0000","last_active":"2024-01-24T17:46:32.044594+0000","last_peered":"2024-01-24T17:46:32.044594+0000","last_clean":"2024-01-24T17:46:32.044594+0000","last_became_active":"2024-01-24T17:46:31.065525+0000","last_became_peered":"2024-01-24T17:46:31.065525+0000","last_unstale":"2024-01-24T17:46:32.044594+0000","last_undegraded":"2024-01-24T17:46:32.044594+0000","last_fullsized":"2024-01-24T17:46:32.044594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:55.521949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044314+0000","last_change":"2024-01-24T17:46:31.038167+0000","last_active":"2024-01-24T17:46:32.044314+0000","last_peered":"2024-01-24T17:46:32.044314+0000","last_clean":"2024-01-24T17:46:32.044314+0000","last_became_active":"2024-01-24T17:46:31.037942+0000","last_became_peered":"2024-01-24T17:46:31.037942+0000","last_unstale":"2024-01-24T17:46:32.044314+0000","last_undegraded":"2024-01-24T17:46:32.044314+0000","last_fullsized":"2024-01-24T17:46:32.044314+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:06:14.187190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041437+0000","last_change":"2024-01-24T17:46:31.061327+0000","last_active":"2024-01-24T17:46:32.041437+0000","last_peered":"2024-01-24T17:46:32.041437+0000","last_clean":"2024-01-24T17:46:32.041437+0000","last_became_active":"2024-01-24T17:46:31.061074+0000","last_became_peered":"2024-01-24T17:46:31.061074+0000","last_unstale":"2024-01-24T17:46:32.041437+0000","last_undegraded":"2024-01-24T17:46:32.041437+0000","last_fullsized":"2024-01-24T17:46:32.041437+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:48:58.770452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.037177+0000","last_change":"2024-01-24T17:46:31.045559+0000","last_active":"2024-01-24T17:46:32.037177+0000","last_peered":"2024-01-24T17:46:32.037177+0000","last_clean":"2024-01-24T17:46:32.037177+0000","last_became_active":"2024-01-24T17:46:31.045140+0000","last_became_peered":"2024-01-24T17:46:31.045140+0000","last_unstale":"2024-01-24T17:46:32.037177+0000","last_undegraded":"2024-01-24T17:46:32.037177+0000","last_fullsized":"2024-01-24T17:46:32.037177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:04:19.310970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.037121+0000","last_change":"2024-01-24T17:46:31.060194+0000","last_active":"2024-01-24T17:46:32.037121+0000","last_peered":"2024-01-24T17:46:32.037121+0000","last_clean":"2024-01-24T17:46:32.037121+0000","last_became_active":"2024-01-24T17:46:31.059896+0000","last_became_peered":"2024-01-24T17:46:31.059896+0000","last_unstale":"2024-01-24T17:46:32.037121+0000","last_undegraded":"2024-01-24T17:46:32.037121+0000","last_fullsized":"2024-01-24T17:46:32.037121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:33:53.027586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043745+0000","last_change":"2024-01-24T17:46:31.058768+0000","last_active":"2024-01-24T17:46:32.043745+0000","last_peered":"2024-01-24T17:46:32.043745+0000","last_clean":"2024-01-24T17:46:32.043745+0000","last_became_active":"2024-01-24T17:46:31.058442+0000","last_became_peered":"2024-01-24T17:46:31.058442+0000","last_unstale":"2024-01-24T17:46:32.043745+0000","last_undegraded":"2024-01-24T17:46:32.043745+0000","last_fullsized":"2024-01-24T17:46:32.043745+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:33:22.421274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036665+0000","last_change":"2024-01-24T17:46:31.049908+0000","last_active":"2024-01-24T17:46:32.036665+0000","last_peered":"2024-01-24T17:46:32.036665+0000","last_clean":"2024-01-24T17:46:32.036665+0000","last_became_active":"2024-01-24T17:46:31.048699+0000","last_became_peered":"2024-01-24T17:46:31.048699+0000","last_unstale":"2024-01-24T17:46:32.036665+0000","last_undegraded":"2024-01-24T17:46:32.036665+0000","last_fullsized":"2024-01-24T17:46:32.036665+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:05:44.318085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042248+0000","last_change":"2024-01-24T17:46:31.045192+0000","last_active":"2024-01-24T17:46:32.042248+0000","last_peered":"2024-01-24T17:46:32.042248+0000","last_clean":"2024-01-24T17:46:32.042248+0000","last_became_active":"2024-01-24T17:46:31.043802+0000","last_became_peered":"2024-01-24T17:46:31.043802+0000","last_unstale":"2024-01-24T17:46:32.042248+0000","last_undegraded":"2024-01-24T17:46:32.042248+0000","last_fullsized":"2024-01-24T17:46:32.042248+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:21:46.987232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045080+0000","last_change":"2024-01-24T17:46:31.040029+0000","last_active":"2024-01-24T17:46:32.045080+0000","last_peered":"2024-01-24T17:46:32.045080+0000","last_clean":"2024-01-24T17:46:32.045080+0000","last_became_active":"2024-01-24T17:46:31.039725+0000","last_became_peered":"2024-01-24T17:46:31.039725+0000","last_unstale":"2024-01-24T17:46:32.045080+0000","last_undegraded":"2024-01-24T17:46:32.045080+0000","last_fullsized":"2024-01-24T17:46:32.045080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:30:54.165656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034990+0000","last_change":"2024-01-24T17:46:31.065042+0000","last_active":"2024-01-24T17:46:32.034990+0000","last_peered":"2024-01-24T17:46:32.034990+0000","last_clean":"2024-01-24T17:46:32.034990+0000","last_became_active":"2024-01-24T17:46:31.062877+0000","last_became_peered":"2024-01-24T17:46:31.062877+0000","last_unstale":"2024-01-24T17:46:32.034990+0000","last_undegraded":"2024-01-24T17:46:32.034990+0000","last_fullsized":"2024-01-24T17:46:32.034990+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:49.085977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041530+0000","last_change":"2024-01-24T17:46:31.051293+0000","last_active":"2024-01-24T17:46:32.041530+0000","last_peered":"2024-01-24T17:46:32.041530+0000","last_clean":"2024-01-24T17:46:32.041530+0000","last_became_active":"2024-01-24T17:46:31.050040+0000","last_became_peered":"2024-01-24T17:46:31.050040+0000","last_unstale":"2024-01-24T17:46:32.041530+0000","last_undegraded":"2024-01-24T17:46:32.041530+0000","last_fullsized":"2024-01-24T17:46:32.041530+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:30:29.679524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035387+0000","last_change":"2024-01-24T17:46:31.059345+0000","last_active":"2024-01-24T17:46:32.035387+0000","last_peered":"2024-01-24T17:46:32.035387+0000","last_clean":"2024-01-24T17:46:32.035387+0000","last_became_active":"2024-01-24T17:46:31.059068+0000","last_became_peered":"2024-01-24T17:46:31.059068+0000","last_unstale":"2024-01-24T17:46:32.035387+0000","last_undegraded":"2024-01-24T17:46:32.035387+0000","last_fullsized":"2024-01-24T17:46:32.035387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:34.489869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044256+0000","last_change":"2024-01-24T17:46:31.062880+0000","last_active":"2024-01-24T17:46:32.044256+0000","last_peered":"2024-01-24T17:46:32.044256+0000","last_clean":"2024-01-24T17:46:32.044256+0000","last_became_active":"2024-01-24T17:46:31.062290+0000","last_became_peered":"2024-01-24T17:46:31.062290+0000","last_unstale":"2024-01-24T17:46:32.044256+0000","last_undegraded":"2024-01-24T17:46:32.044256+0000","last_fullsized":"2024-01-24T17:46:32.044256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:00:33.199885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036785+0000","last_change":"2024-01-24T17:46:31.051510+0000","last_active":"2024-01-24T17:46:32.036785+0000","last_peered":"2024-01-24T17:46:32.036785+0000","last_clean":"2024-01-24T17:46:32.036785+0000","last_became_active":"2024-01-24T17:46:31.051107+0000","last_became_peered":"2024-01-24T17:46:31.051107+0000","last_unstale":"2024-01-24T17:46:32.036785+0000","last_undegraded":"2024-01-24T17:46:32.036785+0000","last_fullsized":"2024-01-24T17:46:32.036785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:04:33.963910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042142+0000","last_change":"2024-01-24T17:46:31.039655+0000","last_active":"2024-01-24T17:46:32.042142+0000","last_peered":"2024-01-24T17:46:32.042142+0000","last_clean":"2024-01-24T17:46:32.042142+0000","last_became_active":"2024-01-24T17:46:31.038771+0000","last_became_peered":"2024-01-24T17:46:31.038771+0000","last_unstale":"2024-01-24T17:46:32.042142+0000","last_undegraded":"2024-01-24T17:46:32.042142+0000","last_fullsized":"2024-01-24T17:46:32.042142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:04.990796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042602+0000","last_change":"2024-01-24T17:46:31.053704+0000","last_active":"2024-01-24T17:46:32.042602+0000","last_peered":"2024-01-24T17:46:32.042602+0000","last_clean":"2024-01-24T17:46:32.042602+0000","last_became_active":"2024-01-24T17:46:31.053419+0000","last_became_peered":"2024-01-24T17:46:31.053419+0000","last_unstale":"2024-01-24T17:46:32.042602+0000","last_undegraded":"2024-01-24T17:46:32.042602+0000","last_fullsized":"2024-01-24T17:46:32.042602+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:08:28.844452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041642+0000","last_change":"2024-01-24T17:46:31.051391+0000","last_active":"2024-01-24T17:46:32.041642+0000","last_peered":"2024-01-24T17:46:32.041642+0000","last_clean":"2024-01-24T17:46:32.041642+0000","last_became_active":"2024-01-24T17:46:31.050828+0000","last_became_peered":"2024-01-24T17:46:31.050828+0000","last_unstale":"2024-01-24T17:46:32.041642+0000","last_undegraded":"2024-01-24T17:46:32.041642+0000","last_fullsized":"2024-01-24T17:46:32.041642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:17:22.948043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042466+0000","last_change":"2024-01-24T17:46:31.038145+0000","last_active":"2024-01-24T17:46:32.042466+0000","last_peered":"2024-01-24T17:46:32.042466+0000","last_clean":"2024-01-24T17:46:32.042466+0000","last_became_active":"2024-01-24T17:46:31.037788+0000","last_became_peered":"2024-01-24T17:46:31.037788+0000","last_unstale":"2024-01-24T17:46:32.042466+0000","last_undegraded":"2024-01-24T17:46:32.042466+0000","last_fullsized":"2024-01-24T17:46:32.042466+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:08:28.844452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042769+0000","last_change":"2024-01-24T17:46:31.040665+0000","last_active":"2024-01-24T17:46:32.042769+0000","last_peered":"2024-01-24T17:46:32.042769+0000","last_clean":"2024-01-24T17:46:32.042769+0000","last_became_active":"2024-01-24T17:46:31.040228+0000","last_became_peered":"2024-01-24T17:46:31.040228+0000","last_unstale":"2024-01-24T17:46:32.042769+0000","last_undegraded":"2024-01-24T17:46:32.042769+0000","last_fullsized":"2024-01-24T17:46:32.042769+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:54:20.923764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.046347+0000","last_change":"2024-01-24T17:46:31.063028+0000","last_active":"2024-01-24T17:46:32.046347+0000","last_peered":"2024-01-24T17:46:32.046347+0000","last_clean":"2024-01-24T17:46:32.046347+0000","last_became_active":"2024-01-24T17:46:31.062619+0000","last_became_peered":"2024-01-24T17:46:31.062619+0000","last_unstale":"2024-01-24T17:46:32.046347+0000","last_undegraded":"2024-01-24T17:46:32.046347+0000","last_fullsized":"2024-01-24T17:46:32.046347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:15:29.774800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"75'4","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045856+0000","last_change":"2024-01-24T17:46:30.568884+0000","last_active":"2024-01-24T17:46:32.045856+0000","last_peered":"2024-01-24T17:46:32.045856+0000","last_clean":"2024-01-24T17:46:32.045856+0000","last_became_active":"2024-01-24T17:46:30.566185+0000","last_became_peered":"2024-01-24T17:46:30.566185+0000","last_unstale":"2024-01-24T17:46:32.045856+0000","last_undegraded":"2024-01-24T17:46:32.045856+0000","last_fullsized":"2024-01-24T17:46:32.045856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:38:52.542242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043220+0000","last_change":"2024-01-24T17:46:31.042247+0000","last_active":"2024-01-24T17:46:32.043220+0000","last_peered":"2024-01-24T17:46:32.043220+0000","last_clean":"2024-01-24T17:46:32.043220+0000","last_became_active":"2024-01-24T17:46:31.041231+0000","last_became_peered":"2024-01-24T17:46:31.041231+0000","last_unstale":"2024-01-24T17:46:32.043220+0000","last_undegraded":"2024-01-24T17:46:32.043220+0000","last_fullsized":"2024-01-24T17:46:32.043220+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:04.990796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.037262+0000","last_change":"2024-01-24T17:46:30.560090+0000","last_active":"2024-01-24T17:46:32.037262+0000","last_peered":"2024-01-24T17:46:32.037262+0000","last_clean":"2024-01-24T17:46:32.037262+0000","last_became_active":"2024-01-24T17:46:30.559793+0000","last_became_peered":"2024-01-24T17:46:30.559793+0000","last_unstale":"2024-01-24T17:46:32.037262+0000","last_undegraded":"2024-01-24T17:46:32.037262+0000","last_fullsized":"2024-01-24T17:46:32.037262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:15:33.655294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045752+0000","last_change":"2024-01-24T17:46:31.060301+0000","last_active":"2024-01-24T17:46:32.045752+0000","last_peered":"2024-01-24T17:46:32.045752+0000","last_clean":"2024-01-24T17:46:32.045752+0000","last_became_active":"2024-01-24T17:46:31.059982+0000","last_became_peered":"2024-01-24T17:46:31.059982+0000","last_unstale":"2024-01-24T17:46:32.045752+0000","last_undegraded":"2024-01-24T17:46:32.045752+0000","last_fullsized":"2024-01-24T17:46:32.045752+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:02:26.610727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044472+0000","last_change":"2024-01-24T17:46:30.576753+0000","last_active":"2024-01-24T17:46:32.044472+0000","last_peered":"2024-01-24T17:46:32.044472+0000","last_clean":"2024-01-24T17:46:32.044472+0000","last_became_active":"2024-01-24T17:46:30.576134+0000","last_became_peered":"2024-01-24T17:46:30.576134+0000","last_unstale":"2024-01-24T17:46:32.044472+0000","last_undegraded":"2024-01-24T17:46:32.044472+0000","last_fullsized":"2024-01-24T17:46:32.044472+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:01:40.209070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035430+0000","last_change":"2024-01-24T17:46:31.064345+0000","last_active":"2024-01-24T17:46:32.035430+0000","last_peered":"2024-01-24T17:46:32.035430+0000","last_clean":"2024-01-24T17:46:32.035430+0000","last_became_active":"2024-01-24T17:46:31.064100+0000","last_became_peered":"2024-01-24T17:46:31.064100+0000","last_unstale":"2024-01-24T17:46:32.035430+0000","last_undegraded":"2024-01-24T17:46:32.035430+0000","last_fullsized":"2024-01-24T17:46:32.035430+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:42:47.498116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045852+0000","last_change":"2024-01-24T17:46:30.564784+0000","last_active":"2024-01-24T17:46:32.045852+0000","last_peered":"2024-01-24T17:46:32.045852+0000","last_clean":"2024-01-24T17:46:32.045852+0000","last_became_active":"2024-01-24T17:46:30.564501+0000","last_became_peered":"2024-01-24T17:46:30.564501+0000","last_unstale":"2024-01-24T17:46:32.045852+0000","last_undegraded":"2024-01-24T17:46:32.045852+0000","last_fullsized":"2024-01-24T17:46:32.045852+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:54:19.911485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041915+0000","last_change":"2024-01-24T17:46:31.052817+0000","last_active":"2024-01-24T17:46:32.041915+0000","last_peered":"2024-01-24T17:46:32.041915+0000","last_clean":"2024-01-24T17:46:32.041915+0000","last_became_active":"2024-01-24T17:46:31.052489+0000","last_became_peered":"2024-01-24T17:46:31.052489+0000","last_unstale":"2024-01-24T17:46:32.041915+0000","last_undegraded":"2024-01-24T17:46:32.041915+0000","last_fullsized":"2024-01-24T17:46:32.041915+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:20:02.096010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044947+0000","last_change":"2024-01-24T17:46:30.570372+0000","last_active":"2024-01-24T17:46:32.044947+0000","last_peered":"2024-01-24T17:46:32.044947+0000","last_clean":"2024-01-24T17:46:32.044947+0000","last_became_active":"2024-01-24T17:46:30.569997+0000","last_became_peered":"2024-01-24T17:46:30.569997+0000","last_unstale":"2024-01-24T17:46:32.044947+0000","last_undegraded":"2024-01-24T17:46:32.044947+0000","last_fullsized":"2024-01-24T17:46:32.044947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:33.477589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045216+0000","last_change":"2024-01-24T17:46:31.055380+0000","last_active":"2024-01-24T17:46:32.045216+0000","last_peered":"2024-01-24T17:46:32.045216+0000","last_clean":"2024-01-24T17:46:32.045216+0000","last_became_active":"2024-01-24T17:46:31.053319+0000","last_became_peered":"2024-01-24T17:46:31.053319+0000","last_unstale":"2024-01-24T17:46:32.045216+0000","last_undegraded":"2024-01-24T17:46:32.045216+0000","last_fullsized":"2024-01-24T17:46:32.045216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:52:12.764326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034763+0000","last_change":"2024-01-24T17:46:30.574055+0000","last_active":"2024-01-24T17:46:32.034763+0000","last_peered":"2024-01-24T17:46:32.034763+0000","last_clean":"2024-01-24T17:46:32.034763+0000","last_became_active":"2024-01-24T17:46:30.573748+0000","last_became_peered":"2024-01-24T17:46:30.573748+0000","last_unstale":"2024-01-24T17:46:32.034763+0000","last_undegraded":"2024-01-24T17:46:32.034763+0000","last_fullsized":"2024-01-24T17:46:32.034763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:37:54.618595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.033810+0000","last_change":"2024-01-24T17:46:31.055136+0000","last_active":"2024-01-24T17:46:32.033810+0000","last_peered":"2024-01-24T17:46:32.033810+0000","last_clean":"2024-01-24T17:46:32.033810+0000","last_became_active":"2024-01-24T17:46:31.054903+0000","last_became_peered":"2024-01-24T17:46:31.054903+0000","last_unstale":"2024-01-24T17:46:32.033810+0000","last_undegraded":"2024-01-24T17:46:32.033810+0000","last_fullsized":"2024-01-24T17:46:32.033810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:38:53.950426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.033784+0000","last_change":"2024-01-24T17:46:30.570178+0000","last_active":"2024-01-24T17:46:32.033784+0000","last_peered":"2024-01-24T17:46:32.033784+0000","last_clean":"2024-01-24T17:46:32.033784+0000","last_became_active":"2024-01-24T17:46:30.569930+0000","last_became_peered":"2024-01-24T17:46:30.569930+0000","last_unstale":"2024-01-24T17:46:32.033784+0000","last_undegraded":"2024-01-24T17:46:32.033784+0000","last_fullsized":"2024-01-24T17:46:32.033784+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:42:46.485837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044515+0000","last_change":"2024-01-24T17:46:31.052827+0000","last_active":"2024-01-24T17:46:32.044515+0000","last_peered":"2024-01-24T17:46:32.044515+0000","last_clean":"2024-01-24T17:46:32.044515+0000","last_became_active":"2024-01-24T17:46:31.052524+0000","last_became_peered":"2024-01-24T17:46:31.052524+0000","last_unstale":"2024-01-24T17:46:32.044515+0000","last_undegraded":"2024-01-24T17:46:32.044515+0000","last_fullsized":"2024-01-24T17:46:32.044515+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:27:39.874755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044601+0000","last_change":"2024-01-24T17:46:30.559701+0000","last_active":"2024-01-24T17:46:32.044601+0000","last_peered":"2024-01-24T17:46:32.044601+0000","last_clean":"2024-01-24T17:46:32.044601+0000","last_became_active":"2024-01-24T17:46:30.559366+0000","last_became_peered":"2024-01-24T17:46:30.559366+0000","last_unstale":"2024-01-24T17:46:32.044601+0000","last_undegraded":"2024-01-24T17:46:32.044601+0000","last_fullsized":"2024-01-24T17:46:32.044601+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:42:46.485837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035770+0000","last_change":"2024-01-24T17:46:31.066489+0000","last_active":"2024-01-24T17:46:32.035770+0000","last_peered":"2024-01-24T17:46:32.035770+0000","last_clean":"2024-01-24T17:46:32.035770+0000","last_became_active":"2024-01-24T17:46:31.066270+0000","last_became_peered":"2024-01-24T17:46:31.066270+0000","last_unstale":"2024-01-24T17:46:32.035770+0000","last_undegraded":"2024-01-24T17:46:32.035770+0000","last_fullsized":"2024-01-24T17:46:32.035770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:03:04.365856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044255+0000","last_change":"2024-01-24T17:46:30.571903+0000","last_active":"2024-01-24T17:46:32.044255+0000","last_peered":"2024-01-24T17:46:32.044255+0000","last_clean":"2024-01-24T17:46:32.044255+0000","last_became_active":"2024-01-24T17:46:30.571610+0000","last_became_peered":"2024-01-24T17:46:30.571610+0000","last_unstale":"2024-01-24T17:46:32.044255+0000","last_undegraded":"2024-01-24T17:46:32.044255+0000","last_fullsized":"2024-01-24T17:46:32.044255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:47:45.575561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043219+0000","last_change":"2024-01-24T17:46:31.059190+0000","last_active":"2024-01-24T17:46:32.043219+0000","last_peered":"2024-01-24T17:46:32.043219+0000","last_clean":"2024-01-24T17:46:32.043219+0000","last_became_active":"2024-01-24T17:46:31.058877+0000","last_became_peered":"2024-01-24T17:46:31.058877+0000","last_unstale":"2024-01-24T17:46:32.043219+0000","last_undegraded":"2024-01-24T17:46:32.043219+0000","last_fullsized":"2024-01-24T17:46:32.043219+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:15:34.667574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043105+0000","last_change":"2024-01-24T17:46:30.572944+0000","last_active":"2024-01-24T17:46:32.043105+0000","last_peered":"2024-01-24T17:46:32.043105+0000","last_clean":"2024-01-24T17:46:32.043105+0000","last_became_active":"2024-01-24T17:46:30.572756+0000","last_became_peered":"2024-01-24T17:46:30.572756+0000","last_unstale":"2024-01-24T17:46:32.043105+0000","last_undegraded":"2024-01-24T17:46:32.043105+0000","last_fullsized":"2024-01-24T17:46:32.043105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:33.477589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043762+0000","last_change":"2024-01-24T17:46:31.060960+0000","last_active":"2024-01-24T17:46:32.043762+0000","last_peered":"2024-01-24T17:46:32.043762+0000","last_clean":"2024-01-24T17:46:32.043762+0000","last_became_active":"2024-01-24T17:46:31.060665+0000","last_became_peered":"2024-01-24T17:46:31.060665+0000","last_unstale":"2024-01-24T17:46:32.043762+0000","last_undegraded":"2024-01-24T17:46:32.043762+0000","last_fullsized":"2024-01-24T17:46:32.043762+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:16:56.913832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"75'3","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045078+0000","last_change":"2024-01-24T17:46:30.559432+0000","last_active":"2024-01-24T17:46:32.045078+0000","last_peered":"2024-01-24T17:46:32.045078+0000","last_clean":"2024-01-24T17:46:32.045078+0000","last_became_active":"2024-01-24T17:46:30.559078+0000","last_became_peered":"2024-01-24T17:46:30.559078+0000","last_unstale":"2024-01-24T17:46:32.045078+0000","last_undegraded":"2024-01-24T17:46:32.045078+0000","last_fullsized":"2024-01-24T17:46:32.045078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:36:12.218321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036878+0000","last_change":"2024-01-24T17:46:31.045618+0000","last_active":"2024-01-24T17:46:32.036878+0000","last_peered":"2024-01-24T17:46:32.036878+0000","last_clean":"2024-01-24T17:46:32.036878+0000","last_became_active":"2024-01-24T17:46:31.045258+0000","last_became_peered":"2024-01-24T17:46:31.045258+0000","last_unstale":"2024-01-24T17:46:32.036878+0000","last_undegraded":"2024-01-24T17:46:32.036878+0000","last_fullsized":"2024-01-24T17:46:32.036878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:47:46.587841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"75'6","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034537+0000","last_change":"2024-01-24T17:46:30.562474+0000","last_active":"2024-01-24T17:46:32.034537+0000","last_peered":"2024-01-24T17:46:32.034537+0000","last_clean":"2024-01-24T17:46:32.034537+0000","last_became_active":"2024-01-24T17:46:30.561846+0000","last_became_peered":"2024-01-24T17:46:30.561846+0000","last_unstale":"2024-01-24T17:46:32.034537+0000","last_undegraded":"2024-01-24T17:46:32.034537+0000","last_fullsized":"2024-01-24T17:46:32.034537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:48.073697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045174+0000","last_change":"2024-01-24T17:46:31.045974+0000","last_active":"2024-01-24T17:46:32.045174+0000","last_peered":"2024-01-24T17:46:32.045174+0000","last_clean":"2024-01-24T17:46:32.045174+0000","last_became_active":"2024-01-24T17:46:31.044058+0000","last_became_peered":"2024-01-24T17:46:31.044058+0000","last_unstale":"2024-01-24T17:46:32.045174+0000","last_undegraded":"2024-01-24T17:46:32.045174+0000","last_fullsized":"2024-01-24T17:46:32.045174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:59:37.246294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034195+0000","last_change":"2024-01-24T17:46:30.566856+0000","last_active":"2024-01-24T17:46:32.034195+0000","last_peered":"2024-01-24T17:46:32.034195+0000","last_clean":"2024-01-24T17:46:32.034195+0000","last_became_active":"2024-01-24T17:46:30.566472+0000","last_became_peered":"2024-01-24T17:46:30.566472+0000","last_unstale":"2024-01-24T17:46:32.034195+0000","last_undegraded":"2024-01-24T17:46:32.034195+0000","last_fullsized":"2024-01-24T17:46:32.034195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:54.139951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034982+0000","last_change":"2024-01-24T17:46:31.063647+0000","last_active":"2024-01-24T17:46:32.034982+0000","last_peered":"2024-01-24T17:46:32.034982+0000","last_clean":"2024-01-24T17:46:32.034982+0000","last_became_active":"2024-01-24T17:46:31.063249+0000","last_became_peered":"2024-01-24T17:46:31.063249+0000","last_unstale":"2024-01-24T17:46:32.034982+0000","last_undegraded":"2024-01-24T17:46:32.034982+0000","last_fullsized":"2024-01-24T17:46:32.034982+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:38:53.950426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034984+0000","last_change":"2024-01-24T17:46:30.564980+0000","last_active":"2024-01-24T17:46:32.034984+0000","last_peered":"2024-01-24T17:46:32.034984+0000","last_clean":"2024-01-24T17:46:32.034984+0000","last_became_active":"2024-01-24T17:46:30.562518+0000","last_became_peered":"2024-01-24T17:46:30.562518+0000","last_unstale":"2024-01-24T17:46:32.034984+0000","last_undegraded":"2024-01-24T17:46:32.034984+0000","last_fullsized":"2024-01-24T17:46:32.034984+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:37:27.535127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035866+0000","last_change":"2024-01-24T17:46:31.044670+0000","last_active":"2024-01-24T17:46:32.035866+0000","last_peered":"2024-01-24T17:46:32.035866+0000","last_clean":"2024-01-24T17:46:32.035866+0000","last_became_active":"2024-01-24T17:46:31.044361+0000","last_became_peered":"2024-01-24T17:46:31.044361+0000","last_unstale":"2024-01-24T17:46:32.035866+0000","last_undegraded":"2024-01-24T17:46:32.035866+0000","last_fullsized":"2024-01-24T17:46:32.035866+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:10:14.348792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035216+0000","last_change":"2024-01-24T17:46:30.559627+0000","last_active":"2024-01-24T17:46:32.035216+0000","last_peered":"2024-01-24T17:46:32.035216+0000","last_clean":"2024-01-24T17:46:32.035216+0000","last_became_active":"2024-01-24T17:46:30.559367+0000","last_became_peered":"2024-01-24T17:46:30.559367+0000","last_unstale":"2024-01-24T17:46:32.035216+0000","last_undegraded":"2024-01-24T17:46:32.035216+0000","last_fullsized":"2024-01-24T17:46:32.035216+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:01:56.871757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042193+0000","last_change":"2024-01-24T17:46:31.034211+0000","last_active":"2024-01-24T17:46:32.042193+0000","last_peered":"2024-01-24T17:46:32.042193+0000","last_clean":"2024-01-24T17:46:32.042193+0000","last_became_active":"2024-01-24T17:46:31.033908+0000","last_became_peered":"2024-01-24T17:46:31.033908+0000","last_unstale":"2024-01-24T17:46:32.042193+0000","last_undegraded":"2024-01-24T17:46:32.042193+0000","last_fullsized":"2024-01-24T17:46:32.042193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:38:53.554522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"75'8","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034656+0000","last_change":"2024-01-24T17:46:30.568532+0000","last_active":"2024-01-24T17:46:32.034656+0000","last_peered":"2024-01-24T17:46:32.034656+0000","last_clean":"2024-01-24T17:46:32.034656+0000","last_became_active":"2024-01-24T17:46:30.568219+0000","last_became_peered":"2024-01-24T17:46:30.568219+0000","last_unstale":"2024-01-24T17:46:32.034656+0000","last_undegraded":"2024-01-24T17:46:32.034656+0000","last_fullsized":"2024-01-24T17:46:32.034656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:38:48.719620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042836+0000","last_change":"2024-01-24T17:46:31.043376+0000","last_active":"2024-01-24T17:46:32.042836+0000","last_peered":"2024-01-24T17:46:32.042836+0000","last_clean":"2024-01-24T17:46:32.042836+0000","last_became_active":"2024-01-24T17:46:31.041172+0000","last_became_peered":"2024-01-24T17:46:31.041172+0000","last_unstale":"2024-01-24T17:46:32.042836+0000","last_undegraded":"2024-01-24T17:46:32.042836+0000","last_fullsized":"2024-01-24T17:46:32.042836+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:58:33.404255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044826+0000","last_change":"2024-01-24T17:46:30.570606+0000","last_active":"2024-01-24T17:46:32.044826+0000","last_peered":"2024-01-24T17:46:32.044826+0000","last_clean":"2024-01-24T17:46:32.044826+0000","last_became_active":"2024-01-24T17:46:30.570188+0000","last_became_peered":"2024-01-24T17:46:30.570188+0000","last_unstale":"2024-01-24T17:46:32.044826+0000","last_undegraded":"2024-01-24T17:46:32.044826+0000","last_fullsized":"2024-01-24T17:46:32.044826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:24:10.438941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043423+0000","last_change":"2024-01-24T17:46:31.044575+0000","last_active":"2024-01-24T17:46:32.043423+0000","last_peered":"2024-01-24T17:46:32.043423+0000","last_clean":"2024-01-24T17:46:32.043423+0000","last_became_active":"2024-01-24T17:46:31.044237+0000","last_became_peered":"2024-01-24T17:46:31.044237+0000","last_unstale":"2024-01-24T17:46:32.043423+0000","last_undegraded":"2024-01-24T17:46:32.043423+0000","last_fullsized":"2024-01-24T17:46:32.043423+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:38:53.554522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043135+0000","last_change":"2024-01-24T17:46:30.555050+0000","last_active":"2024-01-24T17:46:32.043135+0000","last_peered":"2024-01-24T17:46:32.043135+0000","last_clean":"2024-01-24T17:46:32.043135+0000","last_became_active":"2024-01-24T17:46:30.554773+0000","last_became_peered":"2024-01-24T17:46:30.554773+0000","last_unstale":"2024-01-24T17:46:32.043135+0000","last_undegraded":"2024-01-24T17:46:32.043135+0000","last_fullsized":"2024-01-24T17:46:32.043135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:54.139951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042558+0000","last_change":"2024-01-24T17:46:31.041732+0000","last_active":"2024-01-24T17:46:32.042558+0000","last_peered":"2024-01-24T17:46:32.042558+0000","last_clean":"2024-01-24T17:46:32.042558+0000","last_became_active":"2024-01-24T17:46:31.040392+0000","last_became_peered":"2024-01-24T17:46:31.040392+0000","last_unstale":"2024-01-24T17:46:32.042558+0000","last_undegraded":"2024-01-24T17:46:32.042558+0000","last_fullsized":"2024-01-24T17:46:32.042558+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:55.521949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042344+0000","last_change":"2024-01-24T17:46:30.575914+0000","last_active":"2024-01-24T17:46:32.042344+0000","last_peered":"2024-01-24T17:46:32.042344+0000","last_clean":"2024-01-24T17:46:32.042344+0000","last_became_active":"2024-01-24T17:46:30.575637+0000","last_became_peered":"2024-01-24T17:46:30.575637+0000","last_unstale":"2024-01-24T17:46:32.042344+0000","last_undegraded":"2024-01-24T17:46:32.042344+0000","last_fullsized":"2024-01-24T17:46:32.042344+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:56:43.348935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042551+0000","last_change":"2024-01-24T17:46:31.056695+0000","last_active":"2024-01-24T17:46:32.042551+0000","last_peered":"2024-01-24T17:46:32.042551+0000","last_clean":"2024-01-24T17:46:32.042551+0000","last_became_active":"2024-01-24T17:46:31.056432+0000","last_became_peered":"2024-01-24T17:46:31.056432+0000","last_unstale":"2024-01-24T17:46:32.042551+0000","last_undegraded":"2024-01-24T17:46:32.042551+0000","last_fullsized":"2024-01-24T17:46:32.042551+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:24:31.700236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.046005+0000","last_change":"2024-01-24T17:46:30.575906+0000","last_active":"2024-01-24T17:46:32.046005+0000","last_peered":"2024-01-24T17:46:32.046005+0000","last_clean":"2024-01-24T17:46:32.046005+0000","last_became_active":"2024-01-24T17:46:30.575532+0000","last_became_peered":"2024-01-24T17:46:30.575532+0000","last_unstale":"2024-01-24T17:46:32.046005+0000","last_undegraded":"2024-01-24T17:46:32.046005+0000","last_fullsized":"2024-01-24T17:46:32.046005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:58:32.391975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034063+0000","last_change":"2024-01-24T17:46:31.035497+0000","last_active":"2024-01-24T17:46:32.034063+0000","last_peered":"2024-01-24T17:46:32.034063+0000","last_clean":"2024-01-24T17:46:32.034063+0000","last_became_active":"2024-01-24T17:46:31.035159+0000","last_became_peered":"2024-01-24T17:46:31.035159+0000","last_unstale":"2024-01-24T17:46:32.034063+0000","last_undegraded":"2024-01-24T17:46:32.034063+0000","last_fullsized":"2024-01-24T17:46:32.034063+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:00:16.571792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045328+0000","last_change":"2024-01-24T17:46:30.566751+0000","last_active":"2024-01-24T17:46:32.045328+0000","last_peered":"2024-01-24T17:46:32.045328+0000","last_clean":"2024-01-24T17:46:32.045328+0000","last_became_active":"2024-01-24T17:46:30.566192+0000","last_became_peered":"2024-01-24T17:46:30.566192+0000","last_unstale":"2024-01-24T17:46:32.045328+0000","last_undegraded":"2024-01-24T17:46:32.045328+0000","last_fullsized":"2024-01-24T17:46:32.045328+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:06:13.174910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.033556+0000","last_change":"2024-01-24T17:46:31.039959+0000","last_active":"2024-01-24T17:46:32.033556+0000","last_peered":"2024-01-24T17:46:32.033556+0000","last_clean":"2024-01-24T17:46:32.033556+0000","last_became_active":"2024-01-24T17:46:31.038298+0000","last_became_peered":"2024-01-24T17:46:31.038298+0000","last_unstale":"2024-01-24T17:46:32.033556+0000","last_undegraded":"2024-01-24T17:46:32.033556+0000","last_fullsized":"2024-01-24T17:46:32.033556+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:56:12.091199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042768+0000","last_change":"2024-01-24T17:46:30.556835+0000","last_active":"2024-01-24T17:46:32.042768+0000","last_peered":"2024-01-24T17:46:32.042768+0000","last_clean":"2024-01-24T17:46:32.042768+0000","last_became_active":"2024-01-24T17:46:30.556561+0000","last_became_peered":"2024-01-24T17:46:30.556561+0000","last_unstale":"2024-01-24T17:46:32.042768+0000","last_undegraded":"2024-01-24T17:46:32.042768+0000","last_fullsized":"2024-01-24T17:46:32.042768+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:03:03.353576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044453+0000","last_change":"2024-01-24T17:46:31.040557+0000","last_active":"2024-01-24T17:46:32.044453+0000","last_peered":"2024-01-24T17:46:32.044453+0000","last_clean":"2024-01-24T17:46:32.044453+0000","last_became_active":"2024-01-24T17:46:31.040210+0000","last_became_peered":"2024-01-24T17:46:31.040210+0000","last_unstale":"2024-01-24T17:46:32.044453+0000","last_undegraded":"2024-01-24T17:46:32.044453+0000","last_fullsized":"2024-01-24T17:46:32.044453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:38:53.554522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035789+0000","last_change":"2024-01-24T17:46:30.554541+0000","last_active":"2024-01-24T17:46:32.035789+0000","last_peered":"2024-01-24T17:46:32.035789+0000","last_clean":"2024-01-24T17:46:32.035789+0000","last_became_active":"2024-01-24T17:46:30.553996+0000","last_became_peered":"2024-01-24T17:46:30.553996+0000","last_unstale":"2024-01-24T17:46:32.035789+0000","last_undegraded":"2024-01-24T17:46:32.035789+0000","last_fullsized":"2024-01-24T17:46:32.035789+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:21:45.974952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.037192+0000","last_change":"2024-01-24T17:46:31.046837+0000","last_active":"2024-01-24T17:46:32.037192+0000","last_peered":"2024-01-24T17:46:32.037192+0000","last_clean":"2024-01-24T17:46:32.037192+0000","last_became_active":"2024-01-24T17:46:31.046029+0000","last_became_peered":"2024-01-24T17:46:31.046029+0000","last_unstale":"2024-01-24T17:46:32.037192+0000","last_undegraded":"2024-01-24T17:46:32.037192+0000","last_fullsized":"2024-01-24T17:46:32.037192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:57.399522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045009+0000","last_change":"2024-01-24T17:46:30.570966+0000","last_active":"2024-01-24T17:46:32.045009+0000","last_peered":"2024-01-24T17:46:32.045009+0000","last_clean":"2024-01-24T17:46:32.045009+0000","last_became_active":"2024-01-24T17:46:30.570634+0000","last_became_peered":"2024-01-24T17:46:30.570634+0000","last_unstale":"2024-01-24T17:46:32.045009+0000","last_undegraded":"2024-01-24T17:46:32.045009+0000","last_fullsized":"2024-01-24T17:46:32.045009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:04:32.951630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035520+0000","last_change":"2024-01-24T17:46:31.053536+0000","last_active":"2024-01-24T17:46:32.035520+0000","last_peered":"2024-01-24T17:46:32.035520+0000","last_clean":"2024-01-24T17:46:32.035520+0000","last_became_active":"2024-01-24T17:46:31.051508+0000","last_became_peered":"2024-01-24T17:46:31.051508+0000","last_unstale":"2024-01-24T17:46:32.035520+0000","last_undegraded":"2024-01-24T17:46:32.035520+0000","last_fullsized":"2024-01-24T17:46:32.035520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:55.152231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044368+0000","last_change":"2024-01-24T17:46:30.567484+0000","last_active":"2024-01-24T17:46:32.044368+0000","last_peered":"2024-01-24T17:46:32.044368+0000","last_clean":"2024-01-24T17:46:32.044368+0000","last_became_active":"2024-01-24T17:46:30.567016+0000","last_became_peered":"2024-01-24T17:46:30.567016+0000","last_unstale":"2024-01-24T17:46:32.044368+0000","last_undegraded":"2024-01-24T17:46:32.044368+0000","last_fullsized":"2024-01-24T17:46:32.044368+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:50.235864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042412+0000","last_change":"2024-01-24T17:46:31.051429+0000","last_active":"2024-01-24T17:46:32.042412+0000","last_peered":"2024-01-24T17:46:32.042412+0000","last_clean":"2024-01-24T17:46:32.042412+0000","last_became_active":"2024-01-24T17:46:31.051159+0000","last_became_peered":"2024-01-24T17:46:31.051159+0000","last_unstale":"2024-01-24T17:46:32.042412+0000","last_undegraded":"2024-01-24T17:46:32.042412+0000","last_fullsized":"2024-01-24T17:46:32.042412+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:26:43.578208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045079+0000","last_change":"2024-01-24T17:46:30.553986+0000","last_active":"2024-01-24T17:46:32.045079+0000","last_peered":"2024-01-24T17:46:32.045079+0000","last_clean":"2024-01-24T17:46:32.045079+0000","last_became_active":"2024-01-24T17:46:30.553145+0000","last_became_peered":"2024-01-24T17:46:30.553145+0000","last_unstale":"2024-01-24T17:46:32.045079+0000","last_undegraded":"2024-01-24T17:46:32.045079+0000","last_fullsized":"2024-01-24T17:46:32.045079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:54.139951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035370+0000","last_change":"2024-01-24T17:46:31.057010+0000","last_active":"2024-01-24T17:46:32.035370+0000","last_peered":"2024-01-24T17:46:32.035370+0000","last_clean":"2024-01-24T17:46:32.035370+0000","last_became_active":"2024-01-24T17:46:31.055165+0000","last_became_peered":"2024-01-24T17:46:31.055165+0000","last_unstale":"2024-01-24T17:46:32.035370+0000","last_undegraded":"2024-01-24T17:46:32.035370+0000","last_fullsized":"2024-01-24T17:46:32.035370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:33:55.710838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044800+0000","last_change":"2024-01-24T17:46:30.559257+0000","last_active":"2024-01-24T17:46:32.044800+0000","last_peered":"2024-01-24T17:46:32.044800+0000","last_clean":"2024-01-24T17:46:32.044800+0000","last_became_active":"2024-01-24T17:46:30.558943+0000","last_became_peered":"2024-01-24T17:46:30.558943+0000","last_unstale":"2024-01-24T17:46:32.044800+0000","last_undegraded":"2024-01-24T17:46:32.044800+0000","last_fullsized":"2024-01-24T17:46:32.044800+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:03:03.353576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036406+0000","last_change":"2024-01-24T17:46:31.045067+0000","last_active":"2024-01-24T17:46:32.036406+0000","last_peered":"2024-01-24T17:46:32.036406+0000","last_clean":"2024-01-24T17:46:32.036406+0000","last_became_active":"2024-01-24T17:46:31.043603+0000","last_became_peered":"2024-01-24T17:46:31.043603+0000","last_unstale":"2024-01-24T17:46:32.036406+0000","last_undegraded":"2024-01-24T17:46:32.036406+0000","last_fullsized":"2024-01-24T17:46:32.036406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:10:43.456500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045062+0000","last_change":"2024-01-24T17:46:30.573395+0000","last_active":"2024-01-24T17:46:32.045062+0000","last_peered":"2024-01-24T17:46:32.045062+0000","last_clean":"2024-01-24T17:46:32.045062+0000","last_became_active":"2024-01-24T17:46:30.573071+0000","last_became_peered":"2024-01-24T17:46:30.573071+0000","last_unstale":"2024-01-24T17:46:32.045062+0000","last_undegraded":"2024-01-24T17:46:32.045062+0000","last_fullsized":"2024-01-24T17:46:32.045062+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:48.073697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043865+0000","last_change":"2024-01-24T17:46:31.061571+0000","last_active":"2024-01-24T17:46:32.043865+0000","last_peered":"2024-01-24T17:46:32.043865+0000","last_clean":"2024-01-24T17:46:32.043865+0000","last_became_active":"2024-01-24T17:46:31.060833+0000","last_became_peered":"2024-01-24T17:46:31.060833+0000","last_unstale":"2024-01-24T17:46:32.043865+0000","last_undegraded":"2024-01-24T17:46:32.043865+0000","last_fullsized":"2024-01-24T17:46:32.043865+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:18.568407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036403+0000","last_change":"2024-01-24T17:46:30.573214+0000","last_active":"2024-01-24T17:46:32.036403+0000","last_peered":"2024-01-24T17:46:32.036403+0000","last_clean":"2024-01-24T17:46:32.036403+0000","last_became_active":"2024-01-24T17:46:30.572896+0000","last_became_peered":"2024-01-24T17:46:30.572896+0000","last_unstale":"2024-01-24T17:46:32.036403+0000","last_undegraded":"2024-01-24T17:46:32.036403+0000","last_fullsized":"2024-01-24T17:46:32.036403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:00:51.872282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.033836+0000","last_change":"2024-01-24T17:46:31.042793+0000","last_active":"2024-01-24T17:46:32.033836+0000","last_peered":"2024-01-24T17:46:32.033836+0000","last_clean":"2024-01-24T17:46:32.033836+0000","last_became_active":"2024-01-24T17:46:31.042520+0000","last_became_peered":"2024-01-24T17:46:31.042520+0000","last_unstale":"2024-01-24T17:46:32.033836+0000","last_undegraded":"2024-01-24T17:46:32.033836+0000","last_fullsized":"2024-01-24T17:46:32.033836+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:15:38.041920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043023+0000","last_change":"2024-01-24T17:46:30.570868+0000","last_active":"2024-01-24T17:46:32.043023+0000","last_peered":"2024-01-24T17:46:32.043023+0000","last_clean":"2024-01-24T17:46:32.043023+0000","last_became_active":"2024-01-24T17:46:30.570318+0000","last_became_peered":"2024-01-24T17:46:30.570318+0000","last_unstale":"2024-01-24T17:46:32.043023+0000","last_undegraded":"2024-01-24T17:46:32.043023+0000","last_fullsized":"2024-01-24T17:46:32.043023+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:42:46.485837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042518+0000","last_change":"2024-01-24T17:46:31.053274+0000","last_active":"2024-01-24T17:46:32.042518+0000","last_peered":"2024-01-24T17:46:32.042518+0000","last_clean":"2024-01-24T17:46:32.042518+0000","last_became_active":"2024-01-24T17:46:31.053098+0000","last_became_peered":"2024-01-24T17:46:31.053098+0000","last_unstale":"2024-01-24T17:46:32.042518+0000","last_undegraded":"2024-01-24T17:46:32.042518+0000","last_fullsized":"2024-01-24T17:46:32.042518+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:24:31.700236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044732+0000","last_change":"2024-01-24T17:46:30.567485+0000","last_active":"2024-01-24T17:46:32.044732+0000","last_peered":"2024-01-24T17:46:32.044732+0000","last_clean":"2024-01-24T17:46:32.044732+0000","last_became_active":"2024-01-24T17:46:30.565588+0000","last_became_peered":"2024-01-24T17:46:30.565588+0000","last_unstale":"2024-01-24T17:46:32.044732+0000","last_undegraded":"2024-01-24T17:46:32.044732+0000","last_fullsized":"2024-01-24T17:46:32.044732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:03.978516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036285+0000","last_change":"2024-01-24T17:46:31.054334+0000","last_active":"2024-01-24T17:46:32.036285+0000","last_peered":"2024-01-24T17:46:32.036285+0000","last_clean":"2024-01-24T17:46:32.036285+0000","last_became_active":"2024-01-24T17:46:31.052986+0000","last_became_peered":"2024-01-24T17:46:31.052986+0000","last_unstale":"2024-01-24T17:46:32.036285+0000","last_undegraded":"2024-01-24T17:46:32.036285+0000","last_fullsized":"2024-01-24T17:46:32.036285+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:51:52.816440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035085+0000","last_change":"2024-01-24T17:46:30.557919+0000","last_active":"2024-01-24T17:46:32.035085+0000","last_peered":"2024-01-24T17:46:32.035085+0000","last_clean":"2024-01-24T17:46:32.035085+0000","last_became_active":"2024-01-24T17:46:30.557594+0000","last_became_peered":"2024-01-24T17:46:30.557594+0000","last_unstale":"2024-01-24T17:46:32.035085+0000","last_undegraded":"2024-01-24T17:46:32.035085+0000","last_fullsized":"2024-01-24T17:46:32.035085+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:53.356191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":215,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-01-24T17:46:31.043489+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:46:31.043489+0000","last_peered":"2024-01-24T17:46:31.043489+0000","last_clean":"2024-01-24T17:46:31.043489+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:46:31.043489+0000","last_undegraded":"2024-01-24T17:46:31.043489+0000","last_fullsized":"2024-01-24T17:46:31.043489+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":925696,"data_stored":390769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.075,"15min":1.0329999999999999},"min":{"1min":0.66500000000000004,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5269999999999999,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1020000000000001,"15min":1.0509999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.413,"5min":1.917,"15min":1.917},"last":1.0209999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.095,"15min":1.103},"min":{"1min":0.81200000000000006,"5min":0.621,"15min":0.621},"max":{"1min":1.4930000000000001,"5min":2.04,"15min":2.04},"last":1.133},{"interface":"front","average":{"1min":1.141,"5min":1.1870000000000001,"15min":1.1930000000000001},"min":{"1min":0.81200000000000006,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.454,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.194}]},{"osd":2,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1930000000000001,"15min":1.165},"min":{"1min":0.71099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6359999999999999,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1679999999999999,"15min":1.214},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7430000000000001,"5min":2.484,"15min":2.484},"last":1.236}]},{"osd":3,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7230000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.88400000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.9059999999999999,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.389}]},{"osd":4,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.274,"15min":1.3129999999999999},"min":{"1min":0.86499999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.681,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.4139999999999999,"15min":1.431},"min":{"1min":0.73399999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.6859999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.4239999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2869999999999999,"15min":1.341},"min":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9810000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.325},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.359,"15min":1.4219999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.0030000000000001,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.4490000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4590000000000001,"15min":1.522},"min":{"1min":0.755,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.853,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.361},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.431,"15min":1.474},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.8100000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.0920000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":929792,"data_stored":390803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.052,"15min":1.03},"min":{"1min":0.75800000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4810000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.085,"5min":1.0660000000000001,"15min":1.026},"min":{"1min":0.73199999999999998,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.472,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.997}]},{"osd":1,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.071,"15min":1.008},"min":{"1min":0.71399999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.7210000000000001,"5min":1.73,"15min":1.73},"last":1.343},{"interface":"front","average":{"1min":1.093,"5min":1.149,"15min":1.123},"min":{"1min":0.67800000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.3380000000000001,"5min":1.972,"15min":1.972},"last":1.2470000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.1699999999999999,"15min":1.1459999999999999},"min":{"1min":0.85099999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5900000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.103},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1739999999999999,"15min":1.1679999999999999},"min":{"1min":0.69099999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.8009999999999999,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.755}]},{"osd":3,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.22,"15min":1.2470000000000001},"min":{"1min":0.85799999999999998,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.4510000000000001,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.47},{"interface":"front","average":{"1min":1.147,"5min":1.206,"15min":1.244},"min":{"1min":0.89300000000000002,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6639999999999999,"5min":2.46,"15min":2.46},"last":1.0589999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2889999999999999,"15min":1.248},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.8360000000000001,"5min":1.988,"15min":1.988},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.3,"15min":1.3029999999999999},"min":{"1min":0.78600000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6279999999999999,"5min":1.867,"15min":1.867},"last":1.206}]},{"osd":5,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.325,"15min":1.302},"min":{"1min":0.65300000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7330000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.526},{"interface":"front","average":{"1min":1.389,"5min":1.4319999999999999,"15min":1.4370000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.776,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.4219999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3540000000000001,"15min":1.367},"min":{"1min":0.85999999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.704,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.298},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4530000000000001,"15min":1.4630000000000001},"min":{"1min":0.81899999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.7050000000000001,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.5860000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1523712,"data_stored":985963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0389999999999999,"15min":1.0900000000000001},"min":{"1min":0.71199999999999997,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0349999999999999,"15min":1.0329999999999999},"min":{"1min":0.68600000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.399,"5min":1.889,"15min":1.889},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0740000000000001,"15min":1.073},"min":{"1min":0.71799999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.4179999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.1970000000000001,"15min":1.256},"min":{"1min":0.90900000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6040000000000001,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.524}]},{"osd":2,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.226,"15min":1.23},"min":{"1min":0.92700000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.611,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.325},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.278,"15min":1.3120000000000001},"min":{"1min":0.97299999999999998,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.5580000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.22}]},{"osd":3,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.272,"15min":1.3340000000000001},"min":{"1min":0.873,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.5449999999999999,"5min":3.036,"15min":3.036},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.228,"5min":1.2849999999999999,"15min":1.325},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.647,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.121}]},{"osd":4,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4219999999999999,"15min":1.4490000000000001},"min":{"1min":1.1679999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.591,"5min":2.069,"15min":2.069},"last":0.999},{"interface":"front","average":{"1min":1.155,"5min":1.2849999999999999,"15min":1.345},"min":{"1min":0.79300000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5580000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.407}]},{"osd":6,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.325,"15min":1.351},"min":{"1min":1.03,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.649,"5min":2.504,"15min":2.504},"last":1.157},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.49,"15min":1.518},"min":{"1min":1.131,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.8839999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.2330000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.405,"15min":1.4750000000000001},"min":{"1min":0.81999999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.6559999999999999,"5min":3.161,"15min":3.161},"last":1.28},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.5549999999999999,"15min":1.623},"min":{"1min":1.0660000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.7569999999999999,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.603}]}]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":925696,"data_stored":389899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.071,"15min":1.046},"min":{"1min":0.69899999999999995,"5min":0.621,"15min":0.621},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.139,"5min":1.101,"15min":1.099},"min":{"1min":0.76500000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5389999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.56499999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.081,"15min":1.052},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4530000000000001,"5min":1.633,"15min":1.633},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.139,"5min":1.171,"15min":1.2110000000000001},"min":{"1min":0.79700000000000004,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.712,"5min":1.849,"15min":1.849},"last":0.622}]},{"osd":2,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.198,"15min":1.21},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.5669999999999999,"5min":2.105,"15min":2.105},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.22,"5min":1.278,"15min":1.3260000000000001},"min":{"1min":0.69599999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.794,"5min":1.986,"15min":1.986},"last":0.752}]},{"osd":3,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.198,"15min":1.196},"min":{"1min":0.76900000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.514,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.3160000000000001,"15min":1.3420000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.5589999999999999,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":0.73599999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.335,"15min":1.369},"min":{"1min":0.91200000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8759999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2909999999999999,"15min":1.3029999999999999},"min":{"1min":0.90100000000000002,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.661,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.403,"15min":1.4550000000000001},"min":{"1min":0.92100000000000004,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6319999999999999,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4430000000000001,"15min":1.4730000000000001},"min":{"1min":0.91600000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.78,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.4279999999999999,"15min":1.4239999999999999},"min":{"1min":1.1799999999999999,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7450000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.389,"5min":1.5509999999999999,"15min":1.605},"min":{"1min":0.97699999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.6200000000000001,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":0.79300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":925696,"data_stored":389331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.89000000000000001,"15min":0.86399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.5580000000000001,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":1.5580000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96599999999999997,"15min":0.89500000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.6739999999999999,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.68400000000000005}]},{"osd":1,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.88600000000000001,"15min":0.93300000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.05,"15min":1.0589999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.657,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.657}]},{"osd":2,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.087,"15min":1.105},"min":{"1min":0.56399999999999995,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.4790000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.103,"5min":1.1619999999999999,"15min":1.1579999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.7150000000000001,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":1.7150000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.284,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.8660000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":1.022},{"interface":"front","average":{"1min":1.274,"5min":1.3089999999999999,"15min":1.3400000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.0390000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.085}]},{"osd":5,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.387,"15min":1.421},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.1269999999999998,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.379,"15min":1.411},"min":{"1min":0.78400000000000003,"5min":0.746,"15min":0.746},"max":{"1min":1.9430000000000001,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.9430000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5029999999999999,"15min":1.4930000000000001},"min":{"1min":0.86899999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0110000000000001,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":2.0110000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.427,"15min":1.448},"min":{"1min":0.80200000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.9019999999999999,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.151}]},{"osd":7,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.49,"15min":1.5},"min":{"1min":0.93300000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.833,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.5149999999999999,"15min":1.516},"min":{"1min":1.075,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.0990000000000002,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.282}]}]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":917504,"data_stored":389787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.79000000000000004,"15min":0.79400000000000004},"min":{"1min":0.36499999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.1839999999999999,"5min":1.667,"15min":1.667},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.92100000000000004,"15min":0.93500000000000005},"min":{"1min":0.45400000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.7070000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.0740000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.028,"15min":1.052},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1240000000000001,"15min":1.1990000000000001},"min":{"1min":0.51200000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.222}]},{"osd":3,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.96299999999999997,"15min":0.98899999999999999},"min":{"1min":0.50800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.4690000000000001,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.117,"5min":1.1539999999999999,"15min":1.2010000000000001},"min":{"1min":0.57699999999999996,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4370000000000001,"5min":2.996,"15min":2.996},"last":1.3839999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2829999999999999,"15min":1.321},"min":{"1min":0.93200000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0619999999999998,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.264},{"interface":"front","average":{"1min":1.26,"5min":1.276,"15min":1.3839999999999999},"min":{"1min":0.93999999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6719999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.3540000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3149999999999999,"15min":1.3939999999999999},"min":{"1min":0.88600000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.7110000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.494},{"interface":"front","average":{"1min":1.26,"5min":1.248,"15min":1.3149999999999999},"min":{"1min":0.89900000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5489999999999999,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.3100000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3360000000000001,"15min":1.391},"min":{"1min":0.90100000000000002,"5min":0.623,"15min":0.623},"max":{"1min":1.7569999999999999,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.53},{"interface":"front","average":{"1min":1.458,"5min":1.379,"15min":1.4630000000000001},"min":{"1min":0.88100000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0099999999999998,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.4610000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4430000000000001,"15min":1.5589999999999999},"min":{"1min":0.96899999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.1139999999999999,"5min":3.343,"15min":3.343},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.411,"15min":1.4950000000000001},"min":{"1min":0.84999999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9279999999999999,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.427}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1507328,"data_stored":979573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.90000000000000002,"15min":0.86899999999999999},"min":{"1min":0.41299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.722,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.024},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91500000000000004,"15min":0.94799999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.274}]},{"osd":2,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2929999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.282,"5min":1.109,"15min":1.119},"min":{"1min":0.77400000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.70599999999999996}]},{"osd":3,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1679999999999999,"15min":1.1719999999999999},"min":{"1min":0.72899999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.7190000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.597},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.161,"15min":1.2330000000000001},"min":{"1min":0.67600000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.679,"5min":3.149,"15min":3.149},"last":0.94599999999999995}]},{"osd":4,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.3680000000000001,"15min":1.3700000000000001},"min":{"1min":1.0900000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.962,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.093},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.2949999999999999,"15min":1.323},"min":{"1min":0.872,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.7,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.456}]},{"osd":5,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3109999999999999,"15min":1.353},"min":{"1min":1.0269999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.8109999999999999,"5min":2.577,"15min":2.577},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.361,"15min":1.377},"min":{"1min":1.1439999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.885,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":2.0950000000000002}]},{"osd":6,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.397,"15min":1.4430000000000001},"min":{"1min":1.091,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.008,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.9219999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4370000000000001,"15min":1.5229999999999999},"min":{"1min":0.98199999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.032,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.3819999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4590000000000001,"15min":1.4750000000000001},"min":{"1min":0.95199999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.98,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.8919999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.4930000000000001,"15min":1.5509999999999999},"min":{"1min":1.071,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.911,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.6639999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":929792,"data_stored":395575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79000000000000004,"15min":0.81399999999999995},"min":{"1min":0.42899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.26,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.93000000000000005,"15min":0.96999999999999997},"min":{"1min":0.55300000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.94199999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.51300000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0269999999999999,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4850000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.73699999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.117,"15min":1.1830000000000001},"min":{"1min":0.754,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.79,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.167,"15min":1.1890000000000001},"min":{"1min":0.67700000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6140000000000001,"5min":2.407,"15min":2.407},"last":0.67900000000000005}]},{"osd":4,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3200000000000001,"15min":1.407},"min":{"1min":0.90000000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.601,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.256},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.373,"15min":1.474},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.669,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.268}]},{"osd":5,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3160000000000001,"15min":1.391},"min":{"1min":1.0209999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.7290000000000001,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2629999999999999},"min":{"1min":0.77000000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.6659999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.071}]},{"osd":6,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3380000000000001,"15min":1.3600000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.752,"5min":2.335,"15min":2.335},"last":1.325},{"interface":"front","average":{"1min":1.415,"5min":1.401,"15min":1.456},"min":{"1min":0.99199999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.921,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.381}]},{"osd":7,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.429,"15min":1.5229999999999999},"min":{"1min":0.89700000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.788,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.5049999999999999,"15min":1.5940000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.859,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.4890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:46:36.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:46:36.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.289+0000 7f719d000640 1 Processor -- start 2024-01-24T17:46:36.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.289+0000 7f719d000640 1 -- start start 2024-01-24T17:46:36.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.291+0000 7f719d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7198159d00 0x7f71980cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.291+0000 7f719d000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7198071c00 0x7f71980ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.291+0000 7f719d000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71980ccd10 con 0x7f71980b3f20 2024-01-24T17:46:36.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.291+0000 7f719d000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71980ce230 con 0x7f7198159d00 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.291+0000 7f719d000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71980ccff0 con 0x7f7198071c00 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.291+0000 7f7195e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7198159d00 0x7f71980cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.291+0000 7f7195e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7198159d00 0x7f71980cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47258/0 (socket says 172.21.15.39:47258) 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.291+0000 7f7195e00640 1 -- 172.21.15.39:0/919441393 learned_addr learned my addr 172.21.15.39:0/919441393 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.292+0000 7f7196800640 1 --2- 172.21.15.39:0/919441393 >> v2:172.21.15.100:3300/0 conn(0x7f7198071c00 0x7f71980ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.292+0000 7f7195e00640 1 -- 172.21.15.39:0/919441393 >> v2:172.21.15.100:3300/0 conn(0x7f7198071c00 msgr2=0x7f71980ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.292+0000 7f7195e00640 1 --2- 172.21.15.39:0/919441393 >> v2:172.21.15.100:3300/0 conn(0x7f7198071c00 0x7f71980ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.292+0000 7f7195e00640 1 -- 172.21.15.39:0/919441393 >> v1:172.21.15.100:6789/0 conn(0x7f71980b3f20 legacy=0x7f71980714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:36.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.292+0000 7f7195e00640 1 -- 172.21.15.39:0/919441393 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71980ad720 con 0x7f7198159d00 2024-01-24T17:46:36.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.293+0000 7f7195e00640 1 --2- 172.21.15.39:0/919441393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71980cdcf0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f719806fea0 tx=0x7f718802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd718ffa1f84418c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:36.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.293+0000 7f7195400640 1 -- 172.21.15.39:0/919441393 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188031870 con 0x7f7198159d00 2024-01-24T17:46:36.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.293+0000 7f7195400640 1 -- 172.21.15.39:0/919441393 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71880319d0 con 0x7f7198159d00 2024-01-24T17:46:36.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.293+0000 7f7195400640 1 -- 172.21.15.39:0/919441393 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188031d60 con 0x7f7198159d00 2024-01-24T17:46:36.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.294+0000 7f719d000640 1 -- 172.21.15.39:0/919441393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 msgr2=0x7f71980cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.294+0000 7f719d000640 1 --2- 172.21.15.39:0/919441393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71980cdcf0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f719806fea0 tx=0x7f718802b910 comp rx=0 tx=0).stop 2024-01-24T17:46:36.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.294+0000 7f719d000640 1 -- 172.21.15.39:0/919441393 shutdown_connections 2024-01-24T17:46:36.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.294+0000 7f719d000640 1 --2- 172.21.15.39:0/919441393 >> v2:172.21.15.100:3300/0 conn(0x7f7198071c00 0x7f71980ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.294+0000 7f719d000640 1 --2- 172.21.15.39:0/919441393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71980cdcf0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.294+0000 7f719d000640 1 -- 172.21.15.39:0/919441393 >> 172.21.15.39:0/919441393 conn(0x7f719806bf20 msgr2=0x7f719806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:36.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.294+0000 7f719d000640 1 -- 172.21.15.39:0/919441393 shutdown_connections 2024-01-24T17:46:36.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.294+0000 7f719d000640 1 -- 172.21.15.39:0/919441393 wait complete. 2024-01-24T17:46:36.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.296+0000 7f719d000640 1 Processor -- start 2024-01-24T17:46:36.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.296+0000 7f719d000640 1 -- start start 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f719d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7198071c00 0x7f71980c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f719d000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71980b3f20 0x7f71980c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f719d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71981f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f719d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71980c79f0 con 0x7f7198159d00 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f719d000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719814ebe0 con 0x7f71980b3f20 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f719d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71980c70c0 con 0x7f7198071c00 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f7196800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71981f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f7196800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71981f3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47274/0 (socket says 172.21.15.39:47274) 2024-01-24T17:46:36.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.297+0000 7f7196800640 1 -- 172.21.15.39:0/3046722369 learned_addr learned my addr 172.21.15.39:0/3046722369 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f718fe00640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71980b3f20 0x7f71980c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f7195e00640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7198071c00 0x7f71980c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f7196800640 1 -- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7198071c00 msgr2=0x7f71980c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f7196800640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7198071c00 0x7f71980c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f7196800640 1 -- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71980b3f20 msgr2=0x7f71980c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f7196800640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71980b3f20 0x7f71980c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f7196800640 1 -- 172.21.15.39:0/3046722369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71880094d0 con 0x7f7198159d00 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f718fe00640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71980b3f20 0x7f71980c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f7195e00640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7198071c00 0x7f71980c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f7196800640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71981f3800 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7190007e40 tx=0x7f719000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f718e000640 1 -- 172.21.15.39:0/3046722369 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190014620 con 0x7f7198159d00 2024-01-24T17:46:36.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f718e000640 1 -- 172.21.15.39:0/3046722369 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719000ee90 con 0x7f7198159d00 2024-01-24T17:46:36.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f718e000640 1 -- 172.21.15.39:0/3046722369 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71900149e0 con 0x7f7198159d00 2024-01-24T17:46:36.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.298+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71980c73a0 con 0x7f7198159d00 2024-01-24T17:46:36.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.299+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71980c84f0 con 0x7f7198159d00 2024-01-24T17:46:36.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.300+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71980ca810 con 0x7f7198159d00 2024-01-24T17:46:36.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.301+0000 7f718e000640 1 -- 172.21.15.39:0/3046722369 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7190014b40 con 0x7f7198159d00 2024-01-24T17:46:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.302+0000 7f718e000640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f717006a430 0x7f717006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.302+0000 7f718e000640 1 -- 172.21.15.39:0/3046722369 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71900892d0 con 0x7f7198159d00 2024-01-24T17:46:36.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.302+0000 7f7195e00640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f717006a430 0x7f717006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.303+0000 7f7195e00640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f717006a430 0x7f717006c8f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7188036000 tx=0x7f718803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:36.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.306+0000 7f718e000640 1 -- 172.21.15.39:0/3046722369 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f71980ca810 con 0x7f7198159d00 2024-01-24T17:46:36.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.500+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71980adce0 con 0x7f717006a430 2024-01-24T17:46:36.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.522+0000 7f718e000640 1 -- 172.21.15.39:0/3046722369 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287187 (secure 0 0 0) 0x7f7188031c40 con 0x7f717006a430 2024-01-24T17:46:36.526 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:36.528 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:46:36.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f717006a430 msgr2=0x7f717006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f717006a430 0x7f717006c8f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7188036000 tx=0x7f718803c040 comp rx=0 tx=0).stop 2024-01-24T17:46:36.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 msgr2=0x7f71981f3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71981f3800 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7190007e40 tx=0x7f719000e4b0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 shutdown_connections 2024-01-24T17:46:36.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f717006a430 0x7f717006c8f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7198159d00 0x7f71981f3800 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71980b3f20 0x7f71980c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 --2- 172.21.15.39:0/3046722369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7198071c00 0x7f71980c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 >> 172.21.15.39:0/3046722369 conn(0x7f719806bf20 msgr2=0x7f719815c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:36.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 shutdown_connections 2024-01-24T17:46:36.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.529+0000 7f719d000640 1 -- 172.21.15.39:0/3046722369 wait complete. 2024-01-24T17:46:36.549 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-01-24T17:46:36.319990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257364,"kb_used_data":8384,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754717356,"statfs":{"total":773094113280,"available":772830572544,"internally_reserved":0,"allocated":8585216,"data_stored":4311700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002407"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045082+0000","last_change":"2024-01-24T17:46:31.054672+0000","last_active":"2024-01-24T17:46:32.045082+0000","last_peered":"2024-01-24T17:46:32.045082+0000","last_clean":"2024-01-24T17:46:32.045082+0000","last_became_active":"2024-01-24T17:46:31.054438+0000","last_became_peered":"2024-01-24T17:46:31.054438+0000","last_unstale":"2024-01-24T17:46:32.045082+0000","last_undegraded":"2024-01-24T17:46:32.045082+0000","last_fullsized":"2024-01-24T17:46:32.045082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:24:27.292422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042806+0000","last_change":"2024-01-24T17:46:31.061225+0000","last_active":"2024-01-24T17:46:32.042806+0000","last_peered":"2024-01-24T17:46:32.042806+0000","last_clean":"2024-01-24T17:46:32.042806+0000","last_became_active":"2024-01-24T17:46:31.060468+0000","last_became_peered":"2024-01-24T17:46:31.060468+0000","last_unstale":"2024-01-24T17:46:32.042806+0000","last_undegraded":"2024-01-24T17:46:32.042806+0000","last_fullsized":"2024-01-24T17:46:32.042806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:00:52.884562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042381+0000","last_change":"2024-01-24T17:46:31.058896+0000","last_active":"2024-01-24T17:46:32.042381+0000","last_peered":"2024-01-24T17:46:32.042381+0000","last_clean":"2024-01-24T17:46:32.042381+0000","last_became_active":"2024-01-24T17:46:31.058603+0000","last_became_peered":"2024-01-24T17:46:31.058603+0000","last_unstale":"2024-01-24T17:46:32.042381+0000","last_undegraded":"2024-01-24T17:46:32.042381+0000","last_fullsized":"2024-01-24T17:46:32.042381+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:58:33.404255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042742+0000","last_change":"2024-01-24T17:46:31.055283+0000","last_active":"2024-01-24T17:46:32.042742+0000","last_peered":"2024-01-24T17:46:32.042742+0000","last_clean":"2024-01-24T17:46:32.042742+0000","last_became_active":"2024-01-24T17:46:31.054935+0000","last_became_peered":"2024-01-24T17:46:31.054935+0000","last_unstale":"2024-01-24T17:46:32.042742+0000","last_undegraded":"2024-01-24T17:46:32.042742+0000","last_fullsized":"2024-01-24T17:46:32.042742+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:54:20.923764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036460+0000","last_change":"2024-01-24T17:46:31.050700+0000","last_active":"2024-01-24T17:46:32.036460+0000","last_peered":"2024-01-24T17:46:32.036460+0000","last_clean":"2024-01-24T17:46:32.036460+0000","last_became_active":"2024-01-24T17:46:31.050144+0000","last_became_peered":"2024-01-24T17:46:31.050144+0000","last_unstale":"2024-01-24T17:46:32.036460+0000","last_undegraded":"2024-01-24T17:46:32.036460+0000","last_fullsized":"2024-01-24T17:46:32.036460+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:35:11.783864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044887+0000","last_change":"2024-01-24T17:46:31.056008+0000","last_active":"2024-01-24T17:46:32.044887+0000","last_peered":"2024-01-24T17:46:32.044887+0000","last_clean":"2024-01-24T17:46:32.044887+0000","last_became_active":"2024-01-24T17:46:31.055681+0000","last_became_peered":"2024-01-24T17:46:31.055681+0000","last_unstale":"2024-01-24T17:46:32.044887+0000","last_undegraded":"2024-01-24T17:46:32.044887+0000","last_fullsized":"2024-01-24T17:46:32.044887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:03:20.262992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034983+0000","last_change":"2024-01-24T17:46:31.051289+0000","last_active":"2024-01-24T17:46:32.034983+0000","last_peered":"2024-01-24T17:46:32.034983+0000","last_clean":"2024-01-24T17:46:32.034983+0000","last_became_active":"2024-01-24T17:46:31.051035+0000","last_became_peered":"2024-01-24T17:46:31.051035+0000","last_unstale":"2024-01-24T17:46:32.034983+0000","last_undegraded":"2024-01-24T17:46:32.034983+0000","last_fullsized":"2024-01-24T17:46:32.034983+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:36:13.230601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041352+0000","last_change":"2024-01-24T17:46:31.056745+0000","last_active":"2024-01-24T17:46:32.041352+0000","last_peered":"2024-01-24T17:46:32.041352+0000","last_clean":"2024-01-24T17:46:32.041352+0000","last_became_active":"2024-01-24T17:46:31.056492+0000","last_became_peered":"2024-01-24T17:46:31.056492+0000","last_unstale":"2024-01-24T17:46:32.041352+0000","last_undegraded":"2024-01-24T17:46:32.041352+0000","last_fullsized":"2024-01-24T17:46:32.041352+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:01:00.817919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043032+0000","last_change":"2024-01-24T17:46:31.041542+0000","last_active":"2024-01-24T17:46:32.043032+0000","last_peered":"2024-01-24T17:46:32.043032+0000","last_clean":"2024-01-24T17:46:32.043032+0000","last_became_active":"2024-01-24T17:46:31.040890+0000","last_became_peered":"2024-01-24T17:46:31.040890+0000","last_unstale":"2024-01-24T17:46:32.043032+0000","last_undegraded":"2024-01-24T17:46:32.043032+0000","last_fullsized":"2024-01-24T17:46:32.043032+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:27:39.874755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035731+0000","last_change":"2024-01-24T17:46:31.057978+0000","last_active":"2024-01-24T17:46:32.035731+0000","last_peered":"2024-01-24T17:46:32.035731+0000","last_clean":"2024-01-24T17:46:32.035731+0000","last_became_active":"2024-01-24T17:46:31.057652+0000","last_became_peered":"2024-01-24T17:46:31.057652+0000","last_unstale":"2024-01-24T17:46:32.035731+0000","last_undegraded":"2024-01-24T17:46:32.035731+0000","last_fullsized":"2024-01-24T17:46:32.035731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:09:00.744685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044222+0000","last_change":"2024-01-24T17:46:31.058608+0000","last_active":"2024-01-24T17:46:32.044222+0000","last_peered":"2024-01-24T17:46:32.044222+0000","last_clean":"2024-01-24T17:46:32.044222+0000","last_became_active":"2024-01-24T17:46:31.058060+0000","last_became_peered":"2024-01-24T17:46:31.058060+0000","last_unstale":"2024-01-24T17:46:32.044222+0000","last_undegraded":"2024-01-24T17:46:32.044222+0000","last_fullsized":"2024-01-24T17:46:32.044222+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:20:20.136491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044806+0000","last_change":"2024-01-24T17:46:31.046406+0000","last_active":"2024-01-24T17:46:32.044806+0000","last_peered":"2024-01-24T17:46:32.044806+0000","last_clean":"2024-01-24T17:46:32.044806+0000","last_became_active":"2024-01-24T17:46:31.041915+0000","last_became_peered":"2024-01-24T17:46:31.041915+0000","last_unstale":"2024-01-24T17:46:32.044806+0000","last_undegraded":"2024-01-24T17:46:32.044806+0000","last_fullsized":"2024-01-24T17:46:32.044806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:00:16.571792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042629+0000","last_change":"2024-01-24T17:46:31.059008+0000","last_active":"2024-01-24T17:46:32.042629+0000","last_peered":"2024-01-24T17:46:32.042629+0000","last_clean":"2024-01-24T17:46:32.042629+0000","last_became_active":"2024-01-24T17:46:31.058686+0000","last_became_peered":"2024-01-24T17:46:31.058686+0000","last_unstale":"2024-01-24T17:46:32.042629+0000","last_undegraded":"2024-01-24T17:46:32.042629+0000","last_fullsized":"2024-01-24T17:46:32.042629+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:06:14.187190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044594+0000","last_change":"2024-01-24T17:46:31.065860+0000","last_active":"2024-01-24T17:46:32.044594+0000","last_peered":"2024-01-24T17:46:32.044594+0000","last_clean":"2024-01-24T17:46:32.044594+0000","last_became_active":"2024-01-24T17:46:31.065525+0000","last_became_peered":"2024-01-24T17:46:31.065525+0000","last_unstale":"2024-01-24T17:46:32.044594+0000","last_undegraded":"2024-01-24T17:46:32.044594+0000","last_fullsized":"2024-01-24T17:46:32.044594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:55.521949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044314+0000","last_change":"2024-01-24T17:46:31.038167+0000","last_active":"2024-01-24T17:46:32.044314+0000","last_peered":"2024-01-24T17:46:32.044314+0000","last_clean":"2024-01-24T17:46:32.044314+0000","last_became_active":"2024-01-24T17:46:31.037942+0000","last_became_peered":"2024-01-24T17:46:31.037942+0000","last_unstale":"2024-01-24T17:46:32.044314+0000","last_undegraded":"2024-01-24T17:46:32.044314+0000","last_fullsized":"2024-01-24T17:46:32.044314+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:06:14.187190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041437+0000","last_change":"2024-01-24T17:46:31.061327+0000","last_active":"2024-01-24T17:46:32.041437+0000","last_peered":"2024-01-24T17:46:32.041437+0000","last_clean":"2024-01-24T17:46:32.041437+0000","last_became_active":"2024-01-24T17:46:31.061074+0000","last_became_peered":"2024-01-24T17:46:31.061074+0000","last_unstale":"2024-01-24T17:46:32.041437+0000","last_undegraded":"2024-01-24T17:46:32.041437+0000","last_fullsized":"2024-01-24T17:46:32.041437+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:48:58.770452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.037177+0000","last_change":"2024-01-24T17:46:31.045559+0000","last_active":"2024-01-24T17:46:32.037177+0000","last_peered":"2024-01-24T17:46:32.037177+0000","last_clean":"2024-01-24T17:46:32.037177+0000","last_became_active":"2024-01-24T17:46:31.045140+0000","last_became_peered":"2024-01-24T17:46:31.045140+0000","last_unstale":"2024-01-24T17:46:32.037177+0000","last_undegraded":"2024-01-24T17:46:32.037177+0000","last_fullsized":"2024-01-24T17:46:32.037177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:04:19.310970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.037121+0000","last_change":"2024-01-24T17:46:31.060194+0000","last_active":"2024-01-24T17:46:32.037121+0000","last_peered":"2024-01-24T17:46:32.037121+0000","last_clean":"2024-01-24T17:46:32.037121+0000","last_became_active":"2024-01-24T17:46:31.059896+0000","last_became_peered":"2024-01-24T17:46:31.059896+0000","last_unstale":"2024-01-24T17:46:32.037121+0000","last_undegraded":"2024-01-24T17:46:32.037121+0000","last_fullsized":"2024-01-24T17:46:32.037121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:33:53.027586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043745+0000","last_change":"2024-01-24T17:46:31.058768+0000","last_active":"2024-01-24T17:46:32.043745+0000","last_peered":"2024-01-24T17:46:32.043745+0000","last_clean":"2024-01-24T17:46:32.043745+0000","last_became_active":"2024-01-24T17:46:31.058442+0000","last_became_peered":"2024-01-24T17:46:31.058442+0000","last_unstale":"2024-01-24T17:46:32.043745+0000","last_undegraded":"2024-01-24T17:46:32.043745+0000","last_fullsized":"2024-01-24T17:46:32.043745+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:33:22.421274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036665+0000","last_change":"2024-01-24T17:46:31.049908+0000","last_active":"2024-01-24T17:46:32.036665+0000","last_peered":"2024-01-24T17:46:32.036665+0000","last_clean":"2024-01-24T17:46:32.036665+0000","last_became_active":"2024-01-24T17:46:31.048699+0000","last_became_peered":"2024-01-24T17:46:31.048699+0000","last_unstale":"2024-01-24T17:46:32.036665+0000","last_undegraded":"2024-01-24T17:46:32.036665+0000","last_fullsized":"2024-01-24T17:46:32.036665+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:05:44.318085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042248+0000","last_change":"2024-01-24T17:46:31.045192+0000","last_active":"2024-01-24T17:46:32.042248+0000","last_peered":"2024-01-24T17:46:32.042248+0000","last_clean":"2024-01-24T17:46:32.042248+0000","last_became_active":"2024-01-24T17:46:31.043802+0000","last_became_peered":"2024-01-24T17:46:31.043802+0000","last_unstale":"2024-01-24T17:46:32.042248+0000","last_undegraded":"2024-01-24T17:46:32.042248+0000","last_fullsized":"2024-01-24T17:46:32.042248+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:21:46.987232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045080+0000","last_change":"2024-01-24T17:46:31.040029+0000","last_active":"2024-01-24T17:46:32.045080+0000","last_peered":"2024-01-24T17:46:32.045080+0000","last_clean":"2024-01-24T17:46:32.045080+0000","last_became_active":"2024-01-24T17:46:31.039725+0000","last_became_peered":"2024-01-24T17:46:31.039725+0000","last_unstale":"2024-01-24T17:46:32.045080+0000","last_undegraded":"2024-01-24T17:46:32.045080+0000","last_fullsized":"2024-01-24T17:46:32.045080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:30:54.165656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034990+0000","last_change":"2024-01-24T17:46:31.065042+0000","last_active":"2024-01-24T17:46:32.034990+0000","last_peered":"2024-01-24T17:46:32.034990+0000","last_clean":"2024-01-24T17:46:32.034990+0000","last_became_active":"2024-01-24T17:46:31.062877+0000","last_became_peered":"2024-01-24T17:46:31.062877+0000","last_unstale":"2024-01-24T17:46:32.034990+0000","last_undegraded":"2024-01-24T17:46:32.034990+0000","last_fullsized":"2024-01-24T17:46:32.034990+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:49.085977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041530+0000","last_change":"2024-01-24T17:46:31.051293+0000","last_active":"2024-01-24T17:46:32.041530+0000","last_peered":"2024-01-24T17:46:32.041530+0000","last_clean":"2024-01-24T17:46:32.041530+0000","last_became_active":"2024-01-24T17:46:31.050040+0000","last_became_peered":"2024-01-24T17:46:31.050040+0000","last_unstale":"2024-01-24T17:46:32.041530+0000","last_undegraded":"2024-01-24T17:46:32.041530+0000","last_fullsized":"2024-01-24T17:46:32.041530+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:30:29.679524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035387+0000","last_change":"2024-01-24T17:46:31.059345+0000","last_active":"2024-01-24T17:46:32.035387+0000","last_peered":"2024-01-24T17:46:32.035387+0000","last_clean":"2024-01-24T17:46:32.035387+0000","last_became_active":"2024-01-24T17:46:31.059068+0000","last_became_peered":"2024-01-24T17:46:31.059068+0000","last_unstale":"2024-01-24T17:46:32.035387+0000","last_undegraded":"2024-01-24T17:46:32.035387+0000","last_fullsized":"2024-01-24T17:46:32.035387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:34.489869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044256+0000","last_change":"2024-01-24T17:46:31.062880+0000","last_active":"2024-01-24T17:46:32.044256+0000","last_peered":"2024-01-24T17:46:32.044256+0000","last_clean":"2024-01-24T17:46:32.044256+0000","last_became_active":"2024-01-24T17:46:31.062290+0000","last_became_peered":"2024-01-24T17:46:31.062290+0000","last_unstale":"2024-01-24T17:46:32.044256+0000","last_undegraded":"2024-01-24T17:46:32.044256+0000","last_fullsized":"2024-01-24T17:46:32.044256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:00:33.199885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036785+0000","last_change":"2024-01-24T17:46:31.051510+0000","last_active":"2024-01-24T17:46:32.036785+0000","last_peered":"2024-01-24T17:46:32.036785+0000","last_clean":"2024-01-24T17:46:32.036785+0000","last_became_active":"2024-01-24T17:46:31.051107+0000","last_became_peered":"2024-01-24T17:46:31.051107+0000","last_unstale":"2024-01-24T17:46:32.036785+0000","last_undegraded":"2024-01-24T17:46:32.036785+0000","last_fullsized":"2024-01-24T17:46:32.036785+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:04:33.963910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042142+0000","last_change":"2024-01-24T17:46:31.039655+0000","last_active":"2024-01-24T17:46:32.042142+0000","last_peered":"2024-01-24T17:46:32.042142+0000","last_clean":"2024-01-24T17:46:32.042142+0000","last_became_active":"2024-01-24T17:46:31.038771+0000","last_became_peered":"2024-01-24T17:46:31.038771+0000","last_unstale":"2024-01-24T17:46:32.042142+0000","last_undegraded":"2024-01-24T17:46:32.042142+0000","last_fullsized":"2024-01-24T17:46:32.042142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:04.990796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042602+0000","last_change":"2024-01-24T17:46:31.053704+0000","last_active":"2024-01-24T17:46:32.042602+0000","last_peered":"2024-01-24T17:46:32.042602+0000","last_clean":"2024-01-24T17:46:32.042602+0000","last_became_active":"2024-01-24T17:46:31.053419+0000","last_became_peered":"2024-01-24T17:46:31.053419+0000","last_unstale":"2024-01-24T17:46:32.042602+0000","last_undegraded":"2024-01-24T17:46:32.042602+0000","last_fullsized":"2024-01-24T17:46:32.042602+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:08:28.844452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041642+0000","last_change":"2024-01-24T17:46:31.051391+0000","last_active":"2024-01-24T17:46:32.041642+0000","last_peered":"2024-01-24T17:46:32.041642+0000","last_clean":"2024-01-24T17:46:32.041642+0000","last_became_active":"2024-01-24T17:46:31.050828+0000","last_became_peered":"2024-01-24T17:46:31.050828+0000","last_unstale":"2024-01-24T17:46:32.041642+0000","last_undegraded":"2024-01-24T17:46:32.041642+0000","last_fullsized":"2024-01-24T17:46:32.041642+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:17:22.948043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042466+0000","last_change":"2024-01-24T17:46:31.038145+0000","last_active":"2024-01-24T17:46:32.042466+0000","last_peered":"2024-01-24T17:46:32.042466+0000","last_clean":"2024-01-24T17:46:32.042466+0000","last_became_active":"2024-01-24T17:46:31.037788+0000","last_became_peered":"2024-01-24T17:46:31.037788+0000","last_unstale":"2024-01-24T17:46:32.042466+0000","last_undegraded":"2024-01-24T17:46:32.042466+0000","last_fullsized":"2024-01-24T17:46:32.042466+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:08:28.844452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042769+0000","last_change":"2024-01-24T17:46:31.040665+0000","last_active":"2024-01-24T17:46:32.042769+0000","last_peered":"2024-01-24T17:46:32.042769+0000","last_clean":"2024-01-24T17:46:32.042769+0000","last_became_active":"2024-01-24T17:46:31.040228+0000","last_became_peered":"2024-01-24T17:46:31.040228+0000","last_unstale":"2024-01-24T17:46:32.042769+0000","last_undegraded":"2024-01-24T17:46:32.042769+0000","last_fullsized":"2024-01-24T17:46:32.042769+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:54:20.923764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.046347+0000","last_change":"2024-01-24T17:46:31.063028+0000","last_active":"2024-01-24T17:46:32.046347+0000","last_peered":"2024-01-24T17:46:32.046347+0000","last_clean":"2024-01-24T17:46:32.046347+0000","last_became_active":"2024-01-24T17:46:31.062619+0000","last_became_peered":"2024-01-24T17:46:31.062619+0000","last_unstale":"2024-01-24T17:46:32.046347+0000","last_undegraded":"2024-01-24T17:46:32.046347+0000","last_fullsized":"2024-01-24T17:46:32.046347+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:15:29.774800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1f","version":"75'4","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045856+0000","last_change":"2024-01-24T17:46:30.568884+0000","last_active":"2024-01-24T17:46:32.045856+0000","last_peered":"2024-01-24T17:46:32.045856+0000","last_clean":"2024-01-24T17:46:32.045856+0000","last_became_active":"2024-01-24T17:46:30.566185+0000","last_became_peered":"2024-01-24T17:46:30.566185+0000","last_unstale":"2024-01-24T17:46:32.045856+0000","last_undegraded":"2024-01-24T17:46:32.045856+0000","last_fullsized":"2024-01-24T17:46:32.045856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:38:52.542242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043220+0000","last_change":"2024-01-24T17:46:31.042247+0000","last_active":"2024-01-24T17:46:32.043220+0000","last_peered":"2024-01-24T17:46:32.043220+0000","last_clean":"2024-01-24T17:46:32.043220+0000","last_became_active":"2024-01-24T17:46:31.041231+0000","last_became_peered":"2024-01-24T17:46:31.041231+0000","last_unstale":"2024-01-24T17:46:32.043220+0000","last_undegraded":"2024-01-24T17:46:32.043220+0000","last_fullsized":"2024-01-24T17:46:32.043220+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:04.990796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1e","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.037262+0000","last_change":"2024-01-24T17:46:30.560090+0000","last_active":"2024-01-24T17:46:32.037262+0000","last_peered":"2024-01-24T17:46:32.037262+0000","last_clean":"2024-01-24T17:46:32.037262+0000","last_became_active":"2024-01-24T17:46:30.559793+0000","last_became_peered":"2024-01-24T17:46:30.559793+0000","last_unstale":"2024-01-24T17:46:32.037262+0000","last_undegraded":"2024-01-24T17:46:32.037262+0000","last_fullsized":"2024-01-24T17:46:32.037262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:15:33.655294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045752+0000","last_change":"2024-01-24T17:46:31.060301+0000","last_active":"2024-01-24T17:46:32.045752+0000","last_peered":"2024-01-24T17:46:32.045752+0000","last_clean":"2024-01-24T17:46:32.045752+0000","last_became_active":"2024-01-24T17:46:31.059982+0000","last_became_peered":"2024-01-24T17:46:31.059982+0000","last_unstale":"2024-01-24T17:46:32.045752+0000","last_undegraded":"2024-01-24T17:46:32.045752+0000","last_fullsized":"2024-01-24T17:46:32.045752+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:02:26.610727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1d","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044472+0000","last_change":"2024-01-24T17:46:30.576753+0000","last_active":"2024-01-24T17:46:32.044472+0000","last_peered":"2024-01-24T17:46:32.044472+0000","last_clean":"2024-01-24T17:46:32.044472+0000","last_became_active":"2024-01-24T17:46:30.576134+0000","last_became_peered":"2024-01-24T17:46:30.576134+0000","last_unstale":"2024-01-24T17:46:32.044472+0000","last_undegraded":"2024-01-24T17:46:32.044472+0000","last_fullsized":"2024-01-24T17:46:32.044472+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:01:40.209070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035430+0000","last_change":"2024-01-24T17:46:31.064345+0000","last_active":"2024-01-24T17:46:32.035430+0000","last_peered":"2024-01-24T17:46:32.035430+0000","last_clean":"2024-01-24T17:46:32.035430+0000","last_became_active":"2024-01-24T17:46:31.064100+0000","last_became_peered":"2024-01-24T17:46:31.064100+0000","last_unstale":"2024-01-24T17:46:32.035430+0000","last_undegraded":"2024-01-24T17:46:32.035430+0000","last_fullsized":"2024-01-24T17:46:32.035430+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:42:47.498116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1c","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045852+0000","last_change":"2024-01-24T17:46:30.564784+0000","last_active":"2024-01-24T17:46:32.045852+0000","last_peered":"2024-01-24T17:46:32.045852+0000","last_clean":"2024-01-24T17:46:32.045852+0000","last_became_active":"2024-01-24T17:46:30.564501+0000","last_became_peered":"2024-01-24T17:46:30.564501+0000","last_unstale":"2024-01-24T17:46:32.045852+0000","last_undegraded":"2024-01-24T17:46:32.045852+0000","last_fullsized":"2024-01-24T17:46:32.045852+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:54:19.911485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.041915+0000","last_change":"2024-01-24T17:46:31.052817+0000","last_active":"2024-01-24T17:46:32.041915+0000","last_peered":"2024-01-24T17:46:32.041915+0000","last_clean":"2024-01-24T17:46:32.041915+0000","last_became_active":"2024-01-24T17:46:31.052489+0000","last_became_peered":"2024-01-24T17:46:31.052489+0000","last_unstale":"2024-01-24T17:46:32.041915+0000","last_undegraded":"2024-01-24T17:46:32.041915+0000","last_fullsized":"2024-01-24T17:46:32.041915+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:20:02.096010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044947+0000","last_change":"2024-01-24T17:46:30.570372+0000","last_active":"2024-01-24T17:46:32.044947+0000","last_peered":"2024-01-24T17:46:32.044947+0000","last_clean":"2024-01-24T17:46:32.044947+0000","last_became_active":"2024-01-24T17:46:30.569997+0000","last_became_peered":"2024-01-24T17:46:30.569997+0000","last_unstale":"2024-01-24T17:46:32.044947+0000","last_undegraded":"2024-01-24T17:46:32.044947+0000","last_fullsized":"2024-01-24T17:46:32.044947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:33.477589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045216+0000","last_change":"2024-01-24T17:46:31.055380+0000","last_active":"2024-01-24T17:46:32.045216+0000","last_peered":"2024-01-24T17:46:32.045216+0000","last_clean":"2024-01-24T17:46:32.045216+0000","last_became_active":"2024-01-24T17:46:31.053319+0000","last_became_peered":"2024-01-24T17:46:31.053319+0000","last_unstale":"2024-01-24T17:46:32.045216+0000","last_undegraded":"2024-01-24T17:46:32.045216+0000","last_fullsized":"2024-01-24T17:46:32.045216+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:52:12.764326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1a","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034763+0000","last_change":"2024-01-24T17:46:30.574055+0000","last_active":"2024-01-24T17:46:32.034763+0000","last_peered":"2024-01-24T17:46:32.034763+0000","last_clean":"2024-01-24T17:46:32.034763+0000","last_became_active":"2024-01-24T17:46:30.573748+0000","last_became_peered":"2024-01-24T17:46:30.573748+0000","last_unstale":"2024-01-24T17:46:32.034763+0000","last_undegraded":"2024-01-24T17:46:32.034763+0000","last_fullsized":"2024-01-24T17:46:32.034763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:37:54.618595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.033810+0000","last_change":"2024-01-24T17:46:31.055136+0000","last_active":"2024-01-24T17:46:32.033810+0000","last_peered":"2024-01-24T17:46:32.033810+0000","last_clean":"2024-01-24T17:46:32.033810+0000","last_became_active":"2024-01-24T17:46:31.054903+0000","last_became_peered":"2024-01-24T17:46:31.054903+0000","last_unstale":"2024-01-24T17:46:32.033810+0000","last_undegraded":"2024-01-24T17:46:32.033810+0000","last_fullsized":"2024-01-24T17:46:32.033810+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:38:53.950426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.033784+0000","last_change":"2024-01-24T17:46:30.570178+0000","last_active":"2024-01-24T17:46:32.033784+0000","last_peered":"2024-01-24T17:46:32.033784+0000","last_clean":"2024-01-24T17:46:32.033784+0000","last_became_active":"2024-01-24T17:46:30.569930+0000","last_became_peered":"2024-01-24T17:46:30.569930+0000","last_unstale":"2024-01-24T17:46:32.033784+0000","last_undegraded":"2024-01-24T17:46:32.033784+0000","last_fullsized":"2024-01-24T17:46:32.033784+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:42:46.485837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044515+0000","last_change":"2024-01-24T17:46:31.052827+0000","last_active":"2024-01-24T17:46:32.044515+0000","last_peered":"2024-01-24T17:46:32.044515+0000","last_clean":"2024-01-24T17:46:32.044515+0000","last_became_active":"2024-01-24T17:46:31.052524+0000","last_became_peered":"2024-01-24T17:46:31.052524+0000","last_unstale":"2024-01-24T17:46:32.044515+0000","last_undegraded":"2024-01-24T17:46:32.044515+0000","last_fullsized":"2024-01-24T17:46:32.044515+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:27:39.874755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044601+0000","last_change":"2024-01-24T17:46:30.559701+0000","last_active":"2024-01-24T17:46:32.044601+0000","last_peered":"2024-01-24T17:46:32.044601+0000","last_clean":"2024-01-24T17:46:32.044601+0000","last_became_active":"2024-01-24T17:46:30.559366+0000","last_became_peered":"2024-01-24T17:46:30.559366+0000","last_unstale":"2024-01-24T17:46:32.044601+0000","last_undegraded":"2024-01-24T17:46:32.044601+0000","last_fullsized":"2024-01-24T17:46:32.044601+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:42:46.485837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035770+0000","last_change":"2024-01-24T17:46:31.066489+0000","last_active":"2024-01-24T17:46:32.035770+0000","last_peered":"2024-01-24T17:46:32.035770+0000","last_clean":"2024-01-24T17:46:32.035770+0000","last_became_active":"2024-01-24T17:46:31.066270+0000","last_became_peered":"2024-01-24T17:46:31.066270+0000","last_unstale":"2024-01-24T17:46:32.035770+0000","last_undegraded":"2024-01-24T17:46:32.035770+0000","last_fullsized":"2024-01-24T17:46:32.035770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:03:04.365856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044255+0000","last_change":"2024-01-24T17:46:30.571903+0000","last_active":"2024-01-24T17:46:32.044255+0000","last_peered":"2024-01-24T17:46:32.044255+0000","last_clean":"2024-01-24T17:46:32.044255+0000","last_became_active":"2024-01-24T17:46:30.571610+0000","last_became_peered":"2024-01-24T17:46:30.571610+0000","last_unstale":"2024-01-24T17:46:32.044255+0000","last_undegraded":"2024-01-24T17:46:32.044255+0000","last_fullsized":"2024-01-24T17:46:32.044255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:47:45.575561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043219+0000","last_change":"2024-01-24T17:46:31.059190+0000","last_active":"2024-01-24T17:46:32.043219+0000","last_peered":"2024-01-24T17:46:32.043219+0000","last_clean":"2024-01-24T17:46:32.043219+0000","last_became_active":"2024-01-24T17:46:31.058877+0000","last_became_peered":"2024-01-24T17:46:31.058877+0000","last_unstale":"2024-01-24T17:46:32.043219+0000","last_undegraded":"2024-01-24T17:46:32.043219+0000","last_fullsized":"2024-01-24T17:46:32.043219+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:15:34.667574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043105+0000","last_change":"2024-01-24T17:46:30.572944+0000","last_active":"2024-01-24T17:46:32.043105+0000","last_peered":"2024-01-24T17:46:32.043105+0000","last_clean":"2024-01-24T17:46:32.043105+0000","last_became_active":"2024-01-24T17:46:30.572756+0000","last_became_peered":"2024-01-24T17:46:30.572756+0000","last_unstale":"2024-01-24T17:46:32.043105+0000","last_undegraded":"2024-01-24T17:46:32.043105+0000","last_fullsized":"2024-01-24T17:46:32.043105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:33.477589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043762+0000","last_change":"2024-01-24T17:46:31.060960+0000","last_active":"2024-01-24T17:46:32.043762+0000","last_peered":"2024-01-24T17:46:32.043762+0000","last_clean":"2024-01-24T17:46:32.043762+0000","last_became_active":"2024-01-24T17:46:31.060665+0000","last_became_peered":"2024-01-24T17:46:31.060665+0000","last_unstale":"2024-01-24T17:46:32.043762+0000","last_undegraded":"2024-01-24T17:46:32.043762+0000","last_fullsized":"2024-01-24T17:46:32.043762+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:16:56.913832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"75'3","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045078+0000","last_change":"2024-01-24T17:46:30.559432+0000","last_active":"2024-01-24T17:46:32.045078+0000","last_peered":"2024-01-24T17:46:32.045078+0000","last_clean":"2024-01-24T17:46:32.045078+0000","last_became_active":"2024-01-24T17:46:30.559078+0000","last_became_peered":"2024-01-24T17:46:30.559078+0000","last_unstale":"2024-01-24T17:46:32.045078+0000","last_undegraded":"2024-01-24T17:46:32.045078+0000","last_fullsized":"2024-01-24T17:46:32.045078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:36:12.218321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036878+0000","last_change":"2024-01-24T17:46:31.045618+0000","last_active":"2024-01-24T17:46:32.036878+0000","last_peered":"2024-01-24T17:46:32.036878+0000","last_clean":"2024-01-24T17:46:32.036878+0000","last_became_active":"2024-01-24T17:46:31.045258+0000","last_became_peered":"2024-01-24T17:46:31.045258+0000","last_unstale":"2024-01-24T17:46:32.036878+0000","last_undegraded":"2024-01-24T17:46:32.036878+0000","last_fullsized":"2024-01-24T17:46:32.036878+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:47:46.587841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"75'6","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034537+0000","last_change":"2024-01-24T17:46:30.562474+0000","last_active":"2024-01-24T17:46:32.034537+0000","last_peered":"2024-01-24T17:46:32.034537+0000","last_clean":"2024-01-24T17:46:32.034537+0000","last_became_active":"2024-01-24T17:46:30.561846+0000","last_became_peered":"2024-01-24T17:46:30.561846+0000","last_unstale":"2024-01-24T17:46:32.034537+0000","last_undegraded":"2024-01-24T17:46:32.034537+0000","last_fullsized":"2024-01-24T17:46:32.034537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:48.073697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045174+0000","last_change":"2024-01-24T17:46:31.045974+0000","last_active":"2024-01-24T17:46:32.045174+0000","last_peered":"2024-01-24T17:46:32.045174+0000","last_clean":"2024-01-24T17:46:32.045174+0000","last_became_active":"2024-01-24T17:46:31.044058+0000","last_became_peered":"2024-01-24T17:46:31.044058+0000","last_unstale":"2024-01-24T17:46:32.045174+0000","last_undegraded":"2024-01-24T17:46:32.045174+0000","last_fullsized":"2024-01-24T17:46:32.045174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:59:37.246294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034195+0000","last_change":"2024-01-24T17:46:30.566856+0000","last_active":"2024-01-24T17:46:32.034195+0000","last_peered":"2024-01-24T17:46:32.034195+0000","last_clean":"2024-01-24T17:46:32.034195+0000","last_became_active":"2024-01-24T17:46:30.566472+0000","last_became_peered":"2024-01-24T17:46:30.566472+0000","last_unstale":"2024-01-24T17:46:32.034195+0000","last_undegraded":"2024-01-24T17:46:32.034195+0000","last_fullsized":"2024-01-24T17:46:32.034195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:54.139951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034982+0000","last_change":"2024-01-24T17:46:31.063647+0000","last_active":"2024-01-24T17:46:32.034982+0000","last_peered":"2024-01-24T17:46:32.034982+0000","last_clean":"2024-01-24T17:46:32.034982+0000","last_became_active":"2024-01-24T17:46:31.063249+0000","last_became_peered":"2024-01-24T17:46:31.063249+0000","last_unstale":"2024-01-24T17:46:32.034982+0000","last_undegraded":"2024-01-24T17:46:32.034982+0000","last_fullsized":"2024-01-24T17:46:32.034982+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:38:53.950426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034984+0000","last_change":"2024-01-24T17:46:30.564980+0000","last_active":"2024-01-24T17:46:32.034984+0000","last_peered":"2024-01-24T17:46:32.034984+0000","last_clean":"2024-01-24T17:46:32.034984+0000","last_became_active":"2024-01-24T17:46:30.562518+0000","last_became_peered":"2024-01-24T17:46:30.562518+0000","last_unstale":"2024-01-24T17:46:32.034984+0000","last_undegraded":"2024-01-24T17:46:32.034984+0000","last_fullsized":"2024-01-24T17:46:32.034984+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:37:27.535127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035866+0000","last_change":"2024-01-24T17:46:31.044670+0000","last_active":"2024-01-24T17:46:32.035866+0000","last_peered":"2024-01-24T17:46:32.035866+0000","last_clean":"2024-01-24T17:46:32.035866+0000","last_became_active":"2024-01-24T17:46:31.044361+0000","last_became_peered":"2024-01-24T17:46:31.044361+0000","last_unstale":"2024-01-24T17:46:32.035866+0000","last_undegraded":"2024-01-24T17:46:32.035866+0000","last_fullsized":"2024-01-24T17:46:32.035866+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:10:14.348792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035216+0000","last_change":"2024-01-24T17:46:30.559627+0000","last_active":"2024-01-24T17:46:32.035216+0000","last_peered":"2024-01-24T17:46:32.035216+0000","last_clean":"2024-01-24T17:46:32.035216+0000","last_became_active":"2024-01-24T17:46:30.559367+0000","last_became_peered":"2024-01-24T17:46:30.559367+0000","last_unstale":"2024-01-24T17:46:32.035216+0000","last_undegraded":"2024-01-24T17:46:32.035216+0000","last_fullsized":"2024-01-24T17:46:32.035216+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:01:56.871757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042193+0000","last_change":"2024-01-24T17:46:31.034211+0000","last_active":"2024-01-24T17:46:32.042193+0000","last_peered":"2024-01-24T17:46:32.042193+0000","last_clean":"2024-01-24T17:46:32.042193+0000","last_became_active":"2024-01-24T17:46:31.033908+0000","last_became_peered":"2024-01-24T17:46:31.033908+0000","last_unstale":"2024-01-24T17:46:32.042193+0000","last_undegraded":"2024-01-24T17:46:32.042193+0000","last_fullsized":"2024-01-24T17:46:32.042193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:38:53.554522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.10","version":"75'8","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034656+0000","last_change":"2024-01-24T17:46:30.568532+0000","last_active":"2024-01-24T17:46:32.034656+0000","last_peered":"2024-01-24T17:46:32.034656+0000","last_clean":"2024-01-24T17:46:32.034656+0000","last_became_active":"2024-01-24T17:46:30.568219+0000","last_became_peered":"2024-01-24T17:46:30.568219+0000","last_unstale":"2024-01-24T17:46:32.034656+0000","last_undegraded":"2024-01-24T17:46:32.034656+0000","last_fullsized":"2024-01-24T17:46:32.034656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:38:48.719620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042836+0000","last_change":"2024-01-24T17:46:31.043376+0000","last_active":"2024-01-24T17:46:32.042836+0000","last_peered":"2024-01-24T17:46:32.042836+0000","last_clean":"2024-01-24T17:46:32.042836+0000","last_became_active":"2024-01-24T17:46:31.041172+0000","last_became_peered":"2024-01-24T17:46:31.041172+0000","last_unstale":"2024-01-24T17:46:32.042836+0000","last_undegraded":"2024-01-24T17:46:32.042836+0000","last_fullsized":"2024-01-24T17:46:32.042836+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:58:33.404255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044826+0000","last_change":"2024-01-24T17:46:30.570606+0000","last_active":"2024-01-24T17:46:32.044826+0000","last_peered":"2024-01-24T17:46:32.044826+0000","last_clean":"2024-01-24T17:46:32.044826+0000","last_became_active":"2024-01-24T17:46:30.570188+0000","last_became_peered":"2024-01-24T17:46:30.570188+0000","last_unstale":"2024-01-24T17:46:32.044826+0000","last_undegraded":"2024-01-24T17:46:32.044826+0000","last_fullsized":"2024-01-24T17:46:32.044826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:24:10.438941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043423+0000","last_change":"2024-01-24T17:46:31.044575+0000","last_active":"2024-01-24T17:46:32.043423+0000","last_peered":"2024-01-24T17:46:32.043423+0000","last_clean":"2024-01-24T17:46:32.043423+0000","last_became_active":"2024-01-24T17:46:31.044237+0000","last_became_peered":"2024-01-24T17:46:31.044237+0000","last_unstale":"2024-01-24T17:46:32.043423+0000","last_undegraded":"2024-01-24T17:46:32.043423+0000","last_fullsized":"2024-01-24T17:46:32.043423+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:38:53.554522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043135+0000","last_change":"2024-01-24T17:46:30.555050+0000","last_active":"2024-01-24T17:46:32.043135+0000","last_peered":"2024-01-24T17:46:32.043135+0000","last_clean":"2024-01-24T17:46:32.043135+0000","last_became_active":"2024-01-24T17:46:30.554773+0000","last_became_peered":"2024-01-24T17:46:30.554773+0000","last_unstale":"2024-01-24T17:46:32.043135+0000","last_undegraded":"2024-01-24T17:46:32.043135+0000","last_fullsized":"2024-01-24T17:46:32.043135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:54.139951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042558+0000","last_change":"2024-01-24T17:46:31.041732+0000","last_active":"2024-01-24T17:46:32.042558+0000","last_peered":"2024-01-24T17:46:32.042558+0000","last_clean":"2024-01-24T17:46:32.042558+0000","last_became_active":"2024-01-24T17:46:31.040392+0000","last_became_peered":"2024-01-24T17:46:31.040392+0000","last_unstale":"2024-01-24T17:46:32.042558+0000","last_undegraded":"2024-01-24T17:46:32.042558+0000","last_fullsized":"2024-01-24T17:46:32.042558+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:55.521949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.d","version":"75'1","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042344+0000","last_change":"2024-01-24T17:46:30.575914+0000","last_active":"2024-01-24T17:46:32.042344+0000","last_peered":"2024-01-24T17:46:32.042344+0000","last_clean":"2024-01-24T17:46:32.042344+0000","last_became_active":"2024-01-24T17:46:30.575637+0000","last_became_peered":"2024-01-24T17:46:30.575637+0000","last_unstale":"2024-01-24T17:46:32.042344+0000","last_undegraded":"2024-01-24T17:46:32.042344+0000","last_fullsized":"2024-01-24T17:46:32.042344+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:56:43.348935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042551+0000","last_change":"2024-01-24T17:46:31.056695+0000","last_active":"2024-01-24T17:46:32.042551+0000","last_peered":"2024-01-24T17:46:32.042551+0000","last_clean":"2024-01-24T17:46:32.042551+0000","last_became_active":"2024-01-24T17:46:31.056432+0000","last_became_peered":"2024-01-24T17:46:31.056432+0000","last_unstale":"2024-01-24T17:46:32.042551+0000","last_undegraded":"2024-01-24T17:46:32.042551+0000","last_fullsized":"2024-01-24T17:46:32.042551+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:24:31.700236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.046005+0000","last_change":"2024-01-24T17:46:30.575906+0000","last_active":"2024-01-24T17:46:32.046005+0000","last_peered":"2024-01-24T17:46:32.046005+0000","last_clean":"2024-01-24T17:46:32.046005+0000","last_became_active":"2024-01-24T17:46:30.575532+0000","last_became_peered":"2024-01-24T17:46:30.575532+0000","last_unstale":"2024-01-24T17:46:32.046005+0000","last_undegraded":"2024-01-24T17:46:32.046005+0000","last_fullsized":"2024-01-24T17:46:32.046005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:58:32.391975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.034063+0000","last_change":"2024-01-24T17:46:31.035497+0000","last_active":"2024-01-24T17:46:32.034063+0000","last_peered":"2024-01-24T17:46:32.034063+0000","last_clean":"2024-01-24T17:46:32.034063+0000","last_became_active":"2024-01-24T17:46:31.035159+0000","last_became_peered":"2024-01-24T17:46:31.035159+0000","last_unstale":"2024-01-24T17:46:32.034063+0000","last_undegraded":"2024-01-24T17:46:32.034063+0000","last_fullsized":"2024-01-24T17:46:32.034063+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:00:16.571792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045328+0000","last_change":"2024-01-24T17:46:30.566751+0000","last_active":"2024-01-24T17:46:32.045328+0000","last_peered":"2024-01-24T17:46:32.045328+0000","last_clean":"2024-01-24T17:46:32.045328+0000","last_became_active":"2024-01-24T17:46:30.566192+0000","last_became_peered":"2024-01-24T17:46:30.566192+0000","last_unstale":"2024-01-24T17:46:32.045328+0000","last_undegraded":"2024-01-24T17:46:32.045328+0000","last_fullsized":"2024-01-24T17:46:32.045328+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:06:13.174910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.033556+0000","last_change":"2024-01-24T17:46:31.039959+0000","last_active":"2024-01-24T17:46:32.033556+0000","last_peered":"2024-01-24T17:46:32.033556+0000","last_clean":"2024-01-24T17:46:32.033556+0000","last_became_active":"2024-01-24T17:46:31.038298+0000","last_became_peered":"2024-01-24T17:46:31.038298+0000","last_unstale":"2024-01-24T17:46:32.033556+0000","last_undegraded":"2024-01-24T17:46:32.033556+0000","last_fullsized":"2024-01-24T17:46:32.033556+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:56:12.091199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042768+0000","last_change":"2024-01-24T17:46:30.556835+0000","last_active":"2024-01-24T17:46:32.042768+0000","last_peered":"2024-01-24T17:46:32.042768+0000","last_clean":"2024-01-24T17:46:32.042768+0000","last_became_active":"2024-01-24T17:46:30.556561+0000","last_became_peered":"2024-01-24T17:46:30.556561+0000","last_unstale":"2024-01-24T17:46:32.042768+0000","last_undegraded":"2024-01-24T17:46:32.042768+0000","last_fullsized":"2024-01-24T17:46:32.042768+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:03:03.353576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044453+0000","last_change":"2024-01-24T17:46:31.040557+0000","last_active":"2024-01-24T17:46:32.044453+0000","last_peered":"2024-01-24T17:46:32.044453+0000","last_clean":"2024-01-24T17:46:32.044453+0000","last_became_active":"2024-01-24T17:46:31.040210+0000","last_became_peered":"2024-01-24T17:46:31.040210+0000","last_unstale":"2024-01-24T17:46:32.044453+0000","last_undegraded":"2024-01-24T17:46:32.044453+0000","last_fullsized":"2024-01-24T17:46:32.044453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:38:53.554522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035789+0000","last_change":"2024-01-24T17:46:30.554541+0000","last_active":"2024-01-24T17:46:32.035789+0000","last_peered":"2024-01-24T17:46:32.035789+0000","last_clean":"2024-01-24T17:46:32.035789+0000","last_became_active":"2024-01-24T17:46:30.553996+0000","last_became_peered":"2024-01-24T17:46:30.553996+0000","last_unstale":"2024-01-24T17:46:32.035789+0000","last_undegraded":"2024-01-24T17:46:32.035789+0000","last_fullsized":"2024-01-24T17:46:32.035789+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:21:45.974952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.037192+0000","last_change":"2024-01-24T17:46:31.046837+0000","last_active":"2024-01-24T17:46:32.037192+0000","last_peered":"2024-01-24T17:46:32.037192+0000","last_clean":"2024-01-24T17:46:32.037192+0000","last_became_active":"2024-01-24T17:46:31.046029+0000","last_became_peered":"2024-01-24T17:46:31.046029+0000","last_unstale":"2024-01-24T17:46:32.037192+0000","last_undegraded":"2024-01-24T17:46:32.037192+0000","last_fullsized":"2024-01-24T17:46:32.037192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:56:57.399522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045009+0000","last_change":"2024-01-24T17:46:30.570966+0000","last_active":"2024-01-24T17:46:32.045009+0000","last_peered":"2024-01-24T17:46:32.045009+0000","last_clean":"2024-01-24T17:46:32.045009+0000","last_became_active":"2024-01-24T17:46:30.570634+0000","last_became_peered":"2024-01-24T17:46:30.570634+0000","last_unstale":"2024-01-24T17:46:32.045009+0000","last_undegraded":"2024-01-24T17:46:32.045009+0000","last_fullsized":"2024-01-24T17:46:32.045009+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:04:32.951630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035520+0000","last_change":"2024-01-24T17:46:31.053536+0000","last_active":"2024-01-24T17:46:32.035520+0000","last_peered":"2024-01-24T17:46:32.035520+0000","last_clean":"2024-01-24T17:46:32.035520+0000","last_became_active":"2024-01-24T17:46:31.051508+0000","last_became_peered":"2024-01-24T17:46:31.051508+0000","last_unstale":"2024-01-24T17:46:32.035520+0000","last_undegraded":"2024-01-24T17:46:32.035520+0000","last_fullsized":"2024-01-24T17:46:32.035520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:55.152231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044368+0000","last_change":"2024-01-24T17:46:30.567484+0000","last_active":"2024-01-24T17:46:32.044368+0000","last_peered":"2024-01-24T17:46:32.044368+0000","last_clean":"2024-01-24T17:46:32.044368+0000","last_became_active":"2024-01-24T17:46:30.567016+0000","last_became_peered":"2024-01-24T17:46:30.567016+0000","last_unstale":"2024-01-24T17:46:32.044368+0000","last_undegraded":"2024-01-24T17:46:32.044368+0000","last_fullsized":"2024-01-24T17:46:32.044368+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:50.235864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042412+0000","last_change":"2024-01-24T17:46:31.051429+0000","last_active":"2024-01-24T17:46:32.042412+0000","last_peered":"2024-01-24T17:46:32.042412+0000","last_clean":"2024-01-24T17:46:32.042412+0000","last_became_active":"2024-01-24T17:46:31.051159+0000","last_became_peered":"2024-01-24T17:46:31.051159+0000","last_unstale":"2024-01-24T17:46:32.042412+0000","last_undegraded":"2024-01-24T17:46:32.042412+0000","last_fullsized":"2024-01-24T17:46:32.042412+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:26:43.578208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045079+0000","last_change":"2024-01-24T17:46:30.553986+0000","last_active":"2024-01-24T17:46:32.045079+0000","last_peered":"2024-01-24T17:46:32.045079+0000","last_clean":"2024-01-24T17:46:32.045079+0000","last_became_active":"2024-01-24T17:46:30.553145+0000","last_became_peered":"2024-01-24T17:46:30.553145+0000","last_unstale":"2024-01-24T17:46:32.045079+0000","last_undegraded":"2024-01-24T17:46:32.045079+0000","last_fullsized":"2024-01-24T17:46:32.045079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:58:54.139951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035370+0000","last_change":"2024-01-24T17:46:31.057010+0000","last_active":"2024-01-24T17:46:32.035370+0000","last_peered":"2024-01-24T17:46:32.035370+0000","last_clean":"2024-01-24T17:46:32.035370+0000","last_became_active":"2024-01-24T17:46:31.055165+0000","last_became_peered":"2024-01-24T17:46:31.055165+0000","last_unstale":"2024-01-24T17:46:32.035370+0000","last_undegraded":"2024-01-24T17:46:32.035370+0000","last_fullsized":"2024-01-24T17:46:32.035370+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:33:55.710838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044800+0000","last_change":"2024-01-24T17:46:30.559257+0000","last_active":"2024-01-24T17:46:32.044800+0000","last_peered":"2024-01-24T17:46:32.044800+0000","last_clean":"2024-01-24T17:46:32.044800+0000","last_became_active":"2024-01-24T17:46:30.558943+0000","last_became_peered":"2024-01-24T17:46:30.558943+0000","last_unstale":"2024-01-24T17:46:32.044800+0000","last_undegraded":"2024-01-24T17:46:32.044800+0000","last_fullsized":"2024-01-24T17:46:32.044800+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:03:03.353576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036406+0000","last_change":"2024-01-24T17:46:31.045067+0000","last_active":"2024-01-24T17:46:32.036406+0000","last_peered":"2024-01-24T17:46:32.036406+0000","last_clean":"2024-01-24T17:46:32.036406+0000","last_became_active":"2024-01-24T17:46:31.043603+0000","last_became_peered":"2024-01-24T17:46:31.043603+0000","last_unstale":"2024-01-24T17:46:32.036406+0000","last_undegraded":"2024-01-24T17:46:32.036406+0000","last_fullsized":"2024-01-24T17:46:32.036406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:10:43.456500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.045062+0000","last_change":"2024-01-24T17:46:30.573395+0000","last_active":"2024-01-24T17:46:32.045062+0000","last_peered":"2024-01-24T17:46:32.045062+0000","last_clean":"2024-01-24T17:46:32.045062+0000","last_became_active":"2024-01-24T17:46:30.573071+0000","last_became_peered":"2024-01-24T17:46:30.573071+0000","last_unstale":"2024-01-24T17:46:32.045062+0000","last_undegraded":"2024-01-24T17:46:32.045062+0000","last_fullsized":"2024-01-24T17:46:32.045062+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:46:48.073697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043865+0000","last_change":"2024-01-24T17:46:31.061571+0000","last_active":"2024-01-24T17:46:32.043865+0000","last_peered":"2024-01-24T17:46:32.043865+0000","last_clean":"2024-01-24T17:46:32.043865+0000","last_became_active":"2024-01-24T17:46:31.060833+0000","last_became_peered":"2024-01-24T17:46:31.060833+0000","last_unstale":"2024-01-24T17:46:32.043865+0000","last_undegraded":"2024-01-24T17:46:32.043865+0000","last_fullsized":"2024-01-24T17:46:32.043865+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:18.568407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036403+0000","last_change":"2024-01-24T17:46:30.573214+0000","last_active":"2024-01-24T17:46:32.036403+0000","last_peered":"2024-01-24T17:46:32.036403+0000","last_clean":"2024-01-24T17:46:32.036403+0000","last_became_active":"2024-01-24T17:46:30.572896+0000","last_became_peered":"2024-01-24T17:46:30.572896+0000","last_unstale":"2024-01-24T17:46:32.036403+0000","last_undegraded":"2024-01-24T17:46:32.036403+0000","last_fullsized":"2024-01-24T17:46:32.036403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:00:51.872282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.033836+0000","last_change":"2024-01-24T17:46:31.042793+0000","last_active":"2024-01-24T17:46:32.033836+0000","last_peered":"2024-01-24T17:46:32.033836+0000","last_clean":"2024-01-24T17:46:32.033836+0000","last_became_active":"2024-01-24T17:46:31.042520+0000","last_became_peered":"2024-01-24T17:46:31.042520+0000","last_unstale":"2024-01-24T17:46:32.033836+0000","last_undegraded":"2024-01-24T17:46:32.033836+0000","last_fullsized":"2024-01-24T17:46:32.033836+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:15:38.041920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.043023+0000","last_change":"2024-01-24T17:46:30.570868+0000","last_active":"2024-01-24T17:46:32.043023+0000","last_peered":"2024-01-24T17:46:32.043023+0000","last_clean":"2024-01-24T17:46:32.043023+0000","last_became_active":"2024-01-24T17:46:30.570318+0000","last_became_peered":"2024-01-24T17:46:30.570318+0000","last_unstale":"2024-01-24T17:46:32.043023+0000","last_undegraded":"2024-01-24T17:46:32.043023+0000","last_fullsized":"2024-01-24T17:46:32.043023+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:42:46.485837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.042518+0000","last_change":"2024-01-24T17:46:31.053274+0000","last_active":"2024-01-24T17:46:32.042518+0000","last_peered":"2024-01-24T17:46:32.042518+0000","last_clean":"2024-01-24T17:46:32.042518+0000","last_became_active":"2024-01-24T17:46:31.053098+0000","last_became_peered":"2024-01-24T17:46:31.053098+0000","last_unstale":"2024-01-24T17:46:32.042518+0000","last_undegraded":"2024-01-24T17:46:32.042518+0000","last_fullsized":"2024-01-24T17:46:32.042518+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:24:31.700236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.044732+0000","last_change":"2024-01-24T17:46:30.567485+0000","last_active":"2024-01-24T17:46:32.044732+0000","last_peered":"2024-01-24T17:46:32.044732+0000","last_clean":"2024-01-24T17:46:32.044732+0000","last_became_active":"2024-01-24T17:46:30.565588+0000","last_became_peered":"2024-01-24T17:46:30.565588+0000","last_unstale":"2024-01-24T17:46:32.044732+0000","last_undegraded":"2024-01-24T17:46:32.044732+0000","last_fullsized":"2024-01-24T17:46:32.044732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:03.978516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.036285+0000","last_change":"2024-01-24T17:46:31.054334+0000","last_active":"2024-01-24T17:46:32.036285+0000","last_peered":"2024-01-24T17:46:32.036285+0000","last_clean":"2024-01-24T17:46:32.036285+0000","last_became_active":"2024-01-24T17:46:31.052986+0000","last_became_peered":"2024-01-24T17:46:31.052986+0000","last_unstale":"2024-01-24T17:46:32.036285+0000","last_undegraded":"2024-01-24T17:46:32.036285+0000","last_fullsized":"2024-01-24T17:46:32.036285+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:30.514814+0000","last_clean_scrub_stamp":"2024-01-24T17:46:30.514814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:51:52.816440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"75'2","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-01-24T17:46:32.035085+0000","last_change":"2024-01-24T17:46:30.557919+0000","last_active":"2024-01-24T17:46:32.035085+0000","last_peered":"2024-01-24T17:46:32.035085+0000","last_clean":"2024-01-24T17:46:32.035085+0000","last_became_active":"2024-01-24T17:46:30.557594+0000","last_became_peered":"2024-01-24T17:46:30.557594+0000","last_unstale":"2024-01-24T17:46:32.035085+0000","last_undegraded":"2024-01-24T17:46:32.035085+0000","last_fullsized":"2024-01-24T17:46:32.035085+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:46:29.502534+0000","last_clean_scrub_stamp":"2024-01-24T17:46:29.502534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:35:53.356191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":215,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-01-24T17:46:31.043489+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:46:31.043489+0000","last_peered":"2024-01-24T17:46:31.043489+0000","last_clean":"2024-01-24T17:46:31.043489+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:46:31.043489+0000","last_undegraded":"2024-01-24T17:46:31.043489+0000","last_fullsized":"2024-01-24T17:46:31.043489+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":925696,"data_stored":390769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.075,"15min":1.0329999999999999},"min":{"1min":0.66500000000000004,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5269999999999999,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1020000000000001,"15min":1.0509999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.413,"5min":1.917,"15min":1.917},"last":1.0209999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.095,"15min":1.103},"min":{"1min":0.81200000000000006,"5min":0.621,"15min":0.621},"max":{"1min":1.4930000000000001,"5min":2.04,"15min":2.04},"last":1.133},{"interface":"front","average":{"1min":1.141,"5min":1.1870000000000001,"15min":1.1930000000000001},"min":{"1min":0.81200000000000006,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.454,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.194}]},{"osd":2,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1930000000000001,"15min":1.165},"min":{"1min":0.71099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6359999999999999,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1679999999999999,"15min":1.214},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7430000000000001,"5min":2.484,"15min":2.484},"last":1.236}]},{"osd":3,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7230000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.88400000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.9059999999999999,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.389}]},{"osd":4,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.274,"15min":1.3129999999999999},"min":{"1min":0.86499999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.681,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.4139999999999999,"15min":1.431},"min":{"1min":0.73399999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.6859999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.4239999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2869999999999999,"15min":1.341},"min":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9810000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.325},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.359,"15min":1.4219999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.0030000000000001,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.4490000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4590000000000001,"15min":1.522},"min":{"1min":0.755,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.853,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":1.361},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.431,"15min":1.474},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.8100000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.0920000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":929792,"data_stored":390803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.052,"15min":1.03},"min":{"1min":0.75800000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.4810000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.085,"5min":1.0660000000000001,"15min":1.026},"min":{"1min":0.73199999999999998,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.472,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.997}]},{"osd":1,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.071,"15min":1.008},"min":{"1min":0.71399999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.7210000000000001,"5min":1.73,"15min":1.73},"last":1.343},{"interface":"front","average":{"1min":1.093,"5min":1.149,"15min":1.123},"min":{"1min":0.67800000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.3380000000000001,"5min":1.972,"15min":1.972},"last":1.2470000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.1699999999999999,"15min":1.1459999999999999},"min":{"1min":0.85099999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5900000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.103},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.1739999999999999,"15min":1.1679999999999999},"min":{"1min":0.69099999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.8009999999999999,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.755}]},{"osd":3,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.22,"15min":1.2470000000000001},"min":{"1min":0.85799999999999998,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.4510000000000001,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.47},{"interface":"front","average":{"1min":1.147,"5min":1.206,"15min":1.244},"min":{"1min":0.89300000000000002,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.6639999999999999,"5min":2.46,"15min":2.46},"last":1.0589999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2889999999999999,"15min":1.248},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.8360000000000001,"5min":1.988,"15min":1.988},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.3,"15min":1.3029999999999999},"min":{"1min":0.78600000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6279999999999999,"5min":1.867,"15min":1.867},"last":1.206}]},{"osd":5,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.325,"15min":1.302},"min":{"1min":0.65300000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7330000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.526},{"interface":"front","average":{"1min":1.389,"5min":1.4319999999999999,"15min":1.4370000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.776,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.4219999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:46:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3540000000000001,"15min":1.367},"min":{"1min":0.85999999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.704,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":1.298},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4530000000000001,"15min":1.4630000000000001},"min":{"1min":0.81899999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.7050000000000001,"5min":2.6920000000000002,"15min":2.6920000000000002},"last":1.5860000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1523712,"data_stored":985963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0389999999999999,"15min":1.0900000000000001},"min":{"1min":0.71199999999999997,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0349999999999999,"15min":1.0329999999999999},"min":{"1min":0.68600000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.399,"5min":1.889,"15min":1.889},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0740000000000001,"15min":1.073},"min":{"1min":0.71799999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.4179999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.1970000000000001,"15min":1.256},"min":{"1min":0.90900000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6040000000000001,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.524}]},{"osd":2,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.226,"15min":1.23},"min":{"1min":0.92700000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.611,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.325},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.278,"15min":1.3120000000000001},"min":{"1min":0.97299999999999998,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.5580000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":1.22}]},{"osd":3,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.272,"15min":1.3340000000000001},"min":{"1min":0.873,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.5449999999999999,"5min":3.036,"15min":3.036},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.228,"5min":1.2849999999999999,"15min":1.325},"min":{"1min":1.0029999999999999,"5min":0.85599999999999998,"15min":0.85599999999999998},"max":{"1min":1.647,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.121}]},{"osd":4,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4219999999999999,"15min":1.4490000000000001},"min":{"1min":1.1679999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.591,"5min":2.069,"15min":2.069},"last":0.999},{"interface":"front","average":{"1min":1.155,"5min":1.2849999999999999,"15min":1.345},"min":{"1min":0.79300000000000004,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.5580000000000001,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.407}]},{"osd":6,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.325,"15min":1.351},"min":{"1min":1.03,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.649,"5min":2.504,"15min":2.504},"last":1.157},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.49,"15min":1.518},"min":{"1min":1.131,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.8839999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.2330000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:45:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.405,"15min":1.4750000000000001},"min":{"1min":0.81999999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.6559999999999999,"5min":3.161,"15min":3.161},"last":1.28},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.5549999999999999,"15min":1.623},"min":{"1min":1.0660000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.7569999999999999,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.603}]}]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":925696,"data_stored":389899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.071,"15min":1.046},"min":{"1min":0.69899999999999995,"5min":0.621,"15min":0.621},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.139,"5min":1.101,"15min":1.099},"min":{"1min":0.76500000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5389999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.56499999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.081,"15min":1.052},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4530000000000001,"5min":1.633,"15min":1.633},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.139,"5min":1.171,"15min":1.2110000000000001},"min":{"1min":0.79700000000000004,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.712,"5min":1.849,"15min":1.849},"last":0.622}]},{"osd":2,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.198,"15min":1.21},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.5669999999999999,"5min":2.105,"15min":2.105},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.22,"5min":1.278,"15min":1.3260000000000001},"min":{"1min":0.69599999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.794,"5min":1.986,"15min":1.986},"last":0.752}]},{"osd":3,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.198,"15min":1.196},"min":{"1min":0.76900000000000002,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.514,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.22,"5min":1.3160000000000001,"15min":1.3420000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.5589999999999999,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":0.73599999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.335,"15min":1.369},"min":{"1min":0.91200000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8759999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2909999999999999,"15min":1.3029999999999999},"min":{"1min":0.90100000000000002,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.661,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.403,"15min":1.4550000000000001},"min":{"1min":0.92100000000000004,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.6319999999999999,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4430000000000001,"15min":1.4730000000000001},"min":{"1min":0.91600000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.78,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.4279999999999999,"15min":1.4239999999999999},"min":{"1min":1.1799999999999999,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.7450000000000001,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.389,"5min":1.5509999999999999,"15min":1.605},"min":{"1min":0.97699999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.6200000000000001,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":0.79300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574915,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":925696,"data_stored":389331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.89000000000000001,"15min":0.86399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.5580000000000001,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":1.5580000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96599999999999997,"15min":0.89500000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.6739999999999999,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.68400000000000005}]},{"osd":1,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.88600000000000001,"15min":0.93300000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.05,"15min":1.0589999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.657,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.657}]},{"osd":2,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.087,"15min":1.105},"min":{"1min":0.56399999999999995,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.4790000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.103,"5min":1.1619999999999999,"15min":1.1579999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.7150000000000001,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":1.7150000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.284,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.8660000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":1.022},{"interface":"front","average":{"1min":1.274,"5min":1.3089999999999999,"15min":1.3400000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.0390000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.085}]},{"osd":5,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.387,"15min":1.421},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.1269999999999998,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.379,"15min":1.411},"min":{"1min":0.78400000000000003,"5min":0.746,"15min":0.746},"max":{"1min":1.9430000000000001,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.9430000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5029999999999999,"15min":1.4930000000000001},"min":{"1min":0.86899999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0110000000000001,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":2.0110000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.427,"15min":1.448},"min":{"1min":0.80200000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.9019999999999999,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.151}]},{"osd":7,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.49,"15min":1.5},"min":{"1min":0.93300000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.833,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.5149999999999999,"15min":1.516},"min":{"1min":1.075,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.0990000000000002,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.282}]}]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":917504,"data_stored":389787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.79000000000000004,"15min":0.79400000000000004},"min":{"1min":0.36499999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.1839999999999999,"5min":1.667,"15min":1.667},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.92100000000000004,"15min":0.93500000000000005},"min":{"1min":0.45400000000000001,"5min":0.375,"15min":0.375},"max":{"1min":1.7070000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":1.0740000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.028,"15min":1.052},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1240000000000001,"15min":1.1990000000000001},"min":{"1min":0.51200000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.222}]},{"osd":3,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.96299999999999997,"15min":0.98899999999999999},"min":{"1min":0.50800000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.4690000000000001,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.117,"5min":1.1539999999999999,"15min":1.2010000000000001},"min":{"1min":0.57699999999999996,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.4370000000000001,"5min":2.996,"15min":2.996},"last":1.3839999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.2829999999999999,"15min":1.321},"min":{"1min":0.93200000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0619999999999998,"5min":2.7120000000000002,"15min":2.7120000000000002},"last":1.264},{"interface":"front","average":{"1min":1.26,"5min":1.276,"15min":1.3839999999999999},"min":{"1min":0.93999999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.6719999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.3540000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.3149999999999999,"15min":1.3939999999999999},"min":{"1min":0.88600000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.7110000000000001,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.494},{"interface":"front","average":{"1min":1.26,"5min":1.248,"15min":1.3149999999999999},"min":{"1min":0.89900000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5489999999999999,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":1.3100000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.3360000000000001,"15min":1.391},"min":{"1min":0.90100000000000002,"5min":0.623,"15min":0.623},"max":{"1min":1.7569999999999999,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":1.53},{"interface":"front","average":{"1min":1.458,"5min":1.379,"15min":1.4630000000000001},"min":{"1min":0.88100000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.0099999999999998,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.4610000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:46:30 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4430000000000001,"15min":1.5589999999999999},"min":{"1min":0.96899999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.1139999999999999,"5min":3.343,"15min":3.343},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.411,"15min":1.4950000000000001},"min":{"1min":0.84999999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.9279999999999999,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.427}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32588,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339252,"statfs":{"total":96636764160,"available":96603394048,"internally_reserved":0,"allocated":1507328,"data_stored":979573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.90000000000000002,"15min":0.86899999999999999},"min":{"1min":0.41299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.722,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.024},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91500000000000004,"15min":0.94799999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.274}]},{"osd":2,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.93799999999999994,"15min":0.97899999999999998},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2929999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.282,"5min":1.109,"15min":1.119},"min":{"1min":0.77400000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.70599999999999996}]},{"osd":3,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1679999999999999,"15min":1.1719999999999999},"min":{"1min":0.72899999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.7190000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.597},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.161,"15min":1.2330000000000001},"min":{"1min":0.67600000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.679,"5min":3.149,"15min":3.149},"last":0.94599999999999995}]},{"osd":4,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.3680000000000001,"15min":1.3700000000000001},"min":{"1min":1.0900000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.962,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.093},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.2949999999999999,"15min":1.323},"min":{"1min":0.872,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.7,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":1.456}]},{"osd":5,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3109999999999999,"15min":1.353},"min":{"1min":1.0269999999999999,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.8109999999999999,"5min":2.577,"15min":2.577},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.361,"15min":1.377},"min":{"1min":1.1439999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.885,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":2.0950000000000002}]},{"osd":6,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.397,"15min":1.4430000000000001},"min":{"1min":1.091,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.008,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":1.9219999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4370000000000001,"15min":1.5229999999999999},"min":{"1min":0.98199999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.032,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.3819999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4590000000000001,"15min":1.4750000000000001},"min":{"1min":0.95199999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.98,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.8919999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.4930000000000001,"15min":1.5509999999999999},"min":{"1min":1.071,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.911,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.6639999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":929792,"data_stored":395575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79000000000000004,"15min":0.81399999999999995},"min":{"1min":0.42899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.26,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.93000000000000005,"15min":0.96999999999999997},"min":{"1min":0.55300000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.94199999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.51300000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.0269999999999999,"15min":1.0940000000000001},"min":{"1min":0.55800000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4850000000000001,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.73699999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.117,"15min":1.1830000000000001},"min":{"1min":0.754,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.79,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.167,"15min":1.1890000000000001},"min":{"1min":0.67700000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.6140000000000001,"5min":2.407,"15min":2.407},"last":0.67900000000000005}]},{"osd":4,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3200000000000001,"15min":1.407},"min":{"1min":0.90000000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.601,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.256},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.373,"15min":1.474},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.669,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.268}]},{"osd":5,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3160000000000001,"15min":1.391},"min":{"1min":1.0209999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.7290000000000001,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2629999999999999},"min":{"1min":0.77000000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.6659999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.071}]},{"osd":6,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3380000000000001,"15min":1.3600000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.752,"5min":2.335,"15min":2.335},"last":1.325},{"interface":"front","average":{"1min":1.415,"5min":1.401,"15min":1.456},"min":{"1min":0.99199999999999999,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.921,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.381}]},{"osd":7,"last update":"Wed Jan 24 17:45:34 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.429,"15min":1.5229999999999999},"min":{"1min":0.89700000000000002,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.788,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.5049999999999999,"15min":1.5940000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.859,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.4890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:46:36.553 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-24T17:46:36.553 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-24T17:46:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.826+0000 7f7e59e00640 1 Processor -- start 2024-01-24T17:46:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.826+0000 7f7e59e00640 1 -- start start 2024-01-24T17:46:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.827+0000 7f7e59e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e5415cdb0 0x7f7e540d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.827+0000 7f7e59e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e540c63c0 0x7f7e540ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.827+0000 7f7e59e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e540b01b0 con 0x7f7e540d0650 2024-01-24T17:46:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.827+0000 7f7e59e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5415d190 con 0x7f7e5415cdb0 2024-01-24T17:46:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.827+0000 7f7e59e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540b0490 con 0x7f7e540c63c0 2024-01-24T17:46:36.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.828+0000 7f7e53e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7e540d0650 0x7f7e540b1250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60168/0 (socket says 172.21.15.39:60168) 2024-01-24T17:46:36.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.828+0000 7f7e53e00640 1 -- 172.21.15.39:0/3694993543 learned_addr learned my addr 172.21.15.39:0/3694993543 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.828+0000 7f7e59400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e540c63c0 0x7f7e540ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.828+0000 7f7e58a00640 1 --2- 172.21.15.39:0/3694993543 >> v2:172.21.15.100:3300/0 conn(0x7f7e5415cdb0 0x7f7e540d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.828+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689043534 0 0) 0x7f7e540b01b0 con 0x7f7e540d0650 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.828+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e34003520 con 0x7f7e540d0650 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4190451938 0 0) 0x7f7e34003520 con 0x7f7e540d0650 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e540c63c0 msgr2=0x7f7e540ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e540c63c0 0x7f7e540ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 >> v2:172.21.15.100:3300/0 conn(0x7f7e5415cdb0 msgr2=0x7f7e540d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 --2- 172.21.15.39:0/3694993543 >> v2:172.21.15.100:3300/0 conn(0x7f7e5415cdb0 0x7f7e540d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e540b0770 con 0x7f7e540d0650 2024-01-24T17:46:36.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7e3c001670 con 0x7f7e540d0650 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 >> v1:172.21.15.39:6789/0 conn(0x7f7e540d0650 legacy=0x7f7e540b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34003970 0x7f7e34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e540c63c0 0x7f7e340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e34004d00 0x7f7e34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540b0490 con 0x7f7e34004d00 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3400b980 con 0x7f7e34003970 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3400bac0 con 0x7f7e540c63c0 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.829+0000 7f7e59400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e34004d00 0x7f7e34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.830+0000 7f7e58a00640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34003970 0x7f7e34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.830+0000 7f7e53e00640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e540c63c0 0x7f7e340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.830+0000 7f7e59400640 1 -- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e540c63c0 msgr2=0x7f7e340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.830+0000 7f7e59400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e540c63c0 0x7f7e340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.830+0000 7f7e59400640 1 -- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34003970 msgr2=0x7f7e34003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:36.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.830+0000 7f7e59400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34003970 0x7f7e34003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.830+0000 7f7e59400640 1 -- 172.21.15.39:0/3694993543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e34003520 con 0x7f7e34004d00 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.831+0000 7f7e59400640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e34004d00 0x7f7e34005180 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009fc0 tx=0x7f7e340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea2ab1f98994584 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.831+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c02cc20 con 0x7f7e34004d00 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.831+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c00f040 con 0x7f7e34004d00 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.831+0000 7f7e53400640 1 -- 172.21.15.39:0/3694993543 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c034440 con 0x7f7e34004d00 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.832+0000 7f7e59e00640 1 -- 172.21.15.39:0/3694993543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e34004d00 msgr2=0x7f7e34005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.832+0000 7f7e59e00640 1 --2- 172.21.15.39:0/3694993543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e34004d00 0x7f7e34005180 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009fc0 tx=0x7f7e340057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.832+0000 7f7e59400640 1 -- 172.21.15.39:0/3694993543 reap_dead start 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.832+0000 7f7e59e00640 1 -- 172.21.15.39:0/3694993543 shutdown_connections 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.832+0000 7f7e59e00640 1 -- 172.21.15.39:0/3694993543 >> 172.21.15.39:0/3694993543 conn(0x7f7e540d2900 msgr2=0x7f7e540d2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.832+0000 7f7e59e00640 1 -- 172.21.15.39:0/3694993543 shutdown_connections 2024-01-24T17:46:36.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.832+0000 7f7e59e00640 1 -- 172.21.15.39:0/3694993543 wait complete. 2024-01-24T17:46:36.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.834+0000 7f7e59e00640 1 Processor -- start 2024-01-24T17:46:36.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.834+0000 7f7e59e00640 1 -- start start 2024-01-24T17:46:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.835+0000 7f7e59e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34004d00 0x7f7e54157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.835+0000 7f7e59e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e54154f50 0x7f7e540cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.835+0000 7f7e59e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e541583d0 0x7f7e54158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.835+0000 7f7e59e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540cc800 con 0x7f7e541583d0 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.835+0000 7f7e59e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e54158dc0 con 0x7f7e34004d00 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.835+0000 7f7e59e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5409d6e0 con 0x7f7e54154f50 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e541583d0 0x7f7e54158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e541583d0 0x7f7e54158880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47288/0 (socket says 172.21.15.39:47288) 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 -- 172.21.15.39:0/3937612072 learned_addr learned my addr 172.21.15.39:0/3937612072 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e58a00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34004d00 0x7f7e54157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e53e00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e54154f50 0x7f7e540cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 -- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e54154f50 msgr2=0x7f7e540cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e54154f50 0x7f7e540cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 -- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34004d00 msgr2=0x7f7e54157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34004d00 0x7f7e54157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:36.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 -- 172.21.15.39:0/3937612072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e540cbbf0 con 0x7f7e541583d0 2024-01-24T17:46:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.836+0000 7f7e59400640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e541583d0 0x7f7e54158880 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00e230 tx=0x7f7e4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.837+0000 7f7e50c00640 1 -- 172.21.15.39:0/3937612072 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0349c0 con 0x7f7e541583d0 2024-01-24T17:46:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.837+0000 7f7e50c00640 1 -- 172.21.15.39:0/3937612072 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c034b20 con 0x7f7e541583d0 2024-01-24T17:46:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.837+0000 7f7e50c00640 1 -- 172.21.15.39:0/3937612072 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c003990 con 0x7f7e541583d0 2024-01-24T17:46:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.837+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e540cbed0 con 0x7f7e541583d0 2024-01-24T17:46:36.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.837+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e540cc490 con 0x7f7e541583d0 2024-01-24T17:46:36.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.838+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e540b0a30 con 0x7f7e541583d0 2024-01-24T17:46:36.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.839+0000 7f7e50c00640 1 -- 172.21.15.39:0/3937612072 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7e4c047460 con 0x7f7e541583d0 2024-01-24T17:46:36.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.841+0000 7f7e50c00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2c06a460 0x7f7e2c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:36.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.841+0000 7f7e50c00640 1 -- 172.21.15.39:0/3937612072 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e4c0aa6f0 con 0x7f7e541583d0 2024-01-24T17:46:36.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.841+0000 7f7e58a00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2c06a460 0x7f7e2c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:36.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.842+0000 7f7e58a00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2c06a460 0x7f7e2c06c920 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009a80 tx=0x7f7e44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:36.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:36.845+0000 7f7e50c00640 1 -- 172.21.15.39:0/3937612072 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7e540b0a30 con 0x7f7e541583d0 2024-01-24T17:46:37.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.108+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7e540b0a30 con 0x7f7e541583d0 2024-01-24T17:46:37.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.115+0000 7f7e50c00640 1 -- 172.21.15.39:0/3937612072 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7e540b0a30 con 0x7f7e541583d0 2024-01-24T17:46:37.115 INFO:teuthology.orchestra.run.smithi039.stderr:updated caps for client.0 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2c06a460 msgr2=0x7f7e2c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2c06a460 0x7f7e2c06c920 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009a80 tx=0x7f7e44008040 comp rx=0 tx=0).stop 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e541583d0 msgr2=0x7f7e54158880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e541583d0 0x7f7e54158880 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00e230 tx=0x7f7e4c009750 comp rx=0 tx=0).stop 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 shutdown_connections 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7e2c06a460 0x7f7e2c06c920 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e541583d0 0x7f7e54158880 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e54154f50 0x7f7e540cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 --2- 172.21.15.39:0/3937612072 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e34004d00 0x7f7e54157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 >> 172.21.15.39:0/3937612072 conn(0x7f7e540d2900 msgr2=0x7f7e540c88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:37.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 shutdown_connections 2024-01-24T17:46:37.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.118+0000 7f7e59e00640 1 -- 172.21.15.39:0/3937612072 wait complete. 2024-01-24T17:46:37.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:46:37.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.336+0000 7fa1bde00640 1 Processor -- start 2024-01-24T17:46:37.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.336+0000 7fa1bde00640 1 -- start start 2024-01-24T17:46:37.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.338+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80d05d0 0x7fa1b80d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:37.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.338+0000 7fa1bde00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa1b80b1bf0 0x7fa1b80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:37.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.338+0000 7fa1bde00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b80ce2f0 con 0x7fa1b80cf200 2024-01-24T17:46:37.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.338+0000 7fa1bde00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b80cd440 con 0x7fa1b80b1bf0 2024-01-24T17:46:37.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.338+0000 7fa1bde00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80ce780 con 0x7fa1b80d05d0 2024-01-24T17:46:37.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.339+0000 7fa1bd400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa1b80b1bf0 0x7fa1b80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:37.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.339+0000 7fa1afc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80d05d0 0x7fa1b80d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:37.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.339+0000 7fa1bd400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa1b80b1bf0 0x7fa1b80b2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:49644/0 (socket says 172.21.15.39:49644) 2024-01-24T17:46:37.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.339+0000 7fa1bd400640 1 -- 172.21.15.39:0/1371887001 learned_addr learned my addr 172.21.15.39:0/1371887001 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:37.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.340+0000 7fa1afc00640 1 -- 172.21.15.39:0/1371887001 >> v2:172.21.15.100:3300/0 conn(0x7fa1b80b1bf0 msgr2=0x7fa1b80b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:37.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.340+0000 7fa1afc00640 1 --2- 172.21.15.39:0/1371887001 >> v2:172.21.15.100:3300/0 conn(0x7fa1b80b1bf0 0x7fa1b80b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.340+0000 7fa1afc00640 1 -- 172.21.15.39:0/1371887001 >> v1:172.21.15.100:6789/0 conn(0x7fa1b80cf200 legacy=0x7fa1b80cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:37.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.340+0000 7fa1afc00640 1 -- 172.21.15.39:0/1371887001 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b80cd6e0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.341+0000 7fa1afc00640 1 --2- 172.21.15.39:0/1371887001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80d05d0 0x7fa1b80d3310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00097d0 tx=0x7fa1a002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8f1fa2c28fa95cc server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:37.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.342+0000 7fa1b7e00640 1 -- 172.21.15.39:0/1371887001 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0031890 con 0x7fa1b80d05d0 2024-01-24T17:46:37.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.342+0000 7fa1b7e00640 1 -- 172.21.15.39:0/1371887001 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a00319f0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.342+0000 7fa1b7e00640 1 -- 172.21.15.39:0/1371887001 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0031d80 con 0x7fa1b80d05d0 2024-01-24T17:46:37.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.342+0000 7fa1bde00640 1 -- 172.21.15.39:0/1371887001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80d05d0 msgr2=0x7fa1b80d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:37.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.342+0000 7fa1bde00640 1 --2- 172.21.15.39:0/1371887001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80d05d0 0x7fa1b80d3310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00097d0 tx=0x7fa1a002b930 comp rx=0 tx=0).stop 2024-01-24T17:46:37.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.343+0000 7fa1bde00640 1 -- 172.21.15.39:0/1371887001 shutdown_connections 2024-01-24T17:46:37.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.343+0000 7fa1bde00640 1 --2- 172.21.15.39:0/1371887001 >> v2:172.21.15.100:3300/0 conn(0x7fa1b80b1bf0 0x7fa1b80b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.343+0000 7fa1bde00640 1 --2- 172.21.15.39:0/1371887001 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80d05d0 0x7fa1b80d3310 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.343+0000 7fa1bde00640 1 -- 172.21.15.39:0/1371887001 >> 172.21.15.39:0/1371887001 conn(0x7fa1b80d27f0 msgr2=0x7fa1b80d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:37.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.343+0000 7fa1bde00640 1 -- 172.21.15.39:0/1371887001 shutdown_connections 2024-01-24T17:46:37.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.343+0000 7fa1bde00640 1 -- 172.21.15.39:0/1371887001 wait complete. 2024-01-24T17:46:37.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.345+0000 7fa1bde00640 1 Processor -- start 2024-01-24T17:46:37.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.345+0000 7fa1bde00640 1 -- start start 2024-01-24T17:46:37.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.346+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80b1bf0 0x7fa1b8156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.346+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80cf200 0x7fa1b815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.346+0000 7fa1bde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80d05d0 0x7fa1b80b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.346+0000 7fa1bde00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b815aa80 con 0x7fa1b80d05d0 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.346+0000 7fa1bde00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b8158750 con 0x7fa1b80b1bf0 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.346+0000 7fa1bde00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b80b57b0 con 0x7fa1b80cf200 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80d05d0 0x7fa1b80b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80d05d0 0x7fa1b80b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47298/0 (socket says 172.21.15.39:47298) 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bd400640 1 -- 172.21.15.39:0/202130462 learned_addr learned my addr 172.21.15.39:0/202130462 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bca00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80b1bf0 0x7fa1b8156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1afc00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80cf200 0x7fa1b815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bd400640 1 -- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80cf200 msgr2=0x7fa1b815cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:37.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bd400640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80cf200 0x7fa1b815cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bd400640 1 -- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80b1bf0 msgr2=0x7fa1b8156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bd400640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80b1bf0 0x7fa1b8156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1bd400640 1 -- 172.21.15.39:0/202130462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a0009400 con 0x7fa1b80d05d0 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.347+0000 7fa1afc00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80cf200 0x7fa1b815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.348+0000 7fa1bd400640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80d05d0 0x7fa1b80b5db0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009d90 tx=0x7fa1b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.348+0000 7fa1b5600640 1 -- 172.21.15.39:0/202130462 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b000cd20 con 0x7fa1b80d05d0 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.348+0000 7fa1b5600640 1 -- 172.21.15.39:0/202130462 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b000f040 con 0x7fa1b80d05d0 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.348+0000 7fa1b5600640 1 -- 172.21.15.39:0/202130462 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0003bb0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.348+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b81589c0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.348+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b8157310 con 0x7fa1b80d05d0 2024-01-24T17:46:37.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.349+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b8158ee0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.350+0000 7fa1b5600640 1 -- 172.21.15.39:0/202130462 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa1b0003d10 con 0x7fa1b80d05d0 2024-01-24T17:46:37.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.351+0000 7fa1b5600640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:37.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.351+0000 7fa1b5600640 1 -- 172.21.15.39:0/202130462 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1b0088fd0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.352+0000 7fa1bca00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:37.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.353+0000 7fa1bca00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80b87d0 tx=0x7fa1a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:37.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.355+0000 7fa1b5600640 1 -- 172.21.15.39:0/202130462 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa1b8158ee0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.587+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1b8158ee0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.588+0000 7fa1b5600640 1 -- 172.21.15.39:0/202130462 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fa1b8158ee0 con 0x7fa1b80d05d0 2024-01-24T17:46:37.589 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:37.589 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1886779141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1886779141},{"type":"v1","addr":"172.21.15.39:6833","nonce":1886779141}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5854,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2624477336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":2624477336},{"type":"v1","addr":"172.21.15.100:6837","nonce":2624477336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2465319794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2465319794},{"type":"v1","addr":"172.21.15.39:6835","nonce":2465319794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:46:31.015014+0000","modified":"2024-01-24T17:46:34.537399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6836/2467066037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2467066037},{"type":"v1","addr":"172.21.15.100:6836","nonce":2467066037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-24T17:46:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 42 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 msgr2=0x7fa19006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80b87d0 tx=0x7fa1a8009040 comp rx=0 tx=0).stop 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80d05d0 msgr2=0x7fa1b80b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80d05d0 0x7fa1b80b5db0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009d90 tx=0x7fa1b000c490 comp rx=0 tx=0).stop 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 shutdown_connections 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa19006a430 0x7fa19006c8f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1b80d05d0 0x7fa1b80b5db0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1b80cf200 0x7fa1b815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 --2- 172.21.15.39:0/202130462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1b80b1bf0 0x7fa1b8156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 >> 172.21.15.39:0/202130462 conn(0x7fa1b80d27f0 msgr2=0x7fa1b80af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 shutdown_connections 2024-01-24T17:46:37.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:37.591+0000 7fa1bde00640 1 -- 172.21.15.39:0/202130462 wait complete. 2024-01-24T17:46:38.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:46:31.015014+0000', 'modified': '2024-01-24T17:46:34.537399+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6804}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6804': {'gid': 6804, 'name': 'd', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.100:6836/2467066037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6834', 'nonce': 2467066037}, {'type': 'v1', 'addr': '172.21.15.100:6836', 'nonce': 2467066037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:46:38.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T17:46:38.014 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-24T17:46:38.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.243+0000 7f5027200640 1 Processor -- start 2024-01-24T17:46:38.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.243+0000 7f5027200640 1 -- start start 2024-01-24T17:46:38.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.244+0000 7f5027200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f50280cf8a0 0x7f50280d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.244+0000 7f5027200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f502806acb0 0x7f50280d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5027200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50280ce010 con 0x7f502815cbf0 2024-01-24T17:46:38.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5027200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50280ce870 con 0x7f502806acb0 2024-01-24T17:46:38.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5027200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50280ce2b0 con 0x7f50280cf8a0 2024-01-24T17:46:38.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5025e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f502815cbf0 0x7f50280cf190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60184/0 (socket says 172.21.15.39:60184) 2024-01-24T17:46:38.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5025e00640 1 -- 172.21.15.39:0/3380658594 learned_addr learned my addr 172.21.15.39:0/3380658594 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:38.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5026800640 1 --2- 172.21.15.39:0/3380658594 >> v2:172.21.15.39:3300/0 conn(0x7f502806acb0 0x7f50280d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5025400640 1 --2- 172.21.15.39:0/3380658594 >> v2:172.21.15.100:3300/0 conn(0x7f50280cf8a0 0x7f50280d0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5024a00640 1 -- 172.21.15.39:0/3380658594 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168468286 0 0) 0x7f50280ce010 con 0x7f502815cbf0 2024-01-24T17:46:38.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5024a00640 1 -- 172.21.15.39:0/3380658594 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5008003520 con 0x7f502815cbf0 2024-01-24T17:46:38.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5026800640 1 -- 172.21.15.39:0/3380658594 >> v2:172.21.15.100:3300/0 conn(0x7f50280cf8a0 msgr2=0x7f50280d0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5026800640 1 --2- 172.21.15.39:0/3380658594 >> v2:172.21.15.100:3300/0 conn(0x7f50280cf8a0 0x7f50280d0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.245+0000 7f5026800640 1 -- 172.21.15.39:0/3380658594 >> v1:172.21.15.39:6789/0 conn(0x7f502815cbf0 legacy=0x7f50280cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.246+0000 7f5026800640 1 -- 172.21.15.39:0/3380658594 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50280ce590 con 0x7f502806acb0 2024-01-24T17:46:38.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.247+0000 7f5026800640 1 --2- 172.21.15.39:0/3380658594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502806acb0 0x7f50280d0fd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f50280b53b0 tx=0x7f502002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdedc2da1173f944 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.247+0000 7f5024a00640 1 -- 172.21.15.39:0/3380658594 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502002e070 con 0x7f502806acb0 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.247+0000 7f5024a00640 1 -- 172.21.15.39:0/3380658594 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50200315a0 con 0x7f502806acb0 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.247+0000 7f5024a00640 1 -- 172.21.15.39:0/3380658594 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5020031930 con 0x7f502806acb0 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.248+0000 7f5027200640 1 -- 172.21.15.39:0/3380658594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502806acb0 msgr2=0x7f50280d0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.248+0000 7f5027200640 1 --2- 172.21.15.39:0/3380658594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502806acb0 0x7f50280d0fd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f50280b53b0 tx=0x7f502002b840 comp rx=0 tx=0).stop 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.248+0000 7f5027200640 1 -- 172.21.15.39:0/3380658594 shutdown_connections 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.248+0000 7f5027200640 1 --2- 172.21.15.39:0/3380658594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502806acb0 0x7f50280d0fd0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.248+0000 7f5027200640 1 --2- 172.21.15.39:0/3380658594 >> v2:172.21.15.100:3300/0 conn(0x7f50280cf8a0 0x7f50280d0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.248+0000 7f5027200640 1 -- 172.21.15.39:0/3380658594 >> 172.21.15.39:0/3380658594 conn(0x7f502809bad0 msgr2=0x7f50280d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.248+0000 7f5027200640 1 -- 172.21.15.39:0/3380658594 shutdown_connections 2024-01-24T17:46:38.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.248+0000 7f5027200640 1 -- 172.21.15.39:0/3380658594 wait complete. 2024-01-24T17:46:38.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.250+0000 7f5027200640 1 Processor -- start 2024-01-24T17:46:38.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.250+0000 7f5027200640 1 -- start start 2024-01-24T17:46:38.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5027200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502806acb0 0x7f50280a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5027200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50280cf8a0 0x7f50280c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5027200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502815cbf0 0x7f502809e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5027200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502809fcd0 con 0x7f502815cbf0 2024-01-24T17:46:38.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5027200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50280c8ac0 con 0x7f50280cf8a0 2024-01-24T17:46:38.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5027200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502809f050 con 0x7f502806acb0 2024-01-24T17:46:38.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5026800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502815cbf0 0x7f502809e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5026800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502815cbf0 0x7f502809e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47308/0 (socket says 172.21.15.39:47308) 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5026800640 1 -- 172.21.15.39:0/3355410511 learned_addr learned my addr 172.21.15.39:0/3355410511 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5025e00640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502806acb0 0x7f50280a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.251+0000 7f5025400640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50280cf8a0 0x7f50280c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f5026800640 1 -- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502806acb0 msgr2=0x7f50280a00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f5026800640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502806acb0 0x7f50280a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f5026800640 1 -- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50280cf8a0 msgr2=0x7f50280c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f5026800640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50280cf8a0 0x7f50280c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f5026800640 1 -- 172.21.15.39:0/3355410511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5020009400 con 0x7f502815cbf0 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f5026800640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502815cbf0 0x7f502809e570 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f5020009890 tx=0x7f50200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f501e000640 1 -- 172.21.15.39:0/3355410511 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5020032840 con 0x7f502815cbf0 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f501e000640 1 -- 172.21.15.39:0/3355410511 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50200329a0 con 0x7f502815cbf0 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f5027200640 1 -- 172.21.15.39:0/3355410511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502809f330 con 0x7f502815cbf0 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f501e000640 1 -- 172.21.15.39:0/3355410511 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5020032d30 con 0x7f502815cbf0 2024-01-24T17:46:38.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.252+0000 7f5027200640 1 -- 172.21.15.39:0/3355410511 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502809f8f0 con 0x7f502815cbf0 2024-01-24T17:46:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.255+0000 7f501e000640 1 -- 172.21.15.39:0/3355410511 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f502004e430 con 0x7f502815cbf0 2024-01-24T17:46:38.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.255+0000 7f501e000640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f500006a430 0x7f500006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.256+0000 7f501e000640 1 -- 172.21.15.39:0/3355410511 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50200aa880 con 0x7f502815cbf0 2024-01-24T17:46:38.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.256+0000 7f5025e00640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f500006a430 0x7f500006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.256+0000 7f502fe64740 1 Processor -- start 2024-01-24T17:46:38.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.256+0000 7f502fe64740 1 -- start start 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.257+0000 7f502fe64740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4bc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.257+0000 7f502fe64740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48c150 0x56006f4b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.257+0000 7f5025e00640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f500006a430 0x7f500006c8f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5018007fd0 tx=0x7f5018009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.257+0000 7f502fe64740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f48fd50 0x56006f4b9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.257+0000 7f502fe64740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006f4676c0 con 0x56006f4bc1d0 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.257+0000 7f502fe64740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006f3fc070 con 0x56006f48fd50 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.257+0000 7f502fe64740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56006f3f9800 con 0x56006f48c150 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4bc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4bc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47322/0 (socket says 172.21.15.39:47322) 2024-01-24T17:46:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 -- 172.21.15.39:0/2698092207 learned_addr learned my addr 172.21.15.39:0/2698092207 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5026800640 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f48fd50 0x56006f4b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025400640 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48c150 0x56006f4b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 -- 172.21.15.39:0/2698092207 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48c150 msgr2=0x56006f4b9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48c150 0x56006f4b9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 -- 172.21.15.39:0/2698092207 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f48fd50 msgr2=0x56006f4b9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f48fd50 0x56006f4b9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 -- 172.21.15.39:0/2698092207 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56006f3bc780 con 0x56006f4bc1d0 2024-01-24T17:46:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.258+0000 7f5025e00640 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4bc5b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f5018005cd0 tx=0x7f5018010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7e0b31062ab5a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f5016a00640 1 -- 172.21.15.39:0/2698092207 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018007650 con 0x56006f4bc1d0 2024-01-24T17:46:38.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f5016a00640 1 -- 172.21.15.39:0/2698092207 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50180077b0 con 0x56006f4bc1d0 2024-01-24T17:46:38.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f5016a00640 1 -- 172.21.15.39:0/2698092207 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018020450 con 0x56006f4bc1d0 2024-01-24T17:46:38.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 -- 172.21.15.39:0/2698092207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 msgr2=0x56006f4bc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4bc5b0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f5018005cd0 tx=0x7f5018010000 comp rx=0 tx=0).stop 2024-01-24T17:46:38.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 -- 172.21.15.39:0/2698092207 shutdown_connections 2024-01-24T17:46:38.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f48fd50 0x56006f4b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48c150 0x56006f4b9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 --2- 172.21.15.39:0/2698092207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4bc5b0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 -- 172.21.15.39:0/2698092207 >> 172.21.15.39:0/2698092207 conn(0x56006f48e460 msgr2=0x56006f4d6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 -- 172.21.15.39:0/2698092207 shutdown_connections 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.259+0000 7f502fe64740 1 -- 172.21.15.39:0/2698092207 wait complete. 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.260+0000 7f502fe64740 1 -- start start 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.260+0000 7f502fe64740 1 Processor -- start 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.261+0000 7f502fe64740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 0x56006f4b5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.261+0000 7f502fe64740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.261+0000 7f502fe64740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4a2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.261+0000 7f502fe64740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006f3f50f0 con 0x56006f4bc1d0 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.261+0000 7f502fe64740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006f3968c0 con 0x56006f4a2390 2024-01-24T17:46:38.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.261+0000 7f502fe64740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56006f396560 con 0x56006f48fd50 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.261+0000 7f502fe64740 10 client.0 _ll_register_callbacks cb 0x56006f3eea30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47328/0 (socket says 172.21.15.39:47328) 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 -- 172.21.15.39:0/151364837 learned_addr learned my addr 172.21.15.39:0/151364837 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025e00640 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 0x56006f4b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5026800640 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 -- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 msgr2=0x56006f4b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 0x56006f4b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 -- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 msgr2=0x56006f4a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 -- 172.21.15.39:0/151364837 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5008003520 con 0x56006f4bc1d0 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025e00640 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 0x56006f4b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f5025400640 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5bb0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f5010006fd0 tx=0x7f5010010700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daafe10336ff4652 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.262+0000 7f500f400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:46:38.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f500f400640 1 -- 172.21.15.39:0/151364837 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010004440 con 0x56006f4bc1d0 2024-01-24T17:46:38.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f500f400640 1 -- 172.21.15.39:0/151364837 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5010003610 con 0x56006f4bc1d0 2024-01-24T17:46:38.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 1 -- 172.21.15.39:0/151364837 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56006f41f640 con 0x56006f4bc1d0 2024-01-24T17:46:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f500f400640 1 -- 172.21.15.39:0/151364837 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5010010ed0 con 0x56006f4bc1d0 2024-01-24T17:46:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 10 client.6942 fetch_fsmap learned FSMap version 42 2024-01-24T17:46:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 1 -- 172.21.15.39:0/151364837 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x56006f3f3770 con 0x56006f4bc1d0 2024-01-24T17:46:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f500f400640 1 -- 172.21.15.39:0/151364837 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56006f41f640 con 0x56006f4bc1d0 2024-01-24T17:46:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f500f400640 1 -- 172.21.15.39:0/151364837 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f50100037d0 con 0x56006f4bc1d0 2024-01-24T17:46:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 10 client.6942 fetch_fsmap finished waiting for FSMap version 42 2024-01-24T17:46:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 10 client.6942 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-24T17:46:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 20 client.6942 populate_metadata read hostname 'smithi039' 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x56006f4bf1c0 0x56006f4f4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 4 client.6942 mds_command: new command op to 6804 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.263+0000 7f502fe64740 1 -- 172.21.15.39:0/151364837 --> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56006f3e7940 con 0x56006f4bf1c0 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.264+0000 7f5025e00640 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x56006f4bf1c0 0x56006f4f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.265+0000 7f5025e00640 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x56006f4bf1c0 0x56006f4f4180 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d777730ee86b5cd7 server_cookie=1bde8d9070c63b82 in_seq=0 out_seq=0 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.265+0000 7f500f400640 10 client.6942 ms_handle_connect on v2:172.21.15.100:6834/2467066037 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f500f400640 1 -- 172.21.15.39:0/151364837 <== mds.0 v2:172.21.15.100:6834/2467066037 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56006f3e7940 con 0x56006f4bf1c0 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f500f400640 10 client.6942 handle_command_reply: tid=0 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 client.6942 shutdown 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 -- 172.21.15.39:0/151364837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 msgr2=0x56006f4b5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5bb0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f5010006fd0 tx=0x7f5010010700 comp rx=0 tx=0).stop 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 -- 172.21.15.39:0/151364837 shutdown_connections 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x56006f4bf1c0 0x56006f4f4180 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5bb0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f500f400640 0 client.6942 ms_handle_reset on v2:172.21.15.100:6834/2467066037 2024-01-24T17:46:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 --2- 172.21.15.39:0/151364837 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 0x56006f4b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 -- 172.21.15.39:0/151364837 >> 172.21.15.39:0/151364837 conn(0x56006f48c150 msgr2=0x56006f4905d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 -- 172.21.15.39:0/151364837 shutdown_connections 2024-01-24T17:46:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.267+0000 7f502fe64740 1 -- 172.21.15.39:0/151364837 wait complete. 2024-01-24T17:46:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.268+0000 7f502fe64740 20 client.6942 trim_cache size 0 max 16384 2024-01-24T17:46:38.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.284+0000 7f502fe64740 1 Processor -- start 2024-01-24T17:46:38.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.284+0000 7f502fe64740 1 -- start start 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f502fe64740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 0x56006f4bd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f502fe64740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4bd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f502fe64740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f502fe64740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006f4c0440 con 0x56006f4bc1d0 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f502fe64740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006f15cf20 con 0x56006f4a2390 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f502fe64740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56006f4b6fc0 con 0x56006f48fd50 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f5026800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f5026800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47340/0 (socket says 172.21.15.39:47340) 2024-01-24T17:46:38.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f5026800640 1 -- 172.21.15.39:0/1877610381 learned_addr learned my addr 172.21.15.39:0/1877610381 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:38.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f5026800640 1 -- 172.21.15.39:0/1877610381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 msgr2=0x56006f4bd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f5026800640 1 --2- 172.21.15.39:0/1877610381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 0x56006f4bd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f5026800640 1 -- 172.21.15.39:0/1877610381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 msgr2=0x56006f4bd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f5026800640 1 --2- 172.21.15.39:0/1877610381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4bd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.285+0000 7f5026800640 1 -- 172.21.15.39:0/1877610381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56006f4292f0 con 0x56006f4bc1d0 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f5026800640 1 --2- 172.21.15.39:0/1877610381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5650 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f502002be50 tx=0x7f5020009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41bdb665156aba9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f5017400640 1 -- 172.21.15.39:0/1877610381 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50200a6070 con 0x56006f4bc1d0 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f5017400640 1 -- 172.21.15.39:0/1877610381 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5020079b20 con 0x56006f4bc1d0 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f5017400640 1 -- 172.21.15.39:0/1877610381 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502007d950 con 0x56006f4bc1d0 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f502fe64740 1 -- 172.21.15.39:0/1877610381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 msgr2=0x56006f4b5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f502fe64740 1 --2- 172.21.15.39:0/1877610381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5650 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f502002be50 tx=0x7f5020009190 comp rx=0 tx=0).stop 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f502fe64740 1 -- 172.21.15.39:0/1877610381 shutdown_connections 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f502fe64740 1 --2- 172.21.15.39:0/1877610381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f4bc1d0 0x56006f4b5650 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f502fe64740 1 --2- 172.21.15.39:0/1877610381 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4bd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.286+0000 7f502fe64740 1 --2- 172.21.15.39:0/1877610381 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f48fd50 0x56006f4bd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.287+0000 7f502fe64740 1 -- 172.21.15.39:0/1877610381 >> 172.21.15.39:0/1877610381 conn(0x56006f48c150 msgr2=0x56006f487980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:38.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.287+0000 7f502fe64740 1 -- 172.21.15.39:0/1877610381 shutdown_connections 2024-01-24T17:46:38.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.287+0000 7f502fe64740 1 -- 172.21.15.39:0/1877610381 wait complete. 2024-01-24T17:46:38.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.287+0000 7f502fe64740 1 -- start start 2024-01-24T17:46:38.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.287+0000 7f502fe64740 1 Processor -- start 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.288+0000 7f502fe64740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f48fd50 0x56006f4dab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.288+0000 7f502fe64740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.288+0000 7f502fe64740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f4bc1d0 0x56006f4ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.288+0000 7f502fe64740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006f4ebc20 con 0x56006f48fd50 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.288+0000 7f502fe64740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006f48b3c0 con 0x56006f4a2390 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.288+0000 7f502fe64740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56006f53df80 con 0x56006f4bc1d0 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.288+0000 7f502fe64740 10 client.0 _ll_register_callbacks cb 0x7f501000e170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f48fd50 0x56006f4dab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f48fd50 0x56006f4dab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47356/0 (socket says 172.21.15.39:47356) 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 -- 172.21.15.39:0/770113756 learned_addr learned my addr 172.21.15.39:0/770113756 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 -- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f4bc1d0 msgr2=0x56006f4ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f4bc1d0 0x56006f4ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 -- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 msgr2=0x56006f4eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 -- 172.21.15.39:0/770113756 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502809f8f0 con 0x56006f48fd50 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f5025e00640 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f48fd50 0x56006f4dab40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5018009010 tx=0x7f50180013d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aef12475f77a583 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.289+0000 7f500f400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f500f400640 1 -- 172.21.15.39:0/770113756 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018007650 con 0x56006f48fd50 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f500f400640 1 -- 172.21.15.39:0/770113756 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f501802d810 con 0x56006f48fd50 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 1 -- 172.21.15.39:0/770113756 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50180077b0 con 0x56006f48fd50 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f500f400640 1 -- 172.21.15.39:0/770113756 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f501802d9d0 con 0x56006f48fd50 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 10 client.6948 fetch_fsmap learned FSMap version 42 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 1 -- 172.21.15.39:0/770113756 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x56006f395ad0 con 0x56006f48fd50 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f500f400640 1 -- 172.21.15.39:0/770113756 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50180077b0 con 0x56006f48fd50 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f500f400640 1 -- 172.21.15.39:0/770113756 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f501802b050 con 0x56006f48fd50 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 10 client.6948 fetch_fsmap finished waiting for FSMap version 42 2024-01-24T17:46:38.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 10 client.6948 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-24T17:46:38.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 20 client.6948 populate_metadata read hostname 'smithi039' 2024-01-24T17:46:38.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x56006f4d87c0 0x56006f540af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 4 client.6948 mds_command: new command op to 6804 tid=0[{"prefix": "status"}] 2024-01-24T17:46:38.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.290+0000 7f502fe64740 1 -- 172.21.15.39:0/770113756 --> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56006f3ee170 con 0x56006f4d87c0 2024-01-24T17:46:38.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.291+0000 7f5025400640 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x56006f4d87c0 0x56006f540af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.292+0000 7f5025400640 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x56006f4d87c0 0x56006f540af0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10503a3744fb8cee server_cookie=4e7953e4450142ff in_seq=0 out_seq=0 2024-01-24T17:46:38.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.292+0000 7f500f400640 10 client.6948 ms_handle_connect on v2:172.21.15.100:6834/2467066037 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f500f400640 1 -- 172.21.15.39:0/770113756 <== mds.0 v2:172.21.15.100:6834/2467066037 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56006f3ee170 con 0x56006f4d87c0 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f500f400640 10 client.6948 handle_command_reply: tid=0 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 client.6948 shutdown 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 -- 172.21.15.39:0/770113756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f48fd50 msgr2=0x56006f4dab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f48fd50 0x56006f4dab40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5018009010 tx=0x7f50180013d0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 -- 172.21.15.39:0/770113756 shutdown_connections 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x56006f4d87c0 0x56006f540af0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56006f4bc1d0 0x56006f4ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56006f4a2390 0x56006f4eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 --2- 172.21.15.39:0/770113756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56006f48fd50 0x56006f4dab40 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f500f400640 0 client.6948 ms_handle_reset on v2:172.21.15.100:6834/2467066037 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 -- 172.21.15.39:0/770113756 >> 172.21.15.39:0/770113756 conn(0x56006f48c150 msgr2=0x56006f4b5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 -- 172.21.15.39:0/770113756 shutdown_connections 2024-01-24T17:46:38.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 1 -- 172.21.15.39:0/770113756 wait complete. 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.293+0000 7f502fe64740 20 client.6948 trim_cache size 0 max 16384 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 6804, 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:46:38.297 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 7.2728086709999999, 2024-01-24T17:46:38.298 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 42, 2024-01-24T17:46:38.298 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 76, 2024-01-24T17:46:38.298 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 76, 2024-01-24T17:46:38.298 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 13.120157469 2024-01-24T17:46:38.298 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 -- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f500006a430 msgr2=0x7f500006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f500006a430 0x7f500006c8f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5018007fd0 tx=0x7f5018009040 comp rx=0 tx=0).stop 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 -- 172.21.15.39:0/3355410511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502815cbf0 msgr2=0x7f502809e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502815cbf0 0x7f502809e570 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f5020009890 tx=0x7f50200046c0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 -- 172.21.15.39:0/3355410511 shutdown_connections 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f500006a430 0x7f500006c8f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502815cbf0 0x7f502809e570 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50280cf8a0 0x7f50280c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 --2- 172.21.15.39:0/3355410511 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f502806acb0 0x7f50280a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 -- 172.21.15.39:0/3355410511 >> 172.21.15.39:0/3355410511 conn(0x7f502809bad0 msgr2=0x7f50280b0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 -- 172.21.15.39:0/3355410511 shutdown_connections 2024-01-24T17:46:38.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.294+0000 7f5027200640 1 -- 172.21.15.39:0/3355410511 wait complete. 2024-01-24T17:46:38.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:46:38.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.524+0000 7f73cfe00640 1 Processor -- start 2024-01-24T17:46:38.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.524+0000 7f73cfe00640 1 -- start start 2024-01-24T17:46:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.525+0000 7f73cfe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f73d00ca320 0x7f73d015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.526+0000 7f73cfe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d01540f0 con 0x7f73d00cdaf0 2024-01-24T17:46:38.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.526+0000 7f73cfe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d015da00 con 0x7f73d00ca700 2024-01-24T17:46:38.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.526+0000 7f73cfe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d0159fd0 con 0x7f73d00ca320 2024-01-24T17:46:38.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.526+0000 7f73cf400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f73d00cdaf0 0x7f73d006bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60194/0 (socket says 172.21.15.39:60194) 2024-01-24T17:46:38.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.526+0000 7f73cf400640 1 -- 172.21.15.39:0/1486843985 learned_addr learned my addr 172.21.15.39:0/1486843985 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.526+0000 7f73cea00640 1 --2- 172.21.15.39:0/1486843985 >> v2:172.21.15.100:3300/0 conn(0x7f73d00ca320 0x7f73d015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.526+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053389054 0 0) 0x7f73d01540f0 con 0x7f73d00cdaf0 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.526+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b0003520 con 0x7f73d00cdaf0 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4234004692 0 0) 0x7f73b0003520 con 0x7f73d00cdaf0 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 >> v2:172.21.15.100:3300/0 conn(0x7f73d00ca320 msgr2=0x7f73d015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 --2- 172.21.15.39:0/1486843985 >> v2:172.21.15.100:3300/0 conn(0x7f73d00ca320 0x7f73d015d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 >> v1:172.21.15.100:6789/0 conn(0x7f73d00ca700 legacy=0x7f73d015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d015dce0 con 0x7f73d00cdaf0 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f73c80019f0 con 0x7f73d00cdaf0 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 >> v1:172.21.15.39:6789/0 conn(0x7f73d00cdaf0 legacy=0x7f73d006bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b0003970 0x7f73b0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73b00047c0 0x7f73b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00051f0 0x7f73b000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0159fd0 con 0x7f73b00047c0 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b000e360 con 0x7f73b0003970 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b000e520 con 0x7f73b00051f0 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.527+0000 7f73c7e00640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73b00047c0 0x7f73b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.528+0000 7f73cea00640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b0003970 0x7f73b0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.528+0000 7f73cf400640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00051f0 0x7f73b000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.528+0000 7f73c7e00640 1 -- 172.21.15.39:0/1486843985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00051f0 msgr2=0x7f73b000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.528+0000 7f73c7e00640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73b00051f0 0x7f73b000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.528+0000 7f73c7e00640 1 -- 172.21.15.39:0/1486843985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b0003970 msgr2=0x7f73b0003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.528+0000 7f73c7e00640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b0003970 0x7f73b0003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.528+0000 7f73c7e00640 1 -- 172.21.15.39:0/1486843985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b0003520 con 0x7f73b00047c0 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.528+0000 7f73cf400640 1 -- 172.21.15.39:0/1486843985 reap_dead start 2024-01-24T17:46:38.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.529+0000 7f73c7e00640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73b00047c0 0x7f73b0004c40 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f73b8001780 tx=0x7f73b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a43aed5ba096f0c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.529+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b80052d0 con 0x7f73b00047c0 2024-01-24T17:46:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.529+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b802e050 con 0x7f73b00047c0 2024-01-24T17:46:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.529+0000 7f73ce000640 1 -- 172.21.15.39:0/1486843985 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8003850 con 0x7f73b00047c0 2024-01-24T17:46:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.530+0000 7f73cfe00640 1 -- 172.21.15.39:0/1486843985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73b00047c0 msgr2=0x7f73b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.530+0000 7f73cfe00640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73b00047c0 0x7f73b0004c40 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f73b8001780 tx=0x7f73b00057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.530+0000 7f73cfe00640 1 -- 172.21.15.39:0/1486843985 shutdown_connections 2024-01-24T17:46:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.530+0000 7f73cfe00640 1 --2- 172.21.15.39:0/1486843985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73b00047c0 0x7f73b0004c40 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.530+0000 7f73cfe00640 1 -- 172.21.15.39:0/1486843985 >> 172.21.15.39:0/1486843985 conn(0x7f73d006a340 msgr2=0x7f73d00cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.530+0000 7f73cfe00640 1 -- 172.21.15.39:0/1486843985 shutdown_connections 2024-01-24T17:46:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.530+0000 7f73cfe00640 1 -- 172.21.15.39:0/1486843985 wait complete. 2024-01-24T17:46:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.532+0000 7f73cfe00640 1 Processor -- start 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.532+0000 7f73cfe00640 1 -- start start 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cfe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00047c0 0x7f73d01f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73d0153fb0 0x7f73d00a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d00a97a0 0x7f73d00a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00aa160 con 0x7f73d00a97a0 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cfe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00a81d0 con 0x7f73b00047c0 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00a84b0 con 0x7f73d0153fb0 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d00a97a0 0x7f73d00a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d00a97a0 0x7f73d00a9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47370/0 (socket says 172.21.15.39:47370) 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.533+0000 7f73cf400640 1 -- 172.21.15.39:0/1779107543 learned_addr learned my addr 172.21.15.39:0/1779107543 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73c7e00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73d0153fb0 0x7f73d00a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73cea00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00047c0 0x7f73d01f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73cf400640 1 -- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73d0153fb0 msgr2=0x7f73d00a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73cf400640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73d0153fb0 0x7f73d00a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73cf400640 1 -- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00047c0 msgr2=0x7f73d01f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73cf400640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00047c0 0x7f73d01f1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73cf400640 1 -- 172.21.15.39:0/1779107543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00a8790 con 0x7f73d00a97a0 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73c7e00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73d0153fb0 0x7f73d00a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73cf400640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d00a97a0 0x7f73d00a9c20 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f73d006e200 tx=0x7f73c8007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73c7400640 1 -- 172.21.15.39:0/1779107543 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800eaa0 con 0x7f73d00a97a0 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73c7400640 1 -- 172.21.15.39:0/1779107543 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800ec00 con 0x7f73d00a97a0 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73c7400640 1 -- 172.21.15.39:0/1779107543 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8013420 con 0x7f73d00a97a0 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.534+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d00a8a70 con 0x7f73d00a97a0 2024-01-24T17:46:38.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.535+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d00a7ec0 con 0x7f73d00a97a0 2024-01-24T17:46:38.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.536+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d004dff0 con 0x7f73d00a97a0 2024-01-24T17:46:38.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.537+0000 7f73c7400640 1 -- 172.21.15.39:0/1779107543 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f73c8013580 con 0x7f73d00a97a0 2024-01-24T17:46:38.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.537+0000 7f73c7400640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f73a806a430 0x7f73a806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:38.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.538+0000 7f73c7400640 1 -- 172.21.15.39:0/1779107543 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73c8088d50 con 0x7f73d00a97a0 2024-01-24T17:46:38.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.538+0000 7f73cea00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f73a806a430 0x7f73a806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:38.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.539+0000 7f73cea00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f73a806a430 0x7f73a806c8f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f73c00099c0 tx=0x7f73c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:38.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.542+0000 7f73c7400640 1 -- 172.21.15.39:0/1779107543 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f73c80542a0 con 0x7f73d00a97a0 2024-01-24T17:46:38.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.778+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f73d00a7760 con 0x7f73d00a97a0 2024-01-24T17:46:38.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.779+0000 7f73c7400640 1 -- 172.21.15.39:0/1779107543 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7f73c80578c0 con 0x7f73d00a97a0 2024-01-24T17:46:38.780 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f73a806a430 msgr2=0x7f73a806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f73a806a430 0x7f73a806c8f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f73c00099c0 tx=0x7f73c0008040 comp rx=0 tx=0).stop 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d00a97a0 msgr2=0x7f73d00a9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d00a97a0 0x7f73d00a9c20 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f73d006e200 tx=0x7f73c8007bc0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 shutdown_connections 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f73a806a430 0x7f73a806c8f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73d00a97a0 0x7f73d00a9c20 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f73d0153fb0 0x7f73d00a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 --2- 172.21.15.39:0/1779107543 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f73b00047c0 0x7f73d01f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 >> 172.21.15.39:0/1779107543 conn(0x7f73d006a340 msgr2=0x7f73d006bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 shutdown_connections 2024-01-24T17:46:38.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:38.782+0000 7f73cfe00640 1 -- 172.21.15.39:0/1779107543 wait complete. 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:46:38.801 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:46:38.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:38.802 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:46:38.852 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:46:38.852 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:46:38.852 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:46:38.852 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:38.852 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:46:38.852 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:38.852 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:38.852 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42333sec preferred_lft 42333sec 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:38.853 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:46:38.854 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:38.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:38.855 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:46:38.942 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:46:38.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:38.946 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:46:39.003 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:46:39.003 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:46:39.003 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:46:39.003 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:46:39.003 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:46:39.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:39.004 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:46:39.004 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:46:39.004 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:46:39.005 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:46:39.005 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:46:39.005 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:46:39.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:39.193 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:46:39.243 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:46:39.243 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:46:39.244 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:46:39.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:39.244 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:46:39.337 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:46:39.337 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:46:39.337 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:46:39.396 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:46:39.396 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:46:39.396 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:46:39.450 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:46:39.549 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:39.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:39.646 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:39.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:39.651 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:46:39.706 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:46:39.706 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:46:39.711 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:46:39.788 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:39.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:39.860 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:39.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:39.865 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:46:39.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:46:39.896+0000 7f12a52bb480 -1 init, newargv = 0x559fc09b1cb0 newargc=15 2024-01-24T17:46:39.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[63482]: starting ceph client 2024-01-24T17:46:39.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[63482]: starting fuse 2024-01-24T17:46:39.909 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:46:39.909 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:46:39.909 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:46:39.910 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:46:39.911 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:46:40.110 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.63482.asok 2024-01-24T17:46:40.290 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63482.asok 2024-01-24T17:46:40.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:40.290 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63482.asok status 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "63482", 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:46:40.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 6960, 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 6960 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:46:40.531 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1416859915 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1416859915 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.6960 192.168.0.1:0/1416859915", 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/1416859915", 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:46:40.532 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 42, 2024-01-24T17:46:40.533 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 76, 2024-01-24T17:46:40.533 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:46:40.533 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:46:40.533 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:46:40.533 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:46:40.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:40.545 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:46:40.545 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:46:40.603 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:46:40.604 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:46:40.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:40.604 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:46:40.698 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:46:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.920+0000 7f07b7200640 1 Processor -- start 2024-01-24T17:46:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.920+0000 7f07b7200640 1 -- start start 2024-01-24T17:46:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f07b0070540 0x7f07b015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b7200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f07b00b3e00 0x7f07b00c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b7200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b01551b0 con 0x7f07b00c8f70 2024-01-24T17:46:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b7200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b015e410 con 0x7f07b0070540 2024-01-24T17:46:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b7200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b015b1b0 con 0x7f07b00b3e00 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f07b0070540 0x7f07b015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b6800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f07b00c8f70 0x7f07b015e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60196/0 (socket says 172.21.15.39:60196) 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b6800640 1 -- 172.21.15.39:0/3562995197 learned_addr learned my addr 172.21.15.39:0/3562995197 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.922+0000 7f07b5400640 1 --2- 172.21.15.39:0/3562995197 >> v2:172.21.15.100:3300/0 conn(0x7f07b00b3e00 0x7f07b00c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.923+0000 7f07b5e00640 1 -- 172.21.15.39:0/3562995197 >> v2:172.21.15.100:3300/0 conn(0x7f07b00b3e00 msgr2=0x7f07b00c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.923+0000 7f07b5e00640 1 --2- 172.21.15.39:0/3562995197 >> v2:172.21.15.100:3300/0 conn(0x7f07b00b3e00 0x7f07b00c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.923+0000 7f07b4a00640 1 -- 172.21.15.39:0/3562995197 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782161358 0 0) 0x7f07b01551b0 con 0x7f07b00c8f70 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.923+0000 7f07b5e00640 1 -- 172.21.15.39:0/3562995197 >> v1:172.21.15.39:6789/0 conn(0x7f07b00c8f70 legacy=0x7f07b015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.923+0000 7f07b5e00640 1 -- 172.21.15.39:0/3562995197 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b015b480 con 0x7f07b0070540 2024-01-24T17:46:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.923+0000 7f07b5400640 1 --2- 172.21.15.39:0/3562995197 >> v2:172.21.15.100:3300/0 conn(0x7f07b00b3e00 0x7f07b00c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:40.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.924+0000 7f07b5e00640 1 --2- 172.21.15.39:0/3562995197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b0070540 0x7f07b015ac70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0798009790 tx=0x7f079802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1367aeaeaa5c1f7f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:40.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.924+0000 7f07b4a00640 1 -- 172.21.15.39:0/3562995197 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079802d040 con 0x7f07b0070540 2024-01-24T17:46:40.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.924+0000 7f07b4a00640 1 -- 172.21.15.39:0/3562995197 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798031810 con 0x7f07b0070540 2024-01-24T17:46:40.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.924+0000 7f07b4a00640 1 -- 172.21.15.39:0/3562995197 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798031ba0 con 0x7f07b0070540 2024-01-24T17:46:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.925+0000 7f07b7200640 1 -- 172.21.15.39:0/3562995197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b0070540 msgr2=0x7f07b015ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.925+0000 7f07b7200640 1 --2- 172.21.15.39:0/3562995197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b0070540 0x7f07b015ac70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0798009790 tx=0x7f079802b8b0 comp rx=0 tx=0).stop 2024-01-24T17:46:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.925+0000 7f07b7200640 1 -- 172.21.15.39:0/3562995197 shutdown_connections 2024-01-24T17:46:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.925+0000 7f07b7200640 1 --2- 172.21.15.39:0/3562995197 >> v2:172.21.15.100:3300/0 conn(0x7f07b00b3e00 0x7f07b00c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.925+0000 7f07b7200640 1 --2- 172.21.15.39:0/3562995197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b0070540 0x7f07b015ac70 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.925+0000 7f07b7200640 1 -- 172.21.15.39:0/3562995197 >> 172.21.15.39:0/3562995197 conn(0x7f07b006a340 msgr2=0x7f07b00cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.925+0000 7f07b7200640 1 -- 172.21.15.39:0/3562995197 shutdown_connections 2024-01-24T17:46:40.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.925+0000 7f07b7200640 1 -- 172.21.15.39:0/3562995197 wait complete. 2024-01-24T17:46:40.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.927+0000 7f07b7200640 1 Processor -- start 2024-01-24T17:46:40.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.927+0000 7f07b7200640 1 -- start start 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b7200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07b0070540 0x7f07b00c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b7200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b00b3e00 0x7f07b00c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b00c8f70 0x7f07b00c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00c8d40 con 0x7f07b00c8f70 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b7200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00c7ca0 con 0x7f07b00b3e00 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b7200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00c7f80 con 0x7f07b0070540 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b00c8f70 0x7f07b00c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b00c8f70 0x7f07b00c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47400/0 (socket says 172.21.15.39:47400) 2024-01-24T17:46:40.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b6800640 1 -- 172.21.15.39:0/1115196069 learned_addr learned my addr 172.21.15.39:0/1115196069 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b5400640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b00b3e00 0x7f07b00c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.928+0000 7f07b5e00640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07b0070540 0x7f07b00c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b6800640 1 -- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07b0070540 msgr2=0x7f07b00c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b6800640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07b0070540 0x7f07b00c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b6800640 1 -- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b00b3e00 msgr2=0x7f07b00c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b6800640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b00b3e00 0x7f07b00c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b6800640 1 -- 172.21.15.39:0/1115196069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07980093a0 con 0x7f07b00c8f70 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b5e00640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07b0070540 0x7f07b00c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b5400640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b00b3e00 0x7f07b00c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:40.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b6800640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b00c8f70 0x7f07b00c59f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f07a8001eb0 tx=0x7f07a800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:40.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07ae000640 1 -- 172.21.15.39:0/1115196069 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a80074e0 con 0x7f07b00c8f70 2024-01-24T17:46:40.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07ae000640 1 -- 172.21.15.39:0/1115196069 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a8007640 con 0x7f07b00c8f70 2024-01-24T17:46:40.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07ae000640 1 -- 172.21.15.39:0/1115196069 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a80049e0 con 0x7f07b00c8f70 2024-01-24T17:46:40.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b00c8260 con 0x7f07b00c8f70 2024-01-24T17:46:40.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.929+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b00c8820 con 0x7f07b00c8f70 2024-01-24T17:46:40.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.930+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b00c8a10 con 0x7f07b00c8f70 2024-01-24T17:46:40.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.932+0000 7f07ae000640 1 -- 172.21.15.39:0/1115196069 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f07a8004b40 con 0x7f07b00c8f70 2024-01-24T17:46:40.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.932+0000 7f07ae000640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f078806a430 0x7f078806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:40.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.932+0000 7f07ae000640 1 -- 172.21.15.39:0/1115196069 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07a8088650 con 0x7f07b00c8f70 2024-01-24T17:46:40.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.933+0000 7f07b5e00640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f078806a430 0x7f078806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:40.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.934+0000 7f07b5e00640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f078806a430 0x7f078806c8f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0798036000 tx=0x7f079803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:40.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:40.937+0000 7f07ae000640 1 -- 172.21.15.39:0/1115196069 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f07a8053c30 con 0x7f07b00c8f70 2024-01-24T17:46:41.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.176+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f07b004dff0 con 0x7f07b00c8f70 2024-01-24T17:46:41.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.176+0000 7f07ae000640 1 -- 172.21.15.39:0/1115196069 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7f07a8057250 con 0x7f07b00c8f70 2024-01-24T17:46:41.177 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:46:41.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f078806a430 msgr2=0x7f078806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:41.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f078806a430 0x7f078806c8f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0798036000 tx=0x7f079803c040 comp rx=0 tx=0).stop 2024-01-24T17:46:41.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b00c8f70 msgr2=0x7f07b00c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:41.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b00c8f70 0x7f07b00c59f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f07a8001eb0 tx=0x7f07a800c870 comp rx=0 tx=0).stop 2024-01-24T17:46:41.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 shutdown_connections 2024-01-24T17:46:41.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f078806a430 0x7f078806c8f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:41.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07b00c8f70 0x7f07b00c59f0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:41.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b00b3e00 0x7f07b00c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:41.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 --2- 172.21.15.39:0/1115196069 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f07b0070540 0x7f07b00c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:41.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.179+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 >> 172.21.15.39:0/1115196069 conn(0x7f07b006a340 msgr2=0x7f07b00c9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:41.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.180+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 shutdown_connections 2024-01-24T17:46:41.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:41.180+0000 7f07b7200640 1 -- 172.21.15.39:0/1115196069 wait complete. 2024-01-24T17:46:41.200 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:46:41.200 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:46:41.200 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:46:41.201 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-24T17:46:41.201 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:41.201 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:46:41.201 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:46:41.201 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:46:41.201 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:46:41.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:41.201 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:46:41.259 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:46:41.259 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:46:41.259 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:46:41.259 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:41.259 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:46:41.259 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42328sec preferred_lft 42328sec 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:41.260 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:46:41.261 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:46:41.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:41.262 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:46:41.350 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:46:41.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:41.355 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:46:41.404 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:46:41.405 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:46:41.405 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:46:41.405 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:46:41.405 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:46:41.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:41.406 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:46:41.406 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:46:41.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:46:41.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:46:41.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:46:41.406 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:46:41.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:41.588 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:46:41.638 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:46:41.638 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:46:41.638 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:46:41.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:41.639 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:46:41.731 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-24T17:46:41.731 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:46:41.731 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:41.787 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:46:41.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:46:41.788 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:41.837 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:46:41.929 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:41.929 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:42.026 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:42.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:42.031 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:46:42.081 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:46:42.081 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-24T17:46:42.085 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:46:42.156 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:46:42.156 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:46:42.225 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:46:42.228 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:46:42.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:42.228 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:46:42.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:46:42.260+0000 7f0475a3c480 -1 init, newargv = 0x556b6fc6ce20 newargc=15 2024-01-24T17:46:42.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[47154]: starting ceph client 2024-01-24T17:46:42.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[47154]: starting fuse 2024-01-24T17:46:42.273 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:46:42.273 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:46:42.273 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:46:42.273 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:46:42.273 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:46:42.273 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:46:42.274 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:46:42.275 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:46:42.275 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:46:42.275 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:46:42.275 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:46:42.275 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:46:42.275 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:46:42.275 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2024-01-24T17:46:42.275 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:46:42.463 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.47154.asok 2024-01-24T17:46:42.645 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.47154.asok 2024-01-24T17:46:42.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:42.646 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.47154.asok status 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "47154", 2024-01-24T17:46:42.891 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 5962, 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5962 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:46:42.892 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1710098733 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1710098733 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.5962 192.168.0.1:0/1710098733", 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/1710098733", 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 42, 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 76, 2024-01-24T17:46:42.893 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:46:42.894 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:46:42.894 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:46:42.894 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:46:42.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:42.906 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:46:42.906 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:42.959 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:46:42.959 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:42.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:42.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:43.056 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds ms_async_reap_threshold 1 2024-01-24T17:46:43.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.293+0000 7fb31b200640 1 Processor -- start 2024-01-24T17:46:43.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.294+0000 7fb31b200640 1 -- start start 2024-01-24T17:46:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.295+0000 7fb31b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 0x7fb31c0b1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.295+0000 7fb31b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c155170 con 0x7fb31c15ac70 2024-01-24T17:46:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.295+0000 7fb31b200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c15b1e0 con 0x7fb31c0ce8c0 2024-01-24T17:46:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.295+0000 7fb31b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c0b12b0 con 0x7fb31c070460 2024-01-24T17:46:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.295+0000 7fb31a800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb31c15ac70 0x7fb31c0b1c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55532/0 (socket says 172.21.15.39:55532) 2024-01-24T17:46:43.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb31a800640 1 -- 172.21.15.39:0/69316879 learned_addr learned my addr 172.21.15.39:0/69316879 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb319400640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 0x7fb31c0b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985491149 0 0) 0x7fb31c155170 con 0x7fb31c15ac70 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc003520 con 0x7fb31c15ac70 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3722810092 0 0) 0x7fb2fc003520 con 0x7fb31c15ac70 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 msgr2=0x7fb31c0b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 0x7fb31c0b1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 >> v1:172.21.15.100:6789/0 conn(0x7fb31c0ce8c0 legacy=0x7fb31c06fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c15b4c0 con 0x7fb31c15ac70 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fb3140019f0 con 0x7fb31c15ac70 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.296+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 >> v1:172.21.15.39:6789/0 conn(0x7fb31c15ac70 legacy=0x7fb31c0b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb318a00640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 0x7fb2fc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb318a00640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2fc003f20 0x7fb2fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb318a00640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2fc004d00 0x7fb2fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc00fe40 con 0x7fb2fc003f20 2024-01-24T17:46:43.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc00ffb0 con 0x7fb2fc004d00 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc010170 con 0x7fb31c070460 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb319400640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2fc003f20 0x7fb2fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb319e00640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 0x7fb2fc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb31a800640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2fc004d00 0x7fb2fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb319400640 1 -- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 msgr2=0x7fb2fc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb319400640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 0x7fb2fc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb319400640 1 -- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2fc004d00 msgr2=0x7fb2fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb319400640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2fc004d00 0x7fb2fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb319400640 1 -- 172.21.15.39:0/69316879 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc003520 con 0x7fb2fc003f20 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb31a800640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2fc004d00 0x7fb2fc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:43.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.297+0000 7fb319e00640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb31c070460 0x7fb2fc0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.299+0000 7fb319400640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2fc003f20 0x7fb2fc0047c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb304009ac0 tx=0x7fb2fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7c38352ab1e7e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.299+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30402c6b0 con 0x7fb2fc003f20 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.299+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30402c810 con 0x7fb2fc003f20 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.299+0000 7fb318a00640 1 -- 172.21.15.39:0/69316879 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30402cba0 con 0x7fb2fc003f20 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.299+0000 7fb31b200640 1 -- 172.21.15.39:0/69316879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2fc003f20 msgr2=0x7fb2fc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.299+0000 7fb31b200640 1 --2- 172.21.15.39:0/69316879 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2fc003f20 0x7fb2fc0047c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb304009ac0 tx=0x7fb2fc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.299+0000 7fb31a800640 1 -- 172.21.15.39:0/69316879 reap_dead start 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.300+0000 7fb31b200640 1 -- 172.21.15.39:0/69316879 shutdown_connections 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.300+0000 7fb31b200640 1 -- 172.21.15.39:0/69316879 >> 172.21.15.39:0/69316879 conn(0x7fb31c06a200 msgr2=0x7fb31c0affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.300+0000 7fb31b200640 1 -- 172.21.15.39:0/69316879 shutdown_connections 2024-01-24T17:46:43.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.300+0000 7fb31b200640 1 -- 172.21.15.39:0/69316879 wait complete. 2024-01-24T17:46:43.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.301+0000 7fb31b200640 1 Processor -- start 2024-01-24T17:46:43.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.302+0000 7fb31b200640 1 -- start start 2024-01-24T17:46:43.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.302+0000 7fb31b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2fc003f20 0x7fb31c151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb31b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb31c156d50 0x7fb31c1587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb31b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb31c158d00 0x7fb31c159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb31b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c15a2c0 con 0x7fb31c156d50 2024-01-24T17:46:43.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb31b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c15ce50 con 0x7fb31c158d00 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb31b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c1597b0 con 0x7fb2fc003f20 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb31c156d50 0x7fb31c1587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb31c156d50 0x7fb31c1587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44686/0 (socket says 172.21.15.39:44686) 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319400640 1 -- 172.21.15.39:0/4128959000 learned_addr learned my addr 172.21.15.39:0/4128959000 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319e00640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2fc003f20 0x7fb31c151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319400640 1 -- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2fc003f20 msgr2=0x7fb31c151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319400640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2fc003f20 0x7fb31c151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319400640 1 -- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb31c158d00 msgr2=0x7fb31c159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319400640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb31c158d00 0x7fb31c159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.303+0000 7fb319400640 1 -- 172.21.15.39:0/4128959000 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c159a90 con 0x7fb31c156d50 2024-01-24T17:46:43.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.304+0000 7fb319400640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb31c156d50 0x7fb31c1587c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fb304009ac0 tx=0x7fb304004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:43.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.304+0000 7fb312000640 1 -- 172.21.15.39:0/4128959000 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304003810 con 0x7fb31c156d50 2024-01-24T17:46:43.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.304+0000 7fb312000640 1 -- 172.21.15.39:0/4128959000 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb304003970 con 0x7fb31c156d50 2024-01-24T17:46:43.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.304+0000 7fb312000640 1 -- 172.21.15.39:0/4128959000 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304003d00 con 0x7fb31c156d50 2024-01-24T17:46:43.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.304+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c15a580 con 0x7fb31c156d50 2024-01-24T17:46:43.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.304+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c1571b0 con 0x7fb31c156d50 2024-01-24T17:46:43.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.306+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c0738c0 con 0x7fb31c156d50 2024-01-24T17:46:43.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.306+0000 7fb312000640 1 -- 172.21.15.39:0/4128959000 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb304040450 con 0x7fb31c156d50 2024-01-24T17:46:43.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.308+0000 7fb312000640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2f406a430 0x7fb2f406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.308+0000 7fb312000640 1 -- 172.21.15.39:0/4128959000 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3040aa8f0 con 0x7fb31c156d50 2024-01-24T17:46:43.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.308+0000 7fb319e00640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2f406a430 0x7fb2f406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.310+0000 7fb319e00640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2f406a430 0x7fb2f406c8f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb30c001630 tx=0x7fb30c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.313+0000 7fb312000640 1 -- 172.21.15.39:0/4128959000 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb304075ee0 con 0x7fb31c156d50 2024-01-24T17:46:43.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.508+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=ms_async_reap_threshold}] v 0) v1 -- 0x7fb31c0b3ea0 con 0x7fb31c156d50 2024-01-24T17:46:43.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.520+0000 7fb312000640 1 -- 172.21.15.39:0/4128959000 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=ms_async_reap_threshold}]=0 v19)=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7fb304003e60 con 0x7fb31c156d50 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2f406a430 msgr2=0x7fb2f406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2f406a430 0x7fb2f406c8f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb30c001630 tx=0x7fb30c009040 comp rx=0 tx=0).stop 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb31c156d50 msgr2=0x7fb31c1587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb31c156d50 0x7fb31c1587c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fb304009ac0 tx=0x7fb304004730 comp rx=0 tx=0).stop 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 shutdown_connections 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb31c158d00 0x7fb31c159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb2f406a430 0x7fb2f406c8f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb31c156d50 0x7fb31c1587c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 --2- 172.21.15.39:0/4128959000 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb2fc003f20 0x7fb31c151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 >> 172.21.15.39:0/4128959000 conn(0x7fb31c06a200 msgr2=0x7fb31c0af520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 shutdown_connections 2024-01-24T17:46:43.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.523+0000 7fb31b200640 1 -- 172.21.15.39:0/4128959000 wait complete. 2024-01-24T17:46:43.543 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:46:43.544 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:43.595 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:43.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.804+0000 7fa485800640 1 Processor -- start 2024-01-24T17:46:43.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.804+0000 7fa485800640 1 -- start start 2024-01-24T17:46:43.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.805+0000 7fa485800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa48015cbe0 0x7fa4800cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.805+0000 7fa485800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4800b8820 con 0x7fa4800ce510 2024-01-24T17:46:43.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa485800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4800b5790 con 0x7fa4800d15a0 2024-01-24T17:46:43.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa485800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4800b5a50 con 0x7fa48015cbe0 2024-01-24T17:46:43.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa47ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa48015cbe0 0x7fa4800cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa47ea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa48015cbe0 0x7fa4800cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44690/0 (socket says 172.21.15.39:44690) 2024-01-24T17:46:43.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa47ea00640 1 -- 172.21.15.39:0/507866756 learned_addr learned my addr 172.21.15.39:0/507866756 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:43.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa47e000640 1 --1- 172.21.15.39:0/507866756 >> v1:172.21.15.39:6789/0 conn(0x7fa4800ce510 0x7fa4800d0e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55536/0 (socket says 172.21.15.39:55536) 2024-01-24T17:46:43.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa47ea00640 1 -- 172.21.15.39:0/507866756 >> v1:172.21.15.100:6789/0 conn(0x7fa4800d15a0 legacy=0x7fa480166b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:43.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa47ea00640 1 -- 172.21.15.39:0/507866756 >> v1:172.21.15.39:6789/0 conn(0x7fa4800ce510 legacy=0x7fa4800d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.806+0000 7fa47ea00640 1 -- 172.21.15.39:0/507866756 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4800b5d30 con 0x7fa48015cbe0 2024-01-24T17:46:43.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.808+0000 7fa47ea00640 1 --2- 172.21.15.39:0/507866756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa48015cbe0 0x7fa4800cdfd0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa470009ea0 tx=0x7fa47002bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f942476277b3dbc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:43.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.808+0000 7fa47d600640 1 -- 172.21.15.39:0/507866756 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47002cb50 con 0x7fa48015cbe0 2024-01-24T17:46:43.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.808+0000 7fa47d600640 1 -- 172.21.15.39:0/507866756 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47002ccb0 con 0x7fa48015cbe0 2024-01-24T17:46:43.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.808+0000 7fa47d600640 1 -- 172.21.15.39:0/507866756 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470034610 con 0x7fa48015cbe0 2024-01-24T17:46:43.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.809+0000 7fa485800640 1 -- 172.21.15.39:0/507866756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa48015cbe0 msgr2=0x7fa4800cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.809+0000 7fa485800640 1 --2- 172.21.15.39:0/507866756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa48015cbe0 0x7fa4800cdfd0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa470009ea0 tx=0x7fa47002bfb0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.809+0000 7fa485800640 1 -- 172.21.15.39:0/507866756 shutdown_connections 2024-01-24T17:46:43.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.809+0000 7fa485800640 1 --2- 172.21.15.39:0/507866756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa48015cbe0 0x7fa4800cdfd0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.809+0000 7fa485800640 1 -- 172.21.15.39:0/507866756 >> 172.21.15.39:0/507866756 conn(0x7fa48009bc80 msgr2=0x7fa4800d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:43.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.809+0000 7fa485800640 1 -- 172.21.15.39:0/507866756 shutdown_connections 2024-01-24T17:46:43.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.809+0000 7fa485800640 1 -- 172.21.15.39:0/507866756 wait complete. 2024-01-24T17:46:43.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.811+0000 7fa485800640 1 Processor -- start 2024-01-24T17:46:43.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.811+0000 7fa485800640 1 -- start start 2024-01-24T17:46:43.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa485800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4800ce510 0x7fa4801f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa485800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4800d15a0 0x7fa48015f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa485800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48015cbe0 0x7fa48009e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa485800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48015d680 con 0x7fa4800d15a0 2024-01-24T17:46:43.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa485800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4801566c0 con 0x7fa4800ce510 2024-01-24T17:46:43.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa485800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48009d500 con 0x7fa48015cbe0 2024-01-24T17:46:43.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa47e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4800d15a0 0x7fa48015f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa47e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4800d15a0 0x7fa48015f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44698/0 (socket says 172.21.15.39:44698) 2024-01-24T17:46:43.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa47e000640 1 -- 172.21.15.39:0/2622188804 learned_addr learned my addr 172.21.15.39:0/2622188804 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:43.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa47e000640 1 -- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48015cbe0 msgr2=0x7fa48009e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa47e000640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48015cbe0 0x7fa48009e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa47ea00640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4800ce510 0x7fa4801f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.812+0000 7fa47e000640 1 -- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4800ce510 msgr2=0x7fa4801f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:43.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.813+0000 7fa47e000640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4800ce510 0x7fa4801f2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:43.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.813+0000 7fa47e000640 1 -- 172.21.15.39:0/2622188804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa470009b30 con 0x7fa4800d15a0 2024-01-24T17:46:43.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.813+0000 7fa47e000640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4800d15a0 0x7fa48015f970 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c3e0 tx=0x7fa46800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.813+0000 7fa476a00640 1 -- 172.21.15.39:0/2622188804 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4680074e0 con 0x7fa4800d15a0 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.813+0000 7fa476a00640 1 -- 172.21.15.39:0/2622188804 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa468007640 con 0x7fa4800d15a0 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.813+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48009d7e0 con 0x7fa4800d15a0 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.813+0000 7fa476a00640 1 -- 172.21.15.39:0/2622188804 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4680049e0 con 0x7fa4800d15a0 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.813+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48015d2a0 con 0x7fa4800d15a0 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.814+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48015d460 con 0x7fa4800d15a0 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.815+0000 7fa476a00640 1 -- 172.21.15.39:0/2622188804 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa468004b40 con 0x7fa4800d15a0 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.816+0000 7fa476a00640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa45806a430 0x7fa45806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:43.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.816+0000 7fa476a00640 1 -- 172.21.15.39:0/2622188804 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa468088270 con 0x7fa4800d15a0 2024-01-24T17:46:43.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.817+0000 7fa47ea00640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa45806a430 0x7fa45806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:43.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.818+0000 7fa47ea00640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa45806a430 0x7fa45806c8f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa470035000 tx=0x7fa470030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:43.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:43.820+0000 7fa476a00640 1 -- 172.21.15.39:0/2622188804 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa4680538f0 con 0x7fa4800d15a0 2024-01-24T17:46:44.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.016+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa48015d7c0 con 0x7fa4800d15a0 2024-01-24T17:46:44.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.019+0000 7fa476a00640 1 -- 172.21.15.39:0/2622188804 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16040 (secure 0 0 0) 0x7fa48015d7c0 con 0x7fa4800d15a0 2024-01-24T17:46:44.020 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:44.020 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":76,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:32.019249+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:44.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa45806a430 msgr2=0x7fa45806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa45806a430 0x7fa45806c8f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa470035000 tx=0x7fa470030000 comp rx=0 tx=0).stop 2024-01-24T17:46:44.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4800d15a0 msgr2=0x7fa48015f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4800d15a0 0x7fa48015f970 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c3e0 tx=0x7fa46800c8a0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 shutdown_connections 2024-01-24T17:46:44.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa48015cbe0 0x7fa48009e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa45806a430 0x7fa45806c8f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4800d15a0 0x7fa48015f970 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 --2- 172.21.15.39:0/2622188804 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4800ce510 0x7fa4801f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.021+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 >> 172.21.15.39:0/2622188804 conn(0x7fa48009bc80 msgr2=0x7fa4800cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:44.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.022+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 shutdown_connections 2024-01-24T17:46:44.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.022+0000 7fa485800640 1 -- 172.21.15.39:0/2622188804 wait complete. 2024-01-24T17:46:44.042 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:46:29.228451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:46:30.039417+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:46:44.042 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:46:44.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.266+0000 7f716e000640 1 Processor -- start 2024-01-24T17:46:44.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.266+0000 7f716e000640 1 -- start start 2024-01-24T17:46:44.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.268+0000 7f716e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f716815ac70 0x7f71680b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.268+0000 7f716e000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7168155170 con 0x7f7168070460 2024-01-24T17:46:44.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.268+0000 7f716e000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716815b1e0 con 0x7f71680ce8c0 2024-01-24T17:46:44.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.268+0000 7f716e000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71680b12b0 con 0x7f716815ac70 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f715fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7168070460 0x7f71680b1700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55538/0 (socket says 172.21.15.39:55538) 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f715fe00640 1 -- 172.21.15.39:0/4083740743 learned_addr learned my addr 172.21.15.39:0/4083740743 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f716d600640 1 --2- 172.21.15.39:0/4083740743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f716815ac70 0x7f71680b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f7167e00640 1 -- 172.21.15.39:0/4083740743 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956935103 0 0) 0x7f7168155170 con 0x7f7168070460 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f7167e00640 1 -- 172.21.15.39:0/4083740743 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7148003520 con 0x7f7168070460 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f7167e00640 1 -- 172.21.15.39:0/4083740743 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656896611 0 0) 0x7f716815b1e0 con 0x7f71680ce8c0 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f7167e00640 1 -- 172.21.15.39:0/4083740743 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7168155170 con 0x7f71680ce8c0 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f7167e00640 1 -- 172.21.15.39:0/4083740743 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594449178 0 0) 0x7f7148003520 con 0x7f7168070460 2024-01-24T17:46:44.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f716d600640 1 -- 172.21.15.39:0/4083740743 >> v1:172.21.15.100:6789/0 conn(0x7f71680ce8c0 legacy=0x7f716806fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.269+0000 7f716d600640 1 -- 172.21.15.39:0/4083740743 >> v1:172.21.15.39:6789/0 conn(0x7f7168070460 legacy=0x7f71680b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.270+0000 7f716d600640 1 -- 172.21.15.39:0/4083740743 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716815b4c0 con 0x7f716815ac70 2024-01-24T17:46:44.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.271+0000 7f716d600640 1 --2- 172.21.15.39:0/4083740743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f716815ac70 0x7f71680b1e10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7160009dc0 tx=0x7f716002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f78774235d998ccd server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:44.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.271+0000 7f7167e00640 1 -- 172.21.15.39:0/4083740743 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71500015f0 con 0x7f716815ac70 2024-01-24T17:46:44.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.271+0000 7f7167e00640 1 -- 172.21.15.39:0/4083740743 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716002c910 con 0x7f716815ac70 2024-01-24T17:46:44.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.271+0000 7f7167e00640 1 -- 172.21.15.39:0/4083740743 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716002cca0 con 0x7f716815ac70 2024-01-24T17:46:44.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.272+0000 7f716e000640 1 -- 172.21.15.39:0/4083740743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f716815ac70 msgr2=0x7f71680b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.272+0000 7f716e000640 1 --2- 172.21.15.39:0/4083740743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f716815ac70 0x7f71680b1e10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7160009dc0 tx=0x7f716002bde0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.272+0000 7f716e000640 1 -- 172.21.15.39:0/4083740743 shutdown_connections 2024-01-24T17:46:44.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.272+0000 7f716e000640 1 --2- 172.21.15.39:0/4083740743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f716815ac70 0x7f71680b1e10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.272+0000 7f716e000640 1 -- 172.21.15.39:0/4083740743 >> 172.21.15.39:0/4083740743 conn(0x7f716806a200 msgr2=0x7f71680affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:44.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.272+0000 7f716e000640 1 -- 172.21.15.39:0/4083740743 shutdown_connections 2024-01-24T17:46:44.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.272+0000 7f716e000640 1 -- 172.21.15.39:0/4083740743 wait complete. 2024-01-24T17:46:44.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.274+0000 7f716e000640 1 Processor -- start 2024-01-24T17:46:44.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.274+0000 7f716e000640 1 -- start start 2024-01-24T17:46:44.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f716e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7168070460 0x7f7168153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f716e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71680ce8c0 0x7f71681504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f716e000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f716815ac70 0x7f71680c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f716e000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7168153cc0 con 0x7f71680ce8c0 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f716e000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71681526e0 con 0x7f716815ac70 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f716e000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71681529c0 con 0x7f7168070460 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f715fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71680ce8c0 0x7f71681504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f715fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71680ce8c0 0x7f71681504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44708/0 (socket says 172.21.15.39:44708) 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.275+0000 7f715fe00640 1 -- 172.21.15.39:0/3406471979 learned_addr learned my addr 172.21.15.39:0/3406471979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f716cc00640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7168070460 0x7f7168153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f716d600640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f716815ac70 0x7f71680c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f715fe00640 1 -- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7168070460 msgr2=0x7f7168153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f715fe00640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7168070460 0x7f7168153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f715fe00640 1 -- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f716815ac70 msgr2=0x7f71680c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f715fe00640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f716815ac70 0x7f71680c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f715fe00640 1 -- 172.21.15.39:0/3406471979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7160009a70 con 0x7f71680ce8c0 2024-01-24T17:46:44.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f716cc00640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7168070460 0x7f7168153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:44.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f716d600640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f716815ac70 0x7f71680c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:44.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f715fe00640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71680ce8c0 0x7f71681504c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f715000c520 tx=0x7f715000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:44.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f7165600640 1 -- 172.21.15.39:0/3406471979 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150005ce0 con 0x7f71680ce8c0 2024-01-24T17:46:44.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f7165600640 1 -- 172.21.15.39:0/3406471979 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7150005e40 con 0x7f71680ce8c0 2024-01-24T17:46:44.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f7165600640 1 -- 172.21.15.39:0/3406471979 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150007290 con 0x7f71680ce8c0 2024-01-24T17:46:44.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7168152ca0 con 0x7f71680ce8c0 2024-01-24T17:46:44.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.276+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7168153260 con 0x7f71680ce8c0 2024-01-24T17:46:44.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.277+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71680738c0 con 0x7f71680ce8c0 2024-01-24T17:46:44.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.278+0000 7f7165600640 1 -- 172.21.15.39:0/3406471979 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f71500074b0 con 0x7f71680ce8c0 2024-01-24T17:46:44.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.279+0000 7f7165600640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f714006a430 0x7f714006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.279+0000 7f7165600640 1 -- 172.21.15.39:0/3406471979 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7150087fe0 con 0x7f71680ce8c0 2024-01-24T17:46:44.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.280+0000 7f716cc00640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f714006a430 0x7f714006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.281+0000 7f716cc00640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f714006a430 0x7f714006c8f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7158007fd0 tx=0x7f7158009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.285+0000 7f7165600640 1 -- 172.21.15.39:0/3406471979 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7150053600 con 0x7f71680ce8c0 2024-01-24T17:46:44.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.473+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71680b38c0 con 0x7f71680ce8c0 2024-01-24T17:46:44.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.474+0000 7f7165600640 1 -- 172.21.15.39:0/3406471979 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f71500170a0 con 0x7f71680ce8c0 2024-01-24T17:46:44.475 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:46:44.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f714006a430 msgr2=0x7f714006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f714006a430 0x7f714006c8f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7158007fd0 tx=0x7f7158009040 comp rx=0 tx=0).stop 2024-01-24T17:46:44.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71680ce8c0 msgr2=0x7f71681504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71680ce8c0 0x7f71681504c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f715000c520 tx=0x7f715000c9e0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 shutdown_connections 2024-01-24T17:46:44.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f716815ac70 0x7f71680c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f714006a430 0x7f714006c8f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71680ce8c0 0x7f71681504c0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 --2- 172.21.15.39:0/3406471979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7168070460 0x7f7168153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.476+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 >> 172.21.15.39:0/3406471979 conn(0x7f716806a200 msgr2=0x7f716806bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:44.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.477+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 shutdown_connections 2024-01-24T17:46:44.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.477+0000 7f716e000640 1 -- 172.21.15.39:0/3406471979 wait complete. 2024-01-24T17:46:44.496 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:46:44.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.700+0000 7f23e7e00640 1 Processor -- start 2024-01-24T17:46:44.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.700+0000 7f23e7e00640 1 -- start start 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f23e8071e40 0x7f23e8070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23e80cb250 0x7f23e80cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e80cbf90 con 0x7f23e8159530 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e80cc270 con 0x7f23e80cb250 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e80cc550 con 0x7f23e8071e40 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23e80cb250 0x7f23e80cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23e80cb250 0x7f23e80cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44720/0 (socket says 172.21.15.39:44720) 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7400640 1 -- 172.21.15.39:0/54555010 learned_addr learned my addr 172.21.15.39:0/54555010 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e6000640 1 --2- 172.21.15.39:0/54555010 >> v2:172.21.15.100:3300/0 conn(0x7f23e8071e40 0x7f23e8070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7400640 1 -- 172.21.15.39:0/54555010 >> v2:172.21.15.100:3300/0 conn(0x7f23e8071e40 msgr2=0x7f23e8070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:44.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7400640 1 --2- 172.21.15.39:0/54555010 >> v2:172.21.15.100:3300/0 conn(0x7f23e8071e40 0x7f23e8070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7400640 1 -- 172.21.15.39:0/54555010 >> v1:172.21.15.100:6789/0 conn(0x7f23e8159530 legacy=0x7f23e80cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:44.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.702+0000 7f23e7400640 1 -- 172.21.15.39:0/54555010 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80cc830 con 0x7f23e80cb250 2024-01-24T17:46:44.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.703+0000 7f23e7400640 1 --2- 172.21.15.39:0/54555010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80cb250 0x7f23e80cd030 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f23e806e080 tx=0x7f23e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1468821760c7f7e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:44.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.704+0000 7f23e5600640 1 -- 172.21.15.39:0/54555010 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0031870 con 0x7f23e80cb250 2024-01-24T17:46:44.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.704+0000 7f23e5600640 1 -- 172.21.15.39:0/54555010 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e00319d0 con 0x7f23e80cb250 2024-01-24T17:46:44.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.704+0000 7f23e5600640 1 -- 172.21.15.39:0/54555010 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0031d60 con 0x7f23e80cb250 2024-01-24T17:46:44.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.704+0000 7f23e7e00640 1 -- 172.21.15.39:0/54555010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80cb250 msgr2=0x7f23e80cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.704+0000 7f23e7e00640 1 --2- 172.21.15.39:0/54555010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80cb250 0x7f23e80cd030 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f23e806e080 tx=0x7f23e002b910 comp rx=0 tx=0).stop 2024-01-24T17:46:44.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.705+0000 7f23e7e00640 1 -- 172.21.15.39:0/54555010 shutdown_connections 2024-01-24T17:46:44.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.705+0000 7f23e7e00640 1 --2- 172.21.15.39:0/54555010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80cb250 0x7f23e80cd030 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.705+0000 7f23e7e00640 1 --2- 172.21.15.39:0/54555010 >> v2:172.21.15.100:3300/0 conn(0x7f23e8071e40 0x7f23e8070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.705+0000 7f23e7e00640 1 -- 172.21.15.39:0/54555010 >> 172.21.15.39:0/54555010 conn(0x7f23e806a200 msgr2=0x7f23e80affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:44.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.705+0000 7f23e7e00640 1 -- 172.21.15.39:0/54555010 shutdown_connections 2024-01-24T17:46:44.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.705+0000 7f23e7e00640 1 -- 172.21.15.39:0/54555010 wait complete. 2024-01-24T17:46:44.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.707+0000 7f23e7e00640 1 Processor -- start 2024-01-24T17:46:44.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.707+0000 7f23e7e00640 1 -- start start 2024-01-24T17:46:44.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.708+0000 7f23e7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23e80cb250 0x7f23e814f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.708+0000 7f23e7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23e8159530 0x7f23e814c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.708+0000 7f23e7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80b1080 0x7f23e80c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.708+0000 7f23e7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e8166b60 con 0x7f23e80b1080 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.708+0000 7f23e7e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80ac2d0 con 0x7f23e80cb250 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.708+0000 7f23e7e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80ad2a0 con 0x7f23e8159530 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80b1080 0x7f23e80c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80b1080 0x7f23e80c7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44736/0 (socket says 172.21.15.39:44736) 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 -- 172.21.15.39:0/3515475610 learned_addr learned my addr 172.21.15.39:0/3515475610 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e6a00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23e80cb250 0x7f23e814f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e6000640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23e8159530 0x7f23e814c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 -- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23e8159530 msgr2=0x7f23e814c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23e8159530 0x7f23e814c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 -- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23e80cb250 msgr2=0x7f23e814f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23e80cb250 0x7f23e814f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 -- 172.21.15.39:0/3515475610 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e00094d0 con 0x7f23e80b1080 2024-01-24T17:46:44.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.709+0000 7f23e7400640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80b1080 0x7f23e80c7660 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f23e00090b0 tx=0x7f23e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:44.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.710+0000 7f23dea00640 1 -- 172.21.15.39:0/3515475610 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e003ac60 con 0x7f23e80b1080 2024-01-24T17:46:44.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.710+0000 7f23dea00640 1 -- 172.21.15.39:0/3515475610 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e003adc0 con 0x7f23e80b1080 2024-01-24T17:46:44.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.710+0000 7f23dea00640 1 -- 172.21.15.39:0/3515475610 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0044630 con 0x7f23e80b1080 2024-01-24T17:46:44.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.710+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e8166880 con 0x7f23e80b1080 2024-01-24T17:46:44.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.710+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e8167140 con 0x7f23e80b1080 2024-01-24T17:46:44.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.711+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e80738c0 con 0x7f23e80b1080 2024-01-24T17:46:44.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.712+0000 7f23dea00640 1 -- 172.21.15.39:0/3515475610 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f23e0044790 con 0x7f23e80b1080 2024-01-24T17:46:44.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.713+0000 7f23dea00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23c006a430 0x7f23c006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:44.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.713+0000 7f23dea00640 1 -- 172.21.15.39:0/3515475610 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23e002d040 con 0x7f23e80b1080 2024-01-24T17:46:44.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.714+0000 7f23e6a00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23c006a430 0x7f23c006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:44.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.715+0000 7f23e6a00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23c006a430 0x7f23c006c8f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f23e80cd140 tx=0x7f23d80094d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:44.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.717+0000 7f23dea00640 1 -- 172.21.15.39:0/3515475610 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f23e0076630 con 0x7f23e80b1080 2024-01-24T17:46:44.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.909+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f23e8166cc0 con 0x7f23e80b1080 2024-01-24T17:46:44.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.910+0000 7f23dea00640 1 -- 172.21.15.39:0/3515475610 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23e0035020 con 0x7f23e80b1080 2024-01-24T17:46:44.910 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:46:44.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23c006a430 msgr2=0x7f23c006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23c006a430 0x7f23c006c8f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f23e80cd140 tx=0x7f23d80094d0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80b1080 msgr2=0x7f23e80c7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80b1080 0x7f23e80c7660 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f23e00090b0 tx=0x7f23e00046c0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 shutdown_connections 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f23c006a430 0x7f23c006c8f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e80b1080 0x7f23e80c7660 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23e8159530 0x7f23e814c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 --2- 172.21.15.39:0/3515475610 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f23e80cb250 0x7f23e814f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.912+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 >> 172.21.15.39:0/3515475610 conn(0x7f23e806a200 msgr2=0x7f23e80ae200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.913+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 shutdown_connections 2024-01-24T17:46:44.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:44.913+0000 7f23e7e00640 1 -- 172.21.15.39:0/3515475610 wait complete. 2024-01-24T17:46:44.932 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:46:45.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.142+0000 7f894a400640 1 Processor -- start 2024-01-24T17:46:45.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.142+0000 7f894a400640 1 -- start start 2024-01-24T17:46:45.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.143+0000 7f894a400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f89440d0e90 0x7f89441606b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.144+0000 7f894a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89440d13c0 con 0x7f8944160bf0 2024-01-24T17:46:45.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.144+0000 7f894a400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89440d16a0 con 0x7f894415ca30 2024-01-24T17:46:45.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.144+0000 7f894a400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8944161060 con 0x7f89440d0e90 2024-01-24T17:46:45.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.144+0000 7f8943e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8944160bf0 0x7f8944166a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55548/0 (socket says 172.21.15.39:55548) 2024-01-24T17:46:45.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.144+0000 7f8943e00640 1 -- 172.21.15.39:0/3000953849 learned_addr learned my addr 172.21.15.39:0/3000953849 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:45.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.144+0000 7f8942a00640 1 --2- 172.21.15.39:0/3000953849 >> v2:172.21.15.100:3300/0 conn(0x7f89440d0e90 0x7f89441606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.144+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173392644 0 0) 0x7f89440d13c0 con 0x7f8944160bf0 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.144+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8924003520 con 0x7f8944160bf0 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1044339665 0 0) 0x7f8924003520 con 0x7f8944160bf0 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 >> v2:172.21.15.100:3300/0 conn(0x7f89440d0e90 msgr2=0x7f89441606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 --2- 172.21.15.39:0/3000953849 >> v2:172.21.15.100:3300/0 conn(0x7f89440d0e90 0x7f89441606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 >> v1:172.21.15.100:6789/0 conn(0x7f894415ca30 legacy=0x7f89440d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440d1980 con 0x7f8944160bf0 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f893c0019f0 con 0x7f8944160bf0 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 >> v1:172.21.15.39:6789/0 conn(0x7f8944160bf0 legacy=0x7f8944166a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89240039c0 0x7f8924003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89240047c0 0x7f8924004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f892400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8944161060 con 0x7f8924005180 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892400c080 con 0x7f89240039c0 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892400c240 con 0x7f89240047c0 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.145+0000 7f8943e00640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f892400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.146+0000 7f8943400640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89240039c0 0x7f8924003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.146+0000 7f8942a00640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89240047c0 0x7f8924004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.146+0000 7f8943e00640 1 -- 172.21.15.39:0/3000953849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89240047c0 msgr2=0x7f8924004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.146+0000 7f8943e00640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89240047c0 0x7f8924004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.146+0000 7f8943e00640 1 -- 172.21.15.39:0/3000953849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89240039c0 msgr2=0x7f8924003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.146+0000 7f8943e00640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89240039c0 0x7f8924003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.146+0000 7f8943e00640 1 -- 172.21.15.39:0/3000953849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8924003520 con 0x7f8924005180 2024-01-24T17:46:45.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.147+0000 7f8943e00640 1 -- 172.21.15.39:0/3000953849 reap_dead start 2024-01-24T17:46:45.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.147+0000 7f8943e00640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f892400b980 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f893c00a760 tx=0x7f89240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8bef88620637513 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:45.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.147+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c034630 con 0x7f8924005180 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.147+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c02f070 con 0x7f8924005180 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.147+0000 7f8942000640 1 -- 172.21.15.39:0/3000953849 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c034a10 con 0x7f8924005180 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.148+0000 7f894a400640 1 -- 172.21.15.39:0/3000953849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 msgr2=0x7f892400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.148+0000 7f894a400640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f892400b980 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f893c00a760 tx=0x7f89240057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.148+0000 7f894a400640 1 -- 172.21.15.39:0/3000953849 shutdown_connections 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.148+0000 7f894a400640 1 --2- 172.21.15.39:0/3000953849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f892400b980 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.148+0000 7f894a400640 1 -- 172.21.15.39:0/3000953849 >> 172.21.15.39:0/3000953849 conn(0x7f894409bc80 msgr2=0x7f89440d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.148+0000 7f894a400640 1 -- 172.21.15.39:0/3000953849 shutdown_connections 2024-01-24T17:46:45.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.148+0000 7f894a400640 1 -- 172.21.15.39:0/3000953849 wait complete. 2024-01-24T17:46:45.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.150+0000 7f894a400640 1 Processor -- start 2024-01-24T17:46:45.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.150+0000 7f894a400640 1 -- start start 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f894a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f89440cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f894a400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89440b7d80 0x7f89440c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f894a400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89441588b0 0x7f8944158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f894a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89441576d0 con 0x7f8924005180 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f894a400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89440b8200 con 0x7f89440b7d80 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f894a400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8944159270 con 0x7f89441588b0 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f8943400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f89440cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f8943400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f89440cc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44752/0 (socket says 172.21.15.39:44752) 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f8943400640 1 -- 172.21.15.39:0/3649120449 learned_addr learned my addr 172.21.15.39:0/3649120449 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f8943e00640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89441588b0 0x7f8944158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.151+0000 7f8942a00640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89440b7d80 0x7f89440c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f8943400640 1 -- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89441588b0 msgr2=0x7f8944158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f8943400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89441588b0 0x7f8944158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f8943400640 1 -- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89440b7d80 msgr2=0x7f89440c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f8943400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89440b7d80 0x7f89440c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f8943400640 1 -- 172.21.15.39:0/3649120449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440ca100 con 0x7f8924005180 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f8943e00640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89441588b0 0x7f8944158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f8942a00640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89440b7d80 0x7f89440c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f8943400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f89440cc400 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f893400e500 tx=0x7f893400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f893b400640 1 -- 172.21.15.39:0/3649120449 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340052e0 con 0x7f8924005180 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f893b400640 1 -- 172.21.15.39:0/3649120449 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8934005440 con 0x7f8924005180 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f893b400640 1 -- 172.21.15.39:0/3649120449 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934003810 con 0x7f8924005180 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89440c9480 con 0x7f8924005180 2024-01-24T17:46:45.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.152+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89440ca630 con 0x7f8924005180 2024-01-24T17:46:45.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.154+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89440ca850 con 0x7f8924005180 2024-01-24T17:46:45.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.155+0000 7f893b400640 1 -- 172.21.15.39:0/3649120449 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8934003970 con 0x7f8924005180 2024-01-24T17:46:45.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.155+0000 7f893b400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f891c06a460 0x7f891c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.156+0000 7f893b400640 1 -- 172.21.15.39:0/3649120449 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8934088710 con 0x7f8924005180 2024-01-24T17:46:45.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.156+0000 7f8942a00640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f891c06a460 0x7f891c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.158+0000 7f8942a00640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f891c06a460 0x7f891c06c920 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f89440cfcf0 tx=0x7f892c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:45.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.160+0000 7f893b400640 1 -- 172.21.15.39:0/3649120449 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8934053c70 con 0x7f8924005180 2024-01-24T17:46:45.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.352+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f894404dff0 con 0x7f8924005180 2024-01-24T17:46:45.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.353+0000 7f893b400640 1 -- 172.21.15.39:0/3649120449 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8934057290 con 0x7f8924005180 2024-01-24T17:46:45.353 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:46:45.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f891c06a460 msgr2=0x7f891c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f891c06a460 0x7f891c06c920 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f89440cfcf0 tx=0x7f892c008040 comp rx=0 tx=0).stop 2024-01-24T17:46:45.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 msgr2=0x7f89440cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f89440cc400 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f893400e500 tx=0x7f893400e9c0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 shutdown_connections 2024-01-24T17:46:45.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89441588b0 0x7f8944158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89440b7d80 0x7f89440c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f891c06a460 0x7f891c06c920 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 --2- 172.21.15.39:0/3649120449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8924005180 0x7f89440cc400 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 >> 172.21.15.39:0/3649120449 conn(0x7f894409bc80 msgr2=0x7f894415bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.355+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 shutdown_connections 2024-01-24T17:46:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.356+0000 7f894a400640 1 -- 172.21.15.39:0/3649120449 wait complete. 2024-01-24T17:46:45.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:46:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.598+0000 7efe2ac00640 1 Processor -- start 2024-01-24T17:46:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.598+0000 7efe2ac00640 1 -- start start 2024-01-24T17:46:45.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe2ac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efe240b0010 0x7efe2415bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe2ac00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe2415f390 0x7efe240c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe2ac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe24070470 con 0x7efe24070920 2024-01-24T17:46:45.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe2ac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2415ef00 con 0x7efe240b0010 2024-01-24T17:46:45.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe2ac00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe240710a0 con 0x7efe2415f390 2024-01-24T17:46:45.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe23e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efe240b0010 0x7efe2415bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe23e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efe240b0010 0x7efe2415bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44760/0 (socket says 172.21.15.39:44760) 2024-01-24T17:46:45.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe23e00640 1 -- 172.21.15.39:0/108560649 learned_addr learned my addr 172.21.15.39:0/108560649 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:45.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe23e00640 1 -- 172.21.15.39:0/108560649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe2415f390 msgr2=0x7efe240c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe23e00640 1 --2- 172.21.15.39:0/108560649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe2415f390 0x7efe240c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe23e00640 1 -- 172.21.15.39:0/108560649 >> v1:172.21.15.39:6789/0 conn(0x7efe24070920 legacy=0x7efe240b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.600+0000 7efe23e00640 1 -- 172.21.15.39:0/108560649 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2415c0d0 con 0x7efe240b0010 2024-01-24T17:46:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.602+0000 7efe23e00640 1 --2- 172.21.15.39:0/108560649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2415bb90 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7efe14009870 tx=0x7efe1402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10f0bff21805ba4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.602+0000 7efe22a00640 1 -- 172.21.15.39:0/108560649 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1402f830 con 0x7efe240b0010 2024-01-24T17:46:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.602+0000 7efe22a00640 1 -- 172.21.15.39:0/108560649 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1402f990 con 0x7efe240b0010 2024-01-24T17:46:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.602+0000 7efe22a00640 1 -- 172.21.15.39:0/108560649 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1402fd20 con 0x7efe240b0010 2024-01-24T17:46:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.602+0000 7efe2ac00640 1 -- 172.21.15.39:0/108560649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 msgr2=0x7efe2415bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.602+0000 7efe2ac00640 1 --2- 172.21.15.39:0/108560649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2415bb90 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7efe14009870 tx=0x7efe1402ba10 comp rx=0 tx=0).stop 2024-01-24T17:46:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.603+0000 7efe2ac00640 1 -- 172.21.15.39:0/108560649 shutdown_connections 2024-01-24T17:46:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.603+0000 7efe2ac00640 1 --2- 172.21.15.39:0/108560649 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe2415f390 0x7efe240c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.603+0000 7efe2ac00640 1 --2- 172.21.15.39:0/108560649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2415bb90 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.603+0000 7efe2ac00640 1 -- 172.21.15.39:0/108560649 >> 172.21.15.39:0/108560649 conn(0x7efe2406bf20 msgr2=0x7efe2406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.603+0000 7efe2ac00640 1 -- 172.21.15.39:0/108560649 shutdown_connections 2024-01-24T17:46:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.603+0000 7efe2ac00640 1 -- 172.21.15.39:0/108560649 wait complete. 2024-01-24T17:46:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.604+0000 7efe2ac00640 1 Processor -- start 2024-01-24T17:46:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.605+0000 7efe2ac00640 1 -- start start 2024-01-24T17:46:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.605+0000 7efe2ac00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe24070920 0x7efe241510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.605+0000 7efe2ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2414c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.605+0000 7efe2ac00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe2415f390 0x7efe241519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.605+0000 7efe2ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2414f170 con 0x7efe240b0010 2024-01-24T17:46:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.605+0000 7efe2ac00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2414ee60 con 0x7efe2415f390 2024-01-24T17:46:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.605+0000 7efe2ac00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe24153cb0 con 0x7efe24070920 2024-01-24T17:46:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.606+0000 7efe23400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2414c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.606+0000 7efe23400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2414c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44764/0 (socket says 172.21.15.39:44764) 2024-01-24T17:46:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.606+0000 7efe23400640 1 -- 172.21.15.39:0/879913373 learned_addr learned my addr 172.21.15.39:0/879913373 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.606+0000 7efe23400640 1 -- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe24070920 msgr2=0x7efe241510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.606+0000 7efe23400640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe24070920 0x7efe241510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.606+0000 7efe23400640 1 -- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe2415f390 msgr2=0x7efe241519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.606+0000 7efe23400640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe2415f390 0x7efe241519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.606+0000 7efe23400640 1 -- 172.21.15.39:0/879913373 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe140094d0 con 0x7efe240b0010 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.607+0000 7efe23400640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2414c050 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c450 tx=0x7efe0c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.607+0000 7efe1be00640 1 -- 172.21.15.39:0/879913373 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c0052d0 con 0x7efe240b0010 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.607+0000 7efe1be00640 1 -- 172.21.15.39:0/879913373 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0c005430 con 0x7efe240b0010 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.607+0000 7efe1be00640 1 -- 172.21.15.39:0/879913373 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c012870 con 0x7efe240b0010 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.607+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe24153f90 con 0x7efe240b0010 2024-01-24T17:46:45.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.607+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe24151600 con 0x7efe240b0010 2024-01-24T17:46:45.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.608+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2414f2b0 con 0x7efe240b0010 2024-01-24T17:46:45.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.609+0000 7efe1be00640 1 -- 172.21.15.39:0/879913373 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efe0c0129d0 con 0x7efe240b0010 2024-01-24T17:46:45.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.610+0000 7efe1be00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdfc06a430 0x7efdfc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:45.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.611+0000 7efe1be00640 1 -- 172.21.15.39:0/879913373 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe0c088520 con 0x7efe240b0010 2024-01-24T17:46:45.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.611+0000 7efe23e00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdfc06a430 0x7efdfc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:45.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.612+0000 7efe23e00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdfc06a430 0x7efdfc06c8f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7efe14036000 tx=0x7efe14039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:45.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.615+0000 7efe1be00640 1 -- 172.21.15.39:0/879913373 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efe2414f2b0 con 0x7efe240b0010 2024-01-24T17:46:45.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.822+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efe2414f2b0 con 0x7efe240b0010 2024-01-24T17:46:45.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.823+0000 7efe1be00640 1 -- 172.21.15.39:0/879913373 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v76) v1 ==== 80+0+0 (secure 0 0 0) 0x7efe2414f2b0 con 0x7efe240b0010 2024-01-24T17:46:45.823 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:46:45.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdfc06a430 msgr2=0x7efdfc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdfc06a430 0x7efdfc06c8f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7efe14036000 tx=0x7efe14039040 comp rx=0 tx=0).stop 2024-01-24T17:46:45.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 msgr2=0x7efe2414c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:45.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2414c050 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c450 tx=0x7efe0c00c910 comp rx=0 tx=0).stop 2024-01-24T17:46:45.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 shutdown_connections 2024-01-24T17:46:45.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe2415f390 0x7efe241519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efdfc06a430 0x7efdfc06c8f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efe240b0010 0x7efe2414c050 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 --2- 172.21.15.39:0/879913373 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efe24070920 0x7efe241510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:45.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 >> 172.21.15.39:0/879913373 conn(0x7efe2406bf20 msgr2=0x7efe2406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:45.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 shutdown_connections 2024-01-24T17:46:45.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:45.825+0000 7efe2ac00640 1 -- 172.21.15.39:0/879913373 wait complete. 2024-01-24T17:46:45.845 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:46:45.845 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:45.899 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.118+0000 7f2809000640 1 Processor -- start 2024-01-24T17:46:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.118+0000 7f2809000640 1 -- start start 2024-01-24T17:46:46.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.120+0000 7f2809000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 0x7f280415edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.120+0000 7f2809000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28041551b0 con 0x7f2804070540 2024-01-24T17:46:46.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.120+0000 7f2809000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280415e410 con 0x7f28040b3e00 2024-01-24T17:46:46.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.120+0000 7f2809000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280415b480 con 0x7f280415e910 2024-01-24T17:46:46.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.120+0000 7f2803400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2804070540 0x7f280415ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55562/0 (socket says 172.21.15.39:55562) 2024-01-24T17:46:46.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.120+0000 7f2803400640 1 -- 172.21.15.39:0/2247955309 learned_addr learned my addr 172.21.15.39:0/2247955309 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:46.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.120+0000 7f2803e00640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 0x7f280415edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327382449 0 0) 0x7f28041551b0 con 0x7f2804070540 2024-01-24T17:46:46.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e4003520 con 0x7f2804070540 2024-01-24T17:46:46.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1794174171 0 0) 0x7f27e4003520 con 0x7f2804070540 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 msgr2=0x7f280415edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 0x7f280415edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 >> v1:172.21.15.100:6789/0 conn(0x7f28040b3e00 legacy=0x7f28040c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280415b760 con 0x7f2804070540 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f27f40017f0 con 0x7f2804070540 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 >> v1:172.21.15.39:6789/0 conn(0x7f2804070540 legacy=0x7f280415ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27e40039a0 0x7f27e4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e4004350 0x7f27e4004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 0x7f27e4005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280415b480 con 0x7f27e40039a0 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e400b980 con 0x7f27e4004350 2024-01-24T17:46:46.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.121+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e400bb40 con 0x7f280415e910 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2803400640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27e40039a0 0x7f27e4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2803e00640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 0x7f27e4005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2802a00640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e4004350 0x7f27e4004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2803400640 1 -- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 msgr2=0x7f27e4005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2803400640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 0x7f27e4005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2803400640 1 -- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e4004350 msgr2=0x7f27e4004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2803400640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e4004350 0x7f27e4004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2803400640 1 -- 172.21.15.39:0/2247955309 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e4003520 con 0x7f27e40039a0 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2803e00640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f280415e910 0x7f27e4005080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:46.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.122+0000 7f2802a00640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e4004350 0x7f27e4004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.124+0000 7f2803400640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27e40039a0 0x7f27e4003d80 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f27f400c190 tx=0x7f27e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58b1c138996cf22 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.124+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f4007270 con 0x7f27e40039a0 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.124+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f40073d0 con 0x7f27e40039a0 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.124+0000 7f2802000640 1 -- 172.21.15.39:0/2247955309 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f402c850 con 0x7f27e40039a0 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.125+0000 7f2809000640 1 -- 172.21.15.39:0/2247955309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27e40039a0 msgr2=0x7f27e4003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.125+0000 7f2809000640 1 --2- 172.21.15.39:0/2247955309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27e40039a0 0x7f27e4003d80 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f27f400c190 tx=0x7f27e40057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.125+0000 7f2803e00640 1 -- 172.21.15.39:0/2247955309 reap_dead start 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.125+0000 7f2809000640 1 -- 172.21.15.39:0/2247955309 shutdown_connections 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.125+0000 7f2809000640 1 -- 172.21.15.39:0/2247955309 >> 172.21.15.39:0/2247955309 conn(0x7f280406a340 msgr2=0x7f28040cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.125+0000 7f2809000640 1 -- 172.21.15.39:0/2247955309 shutdown_connections 2024-01-24T17:46:46.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.125+0000 7f2809000640 1 -- 172.21.15.39:0/2247955309 wait complete. 2024-01-24T17:46:46.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.127+0000 7f2809000640 1 Processor -- start 2024-01-24T17:46:46.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.127+0000 7f2809000640 1 -- start start 2024-01-24T17:46:46.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2809000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e40039a0 0x7f28040c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2809000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28040c7150 0x7f280415c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2809000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28040c75b0 0x7f280415cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2809000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28040c4e10 con 0x7f28040c7150 2024-01-24T17:46:46.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2809000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28040c3100 con 0x7f27e40039a0 2024-01-24T17:46:46.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2809000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28040c33e0 con 0x7f28040c75b0 2024-01-24T17:46:46.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2802a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28040c7150 0x7f280415c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2802a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28040c7150 0x7f280415c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44772/0 (socket says 172.21.15.39:44772) 2024-01-24T17:46:46.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.128+0000 7f2802a00640 1 -- 172.21.15.39:0/437329697 learned_addr learned my addr 172.21.15.39:0/437329697 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2802a00640 1 -- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28040c75b0 msgr2=0x7f280415cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2802a00640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28040c75b0 0x7f280415cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2803400640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e40039a0 0x7f28040c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2802a00640 1 -- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e40039a0 msgr2=0x7f28040c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2802a00640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e40039a0 0x7f28040c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2802a00640 1 -- 172.21.15.39:0/437329697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28040c36c0 con 0x7f28040c7150 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2803400640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e40039a0 0x7f28040c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2802a00640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28040c7150 0x7f280415c9e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00dae0 tx=0x7f27ec002180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f27fb400640 1 -- 172.21.15.39:0/437329697 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec003da0 con 0x7f28040c7150 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f27fb400640 1 -- 172.21.15.39:0/437329697 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec00b7d0 con 0x7f28040c7150 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f27fb400640 1 -- 172.21.15.39:0/437329697 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00bb40 con 0x7f28040c7150 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.129+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28040c39a0 con 0x7f28040c7150 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.130+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28040c6e40 con 0x7f28040c7150 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.131+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280404dff0 con 0x7f28040c7150 2024-01-24T17:46:46.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.131+0000 7f27fb400640 1 -- 172.21.15.39:0/437329697 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f27ec00bca0 con 0x7f28040c7150 2024-01-24T17:46:46.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.132+0000 7f27fb400640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27dc06a430 0x7f27dc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.133+0000 7f27fb400640 1 -- 172.21.15.39:0/437329697 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27ec088cf0 con 0x7f28040c7150 2024-01-24T17:46:46.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.133+0000 7f2803400640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27dc06a430 0x7f27dc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.134+0000 7f2803400640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27dc06a430 0x7f27dc06c8f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f27f400c190 tx=0x7f27f4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:46.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.137+0000 7f27fb400640 1 -- 172.21.15.39:0/437329697 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f27ec054240 con 0x7f28040c7150 2024-01-24T17:46:46.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.334+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28040cbc10 con 0x7f28040c7150 2024-01-24T17:46:46.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.335+0000 7f27fb400640 1 -- 172.21.15.39:0/437329697 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16040 (secure 0 0 0) 0x7f27ec057860 con 0x7f28040c7150 2024-01-24T17:46:46.337 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:46.337 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":76,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:32.019249+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:46.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27dc06a430 msgr2=0x7f27dc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27dc06a430 0x7f27dc06c8f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f27f400c190 tx=0x7f27f4009030 comp rx=0 tx=0).stop 2024-01-24T17:46:46.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28040c7150 msgr2=0x7f280415c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28040c7150 0x7f280415c9e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00dae0 tx=0x7f27ec002180 comp rx=0 tx=0).stop 2024-01-24T17:46:46.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 shutdown_connections 2024-01-24T17:46:46.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f28040c75b0 0x7f280415cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f27dc06a430 0x7f27dc06c8f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28040c7150 0x7f280415c9e0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 --2- 172.21.15.39:0/437329697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f27e40039a0 0x7f28040c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 >> 172.21.15.39:0/437329697 conn(0x7f280406a340 msgr2=0x7f28040c92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:46.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.338+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 shutdown_connections 2024-01-24T17:46:46.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.339+0000 7f2809000640 1 -- 172.21.15.39:0/437329697 wait complete. 2024-01-24T17:46:46.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:46:29.228451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:46:30.039417+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:46:46.361 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:46:46.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.590+0000 7f66ba800640 1 Processor -- start 2024-01-24T17:46:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.590+0000 7f66ba800640 1 -- start start 2024-01-24T17:46:46.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.591+0000 7f66ba800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f66b40cb990 0x7f66b40ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.591+0000 7f66ba800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b40ca4c0 con 0x7f66b406cec0 2024-01-24T17:46:46.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.591+0000 7f66ba800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b406d9e0 con 0x7f66b415aa60 2024-01-24T17:46:46.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.591+0000 7f66ba800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b40cae90 con 0x7f66b40cb990 2024-01-24T17:46:46.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.592+0000 7f66b3e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f66b406cec0 0x7f66b406d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55572/0 (socket says 172.21.15.39:55572) 2024-01-24T17:46:46.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.592+0000 7f66b3e00640 1 -- 172.21.15.39:0/1972262273 learned_addr learned my addr 172.21.15.39:0/1972262273 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:46.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.592+0000 7f66aa800640 1 --2- 172.21.15.39:0/1972262273 >> v2:172.21.15.100:3300/0 conn(0x7f66b40cb990 0x7f66b40ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.592+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291351772 0 0) 0x7f66b40ca4c0 con 0x7f66b406cec0 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.592+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6694003520 con 0x7f66b406cec0 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2838719666 0 0) 0x7f6694003520 con 0x7f66b406cec0 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 >> v2:172.21.15.100:3300/0 conn(0x7f66b40cb990 msgr2=0x7f66b40ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 --2- 172.21.15.39:0/1972262273 >> v2:172.21.15.100:3300/0 conn(0x7f66b40cb990 0x7f66b40ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 >> v1:172.21.15.100:6789/0 conn(0x7f66b415aa60 legacy=0x7f66b40cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b406dc90 con 0x7f66b406cec0 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f66ac0019f0 con 0x7f66b406cec0 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 >> v1:172.21.15.39:6789/0 conn(0x7f66b406cec0 legacy=0x7f66b406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66940047c0 0x7f6694004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6694005180 0x7f669400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b40cae90 con 0x7f66940047c0 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669400c020 con 0x7f6694003970 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669400c1e0 con 0x7f6694005180 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.593+0000 7f66aa800640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66940047c0 0x7f6694004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66b3400640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66b3e00640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6694005180 0x7f669400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66aa800640 1 -- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6694005180 msgr2=0x7f669400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66aa800640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6694005180 0x7f669400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66aa800640 1 -- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6694003970 msgr2=0x7f6694003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66aa800640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66aa800640 1 -- 172.21.15.39:0/1972262273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694003520 con 0x7f66940047c0 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66b3400640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.594+0000 7f66b3e00640 1 -- 172.21.15.39:0/1972262273 reap_dead start 2024-01-24T17:46:46.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.595+0000 7f66aa800640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66940047c0 0x7f6694004c40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f669c009ac0 tx=0x7f66940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8497760dd6f10d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.595+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c02c6b0 con 0x7f66940047c0 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.595+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c02c810 con 0x7f66940047c0 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.595+0000 7f66b2a00640 1 -- 172.21.15.39:0/1972262273 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c02cba0 con 0x7f66940047c0 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.596+0000 7f66ba800640 1 -- 172.21.15.39:0/1972262273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66940047c0 msgr2=0x7f6694004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.596+0000 7f66ba800640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66940047c0 0x7f6694004c40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f669c009ac0 tx=0x7f66940057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.596+0000 7f66ba800640 1 -- 172.21.15.39:0/1972262273 shutdown_connections 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.596+0000 7f66ba800640 1 --2- 172.21.15.39:0/1972262273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66940047c0 0x7f6694004c40 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.596+0000 7f66ba800640 1 -- 172.21.15.39:0/1972262273 >> 172.21.15.39:0/1972262273 conn(0x7f66b40ce9e0 msgr2=0x7f66b40cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.596+0000 7f66ba800640 1 -- 172.21.15.39:0/1972262273 shutdown_connections 2024-01-24T17:46:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.596+0000 7f66ba800640 1 -- 172.21.15.39:0/1972262273 wait complete. 2024-01-24T17:46:46.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.598+0000 7f66ba800640 1 Processor -- start 2024-01-24T17:46:46.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.598+0000 7f66ba800640 1 -- start start 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66ba800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66940047c0 0x7f66b4155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66ba800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66b415d080 0x7f66b40c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66ba800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40af0a0 0x7f66b40af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66ba800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b40afa90 con 0x7f66b40af0a0 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66ba800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b41574b0 con 0x7f66b415d080 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66ba800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b40c8300 con 0x7f66940047c0 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40af0a0 0x7f66b40af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40af0a0 0x7f66b40af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44788/0 (socket says 172.21.15.39:44788) 2024-01-24T17:46:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3e00640 1 -- 172.21.15.39:0/3719576846 learned_addr learned my addr 172.21.15.39:0/3719576846 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66aa800640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66b415d080 0x7f66b40c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3400640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66940047c0 0x7f66b4155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3e00640 1 -- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66940047c0 msgr2=0x7f66b4155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3e00640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66940047c0 0x7f66b4155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3e00640 1 -- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66b415d080 msgr2=0x7f66b40c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3e00640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66b415d080 0x7f66b40c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66b3e00640 1 -- 172.21.15.39:0/3719576846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b40c7a60 con 0x7f66b40af0a0 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.599+0000 7f66aa800640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66b415d080 0x7f66b40c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.600+0000 7f66b3e00640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40af0a0 0x7f66b40af550 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f66b40adb90 tx=0x7f66ac00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.600+0000 7f66abe00640 1 -- 172.21.15.39:0/3719576846 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ac004d60 con 0x7f66b40af0a0 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.600+0000 7f66abe00640 1 -- 172.21.15.39:0/3719576846 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ac00bb80 con 0x7f66b40af0a0 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.600+0000 7f66abe00640 1 -- 172.21.15.39:0/3719576846 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ac007070 con 0x7f66b40af0a0 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.600+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b40c7d40 con 0x7f66b40af0a0 2024-01-24T17:46:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.600+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b41576e0 con 0x7f66b40af0a0 2024-01-24T17:46:46.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.601+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b404dff0 con 0x7f66b40af0a0 2024-01-24T17:46:46.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.602+0000 7f66abe00640 1 -- 172.21.15.39:0/3719576846 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f66ac0071d0 con 0x7f66b40af0a0 2024-01-24T17:46:46.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.603+0000 7f66abe00640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f668c06a460 0x7f668c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:46.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.603+0000 7f66abe00640 1 -- 172.21.15.39:0/3719576846 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66ac088e10 con 0x7f66b40af0a0 2024-01-24T17:46:46.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.604+0000 7f66b3400640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f668c06a460 0x7f668c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:46.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.605+0000 7f66b3400640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f668c06a460 0x7f668c06c920 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f66b40afc40 tx=0x7f66a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:46.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.608+0000 7f66abe00640 1 -- 172.21.15.39:0/3719576846 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f66ac08c030 con 0x7f66b40af0a0 2024-01-24T17:46:46.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.801+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66b406c8a0 con 0x7f66b40af0a0 2024-01-24T17:46:46.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.802+0000 7f66abe00640 1 -- 172.21.15.39:0/3719576846 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66ac054390 con 0x7f66b40af0a0 2024-01-24T17:46:46.803 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.804+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f668c06a460 msgr2=0x7f668c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f668c06a460 0x7f668c06c920 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f66b40afc40 tx=0x7f66a4009040 comp rx=0 tx=0).stop 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40af0a0 msgr2=0x7f66b40af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40af0a0 0x7f66b40af550 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f66b40adb90 tx=0x7f66ac00b4a0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 shutdown_connections 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f668c06a460 0x7f668c06c920 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66b40af0a0 0x7f66b40af550 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66b415d080 0x7f66b40c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 --2- 172.21.15.39:0/3719576846 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66940047c0 0x7f66b4155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 >> 172.21.15.39:0/3719576846 conn(0x7f66b40ce9e0 msgr2=0x7f66b406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 shutdown_connections 2024-01-24T17:46:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:46.805+0000 7f66ba800640 1 -- 172.21.15.39:0/3719576846 wait complete. 2024-01-24T17:46:46.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:46:47.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.056+0000 7f03ba400640 1 Processor -- start 2024-01-24T17:46:47.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.056+0000 7f03ba400640 1 -- start start 2024-01-24T17:46:47.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.060+0000 7f03ba400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f03b415c920 0x7f03b40cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.060+0000 7f03ba400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b40d1520 0x7f03b40b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.060+0000 7f03ba400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b40d0b40 con 0x7f03b40d0610 2024-01-24T17:46:47.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.060+0000 7f03ba400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b40b1d00 con 0x7f03b415c920 2024-01-24T17:46:47.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.060+0000 7f03ba400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40b1fe0 con 0x7f03b40d1520 2024-01-24T17:46:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.060+0000 7f03b3e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f03b40d0610 0x7f03b40d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55574/0 (socket says 172.21.15.39:55574) 2024-01-24T17:46:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.060+0000 7f03b3e00640 1 -- 172.21.15.39:0/292552357 learned_addr learned my addr 172.21.15.39:0/292552357 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b9000640 1 --2- 172.21.15.39:0/292552357 >> v2:172.21.15.100:3300/0 conn(0x7f03b415c920 0x7f03b40cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b9a00640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b40d1520 0x7f03b40b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159477779 0 0) 0x7f03b40d0b40 con 0x7f03b40d0610 2024-01-24T17:46:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0394003520 con 0x7f03b40d0610 2024-01-24T17:46:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912525101 0 0) 0x7f0394003520 con 0x7f03b40d0610 2024-01-24T17:46:47.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b40d1520 msgr2=0x7f03b40b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:47.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b40d1520 0x7f03b40b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 >> v2:172.21.15.100:3300/0 conn(0x7f03b415c920 msgr2=0x7f03b40cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:47.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 --2- 172.21.15.39:0/292552357 >> v2:172.21.15.100:3300/0 conn(0x7f03b415c920 0x7f03b40cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b40b22c0 con 0x7f03b40d0610 2024-01-24T17:46:47.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.061+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f039c001670 con 0x7f03b40d0610 2024-01-24T17:46:47.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 >> v1:172.21.15.39:6789/0 conn(0x7f03b40d0610 legacy=0x7f03b40d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b3400640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f0394003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b3400640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03940047c0 0x7f0394004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b3400640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b40d1520 0x7f0394005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40b1d00 con 0x7f0394003970 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40b1fe0 con 0x7f03940047c0 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0394014210 con 0x7f03b40d1520 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b9000640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f0394003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b9a00640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b40d1520 0x7f0394005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.062+0000 7f03b3e00640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03940047c0 0x7f0394004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.063+0000 7f03b9000640 1 -- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b40d1520 msgr2=0x7f0394005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.063+0000 7f03b9000640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b40d1520 0x7f0394005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.063+0000 7f03b9000640 1 -- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03940047c0 msgr2=0x7f0394004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.063+0000 7f03b9000640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03940047c0 0x7f0394004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.063+0000 7f03b9000640 1 -- 172.21.15.39:0/292552357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0394003520 con 0x7f0394003970 2024-01-24T17:46:47.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.064+0000 7f03b9000640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f0394003d90 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009ac0 tx=0x7f03940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f6e27ffb49dc7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.064+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a402c710 con 0x7f0394003970 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.064+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a402c870 con 0x7f0394003970 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.064+0000 7f03b3400640 1 -- 172.21.15.39:0/292552357 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a402cc00 con 0x7f0394003970 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.065+0000 7f03ba400640 1 -- 172.21.15.39:0/292552357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 msgr2=0x7f0394003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.065+0000 7f03ba400640 1 --2- 172.21.15.39:0/292552357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f0394003d90 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009ac0 tx=0x7f03940057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.065+0000 7f03b9a00640 1 -- 172.21.15.39:0/292552357 reap_dead start 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.065+0000 7f03ba400640 1 -- 172.21.15.39:0/292552357 shutdown_connections 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.065+0000 7f03ba400640 1 -- 172.21.15.39:0/292552357 >> 172.21.15.39:0/292552357 conn(0x7f03b40d2830 msgr2=0x7f03b40d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.065+0000 7f03ba400640 1 -- 172.21.15.39:0/292552357 shutdown_connections 2024-01-24T17:46:47.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.065+0000 7f03ba400640 1 -- 172.21.15.39:0/292552357 wait complete. 2024-01-24T17:46:47.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.067+0000 7f03ba400640 1 Processor -- start 2024-01-24T17:46:47.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.067+0000 7f03ba400640 1 -- start start 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03ba400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f03b4155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03ba400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03b414d390 0x7f03b4158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03ba400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b4158cb0 0x7f03b4152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03ba400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4159160 con 0x7f0394003970 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03ba400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40ca210 con 0x7f03b414d390 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03ba400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4151b20 con 0x7f03b4158cb0 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03b9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f03b4155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03b9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f03b4155740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44800/0 (socket says 172.21.15.39:44800) 2024-01-24T17:46:47.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.068+0000 7f03b9000640 1 -- 172.21.15.39:0/985030874 learned_addr learned my addr 172.21.15.39:0/985030874 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:47.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b3e00640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03b414d390 0x7f03b4158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b9000640 1 -- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b4158cb0 msgr2=0x7f03b4152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b9000640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b4158cb0 0x7f03b4152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b9000640 1 -- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03b414d390 msgr2=0x7f03b4158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b9000640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03b414d390 0x7f03b4158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b9000640 1 -- 172.21.15.39:0/985030874 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b4151e00 con 0x7f0394003970 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b9000640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f03b4155740 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f03a402c2b0 tx=0x7f03a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b0c00640 1 -- 172.21.15.39:0/985030874 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4003810 con 0x7f0394003970 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b0c00640 1 -- 172.21.15.39:0/985030874 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a4003970 con 0x7f0394003970 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03b0c00640 1 -- 172.21.15.39:0/985030874 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4003d00 con 0x7f0394003970 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.069+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b41520e0 con 0x7f0394003970 2024-01-24T17:46:47.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.070+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b41526a0 con 0x7f0394003970 2024-01-24T17:46:47.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.071+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b404e840 con 0x7f0394003970 2024-01-24T17:46:47.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.072+0000 7f03b0c00640 1 -- 172.21.15.39:0/985030874 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f03a4040450 con 0x7f0394003970 2024-01-24T17:46:47.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.073+0000 7f03b0c00640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f038c06a460 0x7f038c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.073+0000 7f03b0c00640 1 -- 172.21.15.39:0/985030874 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03a40aa7a0 con 0x7f0394003970 2024-01-24T17:46:47.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.073+0000 7f03b3e00640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f038c06a460 0x7f038c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.075+0000 7f03b3e00640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f038c06a460 0x7f038c06c920 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f039c001eb0 tx=0x7f039c000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.077+0000 7f03b0c00640 1 -- 172.21.15.39:0/985030874 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f03a4075d90 con 0x7f0394003970 2024-01-24T17:46:47.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.277+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f03b41576b0 con 0x7f0394003970 2024-01-24T17:46:47.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.278+0000 7f03b0c00640 1 -- 172.21.15.39:0/985030874 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7f03a4003e60 con 0x7f0394003970 2024-01-24T17:46:47.278 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:46:47.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f038c06a460 msgr2=0x7f038c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f038c06a460 0x7f038c06c920 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f039c001eb0 tx=0x7f039c000ed0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 msgr2=0x7f03b4155740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f03b4155740 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f03a402c2b0 tx=0x7f03a4004730 comp rx=0 tx=0).stop 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 shutdown_connections 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f03b4158cb0 0x7f03b4152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f03b414d390 0x7f03b4158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f038c06a460 0x7f038c06c920 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 --2- 172.21.15.39:0/985030874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0394003970 0x7f03b4155740 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.280+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 >> 172.21.15.39:0/985030874 conn(0x7f03b40d2830 msgr2=0x7f03b40ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.281+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 shutdown_connections 2024-01-24T17:46:47.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.281+0000 7f03ba400640 1 -- 172.21.15.39:0/985030874 wait complete. 2024-01-24T17:46:47.300 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:46:47.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.522+0000 7fec1f400640 1 Processor -- start 2024-01-24T17:46:47.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.523+0000 7fec1f400640 1 -- start start 2024-01-24T17:46:47.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.524+0000 7fec1f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec1815ca30 0x7fec180d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.524+0000 7fec1f400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fec180d0e90 0x7fec181606b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.524+0000 7fec1f400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec18161040 con 0x7fec18160bf0 2024-01-24T17:46:47.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.524+0000 7fec1f400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec180d37a0 con 0x7fec1815ca30 2024-01-24T17:46:47.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.524+0000 7fec1f400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec180d13c0 con 0x7fec180d0e90 2024-01-24T17:46:47.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.524+0000 7fec1ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec1815ca30 0x7fec180d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.524+0000 7fec1ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec1815ca30 0x7fec180d3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44804/0 (socket says 172.21.15.39:44804) 2024-01-24T17:46:47.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.524+0000 7fec1ca00640 1 -- 172.21.15.39:0/628228140 learned_addr learned my addr 172.21.15.39:0/628228140 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:47.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.525+0000 7fec1ca00640 1 -- 172.21.15.39:0/628228140 >> v2:172.21.15.100:3300/0 conn(0x7fec180d0e90 msgr2=0x7fec181606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:47.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.525+0000 7fec0fc00640 1 --2- 172.21.15.39:0/628228140 >> v2:172.21.15.100:3300/0 conn(0x7fec180d0e90 0x7fec181606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.525+0000 7fec17e00640 1 -- 172.21.15.39:0/628228140 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826412266 0 0) 0x7fec10001110 con 0x7fec18160bf0 2024-01-24T17:46:47.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.525+0000 7fec1ca00640 1 --2- 172.21.15.39:0/628228140 >> v2:172.21.15.100:3300/0 conn(0x7fec180d0e90 0x7fec181606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.525+0000 7fec1ca00640 1 -- 172.21.15.39:0/628228140 >> v1:172.21.15.39:6789/0 conn(0x7fec18160bf0 legacy=0x7fec180cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.525+0000 7fec1ca00640 1 -- 172.21.15.39:0/628228140 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec180d16a0 con 0x7fec1815ca30 2024-01-24T17:46:47.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.525+0000 7fec0fc00640 1 --2- 172.21.15.39:0/628228140 >> v2:172.21.15.100:3300/0 conn(0x7fec180d0e90 0x7fec181606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:47.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.526+0000 7fec1ca00640 1 --2- 172.21.15.39:0/628228140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec1815ca30 0x7fec180d3260 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fec1808bac0 tx=0x7fec0802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2184edfcb66e528 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.526+0000 7fec17e00640 1 -- 172.21.15.39:0/628228140 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0802cb10 con 0x7fec1815ca30 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.526+0000 7fec17e00640 1 -- 172.21.15.39:0/628228140 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0802cc70 con 0x7fec1815ca30 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.526+0000 7fec17e00640 1 -- 172.21.15.39:0/628228140 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec080343f0 con 0x7fec1815ca30 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.527+0000 7fec1f400640 1 -- 172.21.15.39:0/628228140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec1815ca30 msgr2=0x7fec180d3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.527+0000 7fec1f400640 1 --2- 172.21.15.39:0/628228140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec1815ca30 0x7fec180d3260 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fec1808bac0 tx=0x7fec0802bf70 comp rx=0 tx=0).stop 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.527+0000 7fec1f400640 1 -- 172.21.15.39:0/628228140 shutdown_connections 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.527+0000 7fec1f400640 1 --2- 172.21.15.39:0/628228140 >> v2:172.21.15.100:3300/0 conn(0x7fec180d0e90 0x7fec181606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.527+0000 7fec1f400640 1 --2- 172.21.15.39:0/628228140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec1815ca30 0x7fec180d3260 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.527+0000 7fec1f400640 1 -- 172.21.15.39:0/628228140 >> 172.21.15.39:0/628228140 conn(0x7fec1809bc80 msgr2=0x7fec180d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.527+0000 7fec1f400640 1 -- 172.21.15.39:0/628228140 shutdown_connections 2024-01-24T17:46:47.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.527+0000 7fec1f400640 1 -- 172.21.15.39:0/628228140 wait complete. 2024-01-24T17:46:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.529+0000 7fec1f400640 1 Processor -- start 2024-01-24T17:46:47.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.529+0000 7fec1f400640 1 -- start start 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.530+0000 7fec1f400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec180d0e90 0x7fec18148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.530+0000 7fec1f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec1815ca30 0x7fec180b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.530+0000 7fec1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec18160bf0 0x7fec180b6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.530+0000 7fec1f400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec180b72a0 con 0x7fec18160bf0 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.530+0000 7fec1f400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec180b7580 con 0x7fec180d0e90 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.530+0000 7fec1f400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec180b7860 con 0x7fec1815ca30 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.530+0000 7fec1d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec18160bf0 0x7fec180b6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec18160bf0 0x7fec180b6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44814/0 (socket says 172.21.15.39:44814) 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1d400640 1 -- 172.21.15.39:0/3490682695 learned_addr learned my addr 172.21.15.39:0/3490682695 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1ca00640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec180d0e90 0x7fec18148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec0fc00640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec1815ca30 0x7fec180b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1d400640 1 -- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec1815ca30 msgr2=0x7fec180b66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1d400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec1815ca30 0x7fec180b66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1d400640 1 -- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec180d0e90 msgr2=0x7fec18148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1d400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec180d0e90 0x7fec18148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1d400640 1 -- 172.21.15.39:0/3490682695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec08009b30 con 0x7fec18160bf0 2024-01-24T17:46:47.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.531+0000 7fec1d400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec18160bf0 0x7fec180b6c20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fec1000c390 tx=0x7fec1000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.532+0000 7fec15600640 1 -- 172.21.15.39:0/3490682695 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec100035f0 con 0x7fec18160bf0 2024-01-24T17:46:47.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.532+0000 7fec15600640 1 -- 172.21.15.39:0/3490682695 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec10003750 con 0x7fec18160bf0 2024-01-24T17:46:47.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.532+0000 7fec15600640 1 -- 172.21.15.39:0/3490682695 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10003b00 con 0x7fec18160bf0 2024-01-24T17:46:47.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.532+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec180b7b40 con 0x7fec18160bf0 2024-01-24T17:46:47.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.532+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec180cc600 con 0x7fec18160bf0 2024-01-24T17:46:47.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.533+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1806cd10 con 0x7fec18160bf0 2024-01-24T17:46:47.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.534+0000 7fec15600640 1 -- 172.21.15.39:0/3490682695 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fec10003c60 con 0x7fec18160bf0 2024-01-24T17:46:47.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.535+0000 7fec15600640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febf006a430 0x7febf006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.535+0000 7fec15600640 1 -- 172.21.15.39:0/3490682695 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec10087d90 con 0x7fec18160bf0 2024-01-24T17:46:47.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.536+0000 7fec1ca00640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febf006a430 0x7febf006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.537+0000 7fec1ca00640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febf006a430 0x7febf006c8f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fec08035000 tx=0x7fec08030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.542+0000 7fec15600640 1 -- 172.21.15.39:0/3490682695 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fec100532e0 con 0x7fec18160bf0 2024-01-24T17:46:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.734+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fec18161040 con 0x7fec18160bf0 2024-01-24T17:46:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.735+0000 7fec15600640 1 -- 172.21.15.39:0/3490682695 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7fec18161040 con 0x7fec18160bf0 2024-01-24T17:46:47.736 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.737+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febf006a430 msgr2=0x7febf006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.737+0000 7fec1f400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febf006a430 0x7febf006c8f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fec08035000 tx=0x7fec08030000 comp rx=0 tx=0).stop 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.737+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec18160bf0 msgr2=0x7fec180b6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.737+0000 7fec1f400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec18160bf0 0x7fec180b6c20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fec1000c390 tx=0x7fec1000c850 comp rx=0 tx=0).stop 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.738+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 shutdown_connections 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.738+0000 7fec1f400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febf006a430 0x7febf006c8f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.738+0000 7fec1f400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec18160bf0 0x7fec180b6c20 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.738+0000 7fec1f400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec1815ca30 0x7fec180b66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.738+0000 7fec1f400640 1 --2- 172.21.15.39:0/3490682695 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec180d0e90 0x7fec18148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.738+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 >> 172.21.15.39:0/3490682695 conn(0x7fec1809bc80 msgr2=0x7fec180c6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.738+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 shutdown_connections 2024-01-24T17:46:47.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.738+0000 7fec1f400640 1 -- 172.21.15.39:0/3490682695 wait complete. 2024-01-24T17:46:47.759 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:46:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.980+0000 7ff87ac00640 1 Processor -- start 2024-01-24T17:46:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.981+0000 7ff87ac00640 1 -- start start 2024-01-24T17:46:47.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.982+0000 7ff87ac00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff87415cbe0 0x7ff8740cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.982+0000 7ff87ac00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff8740d15a0 0x7ff874166b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.982+0000 7ff87ac00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8740b8820 con 0x7ff8740ce510 2024-01-24T17:46:47.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.982+0000 7ff87ac00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8740b5790 con 0x7ff8740d15a0 2024-01-24T17:46:47.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.982+0000 7ff87ac00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8740b5a50 con 0x7ff87415cbe0 2024-01-24T17:46:47.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.982+0000 7ff873400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff8740ce510 0x7ff8740d0e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:55590/0 (socket says 172.21.15.39:55590) 2024-01-24T17:46:47.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.982+0000 7ff873400640 1 -- 172.21.15.39:0/3295359116 learned_addr learned my addr 172.21.15.39:0/3295359116 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:47.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.983+0000 7ff878c00640 1 --2- 172.21.15.39:0/3295359116 >> v2:172.21.15.39:3300/0 conn(0x7ff8740d15a0 0x7ff874166b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.983+0000 7ff873e00640 1 --2- 172.21.15.39:0/3295359116 >> v2:172.21.15.100:3300/0 conn(0x7ff87415cbe0 0x7ff8740cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.983+0000 7ff872a00640 1 -- 172.21.15.39:0/3295359116 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836237877 0 0) 0x7ff8740b8820 con 0x7ff8740ce510 2024-01-24T17:46:47.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.983+0000 7ff872a00640 1 -- 172.21.15.39:0/3295359116 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff854003520 con 0x7ff8740ce510 2024-01-24T17:46:47.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.983+0000 7ff878c00640 1 -- 172.21.15.39:0/3295359116 >> v2:172.21.15.100:3300/0 conn(0x7ff87415cbe0 msgr2=0x7ff8740cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:47.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.983+0000 7ff878c00640 1 --2- 172.21.15.39:0/3295359116 >> v2:172.21.15.100:3300/0 conn(0x7ff87415cbe0 0x7ff8740cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.983+0000 7ff878c00640 1 -- 172.21.15.39:0/3295359116 >> v1:172.21.15.39:6789/0 conn(0x7ff8740ce510 legacy=0x7ff8740d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.983+0000 7ff878c00640 1 -- 172.21.15.39:0/3295359116 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8740b5d30 con 0x7ff8740d15a0 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.984+0000 7ff878c00640 1 --2- 172.21.15.39:0/3295359116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8740d15a0 0x7ff874166b50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff87408baa0 tx=0x7ff86c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9a6dc66ad20686 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff872a00640 1 -- 172.21.15.39:0/3295359116 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c02cc80 con 0x7ff8740d15a0 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff872a00640 1 -- 172.21.15.39:0/3295359116 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c02cde0 con 0x7ff8740d15a0 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff872a00640 1 -- 172.21.15.39:0/3295359116 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c034660 con 0x7ff8740d15a0 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff87ac00640 1 -- 172.21.15.39:0/3295359116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8740d15a0 msgr2=0x7ff874166b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3295359116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8740d15a0 0x7ff874166b50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff87408baa0 tx=0x7ff86c02bed0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff87ac00640 1 -- 172.21.15.39:0/3295359116 shutdown_connections 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3295359116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8740d15a0 0x7ff874166b50 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3295359116 >> v2:172.21.15.100:3300/0 conn(0x7ff87415cbe0 0x7ff8740cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff87ac00640 1 -- 172.21.15.39:0/3295359116 >> 172.21.15.39:0/3295359116 conn(0x7ff87409bc80 msgr2=0x7ff8740d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:47.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.985+0000 7ff87ac00640 1 -- 172.21.15.39:0/3295359116 shutdown_connections 2024-01-24T17:46:47.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.986+0000 7ff87ac00640 1 -- 172.21.15.39:0/3295359116 wait complete. 2024-01-24T17:46:47.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.987+0000 7ff87ac00640 1 Processor -- start 2024-01-24T17:46:47.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.987+0000 7ff87ac00640 1 -- start start 2024-01-24T17:46:47.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.988+0000 7ff87ac00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8740ce510 0x7ff8741591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.988+0000 7ff87ac00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8740d15a0 0x7ff87415f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.988+0000 7ff87ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87415cbe0 0x7ff8740cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.988+0000 7ff87ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740c9d60 con 0x7ff87415cbe0 2024-01-24T17:46:47.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.988+0000 7ff87ac00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740ca040 con 0x7ff8740ce510 2024-01-24T17:46:47.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.988+0000 7ff87ac00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8740ca320 con 0x7ff8740d15a0 2024-01-24T17:46:47.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff878c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87415cbe0 0x7ff8740cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff878c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87415cbe0 0x7ff8740cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44830/0 (socket says 172.21.15.39:44830) 2024-01-24T17:46:47.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff878c00640 1 -- 172.21.15.39:0/3325759186 learned_addr learned my addr 172.21.15.39:0/3325759186 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:47.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff873e00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8740ce510 0x7ff8741591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff873400640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8740d15a0 0x7ff87415f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff878c00640 1 -- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8740d15a0 msgr2=0x7ff87415f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff878c00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8740d15a0 0x7ff87415f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff878c00640 1 -- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8740ce510 msgr2=0x7ff8741591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff878c00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8740ce510 0x7ff8741591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff878c00640 1 -- 172.21.15.39:0/3325759186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c009a70 con 0x7ff87415cbe0 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.989+0000 7ff873e00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8740ce510 0x7ff8741591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.990+0000 7ff878c00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87415cbe0 0x7ff8740cb8f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff86c02c4e0 tx=0x7ff86c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.990+0000 7ff86be00640 1 -- 172.21.15.39:0/3325759186 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c03e610 con 0x7ff87415cbe0 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.990+0000 7ff86be00640 1 -- 172.21.15.39:0/3325759186 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c03e770 con 0x7ff87415cbe0 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.990+0000 7ff86be00640 1 -- 172.21.15.39:0/3325759186 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c03eb60 con 0x7ff87415cbe0 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.990+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8740ca600 con 0x7ff87415cbe0 2024-01-24T17:46:47.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.990+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff874159730 con 0x7ff87415cbe0 2024-01-24T17:46:47.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.991+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8740cab20 con 0x7ff87415cbe0 2024-01-24T17:46:47.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.992+0000 7ff86be00640 1 -- 172.21.15.39:0/3325759186 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff86c03b050 con 0x7ff87415cbe0 2024-01-24T17:46:47.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.993+0000 7ff86be00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff84c06a430 0x7ff84c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:47.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.993+0000 7ff86be00640 1 -- 172.21.15.39:0/3325759186 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff86c0aab70 con 0x7ff87415cbe0 2024-01-24T17:46:47.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.994+0000 7ff873e00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff84c06a430 0x7ff84c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:47.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.995+0000 7ff873e00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff84c06a430 0x7ff84c06c8f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff864005fd0 tx=0x7ff864008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:47.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:47.998+0000 7ff86be00640 1 -- 172.21.15.39:0/3325759186 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff86c0760c0 con 0x7ff87415cbe0 2024-01-24T17:46:48.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.207+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff87404dff0 con 0x7ff87415cbe0 2024-01-24T17:46:48.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.208+0000 7ff86be00640 1 -- 172.21.15.39:0/3325759186 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v76) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff86c04d080 con 0x7ff87415cbe0 2024-01-24T17:46:48.209 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff84c06a430 msgr2=0x7ff84c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff84c06a430 0x7ff84c06c8f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff864005fd0 tx=0x7ff864008040 comp rx=0 tx=0).stop 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87415cbe0 msgr2=0x7ff8740cb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87415cbe0 0x7ff8740cb8f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7ff86c02c4e0 tx=0x7ff86c004a40 comp rx=0 tx=0).stop 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 shutdown_connections 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff84c06a430 0x7ff84c06c8f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87415cbe0 0x7ff8740cb8f0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff8740d15a0 0x7ff87415f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 --2- 172.21.15.39:0/3325759186 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff8740ce510 0x7ff8741591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 >> 172.21.15.39:0/3325759186 conn(0x7ff87409bc80 msgr2=0x7ff8740b6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:48.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 shutdown_connections 2024-01-24T17:46:48.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:48.211+0000 7ff87ac00640 1 -- 172.21.15.39:0/3325759186 wait complete. 2024-01-24T17:46:48.229 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:46:48.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:48.229 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:46:48.323 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[63482]: fuse finished with error 0 and tester_r 0 2024-01-24T17:46:48.327 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:46:51.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:46:51.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:46:54.429 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:46:54.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:54.429 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:46:54.491 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:54.492 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:54.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:54.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.777+0000 7f0bfea00640 1 Processor -- start 2024-01-24T17:46:54.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.778+0000 7f0bfea00640 1 -- start start 2024-01-24T17:46:54.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.779+0000 7f0bfea00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0bf80b2980 0x7f0bf80cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:54.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.779+0000 7f0bfea00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf80cf2e0 con 0x7f0bf815ce70 2024-01-24T17:46:54.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.779+0000 7f0bfea00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf806c820 con 0x7f0bf80cc840 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.779+0000 7f0bfea00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf80ccc50 con 0x7f0bf80b2980 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.780+0000 7f0bf7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0bf815ce70 0x7f0bf80b2270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41660/0 (socket says 172.21.15.39:41660) 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.780+0000 7f0bf7e00640 1 -- 172.21.15.39:0/3472711831 learned_addr learned my addr 172.21.15.39:0/3472711831 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.780+0000 7f0bf7400640 1 --2- 172.21.15.39:0/3472711831 >> v2:172.21.15.100:3300/0 conn(0x7f0bf80b2980 0x7f0bf80cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.780+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766995833 0 0) 0x7f0bf80cf2e0 con 0x7f0bf815ce70 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.780+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bd8003520 con 0x7f0bf815ce70 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 550299232 0 0) 0x7f0bd8003520 con 0x7f0bf815ce70 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 >> v2:172.21.15.100:3300/0 conn(0x7f0bf80b2980 msgr2=0x7f0bf80cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 --2- 172.21.15.39:0/3472711831 >> v2:172.21.15.100:3300/0 conn(0x7f0bf80b2980 0x7f0bf80cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 >> v1:172.21.15.100:6789/0 conn(0x7f0bf80cc840 legacy=0x7f0bf80cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf806cb00 con 0x7f0bf815ce70 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f0be80019f0 con 0x7f0bf815ce70 2024-01-24T17:46:54.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 >> v1:172.21.15.39:6789/0 conn(0x7f0bf815ce70 legacy=0x7f0bf80b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd80039c0 0x7f0bd8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd80047c0 0x7f0bd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd8005180 0x7f0bd800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80ccc50 con 0x7f0bd80047c0 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd800c080 con 0x7f0bd8005180 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd800c240 con 0x7f0bd80039c0 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.781+0000 7f0bf7400640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd80047c0 0x7f0bd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.782+0000 7f0bf7e00640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd80039c0 0x7f0bd8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.782+0000 7f0bfca00640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd8005180 0x7f0bd800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.782+0000 7f0bf7400640 1 -- 172.21.15.39:0/3472711831 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd80039c0 msgr2=0x7f0bd8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.782+0000 7f0bf7400640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bd80039c0 0x7f0bd8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.782+0000 7f0bf7400640 1 -- 172.21.15.39:0/3472711831 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd8005180 msgr2=0x7f0bd800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.782+0000 7f0bf7400640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd8005180 0x7f0bd800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.782+0000 7f0bfca00640 1 -- 172.21.15.39:0/3472711831 reap_dead start 2024-01-24T17:46:54.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.782+0000 7f0bf7400640 1 -- 172.21.15.39:0/3472711831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd8003520 con 0x7f0bd80047c0 2024-01-24T17:46:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.783+0000 7f0bf7400640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd80047c0 0x7f0bd8004c40 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009ac0 tx=0x7f0bd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17cc069ec85da106 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.783+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be002c6b0 con 0x7f0bd80047c0 2024-01-24T17:46:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.783+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be002c810 con 0x7f0bd80047c0 2024-01-24T17:46:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.783+0000 7f0bf6a00640 1 -- 172.21.15.39:0/3472711831 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be002cba0 con 0x7f0bd80047c0 2024-01-24T17:46:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.784+0000 7f0bfea00640 1 -- 172.21.15.39:0/3472711831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd80047c0 msgr2=0x7f0bd8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:54.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.784+0000 7f0bfea00640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd80047c0 0x7f0bd8004c40 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009ac0 tx=0x7f0bd80057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.784+0000 7f0bfea00640 1 -- 172.21.15.39:0/3472711831 shutdown_connections 2024-01-24T17:46:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.784+0000 7f0bfea00640 1 --2- 172.21.15.39:0/3472711831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bd80047c0 0x7f0bd8004c40 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.784+0000 7f0bfea00640 1 -- 172.21.15.39:0/3472711831 >> 172.21.15.39:0/3472711831 conn(0x7f0bf80cd270 msgr2=0x7f0bf80cd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.784+0000 7f0bfea00640 1 -- 172.21.15.39:0/3472711831 shutdown_connections 2024-01-24T17:46:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.784+0000 7f0bfea00640 1 -- 172.21.15.39:0/3472711831 wait complete. 2024-01-24T17:46:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.786+0000 7f0bfea00640 1 Processor -- start 2024-01-24T17:46:54.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.786+0000 7f0bfea00640 1 -- start start 2024-01-24T17:46:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bfea00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd80047c0 0x7f0bf814cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bfea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bf80c78c0 0x7f0bf80c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bfea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bf80c7d40 0x7f0bf80c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bfea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80b0530 con 0x7f0bf80c78c0 2024-01-24T17:46:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bfea00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80b4330 con 0x7f0bd80047c0 2024-01-24T17:46:54.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bfea00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80b4610 con 0x7f0bf80c7d40 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bf7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bf80c78c0 0x7f0bf80c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bf7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bf80c78c0 0x7f0bf80c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52164/0 (socket says 172.21.15.39:52164) 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bf7400640 1 -- 172.21.15.39:0/3048625440 learned_addr learned my addr 172.21.15.39:0/3048625440 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bfca00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bf80c7d40 0x7f0bf80c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.788+0000 7f0bf7e00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd80047c0 0x7f0bf814cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0bf7400640 1 -- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bf80c7d40 msgr2=0x7f0bf80c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0bf7400640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bf80c7d40 0x7f0bf80c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0bf7400640 1 -- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd80047c0 msgr2=0x7f0bf814cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0bf7400640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd80047c0 0x7f0bf814cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0bf7400640 1 -- 172.21.15.39:0/3048625440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf80afee0 con 0x7f0bf80c78c0 2024-01-24T17:46:54.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0bf7400640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bf80c78c0 0x7f0bf80c9620 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f0be002c250 tx=0x7f0be0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:54.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0befe00640 1 -- 172.21.15.39:0/3048625440 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be0003810 con 0x7f0bf80c78c0 2024-01-24T17:46:54.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0befe00640 1 -- 172.21.15.39:0/3048625440 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be0003970 con 0x7f0bf80c78c0 2024-01-24T17:46:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0befe00640 1 -- 172.21.15.39:0/3048625440 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be0003d00 con 0x7f0bf80c78c0 2024-01-24T17:46:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.789+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf80b01c0 con 0x7f0bf80c78c0 2024-01-24T17:46:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.790+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf80b0ba0 con 0x7f0bf80c78c0 2024-01-24T17:46:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.791+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf80b0380 con 0x7f0bf80c78c0 2024-01-24T17:46:54.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.792+0000 7f0befe00640 1 -- 172.21.15.39:0/3048625440 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0be0034440 con 0x7f0bf80c78c0 2024-01-24T17:46:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.793+0000 7f0befe00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bd006a430 0x7f0bd006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:54.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.793+0000 7f0befe00640 1 -- 172.21.15.39:0/3048625440 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0be00aac80 con 0x7f0bf80c78c0 2024-01-24T17:46:54.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.793+0000 7f0bf7e00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bd006a430 0x7f0bd006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:54.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.795+0000 7f0bf7e00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bd006a430 0x7f0bd006c8f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0be8000f80 tx=0x7f0be8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:54.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.798+0000 7f0befe00640 1 -- 172.21.15.39:0/3048625440 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0be0076320 con 0x7f0bf80c78c0 2024-01-24T17:46:54.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.993+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bf804dff0 con 0x7f0bf80c78c0 2024-01-24T17:46:54.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.994+0000 7f0befe00640 1 -- 172.21.15.39:0/3048625440 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16040 (secure 0 0 0) 0x7f0be004c090 con 0x7f0bf80c78c0 2024-01-24T17:46:54.995 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:54.995 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":76,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:32.019249+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:54.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bd006a430 msgr2=0x7f0bd006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:54.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bd006a430 0x7f0bd006c8f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0be8000f80 tx=0x7f0be8009040 comp rx=0 tx=0).stop 2024-01-24T17:46:54.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bf80c78c0 msgr2=0x7f0bf80c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:54.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bf80c78c0 0x7f0bf80c9620 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f0be002c250 tx=0x7f0be0004730 comp rx=0 tx=0).stop 2024-01-24T17:46:54.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 shutdown_connections 2024-01-24T17:46:54.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0bf80c7d40 0x7f0bf80c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0bd006a430 0x7f0bd006c8f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0bf80c78c0 0x7f0bf80c9620 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 --2- 172.21.15.39:0/3048625440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0bd80047c0 0x7f0bf814cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:54.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 >> 172.21.15.39:0/3048625440 conn(0x7f0bf80cd270 msgr2=0x7f0bf80cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:54.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 shutdown_connections 2024-01-24T17:46:54.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:54.997+0000 7f0bfea00640 1 -- 172.21.15.39:0/3048625440 wait complete. 2024-01-24T17:46:55.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:46:29.228451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:46:30.039417+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:46:55.018 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:46:55.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.241+0000 7f62f1600640 1 Processor -- start 2024-01-24T17:46:55.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.241+0000 7f62f1600640 1 -- start start 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62f1600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62f1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f62ec06ddb0 0x7f62ec0b2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62f1600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec06a8f0 con 0x7f62ec15a9c0 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62f1600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ec069ce0 con 0x7f62ec06ddb0 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62f1600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec069fc0 con 0x7f62ec065a10 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62f0c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f62ec06ddb0 0x7f62ec0b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62f0c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f62ec06ddb0 0x7f62ec0b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52172/0 (socket says 172.21.15.39:52172) 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62f0c00640 1 -- 172.21.15.39:0/1590507233 learned_addr learned my addr 172.21.15.39:0/1590507233 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:55.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.243+0000 7f62eb400640 1 --2- 172.21.15.39:0/1590507233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.244+0000 7f62f0c00640 1 -- 172.21.15.39:0/1590507233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 msgr2=0x7f62ec0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:55.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.244+0000 7f62f0c00640 1 --2- 172.21.15.39:0/1590507233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec0b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.244+0000 7f62f0c00640 1 -- 172.21.15.39:0/1590507233 >> v1:172.21.15.100:6789/0 conn(0x7f62ec15a9c0 legacy=0x7f62ec06ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.244+0000 7f62f0c00640 1 -- 172.21.15.39:0/1590507233 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec06a2a0 con 0x7f62ec06ddb0 2024-01-24T17:46:55.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.245+0000 7f62f0c00640 1 --2- 172.21.15.39:0/1590507233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec0b2030 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009e50 tx=0x7f62e402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b5c23f74f36a62 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:55.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.245+0000 7f62eaa00640 1 -- 172.21.15.39:0/1590507233 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e402cb00 con 0x7f62ec06ddb0 2024-01-24T17:46:55.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.245+0000 7f62eaa00640 1 -- 172.21.15.39:0/1590507233 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e402cc60 con 0x7f62ec06ddb0 2024-01-24T17:46:55.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.245+0000 7f62eaa00640 1 -- 172.21.15.39:0/1590507233 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e40343f0 con 0x7f62ec06ddb0 2024-01-24T17:46:55.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.246+0000 7f62f1600640 1 -- 172.21.15.39:0/1590507233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 msgr2=0x7f62ec0b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.246+0000 7f62f1600640 1 --2- 172.21.15.39:0/1590507233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec0b2030 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009e50 tx=0x7f62e402bf60 comp rx=0 tx=0).stop 2024-01-24T17:46:55.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.246+0000 7f62f1600640 1 -- 172.21.15.39:0/1590507233 shutdown_connections 2024-01-24T17:46:55.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.246+0000 7f62f1600640 1 --2- 172.21.15.39:0/1590507233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec0b2030 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.246+0000 7f62f1600640 1 --2- 172.21.15.39:0/1590507233 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec0b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.246+0000 7f62f1600640 1 -- 172.21.15.39:0/1590507233 >> 172.21.15.39:0/1590507233 conn(0x7f62ec0b0790 msgr2=0x7f62ec0b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:55.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.246+0000 7f62f1600640 1 -- 172.21.15.39:0/1590507233 shutdown_connections 2024-01-24T17:46:55.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.246+0000 7f62f1600640 1 -- 172.21.15.39:0/1590507233 wait complete. 2024-01-24T17:46:55.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.248+0000 7f62f1600640 1 Processor -- start 2024-01-24T17:46:55.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.248+0000 7f62f1600640 1 -- start start 2024-01-24T17:46:55.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.249+0000 7f62f1600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec1f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.249+0000 7f62f1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.249+0000 7f62f1600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ec15a9c0 0x7f62ec0cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.249+0000 7f62f1600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec1e2930 con 0x7f62ec06ddb0 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.249+0000 7f62f1600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec158da0 con 0x7f62ec15a9c0 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.249+0000 7f62f1600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec0c5300 con 0x7f62ec065a10 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.249+0000 7f62eb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62eb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52186/0 (socket says 172.21.15.39:52186) 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62eb400640 1 -- 172.21.15.39:0/1574717849 learned_addr learned my addr 172.21.15.39:0/1574717849 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62ebe00640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec1f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62eb400640 1 -- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 msgr2=0x7f62ec1f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62eb400640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec1f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62eb400640 1 -- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ec15a9c0 msgr2=0x7f62ec0cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62eb400640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ec15a9c0 0x7f62ec0cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62eb400640 1 -- 172.21.15.39:0/1574717849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4009b30 con 0x7f62ec06ddb0 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62ebe00640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec1f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:55.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.250+0000 7f62eb400640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec158860 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f62d400e9b0 tx=0x7f62d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:55.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.251+0000 7f62e3e00640 1 -- 172.21.15.39:0/1574717849 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4009670 con 0x7f62ec06ddb0 2024-01-24T17:46:55.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.251+0000 7f62e3e00640 1 -- 172.21.15.39:0/1574717849 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d400c9a0 con 0x7f62ec06ddb0 2024-01-24T17:46:55.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.251+0000 7f62e3e00640 1 -- 172.21.15.39:0/1574717849 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d400cd50 con 0x7f62ec06ddb0 2024-01-24T17:46:55.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.251+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec155920 con 0x7f62ec06ddb0 2024-01-24T17:46:55.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.251+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec0cdbb0 con 0x7f62ec06ddb0 2024-01-24T17:46:55.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.252+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec1e2a70 con 0x7f62ec06ddb0 2024-01-24T17:46:55.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.253+0000 7f62e3e00640 1 -- 172.21.15.39:0/1574717849 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f62d4013470 con 0x7f62ec06ddb0 2024-01-24T17:46:55.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.254+0000 7f62e3e00640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62c406a430 0x7f62c406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.254+0000 7f62e3e00640 1 -- 172.21.15.39:0/1574717849 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62d4088c80 con 0x7f62ec06ddb0 2024-01-24T17:46:55.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.254+0000 7f62ebe00640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62c406a430 0x7f62c406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.256+0000 7f62ebe00640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62c406a430 0x7f62c406c8f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f62dc000f60 tx=0x7f62dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:55.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.258+0000 7f62e3e00640 1 -- 172.21.15.39:0/1574717849 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f62ec1e2a70 con 0x7f62ec06ddb0 2024-01-24T17:46:55.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.454+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62ec1e2a70 con 0x7f62ec06ddb0 2024-01-24T17:46:55.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.455+0000 7f62e3e00640 1 -- 172.21.15.39:0/1574717849 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62ec1e2a70 con 0x7f62ec06ddb0 2024-01-24T17:46:55.456 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:46:55.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.457+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62c406a430 msgr2=0x7f62c406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.457+0000 7f62f1600640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62c406a430 0x7f62c406c8f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f62dc000f60 tx=0x7f62dc009040 comp rx=0 tx=0).stop 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.457+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 msgr2=0x7f62ec158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.457+0000 7f62f1600640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec158860 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f62d400e9b0 tx=0x7f62d400ee70 comp rx=0 tx=0).stop 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.458+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 shutdown_connections 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.458+0000 7f62f1600640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f62ec15a9c0 0x7f62ec0cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.458+0000 7f62f1600640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f62c406a430 0x7f62c406c8f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.458+0000 7f62f1600640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62ec06ddb0 0x7f62ec158860 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.458+0000 7f62f1600640 1 --2- 172.21.15.39:0/1574717849 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f62ec065a10 0x7f62ec1f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.458+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 >> 172.21.15.39:0/1574717849 conn(0x7f62ec0b0790 msgr2=0x7f62ec0b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.458+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 shutdown_connections 2024-01-24T17:46:55.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.458+0000 7f62f1600640 1 -- 172.21.15.39:0/1574717849 wait complete. 2024-01-24T17:46:55.478 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:46:55.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.707+0000 7fb9e6e00640 1 Processor -- start 2024-01-24T17:46:55.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.707+0000 7fb9e6e00640 1 -- start start 2024-01-24T17:46:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.708+0000 7fb9e6e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb9e00cb990 0x7fb9e00ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.708+0000 7fb9e6e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb9e006cec0 0x7fb9e006d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.708+0000 7fb9e6e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00ca4c0 con 0x7fb9e015aa60 2024-01-24T17:46:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.708+0000 7fb9e6e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e00cae90 con 0x7fb9e00cb990 2024-01-24T17:46:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.708+0000 7fb9e6e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e006dc90 con 0x7fb9e006cec0 2024-01-24T17:46:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.708+0000 7fb9dfe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb9e015aa60 0x7fb9e00cb280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41676/0 (socket says 172.21.15.39:41676) 2024-01-24T17:46:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9dfe00640 1 -- 172.21.15.39:0/2082131663 learned_addr learned my addr 172.21.15.39:0/2082131663 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9d7200640 1 --2- 172.21.15.39:0/2082131663 >> v2:172.21.15.39:3300/0 conn(0x7fb9e00cb990 0x7fb9e00ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9e4e00640 1 --2- 172.21.15.39:0/2082131663 >> v2:172.21.15.100:3300/0 conn(0x7fb9e006cec0 0x7fb9e006d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9d7200640 1 -- 172.21.15.39:0/2082131663 >> v2:172.21.15.100:3300/0 conn(0x7fb9e006cec0 msgr2=0x7fb9e006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9d7200640 1 --2- 172.21.15.39:0/2082131663 >> v2:172.21.15.100:3300/0 conn(0x7fb9e006cec0 0x7fb9e006d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9d7200640 1 -- 172.21.15.39:0/2082131663 >> v1:172.21.15.39:6789/0 conn(0x7fb9e015aa60 legacy=0x7fb9e00cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9df400640 1 -- 172.21.15.39:0/2082131663 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706280480 0 0) 0x7fb9e00ca4c0 con 0x7fb9e015aa60 2024-01-24T17:46:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9d7200640 1 -- 172.21.15.39:0/2082131663 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e00c9e70 con 0x7fb9e00cb990 2024-01-24T17:46:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.709+0000 7fb9e4e00640 1 --2- 172.21.15.39:0/2082131663 >> v2:172.21.15.100:3300/0 conn(0x7fb9e006cec0 0x7fb9e006d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.710+0000 7fb9d7200640 1 --2- 172.21.15.39:0/2082131663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e00cb990 0x7fb9e00ca950 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00ad600 tx=0x7fb9c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159194403af233ec server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:55.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9df400640 1 -- 172.21.15.39:0/2082131663 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8031830 con 0x7fb9e00cb990 2024-01-24T17:46:55.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9df400640 1 -- 172.21.15.39:0/2082131663 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c8031990 con 0x7fb9e00cb990 2024-01-24T17:46:55.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9df400640 1 -- 172.21.15.39:0/2082131663 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8031d20 con 0x7fb9e00cb990 2024-01-24T17:46:55.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9e6e00640 1 -- 172.21.15.39:0/2082131663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e00cb990 msgr2=0x7fb9e00ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/2082131663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e00cb990 0x7fb9e00ca950 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00ad600 tx=0x7fb9c802b8d0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9e6e00640 1 -- 172.21.15.39:0/2082131663 shutdown_connections 2024-01-24T17:46:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/2082131663 >> v2:172.21.15.100:3300/0 conn(0x7fb9e006cec0 0x7fb9e006d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/2082131663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e00cb990 0x7fb9e00ca950 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9e6e00640 1 -- 172.21.15.39:0/2082131663 >> 172.21.15.39:0/2082131663 conn(0x7fb9e00ce9e0 msgr2=0x7fb9e00cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9e6e00640 1 -- 172.21.15.39:0/2082131663 shutdown_connections 2024-01-24T17:46:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.711+0000 7fb9e6e00640 1 -- 172.21.15.39:0/2082131663 wait complete. 2024-01-24T17:46:55.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.713+0000 7fb9e6e00640 1 Processor -- start 2024-01-24T17:46:55.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.713+0000 7fb9e6e00640 1 -- start start 2024-01-24T17:46:55.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.714+0000 7fb9e6e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9e006cec0 0x7fb9e01f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.714+0000 7fb9e6e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9e00cb990 0x7fb9e0071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.714+0000 7fb9e6e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e015aa60 0x7fb9e00c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.714+0000 7fb9e6e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00cc190 con 0x7fb9e015aa60 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.714+0000 7fb9e6e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e0152740 con 0x7fb9e00cb990 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.714+0000 7fb9e6e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00cc3e0 con 0x7fb9e006cec0 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e015aa60 0x7fb9e00c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e015aa60 0x7fb9e00c3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52196/0 (socket says 172.21.15.39:52196) 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 -- 172.21.15.39:0/1684396183 learned_addr learned my addr 172.21.15.39:0/1684396183 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9dfe00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9e006cec0 0x7fb9e01f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9d7200640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9e00cb990 0x7fb9e0071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 -- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9e006cec0 msgr2=0x7fb9e01f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9e006cec0 0x7fb9e01f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 -- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9e00cb990 msgr2=0x7fb9e0071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9e00cb990 0x7fb9e0071370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 -- 172.21.15.39:0/1684396183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c8009400 con 0x7fb9e015aa60 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9d7200640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9e00cb990 0x7fb9e0071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.715+0000 7fb9e4e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e015aa60 0x7fb9e00c3f60 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c7c0 tx=0x7fb9d800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.716+0000 7fb9dcc00640 1 -- 172.21.15.39:0/1684396183 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8004230 con 0x7fb9e015aa60 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.716+0000 7fb9dcc00640 1 -- 172.21.15.39:0/1684396183 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d8004390 con 0x7fb9e015aa60 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.716+0000 7fb9dcc00640 1 -- 172.21.15.39:0/1684396183 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8007860 con 0x7fb9e015aa60 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.716+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e00cc6c0 con 0x7fb9e015aa60 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.716+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e00ccc80 con 0x7fb9e015aa60 2024-01-24T17:46:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.717+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e00ccea0 con 0x7fb9e015aa60 2024-01-24T17:46:55.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.718+0000 7fb9dcc00640 1 -- 172.21.15.39:0/1684396183 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb9d80079c0 con 0x7fb9e015aa60 2024-01-24T17:46:55.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.719+0000 7fb9dcc00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb9b806a430 0x7fb9b806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:55.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.719+0000 7fb9dcc00640 1 -- 172.21.15.39:0/1684396183 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9d8088c70 con 0x7fb9e015aa60 2024-01-24T17:46:55.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.720+0000 7fb9dfe00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb9b806a430 0x7fb9b806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.721+0000 7fb9dfe00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb9b806a430 0x7fb9b806c8f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00012e0 tx=0x7fb9d0007ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.724+0000 7fb9dcc00640 1 -- 172.21.15.39:0/1684396183 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb9d8054280 con 0x7fb9e015aa60 2024-01-24T17:46:55.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.914+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9e004dff0 con 0x7fb9e015aa60 2024-01-24T17:46:55.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.915+0000 7fb9dcc00640 1 -- 172.21.15.39:0/1684396183 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb9d80578a0 con 0x7fb9e015aa60 2024-01-24T17:46:55.916 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb9b806a430 msgr2=0x7fb9b806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb9b806a430 0x7fb9b806c8f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00012e0 tx=0x7fb9d0007ed0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e015aa60 msgr2=0x7fb9e00c3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e015aa60 0x7fb9e00c3f60 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c7c0 tx=0x7fb9d800cc80 comp rx=0 tx=0).stop 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 shutdown_connections 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb9b806a430 0x7fb9b806c8f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9e015aa60 0x7fb9e00c3f60 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9e00cb990 0x7fb9e0071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 --2- 172.21.15.39:0/1684396183 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb9e006cec0 0x7fb9e01f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:55.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 >> 172.21.15.39:0/1684396183 conn(0x7fb9e00ce9e0 msgr2=0x7fb9e015e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:55.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 shutdown_connections 2024-01-24T17:46:55.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:55.918+0000 7fb9e6e00640 1 -- 172.21.15.39:0/1684396183 wait complete. 2024-01-24T17:46:55.938 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:46:56.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.157+0000 7fc6c7400640 1 Processor -- start 2024-01-24T17:46:56.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.157+0000 7fc6c7400640 1 -- start start 2024-01-24T17:46:56.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6c0070460 0x7fc6c00b1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c7400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc6c015ac70 0x7fc6c00b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c7400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c0155170 con 0x7fc6c00ce8c0 2024-01-24T17:46:56.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c7400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c00b12b0 con 0x7fc6c0070460 2024-01-24T17:46:56.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c7400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c015b1e0 con 0x7fc6c015ac70 2024-01-24T17:46:56.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c5600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6c0070460 0x7fc6c00b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c5600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc6c0070460 0x7fc6c00b1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52206/0 (socket says 172.21.15.39:52206) 2024-01-24T17:46:56.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c5600640 1 -- 172.21.15.39:0/4150425946 learned_addr learned my addr 172.21.15.39:0/4150425946 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:56.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.159+0000 7fc6c6a00640 1 --2- 172.21.15.39:0/4150425946 >> v2:172.21.15.100:3300/0 conn(0x7fc6c015ac70 0x7fc6c00b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.160+0000 7fc6c5600640 1 -- 172.21.15.39:0/4150425946 >> v2:172.21.15.100:3300/0 conn(0x7fc6c015ac70 msgr2=0x7fc6c00b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:56.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.160+0000 7fc6c5600640 1 --2- 172.21.15.39:0/4150425946 >> v2:172.21.15.100:3300/0 conn(0x7fc6c015ac70 0x7fc6c00b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.160+0000 7fc6c5600640 1 -- 172.21.15.39:0/4150425946 >> v1:172.21.15.100:6789/0 conn(0x7fc6c00ce8c0 legacy=0x7fc6c006fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:56.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.160+0000 7fc6c5600640 1 -- 172.21.15.39:0/4150425946 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c015b4c0 con 0x7fc6c0070460 2024-01-24T17:46:56.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.161+0000 7fc6c5600640 1 --2- 172.21.15.39:0/4150425946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c0070460 0x7fc6c00b1700 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0071c30 tx=0x7fc6a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad17d32c39f54d75 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:56.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.161+0000 7fc6c4c00640 1 -- 172.21.15.39:0/4150425946 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a802ca50 con 0x7fc6c0070460 2024-01-24T17:46:56.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.161+0000 7fc6c4c00640 1 -- 172.21.15.39:0/4150425946 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a802cbb0 con 0x7fc6c0070460 2024-01-24T17:46:56.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.161+0000 7fc6c4c00640 1 -- 172.21.15.39:0/4150425946 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a80343f0 con 0x7fc6c0070460 2024-01-24T17:46:56.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.162+0000 7fc6c7400640 1 -- 172.21.15.39:0/4150425946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c0070460 msgr2=0x7fc6c00b1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.162+0000 7fc6c7400640 1 --2- 172.21.15.39:0/4150425946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c0070460 0x7fc6c00b1700 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0071c30 tx=0x7fc6a802beb0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.162+0000 7fc6c7400640 1 -- 172.21.15.39:0/4150425946 shutdown_connections 2024-01-24T17:46:56.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.162+0000 7fc6c7400640 1 --2- 172.21.15.39:0/4150425946 >> v2:172.21.15.100:3300/0 conn(0x7fc6c015ac70 0x7fc6c00b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.162+0000 7fc6c7400640 1 --2- 172.21.15.39:0/4150425946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c0070460 0x7fc6c00b1700 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.162+0000 7fc6c7400640 1 -- 172.21.15.39:0/4150425946 >> 172.21.15.39:0/4150425946 conn(0x7fc6c006a200 msgr2=0x7fc6c00affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:56.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.162+0000 7fc6c7400640 1 -- 172.21.15.39:0/4150425946 shutdown_connections 2024-01-24T17:46:56.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.162+0000 7fc6c7400640 1 -- 172.21.15.39:0/4150425946 wait complete. 2024-01-24T17:46:56.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.164+0000 7fc6c7400640 1 Processor -- start 2024-01-24T17:46:56.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.164+0000 7fc6c7400640 1 -- start start 2024-01-24T17:46:56.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.165+0000 7fc6c7400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6c0070460 0x7fc6c0150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.165+0000 7fc6c7400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6c00ce8c0 0x7fc6c014e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.165+0000 7fc6c7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c015ac70 0x7fc6c0158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.165+0000 7fc6c7400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c0159040 con 0x7fc6c015ac70 2024-01-24T17:46:56.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.165+0000 7fc6c7400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c01506f0 con 0x7fc6c00ce8c0 2024-01-24T17:46:56.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.165+0000 7fc6c7400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c01509d0 con 0x7fc6c0070460 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.165+0000 7fc6c6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c015ac70 0x7fc6c0158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c015ac70 0x7fc6c0158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52220/0 (socket says 172.21.15.39:52220) 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6a00640 1 -- 172.21.15.39:0/1851768795 learned_addr learned my addr 172.21.15.39:0/1851768795 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6a00640 1 -- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6c0070460 msgr2=0x7fc6c0150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c5600640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6c00ce8c0 0x7fc6c014e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6000640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6c0070460 0x7fc6c0150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6a00640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6c0070460 0x7fc6c0150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6a00640 1 -- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6c00ce8c0 msgr2=0x7fc6c014e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6a00640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6c00ce8c0 0x7fc6c014e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6a00640 1 -- 172.21.15.39:0/1851768795 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8009b30 con 0x7fc6c015ac70 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c5600640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6c00ce8c0 0x7fc6c014e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.166+0000 7fc6c6a00640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c015ac70 0x7fc6c0158b00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800d5d0 tx=0x7fc6b800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:56.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.167+0000 7fc6be000640 1 -- 172.21.15.39:0/1851768795 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80125f0 con 0x7fc6c015ac70 2024-01-24T17:46:56.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.167+0000 7fc6be000640 1 -- 172.21.15.39:0/1851768795 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b8012770 con 0x7fc6c015ac70 2024-01-24T17:46:56.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.167+0000 7fc6be000640 1 -- 172.21.15.39:0/1851768795 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8012b20 con 0x7fc6c015ac70 2024-01-24T17:46:56.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.167+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c0150cb0 con 0x7fc6c015ac70 2024-01-24T17:46:56.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.167+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c0151270 con 0x7fc6c015ac70 2024-01-24T17:46:56.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.168+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c00738c0 con 0x7fc6c015ac70 2024-01-24T17:46:56.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.169+0000 7fc6be000640 1 -- 172.21.15.39:0/1851768795 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc6b8012c80 con 0x7fc6c015ac70 2024-01-24T17:46:56.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.170+0000 7fc6be000640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69806a430 0x7fc69806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.170+0000 7fc6be000640 1 -- 172.21.15.39:0/1851768795 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6b8088e50 con 0x7fc6c015ac70 2024-01-24T17:46:56.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.170+0000 7fc6c6000640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69806a430 0x7fc69806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.172+0000 7fc6c6000640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69806a430 0x7fc69806c8f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0003d90 tx=0x7fc6b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:56.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.174+0000 7fc6be000640 1 -- 172.21.15.39:0/1851768795 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc6b80543b0 con 0x7fc6c015ac70 2024-01-24T17:46:56.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.371+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6c0151460 con 0x7fc6c015ac70 2024-01-24T17:46:56.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.372+0000 7fc6be000640 1 -- 172.21.15.39:0/1851768795 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc6b80579d0 con 0x7fc6c015ac70 2024-01-24T17:46:56.373 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:46:56.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69806a430 msgr2=0x7fc69806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69806a430 0x7fc69806c8f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0003d90 tx=0x7fc6b0009040 comp rx=0 tx=0).stop 2024-01-24T17:46:56.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c015ac70 msgr2=0x7fc6c0158b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c015ac70 0x7fc6c0158b00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800d5d0 tx=0x7fc6b800da90 comp rx=0 tx=0).stop 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 shutdown_connections 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc69806a430 0x7fc69806c8f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6c015ac70 0x7fc6c0158b00 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6c00ce8c0 0x7fc6c014e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 --2- 172.21.15.39:0/1851768795 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc6c0070460 0x7fc6c0150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 >> 172.21.15.39:0/1851768795 conn(0x7fc6c006a200 msgr2=0x7fc6c006bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 shutdown_connections 2024-01-24T17:46:56.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.375+0000 7fc6c7400640 1 -- 172.21.15.39:0/1851768795 wait complete. 2024-01-24T17:46:56.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:46:56.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.618+0000 7f0fbbc00640 1 Processor -- start 2024-01-24T17:46:56.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.618+0000 7f0fbbc00640 1 -- start start 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fbbc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0fb40d0e90 0x7f0fb40d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fbbc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb4164b40 0x7f0fb4166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fbbc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb40d1820 con 0x7f0fb415cbe0 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fbbc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb40b8820 con 0x7f0fb40d0e90 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fbbc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40b5790 con 0x7f0fb4164b40 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fb3e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0fb40d0e90 0x7f0fb40d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fb9c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb4164b40 0x7f0fb4166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fb3e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0fb40d0e90 0x7f0fb40d12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:33018/0 (socket says 172.21.15.39:33018) 2024-01-24T17:46:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.620+0000 7f0fb3e00640 1 -- 172.21.15.39:0/2440868801 learned_addr learned my addr 172.21.15.39:0/2440868801 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:56.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.621+0000 7f0fb3e00640 1 -- 172.21.15.39:0/2440868801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb4164b40 msgr2=0x7f0fb4166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:56.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.621+0000 7f0fb3e00640 1 --2- 172.21.15.39:0/2440868801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb4164b40 0x7f0fb4166f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.621+0000 7f0fb3e00640 1 -- 172.21.15.39:0/2440868801 >> v1:172.21.15.100:6789/0 conn(0x7f0fb415cbe0 legacy=0x7f0fb40cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.621+0000 7f0fb3e00640 1 -- 172.21.15.39:0/2440868801 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb40b5a50 con 0x7f0fb40d0e90 2024-01-24T17:46:56.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.622+0000 7f0fb3e00640 1 --2- 172.21.15.39:0/2440868801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb40d0e90 0x7f0fb40d12e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009870 tx=0x7f0f9c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b880191f61e52c0c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:56.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.623+0000 7f0fb3400640 1 -- 172.21.15.39:0/2440868801 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c030840 con 0x7f0fb40d0e90 2024-01-24T17:46:56.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.623+0000 7f0fb3400640 1 -- 172.21.15.39:0/2440868801 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c0309a0 con 0x7f0fb40d0e90 2024-01-24T17:46:56.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.623+0000 7f0fb3400640 1 -- 172.21.15.39:0/2440868801 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c030d30 con 0x7f0fb40d0e90 2024-01-24T17:46:56.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.624+0000 7f0fbbc00640 1 -- 172.21.15.39:0/2440868801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb40d0e90 msgr2=0x7f0fb40d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.624+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/2440868801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb40d0e90 0x7f0fb40d12e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009870 tx=0x7f0f9c02b980 comp rx=0 tx=0).stop 2024-01-24T17:46:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.624+0000 7f0fbbc00640 1 -- 172.21.15.39:0/2440868801 shutdown_connections 2024-01-24T17:46:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.624+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/2440868801 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb40d0e90 0x7f0fb40d12e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.624+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/2440868801 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb4164b40 0x7f0fb4166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.624+0000 7f0fbbc00640 1 -- 172.21.15.39:0/2440868801 >> 172.21.15.39:0/2440868801 conn(0x7f0fb409bc80 msgr2=0x7f0fb40d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.624+0000 7f0fbbc00640 1 -- 172.21.15.39:0/2440868801 shutdown_connections 2024-01-24T17:46:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.624+0000 7f0fbbc00640 1 -- 172.21.15.39:0/2440868801 wait complete. 2024-01-24T17:46:56.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.626+0000 7f0fbbc00640 1 Processor -- start 2024-01-24T17:46:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.626+0000 7f0fbbc00640 1 -- start start 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.627+0000 7f0fbbc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb40d0e90 0x7f0fb40c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.627+0000 7f0fbbc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb415cbe0 0x7f0fb4150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.627+0000 7f0fbbc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fb4164b40 0x7f0fb4159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.627+0000 7f0fbbc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40c8450 con 0x7f0fb4164b40 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.627+0000 7f0fbbc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40c88e0 con 0x7f0fb415cbe0 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.627+0000 7f0fbbc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4152410 con 0x7f0fb40d0e90 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fb4164b40 0x7f0fb4159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fb4164b40 0x7f0fb4159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52236/0 (socket says 172.21.15.39:52236) 2024-01-24T17:46:56.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9c00640 1 -- 172.21.15.39:0/3850412739 learned_addr learned my addr 172.21.15.39:0/3850412739 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9200640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb40d0e90 0x7f0fb40c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb3e00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb415cbe0 0x7f0fb4150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9c00640 1 -- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb40d0e90 msgr2=0x7f0fb40c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9c00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb40d0e90 0x7f0fb40c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9c00640 1 -- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb415cbe0 msgr2=0x7f0fb4150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9c00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb415cbe0 0x7f0fb4150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9c00640 1 -- 172.21.15.39:0/3850412739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c0094d0 con 0x7f0fb4164b40 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.628+0000 7f0fb9200640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb40d0e90 0x7f0fb40c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.629+0000 7f0fb9c00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fb4164b40 0x7f0fb4159720 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00efc0 tx=0x7f0fac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.629+0000 7f0fb0c00640 1 -- 172.21.15.39:0/3850412739 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac009b20 con 0x7f0fb4164b40 2024-01-24T17:46:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.629+0000 7f0fb0c00640 1 -- 172.21.15.39:0/3850412739 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac00f040 con 0x7f0fb4164b40 2024-01-24T17:46:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.629+0000 7f0fb0c00640 1 -- 172.21.15.39:0/3850412739 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0146c0 con 0x7f0fb4164b40 2024-01-24T17:46:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.629+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb40c7b20 con 0x7f0fb4164b40 2024-01-24T17:46:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.629+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb40c80e0 con 0x7f0fb4164b40 2024-01-24T17:46:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.630+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb40d0840 con 0x7f0fb4164b40 2024-01-24T17:46:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.631+0000 7f0fb0c00640 1 -- 172.21.15.39:0/3850412739 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0fac014820 con 0x7f0fb4164b40 2024-01-24T17:46:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.632+0000 7f0fb0c00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f8c06a460 0x7f0f8c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.632+0000 7f0fb0c00640 1 -- 172.21.15.39:0/3850412739 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fac089750 con 0x7f0fb4164b40 2024-01-24T17:46:56.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.633+0000 7f0fb9200640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f8c06a460 0x7f0f8c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:56.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.634+0000 7f0fb9200640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f8c06a460 0x7f0f8c06c920 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4000fb0 tx=0x7f0fa4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:56.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.637+0000 7f0fb0c00640 1 -- 172.21.15.39:0/3850412739 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0fac054ca0 con 0x7f0fb4164b40 2024-01-24T17:46:56.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.844+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0fb404dff0 con 0x7f0fb4164b40 2024-01-24T17:46:56.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.845+0000 7f0fb0c00640 1 -- 172.21.15.39:0/3850412739 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v76) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0fac009e60 con 0x7f0fb4164b40 2024-01-24T17:46:56.846 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f8c06a460 msgr2=0x7f0f8c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f8c06a460 0x7f0f8c06c920 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4000fb0 tx=0x7f0fa4009040 comp rx=0 tx=0).stop 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fb4164b40 msgr2=0x7f0fb4159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fb4164b40 0x7f0fb4159720 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00efc0 tx=0x7f0fac00c490 comp rx=0 tx=0).stop 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 shutdown_connections 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f8c06a460 0x7f0f8c06c920 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fb4164b40 0x7f0fb4159720 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fb415cbe0 0x7f0fb4150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 --2- 172.21.15.39:0/3850412739 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fb40d0e90 0x7f0fb40c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 >> 172.21.15.39:0/3850412739 conn(0x7f0fb409bc80 msgr2=0x7f0fb41573e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 shutdown_connections 2024-01-24T17:46:56.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:56.848+0000 7f0fbbc00640 1 -- 172.21.15.39:0/3850412739 wait complete. 2024-01-24T17:46:56.870 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:46:56.870 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:46:56.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:46:57.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.156+0000 7f930f200640 1 Processor -- start 2024-01-24T17:46:57.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.156+0000 7f930f200640 1 -- start start 2024-01-24T17:46:57.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.158+0000 7f930f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308070540 0x7f930815ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.158+0000 7f930f200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f930815e910 0x7f930815eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.158+0000 7f930f200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93081551b0 con 0x7f93080b3e00 2024-01-24T17:46:57.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.158+0000 7f930f200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930815e410 con 0x7f930815e910 2024-01-24T17:46:57.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.158+0000 7f930f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930815b1b0 con 0x7f9308070540 2024-01-24T17:46:57.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.158+0000 7f930d400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f93080b3e00 0x7f93080c8a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41690/0 (socket says 172.21.15.39:41690) 2024-01-24T17:46:57.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.158+0000 7f930d400640 1 -- 172.21.15.39:0/2419677174 learned_addr learned my addr 172.21.15.39:0/2419677174 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:57.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.158+0000 7f930de00640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308070540 0x7f930815ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930e800640 1 --2- 172.21.15.39:0/2419677174 >> v2:172.21.15.100:3300/0 conn(0x7f930815e910 0x7f930815eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486694542 0 0) 0x7f93081551b0 con 0x7f93080b3e00 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e8003520 con 0x7f93080b3e00 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885300806 0 0) 0x7f92e8003520 con 0x7f93080b3e00 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308070540 msgr2=0x7f930815ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308070540 0x7f930815ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 >> v2:172.21.15.100:3300/0 conn(0x7f930815e910 msgr2=0x7f930815eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 --2- 172.21.15.39:0/2419677174 >> v2:172.21.15.100:3300/0 conn(0x7f930815e910 0x7f930815eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930815b480 con 0x7f93080b3e00 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f92f0001670 con 0x7f93080b3e00 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 >> v1:172.21.15.39:6789/0 conn(0x7f93080b3e00 legacy=0x7f93080c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308070540 0x7f92e8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92e8003ef0 0x7f92e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8004d00 0x7f92e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930815e410 con 0x7f92e8003ef0 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930815b1b0 con 0x7f92e8004d00 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e800b980 con 0x7f9308070540 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.159+0000 7f930d400640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92e8003ef0 0x7f92e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.160+0000 7f930e800640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8004d00 0x7f92e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.160+0000 7f930de00640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308070540 0x7f92e8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.160+0000 7f930d400640 1 -- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308070540 msgr2=0x7f92e8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.160+0000 7f930d400640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308070540 0x7f92e8003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.160+0000 7f930d400640 1 -- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8004d00 msgr2=0x7f92e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:57.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.160+0000 7f930d400640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8004d00 0x7f92e8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.160+0000 7f930d400640 1 -- 172.21.15.39:0/2419677174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8003520 con 0x7f92e8003ef0 2024-01-24T17:46:57.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.161+0000 7f930d400640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92e8003ef0 0x7f92e80047c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c240 tx=0x7f92e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d173247279bcb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.161+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f002cae0 con 0x7f92e8003ef0 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.161+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f002cc40 con 0x7f92e8003ef0 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.161+0000 7f930ca00640 1 -- 172.21.15.39:0/2419677174 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0005af0 con 0x7f92e8003ef0 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.162+0000 7f930f200640 1 -- 172.21.15.39:0/2419677174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92e8003ef0 msgr2=0x7f92e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.162+0000 7f930f200640 1 --2- 172.21.15.39:0/2419677174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92e8003ef0 0x7f92e80047c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c240 tx=0x7f92e80057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.162+0000 7f930e800640 1 -- 172.21.15.39:0/2419677174 reap_dead start 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.162+0000 7f930f200640 1 -- 172.21.15.39:0/2419677174 shutdown_connections 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.162+0000 7f930f200640 1 -- 172.21.15.39:0/2419677174 >> 172.21.15.39:0/2419677174 conn(0x7f930806a340 msgr2=0x7f93080cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:57.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.162+0000 7f930f200640 1 -- 172.21.15.39:0/2419677174 shutdown_connections 2024-01-24T17:46:57.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.162+0000 7f930f200640 1 -- 172.21.15.39:0/2419677174 wait complete. 2024-01-24T17:46:57.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.164+0000 7f930f200640 1 Processor -- start 2024-01-24T17:46:57.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.164+0000 7f930f200640 1 -- start start 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.165+0000 7f930f200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8003ef0 0x7f93081490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.165+0000 7f930f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93080c4830 0x7f93080c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.165+0000 7f930f200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308159770 0x7f9308159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.165+0000 7f930f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93080c8420 con 0x7f93080c4830 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.165+0000 7f930f200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930815a130 con 0x7f92e8003ef0 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.165+0000 7f930f200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93080c77a0 con 0x7f9308159770 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93080c4830 0x7f93080c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93080c4830 0x7f93080c6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52254/0 (socket says 172.21.15.39:52254) 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930d400640 1 -- 172.21.15.39:0/2484696210 learned_addr learned my addr 172.21.15.39:0/2484696210 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:57.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930de00640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8003ef0 0x7f93081490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930e800640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308159770 0x7f9308159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930d400640 1 -- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308159770 msgr2=0x7f9308159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930d400640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308159770 0x7f9308159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930d400640 1 -- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8003ef0 msgr2=0x7f93081490e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930d400640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8003ef0 0x7f93081490e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930d400640 1 -- 172.21.15.39:0/2484696210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93080c7a80 con 0x7f93080c4830 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.166+0000 7f930e800640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308159770 0x7f9308159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.167+0000 7f930d400640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93080c4830 0x7f93080c6ec0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c240 tx=0x7f92f0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.167+0000 7f9306000640 1 -- 172.21.15.39:0/2484696210 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0039030 con 0x7f93080c4830 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.167+0000 7f9306000640 1 -- 172.21.15.39:0/2484696210 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f0004bf0 con 0x7f93080c4830 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.167+0000 7f9306000640 1 -- 172.21.15.39:0/2484696210 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f003c040 con 0x7f93080c4830 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.167+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93080c7d60 con 0x7f93080c4830 2024-01-24T17:46:57.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.167+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93080c4c90 con 0x7f93080c4830 2024-01-24T17:46:57.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.168+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930804dff0 con 0x7f93080c4830 2024-01-24T17:46:57.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.169+0000 7f9306000640 1 -- 172.21.15.39:0/2484696210 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f92f0032070 con 0x7f93080c4830 2024-01-24T17:46:57.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.171+0000 7f9306000640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f92e006a430 0x7f92e006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.171+0000 7f9306000640 1 -- 172.21.15.39:0/2484696210 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92f00a9c00 con 0x7f93080c4830 2024-01-24T17:46:57.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.171+0000 7f930de00640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f92e006a430 0x7f92e006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.173+0000 7f930de00640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f92e006a430 0x7f92e006c8f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f92f80099d0 tx=0x7f92f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:57.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.175+0000 7f9306000640 1 -- 172.21.15.39:0/2484696210 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f92f00b0080 con 0x7f93080c4830 2024-01-24T17:46:57.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.373+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93080cbc10 con 0x7f93080c4830 2024-01-24T17:46:57.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.376+0000 7f9306000640 1 -- 172.21.15.39:0/2484696210 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16040 (secure 0 0 0) 0x7f92f0078d10 con 0x7f93080c4830 2024-01-24T17:46:57.377 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:46:57.377 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":76,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:32.019249+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f92e006a430 msgr2=0x7f92e006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f92e006a430 0x7f92e006c8f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f92f80099d0 tx=0x7f92f8008040 comp rx=0 tx=0).stop 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93080c4830 msgr2=0x7f93080c6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93080c4830 0x7f93080c6ec0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c240 tx=0x7f92f0001a00 comp rx=0 tx=0).stop 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 shutdown_connections 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9308159770 0x7f9308159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f92e006a430 0x7f92e006c8f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93080c4830 0x7f93080c6ec0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 --2- 172.21.15.39:0/2484696210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f92e8003ef0 0x7f93081490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 >> 172.21.15.39:0/2484696210 conn(0x7f930806a340 msgr2=0x7f9308072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:57.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 shutdown_connections 2024-01-24T17:46:57.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.379+0000 7f930f200640 1 -- 172.21.15.39:0/2484696210 wait complete. 2024-01-24T17:46:57.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:46:29.228451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:46:30.039417+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:46:57.401 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:46:57.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.631+0000 7f5b45000640 1 Processor -- start 2024-01-24T17:46:57.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.631+0000 7f5b45000640 1 -- start start 2024-01-24T17:46:57.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.632+0000 7f5b45000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5b40160dc0 0x7f5b40167a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.632+0000 7f5b45000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b400d13c0 con 0x7f5b400d0e90 2024-01-24T17:46:57.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.632+0000 7f5b45000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b400d16a0 con 0x7f5b4015ca30 2024-01-24T17:46:57.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.633+0000 7f5b45000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b400d1980 con 0x7f5b40160dc0 2024-01-24T17:46:57.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.633+0000 7f5b3d400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f5b400d0e90 0x7f5b401606b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41698/0 (socket says 172.21.15.39:41698) 2024-01-24T17:46:57.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.633+0000 7f5b3d400640 1 -- 172.21.15.39:0/2649541932 learned_addr learned my addr 172.21.15.39:0/2649541932 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:57.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.633+0000 7f5b3e800640 1 --2- 172.21.15.39:0/2649541932 >> v2:172.21.15.100:3300/0 conn(0x7f5b40160dc0 0x7f5b40167a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.634+0000 7f5b3ca00640 1 -- 172.21.15.39:0/2649541932 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011472450 0 0) 0x7f5b400d13c0 con 0x7f5b400d0e90 2024-01-24T17:46:57.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.634+0000 7f5b3ca00640 1 -- 172.21.15.39:0/2649541932 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b20003520 con 0x7f5b400d0e90 2024-01-24T17:46:57.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.634+0000 7f5b3e800640 1 -- 172.21.15.39:0/2649541932 >> v1:172.21.15.100:6789/0 conn(0x7f5b4015ca30 legacy=0x7f5b400d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.634+0000 7f5b3e800640 1 -- 172.21.15.39:0/2649541932 >> v1:172.21.15.39:6789/0 conn(0x7f5b400d0e90 legacy=0x7f5b401606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.634+0000 7f5b3e800640 1 -- 172.21.15.39:0/2649541932 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b400af510 con 0x7f5b40160dc0 2024-01-24T17:46:57.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.635+0000 7f5b3e800640 1 --2- 172.21.15.39:0/2649541932 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b40160dc0 0x7f5b40167a30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5b380097a0 tx=0x7f5b3802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f93f71b183a4b8f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:57.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.636+0000 7f5b3ca00640 1 -- 172.21.15.39:0/2649541932 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38030610 con 0x7f5b40160dc0 2024-01-24T17:46:57.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.636+0000 7f5b3ca00640 1 -- 172.21.15.39:0/2649541932 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b38030770 con 0x7f5b40160dc0 2024-01-24T17:46:57.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.636+0000 7f5b3ca00640 1 -- 172.21.15.39:0/2649541932 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38030b00 con 0x7f5b40160dc0 2024-01-24T17:46:57.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.636+0000 7f5b45000640 1 -- 172.21.15.39:0/2649541932 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b40160dc0 msgr2=0x7f5b40167a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.636+0000 7f5b45000640 1 --2- 172.21.15.39:0/2649541932 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b40160dc0 0x7f5b40167a30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5b380097a0 tx=0x7f5b3802b960 comp rx=0 tx=0).stop 2024-01-24T17:46:57.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.637+0000 7f5b45000640 1 -- 172.21.15.39:0/2649541932 shutdown_connections 2024-01-24T17:46:57.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.637+0000 7f5b45000640 1 --2- 172.21.15.39:0/2649541932 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b40160dc0 0x7f5b40167a30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.637+0000 7f5b45000640 1 -- 172.21.15.39:0/2649541932 >> 172.21.15.39:0/2649541932 conn(0x7f5b4009bc80 msgr2=0x7f5b400d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:57.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.637+0000 7f5b45000640 1 -- 172.21.15.39:0/2649541932 shutdown_connections 2024-01-24T17:46:57.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.637+0000 7f5b45000640 1 -- 172.21.15.39:0/2649541932 wait complete. 2024-01-24T17:46:57.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.638+0000 7f5b45000640 1 Processor -- start 2024-01-24T17:46:57.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.639+0000 7f5b45000640 1 -- start start 2024-01-24T17:46:57.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b45000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b400d0e90 0x7f5b401f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b45000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b4015ca30 0x7f5b401f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b45000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b40160dc0 0x7f5b400b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b45000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4015e2d0 con 0x7f5b40160dc0 2024-01-24T17:46:57.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b45000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b401f68d0 con 0x7f5b4015ca30 2024-01-24T17:46:57.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b45000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4015d650 con 0x7f5b400d0e90 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b40160dc0 0x7f5b400b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b40160dc0 0x7f5b400b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52258/0 (socket says 172.21.15.39:52258) 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3e800640 1 -- 172.21.15.39:0/2524774048 learned_addr learned my addr 172.21.15.39:0/2524774048 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3d400640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b4015ca30 0x7f5b401f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3e800640 1 -- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b400d0e90 msgr2=0x7f5b401f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3e800640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b400d0e90 0x7f5b401f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3e800640 1 -- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b4015ca30 msgr2=0x7f5b401f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3e800640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b4015ca30 0x7f5b401f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3e800640 1 -- 172.21.15.39:0/2524774048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b38009400 con 0x7f5b40160dc0 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.640+0000 7f5b3d400640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b4015ca30 0x7f5b401f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.641+0000 7f5b3e800640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b40160dc0 0x7f5b400b27e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5b380098c0 tx=0x7f5b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.641+0000 7f5b36000640 1 -- 172.21.15.39:0/2524774048 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3803e070 con 0x7f5b40160dc0 2024-01-24T17:46:57.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.641+0000 7f5b36000640 1 -- 172.21.15.39:0/2524774048 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b38030e10 con 0x7f5b40160dc0 2024-01-24T17:46:57.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.641+0000 7f5b36000640 1 -- 172.21.15.39:0/2524774048 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38039bd0 con 0x7f5b40160dc0 2024-01-24T17:46:57.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.641+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4015d930 con 0x7f5b40160dc0 2024-01-24T17:46:57.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.641+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4015def0 con 0x7f5b40160dc0 2024-01-24T17:46:57.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.642+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4004dff0 con 0x7f5b40160dc0 2024-01-24T17:46:57.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.643+0000 7f5b36000640 1 -- 172.21.15.39:0/2524774048 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5b38039d30 con 0x7f5b40160dc0 2024-01-24T17:46:57.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.644+0000 7f5b36000640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5b1806a430 0x7f5b1806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:57.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.644+0000 7f5b36000640 1 -- 172.21.15.39:0/2524774048 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b380aaca0 con 0x7f5b40160dc0 2024-01-24T17:46:57.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.645+0000 7f5b3de00640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5b1806a430 0x7f5b1806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:57.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.646+0000 7f5b3de00640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5b1806a430 0x7f5b1806c8f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5b300012e0 tx=0x7f5b30007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:57.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.648+0000 7f5b36000640 1 -- 172.21.15.39:0/2524774048 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5b38076280 con 0x7f5b40160dc0 2024-01-24T17:46:57.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.843+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b400b5110 con 0x7f5b40160dc0 2024-01-24T17:46:57.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.844+0000 7f5b36000640 1 -- 172.21.15.39:0/2524774048 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b380340a0 con 0x7f5b40160dc0 2024-01-24T17:46:57.845 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.846+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5b1806a430 msgr2=0x7f5b1806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.846+0000 7f5b45000640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5b1806a430 0x7f5b1806c8f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5b300012e0 tx=0x7f5b30007d30 comp rx=0 tx=0).stop 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.846+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b40160dc0 msgr2=0x7f5b400b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.846+0000 7f5b45000640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b40160dc0 0x7f5b400b27e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5b380098c0 tx=0x7f5b380046c0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.847+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 shutdown_connections 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.847+0000 7f5b45000640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5b1806a430 0x7f5b1806c8f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.847+0000 7f5b45000640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b40160dc0 0x7f5b400b27e0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.847+0000 7f5b45000640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5b4015ca30 0x7f5b401f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.847+0000 7f5b45000640 1 --2- 172.21.15.39:0/2524774048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5b400d0e90 0x7f5b401f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.847+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 >> 172.21.15.39:0/2524774048 conn(0x7f5b4009bc80 msgr2=0x7f5b4006bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:57.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.847+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 shutdown_connections 2024-01-24T17:46:57.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:57.847+0000 7f5b45000640 1 -- 172.21.15.39:0/2524774048 wait complete. 2024-01-24T17:46:57.867 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:46:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.089+0000 7efcc4e00640 1 Processor -- start 2024-01-24T17:46:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.089+0000 7efcc4e00640 1 -- start start 2024-01-24T17:46:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.090+0000 7efcc4e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc015ac70 0x7efcc00b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.090+0000 7efcc4e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc0155170 con 0x7efcc00ce8c0 2024-01-24T17:46:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.090+0000 7efcc4e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc015b1e0 con 0x7efcc0070460 2024-01-24T17:46:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.090+0000 7efcc4e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00b12b0 con 0x7efcc015ac70 2024-01-24T17:46:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.091+0000 7efcbf400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efcc00ce8c0 0x7efcc006fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41702/0 (socket says 172.21.15.39:41702) 2024-01-24T17:46:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.091+0000 7efcbf400640 1 -- 172.21.15.39:0/1699368573 learned_addr learned my addr 172.21.15.39:0/1699368573 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.091+0000 7efcbfe00640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc015ac70 0x7efcc00b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.091+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275058833 0 0) 0x7efcc0155170 con 0x7efcc00ce8c0 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.091+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0003520 con 0x7efcc00ce8c0 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228371026 0 0) 0x7efca0003520 con 0x7efcc00ce8c0 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc015ac70 msgr2=0x7efcc00b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc015ac70 0x7efcc00b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 >> v1:172.21.15.100:6789/0 conn(0x7efcc0070460 legacy=0x7efcc00b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc015b4c0 con 0x7efcc00ce8c0 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7efcb0001670 con 0x7efcc00ce8c0 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 >> v1:172.21.15.39:6789/0 conn(0x7efcc00ce8c0 legacy=0x7efcc006fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efca00039a0 0x7efca0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca0004350 0x7efca0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc015ac70 0x7efca0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca000b980 con 0x7efca00039a0 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca000bac0 con 0x7efca0004350 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca000bc80 con 0x7efcc015ac70 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.092+0000 7efcbf400640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efca00039a0 0x7efca0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.093+0000 7efcbea00640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca0004350 0x7efca0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.093+0000 7efcbf400640 1 -- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc015ac70 msgr2=0x7efca0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.093+0000 7efcbf400640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc015ac70 0x7efca0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.093+0000 7efcbf400640 1 -- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca0004350 msgr2=0x7efca0004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.093+0000 7efcbf400640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca0004350 0x7efca0004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.093+0000 7efcbf400640 1 -- 172.21.15.39:0/1699368573 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0003520 con 0x7efca00039a0 2024-01-24T17:46:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.093+0000 7efcbea00640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca0004350 0x7efca0004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:58.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.094+0000 7efcbf400640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efca00039a0 0x7efca0003d80 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c210 tx=0x7efca00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5fe56eb0208f48 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:58.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.094+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb002cae0 con 0x7efca00039a0 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.094+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb002cc60 con 0x7efca00039a0 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.094+0000 7efcbe000640 1 -- 172.21.15.39:0/1699368573 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb00058d0 con 0x7efca00039a0 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.095+0000 7efcc4e00640 1 -- 172.21.15.39:0/1699368573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efca00039a0 msgr2=0x7efca0003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.095+0000 7efcc4e00640 1 --2- 172.21.15.39:0/1699368573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efca00039a0 0x7efca0003d80 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c210 tx=0x7efca00057f0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.095+0000 7efcbfe00640 1 -- 172.21.15.39:0/1699368573 reap_dead start 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.095+0000 7efcc4e00640 1 -- 172.21.15.39:0/1699368573 shutdown_connections 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.095+0000 7efcc4e00640 1 -- 172.21.15.39:0/1699368573 >> 172.21.15.39:0/1699368573 conn(0x7efcc006a200 msgr2=0x7efcc00affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.095+0000 7efcc4e00640 1 -- 172.21.15.39:0/1699368573 shutdown_connections 2024-01-24T17:46:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.095+0000 7efcc4e00640 1 -- 172.21.15.39:0/1699368573 wait complete. 2024-01-24T17:46:58.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.097+0000 7efcc4e00640 1 Processor -- start 2024-01-24T17:46:58.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.098+0000 7efcc4e00640 1 -- start start 2024-01-24T17:46:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcc4e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca00039a0 0x7efcc00c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcc4e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc0153920 0x7efcc015a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcc4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc015a770 0x7efcc0158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcc4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc01f4490 con 0x7efcc015a770 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcc4e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00c6ae0 con 0x7efca00039a0 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcc4e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc00c5ed0 con 0x7efcc0153920 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcbfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc015a770 0x7efcc0158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcbfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc015a770 0x7efcc0158220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52276/0 (socket says 172.21.15.39:52276) 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.099+0000 7efcbfe00640 1 -- 172.21.15.39:0/3904383622 learned_addr learned my addr 172.21.15.39:0/3904383622 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbf400640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca00039a0 0x7efcc00c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbea00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc0153920 0x7efcc015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbfe00640 1 -- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc0153920 msgr2=0x7efcc015a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbfe00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc0153920 0x7efcc015a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbfe00640 1 -- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca00039a0 msgr2=0x7efcc00c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbfe00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca00039a0 0x7efcc00c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbfe00640 1 -- 172.21.15.39:0/3904383622 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc00c61b0 con 0x7efcc015a770 2024-01-24T17:46:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbea00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc0153920 0x7efcc015a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcbfe00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc015a770 0x7efcc0158220 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7efcc006de40 tx=0x7efcb800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcb7400640 1 -- 172.21.15.39:0/3904383622 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8010040 con 0x7efcc015a770 2024-01-24T17:46:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcb7400640 1 -- 172.21.15.39:0/3904383622 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb800bb10 con 0x7efcc015a770 2024-01-24T17:46:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcb7400640 1 -- 172.21.15.39:0/3904383622 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb800bea0 con 0x7efcc015a770 2024-01-24T17:46:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.100+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc00c6490 con 0x7efcc015a770 2024-01-24T17:46:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.101+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc00c74b0 con 0x7efcc015a770 2024-01-24T17:46:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.102+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc00738c0 con 0x7efcc015a770 2024-01-24T17:46:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.103+0000 7efcb7400640 1 -- 172.21.15.39:0/3904383622 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efcb801e410 con 0x7efcc015a770 2024-01-24T17:46:58.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.104+0000 7efcb7400640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9806a430 0x7efc9806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.104+0000 7efcb7400640 1 -- 172.21.15.39:0/3904383622 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcb8088b90 con 0x7efcc015a770 2024-01-24T17:46:58.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.104+0000 7efcbf400640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9806a430 0x7efc9806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.105+0000 7efcbf400640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9806a430 0x7efc9806c8f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7efcb002cfd0 tx=0x7efcb0039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:58.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.108+0000 7efcb7400640 1 -- 172.21.15.39:0/3904383622 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efcb80540e0 con 0x7efcc015a770 2024-01-24T17:46:58.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.302+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efcc00c6c70 con 0x7efcc015a770 2024-01-24T17:46:58.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.303+0000 7efcb7400640 1 -- 172.21.15.39:0/3904383622 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7efcb8057700 con 0x7efcc015a770 2024-01-24T17:46:58.304 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:46:58.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.305+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9806a430 msgr2=0x7efc9806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.305+0000 7efcc4e00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9806a430 0x7efc9806c8f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7efcb002cfd0 tx=0x7efcb0039040 comp rx=0 tx=0).stop 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.305+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc015a770 msgr2=0x7efcc0158220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.305+0000 7efcc4e00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc015a770 0x7efcc0158220 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7efcc006de40 tx=0x7efcb800b480 comp rx=0 tx=0).stop 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.306+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 shutdown_connections 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.306+0000 7efcc4e00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efc9806a430 0x7efc9806c8f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.306+0000 7efcc4e00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcc015a770 0x7efcc0158220 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.306+0000 7efcc4e00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efcc0153920 0x7efcc015a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.306+0000 7efcc4e00640 1 --2- 172.21.15.39:0/3904383622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efca00039a0 0x7efcc00c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.306+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 >> 172.21.15.39:0/3904383622 conn(0x7efcc006a200 msgr2=0x7efcc015c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.306+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 shutdown_connections 2024-01-24T17:46:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.306+0000 7efcc4e00640 1 -- 172.21.15.39:0/3904383622 wait complete. 2024-01-24T17:46:58.328 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:46:58.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.549+0000 7fde75000640 1 Processor -- start 2024-01-24T17:46:58.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.550+0000 7fde75000640 1 -- start start 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.551+0000 7fde75000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fde70070460 0x7fde700b1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.551+0000 7fde75000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde7015ac70 0x7fde700b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.551+0000 7fde75000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70155170 con 0x7fde700ce8c0 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.551+0000 7fde75000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde700b12b0 con 0x7fde70070460 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.551+0000 7fde75000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7015b1e0 con 0x7fde7015ac70 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.551+0000 7fde66800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fde70070460 0x7fde700b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.551+0000 7fde66800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fde70070460 0x7fde700b1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52290/0 (socket says 172.21.15.39:52290) 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.552+0000 7fde66800640 1 -- 172.21.15.39:0/1383377863 learned_addr learned my addr 172.21.15.39:0/1383377863 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.551+0000 7fde6f400640 1 --1- 172.21.15.39:0/1383377863 >> v1:172.21.15.39:6789/0 conn(0x7fde700ce8c0 0x7fde7006fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:41706/0 (socket says 172.21.15.39:41706) 2024-01-24T17:46:58.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.552+0000 7fde6fe00640 1 --2- 172.21.15.39:0/1383377863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde7015ac70 0x7fde700b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.552+0000 7fde66800640 1 -- 172.21.15.39:0/1383377863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde7015ac70 msgr2=0x7fde700b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:58.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.552+0000 7fde66800640 1 --2- 172.21.15.39:0/1383377863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde7015ac70 0x7fde700b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.552+0000 7fde66800640 1 -- 172.21.15.39:0/1383377863 >> v1:172.21.15.39:6789/0 conn(0x7fde700ce8c0 legacy=0x7fde7006fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.552+0000 7fde66800640 1 -- 172.21.15.39:0/1383377863 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7015b4c0 con 0x7fde70070460 2024-01-24T17:46:58.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.552+0000 7fde6fe00640 1 --2- 172.21.15.39:0/1383377863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde7015ac70 0x7fde700b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:58.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.553+0000 7fde66800640 1 --2- 172.21.15.39:0/1383377863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde70070460 0x7fde700b1700 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fde58009870 tx=0x7fde5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd84301aecd68864 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:58.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.553+0000 7fde6ea00640 1 -- 172.21.15.39:0/1383377863 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58030840 con 0x7fde70070460 2024-01-24T17:46:58.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde6ea00640 1 -- 172.21.15.39:0/1383377863 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde580309a0 con 0x7fde70070460 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde6ea00640 1 -- 172.21.15.39:0/1383377863 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58030d30 con 0x7fde70070460 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde75000640 1 -- 172.21.15.39:0/1383377863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde70070460 msgr2=0x7fde700b1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde75000640 1 --2- 172.21.15.39:0/1383377863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde70070460 0x7fde700b1700 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fde58009870 tx=0x7fde5802b980 comp rx=0 tx=0).stop 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde75000640 1 -- 172.21.15.39:0/1383377863 shutdown_connections 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde75000640 1 --2- 172.21.15.39:0/1383377863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde70070460 0x7fde700b1700 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde75000640 1 --2- 172.21.15.39:0/1383377863 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde7015ac70 0x7fde700b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde75000640 1 -- 172.21.15.39:0/1383377863 >> 172.21.15.39:0/1383377863 conn(0x7fde7006a200 msgr2=0x7fde700affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.554+0000 7fde75000640 1 -- 172.21.15.39:0/1383377863 shutdown_connections 2024-01-24T17:46:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.555+0000 7fde75000640 1 -- 172.21.15.39:0/1383377863 wait complete. 2024-01-24T17:46:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.556+0000 7fde75000640 1 Processor -- start 2024-01-24T17:46:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.557+0000 7fde75000640 1 -- start start 2024-01-24T17:46:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde75000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde70070460 0x7fde700c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde75000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde700ce8c0 0x7fde700c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde75000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde7015ac70 0x7fde700c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde75000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7015c5b0 con 0x7fde7015ac70 2024-01-24T17:46:58.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde75000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7015c800 con 0x7fde70070460 2024-01-24T17:46:58.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde75000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7015cae0 con 0x7fde700ce8c0 2024-01-24T17:46:58.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde7015ac70 0x7fde700c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde7015ac70 0x7fde700c5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52294/0 (socket says 172.21.15.39:52294) 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde6fe00640 1 -- 172.21.15.39:0/716268079 learned_addr learned my addr 172.21.15.39:0/716268079 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde66800640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde700ce8c0 0x7fde700c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde6f400640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde70070460 0x7fde700c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.558+0000 7fde6fe00640 1 -- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde700ce8c0 msgr2=0x7fde700c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde6fe00640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde700ce8c0 0x7fde700c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde6fe00640 1 -- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde70070460 msgr2=0x7fde700c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde6fe00640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde70070460 0x7fde700c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde6fe00640 1 -- 172.21.15.39:0/716268079 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580094d0 con 0x7fde7015ac70 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde6f400640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde70070460 0x7fde700c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde66800640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde700ce8c0 0x7fde700c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde6fe00640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde7015ac70 0x7fde700c5e20 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fde6800cd50 tx=0x7fde6800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:58.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde67e00640 1 -- 172.21.15.39:0/716268079 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6800f680 con 0x7fde7015ac70 2024-01-24T17:46:58.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde67e00640 1 -- 172.21.15.39:0/716268079 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6800f800 con 0x7fde7015ac70 2024-01-24T17:46:58.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7015cdc0 con 0x7fde7015ac70 2024-01-24T17:46:58.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde67e00640 1 -- 172.21.15.39:0/716268079 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6800fbb0 con 0x7fde7015ac70 2024-01-24T17:46:58.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.559+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde700c5710 con 0x7fde7015ac70 2024-01-24T17:46:58.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.560+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde700738c0 con 0x7fde7015ac70 2024-01-24T17:46:58.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.561+0000 7fde67e00640 1 -- 172.21.15.39:0/716268079 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fde6800b040 con 0x7fde7015ac70 2024-01-24T17:46:58.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.562+0000 7fde67e00640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fde4806a430 0x7fde4806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:58.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.562+0000 7fde67e00640 1 -- 172.21.15.39:0/716268079 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde68089390 con 0x7fde7015ac70 2024-01-24T17:46:58.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.563+0000 7fde6f400640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fde4806a430 0x7fde4806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:58.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.564+0000 7fde6f400640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fde4806a430 0x7fde4806c8f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fde60000f60 tx=0x7fde60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:58.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.568+0000 7fde67e00640 1 -- 172.21.15.39:0/716268079 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fde680548e0 con 0x7fde7015ac70 2024-01-24T17:46:58.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.761+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde7015d2e0 con 0x7fde7015ac70 2024-01-24T17:46:58.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.762+0000 7fde67e00640 1 -- 172.21.15.39:0/716268079 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde7015d2e0 con 0x7fde7015ac70 2024-01-24T17:46:58.763 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fde4806a430 msgr2=0x7fde4806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fde4806a430 0x7fde4806c8f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fde60000f60 tx=0x7fde60009040 comp rx=0 tx=0).stop 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde7015ac70 msgr2=0x7fde700c5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde7015ac70 0x7fde700c5e20 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fde6800cd50 tx=0x7fde6800a650 comp rx=0 tx=0).stop 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 shutdown_connections 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fde4806a430 0x7fde4806c8f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde7015ac70 0x7fde700c5e20 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fde700ce8c0 0x7fde700c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 --2- 172.21.15.39:0/716268079 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fde70070460 0x7fde700c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 >> 172.21.15.39:0/716268079 conn(0x7fde7006a200 msgr2=0x7fde700b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 shutdown_connections 2024-01-24T17:46:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:58.766+0000 7fde75000640 1 -- 172.21.15.39:0/716268079 wait complete. 2024-01-24T17:46:58.785 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:46:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.004+0000 7f6583a00640 1 Processor -- start 2024-01-24T17:46:59.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.004+0000 7f6583a00640 1 -- start start 2024-01-24T17:46:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.006+0000 7f6583a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f657c159d00 0x7f657c0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.006+0000 7f6583a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c0b3f20 0x7f657c0714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.006+0000 7f6583a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c0ccd10 con 0x7f657c071a30 2024-01-24T17:46:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.006+0000 7f6583a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c0ce230 con 0x7f657c159d00 2024-01-24T17:46:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.006+0000 7f6583a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0ccff0 con 0x7f657c0b3f20 2024-01-24T17:46:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6581000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f657c159d00 0x7f657c0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6573c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c0b3f20 0x7f657c0714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6581000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f657c159d00 0x7f657c0cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:33128/0 (socket says 172.21.15.39:33128) 2024-01-24T17:46:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6573c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c0b3f20 0x7f657c0714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:38302/0 (socket says 172.21.15.39:38302) 2024-01-24T17:46:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6581000640 1 -- 172.21.15.39:0/146571142 learned_addr learned my addr 172.21.15.39:0/146571142 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6573c00640 1 -- 172.21.15.39:0/146571142 >> v2:172.21.15.100:3300/0 conn(0x7f657c159d00 msgr2=0x7f657c0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:46:59.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6573c00640 1 --2- 172.21.15.39:0/146571142 >> v2:172.21.15.100:3300/0 conn(0x7f657c159d00 0x7f657c0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6573c00640 1 -- 172.21.15.39:0/146571142 >> v1:172.21.15.100:6789/0 conn(0x7f657c071a30 legacy=0x7f657c0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:59.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6573c00640 1 -- 172.21.15.39:0/146571142 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c0ad720 con 0x7f657c0b3f20 2024-01-24T17:46:59.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.007+0000 7f6581000640 1 --2- 172.21.15.39:0/146571142 >> v2:172.21.15.100:3300/0 conn(0x7f657c159d00 0x7f657c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:46:59.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.009+0000 7f6573c00640 1 --2- 172.21.15.39:0/146571142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c0b3f20 0x7f657c0714f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6564009d90 tx=0x7f656402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79947d89212b6b34 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:59.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.009+0000 7f657be00640 1 -- 172.21.15.39:0/146571142 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656402cc50 con 0x7f657c0b3f20 2024-01-24T17:46:59.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.009+0000 7f657be00640 1 -- 172.21.15.39:0/146571142 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656402cdb0 con 0x7f657c0b3f20 2024-01-24T17:46:59.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.009+0000 7f657be00640 1 -- 172.21.15.39:0/146571142 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564034660 con 0x7f657c0b3f20 2024-01-24T17:46:59.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.010+0000 7f6583a00640 1 -- 172.21.15.39:0/146571142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c0b3f20 msgr2=0x7f657c0714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:59.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.010+0000 7f6583a00640 1 --2- 172.21.15.39:0/146571142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c0b3f20 0x7f657c0714f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6564009d90 tx=0x7f656402bea0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.010+0000 7f6583a00640 1 -- 172.21.15.39:0/146571142 shutdown_connections 2024-01-24T17:46:59.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.010+0000 7f6583a00640 1 --2- 172.21.15.39:0/146571142 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c0b3f20 0x7f657c0714f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.010+0000 7f6583a00640 1 --2- 172.21.15.39:0/146571142 >> v2:172.21.15.100:3300/0 conn(0x7f657c159d00 0x7f657c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.010+0000 7f6583a00640 1 -- 172.21.15.39:0/146571142 >> 172.21.15.39:0/146571142 conn(0x7f657c06bf20 msgr2=0x7f657c06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:59.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.010+0000 7f6583a00640 1 -- 172.21.15.39:0/146571142 shutdown_connections 2024-01-24T17:46:59.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.010+0000 7f6583a00640 1 -- 172.21.15.39:0/146571142 wait complete. 2024-01-24T17:46:59.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.012+0000 7f6583a00640 1 Processor -- start 2024-01-24T17:46:59.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.012+0000 7f6583a00640 1 -- start start 2024-01-24T17:46:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6583a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657c071a30 0x7f657c153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6583a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f657c0b3f20 0x7f657c1f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6583a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c159d00 0x7f657c0c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6583a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c154050 con 0x7f657c071a30 2024-01-24T17:46:59.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6583a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c15c740 con 0x7f657c0b3f20 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6583a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c15ca20 con 0x7f657c159d00 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6581000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657c071a30 0x7f657c153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6581000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657c071a30 0x7f657c153b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52302/0 (socket says 172.21.15.39:52302) 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.013+0000 7f6581000640 1 -- 172.21.15.39:0/2571348306 learned_addr learned my addr 172.21.15.39:0/2571348306 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6573c00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f657c0b3f20 0x7f657c1f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6581a00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c159d00 0x7f657c0c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6581000640 1 -- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c159d00 msgr2=0x7f657c0c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6581000640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c159d00 0x7f657c0c47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6581000640 1 -- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f657c0b3f20 msgr2=0x7f657c1f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6581000640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f657c0b3f20 0x7f657c1f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6581000640 1 -- 172.21.15.39:0/2571348306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c00b770 con 0x7f657c071a30 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6581a00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c159d00 0x7f657c0c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6581000640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657c071a30 0x7f657c153b10 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f656c00ed40 tx=0x7f656c00d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:59.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.014+0000 7f6579600640 1 -- 172.21.15.39:0/2571348306 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c019680 con 0x7f657c071a30 2024-01-24T17:46:59.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.015+0000 7f6579600640 1 -- 172.21.15.39:0/2571348306 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c0197e0 con 0x7f657c071a30 2024-01-24T17:46:59.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.015+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6564009a70 con 0x7f657c071a30 2024-01-24T17:46:59.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.015+0000 7f6579600640 1 -- 172.21.15.39:0/2571348306 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c019b50 con 0x7f657c071a30 2024-01-24T17:46:59.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.015+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c0c6880 con 0x7f657c071a30 2024-01-24T17:46:59.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.016+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657c04dff0 con 0x7f657c071a30 2024-01-24T17:46:59.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.017+0000 7f6579600640 1 -- 172.21.15.39:0/2571348306 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f656c010440 con 0x7f657c071a30 2024-01-24T17:46:59.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.018+0000 7f6579600640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f655406a460 0x7f655406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:46:59.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.019+0000 7f6579600640 1 -- 172.21.15.39:0/2571348306 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f656c014070 con 0x7f657c071a30 2024-01-24T17:46:59.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.019+0000 7f6573c00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f655406a460 0x7f655406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:46:59.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.020+0000 7f6573c00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f655406a460 0x7f655406c920 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6564036000 tx=0x7f6564030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:46:59.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.023+0000 7f6579600640 1 -- 172.21.15.39:0/2571348306 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f656c054e30 con 0x7f657c071a30 2024-01-24T17:46:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.228+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f657c0cb640 con 0x7f657c071a30 2024-01-24T17:46:59.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.229+0000 7f6579600640 1 -- 172.21.15.39:0/2571348306 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v76) v1 ==== 80+0+0 (secure 0 0 0) 0x7f656c058450 con 0x7f657c071a30 2024-01-24T17:46:59.230 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:46:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f655406a460 msgr2=0x7f655406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f655406a460 0x7f655406c920 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6564036000 tx=0x7f6564030000 comp rx=0 tx=0).stop 2024-01-24T17:46:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657c071a30 msgr2=0x7f657c153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:46:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657c071a30 0x7f657c153b10 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f656c00ed40 tx=0x7f656c00d720 comp rx=0 tx=0).stop 2024-01-24T17:46:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 shutdown_connections 2024-01-24T17:46:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f657c159d00 0x7f657c0c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f657c0b3f20 0x7f657c1f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f655406a460 0x7f655406c920 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 --2- 172.21.15.39:0/2571348306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657c071a30 0x7f657c153b10 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:46:59.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.232+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 >> 172.21.15.39:0/2571348306 conn(0x7f657c06bf20 msgr2=0x7f657c158bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:46:59.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.233+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 shutdown_connections 2024-01-24T17:46:59.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:46:59.233+0000 7f6583a00640 1 -- 172.21.15.39:0/2571348306 wait complete. 2024-01-24T17:46:59.252 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:46:59.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:46:59.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:46:59.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[47154]: fuse finished with error 0 and tester_r 0 2024-01-24T17:46:59.356 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:47:05.458 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:47:05.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:05.458 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:47:05.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:47:05.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.746+0000 7fa73be00640 1 Processor -- start 2024-01-24T17:47:05.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.747+0000 7fa73be00640 1 -- start start 2024-01-24T17:47:05.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa73be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa73c0ca320 0x7fa73c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:05.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa73be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa73c0ca700 0x7fa73c15a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:05.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa73be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa73c1540f0 con 0x7fa73c15a910 2024-01-24T17:47:05.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa73be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa73c159fd0 con 0x7fa73c0ca700 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa73be00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa73c15da00 con 0x7fa73c0ca320 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa731e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa73c0ca700 0x7fa73c15a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa731e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa73c0ca700 0x7fa73c15a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44682/0 (socket says 172.21.15.39:44682) 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa731e00640 1 -- 172.21.15.39:0/2525077888 learned_addr learned my addr 172.21.15.39:0/2525077888 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.748+0000 7fa73b400640 1 --1- 172.21.15.39:0/2525077888 >> v1:172.21.15.39:6789/0 conn(0x7fa73c15a910 0x7fa73c06bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49750/0 (socket says 172.21.15.39:49750) 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.749+0000 7fa73aa00640 1 --2- 172.21.15.39:0/2525077888 >> v2:172.21.15.100:3300/0 conn(0x7fa73c0ca320 0x7fa73c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.749+0000 7fa731e00640 1 -- 172.21.15.39:0/2525077888 >> v2:172.21.15.100:3300/0 conn(0x7fa73c0ca320 msgr2=0x7fa73c15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.749+0000 7fa731e00640 1 --2- 172.21.15.39:0/2525077888 >> v2:172.21.15.100:3300/0 conn(0x7fa73c0ca320 0x7fa73c15d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.749+0000 7fa731e00640 1 -- 172.21.15.39:0/2525077888 >> v1:172.21.15.39:6789/0 conn(0x7fa73c15a910 legacy=0x7fa73c06bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:05.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.749+0000 7fa731e00640 1 -- 172.21.15.39:0/2525077888 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c15dce0 con 0x7fa73c0ca700 2024-01-24T17:47:05.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.750+0000 7fa731e00640 1 --2- 172.21.15.39:0/2525077888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c15a3d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fa73c072220 tx=0x7fa72402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82493c23232abec server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.750+0000 7fa73a000640 1 -- 172.21.15.39:0/2525077888 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72402ee00 con 0x7fa73c0ca700 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.750+0000 7fa73a000640 1 -- 172.21.15.39:0/2525077888 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa724036440 con 0x7fa73c0ca700 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.750+0000 7fa73a000640 1 -- 172.21.15.39:0/2525077888 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7240367d0 con 0x7fa73c0ca700 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.751+0000 7fa73be00640 1 -- 172.21.15.39:0/2525077888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 msgr2=0x7fa73c15a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.751+0000 7fa73be00640 1 --2- 172.21.15.39:0/2525077888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c15a3d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fa73c072220 tx=0x7fa72402e050 comp rx=0 tx=0).stop 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.751+0000 7fa73be00640 1 -- 172.21.15.39:0/2525077888 shutdown_connections 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.751+0000 7fa73be00640 1 --2- 172.21.15.39:0/2525077888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c15a3d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.751+0000 7fa73be00640 1 --2- 172.21.15.39:0/2525077888 >> v2:172.21.15.100:3300/0 conn(0x7fa73c0ca320 0x7fa73c15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.751+0000 7fa73be00640 1 -- 172.21.15.39:0/2525077888 >> 172.21.15.39:0/2525077888 conn(0x7fa73c06a340 msgr2=0x7fa73c0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.751+0000 7fa73be00640 1 -- 172.21.15.39:0/2525077888 shutdown_connections 2024-01-24T17:47:05.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.751+0000 7fa73be00640 1 -- 172.21.15.39:0/2525077888 wait complete. 2024-01-24T17:47:05.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.753+0000 7fa73be00640 1 Processor -- start 2024-01-24T17:47:05.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.753+0000 7fa73be00640 1 -- start start 2024-01-24T17:47:05.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa73be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa73c0ca320 0x7fa73c14c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa73be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c0c32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa73be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa73c15a910 0x7fa73c0c3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa73be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c14f1e0 con 0x7fa73c0ca700 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa73be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c14cf10 con 0x7fa73c0ca320 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa73be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c155710 con 0x7fa73c15a910 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa731e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c0c32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa731e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c0c32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44688/0 (socket says 172.21.15.39:44688) 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.754+0000 7fa731e00640 1 -- 172.21.15.39:0/3475881354 learned_addr learned my addr 172.21.15.39:0/3475881354 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa73aa00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa73c0ca320 0x7fa73c14c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa731e00640 1 -- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa73c15a910 msgr2=0x7fa73c0c3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa731e00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa73c15a910 0x7fa73c0c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa731e00640 1 -- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa73c0ca320 msgr2=0x7fa73c14c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa731e00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa73c0ca320 0x7fa73c14c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:05.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa731e00640 1 -- 172.21.15.39:0/3475881354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72400b770 con 0x7fa73c0ca700 2024-01-24T17:47:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa73aa00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa73c0ca320 0x7fa73c14c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa731e00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c0c32d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fa724000c00 tx=0x7fa724004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa733e00640 1 -- 172.21.15.39:0/3475881354 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7240407f0 con 0x7fa73c0ca700 2024-01-24T17:47:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa733e00640 1 -- 172.21.15.39:0/3475881354 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa724040950 con 0x7fa73c0ca700 2024-01-24T17:47:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa733e00640 1 -- 172.21.15.39:0/3475881354 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724040d00 con 0x7fa73c0ca700 2024-01-24T17:47:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73c14e840 con 0x7fa73c0ca700 2024-01-24T17:47:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.755+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73c14ee00 con 0x7fa73c0ca700 2024-01-24T17:47:05.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.757+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73c14f320 con 0x7fa73c0ca700 2024-01-24T17:47:05.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.757+0000 7fa733e00640 1 -- 172.21.15.39:0/3475881354 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa724041480 con 0x7fa73c0ca700 2024-01-24T17:47:05.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.759+0000 7fa733e00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa71406a430 0x7fa71406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:05.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.759+0000 7fa733e00640 1 -- 172.21.15.39:0/3475881354 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7240ab900 con 0x7fa73c0ca700 2024-01-24T17:47:05.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.759+0000 7fa73aa00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa71406a430 0x7fa71406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:05.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.761+0000 7fa73aa00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa71406a430 0x7fa71406c8f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa72c006fd0 tx=0x7fa72c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:05.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.764+0000 7fa733e00640 1 -- 172.21.15.39:0/3475881354 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa73c14f320 con 0x7fa73c0ca700 2024-01-24T17:47:06.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:05.999+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa73c14f320 con 0x7fa73c0ca700 2024-01-24T17:47:06.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.000+0000 7fa733e00640 1 -- 172.21.15.39:0/3475881354 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fa73c14f320 con 0x7fa73c0ca700 2024-01-24T17:47:06.002 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:06.002 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1886779141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1886779141},{"type":"v1","addr":"172.21.15.39:6833","nonce":1886779141}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5854,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2624477336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":2624477336},{"type":"v1","addr":"172.21.15.100:6837","nonce":2624477336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2465319794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2465319794},{"type":"v1","addr":"172.21.15.39:6835","nonce":2465319794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:46:31.015014+0000","modified":"2024-01-24T17:46:34.537399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6836/2467066037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2467066037},{"type":"v1","addr":"172.21.15.100:6836","nonce":2467066037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-24T17:47:06.002 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 42 2024-01-24T17:47:06.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa71406a430 msgr2=0x7fa71406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa71406a430 0x7fa71406c8f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa72c006fd0 tx=0x7fa72c008040 comp rx=0 tx=0).stop 2024-01-24T17:47:06.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 msgr2=0x7fa73c0c32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c0c32d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fa724000c00 tx=0x7fa724004730 comp rx=0 tx=0).stop 2024-01-24T17:47:06.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 shutdown_connections 2024-01-24T17:47:06.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa73c15a910 0x7fa73c0c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa71406a430 0x7fa71406c8f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa73c0ca700 0x7fa73c0c32d0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 --2- 172.21.15.39:0/3475881354 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa73c0ca320 0x7fa73c14c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 >> 172.21.15.39:0/3475881354 conn(0x7fa73c06a340 msgr2=0x7fa73c0c98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.003+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 shutdown_connections 2024-01-24T17:47:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.004+0000 7fa73be00640 1 -- 172.21.15.39:0/3475881354 wait complete. 2024-01-24T17:47:06.026 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-01-24T17:47:06.261 INFO:teuthology.orchestra.run.smithi100.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":74,"msgr_send_messages":69,"msgr_recv_bytes":115515,"msgr_send_bytes":20080,"msgr_created_connections":6,"msgr_active_connections":2,"msgr_running_total_time":0.017855664,"msgr_running_send_time":0.003369149,"msgr_running_recv_time":0.014483548,"msgr_running_fast_dispatch_time":0.000778652,"msgr_send_messages_queue_lat":{"avgcount":69,"sum":0.233474981,"avgtime":0.003383695},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":112224,"msgr_send_encrypted_bytes":14384},"AsyncMessenger::Worker-1":{"msgr_recv_messages":31,"msgr_send_messages":33,"msgr_recv_bytes":9479,"msgr_send_bytes":124961,"msgr_created_connections":8,"msgr_active_connections":3,"msgr_running_total_time":0.016840369,"msgr_running_send_time":0.004007241,"msgr_running_recv_time":0.011415007,"msgr_running_fast_dispatch_time":0.000891314,"msgr_send_messages_queue_lat":{"avgcount":32,"sum":0.114274840,"avgtime":0.003571088},"msgr_handle_ack_lat":{"avgcount":22,"sum":0.000006115,"avgtime":0.000000277},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":100288},"AsyncMessenger::Worker-2":{"msgr_recv_messages":44,"msgr_send_messages":29,"msgr_recv_bytes":13957,"msgr_send_bytes":19492,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":0.012737168,"msgr_running_send_time":0.003293892,"msgr_running_recv_time":0.006660533,"msgr_running_fast_dispatch_time":0.002293695,"msgr_send_messages_queue_lat":{"avgcount":27,"sum":0.181927623,"avgtime":0.006738060},"msgr_handle_ack_lat":{"avgcount":26,"sum":0.000006328,"avgtime":0.000000243},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":53,"sum":0.004248498,"avgtime":0.000080160}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2,"sum":0.000024425,"avgtime":0.000012212}},"mds":{"request":3,"reply":3,"reply_latency":{"avgcount":3,"sum":0.001398824,"avgtime":0.000466274},"slow_reply":0,"forward":0,"dir_fetch_complete":0,"dir_fetch_keys":0,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":0,"inodes_bottom":10,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":0,"caps":0,"subtrees":2,"traverse":5,"traverse_hit":5,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":0,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":0,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":6,"evex":0,"evtrm":0,"ev":6,"evexg":0,"evexd":0,"segadd":1,"segex":0,"segtrm":0,"seg":1,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4197177,"rdpos":0,"jlat":{"avgcount":5,"sum":0.021097703,"avgtime":0.004219540},"replayed":0},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":0,"cap+":2,"cap-":2,"rss":39296,"heap":207196},"mds_server":{"dispatch_client_request":3,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":8,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":2,"sum":0.000755106,"avgtime":0.000377553},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":1,"sum":0.000643718,"avgtime":0.000643718},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":0,"session_add":2,"session_remove":2,"sessions_open":0,"sessions_stale":0,"total_load":2,"average_load":1,"avg_session_uptime":12,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":221411,"buffer_anon_items":62,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5312,"osdmap_items":75,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51440,"mds_co_items":211,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":44,"op_send_bytes":14083,"op_resend":37,"op_reply":44,"op_latency":{"avgcount":44,"sum":1.079972295,"avgtime":0.024544824},"op_inflight":0,"oplen_avg":{"avgcount":44,"sum":68},"op":44,"op_r":0,"op_w":44,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":11,"osdop_read":0,"osdop_write":5,"osdop_writefull":8,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":22,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":76,"map_full":0,"map_inc":1,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":3},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":145,"get_sum":121394,"get_or_fail_fail":0,"get_or_fail_success":145,"take":0,"take_sum":0,"put":145,"put_sum":121394,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":44,"take_sum":14083,"put":44,"put_sum":14083,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":44,"take_sum":44,"put":44,"put_sum":44,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x558a64cce0a0":{"val":0,"max":3758096384,"get_started":0,"get":6,"get_sum":2873,"get_or_fail_fail":0,"get_or_fail_success":6,"take":0,"take_sum":0,"put":5,"put_sum":2873,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-01-24T17:47:06.263 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 2, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 115515, "msgr_recv_encrypted_bytes": 112224, "msgr_recv_messages": 74, "msgr_running_fast_dispatch_time": 0.000778652, "msgr_running_recv_time": 0.014483548, "msgr_running_send_time": 0.003369149, "msgr_running_total_time": 0.017855664, "msgr_send_bytes": 20080, "msgr_send_encrypted_bytes": 14384, "msgr_send_messages": 69, "msgr_send_messages_queue_lat": { "avgcount": 69, "avgtime": 0.003383695, "sum": 0.233474981 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 22, "avgtime": 2.77e-07, "sum": 6.115e-06 }, "msgr_recv_bytes": 9479, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 31, "msgr_running_fast_dispatch_time": 0.000891314, "msgr_running_recv_time": 0.011415007, "msgr_running_send_time": 0.004007241, "msgr_running_total_time": 0.016840369, "msgr_send_bytes": 124961, "msgr_send_encrypted_bytes": 100288, "msgr_send_messages": 33, "msgr_send_messages_queue_lat": { "avgcount": 32, "avgtime": 0.003571088, "sum": 0.11427484 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 26, "avgtime": 2.43e-07, "sum": 6.328e-06 }, "msgr_recv_bytes": 13957, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 44, "msgr_running_fast_dispatch_time": 0.002293695, "msgr_running_recv_time": 0.006660533, "msgr_running_send_time": 0.003293892, "msgr_running_total_time": 0.012737168, "msgr_send_bytes": 19492, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 29, "msgr_send_messages_queue_lat": { "avgcount": 27, "avgtime": 0.00673806, "sum": 0.181927623 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 53, "avgtime": 8.016e-05, "sum": 0.004248498 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2, "avgtime": 1.2212e-05, "sum": 2.4425e-05 }, "queue_len": 0 }, "mds": { "caps": 0, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 0, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 0, "dir_fetch_keys": 0, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 0, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 10, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 0, "inodes_with_caps": 0, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 3, "reply_latency": { "avgcount": 3, "avgtime": 0.000466274, "sum": 0.001398824 }, "request": 3, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6, "evadd": 6, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 5, "avgtime": 0.00421954, "sum": 0.021097703 }, "rdpos": 0, "replayed": 0, "seg": 1, "segadd": 1, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 4197177 }, "mds_mem": { "cap": 0, "cap+": 2, "cap-": 2, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 207196, "ino": 13, "ino+": 13, "ino-": 0, "rss": 39296 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 8, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000377553, "sum": 0.000755106 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 1, "avgtime": 0.000643718, "sum": 0.000643718 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 12, "mdthresh_evicted": 0, "session_add": 2, "session_count": 0, "session_remove": 2, "sessions_open": 0, "sessions_stale": 0, "total_load": 2 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 221411, "buffer_anon_items": 62, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51440, "mds_co_items": 211, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5312, "osdmap_items": 75, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 76, "map_full": 0, "map_inc": 1, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 44, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 44, "avgtime": 0.024544824, "sum": 1.079972295 }, "op_pg": 0, "op_r": 0, "op_reply": 44, "op_resend": 37, "op_rmw": 0, "op_send": 44, "op_send_bytes": 14083, "op_w": 44, "oplen_avg": { "avgcount": 44, "sum": 68 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 11, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 22, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 5, "osdop_writefull": 8, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 1, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 3 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 145, "get_or_fail_fail": 0, "get_or_fail_success": 145, "get_started": 0, "get_sum": 121394, "max": 104857600, "put": 145, "put_sum": 121394, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 44, "put_sum": 14083, "take": 44, "take_sum": 14083, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 44, "put_sum": 44, "take": 44, "take_sum": 44, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x558a64cce0a0": { "get": 6, "get_or_fail_fail": 0, "get_or_fail_success": 6, "get_started": 0, "get_sum": 2873, "max": 3758096384, "put": 5, "put_sum": 2873, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-01-24T17:47:06.263 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 session ls 2024-01-24T17:47:06.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.484+0000 7f963a000640 1 Processor -- start 2024-01-24T17:47:06.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.484+0000 7f963a000640 1 -- start start 2024-01-24T17:47:06.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f963a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f963415c920 0x7f96340cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f963a000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f96340d1520 0x7f96340b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f963a000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96340d0b40 con 0x7f96340d0610 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f963a000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96340b1d00 con 0x7f963415c920 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f963a000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96340b1fe0 con 0x7f96340d1520 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f9638c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f963415c920 0x7f96340cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f9638c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f963415c920 0x7f96340cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44692/0 (socket says 172.21.15.39:44692) 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f9638c00640 1 -- 172.21.15.39:0/3862367595 learned_addr learned my addr 172.21.15.39:0/3862367595 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f9633e00640 1 --1- 172.21.15.39:0/3862367595 >> v1:172.21.15.39:6789/0 conn(0x7f96340d0610 0x7f96340d0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49756/0 (socket says 172.21.15.39:49756) 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.486+0000 7f9639600640 1 --2- 172.21.15.39:0/3862367595 >> v2:172.21.15.100:3300/0 conn(0x7f96340d1520 0x7f96340b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.487+0000 7f9638c00640 1 -- 172.21.15.39:0/3862367595 >> v2:172.21.15.100:3300/0 conn(0x7f96340d1520 msgr2=0x7f96340b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.487+0000 7f9638c00640 1 --2- 172.21.15.39:0/3862367595 >> v2:172.21.15.100:3300/0 conn(0x7f96340d1520 0x7f96340b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.487+0000 7f9638c00640 1 -- 172.21.15.39:0/3862367595 >> v1:172.21.15.39:6789/0 conn(0x7f96340d0610 legacy=0x7f96340d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.487+0000 7f9638c00640 1 -- 172.21.15.39:0/3862367595 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96340b22c0 con 0x7f963415c920 2024-01-24T17:47:06.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.487+0000 7f9639600640 1 --2- 172.21.15.39:0/3862367595 >> v2:172.21.15.100:3300/0 conn(0x7f96340d1520 0x7f96340b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.488+0000 7f9638c00640 1 --2- 172.21.15.39:0/3862367595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f963415c920 0x7f96340cf870 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f96340b42c0 tx=0x7f962402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92ceae5e3325335 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.488+0000 7f9633400640 1 -- 172.21.15.39:0/3862367595 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962402f850 con 0x7f963415c920 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.488+0000 7f9633400640 1 -- 172.21.15.39:0/3862367595 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962402f9b0 con 0x7f963415c920 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.488+0000 7f9633400640 1 -- 172.21.15.39:0/3862367595 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962402fd40 con 0x7f963415c920 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.489+0000 7f963a000640 1 -- 172.21.15.39:0/3862367595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f963415c920 msgr2=0x7f96340cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.489+0000 7f963a000640 1 --2- 172.21.15.39:0/3862367595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f963415c920 0x7f96340cf870 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f96340b42c0 tx=0x7f962402ba30 comp rx=0 tx=0).stop 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.489+0000 7f963a000640 1 -- 172.21.15.39:0/3862367595 shutdown_connections 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.489+0000 7f963a000640 1 --2- 172.21.15.39:0/3862367595 >> v2:172.21.15.100:3300/0 conn(0x7f96340d1520 0x7f96340b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.489+0000 7f963a000640 1 --2- 172.21.15.39:0/3862367595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f963415c920 0x7f96340cf870 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.489+0000 7f963a000640 1 -- 172.21.15.39:0/3862367595 >> 172.21.15.39:0/3862367595 conn(0x7f96340d2830 msgr2=0x7f96340d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:06.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.489+0000 7f963a000640 1 -- 172.21.15.39:0/3862367595 shutdown_connections 2024-01-24T17:47:06.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.489+0000 7f963a000640 1 -- 172.21.15.39:0/3862367595 wait complete. 2024-01-24T17:47:06.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.491+0000 7f963a000640 1 Processor -- start 2024-01-24T17:47:06.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.491+0000 7f963a000640 1 -- start start 2024-01-24T17:47:06.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f963a000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96340d0610 0x7f963415fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f963a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96340d1520 0x7f96340b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f963a000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f963415c920 0x7f96340b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f963a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963414d1a0 con 0x7f96340d1520 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f963a000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634160420 con 0x7f96340d0610 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f963a000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963414c590 con 0x7f963415c920 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f9633e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96340d1520 0x7f96340b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f9633e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96340d1520 0x7f96340b0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44710/0 (socket says 172.21.15.39:44710) 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.492+0000 7f9633e00640 1 -- 172.21.15.39:0/71316481 learned_addr learned my addr 172.21.15.39:0/71316481 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9638c00640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96340d0610 0x7f963415fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9639600640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f963415c920 0x7f96340b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9633e00640 1 -- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f963415c920 msgr2=0x7f96340b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9633e00640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f963415c920 0x7f96340b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9633e00640 1 -- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96340d0610 msgr2=0x7f963415fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9633e00640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96340d0610 0x7f963415fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9633e00640 1 -- 172.21.15.39:0/71316481 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96240094d0 con 0x7f96340d1520 2024-01-24T17:47:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9633e00640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96340d1520 0x7f96340b0730 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f961c00c450 tx=0x7f961c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.493+0000 7f9630c00640 1 -- 172.21.15.39:0/71316481 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c007480 con 0x7f96340d1520 2024-01-24T17:47:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.494+0000 7f9630c00640 1 -- 172.21.15.39:0/71316481 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961c0075e0 con 0x7f96340d1520 2024-01-24T17:47:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.494+0000 7f9630c00640 1 -- 172.21.15.39:0/71316481 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c004a00 con 0x7f96340d1520 2024-01-24T17:47:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.494+0000 7f963a000640 1 -- 172.21.15.39:0/71316481 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963414c870 con 0x7f96340d1520 2024-01-24T17:47:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.494+0000 7f963a000640 1 -- 172.21.15.39:0/71316481 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963414ce30 con 0x7f96340d1520 2024-01-24T17:47:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.496+0000 7f9630c00640 1 -- 172.21.15.39:0/71316481 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f961c004b60 con 0x7f96340d1520 2024-01-24T17:47:06.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.497+0000 7f9630c00640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f960c06a460 0x7f960c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.497+0000 7f9630c00640 1 -- 172.21.15.39:0/71316481 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f961c087a30 con 0x7f96340d1520 2024-01-24T17:47:06.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.497+0000 7f9638c00640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f960c06a460 0x7f960c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.498+0000 7f963e213740 1 Processor -- start 2024-01-24T17:47:06.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.498+0000 7f963e213740 1 -- start start 2024-01-24T17:47:06.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f960c06a460 0x7f960c06c920 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9624036000 tx=0x7f9624008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:06.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f963e213740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a44dc280 0x55a0a44af410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f963e213740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 0x55a0a44df010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f963e213740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44af950 0x55a0a44df550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f963e213740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a441f210 con 0x55a0a44dc280 2024-01-24T17:47:06.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f963e213740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a441c9a0 con 0x55a0a44dc660 2024-01-24T17:47:06.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f963e213740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0a4418290 con 0x55a0a44af950 2024-01-24T17:47:06.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a44dc280 0x55a0a44af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a44dc280 0x55a0a44af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44712/0 (socket says 172.21.15.39:44712) 2024-01-24T17:47:06.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 -- 172.21.15.39:0/4216526615 learned_addr learned my addr 172.21.15.39:0/4216526615 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:06.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9633e00640 1 --2- 172.21.15.39:0/4216526615 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 0x55a0a44df010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 -- 172.21.15.39:0/4216526615 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44af950 msgr2=0x55a0a44df550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 --2- 172.21.15.39:0/4216526615 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44af950 0x55a0a44df550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 -- 172.21.15.39:0/4216526615 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 msgr2=0x55a0a44df010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 --2- 172.21.15.39:0/4216526615 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 0x55a0a44df010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.499+0000 7f9638c00640 1 -- 172.21.15.39:0/4216526615 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0a43d4f20 con 0x55a0a44dc280 2024-01-24T17:47:06.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.500+0000 7f9638c00640 1 --2- 172.21.15.39:0/4216526615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a44dc280 0x55a0a44af410 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f962402fd40 tx=0x7f9624030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fe33921ddf4358 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:06.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.500+0000 7f9629600640 1 -- 172.21.15.39:0/4216526615 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624048070 con 0x55a0a44dc280 2024-01-24T17:47:06.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.500+0000 7f9629600640 1 -- 172.21.15.39:0/4216526615 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962402fe30 con 0x55a0a44dc280 2024-01-24T17:47:06.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.500+0000 7f9629600640 1 -- 172.21.15.39:0/4216526615 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962404e420 con 0x55a0a44dc280 2024-01-24T17:47:06.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 -- 172.21.15.39:0/4216526615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a44dc280 msgr2=0x55a0a44af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 --2- 172.21.15.39:0/4216526615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a44dc280 0x55a0a44af410 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f962402fd40 tx=0x7f9624030000 comp rx=0 tx=0).stop 2024-01-24T17:47:06.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 -- 172.21.15.39:0/4216526615 shutdown_connections 2024-01-24T17:47:06.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 --2- 172.21.15.39:0/4216526615 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44af950 0x55a0a44df550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 --2- 172.21.15.39:0/4216526615 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 0x55a0a44df010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 --2- 172.21.15.39:0/4216526615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a44dc280 0x55a0a44af410 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 -- 172.21.15.39:0/4216526615 >> 172.21.15.39:0/4216526615 conn(0x55a0a44b1720 msgr2=0x55a0a44fa010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:06.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 -- 172.21.15.39:0/4216526615 shutdown_connections 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.501+0000 7f963e213740 1 -- 172.21.15.39:0/4216526615 wait complete. 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.502+0000 7f963e213740 1 -- start start 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.502+0000 7f963e213740 1 Processor -- start 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f963e213740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 0x55a0a450aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f963e213740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f963e213740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a44f2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f963e213740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a43b9890 con 0x55a0a450b000 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f963e213740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a43b9530 con 0x55a0a44dc280 2024-01-24T17:47:06.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f963e213740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0a4115f90 con 0x55a0a44dc660 2024-01-24T17:47:06.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f963e213740 10 client.0 _ll_register_callbacks cb 0x55a0a4411bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:47:06.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f9639600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a44f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f9639600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a44f2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44718/0 (socket says 172.21.15.39:44718) 2024-01-24T17:47:06.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.503+0000 7f9639600640 1 -- 172.21.15.39:0/3062666551 learned_addr learned my addr 172.21.15.39:0/3062666551 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:06.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9633e00640 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9638c00640 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 0x55a0a450aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9639600640 1 -- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 msgr2=0x55a0a44f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9639600640 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9639600640 1 -- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 msgr2=0x55a0a450aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9639600640 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 0x55a0a450aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9639600640 1 -- 172.21.15.39:0/3062666551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96340b22c0 con 0x55a0a450b000 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9633e00640 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9639600640 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a44f2170 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f962c00eae0 tx=0x7f962c00c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88f5820b38c2da3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9622000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9622000640 1 -- 172.21.15.39:0/3062666551 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c00a410 con 0x55a0a450b000 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f9622000640 1 -- 172.21.15.39:0/3062666551 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f962c00a570 con 0x55a0a450b000 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.504+0000 7f963e213740 1 -- 172.21.15.39:0/3062666551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0a44435a0 con 0x55a0a450b000 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f9622000640 1 -- 172.21.15.39:0/3062666551 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f962c0079c0 con 0x55a0a450b000 2024-01-24T17:47:06.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f963e213740 10 client.7134 fetch_fsmap learned FSMap version 42 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f963e213740 1 -- 172.21.15.39:0/3062666551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55a0a4416910 con 0x55a0a450b000 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f9622000640 1 -- 172.21.15.39:0/3062666551 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a0a44435a0 con 0x55a0a450b000 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f9622000640 1 -- 172.21.15.39:0/3062666551 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f962c008040 con 0x55a0a450b000 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f963e213740 10 client.7134 fetch_fsmap finished waiting for FSMap version 42 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f963e213740 10 client.7134 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.d 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f963e213740 20 client.7134 populate_metadata read hostname 'smithi039' 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f963e213740 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x55a0a44f2ff0 0x55a0a44f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f963e213740 4 client.7134 mds_command: new command op to 6804 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.505+0000 7f963e213740 1 -- 172.21.15.39:0/3062666551 --> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0a440bf40 con 0x55a0a44f2ff0 2024-01-24T17:47:06.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.506+0000 7f9638c00640 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x55a0a44f2ff0 0x55a0a44f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.507+0000 7f9638c00640 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x55a0a44f2ff0 0x55a0a44f33d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4ab21a14bb12a4a server_cookie=90ce25d6d8f6553a in_seq=0 out_seq=0 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.507+0000 7f9622000640 10 client.7134 ms_handle_connect on v2:172.21.15.100:6834/2467066037 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.508+0000 7f9622000640 1 -- 172.21.15.39:0/3062666551 <== mds.0 v2:172.21.15.100:6834/2467066037 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a0a440bf40 con 0x55a0a44f2ff0 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.508+0000 7f9622000640 10 client.7134 handle_command_reply: tid=0 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.508+0000 7f963e213740 1 client.7134 shutdown 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.508+0000 7f963e213740 1 -- 172.21.15.39:0/3062666551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 msgr2=0x55a0a44f2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.508+0000 7f963e213740 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a44f2170 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f962c00eae0 tx=0x7f962c00c3c0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 1 -- 172.21.15.39:0/3062666551 shutdown_connections 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x55a0a44f2ff0 0x55a0a44f33d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a44f2170 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 1 --2- 172.21.15.39:0/3062666551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 0x55a0a450aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f9622000640 0 client.7134 ms_handle_reset on v2:172.21.15.100:6834/2467066037 2024-01-24T17:47:06.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 1 -- 172.21.15.39:0/3062666551 >> 172.21.15.39:0/3062666551 conn(0x55a0a44af950 msgr2=0x55a0a44b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:06.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 1 -- 172.21.15.39:0/3062666551 shutdown_connections 2024-01-24T17:47:06.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 1 -- 172.21.15.39:0/3062666551 wait complete. 2024-01-24T17:47:06.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.509+0000 7f963e213740 20 client.7134 trim_cache size 0 max 16384 2024-01-24T17:47:06.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.526+0000 7f963e213740 1 Processor -- start 2024-01-24T17:47:06.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.526+0000 7f963e213740 1 -- start start 2024-01-24T17:47:06.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f963e213740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 0x55a0a44f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f963e213740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f963e213740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a4556150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f963e213740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a451b0b0 con 0x55a0a450b000 2024-01-24T17:47:06.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f963e213740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a451b300 con 0x55a0a44dc280 2024-01-24T17:47:06.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f963e213740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0a44fabf0 con 0x55a0a44dc660 2024-01-24T17:47:06.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9639600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a4556150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9639600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a4556150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44726/0 (socket says 172.21.15.39:44726) 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9639600640 1 -- 172.21.15.39:0/4293815361 learned_addr learned my addr 172.21.15.39:0/4293815361 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9639600640 1 -- 172.21.15.39:0/4293815361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 msgr2=0x55a0a44f33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9633e00640 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9638c00640 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 0x55a0a44f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9639600640 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9639600640 1 -- 172.21.15.39:0/4293815361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 msgr2=0x55a0a44f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9639600640 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 0x55a0a44f2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.527+0000 7f9639600640 1 -- 172.21.15.39:0/4293815361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0a444d0a0 con 0x55a0a450b000 2024-01-24T17:47:06.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.528+0000 7f9639600640 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a4556150 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f962c007f60 tx=0x7f962c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6151e9d50e3585 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:06.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.528+0000 7f962a000640 1 -- 172.21.15.39:0/4293815361 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c00c8c0 con 0x55a0a450b000 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.528+0000 7f962a000640 1 -- 172.21.15.39:0/4293815361 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c00a910 con 0x55a0a450b000 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.528+0000 7f962a000640 1 -- 172.21.15.39:0/4293815361 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c00ab20 con 0x55a0a450b000 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 -- 172.21.15.39:0/4293815361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 msgr2=0x55a0a4556150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a4556150 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f962c007f60 tx=0x7f962c011000 comp rx=0 tx=0).stop 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 -- 172.21.15.39:0/4293815361 shutdown_connections 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a4556150 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc660 0x55a0a44f33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 --2- 172.21.15.39:0/4293815361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc280 0x55a0a44f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 -- 172.21.15.39:0/4293815361 >> 172.21.15.39:0/4293815361 conn(0x55a0a44af950 msgr2=0x55a0a44b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 -- 172.21.15.39:0/4293815361 shutdown_connections 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.529+0000 7f963e213740 1 -- 172.21.15.39:0/4293815361 wait complete. 2024-01-24T17:47:06.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.530+0000 7f963e213740 1 -- start start 2024-01-24T17:47:06.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.530+0000 7f963e213740 1 Processor -- start 2024-01-24T17:47:06.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f963e213740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc280 0x55a0a44efda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f963e213740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 0x55a0a4555390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f963e213740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a454dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f963e213740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a44f37c0 con 0x55a0a450b000 2024-01-24T17:47:06.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f963e213740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a454eb20 con 0x55a0a44dc660 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f963e213740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0a45128e0 con 0x55a0a44dc280 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f963e213740 10 client.0 _ll_register_callbacks cb 0x7f962c007700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f9639600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a454dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f9639600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a454dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44740/0 (socket says 172.21.15.39:44740) 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f9639600640 1 -- 172.21.15.39:0/2683378172 learned_addr learned my addr 172.21.15.39:0/2683378172 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.531+0000 7f9633e00640 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 0x55a0a4555390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9639600640 1 -- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc280 msgr2=0x55a0a44efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9639600640 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc280 0x55a0a44efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9639600640 1 -- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 msgr2=0x55a0a4555390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9639600640 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 0x55a0a4555390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9639600640 1 -- 172.21.15.39:0/2683378172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0a4416910 con 0x55a0a450b000 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9639600640 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a454dff0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f962c00a8e0 tx=0x7f962c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10274428c33e569 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9622000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9622000640 1 -- 172.21.15.39:0/2683378172 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c00ae50 con 0x55a0a450b000 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9622000640 1 -- 172.21.15.39:0/2683378172 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f962c00cda0 con 0x55a0a450b000 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f963e213740 1 -- 172.21.15.39:0/2683378172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f962402fe30 con 0x55a0a450b000 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f9622000640 1 -- 172.21.15.39:0/2683378172 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f962c018600 con 0x55a0a450b000 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f963e213740 10 client.7140 fetch_fsmap learned FSMap version 42 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.532+0000 7f963e213740 1 -- 172.21.15.39:0/2683378172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55a0a43b8c40 con 0x55a0a450b000 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.533+0000 7f9622000640 1 -- 172.21.15.39:0/2683378172 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962402fe30 con 0x55a0a450b000 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.533+0000 7f9622000640 1 -- 172.21.15.39:0/2683378172 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f962c018760 con 0x55a0a450b000 2024-01-24T17:47:06.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.533+0000 7f963e213740 10 client.7140 fetch_fsmap finished waiting for FSMap version 42 2024-01-24T17:47:06.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.533+0000 7f963e213740 10 client.7140 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.d 2024-01-24T17:47:06.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.533+0000 7f963e213740 20 client.7140 populate_metadata read hostname 'smithi039' 2024-01-24T17:47:06.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.533+0000 7f963e213740 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x55a0a4550ab0 0x55a0a4511410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:06.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.533+0000 7f963e213740 4 client.7140 mds_command: new command op to 6804 tid=0[{"prefix": "session ls"}] 2024-01-24T17:47:06.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.533+0000 7f963e213740 1 -- 172.21.15.39:0/2683378172 --> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] -- command(tid 0: {"prefix": "session ls"}) v1 -- 0x55a0a4411310 con 0x55a0a4550ab0 2024-01-24T17:47:06.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.534+0000 7f9638c00640 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x55a0a4550ab0 0x55a0a4511410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:06.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f9638c00640 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x55a0a4550ab0 0x55a0a4511410 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ec2adc5a334058 server_cookie=acc8cadab1db3115 in_seq=0 out_seq=0 2024-01-24T17:47:06.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f9622000640 10 client.7140 ms_handle_connect on v2:172.21.15.100:6834/2467066037 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f9622000640 1 -- 172.21.15.39:0/2683378172 <== mds.0 v2:172.21.15.100:6834/2467066037 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a0a4411310 con 0x55a0a4550ab0 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f9622000640 10 client.7140 handle_command_reply: tid=0 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 client.7140 shutdown 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 -- 172.21.15.39:0/2683378172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 msgr2=0x55a0a454dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a454dff0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f962c00a8e0 tx=0x7f962c004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 -- 172.21.15.39:0/2683378172 shutdown_connections 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:6834/2467066037,v1:172.21.15.100:6836/2467066037] conn(0x55a0a4550ab0 0x55a0a4511410 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0a450b000 0x55a0a454dff0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55a0a44dc660 0x55a0a4555390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 --2- 172.21.15.39:0/2683378172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55a0a44dc280 0x55a0a44efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f9622000640 0 client.7140 ms_handle_reset on v2:172.21.15.100:6834/2467066037 2024-01-24T17:47:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 -- 172.21.15.39:0/2683378172 >> 172.21.15.39:0/2683378172 conn(0x55a0a44af950 msgr2=0x55a0a44f3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 -- 172.21.15.39:0/2683378172 shutdown_connections 2024-01-24T17:47:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.535+0000 7f963e213740 1 -- 172.21.15.39:0/2683378172 wait complete. 2024-01-24T17:47:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963e213740 20 client.7140 trim_cache size 0 max 16384 2024-01-24T17:47:06.541 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-01-24T17:47:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 -- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f960c06a460 msgr2=0x7f960c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f960c06a460 0x7f960c06c920 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9624036000 tx=0x7f9624008030 comp rx=0 tx=0).stop 2024-01-24T17:47:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 -- 172.21.15.39:0/71316481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96340d1520 msgr2=0x7f96340b0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96340d1520 0x7f96340b0730 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f961c00c450 tx=0x7f961c00c910 comp rx=0 tx=0).stop 2024-01-24T17:47:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 -- 172.21.15.39:0/71316481 shutdown_connections 2024-01-24T17:47:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f963415c920 0x7f96340b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f960c06a460 0x7f960c06c920 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96340d1520 0x7f96340b0730 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 --2- 172.21.15.39:0/71316481 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96340d0610 0x7f963415fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 -- 172.21.15.39:0/71316481 >> 172.21.15.39:0/71316481 conn(0x7f96340d2830 msgr2=0x7f963415ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 -- 172.21.15.39:0/71316481 shutdown_connections 2024-01-24T17:47:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:06.536+0000 7f963a000640 1 -- 172.21.15.39:0/71316481 wait complete. 2024-01-24T17:47:06.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump 2024-01-24T17:47:06.787 INFO:teuthology.orchestra.run.smithi100.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":74,"msgr_send_messages":69,"msgr_recv_bytes":115515,"msgr_send_bytes":20080,"msgr_created_connections":6,"msgr_active_connections":2,"msgr_running_total_time":0.018291863,"msgr_running_send_time":0.003369149,"msgr_running_recv_time":0.014483548,"msgr_running_fast_dispatch_time":0.000778652,"msgr_send_messages_queue_lat":{"avgcount":69,"sum":0.233474981,"avgtime":0.003383695},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":112224,"msgr_send_encrypted_bytes":14384},"AsyncMessenger::Worker-1":{"msgr_recv_messages":33,"msgr_send_messages":35,"msgr_recv_bytes":9769,"msgr_send_bytes":142979,"msgr_created_connections":10,"msgr_active_connections":3,"msgr_running_total_time":0.018978783,"msgr_running_send_time":0.004181505,"msgr_running_recv_time":0.013173474,"msgr_running_fast_dispatch_time":0.000891314,"msgr_send_messages_queue_lat":{"avgcount":34,"sum":0.114361105,"avgtime":0.003363561},"msgr_handle_ack_lat":{"avgcount":24,"sum":0.000006442,"avgtime":0.000000268},"msgr_recv_encrypted_bytes":1728,"msgr_send_encrypted_bytes":100288},"AsyncMessenger::Worker-2":{"msgr_recv_messages":44,"msgr_send_messages":29,"msgr_recv_bytes":13957,"msgr_send_bytes":19492,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":0.012737168,"msgr_running_send_time":0.003293892,"msgr_running_recv_time":0.006660533,"msgr_running_fast_dispatch_time":0.002293695,"msgr_send_messages_queue_lat":{"avgcount":27,"sum":0.181927623,"avgtime":0.006738060},"msgr_handle_ack_lat":{"avgcount":26,"sum":0.000006328,"avgtime":0.000000243},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":53,"sum":0.004248498,"avgtime":0.000080160}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2,"sum":0.000024425,"avgtime":0.000012212}},"mds":{"request":3,"reply":3,"reply_latency":{"avgcount":3,"sum":0.001398824,"avgtime":0.000466274},"slow_reply":0,"forward":0,"dir_fetch_complete":0,"dir_fetch_keys":0,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":0,"inodes_bottom":10,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":0,"caps":0,"subtrees":2,"traverse":5,"traverse_hit":5,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":0,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":0,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evadd":6,"evex":0,"evtrm":0,"ev":6,"evexg":0,"evexd":0,"segadd":1,"segex":0,"segtrm":0,"seg":1,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4197177,"rdpos":0,"jlat":{"avgcount":5,"sum":0.021097703,"avgtime":0.004219540},"replayed":0},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":0,"cap+":2,"cap-":2,"rss":39296,"heap":207196},"mds_server":{"dispatch_client_request":3,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":8,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":2,"sum":0.000755106,"avgtime":0.000377553},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":1,"sum":0.000643718,"avgtime":0.000643718},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":0,"session_add":2,"session_remove":2,"sessions_open":0,"sessions_stale":0,"total_load":2,"average_load":1,"avg_session_uptime":12,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":221411,"buffer_anon_items":62,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5312,"osdmap_items":75,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51440,"mds_co_items":211,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":44,"op_send_bytes":14083,"op_resend":37,"op_reply":44,"op_latency":{"avgcount":44,"sum":1.079972295,"avgtime":0.024544824},"op_inflight":0,"oplen_avg":{"avgcount":44,"sum":68},"op":44,"op_r":0,"op_w":44,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":11,"osdop_read":0,"osdop_write":5,"osdop_writefull":8,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":22,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":76,"map_full":0,"map_inc":1,"osd_sessions":7,"osd_session_open":7,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":3},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":147,"get_sum":121504,"get_or_fail_fail":0,"get_or_fail_success":147,"take":0,"take_sum":0,"put":147,"put_sum":121504,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":44,"take_sum":14083,"put":44,"put_sum":14083,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":44,"take_sum":44,"put":44,"put_sum":44,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x558a64cce0a0":{"val":0,"max":3758096384,"get_started":0,"get":6,"get_sum":2873,"get_or_fail_fail":0,"get_or_fail_success":6,"take":0,"take_sum":0,"put":5,"put_sum":2873,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-01-24T17:47:06.788 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 2, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 115515, "msgr_recv_encrypted_bytes": 112224, "msgr_recv_messages": 74, "msgr_running_fast_dispatch_time": 0.000778652, "msgr_running_recv_time": 0.014483548, "msgr_running_send_time": 0.003369149, "msgr_running_total_time": 0.018291863, "msgr_send_bytes": 20080, "msgr_send_encrypted_bytes": 14384, "msgr_send_messages": 69, "msgr_send_messages_queue_lat": { "avgcount": 69, "avgtime": 0.003383695, "sum": 0.233474981 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 10, "msgr_handle_ack_lat": { "avgcount": 24, "avgtime": 2.68e-07, "sum": 6.442e-06 }, "msgr_recv_bytes": 9769, "msgr_recv_encrypted_bytes": 1728, "msgr_recv_messages": 33, "msgr_running_fast_dispatch_time": 0.000891314, "msgr_running_recv_time": 0.013173474, "msgr_running_send_time": 0.004181505, "msgr_running_total_time": 0.018978783, "msgr_send_bytes": 142979, "msgr_send_encrypted_bytes": 100288, "msgr_send_messages": 35, "msgr_send_messages_queue_lat": { "avgcount": 34, "avgtime": 0.003363561, "sum": 0.114361105 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 26, "avgtime": 2.43e-07, "sum": 6.328e-06 }, "msgr_recv_bytes": 13957, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 44, "msgr_running_fast_dispatch_time": 0.002293695, "msgr_running_recv_time": 0.006660533, "msgr_running_send_time": 0.003293892, "msgr_running_total_time": 0.012737168, "msgr_send_bytes": 19492, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 29, "msgr_send_messages_queue_lat": { "avgcount": 27, "avgtime": 0.00673806, "sum": 0.181927623 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 53, "avgtime": 8.016e-05, "sum": 0.004248498 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2, "avgtime": 1.2212e-05, "sum": 2.4425e-05 }, "queue_len": 0 }, "mds": { "caps": 0, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 0, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 0, "dir_fetch_keys": 0, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 0, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 10, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 0, "inodes_with_caps": 0, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 3, "reply_latency": { "avgcount": 3, "avgtime": 0.000466274, "sum": 0.001398824 }, "request": 3, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6, "evadd": 6, "evex": 0, "evexd": 0, "evexg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 5, "avgtime": 0.00421954, "sum": 0.021097703 }, "rdpos": 0, "replayed": 0, "seg": 1, "segadd": 1, "segex": 0, "segexd": 0, "segexg": 0, "segtrm": 0, "wrpos": 4197177 }, "mds_mem": { "cap": 0, "cap+": 2, "cap-": 2, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 207196, "ino": 13, "ino+": 13, "ino-": 0, "rss": 39296 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 8, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000377553, "sum": 0.000755106 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 1, "avgtime": 0.000643718, "sum": 0.000643718 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 12, "mdthresh_evicted": 0, "session_add": 2, "session_count": 0, "session_remove": 2, "sessions_open": 0, "sessions_stale": 0, "total_load": 2 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 221411, "buffer_anon_items": 62, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51440, "mds_co_items": 211, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5312, "osdmap_items": 75, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 76, "map_full": 0, "map_inc": 1, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 44, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 44, "avgtime": 0.024544824, "sum": 1.079972295 }, "op_pg": 0, "op_r": 0, "op_reply": 44, "op_resend": 37, "op_rmw": 0, "op_send": 44, "op_send_bytes": 14083, "op_w": 44, "oplen_avg": { "avgcount": 44, "sum": 68 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 7, "osd_sessions": 7, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 11, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 22, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 5, "osdop_writefull": 8, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 1, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 3 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 147, "get_or_fail_fail": 0, "get_or_fail_success": 147, "get_started": 0, "get_sum": 121504, "max": 104857600, "put": 147, "put_sum": 121504, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 44, "put_sum": 14083, "take": 44, "take_sum": 14083, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 44, "put_sum": 44, "take": 44, "take_sum": 44, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x558a64cce0a0": { "get": 6, "get_or_fail_fail": 0, "get_or_fail_success": 6, "get_started": 0, "get_sum": 2873, "max": 3758096384, "put": 5, "put_sum": 2873, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-01-24T17:47:06.788 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-01-24T17:47:06.789 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:47:06.789 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:47:06.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:47:07.030 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:47:07.030 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:47:07.030 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:07.090 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:47:07.091 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:47:07.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:07.091 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:47:07.147 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-24T17:47:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:47:07.148 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:47:07.148 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:07.206 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:47:07.206 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:47:07.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:07.207 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:47:07.259 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-24T17:47:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:47:07.260 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:47:07.260 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:07.322 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:47:07.322 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:47:07.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:07.322 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-24T17:47:07.380 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-24T17:47:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:47:07.381 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:47:07.382 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:07.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:47:07.438 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:47:07.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:07.439 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-24T17:47:07.489 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-24T17:47:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:47:07.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:47:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.722+0000 7ffb12800640 1 Processor -- start 2024-01-24T17:47:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.722+0000 7ffb12800640 1 -- start start 2024-01-24T17:47:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.723+0000 7ffb12800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffb0c0ca320 0x7ffb0c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.723+0000 7ffb12800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ffb0c0ca700 0x7ffb0c15a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.723+0000 7ffb12800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c15a910 0x7ffb0c06bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.723+0000 7ffb12800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c159fd0 con 0x7ffb0c0ca320 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb12800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c1540f0 con 0x7ffb0c0ca700 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb12800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c15da00 con 0x7ffb0c15a910 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffb0c0ca320 0x7ffb0c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffb0c0ca320 0x7ffb0c15d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44742/0 (socket says 172.21.15.39:44742) 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11400640 1 -- 172.21.15.39:0/2778769648 learned_addr learned my addr 172.21.15.39:0/2778769648 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb10a00640 1 --2- 172.21.15.39:0/2778769648 >> v2:172.21.15.100:3300/0 conn(0x7ffb0c0ca700 0x7ffb0c15a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11e00640 1 --2- 172.21.15.39:0/2778769648 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c15a910 0x7ffb0c06bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11400640 1 -- 172.21.15.39:0/2778769648 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c15a910 msgr2=0x7ffb0c06bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11400640 1 --2- 172.21.15.39:0/2778769648 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c15a910 0x7ffb0c06bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11400640 1 -- 172.21.15.39:0/2778769648 >> v2:172.21.15.100:3300/0 conn(0x7ffb0c0ca700 msgr2=0x7ffb0c15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11400640 1 --2- 172.21.15.39:0/2778769648 >> v2:172.21.15.100:3300/0 conn(0x7ffb0c0ca700 0x7ffb0c15a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb11400640 1 -- 172.21.15.39:0/2778769648 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c15dce0 con 0x7ffb0c0ca320 2024-01-24T17:47:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.724+0000 7ffb10a00640 1 --2- 172.21.15.39:0/2778769648 >> v2:172.21.15.100:3300/0 conn(0x7ffb0c0ca700 0x7ffb0c15a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.726+0000 7ffb11400640 1 --2- 172.21.15.39:0/2778769648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c0ca320 0x7ffb0c15d1b0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c070600 tx=0x7ffafc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedf83f0ad2399b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.726+0000 7ffb0be00640 1 -- 172.21.15.39:0/2778769648 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc004230 con 0x7ffb0c0ca320 2024-01-24T17:47:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.726+0000 7ffb0be00640 1 -- 172.21.15.39:0/2778769648 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc02f050 con 0x7ffb0c0ca320 2024-01-24T17:47:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.726+0000 7ffb0be00640 1 -- 172.21.15.39:0/2778769648 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc003610 con 0x7ffb0c0ca320 2024-01-24T17:47:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 -- 172.21.15.39:0/2778769648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c0ca320 msgr2=0x7ffb0c15d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 --2- 172.21.15.39:0/2778769648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c0ca320 0x7ffb0c15d1b0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c070600 tx=0x7ffafc02b7b0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 -- 172.21.15.39:0/2778769648 shutdown_connections 2024-01-24T17:47:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 --2- 172.21.15.39:0/2778769648 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c15a910 0x7ffb0c06bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 --2- 172.21.15.39:0/2778769648 >> v2:172.21.15.100:3300/0 conn(0x7ffb0c0ca700 0x7ffb0c15a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 --2- 172.21.15.39:0/2778769648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c0ca320 0x7ffb0c15d1b0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 -- 172.21.15.39:0/2778769648 >> 172.21.15.39:0/2778769648 conn(0x7ffb0c06a340 msgr2=0x7ffb0c0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 -- 172.21.15.39:0/2778769648 shutdown_connections 2024-01-24T17:47:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.727+0000 7ffb12800640 1 -- 172.21.15.39:0/2778769648 wait complete. 2024-01-24T17:47:07.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.729+0000 7ffb12800640 1 Processor -- start 2024-01-24T17:47:07.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.729+0000 7ffb12800640 1 -- start start 2024-01-24T17:47:07.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb12800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c0ca320 0x7ffb0c151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:07.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb12800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb0c0ca700 0x7ffb0c0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb12800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c15a910 0x7ffb0c158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb12800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0c9d40 con 0x7ffb0c15a910 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb12800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0c9a60 con 0x7ffb0c0ca700 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb12800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c164460 con 0x7ffb0c0ca320 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c15a910 0x7ffb0c158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c15a910 0x7ffb0c158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44758/0 (socket says 172.21.15.39:44758) 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11e00640 1 -- 172.21.15.39:0/4066435993 learned_addr learned my addr 172.21.15.39:0/4066435993 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11400640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c0ca320 0x7ffb0c151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb10a00640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb0c0ca700 0x7ffb0c0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11e00640 1 -- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c0ca320 msgr2=0x7ffb0c151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11e00640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c0ca320 0x7ffb0c151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11e00640 1 -- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb0c0ca700 msgr2=0x7ffb0c0c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11e00640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb0c0ca700 0x7ffb0c0c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11e00640 1 -- 172.21.15.39:0/4066435993 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc009460 con 0x7ffb0c15a910 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb10a00640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb0c0ca700 0x7ffb0c0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.730+0000 7ffb11400640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c0ca320 0x7ffb0c151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.731+0000 7ffb11e00640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c15a910 0x7ffb0c158690 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400c550 tx=0x7ffb0400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.731+0000 7ffb09600640 1 -- 172.21.15.39:0/4066435993 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04004220 con 0x7ffb0c15a910 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.731+0000 7ffb09600640 1 -- 172.21.15.39:0/4066435993 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04004380 con 0x7ffb0c15a910 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.731+0000 7ffb09600640 1 -- 172.21.15.39:0/4066435993 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003830 con 0x7ffb0c15a910 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.731+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c164740 con 0x7ffb0c15a910 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.731+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c151db0 con 0x7ffb0c15a910 2024-01-24T17:47:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.732+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c164c60 con 0x7ffb0c15a910 2024-01-24T17:47:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.733+0000 7ffb09600640 1 -- 172.21.15.39:0/4066435993 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ffb04003990 con 0x7ffb0c15a910 2024-01-24T17:47:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.734+0000 7ffb09600640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffae406a460 0x7ffae406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.734+0000 7ffb09600640 1 -- 172.21.15.39:0/4066435993 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb04088580 con 0x7ffb0c15a910 2024-01-24T17:47:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.735+0000 7ffb11400640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffae406a460 0x7ffae406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:07.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.736+0000 7ffb11400640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffae406a460 0x7ffae406c920 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ffafc035000 tx=0x7ffafc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.738+0000 7ffb09600640 1 -- 172.21.15.39:0/4066435993 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ffb0c164c60 con 0x7ffb0c15a910 2024-01-24T17:47:07.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.976+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb0c164c60 con 0x7ffb0c15a910 2024-01-24T17:47:07.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.977+0000 7ffb09600640 1 -- 172.21.15.39:0/4066435993 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4742 (secure 0 0 0) 0x7ffb0c164c60 con 0x7ffb0c15a910 2024-01-24T17:47:07.979 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:07.979 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1886779141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1886779141},{"type":"v1","addr":"172.21.15.39:6833","nonce":1886779141}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5854,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2624477336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":2624477336},{"type":"v1","addr":"172.21.15.100:6837","nonce":2624477336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2465319794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2465319794},{"type":"v1","addr":"172.21.15.39:6835","nonce":2465319794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:46:31.015014+0000","modified":"2024-01-24T17:46:34.537399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6836/2467066037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2467066037},{"type":"v1","addr":"172.21.15.100:6836","nonce":2467066037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-24T17:47:07.979 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 42 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.979+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffae406a460 msgr2=0x7ffae406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.979+0000 7ffb12800640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffae406a460 0x7ffae406c920 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ffafc035000 tx=0x7ffafc008030 comp rx=0 tx=0).stop 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c15a910 msgr2=0x7ffb0c158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c15a910 0x7ffb0c158690 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400c550 tx=0x7ffb0400ca10 comp rx=0 tx=0).stop 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 shutdown_connections 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffae406a460 0x7ffae406c920 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0c15a910 0x7ffb0c158690 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb0c0ca700 0x7ffb0c0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 --2- 172.21.15.39:0/4066435993 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb0c0ca320 0x7ffb0c151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 >> 172.21.15.39:0/4066435993 conn(0x7ffb0c06a340 msgr2=0x7ffb0c15c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 shutdown_connections 2024-01-24T17:47:07.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:07.980+0000 7ffb12800640 1 -- 172.21.15.39:0/4066435993 wait complete. 2024-01-24T17:47:08.000 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:08.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.220+0000 7fc9c4200640 1 Processor -- start 2024-01-24T17:47:08.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.221+0000 7fc9c4200640 1 -- start start 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.222+0000 7fc9c4200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.222+0000 7fc9c4200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc9bc0b3f20 0x7fc9bc0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.222+0000 7fc9c4200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9bc0ccd10 con 0x7fc9bc159d00 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.222+0000 7fc9c4200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9bc0ce230 con 0x7fc9bc0b3f20 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.222+0000 7fc9c4200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0ccff0 con 0x7fc9bc071c00 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.222+0000 7fc9c1800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc9bc159d00 0x7fc9bc0714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49772/0 (socket says 172.21.15.39:49772) 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.222+0000 7fc9c1800640 1 -- 172.21.15.39:0/4126246865 learned_addr learned my addr 172.21.15.39:0/4126246865 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9c2200640 1 --2- 172.21.15.39:0/4126246865 >> v2:172.21.15.39:3300/0 conn(0x7fc9bc0b3f20 0x7fc9bc0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/4126246865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9bbe00640 1 -- 172.21.15.39:0/4126246865 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635134720 0 0) 0x7fc9bc0ccd10 con 0x7fc9bc159d00 2024-01-24T17:47:08.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9bbe00640 1 -- 172.21.15.39:0/4126246865 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc99c003520 con 0x7fc9bc159d00 2024-01-24T17:47:08.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9c2200640 1 -- 172.21.15.39:0/4126246865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 msgr2=0x7fc9bc0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:08.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9c2200640 1 --2- 172.21.15.39:0/4126246865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9c2200640 1 -- 172.21.15.39:0/4126246865 >> v1:172.21.15.39:6789/0 conn(0x7fc9bc159d00 legacy=0x7fc9bc0714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9c2200640 1 -- 172.21.15.39:0/4126246865 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc0ad720 con 0x7fc9bc0b3f20 2024-01-24T17:47:08.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.223+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/4126246865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.224+0000 7fc9c2200640 1 --2- 172.21.15.39:0/4126246865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc0b3f20 0x7fc9bc0ca2f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009d90 tx=0x7fc9b402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a535bd6041739467 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9bbe00640 1 -- 172.21.15.39:0/4126246865 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b402cc60 con 0x7fc9bc0b3f20 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9bbe00640 1 -- 172.21.15.39:0/4126246865 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b402cdc0 con 0x7fc9bc0b3f20 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9bbe00640 1 -- 172.21.15.39:0/4126246865 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4034660 con 0x7fc9bc0b3f20 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9c4200640 1 -- 172.21.15.39:0/4126246865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc0b3f20 msgr2=0x7fc9bc0ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9c4200640 1 --2- 172.21.15.39:0/4126246865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc0b3f20 0x7fc9bc0ca2f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009d90 tx=0x7fc9b402beb0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9c4200640 1 -- 172.21.15.39:0/4126246865 shutdown_connections 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9c4200640 1 --2- 172.21.15.39:0/4126246865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc0b3f20 0x7fc9bc0ca2f0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9c4200640 1 --2- 172.21.15.39:0/4126246865 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.225+0000 7fc9c4200640 1 -- 172.21.15.39:0/4126246865 >> 172.21.15.39:0/4126246865 conn(0x7fc9bc06bf20 msgr2=0x7fc9bc06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.226+0000 7fc9c4200640 1 -- 172.21.15.39:0/4126246865 shutdown_connections 2024-01-24T17:47:08.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.226+0000 7fc9c4200640 1 -- 172.21.15.39:0/4126246865 wait complete. 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.227+0000 7fc9c4200640 1 Processor -- start 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.227+0000 7fc9c4200640 1 -- start start 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.228+0000 7fc9c4200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc1f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.228+0000 7fc9c4200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9bc0b3f20 0x7fc9bc0c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.228+0000 7fc9c4200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc159d00 0x7fc9bc0c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.228+0000 7fc9c4200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0c7a20 con 0x7fc9bc159d00 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.228+0000 7fc9c4200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc1f4560 con 0x7fc9bc0b3f20 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.228+0000 7fc9c4200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0c8900 con 0x7fc9bc071c00 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c2200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc159d00 0x7fc9bc0c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c2200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc159d00 0x7fc9bc0c6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44790/0 (socket says 172.21.15.39:44790) 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c2200640 1 -- 172.21.15.39:0/1182170299 learned_addr learned my addr 172.21.15.39:0/1182170299 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c1800640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc1f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c0e00640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9bc0b3f20 0x7fc9bc0c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c2200640 1 -- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 msgr2=0x7fc9bc1f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c2200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc1f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c2200640 1 -- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9bc0b3f20 msgr2=0x7fc9bc0c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c2200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9bc0b3f20 0x7fc9bc0c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.229+0000 7fc9c2200640 1 -- 172.21.15.39:0/1182170299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4009a70 con 0x7fc9bc159d00 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.230+0000 7fc9c2200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc159d00 0x7fc9bc0c6700 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009d90 tx=0x7fc9b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.230+0000 7fc9b9600640 1 -- 172.21.15.39:0/1182170299 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b403e610 con 0x7fc9bc159d00 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.230+0000 7fc9b9600640 1 -- 172.21.15.39:0/1182170299 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b403e770 con 0x7fc9bc159d00 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.230+0000 7fc9b9600640 1 -- 172.21.15.39:0/1182170299 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b403eb20 con 0x7fc9bc159d00 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.230+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc0c8620 con 0x7fc9bc159d00 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.230+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc1f15f0 con 0x7fc9bc159d00 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.231+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc04dff0 con 0x7fc9bc159d00 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.232+0000 7fc9b9600640 1 -- 172.21.15.39:0/1182170299 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc9b403b050 con 0x7fc9bc159d00 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.233+0000 7fc9b9600640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc99406a430 0x7fc99406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.234+0000 7fc9b9600640 1 -- 172.21.15.39:0/1182170299 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9b40ab3e0 con 0x7fc9bc159d00 2024-01-24T17:47:08.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.234+0000 7fc9c1800640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc99406a430 0x7fc99406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.235+0000 7fc9c1800640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc99406a430 0x7fc99406c8f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0011e0 tx=0x7fc9ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:08.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.238+0000 7fc9b9600640 1 -- 172.21.15.39:0/1182170299 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc9b4076ae0 con 0x7fc9bc159d00 2024-01-24T17:47:08.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.433+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9bc06ffb0 con 0x7fc9bc159d00 2024-01-24T17:47:08.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.437+0000 7fc9b9600640 1 -- 172.21.15.39:0/1182170299 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16040 (secure 0 0 0) 0x7fc9b4009450 con 0x7fc9bc159d00 2024-01-24T17:47:08.439 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:08.439 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":76,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:32.019249+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc99406a430 msgr2=0x7fc99406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc99406a430 0x7fc99406c8f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0011e0 tx=0x7fc9ac009040 comp rx=0 tx=0).stop 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc159d00 msgr2=0x7fc9bc0c6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc159d00 0x7fc9bc0c6700 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009d90 tx=0x7fc9b4004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 shutdown_connections 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc99406a430 0x7fc99406c8f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9bc159d00 0x7fc9bc0c6700 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc9bc0b3f20 0x7fc9bc0c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 --2- 172.21.15.39:0/1182170299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc9bc071c00 0x7fc9bc1f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 >> 172.21.15.39:0/1182170299 conn(0x7fc9bc06bf20 msgr2=0x7fc9bc0c20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 shutdown_connections 2024-01-24T17:47:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.440+0000 7fc9c4200640 1 -- 172.21.15.39:0/1182170299 wait complete. 2024-01-24T17:47:08.462 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:46:29.228451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:46:30.039417+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:47:08.462 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:47:08.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.681+0000 7fe219200640 1 Processor -- start 2024-01-24T17:47:08.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.681+0000 7fe219200640 1 -- start start 2024-01-24T17:47:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.682+0000 7fe219200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe21406a2e0 0x7fe2140cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.682+0000 7fe219200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140ae5b0 0x7fe2140c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.682+0000 7fe219200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21406e550 con 0x7fe21406e170 2024-01-24T17:47:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.682+0000 7fe219200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2140ca030 con 0x7fe21406a2e0 2024-01-24T17:47:08.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.682+0000 7fe219200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21415dbb0 con 0x7fe2140ae5b0 2024-01-24T17:47:08.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.683+0000 7fe213400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe21406e170 0x7fe214069bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49786/0 (socket says 172.21.15.39:49786) 2024-01-24T17:47:08.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.683+0000 7fe213400640 1 -- 172.21.15.39:0/1034176453 learned_addr learned my addr 172.21.15.39:0/1034176453 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:08.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.683+0000 7fe212a00640 1 --2- 172.21.15.39:0/1034176453 >> v2:172.21.15.100:3300/0 conn(0x7fe21406a2e0 0x7fe2140cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.683+0000 7fe213e00640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140ae5b0 0x7fe2140c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.683+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930468787 0 0) 0x7fe21406e550 con 0x7fe21406e170 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.683+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f4003520 con 0x7fe21406e170 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3988626102 0 0) 0x7fe1f4003520 con 0x7fe21406e170 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140ae5b0 msgr2=0x7fe2140c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140ae5b0 0x7fe2140c9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 >> v2:172.21.15.100:3300/0 conn(0x7fe21406a2e0 msgr2=0x7fe2140cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 --2- 172.21.15.39:0/1034176453 >> v2:172.21.15.100:3300/0 conn(0x7fe21406a2e0 0x7fe2140cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2140ce630 con 0x7fe21406e170 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fe204001670 con 0x7fe21406e170 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 >> v1:172.21.15.39:6789/0 conn(0x7fe21406e170 legacy=0x7fe214069bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140ae5b0 0x7fe1f4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f4003f40 0x7fe1f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4004d00 0x7fe1f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140ca030 con 0x7fe1f4003f40 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21415dbb0 con 0x7fe1f4004d00 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f400b980 con 0x7fe2140ae5b0 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.684+0000 7fe212a00640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f4003f40 0x7fe1f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.685+0000 7fe213400640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140ae5b0 0x7fe1f4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.685+0000 7fe212a00640 1 -- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140ae5b0 msgr2=0x7fe1f4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.685+0000 7fe212a00640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140ae5b0 0x7fe1f4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.685+0000 7fe212a00640 1 -- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4004d00 msgr2=0x7fe1f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.685+0000 7fe212a00640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4004d00 0x7fe1f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.685+0000 7fe212a00640 1 -- 172.21.15.39:0/1034176453 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f4003520 con 0x7fe1f4003f40 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.686+0000 7fe212a00640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f4003f40 0x7fe1f40047c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009af0 tx=0x7fe1f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10fec1fb4711201 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.686+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc02c710 con 0x7fe1f4003f40 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.686+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc02c870 con 0x7fe1f4003f40 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.686+0000 7fe212000640 1 -- 172.21.15.39:0/1034176453 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc02cc00 con 0x7fe1f4003f40 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.687+0000 7fe219200640 1 -- 172.21.15.39:0/1034176453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f4003f40 msgr2=0x7fe1f40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.687+0000 7fe219200640 1 --2- 172.21.15.39:0/1034176453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f4003f40 0x7fe1f40047c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009af0 tx=0x7fe1f40057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.687+0000 7fe213e00640 1 -- 172.21.15.39:0/1034176453 reap_dead start 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.687+0000 7fe219200640 1 -- 172.21.15.39:0/1034176453 shutdown_connections 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.687+0000 7fe219200640 1 -- 172.21.15.39:0/1034176453 >> 172.21.15.39:0/1034176453 conn(0x7fe2140cf4a0 msgr2=0x7fe2140cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.687+0000 7fe219200640 1 -- 172.21.15.39:0/1034176453 shutdown_connections 2024-01-24T17:47:08.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.687+0000 7fe219200640 1 -- 172.21.15.39:0/1034176453 wait complete. 2024-01-24T17:47:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.689+0000 7fe219200640 1 Processor -- start 2024-01-24T17:47:08.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.689+0000 7fe219200640 1 -- start start 2024-01-24T17:47:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe219200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4003f40 0x7fe21414a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe219200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2140cd1b0 0x7fe2140cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe219200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140acd30 0x7fe2140ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe219200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140ad720 con 0x7fe2140cd1b0 2024-01-24T17:47:08.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe219200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140c9420 con 0x7fe1f4003f40 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe219200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140c87a0 con 0x7fe2140acd30 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe212a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2140cd1b0 0x7fe2140cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe212a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2140cd1b0 0x7fe2140cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44812/0 (socket says 172.21.15.39:44812) 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.690+0000 7fe212a00640 1 -- 172.21.15.39:0/3000264718 learned_addr learned my addr 172.21.15.39:0/3000264718 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe213400640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4003f40 0x7fe21414a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe212a00640 1 -- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140acd30 msgr2=0x7fe2140ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe213e00640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140acd30 0x7fe2140ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe212a00640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140acd30 0x7fe2140ad1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe212a00640 1 -- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4003f40 msgr2=0x7fe21414a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe212a00640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4003f40 0x7fe21414a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe212a00640 1 -- 172.21.15.39:0/3000264718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2140c8a80 con 0x7fe2140cd1b0 2024-01-24T17:47:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe213400640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4003f40 0x7fe21414a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe212a00640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2140cd1b0 0x7fe2140cd630 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009af0 tx=0x7fe1fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.691+0000 7fe20b400640 1 -- 172.21.15.39:0/3000264718 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc003810 con 0x7fe2140cd1b0 2024-01-24T17:47:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.692+0000 7fe20b400640 1 -- 172.21.15.39:0/3000264718 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc02de90 con 0x7fe2140cd1b0 2024-01-24T17:47:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.692+0000 7fe20b400640 1 -- 172.21.15.39:0/3000264718 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc003b70 con 0x7fe2140cd1b0 2024-01-24T17:47:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.692+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2140c8d60 con 0x7fe2140cd1b0 2024-01-24T17:47:08.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.692+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21414ad00 con 0x7fe2140cd1b0 2024-01-24T17:47:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.693+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21404e840 con 0x7fe2140cd1b0 2024-01-24T17:47:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.694+0000 7fe20b400640 1 -- 172.21.15.39:0/3000264718 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe1fc003cd0 con 0x7fe2140cd1b0 2024-01-24T17:47:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.695+0000 7fe20b400640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1ec06a4b0 0x7fe1ec06c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:08.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.695+0000 7fe20b400640 1 -- 172.21.15.39:0/3000264718 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1fc0aa7c0 con 0x7fe2140cd1b0 2024-01-24T17:47:08.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.695+0000 7fe213400640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1ec06a4b0 0x7fe1ec06c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:08.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.696+0000 7fe213400640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1ec06a4b0 0x7fe1ec06c970 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe204001eb0 tx=0x7fe204000f70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:08.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.699+0000 7fe20b400640 1 -- 172.21.15.39:0/3000264718 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe1fc075ed0 con 0x7fe2140cd1b0 2024-01-24T17:47:08.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.893+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2140b5330 con 0x7fe2140cd1b0 2024-01-24T17:47:08.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.895+0000 7fe20b400640 1 -- 172.21.15.39:0/3000264718 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe1fc0794f0 con 0x7fe2140cd1b0 2024-01-24T17:47:08.896 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:47:08.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.897+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1ec06a4b0 msgr2=0x7fe1ec06c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.897+0000 7fe219200640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1ec06a4b0 0x7fe1ec06c970 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe204001eb0 tx=0x7fe204000f70 comp rx=0 tx=0).stop 2024-01-24T17:47:08.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.897+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2140cd1b0 msgr2=0x7fe2140cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:08.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.897+0000 7fe219200640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2140cd1b0 0x7fe2140cd630 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009af0 tx=0x7fe1fc004730 comp rx=0 tx=0).stop 2024-01-24T17:47:08.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.898+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 shutdown_connections 2024-01-24T17:47:08.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.898+0000 7fe219200640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2140acd30 0x7fe2140ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.898+0000 7fe219200640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1ec06a4b0 0x7fe1ec06c970 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.898+0000 7fe219200640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2140cd1b0 0x7fe2140cd630 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.898+0000 7fe219200640 1 --2- 172.21.15.39:0/3000264718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe1f4003f40 0x7fe21414a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:08.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.898+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 >> 172.21.15.39:0/3000264718 conn(0x7fe2140cf4a0 msgr2=0x7fe21415c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:08.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.898+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 shutdown_connections 2024-01-24T17:47:08.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:08.898+0000 7fe219200640 1 -- 172.21.15.39:0/3000264718 wait complete. 2024-01-24T17:47:08.919 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:47:09.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.138+0000 7f0442a00640 1 Processor -- start 2024-01-24T17:47:09.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.138+0000 7f0442a00640 1 -- start start 2024-01-24T17:47:09.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.139+0000 7f0442a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f043c15aa60 0x7f043c0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0442a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 0x7f043c0cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0442a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f043c06cec0 0x7f043c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0442a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c0cae90 con 0x7f043c06cec0 2024-01-24T17:47:09.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0442a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c0cbc20 con 0x7f043c15aa60 2024-01-24T17:47:09.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0442a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0ca4c0 con 0x7f043c0cb280 2024-01-24T17:47:09.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0440a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f043c06cec0 0x7f043c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0440a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f043c06cec0 0x7f043c06d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44820/0 (socket says 172.21.15.39:44820) 2024-01-24T17:47:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0440a00640 1 -- 172.21.15.39:0/3367502134 learned_addr learned my addr 172.21.15.39:0/3367502134 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f0433200640 1 --2- 172.21.15.39:0/3367502134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 0x7f043c0cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.140+0000 7f043be00640 1 --2- 172.21.15.39:0/3367502134 >> v2:172.21.15.100:3300/0 conn(0x7f043c15aa60 0x7f043c0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.141+0000 7f0440a00640 1 -- 172.21.15.39:0/3367502134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 msgr2=0x7f043c0cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.141+0000 7f0440a00640 1 --2- 172.21.15.39:0/3367502134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 0x7f043c0cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.141+0000 7f0440a00640 1 -- 172.21.15.39:0/3367502134 >> v2:172.21.15.100:3300/0 conn(0x7f043c15aa60 msgr2=0x7f043c0ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.141+0000 7f0440a00640 1 --2- 172.21.15.39:0/3367502134 >> v2:172.21.15.100:3300/0 conn(0x7f043c15aa60 0x7f043c0ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.141+0000 7f0440a00640 1 -- 172.21.15.39:0/3367502134 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c06dc90 con 0x7f043c06cec0 2024-01-24T17:47:09.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.142+0000 7f0440a00640 1 --2- 172.21.15.39:0/3367502134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c06cec0 0x7f043c06d330 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f043c06c190 tx=0x7f043402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef27ddd0883291d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:09.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.142+0000 7f043b400640 1 -- 172.21.15.39:0/3367502134 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434031640 con 0x7f043c06cec0 2024-01-24T17:47:09.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.142+0000 7f043b400640 1 -- 172.21.15.39:0/3367502134 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04340317a0 con 0x7f043c06cec0 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.142+0000 7f043b400640 1 -- 172.21.15.39:0/3367502134 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434031b30 con 0x7f043c06cec0 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 -- 172.21.15.39:0/3367502134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c06cec0 msgr2=0x7f043c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 --2- 172.21.15.39:0/3367502134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c06cec0 0x7f043c06d330 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f043c06c190 tx=0x7f043402b8f0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 -- 172.21.15.39:0/3367502134 shutdown_connections 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 --2- 172.21.15.39:0/3367502134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c06cec0 0x7f043c06d330 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 --2- 172.21.15.39:0/3367502134 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 0x7f043c0cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 --2- 172.21.15.39:0/3367502134 >> v2:172.21.15.100:3300/0 conn(0x7f043c15aa60 0x7f043c0ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 -- 172.21.15.39:0/3367502134 >> 172.21.15.39:0/3367502134 conn(0x7f043c0ce9e0 msgr2=0x7f043c0cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 -- 172.21.15.39:0/3367502134 shutdown_connections 2024-01-24T17:47:09.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.143+0000 7f0442a00640 1 -- 172.21.15.39:0/3367502134 wait complete. 2024-01-24T17:47:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.144+0000 7f0442a00640 1 Processor -- start 2024-01-24T17:47:09.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.145+0000 7f0442a00640 1 -- start start 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.145+0000 7f0442a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c06cec0 0x7f043c0c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0442a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 0x7f043c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0442a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c15aa60 0x7f043c15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0442a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0c6ab0 con 0x7f043c15aa60 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0442a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0c6d50 con 0x7f043c06cec0 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0442a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0c7030 con 0x7f043c0cb280 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0440a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c15aa60 0x7f043c15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0440a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c15aa60 0x7f043c15b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44822/0 (socket says 172.21.15.39:44822) 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0440a00640 1 -- 172.21.15.39:0/1474711360 learned_addr learned my addr 172.21.15.39:0/1474711360 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:09.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0440a00640 1 -- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 msgr2=0x7f043c153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f043be00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c06cec0 0x7f043c0c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0440a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 0x7f043c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0440a00640 1 -- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c06cec0 msgr2=0x7f043c0c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0440a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c06cec0 0x7f043c0c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.146+0000 7f0440a00640 1 -- 172.21.15.39:0/1474711360 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04340094d0 con 0x7f043c15aa60 2024-01-24T17:47:09.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.147+0000 7f0440a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c15aa60 0x7f043c15b8e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f0434009870 tx=0x7f04340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:09.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.147+0000 7f0438c00640 1 -- 172.21.15.39:0/1474711360 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043403ac60 con 0x7f043c15aa60 2024-01-24T17:47:09.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.147+0000 7f0438c00640 1 -- 172.21.15.39:0/1474711360 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043403ade0 con 0x7f043c15aa60 2024-01-24T17:47:09.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.147+0000 7f0438c00640 1 -- 172.21.15.39:0/1474711360 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434044630 con 0x7f043c15aa60 2024-01-24T17:47:09.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.147+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c0c7310 con 0x7f043c15aa60 2024-01-24T17:47:09.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.147+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c154900 con 0x7f043c15aa60 2024-01-24T17:47:09.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.148+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043c0c7830 con 0x7f043c15aa60 2024-01-24T17:47:09.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.149+0000 7f0438c00640 1 -- 172.21.15.39:0/1474711360 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f043402f050 con 0x7f043c15aa60 2024-01-24T17:47:09.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.150+0000 7f0438c00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.150+0000 7f0438c00640 1 -- 172.21.15.39:0/1474711360 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04340aa830 con 0x7f043c15aa60 2024-01-24T17:47:09.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.151+0000 7f043be00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.152+0000 7f043be00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f042c009eb0 tx=0x7f042c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:09.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.154+0000 7f0438c00640 1 -- 172.21.15.39:0/1474711360 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f043c0c7830 con 0x7f043c15aa60 2024-01-24T17:47:09.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.349+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f043c0c7830 con 0x7f043c15aa60 2024-01-24T17:47:09.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.350+0000 7f0438c00640 1 -- 172.21.15.39:0/1474711360 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7f043c0c7830 con 0x7f043c15aa60 2024-01-24T17:47:09.352 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:47:09.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.352+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 msgr2=0x7f041406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.352+0000 7f0442a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f042c009eb0 tx=0x7f042c00b040 comp rx=0 tx=0).stop 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.352+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c15aa60 msgr2=0x7f043c15b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.352+0000 7f0442a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c15aa60 0x7f043c15b8e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f0434009870 tx=0x7f04340046c0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.353+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 shutdown_connections 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.353+0000 7f0442a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.353+0000 7f0442a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c15aa60 0x7f043c15b8e0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.353+0000 7f0442a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c0cb280 0x7f043c153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.353+0000 7f0442a00640 1 --2- 172.21.15.39:0/1474711360 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c06cec0 0x7f043c0c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.353+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 >> 172.21.15.39:0/1474711360 conn(0x7f043c0ce9e0 msgr2=0x7f043c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.353+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 shutdown_connections 2024-01-24T17:47:09.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.353+0000 7f0442a00640 1 -- 172.21.15.39:0/1474711360 wait complete. 2024-01-24T17:47:09.372 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:47:09.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.590+0000 7fcbd7e00640 1 Processor -- start 2024-01-24T17:47:09.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.590+0000 7fcbd7e00640 1 -- start start 2024-01-24T17:47:09.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 0x7fcbd8070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcbd8071e40 0x7fcbd80cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcbd80cdb90 0x7fcbd80cab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd80cbf90 con 0x7fcbd80cdb90 2024-01-24T17:47:09.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd80cc270 con 0x7fcbd8071e40 2024-01-24T17:47:09.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80cc550 con 0x7fcbd8159530 2024-01-24T17:47:09.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcbd80cdb90 0x7fcbd80cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcbd80cdb90 0x7fcbd80cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44836/0 (socket says 172.21.15.39:44836) 2024-01-24T17:47:09.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7400640 1 -- 172.21.15.39:0/535111609 learned_addr learned my addr 172.21.15.39:0/535111609 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:09.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd6000640 1 --2- 172.21.15.39:0/535111609 >> v2:172.21.15.100:3300/0 conn(0x7fcbd8071e40 0x7fcbd80cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.592+0000 7fcbd7400640 1 -- 172.21.15.39:0/535111609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 msgr2=0x7fcbd8070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:09.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.593+0000 7fcbd6a00640 1 --2- 172.21.15.39:0/535111609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 0x7fcbd8070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.593+0000 7fcbd7400640 1 --2- 172.21.15.39:0/535111609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 0x7fcbd8070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.593+0000 7fcbd7400640 1 -- 172.21.15.39:0/535111609 >> v2:172.21.15.100:3300/0 conn(0x7fcbd8071e40 msgr2=0x7fcbd80cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:09.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.593+0000 7fcbd7400640 1 --2- 172.21.15.39:0/535111609 >> v2:172.21.15.100:3300/0 conn(0x7fcbd8071e40 0x7fcbd80cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.593+0000 7fcbd7400640 1 -- 172.21.15.39:0/535111609 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd80cc830 con 0x7fcbd80cdb90 2024-01-24T17:47:09.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.593+0000 7fcbd6000640 1 --2- 172.21.15.39:0/535111609 >> v2:172.21.15.100:3300/0 conn(0x7fcbd8071e40 0x7fcbd80cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:09.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.593+0000 7fcbd6a00640 1 --2- 172.21.15.39:0/535111609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 0x7fcbd8070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:09.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.594+0000 7fcbd7400640 1 --2- 172.21.15.39:0/535111609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80cab40 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fcbd806df00 tx=0x7fcbc802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932d17ac1e283e3a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:09.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.594+0000 7fcbd5600640 1 -- 172.21.15.39:0/535111609 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8031860 con 0x7fcbd80cdb90 2024-01-24T17:47:09.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.594+0000 7fcbd5600640 1 -- 172.21.15.39:0/535111609 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc80319c0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.594+0000 7fcbd5600640 1 -- 172.21.15.39:0/535111609 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8031d50 con 0x7fcbd80cdb90 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 -- 172.21.15.39:0/535111609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 msgr2=0x7fcbd80cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/535111609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80cab40 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fcbd806df00 tx=0x7fcbc802b900 comp rx=0 tx=0).stop 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 -- 172.21.15.39:0/535111609 shutdown_connections 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/535111609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80cab40 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/535111609 >> v2:172.21.15.100:3300/0 conn(0x7fcbd8071e40 0x7fcbd80cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/535111609 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 0x7fcbd8070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 -- 172.21.15.39:0/535111609 >> 172.21.15.39:0/535111609 conn(0x7fcbd806a200 msgr2=0x7fcbd80affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 -- 172.21.15.39:0/535111609 shutdown_connections 2024-01-24T17:47:09.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.595+0000 7fcbd7e00640 1 -- 172.21.15.39:0/535111609 wait complete. 2024-01-24T17:47:09.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.597+0000 7fcbd7e00640 1 Processor -- start 2024-01-24T17:47:09.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.597+0000 7fcbd7e00640 1 -- start start 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 0x7fcbd80caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbd80ac440 0x7fcbd80c06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8154d50 con 0x7fcbd80cdb90 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd7e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd8154fa0 con 0x7fcbd80ac440 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd7e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80ab960 con 0x7fcbd8159530 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80ad0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44844/0 (socket says 172.21.15.39:44844) 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.598+0000 7fcbd6a00640 1 -- 172.21.15.39:0/939540090 learned_addr learned my addr 172.21.15.39:0/939540090 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbd6a00640 1 -- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 msgr2=0x7fcbd80caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbd6a00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 0x7fcbd80caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbd6a00640 1 -- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbd80ac440 msgr2=0x7fcbd80c06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbd6a00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbd80ac440 0x7fcbd80c06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbd6a00640 1 -- 172.21.15.39:0/939540090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc80094d0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbd6a00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80ad0c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000ce00 tx=0x7fcbd000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbcf400640 1 -- 172.21.15.39:0/939540090 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000f890 con 0x7fcbd80cdb90 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbcf400640 1 -- 172.21.15.39:0/939540090 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd000f9f0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd80abc40 con 0x7fcbd80cdb90 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbcf400640 1 -- 172.21.15.39:0/939540090 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd000fdc0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.599+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd80c0be0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.600+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd80738c0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.601+0000 7fcbcf400640 1 -- 172.21.15.39:0/939540090 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcbd000b040 con 0x7fcbd80cdb90 2024-01-24T17:47:09.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.602+0000 7fcbcf400640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcbb006a430 0x7fcbb006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:09.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.602+0000 7fcbcf400640 1 -- 172.21.15.39:0/939540090 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcbd00893b0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.603+0000 7fcbd6000640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcbb006a430 0x7fcbb006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:09.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.604+0000 7fcbd6000640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcbb006a430 0x7fcbb006c8f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0006fd0 tx=0x7fcbc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:09.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.607+0000 7fcbcf400640 1 -- 172.21.15.39:0/939540090 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcbd00549b0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.802+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcbd804e840 con 0x7fcbd80cdb90 2024-01-24T17:47:09.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.803+0000 7fcbcf400640 1 -- 172.21.15.39:0/939540090 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcbd0057fd0 con 0x7fcbd80cdb90 2024-01-24T17:47:09.805 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:47:09.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.805+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcbb006a430 msgr2=0x7fcbb006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.805+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcbb006a430 0x7fcbb006c8f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0006fd0 tx=0x7fcbc0008040 comp rx=0 tx=0).stop 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.805+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 msgr2=0x7fcbd80ad0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.805+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80ad0c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000ce00 tx=0x7fcbd000a650 comp rx=0 tx=0).stop 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.806+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 shutdown_connections 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.806+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcbd80ac440 0x7fcbd80c06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.806+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcbd8159530 0x7fcbd80caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.806+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcbb006a430 0x7fcbb006c8f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.806+0000 7fcbd7e00640 1 --2- 172.21.15.39:0/939540090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd80cdb90 0x7fcbd80ad0c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.806+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 >> 172.21.15.39:0/939540090 conn(0x7fcbd806a200 msgr2=0x7fcbd80c1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.806+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 shutdown_connections 2024-01-24T17:47:09.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:09.806+0000 7fcbd7e00640 1 -- 172.21.15.39:0/939540090 wait complete. 2024-01-24T17:47:09.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:47:10.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.021+0000 7f2fa5a00640 1 Processor -- start 2024-01-24T17:47:10.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.021+0000 7f2fa5a00640 1 -- start start 2024-01-24T17:47:10.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.023+0000 7f2fa5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca320 0x7f2fa015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.023+0000 7f2fa5a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2fa00cdaf0 0x7f2fa006bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.023+0000 7f2fa5a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa01540f0 con 0x7f2fa00ca700 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.023+0000 7f2fa5a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa0159fd0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.023+0000 7f2fa5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa015da00 con 0x7f2fa00ca320 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.023+0000 7f2fa5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2fa00cdaf0 0x7f2fa006bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.023+0000 7f2fa5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2fa00cdaf0 0x7f2fa006bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44854/0 (socket says 172.21.15.39:44854) 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.023+0000 7f2fa5000640 1 -- 172.21.15.39:0/1090287429 learned_addr learned my addr 172.21.15.39:0/1090287429 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.024+0000 7f2f9fe00640 1 --2- 172.21.15.39:0/1090287429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca320 0x7f2fa015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.024+0000 7f2fa5000640 1 -- 172.21.15.39:0/1090287429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca320 msgr2=0x7f2fa015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.024+0000 7f2fa5000640 1 --2- 172.21.15.39:0/1090287429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca320 0x7f2fa015d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.024+0000 7f2fa5000640 1 -- 172.21.15.39:0/1090287429 >> v1:172.21.15.100:6789/0 conn(0x7f2fa00ca700 legacy=0x7f2fa015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.024+0000 7f2fa5000640 1 -- 172.21.15.39:0/1090287429 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa015dce0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.025+0000 7f2fa5000640 1 --2- 172.21.15.39:0/1090287429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa006bda0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009e50 tx=0x7f2f9802dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2ea6bb0ed1fa8a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.025+0000 7f2f9ea00640 1 -- 172.21.15.39:0/1090287429 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9802eb60 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.025+0000 7f2f9ea00640 1 -- 172.21.15.39:0/1090287429 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9802ecc0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.025+0000 7f2f9ea00640 1 -- 172.21.15.39:0/1090287429 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98036610 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.026+0000 7f2fa5a00640 1 -- 172.21.15.39:0/1090287429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 msgr2=0x7f2fa006bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.026+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/1090287429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa006bda0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009e50 tx=0x7f2f9802dfc0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.026+0000 7f2fa5a00640 1 -- 172.21.15.39:0/1090287429 shutdown_connections 2024-01-24T17:47:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.026+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/1090287429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa006bda0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.026+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/1090287429 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca320 0x7f2fa015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.026+0000 7f2fa5a00640 1 -- 172.21.15.39:0/1090287429 >> 172.21.15.39:0/1090287429 conn(0x7f2fa006a340 msgr2=0x7f2fa00cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.026+0000 7f2fa5a00640 1 -- 172.21.15.39:0/1090287429 shutdown_connections 2024-01-24T17:47:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.026+0000 7f2fa5a00640 1 -- 172.21.15.39:0/1090287429 wait complete. 2024-01-24T17:47:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.028+0000 7f2fa5a00640 1 Processor -- start 2024-01-24T17:47:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.028+0000 7f2fa5a00640 1 -- start start 2024-01-24T17:47:10.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fa00ca320 0x7f2fa0144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca700 0x7f2fa0154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa01f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa00c4c60 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa00c5220 con 0x7f2fa00ca320 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa0141fd0 con 0x7f2fa00ca700 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa01f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa01f0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44856/0 (socket says 172.21.15.39:44856) 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.029+0000 7f2fa5000640 1 -- 172.21.15.39:0/4015172528 learned_addr learned my addr 172.21.15.39:0/4015172528 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2f9fe00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fa00ca320 0x7f2fa0144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2f9f400640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca700 0x7f2fa0154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2fa5000640 1 -- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca700 msgr2=0x7f2fa0154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2fa5000640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca700 0x7f2fa0154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2fa5000640 1 -- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fa00ca320 msgr2=0x7f2fa0144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2fa5000640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fa00ca320 0x7f2fa0144a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2fa5000640 1 -- 172.21.15.39:0/4015172528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f98009b30 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2f9f400640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca700 0x7f2fa0154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2fa5000640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa01f0f10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009e50 tx=0x7f2f98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2f97e00640 1 -- 172.21.15.39:0/4015172528 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98040450 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa00c85c0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2f97e00640 1 -- 172.21.15.39:0/4015172528 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f980405d0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.030+0000 7f2f97e00640 1 -- 172.21.15.39:0/4015172528 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98040980 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.031+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa00c4eb0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.032+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa00c5ea0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.033+0000 7f2f97e00640 1 -- 172.21.15.39:0/4015172528 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2f98040ae0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.034+0000 7f2f97e00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f7806a430 0x7f2f7806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.034+0000 7f2f97e00640 1 -- 172.21.15.39:0/4015172528 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f980ad8c0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.034+0000 7f2f9fe00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f7806a430 0x7f2f7806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.035+0000 7f2f9fe00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f7806a430 0x7f2f7806c8f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2f900096d0 tx=0x7f2f90008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.039+0000 7f2f97e00640 1 -- 172.21.15.39:0/4015172528 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2f98078d90 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.273+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fa015df90 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.274+0000 7f2f97e00640 1 -- 172.21.15.39:0/4015172528 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f2f9807c3b0 con 0x7f2fa00cdaf0 2024-01-24T17:47:10.276 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:10.276 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1886779141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1886779141},{"type":"v1","addr":"172.21.15.39:6833","nonce":1886779141}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5854,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2624477336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":2624477336},{"type":"v1","addr":"172.21.15.100:6837","nonce":2624477336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2465319794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2465319794},{"type":"v1","addr":"172.21.15.39:6835","nonce":2465319794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:46:31.015014+0000","modified":"2024-01-24T17:46:34.537399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6836/2467066037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2467066037},{"type":"v1","addr":"172.21.15.100:6836","nonce":2467066037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-24T17:47:10.277 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 42 2024-01-24T17:47:10.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f7806a430 msgr2=0x7f2f7806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f7806a430 0x7f2f7806c8f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2f900096d0 tx=0x7f2f90008040 comp rx=0 tx=0).stop 2024-01-24T17:47:10.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 msgr2=0x7f2fa01f0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa01f0f10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009e50 tx=0x7f2f98004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:10.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 shutdown_connections 2024-01-24T17:47:10.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2f7806a430 0x7f2f7806c8f0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fa00cdaf0 0x7f2fa01f0f10 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2fa00ca700 0x7f2fa0154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.277+0000 7f2fa5a00640 1 --2- 172.21.15.39:0/4015172528 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2fa00ca320 0x7f2fa0144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.278+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 >> 172.21.15.39:0/4015172528 conn(0x7f2fa006a340 msgr2=0x7f2fa00cad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:10.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.278+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 shutdown_connections 2024-01-24T17:47:10.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.278+0000 7f2fa5a00640 1 -- 172.21.15.39:0/4015172528 wait complete. 2024-01-24T17:47:10.299 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:10.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.518+0000 7f1e4b200640 1 Processor -- start 2024-01-24T17:47:10.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.519+0000 7f1e4b200640 1 -- start start 2024-01-24T17:47:10.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e4b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e440d3310 0x7f1e440b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e4b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e440ce2f0 con 0x7f1e440d05d0 2024-01-24T17:47:10.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e4b200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e440cd440 con 0x7f1e440cf200 2024-01-24T17:47:10.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e4b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e440d37a0 con 0x7f1e440d3310 2024-01-24T17:47:10.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e49400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1e440d05d0 0x7f1e440b1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49798/0 (socket says 172.21.15.39:49798) 2024-01-24T17:47:10.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e49400640 1 -- 172.21.15.39:0/128778325 learned_addr learned my addr 172.21.15.39:0/128778325 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e4a800640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e440d3310 0x7f1e440b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748925428 0 0) 0x7f1e440ce2f0 con 0x7f1e440d05d0 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.521+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e24003520 con 0x7f1e440d05d0 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888349617 0 0) 0x7f1e24003520 con 0x7f1e440d05d0 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e440d3310 msgr2=0x7f1e440b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e440d3310 0x7f1e440b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 >> v1:172.21.15.100:6789/0 conn(0x7f1e440cf200 legacy=0x7f1e440cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e440ce780 con 0x7f1e440d05d0 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1e2c001670 con 0x7f1e440d05d0 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 >> v1:172.21.15.39:6789/0 conn(0x7f1e440d05d0 legacy=0x7f1e440b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e240039a0 0x7f1e24003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240047c0 0x7f1e24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e440d3310 0x7f1e24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e440d37a0 con 0x7f1e240039a0 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2400baf0 con 0x7f1e240047c0 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2400bcb0 con 0x7f1e440d3310 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.522+0000 7f1e49e00640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e240039a0 0x7f1e24003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.523+0000 7f1e4a800640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e440d3310 0x7f1e24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.523+0000 7f1e49400640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240047c0 0x7f1e24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.523+0000 7f1e49e00640 1 -- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e440d3310 msgr2=0x7f1e24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.523+0000 7f1e49e00640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e440d3310 0x7f1e24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.523+0000 7f1e49e00640 1 -- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240047c0 msgr2=0x7f1e24004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.523+0000 7f1e49e00640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240047c0 0x7f1e24004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.523+0000 7f1e49e00640 1 -- 172.21.15.39:0/128778325 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e24003520 con 0x7f1e240039a0 2024-01-24T17:47:10.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.524+0000 7f1e49e00640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e240039a0 0x7f1e24003dc0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1e34007fd0 tx=0x7f1e240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28739c6bda76e65d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.524+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e340052d0 con 0x7f1e240039a0 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.524+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3402e050 con 0x7f1e240039a0 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.524+0000 7f1e48a00640 1 -- 172.21.15.39:0/128778325 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e34003640 con 0x7f1e240039a0 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.525+0000 7f1e4b200640 1 -- 172.21.15.39:0/128778325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e240039a0 msgr2=0x7f1e24003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.525+0000 7f1e4b200640 1 --2- 172.21.15.39:0/128778325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e240039a0 0x7f1e24003dc0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1e34007fd0 tx=0x7f1e240057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.525+0000 7f1e4a800640 1 -- 172.21.15.39:0/128778325 reap_dead start 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.525+0000 7f1e4b200640 1 -- 172.21.15.39:0/128778325 shutdown_connections 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.525+0000 7f1e4b200640 1 -- 172.21.15.39:0/128778325 >> 172.21.15.39:0/128778325 conn(0x7f1e440d27f0 msgr2=0x7f1e440d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.525+0000 7f1e4b200640 1 -- 172.21.15.39:0/128778325 shutdown_connections 2024-01-24T17:47:10.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.525+0000 7f1e4b200640 1 -- 172.21.15.39:0/128778325 wait complete. 2024-01-24T17:47:10.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.527+0000 7f1e4b200640 1 Processor -- start 2024-01-24T17:47:10.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.527+0000 7f1e4b200640 1 -- start start 2024-01-24T17:47:10.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.528+0000 7f1e4b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240039a0 0x7f1e44147be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.528+0000 7f1e4b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e44157180 0x7f1e440b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.528+0000 7f1e4b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e440b6320 0x7f1e440b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.528+0000 7f1e4b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e440ca290 con 0x7f1e440b6320 2024-01-24T17:47:10.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.528+0000 7f1e4b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e44148120 con 0x7f1e240039a0 2024-01-24T17:47:10.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.528+0000 7f1e4b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e440c9960 con 0x7f1e44157180 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.528+0000 7f1e4a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e440b6320 0x7f1e440b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e4a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e440b6320 0x7f1e440b70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44882/0 (socket says 172.21.15.39:44882) 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e4a800640 1 -- 172.21.15.39:0/2046366819 learned_addr learned my addr 172.21.15.39:0/2046366819 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e49400640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e44157180 0x7f1e440b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e49e00640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240039a0 0x7f1e44147be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e4a800640 1 -- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e44157180 msgr2=0x7f1e440b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e4a800640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e44157180 0x7f1e440b5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e4a800640 1 -- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240039a0 msgr2=0x7f1e44147be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e4a800640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240039a0 0x7f1e44147be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e4a800640 1 -- 172.21.15.39:0/2046366819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e440c9c40 con 0x7f1e440b6320 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e4a800640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e440b6320 0x7f1e440b70f0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1e440d0b60 tx=0x7f1e3c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.529+0000 7f1e42000640 1 -- 172.21.15.39:0/2046366819 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c00e070 con 0x7f1e440b6320 2024-01-24T17:47:10.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.530+0000 7f1e42000640 1 -- 172.21.15.39:0/2046366819 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3c003990 con 0x7f1e440b6320 2024-01-24T17:47:10.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.530+0000 7f1e42000640 1 -- 172.21.15.39:0/2046366819 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c003b40 con 0x7f1e440b6320 2024-01-24T17:47:10.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.530+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e440c9f20 con 0x7f1e440b6320 2024-01-24T17:47:10.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.530+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e44157600 con 0x7f1e440b6320 2024-01-24T17:47:10.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.531+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4404dff0 con 0x7f1e440b6320 2024-01-24T17:47:10.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.532+0000 7f1e42000640 1 -- 172.21.15.39:0/2046366819 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1e3c003ca0 con 0x7f1e440b6320 2024-01-24T17:47:10.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.533+0000 7f1e42000640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e1c06a430 0x7f1e1c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.533+0000 7f1e42000640 1 -- 172.21.15.39:0/2046366819 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e3c088fe0 con 0x7f1e440b6320 2024-01-24T17:47:10.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.534+0000 7f1e49e00640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e1c06a430 0x7f1e1c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.535+0000 7f1e49e00640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e1c06a430 0x7f1e1c06c8f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1e34033000 tx=0x7f1e34009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.538+0000 7f1e42000640 1 -- 172.21.15.39:0/2046366819 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1e3c054530 con 0x7f1e440b6320 2024-01-24T17:47:10.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.735+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e440d0bb0 con 0x7f1e440b6320 2024-01-24T17:47:10.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.738+0000 7f1e42000640 1 -- 172.21.15.39:0/2046366819 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16040 (secure 0 0 0) 0x7f1e3c057b50 con 0x7f1e440b6320 2024-01-24T17:47:10.740 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:10.740 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":76,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:32.019249+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:10.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.740+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e1c06a430 msgr2=0x7f1e1c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.740+0000 7f1e4b200640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e1c06a430 0x7f1e1c06c8f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1e34033000 tx=0x7f1e34009030 comp rx=0 tx=0).stop 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.740+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e440b6320 msgr2=0x7f1e440b70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.740+0000 7f1e4b200640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e440b6320 0x7f1e440b70f0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1e440d0b60 tx=0x7f1e3c0115a0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.741+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 shutdown_connections 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.741+0000 7f1e4b200640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e1c06a430 0x7f1e1c06c8f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.741+0000 7f1e4b200640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e440b6320 0x7f1e440b70f0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.741+0000 7f1e4b200640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e44157180 0x7f1e440b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.741+0000 7f1e4b200640 1 --2- 172.21.15.39:0/2046366819 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e240039a0 0x7f1e44147be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.741+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 >> 172.21.15.39:0/2046366819 conn(0x7f1e440d27f0 msgr2=0x7f1e44155ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.741+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 shutdown_connections 2024-01-24T17:47:10.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.741+0000 7f1e4b200640 1 -- 172.21.15.39:0/2046366819 wait complete. 2024-01-24T17:47:10.761 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-24T17:47:10.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:47:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.980+0000 7f41d1e00640 1 Processor -- start 2024-01-24T17:47:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.980+0000 7f41d1e00640 1 -- start start 2024-01-24T17:47:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.982+0000 7f41d1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc15bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.982+0000 7f41d1e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f41cc070920 0x7f41cc0b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.982+0000 7f41d1e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41cc070470 con 0x7f41cc15f390 2024-01-24T17:47:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.982+0000 7f41d1e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41cc15ef00 con 0x7f41cc070920 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.982+0000 7f41d1e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc0710a0 con 0x7f41cc0b0010 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.982+0000 7f41ca000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f41cc070920 0x7f41cc0b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.983+0000 7f41ca000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f41cc070920 0x7f41cc0b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44884/0 (socket says 172.21.15.39:44884) 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.983+0000 7f41ca000640 1 -- 172.21.15.39:0/3549996583 learned_addr learned my addr 172.21.15.39:0/3549996583 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.983+0000 7f41caa00640 1 --2- 172.21.15.39:0/3549996583 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc15bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.983+0000 7f41ca000640 1 -- 172.21.15.39:0/3549996583 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 msgr2=0x7f41cc15bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.983+0000 7f41ca000640 1 --2- 172.21.15.39:0/3549996583 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc15bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.983+0000 7f41ca000640 1 -- 172.21.15.39:0/3549996583 >> v1:172.21.15.100:6789/0 conn(0x7f41cc15f390 legacy=0x7f41cc0cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.983+0000 7f41ca000640 1 -- 172.21.15.39:0/3549996583 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc15c0d0 con 0x7f41cc070920 2024-01-24T17:47:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.984+0000 7f41ca000640 1 --2- 172.21.15.39:0/3549996583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc0b03f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009870 tx=0x7f41b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6f4cc64278e465 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41c9600640 1 -- 172.21.15.39:0/3549996583 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4030840 con 0x7f41cc070920 2024-01-24T17:47:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41c9600640 1 -- 172.21.15.39:0/3549996583 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b40309a0 con 0x7f41cc070920 2024-01-24T17:47:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41c9600640 1 -- 172.21.15.39:0/3549996583 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4030d30 con 0x7f41cc070920 2024-01-24T17:47:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41d1e00640 1 -- 172.21.15.39:0/3549996583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 msgr2=0x7f41cc0b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41d1e00640 1 --2- 172.21.15.39:0/3549996583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc0b03f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009870 tx=0x7f41b402b980 comp rx=0 tx=0).stop 2024-01-24T17:47:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41d1e00640 1 -- 172.21.15.39:0/3549996583 shutdown_connections 2024-01-24T17:47:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41d1e00640 1 --2- 172.21.15.39:0/3549996583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc0b03f0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41d1e00640 1 --2- 172.21.15.39:0/3549996583 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc15bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41d1e00640 1 -- 172.21.15.39:0/3549996583 >> 172.21.15.39:0/3549996583 conn(0x7f41cc06bf20 msgr2=0x7f41cc06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41d1e00640 1 -- 172.21.15.39:0/3549996583 shutdown_connections 2024-01-24T17:47:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.985+0000 7f41d1e00640 1 -- 172.21.15.39:0/3549996583 wait complete. 2024-01-24T17:47:10.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.987+0000 7f41d1e00640 1 Processor -- start 2024-01-24T17:47:10.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.987+0000 7f41d1e00640 1 -- start start 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.988+0000 7f41d1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc1527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.988+0000 7f41d1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.988+0000 7f41d1e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41cc15f390 0x7f41cc158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.988+0000 7f41d1e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc159f30 con 0x7f41cc070920 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.988+0000 7f41d1e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc159320 con 0x7f41cc15f390 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.988+0000 7f41d1e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41cc159600 con 0x7f41cc0b0010 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.988+0000 7f41caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc1527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc1527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44894/0 (socket says 172.21.15.39:44894) 2024-01-24T17:47:10.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41caa00640 1 -- 172.21.15.39:0/1860925005 learned_addr learned my addr 172.21.15.39:0/1860925005 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41ca000640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41cb400640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41cc15f390 0x7f41cc158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41caa00640 1 -- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 msgr2=0x7f41cc152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41caa00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41caa00640 1 -- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41cc15f390 msgr2=0x7f41cc158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41caa00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41cc15f390 0x7f41cc158ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41caa00640 1 -- 172.21.15.39:0/1860925005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b40094d0 con 0x7f41cc070920 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41ca000640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:10.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41cb400640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41cc15f390 0x7f41cc158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:10.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.989+0000 7f41caa00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc1527e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00d800 tx=0x7f41bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.990+0000 7f41c3400640 1 -- 172.21.15.39:0/1860925005 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc00b6c0 con 0x7f41cc070920 2024-01-24T17:47:10.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.990+0000 7f41c3400640 1 -- 172.21.15.39:0/1860925005 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41bc00b820 con 0x7f41cc070920 2024-01-24T17:47:10.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.990+0000 7f41c3400640 1 -- 172.21.15.39:0/1860925005 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc00bbb0 con 0x7f41cc070920 2024-01-24T17:47:10.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.990+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41cc1598e0 con 0x7f41cc070920 2024-01-24T17:47:10.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.990+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41cc159010 con 0x7f41cc070920 2024-01-24T17:47:10.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.991+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41cc04dff0 con 0x7f41cc070920 2024-01-24T17:47:10.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.992+0000 7f41c3400640 1 -- 172.21.15.39:0/1860925005 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f41bc00bd10 con 0x7f41cc070920 2024-01-24T17:47:10.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.993+0000 7f41c3400640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f41a406a430 0x7f41a406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:10.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.993+0000 7f41c3400640 1 -- 172.21.15.39:0/1860925005 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41bc088e70 con 0x7f41cc070920 2024-01-24T17:47:10.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.993+0000 7f41ca000640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f41a406a430 0x7f41a406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:10.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.994+0000 7f41ca000640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f41a406a430 0x7f41a406c8f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f41b4036000 tx=0x7f41b403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:10.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:10.997+0000 7f41c3400640 1 -- 172.21.15.39:0/1860925005 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f41bc054450 con 0x7f41cc070920 2024-01-24T17:47:11.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.235+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41cc15bd30 con 0x7f41cc070920 2024-01-24T17:47:11.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.236+0000 7f41c3400640 1 -- 172.21.15.39:0/1860925005 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f41bc057a70 con 0x7f41cc070920 2024-01-24T17:47:11.237 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:11.237 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1886779141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1886779141},{"type":"v1","addr":"172.21.15.39:6833","nonce":1886779141}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5854,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2624477336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":2624477336},{"type":"v1","addr":"172.21.15.100:6837","nonce":2624477336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2465319794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2465319794},{"type":"v1","addr":"172.21.15.39:6835","nonce":2465319794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:46:31.015014+0000","modified":"2024-01-24T17:46:34.537399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6836/2467066037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2467066037},{"type":"v1","addr":"172.21.15.100:6836","nonce":2467066037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-24T17:47:11.238 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 42 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f41a406a430 msgr2=0x7f41a406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f41a406a430 0x7f41a406c8f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f41b4036000 tx=0x7f41b403b040 comp rx=0 tx=0).stop 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 msgr2=0x7f41cc1527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc1527e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00d800 tx=0x7f41bc00dcc0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 shutdown_connections 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41cc15f390 0x7f41cc158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41cc0b0010 0x7f41cc152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f41a406a430 0x7f41a406c8f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 --2- 172.21.15.39:0/1860925005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41cc070920 0x7f41cc1527e0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 >> 172.21.15.39:0/1860925005 conn(0x7f41cc06bf20 msgr2=0x7f41cc0ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 shutdown_connections 2024-01-24T17:47:11.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.239+0000 7f41d1e00640 1 -- 172.21.15.39:0/1860925005 wait complete. 2024-01-24T17:47:11.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:47:11.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.473+0000 7fa08e400640 1 Processor -- start 2024-01-24T17:47:11.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.473+0000 7fa08e400640 1 -- start start 2024-01-24T17:47:11.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa08e400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0880b0010 0x7fa08815bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa08e400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa088070920 0x7fa0880b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa08e400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa088070470 con 0x7fa08815f390 2024-01-24T17:47:11.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa08e400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa08815ef00 con 0x7fa088070920 2024-01-24T17:47:11.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa08e400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0880710a0 con 0x7fa0880b0010 2024-01-24T17:47:11.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa07fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa088070920 0x7fa0880b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa07fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa088070920 0x7fa0880b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44900/0 (socket says 172.21.15.39:44900) 2024-01-24T17:47:11.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa07fe00640 1 -- 172.21.15.39:0/1306269260 learned_addr learned my addr 172.21.15.39:0/1306269260 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:11.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa087400640 1 --2- 172.21.15.39:0/1306269260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0880b0010 0x7fa08815bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa07fe00640 1 -- 172.21.15.39:0/1306269260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0880b0010 msgr2=0x7fa08815bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:11.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.475+0000 7fa07fe00640 1 --2- 172.21.15.39:0/1306269260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0880b0010 0x7fa08815bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.476+0000 7fa07fe00640 1 -- 172.21.15.39:0/1306269260 >> v1:172.21.15.39:6789/0 conn(0x7fa08815f390 legacy=0x7fa0880cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.476+0000 7fa07fe00640 1 -- 172.21.15.39:0/1306269260 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08815c0d0 con 0x7fa088070920 2024-01-24T17:47:11.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.477+0000 7fa07fe00640 1 --2- 172.21.15.39:0/1306269260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa088070920 0x7fa0880b03f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa070009870 tx=0x7fa07002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291d1120422b00b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:11.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.477+0000 7fa086a00640 1 -- 172.21.15.39:0/1306269260 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07002f830 con 0x7fa088070920 2024-01-24T17:47:11.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.477+0000 7fa086a00640 1 -- 172.21.15.39:0/1306269260 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa07002f990 con 0x7fa088070920 2024-01-24T17:47:11.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.477+0000 7fa086a00640 1 -- 172.21.15.39:0/1306269260 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07002fd20 con 0x7fa088070920 2024-01-24T17:47:11.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.478+0000 7fa08e400640 1 -- 172.21.15.39:0/1306269260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa088070920 msgr2=0x7fa0880b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.478+0000 7fa08e400640 1 --2- 172.21.15.39:0/1306269260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa088070920 0x7fa0880b03f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fa070009870 tx=0x7fa07002ba10 comp rx=0 tx=0).stop 2024-01-24T17:47:11.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.478+0000 7fa08e400640 1 -- 172.21.15.39:0/1306269260 shutdown_connections 2024-01-24T17:47:11.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.478+0000 7fa08e400640 1 --2- 172.21.15.39:0/1306269260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa088070920 0x7fa0880b03f0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.478+0000 7fa08e400640 1 --2- 172.21.15.39:0/1306269260 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa0880b0010 0x7fa08815bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.478+0000 7fa08e400640 1 -- 172.21.15.39:0/1306269260 >> 172.21.15.39:0/1306269260 conn(0x7fa08806bf20 msgr2=0x7fa08806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:11.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.478+0000 7fa08e400640 1 -- 172.21.15.39:0/1306269260 shutdown_connections 2024-01-24T17:47:11.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.478+0000 7fa08e400640 1 -- 172.21.15.39:0/1306269260 wait complete. 2024-01-24T17:47:11.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.480+0000 7fa08e400640 1 Processor -- start 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.480+0000 7fa08e400640 1 -- start start 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa08e400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa088070920 0x7fa0880ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa08e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0880b0010 0x7fa08814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa08e400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa08815f390 0x7fa0880c1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa08e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08814d050 con 0x7fa0880b0010 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa08e400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0880ccb60 con 0x7fa08815f390 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa08e400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0880c2480 con 0x7fa088070920 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa07fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0880b0010 0x7fa08814cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa07fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0880b0010 0x7fa08814cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44914/0 (socket says 172.21.15.39:44914) 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa07fe00640 1 -- 172.21.15.39:0/3311711200 learned_addr learned my addr 172.21.15.39:0/3311711200 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa07fe00640 1 -- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa088070920 msgr2=0x7fa0880ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa07fe00640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa088070920 0x7fa0880ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa07fe00640 1 -- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa08815f390 msgr2=0x7fa0880c1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa07fe00640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa08815f390 0x7fa0880c1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.481+0000 7fa07fe00640 1 -- 172.21.15.39:0/3311711200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0700094d0 con 0x7fa0880b0010 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.482+0000 7fa07fe00640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0880b0010 0x7fa08814cb10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fa070009990 tx=0x7fa070004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.482+0000 7fa07f400640 1 -- 172.21.15.39:0/3311711200 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07002f800 con 0x7fa0880b0010 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.482+0000 7fa07f400640 1 -- 172.21.15.39:0/3311711200 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa070042480 con 0x7fa0880b0010 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.482+0000 7fa07f400640 1 -- 172.21.15.39:0/3311711200 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070042850 con 0x7fa0880b0010 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.482+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0880cc230 con 0x7fa0880b0010 2024-01-24T17:47:11.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.482+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0880cc7f0 con 0x7fa0880b0010 2024-01-24T17:47:11.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.484+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08804dff0 con 0x7fa0880b0010 2024-01-24T17:47:11.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.484+0000 7fa07f400640 1 -- 172.21.15.39:0/3311711200 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa07002d3c0 con 0x7fa0880b0010 2024-01-24T17:47:11.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.485+0000 7fa07f400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa06006a430 0x7fa06006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.485+0000 7fa07f400640 1 -- 172.21.15.39:0/3311711200 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa07003c070 con 0x7fa0880b0010 2024-01-24T17:47:11.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.486+0000 7fa087400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa06006a430 0x7fa06006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.487+0000 7fa087400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa06006a430 0x7fa06006c8f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa078006fd0 tx=0x7fa078008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:11.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.489+0000 7fa07f400640 1 -- 172.21.15.39:0/3311711200 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa070076a50 con 0x7fa0880b0010 2024-01-24T17:47:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.725+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa088070700 con 0x7fa0880b0010 2024-01-24T17:47:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.726+0000 7fa07f400640 1 -- 172.21.15.39:0/3311711200 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fa07007a070 con 0x7fa0880b0010 2024-01-24T17:47:11.728 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:11.728 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1886779141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1886779141},{"type":"v1","addr":"172.21.15.39:6833","nonce":1886779141}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5854,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2624477336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":2624477336},{"type":"v1","addr":"172.21.15.100:6837","nonce":2624477336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2465319794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2465319794},{"type":"v1","addr":"172.21.15.39:6835","nonce":2465319794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:46:31.015014+0000","modified":"2024-01-24T17:46:34.537399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"d","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.100:6836/2467066037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2467066037},{"type":"v1","addr":"172.21.15.100:6836","nonce":2467066037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-24T17:47:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 42 2024-01-24T17:47:11.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa06006a430 msgr2=0x7fa06006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa06006a430 0x7fa06006c8f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa078006fd0 tx=0x7fa078008040 comp rx=0 tx=0).stop 2024-01-24T17:47:11.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0880b0010 msgr2=0x7fa08814cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0880b0010 0x7fa08814cb10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fa070009990 tx=0x7fa070004730 comp rx=0 tx=0).stop 2024-01-24T17:47:11.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 shutdown_connections 2024-01-24T17:47:11.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa08815f390 0x7fa0880c1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa06006a430 0x7fa06006c8f0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0880b0010 0x7fa08814cb10 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 --2- 172.21.15.39:0/3311711200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa088070920 0x7fa0880ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.730+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 >> 172.21.15.39:0/3311711200 conn(0x7fa08806bf20 msgr2=0x7fa08814d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:11.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.731+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 shutdown_connections 2024-01-24T17:47:11.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.731+0000 7fa08e400640 1 -- 172.21.15.39:0/3311711200 wait complete. 2024-01-24T17:47:11.750 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:11.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.975+0000 7f638dc00640 1 Processor -- start 2024-01-24T17:47:11.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.976+0000 7f638dc00640 1 -- start start 2024-01-24T17:47:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.977+0000 7f638dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63880d05d0 0x7f63880d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.977+0000 7f638dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880b1bf0 0x7f63880b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.977+0000 7f638dc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63880ce2f0 con 0x7f63880cf200 2024-01-24T17:47:11.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.977+0000 7f638dc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63880cd440 con 0x7f63880d05d0 2024-01-24T17:47:11.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.977+0000 7f638dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63880ce780 con 0x7f63880b1bf0 2024-01-24T17:47:11.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.978+0000 7f6387400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63880d05d0 0x7f63880d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.978+0000 7f6387400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63880d05d0 0x7f63880d3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44928/0 (socket says 172.21.15.39:44928) 2024-01-24T17:47:11.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.978+0000 7f6387400640 1 -- 172.21.15.39:0/1736807435 learned_addr learned my addr 172.21.15.39:0/1736807435 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:11.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.978+0000 7f638d200640 1 --2- 172.21.15.39:0/1736807435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880b1bf0 0x7f63880b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.978+0000 7f6387400640 1 -- 172.21.15.39:0/1736807435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880b1bf0 msgr2=0x7f63880b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:11.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.978+0000 7f6387400640 1 --2- 172.21.15.39:0/1736807435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880b1bf0 0x7f63880b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.978+0000 7f6387400640 1 -- 172.21.15.39:0/1736807435 >> v1:172.21.15.100:6789/0 conn(0x7f63880cf200 legacy=0x7f63880cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.978+0000 7f6387400640 1 -- 172.21.15.39:0/1736807435 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63880cd6e0 con 0x7f63880d05d0 2024-01-24T17:47:11.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.979+0000 7f6387400640 1 --2- 172.21.15.39:0/1736807435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880d3310 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f6370009e50 tx=0x7f637002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e68cc58caf12f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:11.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f6386a00640 1 -- 172.21.15.39:0/1736807435 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637002cb90 con 0x7f63880d05d0 2024-01-24T17:47:11.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f6386a00640 1 -- 172.21.15.39:0/1736807435 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637002ccf0 con 0x7f63880d05d0 2024-01-24T17:47:11.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f6386a00640 1 -- 172.21.15.39:0/1736807435 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370034610 con 0x7f63880d05d0 2024-01-24T17:47:11.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f638dc00640 1 -- 172.21.15.39:0/1736807435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 msgr2=0x7f63880d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f638dc00640 1 --2- 172.21.15.39:0/1736807435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880d3310 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f6370009e50 tx=0x7f637002bff0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f638dc00640 1 -- 172.21.15.39:0/1736807435 shutdown_connections 2024-01-24T17:47:11.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f638dc00640 1 --2- 172.21.15.39:0/1736807435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880d3310 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f638dc00640 1 --2- 172.21.15.39:0/1736807435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880b1bf0 0x7f63880b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f638dc00640 1 -- 172.21.15.39:0/1736807435 >> 172.21.15.39:0/1736807435 conn(0x7f63880d27f0 msgr2=0x7f63880d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:11.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f638dc00640 1 -- 172.21.15.39:0/1736807435 shutdown_connections 2024-01-24T17:47:11.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.980+0000 7f638dc00640 1 -- 172.21.15.39:0/1736807435 wait complete. 2024-01-24T17:47:11.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.982+0000 7f638dc00640 1 Processor -- start 2024-01-24T17:47:11.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.982+0000 7f638dc00640 1 -- start start 2024-01-24T17:47:11.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638dc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63880b1bf0 0x7f6388153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880cf200 0x7f63880b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63881542c0 con 0x7f63880d05d0 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638dc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63881f2520 con 0x7f63880b1bf0 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63880cbe30 con 0x7f63880cf200 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44944/0 (socket says 172.21.15.39:44944) 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.983+0000 7f638d200640 1 -- 172.21.15.39:0/2581802959 learned_addr learned my addr 172.21.15.39:0/2581802959 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.984+0000 7f6387e00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63880b1bf0 0x7f6388153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.984+0000 7f6387400640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880cf200 0x7f63880b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.984+0000 7f638d200640 1 -- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880cf200 msgr2=0x7f63880b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.984+0000 7f638d200640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880cf200 0x7f63880b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.984+0000 7f638d200640 1 -- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63880b1bf0 msgr2=0x7f6388153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.984+0000 7f638d200640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63880b1bf0 0x7f6388153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.984+0000 7f638d200640 1 -- 172.21.15.39:0/2581802959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6370009b30 con 0x7f63880d05d0 2024-01-24T17:47:11.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.984+0000 7f638d200640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880b0b20 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f638000d800 tx=0x7f638000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:11.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.985+0000 7f637fe00640 1 -- 172.21.15.39:0/2581802959 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000b6c0 con 0x7f63880d05d0 2024-01-24T17:47:11.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.985+0000 7f637fe00640 1 -- 172.21.15.39:0/2581802959 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638000b840 con 0x7f63880d05d0 2024-01-24T17:47:11.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.985+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63880cc110 con 0x7f63880d05d0 2024-01-24T17:47:11.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.985+0000 7f637fe00640 1 -- 172.21.15.39:0/2581802959 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000bbf0 con 0x7f63880d05d0 2024-01-24T17:47:11.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.985+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63880cc6d0 con 0x7f63880d05d0 2024-01-24T17:47:11.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.986+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63881f2740 con 0x7f63880d05d0 2024-01-24T17:47:11.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.987+0000 7f637fe00640 1 -- 172.21.15.39:0/2581802959 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f638000bd50 con 0x7f63880d05d0 2024-01-24T17:47:11.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.988+0000 7f637fe00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f636006a430 0x7f636006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:11.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.988+0000 7f637fe00640 1 -- 172.21.15.39:0/2581802959 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6380088eb0 con 0x7f63880d05d0 2024-01-24T17:47:11.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.989+0000 7f6387e00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f636006a430 0x7f636006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:11.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.990+0000 7f6387e00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f636006a430 0x7f636006c8f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6378000fb0 tx=0x7f6378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:11.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:11.993+0000 7f637fe00640 1 -- 172.21.15.39:0/2581802959 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f63881f2740 con 0x7f63880d05d0 2024-01-24T17:47:12.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.189+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63881f26e0 con 0x7f63880d05d0 2024-01-24T17:47:12.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.191+0000 7f637fe00640 1 -- 172.21.15.39:0/2581802959 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16040 (secure 0 0 0) 0x7f6380054400 con 0x7f63880d05d0 2024-01-24T17:47:12.193 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:12.193 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":76,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:46:32.019249+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:46:29.228451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-24T17:46:30.039417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.193+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f636006a430 msgr2=0x7f636006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.193+0000 7f638dc00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f636006a430 0x7f636006c8f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6378000fb0 tx=0x7f6378009040 comp rx=0 tx=0).stop 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.193+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 msgr2=0x7f63880b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.193+0000 7f638dc00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880b0b20 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f638000d800 tx=0x7f638000dcc0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.194+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 shutdown_connections 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.194+0000 7f638dc00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f636006a430 0x7f636006c8f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.194+0000 7f638dc00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63880d05d0 0x7f63880b0b20 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.194+0000 7f638dc00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63880cf200 0x7f63880b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.194+0000 7f638dc00640 1 --2- 172.21.15.39:0/2581802959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63880b1bf0 0x7f6388153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.194+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 >> 172.21.15.39:0/2581802959 conn(0x7f63880d27f0 msgr2=0x7f63880cf5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:12.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.194+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 shutdown_connections 2024-01-24T17:47:12.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.194+0000 7f638dc00640 1 -- 172.21.15.39:0/2581802959 wait complete. 2024-01-24T17:47:12.215 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-24T17:47:12.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.438+0000 7fe6ba600640 1 Processor -- start 2024-01-24T17:47:12.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.438+0000 7fe6ba600640 1 -- start start 2024-01-24T17:47:12.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.439+0000 7fe6ba600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe6b40cadc0 0x7fe6b40cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.439+0000 7fe6ba600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b415cb50 0x7fe6b40cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.439+0000 7fe6ba600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b40ce1d0 con 0x7fe6b40cb1a0 2024-01-24T17:47:12.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.439+0000 7fe6ba600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b40cd350 con 0x7fe6b40cadc0 2024-01-24T17:47:12.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.439+0000 7fe6ba600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40ccab0 con 0x7fe6b415cb50 2024-01-24T17:47:12.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.440+0000 7fe6b9c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe6b40cb1a0 0x7fe6b4160460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49814/0 (socket says 172.21.15.39:49814) 2024-01-24T17:47:12.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.440+0000 7fe6b9c00640 1 -- 172.21.15.39:0/2391113028 learned_addr learned my addr 172.21.15.39:0/2391113028 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.440+0000 7fe6b3e00640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b415cb50 0x7fe6b40cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.440+0000 7fe6b9200640 1 --2- 172.21.15.39:0/2391113028 >> v2:172.21.15.100:3300/0 conn(0x7fe6b40cadc0 0x7fe6b40cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.440+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443886675 0 0) 0x7fe6b40ce1d0 con 0x7fe6b40cb1a0 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe694003520 con 0x7fe6b40cb1a0 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3276294640 0 0) 0x7fe694003520 con 0x7fe6b40cb1a0 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b415cb50 msgr2=0x7fe6b40cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b415cb50 0x7fe6b40cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 >> v2:172.21.15.100:3300/0 conn(0x7fe6b40cadc0 msgr2=0x7fe6b40cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 --2- 172.21.15.39:0/2391113028 >> v2:172.21.15.100:3300/0 conn(0x7fe6b40cadc0 0x7fe6b40cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b40ccd90 con 0x7fe6b40cb1a0 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fe6ac001670 con 0x7fe6b40cb1a0 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 >> v1:172.21.15.39:6789/0 conn(0x7fe6b40cb1a0 legacy=0x7fe6b4160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694003970 0x7fe694003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b415cb50 0x7fe6940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe694004d00 0x7fe694005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40cd350 con 0x7fe694004d00 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40ccab0 con 0x7fe694003970 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69400b980 con 0x7fe6b415cb50 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.441+0000 7fe6b9c00640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe694004d00 0x7fe694005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.442+0000 7fe6b9200640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694003970 0x7fe694003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.442+0000 7fe6b3e00640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b415cb50 0x7fe6940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.442+0000 7fe6b9c00640 1 -- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b415cb50 msgr2=0x7fe6940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.442+0000 7fe6b9c00640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b415cb50 0x7fe6940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.442+0000 7fe6b9c00640 1 -- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694003970 msgr2=0x7fe694003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.442+0000 7fe6b9c00640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694003970 0x7fe694003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.442+0000 7fe6b9c00640 1 -- 172.21.15.39:0/2391113028 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe694003520 con 0x7fe694004d00 2024-01-24T17:47:12.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.443+0000 7fe6b9c00640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe694004d00 0x7fe694005180 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac00c110 tx=0x7fe6940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a160aaef1d6c68cf server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.443+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac005ad0 con 0x7fe694004d00 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.443+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ac005c50 con 0x7fe694004d00 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.443+0000 7fe6b3400640 1 -- 172.21.15.39:0/2391113028 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac02c830 con 0x7fe694004d00 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.444+0000 7fe6ba600640 1 -- 172.21.15.39:0/2391113028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe694004d00 msgr2=0x7fe694005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.444+0000 7fe6ba600640 1 --2- 172.21.15.39:0/2391113028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe694004d00 0x7fe694005180 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac00c110 tx=0x7fe6940057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.444+0000 7fe6b9c00640 1 -- 172.21.15.39:0/2391113028 reap_dead start 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.444+0000 7fe6ba600640 1 -- 172.21.15.39:0/2391113028 shutdown_connections 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.444+0000 7fe6ba600640 1 -- 172.21.15.39:0/2391113028 >> 172.21.15.39:0/2391113028 conn(0x7fe6b406a200 msgr2=0x7fe6b40aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.444+0000 7fe6ba600640 1 -- 172.21.15.39:0/2391113028 shutdown_connections 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.444+0000 7fe6ba600640 1 -- 172.21.15.39:0/2391113028 wait complete. 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.446+0000 7fe6ba600640 1 Processor -- start 2024-01-24T17:47:12.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.446+0000 7fe6ba600640 1 -- start start 2024-01-24T17:47:12.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6ba600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694004d00 0x7fe6b4071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6ba600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b415d820 0x7fe6b41485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6ba600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b414b790 0x7fe6b415a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6ba600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40c5830 con 0x7fe6b415d820 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6ba600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40705d0 con 0x7fe694004d00 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6ba600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4070e70 con 0x7fe6b414b790 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6b3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b415d820 0x7fe6b41485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6b3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b415d820 0x7fe6b41485c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44948/0 (socket says 172.21.15.39:44948) 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.447+0000 7fe6b3e00640 1 -- 172.21.15.39:0/3468235412 learned_addr learned my addr 172.21.15.39:0/3468235412 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b3e00640 1 -- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b414b790 msgr2=0x7fe6b415a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b9c00640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b414b790 0x7fe6b415a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b9200640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694004d00 0x7fe6b4071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b3e00640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b414b790 0x7fe6b415a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b3e00640 1 -- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694004d00 msgr2=0x7fe6b4071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b3e00640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694004d00 0x7fe6b4071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b3e00640 1 -- 172.21.15.39:0/3468235412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b415dd20 con 0x7fe6b415d820 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b3e00640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b415d820 0x7fe6b41485c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe69c011630 tx=0x7fe69c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:12.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.448+0000 7fe6b0c00640 1 -- 172.21.15.39:0/3468235412 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c00f8c0 con 0x7fe6b415d820 2024-01-24T17:47:12.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.449+0000 7fe6b0c00640 1 -- 172.21.15.39:0/3468235412 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c00fa20 con 0x7fe6b415d820 2024-01-24T17:47:12.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.449+0000 7fe6b0c00640 1 -- 172.21.15.39:0/3468235412 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c00fd90 con 0x7fe6b415d820 2024-01-24T17:47:12.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.449+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b4070820 con 0x7fe6b415d820 2024-01-24T17:47:12.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.449+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b414bc10 con 0x7fe6b415d820 2024-01-24T17:47:12.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.450+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b404dff0 con 0x7fe6b415d820 2024-01-24T17:47:12.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.451+0000 7fe6b0c00640 1 -- 172.21.15.39:0/3468235412 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe69c015460 con 0x7fe6b415d820 2024-01-24T17:47:12.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.452+0000 7fe6b0c00640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe68c06a460 0x7fe68c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.452+0000 7fe6b0c00640 1 -- 172.21.15.39:0/3468235412 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe69c088e50 con 0x7fe6b415d820 2024-01-24T17:47:12.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.452+0000 7fe6b9200640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe68c06a460 0x7fe68c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.453+0000 7fe6b9200640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe68c06a460 0x7fe68c06c920 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4007440 tx=0x7fe6a400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:12.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.456+0000 7fe6b0c00640 1 -- 172.21.15.39:0/3468235412 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe69c0543a0 con 0x7fe6b415d820 2024-01-24T17:47:12.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.687+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe6b4070fb0 con 0x7fe6b415d820 2024-01-24T17:47:12.690 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:47:12.689+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:47:12.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.709+0000 7fe6b0c00640 1 -- 172.21.15.39:0/3468235412 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe69c0579c0 con 0x7fe6b415d820 2024-01-24T17:47:12.710 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-24T17:47:12.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.711+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe68c06a460 msgr2=0x7fe68c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.711+0000 7fe6ba600640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe68c06a460 0x7fe68c06c920 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4007440 tx=0x7fe6a400b040 comp rx=0 tx=0).stop 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b415d820 msgr2=0x7fe6b41485c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b415d820 0x7fe6b41485c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe69c011630 tx=0x7fe69c011af0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 shutdown_connections 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe6b414b790 0x7fe6b415a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe68c06a460 0x7fe68c06c920 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b415d820 0x7fe6b41485c0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 --2- 172.21.15.39:0/3468235412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe694004d00 0x7fe6b4071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 >> 172.21.15.39:0/3468235412 conn(0x7fe6b406a200 msgr2=0x7fe6b40c9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 shutdown_connections 2024-01-24T17:47:12.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.712+0000 7fe6ba600640 1 -- 172.21.15.39:0/3468235412 wait complete. 2024-01-24T17:47:12.728 INFO:tasks.ceph.mds.d.smithi100.stderr: -1735> 2024-01-24T17:46:35.488+0000 7faf32c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:47:12.734 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-24T17:47:12.804 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:47:12.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.955+0000 7efec3400640 1 Processor -- start 2024-01-24T17:47:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.956+0000 7efec3400640 1 -- start start 2024-01-24T17:47:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.957+0000 7efec3400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efebc0d0e90 0x7efebc0d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.957+0000 7efec3400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efebc164b40 0x7efebc166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.957+0000 7efec3400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efebc0d1820 con 0x7efebc15cbe0 2024-01-24T17:47:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.957+0000 7efec3400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efebc0b8820 con 0x7efebc0d0e90 2024-01-24T17:47:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.957+0000 7efec3400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efebc0b5790 con 0x7efebc164b40 2024-01-24T17:47:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.957+0000 7efebbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efebc0d0e90 0x7efebc0d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.957+0000 7efebbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efebc0d0e90 0x7efebc0d12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44950/0 (socket says 172.21.15.39:44950) 2024-01-24T17:47:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.957+0000 7efebbe00640 1 -- 172.21.15.39:0/1064142026 learned_addr learned my addr 172.21.15.39:0/1064142026 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.958+0000 7efec1400640 1 --2- 172.21.15.39:0/1064142026 >> v2:172.21.15.100:3300/0 conn(0x7efebc164b40 0x7efebc166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.958+0000 7efebbe00640 1 -- 172.21.15.39:0/1064142026 >> v2:172.21.15.100:3300/0 conn(0x7efebc164b40 msgr2=0x7efebc166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.958+0000 7efebbe00640 1 --2- 172.21.15.39:0/1064142026 >> v2:172.21.15.100:3300/0 conn(0x7efebc164b40 0x7efebc166f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.958+0000 7efebbe00640 1 -- 172.21.15.39:0/1064142026 >> v1:172.21.15.100:6789/0 conn(0x7efebc15cbe0 legacy=0x7efebc0cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.958+0000 7efebbe00640 1 -- 172.21.15.39:0/1064142026 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc0b5a50 con 0x7efebc0d0e90 2024-01-24T17:47:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.959+0000 7efebbe00640 1 --2- 172.21.15.39:0/1064142026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc0d0e90 0x7efebc0d12e0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7efebc08c200 tx=0x7efea402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5646e38db6f0d025 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.959+0000 7efebb400640 1 -- 172.21.15.39:0/1064142026 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea4031870 con 0x7efebc0d0e90 2024-01-24T17:47:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.959+0000 7efebb400640 1 -- 172.21.15.39:0/1064142026 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea40319d0 con 0x7efebc0d0e90 2024-01-24T17:47:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.959+0000 7efebb400640 1 -- 172.21.15.39:0/1064142026 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea4031d60 con 0x7efebc0d0e90 2024-01-24T17:47:12.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.960+0000 7efec3400640 1 -- 172.21.15.39:0/1064142026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc0d0e90 msgr2=0x7efebc0d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.960+0000 7efec3400640 1 --2- 172.21.15.39:0/1064142026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc0d0e90 0x7efebc0d12e0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7efebc08c200 tx=0x7efea402b910 comp rx=0 tx=0).stop 2024-01-24T17:47:12.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.960+0000 7efec3400640 1 -- 172.21.15.39:0/1064142026 shutdown_connections 2024-01-24T17:47:12.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.960+0000 7efec3400640 1 --2- 172.21.15.39:0/1064142026 >> v2:172.21.15.100:3300/0 conn(0x7efebc164b40 0x7efebc166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.960+0000 7efec3400640 1 --2- 172.21.15.39:0/1064142026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc0d0e90 0x7efebc0d12e0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.960+0000 7efec3400640 1 -- 172.21.15.39:0/1064142026 >> 172.21.15.39:0/1064142026 conn(0x7efebc09bc80 msgr2=0x7efebc0d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.960+0000 7efec3400640 1 -- 172.21.15.39:0/1064142026 shutdown_connections 2024-01-24T17:47:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.960+0000 7efec3400640 1 -- 172.21.15.39:0/1064142026 wait complete. 2024-01-24T17:47:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.962+0000 7efec3400640 1 Processor -- start 2024-01-24T17:47:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.962+0000 7efec3400640 1 -- start start 2024-01-24T17:47:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec3400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efebc0d0e90 0x7efebc0c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec3400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efebc15cbe0 0x7efebc159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc164b40 0x7efebc0c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec3400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc0c8900 con 0x7efebc164b40 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec3400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc159f40 con 0x7efebc0d0e90 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec3400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efebc0c7cf0 con 0x7efebc15cbe0 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc164b40 0x7efebc0c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc164b40 0x7efebc0c73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44962/0 (socket says 172.21.15.39:44962) 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.963+0000 7efec1400640 1 -- 172.21.15.39:0/1848880792 learned_addr learned my addr 172.21.15.39:0/1848880792 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efebbe00640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efebc15cbe0 0x7efebc159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efec0a00640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efebc0d0e90 0x7efebc0c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efec1400640 1 -- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efebc15cbe0 msgr2=0x7efebc159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efec1400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efebc15cbe0 0x7efebc159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efec1400640 1 -- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efebc0d0e90 msgr2=0x7efebc0c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efec1400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efebc0d0e90 0x7efebc0c8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efec1400640 1 -- 172.21.15.39:0/1848880792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea40094d0 con 0x7efebc164b40 2024-01-24T17:47:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efec1400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc164b40 0x7efebc0c73f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7efeb4007e60 tx=0x7efeb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.964+0000 7efeb8c00640 1 -- 172.21.15.39:0/1848880792 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4014420 con 0x7efebc164b40 2024-01-24T17:47:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.965+0000 7efeb8c00640 1 -- 172.21.15.39:0/1848880792 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb40145a0 con 0x7efebc164b40 2024-01-24T17:47:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.965+0000 7efeb8c00640 1 -- 172.21.15.39:0/1848880792 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb4014950 con 0x7efebc164b40 2024-01-24T17:47:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.965+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc0c7fd0 con 0x7efebc164b40 2024-01-24T17:47:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.965+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc0c8590 con 0x7efebc164b40 2024-01-24T17:47:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.966+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc0c8ab0 con 0x7efebc164b40 2024-01-24T17:47:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.967+0000 7efeb8c00640 1 -- 172.21.15.39:0/1848880792 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efeb4014ab0 con 0x7efebc164b40 2024-01-24T17:47:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.968+0000 7efeb8c00640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9406a460 0x7efe9406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.968+0000 7efeb8c00640 1 -- 172.21.15.39:0/1848880792 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efeb40892a0 con 0x7efebc164b40 2024-01-24T17:47:12.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.968+0000 7efec0a00640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9406a460 0x7efe9406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.970+0000 7efec0a00640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9406a460 0x7efe9406c920 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efebc0b4680 tx=0x7efeac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:12.972+0000 7efeb8c00640 1 -- 172.21.15.39:0/1848880792 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efebc0c8ab0 con 0x7efebc164b40 2024-01-24T17:47:13.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.209+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efebc0c8ab0 con 0x7efebc164b40 2024-01-24T17:47:13.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.713+0000 7efeb8c00640 1 -- 172.21.15.39:0/1848880792 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7efebc0c8ab0 con 0x7efebc164b40 2024-01-24T17:47:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9406a460 msgr2=0x7efe9406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9406a460 0x7efe9406c920 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7efebc0b4680 tx=0x7efeac009040 comp rx=0 tx=0).stop 2024-01-24T17:47:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc164b40 msgr2=0x7efebc0c73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc164b40 0x7efebc0c73f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7efeb4007e60 tx=0x7efeb400e4b0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 shutdown_connections 2024-01-24T17:47:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efe9406a460 0x7efe9406c920 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efebc164b40 0x7efebc0c73f0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efebc15cbe0 0x7efebc159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 --2- 172.21.15.39:0/1848880792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efebc0d0e90 0x7efebc0c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 >> 172.21.15.39:0/1848880792 conn(0x7efebc09bc80 msgr2=0x7efebc0cc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:13.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 shutdown_connections 2024-01-24T17:47:13.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.717+0000 7efec3400640 1 -- 172.21.15.39:0/1848880792 wait complete. 2024-01-24T17:47:13.738 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-24T17:47:13.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.960+0000 7f8b3f400640 1 Processor -- start 2024-01-24T17:47:13.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.961+0000 7f8b3f400640 1 -- start start 2024-01-24T17:47:13.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.962+0000 7f8b3f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c8c30 0x7f8b380cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:13.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.962+0000 7f8b3f400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b380701a0 con 0x7f8b3815caa0 2024-01-24T17:47:13.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.962+0000 7f8b3f400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b380c9280 con 0x7f8b380ceac0 2024-01-24T17:47:13.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.962+0000 7f8b3f400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b380cc530 con 0x7f8b380c8c30 2024-01-24T17:47:13.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.963+0000 7f8b3e000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8b3815caa0 0x7f8b380ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54804/0 (socket says 172.21.15.39:54804) 2024-01-24T17:47:13.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.963+0000 7f8b3e000640 1 -- 172.21.15.39:0/3892777248 learned_addr learned my addr 172.21.15.39:0/3892777248 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:13.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.963+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057586494 0 0) 0x7f8b380701a0 con 0x7f8b3815caa0 2024-01-24T17:47:13.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.963+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b18003520 con 0x7f8b3815caa0 2024-01-24T17:47:13.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.963+0000 7f8b3ea00640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c8c30 0x7f8b380cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:13.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179106113 0 0) 0x7f8b18003520 con 0x7f8b3815caa0 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c8c30 msgr2=0x7f8b380cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c8c30 0x7f8b380cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 >> v1:172.21.15.100:6789/0 conn(0x7f8b380ceac0 legacy=0x7f8b38070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b380c9560 con 0x7f8b3815caa0 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8b28001670 con 0x7f8b3815caa0 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 >> v1:172.21.15.39:6789/0 conn(0x7f8b3815caa0 legacy=0x7f8b380ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b180039a0 0x7f8b18003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b180047c0 0x7f8b18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b180052a0 0x7f8b1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b380cc530 con 0x7f8b180039a0 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1800c1c0 con 0x7f8b180052a0 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1800c380 con 0x7f8b180047c0 2024-01-24T17:47:13.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3ea00640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c8c30 0x7f8b380cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:47:13.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3e000640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b180039a0 0x7f8b18003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:13.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.964+0000 7f8b3d600640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b180047c0 0x7f8b18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:13.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.965+0000 7f8b3e000640 1 -- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b180047c0 msgr2=0x7f8b18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:13.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.965+0000 7f8b3e000640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b180047c0 0x7f8b18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.965+0000 7f8b3e000640 1 -- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b180052a0 msgr2=0x7f8b1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:13.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.965+0000 7f8b3e000640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b180052a0 0x7f8b1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.965+0000 7f8b3e000640 1 -- 172.21.15.39:0/3892777248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18003520 con 0x7f8b180039a0 2024-01-24T17:47:13.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.965+0000 7f8b3d600640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b180047c0 0x7f8b18004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:13.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.966+0000 7f8b3e000640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b180039a0 0x7f8b18003dc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800c0e0 tx=0x7f8b180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16f666d0ba0c9d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:13.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.966+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28005ad0 con 0x7f8b180039a0 2024-01-24T17:47:13.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.966+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b28005c30 con 0x7f8b180039a0 2024-01-24T17:47:13.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.966+0000 7f8b3cc00640 1 -- 172.21.15.39:0/3892777248 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2802c7f0 con 0x7f8b180039a0 2024-01-24T17:47:13.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.967+0000 7f8b3f400640 1 -- 172.21.15.39:0/3892777248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b180039a0 msgr2=0x7f8b18003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:13.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.967+0000 7f8b3f400640 1 --2- 172.21.15.39:0/3892777248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b180039a0 0x7f8b18003dc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800c0e0 tx=0x7f8b180057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.967+0000 7f8b3ea00640 1 -- 172.21.15.39:0/3892777248 reap_dead start 2024-01-24T17:47:13.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.967+0000 7f8b3f400640 1 -- 172.21.15.39:0/3892777248 shutdown_connections 2024-01-24T17:47:13.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.967+0000 7f8b3f400640 1 -- 172.21.15.39:0/3892777248 >> 172.21.15.39:0/3892777248 conn(0x7f8b3806a200 msgr2=0x7f8b380affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:13.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.967+0000 7f8b3f400640 1 -- 172.21.15.39:0/3892777248 shutdown_connections 2024-01-24T17:47:13.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.967+0000 7f8b3f400640 1 -- 172.21.15.39:0/3892777248 wait complete. 2024-01-24T17:47:13.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.969+0000 7f8b3f400640 1 Processor -- start 2024-01-24T17:47:13.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.969+0000 7f8b3f400640 1 -- start start 2024-01-24T17:47:13.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3f400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b180039a0 0x7f8b38156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:13.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b380b02d0 0x7f8b380ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:13.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c54f0 0x7f8b380cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:13.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3f400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b380b10a0 con 0x7f8b380b02d0 2024-01-24T17:47:13.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3f400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b38157110 con 0x7f8b180039a0 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3f400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b380c61d0 con 0x7f8b380c54f0 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b380b02d0 0x7f8b380ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b380b02d0 0x7f8b380ccef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48930/0 (socket says 172.21.15.39:48930) 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.970+0000 7f8b3d600640 1 -- 172.21.15.39:0/804135850 learned_addr learned my addr 172.21.15.39:0/804135850 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3d600640 1 -- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c54f0 msgr2=0x7f8b380cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3e000640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b180039a0 0x7f8b38156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3d600640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c54f0 0x7f8b380cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3d600640 1 -- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b180039a0 msgr2=0x7f8b38156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3d600640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b180039a0 0x7f8b38156bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3d600640 1 -- 172.21.15.39:0/804135850 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b380c64b0 con 0x7f8b380b02d0 2024-01-24T17:47:13.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3d600640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b380b02d0 0x7f8b380ccef0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000d770 tx=0x7f8b2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:13.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b36000640 1 -- 172.21.15.39:0/804135850 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20005270 con 0x7f8b380b02d0 2024-01-24T17:47:13.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b36000640 1 -- 172.21.15.39:0/804135850 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b200053d0 con 0x7f8b380b02d0 2024-01-24T17:47:13.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b380ae7e0 con 0x7f8b380b02d0 2024-01-24T17:47:13.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b36000640 1 -- 172.21.15.39:0/804135850 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000b8d0 con 0x7f8b380b02d0 2024-01-24T17:47:13.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.971+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b380ae4e0 con 0x7f8b380b02d0 2024-01-24T17:47:13.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.972+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b380c66f0 con 0x7f8b380b02d0 2024-01-24T17:47:13.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.973+0000 7f8b36000640 1 -- 172.21.15.39:0/804135850 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8b2000ba30 con 0x7f8b380b02d0 2024-01-24T17:47:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.974+0000 7f8b36000640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8b1006a430 0x7f8b1006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.974+0000 7f8b36000640 1 -- 172.21.15.39:0/804135850 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8b200888f0 con 0x7f8b380b02d0 2024-01-24T17:47:13.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.975+0000 7f8b3e000640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8b1006a430 0x7f8b1006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:13.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.976+0000 7f8b3e000640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8b1006a430 0x7f8b1006c8f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8b28034000 tx=0x7f8b28009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:13.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:13.979+0000 7f8b36000640 1 -- 172.21.15.39:0/804135850 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8b20053e70 con 0x7f8b380b02d0 2024-01-24T17:47:14.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.173+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8b3804e840 con 0x7f8b380b02d0 2024-01-24T17:47:14.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.719+0000 7f8b36000640 1 -- 172.21.15.39:0/804135850 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v78) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8b20057490 con 0x7f8b380b02d0 2024-01-24T17:47:14.721 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8b1006a430 msgr2=0x7f8b1006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8b1006a430 0x7f8b1006c8f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8b28034000 tx=0x7f8b28009030 comp rx=0 tx=0).stop 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b380b02d0 msgr2=0x7f8b380ccef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b380b02d0 0x7f8b380ccef0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000d770 tx=0x7f8b2000dc30 comp rx=0 tx=0).stop 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 shutdown_connections 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8b380c54f0 0x7f8b380cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8b1006a430 0x7f8b1006c8f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8b380b02d0 0x7f8b380ccef0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 --2- 172.21.15.39:0/804135850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8b180039a0 0x7f8b38156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 >> 172.21.15.39:0/804135850 conn(0x7f8b3806a200 msgr2=0x7f8b380c99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 shutdown_connections 2024-01-24T17:47:14.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.724+0000 7f8b3f400640 1 -- 172.21.15.39:0/804135850 wait complete. 2024-01-24T17:47:14.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-24T17:47:14.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.961+0000 7f994e200640 1 Processor -- start 2024-01-24T17:47:14.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.961+0000 7f994e200640 1 -- start start 2024-01-24T17:47:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.962+0000 7f994e200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99481516c0 0x7f99480b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.962+0000 7f994e200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f994815ac70 0x7f994806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.962+0000 7f994e200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480b12b0 con 0x7f9948070260 2024-01-24T17:47:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.962+0000 7f994e200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9948070710 con 0x7f994815ac70 2024-01-24T17:47:14.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.962+0000 7f994e200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994815b170 con 0x7f99481516c0 2024-01-24T17:47:14.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f994d800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9948070260 0x7f99480ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54820/0 (socket says 172.21.15.39:54820) 2024-01-24T17:47:14.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f994d800640 1 -- 172.21.15.39:0/838275570 learned_addr learned my addr 172.21.15.39:0/838275570 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f993fc00640 1 --2- 172.21.15.39:0/838275570 >> v2:172.21.15.100:3300/0 conn(0x7f994815ac70 0x7f994806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f994ce00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99481516c0 0x7f99480b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837679187 0 0) 0x7f99480b12b0 con 0x7f9948070260 2024-01-24T17:47:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9928003520 con 0x7f9948070260 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 11337101 0 0) 0x7f9928003520 con 0x7f9948070260 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99481516c0 msgr2=0x7f99480b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99481516c0 0x7f99480b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 >> v2:172.21.15.100:3300/0 conn(0x7f994815ac70 msgr2=0x7f994806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 --2- 172.21.15.39:0/838275570 >> v2:172.21.15.100:3300/0 conn(0x7f994815ac70 0x7f994806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994815b450 con 0x7f9948070260 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.963+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f9940001670 con 0x7f9948070260 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 >> v1:172.21.15.39:6789/0 conn(0x7f9948070260 legacy=0x7f99480ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f9947e00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9928003970 0x7f9928003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f9947e00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99280047c0 0x7f9928004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f9947e00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99481516c0 0x7f9928005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9948070710 con 0x7f99280047c0 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994815b170 con 0x7f9928003970 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992800ba30 con 0x7f99481516c0 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f993fc00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99280047c0 0x7f9928004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f994ce00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9928003970 0x7f9928003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f993fc00640 1 -- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99481516c0 msgr2=0x7f9928005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f993fc00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99481516c0 0x7f9928005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f993fc00640 1 -- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9928003970 msgr2=0x7f9928003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f993fc00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9928003970 0x7f9928003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f993fc00640 1 -- 172.21.15.39:0/838275570 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9928003520 con 0x7f99280047c0 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.964+0000 7f994ce00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9928003970 0x7f9928003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.965+0000 7f993fc00640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99280047c0 0x7f9928004c40 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f993000ba00 tx=0x7f99280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69849049d6290639 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.966+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930032620 con 0x7f99280047c0 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.966+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930032780 con 0x7f99280047c0 2024-01-24T17:47:14.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.966+0000 7f9947e00640 1 -- 172.21.15.39:0/838275570 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930032b10 con 0x7f99280047c0 2024-01-24T17:47:14.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.966+0000 7f994e200640 1 -- 172.21.15.39:0/838275570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99280047c0 msgr2=0x7f9928004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.966+0000 7f994e200640 1 --2- 172.21.15.39:0/838275570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99280047c0 0x7f9928004c40 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f993000ba00 tx=0x7f99280057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.966+0000 7f994d800640 1 -- 172.21.15.39:0/838275570 reap_dead start 2024-01-24T17:47:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.966+0000 7f994e200640 1 -- 172.21.15.39:0/838275570 shutdown_connections 2024-01-24T17:47:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.966+0000 7f994e200640 1 -- 172.21.15.39:0/838275570 >> 172.21.15.39:0/838275570 conn(0x7f994806a200 msgr2=0x7f99480affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.967+0000 7f994e200640 1 -- 172.21.15.39:0/838275570 shutdown_connections 2024-01-24T17:47:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.967+0000 7f994e200640 1 -- 172.21.15.39:0/838275570 wait complete. 2024-01-24T17:47:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.968+0000 7f994e200640 1 Processor -- start 2024-01-24T17:47:14.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.969+0000 7f994e200640 1 -- start start 2024-01-24T17:47:14.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994e200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99280047c0 0x7f99480c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994e200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99480cd810 0x7f994815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99481f6010 0x7f99481571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994e200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480cdff0 con 0x7f99481f6010 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994e200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99480cdd10 con 0x7f99480cd810 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994e200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99481f5530 con 0x7f99280047c0 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99481f6010 0x7f99481571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99481f6010 0x7f99481571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48946/0 (socket says 172.21.15.39:48946) 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994d800640 1 -- 172.21.15.39:0/816147994 learned_addr learned my addr 172.21.15.39:0/816147994 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994ce00640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99280047c0 0x7f99480c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f993fc00640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99480cd810 0x7f994815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994d800640 1 -- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99280047c0 msgr2=0x7f99480c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994d800640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99280047c0 0x7f99480c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994d800640 1 -- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99480cd810 msgr2=0x7f994815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994d800640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99480cd810 0x7f994815c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:14.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994d800640 1 -- 172.21.15.39:0/816147994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99481f5810 con 0x7f99481f6010 2024-01-24T17:47:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.970+0000 7f994ce00640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99280047c0 0x7f99480c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.971+0000 7f994d800640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99481f6010 0x7f99481571b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f9940000ff0 tx=0x7f994000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.971+0000 7f9945600640 1 -- 172.21.15.39:0/816147994 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99400074e0 con 0x7f99481f6010 2024-01-24T17:47:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.971+0000 7f9945600640 1 -- 172.21.15.39:0/816147994 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940007680 con 0x7f99481f6010 2024-01-24T17:47:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.971+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99481f5af0 con 0x7f99481f6010 2024-01-24T17:47:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.971+0000 7f9945600640 1 -- 172.21.15.39:0/816147994 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940004a00 con 0x7f99481f6010 2024-01-24T17:47:14.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.971+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994815ceb0 con 0x7f99481f6010 2024-01-24T17:47:14.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.972+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99481f5ce0 con 0x7f99481f6010 2024-01-24T17:47:14.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.973+0000 7f9945600640 1 -- 172.21.15.39:0/816147994 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9940004b60 con 0x7f99481f6010 2024-01-24T17:47:14.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.974+0000 7f9945600640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f992006a430 0x7f992006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:14.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.974+0000 7f9945600640 1 -- 172.21.15.39:0/816147994 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f9940088120 con 0x7f99481f6010 2024-01-24T17:47:14.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.975+0000 7f994ce00640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f992006a430 0x7f992006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:14.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.976+0000 7f994ce00640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f992006a430 0x7f992006c8f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9938009fd0 tx=0x7f993800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:14.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:14.979+0000 7f9945600640 1 -- 172.21.15.39:0/816147994 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9940053860 con 0x7f99481f6010 2024-01-24T17:47:15.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.171+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f994815b9e0 con 0x7f99481f6010 2024-01-24T17:47:15.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.721+0000 7f9945600640 1 -- 172.21.15.39:0/816147994 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v79) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9940056e80 con 0x7f99481f6010 2024-01-24T17:47:15.723 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-01-24T17:47:15.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.724+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f992006a430 msgr2=0x7f992006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:15.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.724+0000 7f994e200640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f992006a430 0x7f992006c8f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9938009fd0 tx=0x7f993800b040 comp rx=0 tx=0).stop 2024-01-24T17:47:15.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.724+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99481f6010 msgr2=0x7f99481571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:15.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.724+0000 7f994e200640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99481f6010 0x7f99481571b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f9940000ff0 tx=0x7f994000c890 comp rx=0 tx=0).stop 2024-01-24T17:47:15.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.725+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 shutdown_connections 2024-01-24T17:47:15.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.725+0000 7f994e200640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f992006a430 0x7f992006c8f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.725+0000 7f994e200640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99481f6010 0x7f99481571b0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.725+0000 7f994e200640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f99480cd810 0x7f994815c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.725+0000 7f994e200640 1 --2- 172.21.15.39:0/816147994 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f99280047c0 0x7f99480c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.725+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 >> 172.21.15.39:0/816147994 conn(0x7f994806a200 msgr2=0x7f9948157f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:15.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.725+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 shutdown_connections 2024-01-24T17:47:15.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.725+0000 7f994e200640 1 -- 172.21.15.39:0/816147994 wait complete. 2024-01-24T17:47:15.750 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-24T17:47:15.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.976+0000 7f6fe3200640 1 Processor -- start 2024-01-24T17:47:15.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.976+0000 7f6fe3200640 1 -- start start 2024-01-24T17:47:15.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.977+0000 7f6fe3200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6fe40b1c40 0x7f6fe40d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:15.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.977+0000 7f6fe3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe406acb0 con 0x7f6fe40d0de0 2024-01-24T17:47:15.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.977+0000 7f6fe3200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe415e0f0 con 0x7f6fe40d1220 2024-01-24T17:47:15.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.977+0000 7f6fe3200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe415d300 con 0x7f6fe40b1c40 2024-01-24T17:47:15.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.978+0000 7f6fe1400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f6fe40d0de0 0x7f6fe415c4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54828/0 (socket says 172.21.15.39:54828) 2024-01-24T17:47:15.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.978+0000 7f6fe1400640 1 -- 172.21.15.39:0/1984614303 learned_addr learned my addr 172.21.15.39:0/1984614303 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:15.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.978+0000 7f6fe1e00640 1 --2- 172.21.15.39:0/1984614303 >> v2:172.21.15.100:3300/0 conn(0x7f6fe40b1c40 0x7f6fe40d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:15.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.978+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622743922 0 0) 0x7f6fe406acb0 con 0x7f6fe40d0de0 2024-01-24T17:47:15.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.978+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4003520 con 0x7f6fe40d0de0 2024-01-24T17:47:15.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2141104393 0 0) 0x7f6fc4003520 con 0x7f6fe40d0de0 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 >> v2:172.21.15.100:3300/0 conn(0x7f6fe40b1c40 msgr2=0x7f6fe40d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 --2- 172.21.15.39:0/1984614303 >> v2:172.21.15.100:3300/0 conn(0x7f6fe40b1c40 0x7f6fe40d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 >> v1:172.21.15.100:6789/0 conn(0x7f6fe40d1220 legacy=0x7f6fe415cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe415d4e0 con 0x7f6fe40d0de0 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f6fcc001670 con 0x7f6fe40d0de0 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 >> v1:172.21.15.39:6789/0 conn(0x7f6fe40d0de0 legacy=0x7f6fe415c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fc40047c0 0x7f6fc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fc4005180 0x7f6fc400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe415d300 con 0x7f6fc4003970 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc400c030 con 0x7f6fc4005180 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc400c1f0 con 0x7f6fc40047c0 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.979+0000 7f6fe1e00640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe2800640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fc4005180 0x7f6fc400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:15.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe1400640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fc40047c0 0x7f6fc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:15.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe1e00640 1 -- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fc40047c0 msgr2=0x7f6fc4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:15.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe1e00640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fc40047c0 0x7f6fc4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe1e00640 1 -- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fc4005180 msgr2=0x7f6fc400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:15.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe1e00640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fc4005180 0x7f6fc400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe1e00640 1 -- 172.21.15.39:0/1984614303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4003520 con 0x7f6fc4003970 2024-01-24T17:47:15.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe2800640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fc4005180 0x7f6fc400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:15.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.980+0000 7f6fe2800640 1 -- 172.21.15.39:0/1984614303 reap_dead start 2024-01-24T17:47:15.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.981+0000 7f6fe1e00640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fc4003d90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400e170 tx=0x7f6fc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a7412846ddb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:15.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.981+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd402cc60 con 0x7f6fc4003970 2024-01-24T17:47:15.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.981+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd400f040 con 0x7f6fc4003970 2024-01-24T17:47:15.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.981+0000 7f6fe0a00640 1 -- 172.21.15.39:0/1984614303 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4034440 con 0x7f6fc4003970 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.982+0000 7f6fe3200640 1 -- 172.21.15.39:0/1984614303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 msgr2=0x7f6fc4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.982+0000 7f6fe3200640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fc4003d90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400e170 tx=0x7f6fc40057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.982+0000 7f6fe3200640 1 -- 172.21.15.39:0/1984614303 shutdown_connections 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.982+0000 7f6fe3200640 1 --2- 172.21.15.39:0/1984614303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fc4003d90 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.982+0000 7f6fe3200640 1 -- 172.21.15.39:0/1984614303 >> 172.21.15.39:0/1984614303 conn(0x7f6fe409bad0 msgr2=0x7f6fe40d2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.982+0000 7f6fe3200640 1 -- 172.21.15.39:0/1984614303 shutdown_connections 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.982+0000 7f6fe3200640 1 -- 172.21.15.39:0/1984614303 wait complete. 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.984+0000 7f6fe3200640 1 Processor -- start 2024-01-24T17:47:15.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.984+0000 7f6fe3200640 1 -- start start 2024-01-24T17:47:15.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fe415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:15.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe3200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fe40b6030 0x7f6fe41f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:15.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe3200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fe41f2b30 0x7f6fe406bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:15.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40b78d0 con 0x7f6fc4003970 2024-01-24T17:47:15.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe3200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe41f31b0 con 0x7f6fe40b6030 2024-01-24T17:47:15.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe3200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40b5550 con 0x7f6fe41f2b30 2024-01-24T17:47:15.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fe415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:15.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fe415b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48968/0 (socket says 172.21.15.39:48968) 2024-01-24T17:47:15.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1e00640 1 -- 172.21.15.39:0/1206776391 learned_addr learned my addr 172.21.15.39:0/1206776391 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1400640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fe40b6030 0x7f6fe41f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1e00640 1 -- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fe41f2b30 msgr2=0x7f6fe406bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1e00640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fe41f2b30 0x7f6fe406bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1e00640 1 -- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fe40b6030 msgr2=0x7f6fe41f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:15.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1e00640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fe40b6030 0x7f6fe41f25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.985+0000 7f6fe1e00640 1 -- 172.21.15.39:0/1206776391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40b5830 con 0x7f6fc4003970 2024-01-24T17:47:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.986+0000 7f6fe1e00640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fe415b0a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f6fd402c600 tx=0x7f6fd4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.986+0000 7f6fda000640 1 -- 172.21.15.39:0/1206776391 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4040450 con 0x7f6fc4003970 2024-01-24T17:47:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.986+0000 7f6fda000640 1 -- 172.21.15.39:0/1206776391 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd40405d0 con 0x7f6fc4003970 2024-01-24T17:47:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.986+0000 7f6fda000640 1 -- 172.21.15.39:0/1206776391 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4040980 con 0x7f6fc4003970 2024-01-24T17:47:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.986+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe40b5b10 con 0x7f6fc4003970 2024-01-24T17:47:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.986+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe406c4f0 con 0x7f6fc4003970 2024-01-24T17:47:15.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.987+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe41f2f80 con 0x7f6fc4003970 2024-01-24T17:47:15.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.988+0000 7f6fda000640 1 -- 172.21.15.39:0/1206776391 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6fd4040ae0 con 0x7f6fc4003970 2024-01-24T17:47:15.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.989+0000 7f6fda000640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a430 0x7f6fbc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:15.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.989+0000 7f6fda000640 1 -- 172.21.15.39:0/1206776391 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6fd40aafc0 con 0x7f6fc4003970 2024-01-24T17:47:15.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.990+0000 7f6fe1400640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a430 0x7f6fbc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:15.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.991+0000 7f6fe1400640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a430 0x7f6fbc06c8f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc001200 tx=0x7f6fcc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:15.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:15.994+0000 7f6fda000640 1 -- 172.21.15.39:0/1206776391 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6fd4076930 con 0x7f6fc4003970 2024-01-24T17:47:16.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.195+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6fe415b5e0 con 0x7f6fc4003970 2024-01-24T17:47:16.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.201+0000 7f6fda000640 1 -- 172.21.15.39:0/1206776391 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6fd4079f50 con 0x7f6fc4003970 2024-01-24T17:47:16.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a430 msgr2=0x7f6fbc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a430 0x7f6fbc06c8f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc001200 tx=0x7f6fcc009040 comp rx=0 tx=0).stop 2024-01-24T17:47:16.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 msgr2=0x7f6fe415b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fe415b0a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f6fd402c600 tx=0x7f6fd4009750 comp rx=0 tx=0).stop 2024-01-24T17:47:16.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 shutdown_connections 2024-01-24T17:47:16.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6fe41f2b30 0x7f6fe406bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6fe40b6030 0x7f6fe41f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6fbc06a430 0x7f6fbc06c8f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 --2- 172.21.15.39:0/1206776391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fc4003970 0x7f6fe415b0a0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 >> 172.21.15.39:0/1206776391 conn(0x7f6fe409bad0 msgr2=0x7f6fe40cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:16.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 shutdown_connections 2024-01-24T17:47:16.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.204+0000 7f6fe3200640 1 -- 172.21.15.39:0/1206776391 wait complete. 2024-01-24T17:47:16.227 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds ms_async_reap_threshold 2024-01-24T17:47:16.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.449+0000 7fbf14a00640 1 Processor -- start 2024-01-24T17:47:16.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.449+0000 7fbf14a00640 1 -- start start 2024-01-24T17:47:16.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.451+0000 7fbf14a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 0x7fbf100b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.451+0000 7fbf14a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbf1015ac70 0x7fbf1006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.451+0000 7fbf14a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100b12b0 con 0x7fbf10070260 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.451+0000 7fbf14a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf10070710 con 0x7fbf1015ac70 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.451+0000 7fbf14a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1015b170 con 0x7fbf101516c0 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.451+0000 7fbf0de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbf1015ac70 0x7fbf1006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.451+0000 7fbf0de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbf1015ac70 0x7fbf1006fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48976/0 (socket says 172.21.15.39:48976) 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.451+0000 7fbf0de00640 1 -- 172.21.15.39:0/2709436354 learned_addr learned my addr 172.21.15.39:0/2709436354 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.452+0000 7fbf0e800640 1 --2- 172.21.15.39:0/2709436354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 0x7fbf100b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.452+0000 7fbf0de00640 1 -- 172.21.15.39:0/2709436354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 msgr2=0x7fbf100b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.452+0000 7fbf0de00640 1 --2- 172.21.15.39:0/2709436354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 0x7fbf100b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.452+0000 7fbf0de00640 1 -- 172.21.15.39:0/2709436354 >> v1:172.21.15.100:6789/0 conn(0x7fbf10070260 legacy=0x7fbf100ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:16.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.452+0000 7fbf0de00640 1 -- 172.21.15.39:0/2709436354 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1015b450 con 0x7fbf1015ac70 2024-01-24T17:47:16.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.453+0000 7fbf0de00640 1 --2- 172.21.15.39:0/2709436354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf1015ac70 0x7fbf1006fd20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009870 tx=0x7fbef802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d046bfd1e89f4ed5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:16.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.453+0000 7fbf0d400640 1 -- 172.21.15.39:0/2709436354 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8031640 con 0x7fbf1015ac70 2024-01-24T17:47:16.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.454+0000 7fbf0d400640 1 -- 172.21.15.39:0/2709436354 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef80317a0 con 0x7fbf1015ac70 2024-01-24T17:47:16.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.454+0000 7fbf0d400640 1 -- 172.21.15.39:0/2709436354 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8031b30 con 0x7fbf1015ac70 2024-01-24T17:47:16.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.454+0000 7fbf14a00640 1 -- 172.21.15.39:0/2709436354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf1015ac70 msgr2=0x7fbf1006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.454+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2709436354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf1015ac70 0x7fbf1006fd20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009870 tx=0x7fbef802b8f0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.454+0000 7fbf14a00640 1 -- 172.21.15.39:0/2709436354 shutdown_connections 2024-01-24T17:47:16.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.454+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2709436354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf1015ac70 0x7fbf1006fd20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.454+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2709436354 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 0x7fbf100b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.454+0000 7fbf14a00640 1 -- 172.21.15.39:0/2709436354 >> 172.21.15.39:0/2709436354 conn(0x7fbf1006a200 msgr2=0x7fbf100affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:16.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.455+0000 7fbf14a00640 1 -- 172.21.15.39:0/2709436354 shutdown_connections 2024-01-24T17:47:16.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.455+0000 7fbf14a00640 1 -- 172.21.15.39:0/2709436354 wait complete. 2024-01-24T17:47:16.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.457+0000 7fbf14a00640 1 Processor -- start 2024-01-24T17:47:16.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.457+0000 7fbf14a00640 1 -- start start 2024-01-24T17:47:16.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.458+0000 7fbf14a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf10070260 0x7fbf101f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.458+0000 7fbf14a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 0x7fbf100c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.458+0000 7fbf14a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1015ac70 0x7fbf100addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.458+0000 7fbf14a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100afde0 con 0x7fbf10070260 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.458+0000 7fbf14a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100c3e60 con 0x7fbf1015ac70 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.458+0000 7fbf14a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf101f3a60 con 0x7fbf101516c0 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf10070260 0x7fbf101f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf10070260 0x7fbf101f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48986/0 (socket says 172.21.15.39:48986) 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 -- 172.21.15.39:0/2499332179 learned_addr learned my addr 172.21.15.39:0/2499332179 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 -- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 msgr2=0x7fbf100c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 0x7fbf100c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 -- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1015ac70 msgr2=0x7fbf100addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1015ac70 0x7fbf100addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 -- 172.21.15.39:0/2499332179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef80094d0 con 0x7fbf10070260 2024-01-24T17:47:16.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.459+0000 7fbf0e800640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf10070260 0x7fbf101f3520 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000d5f0 tx=0x7fbf0000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:16.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.460+0000 7fbf06a00640 1 -- 172.21.15.39:0/2499332179 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf000125f0 con 0x7fbf10070260 2024-01-24T17:47:16.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.460+0000 7fbf06a00640 1 -- 172.21.15.39:0/2499332179 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf00012750 con 0x7fbf10070260 2024-01-24T17:47:16.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.460+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf101f3c90 con 0x7fbf10070260 2024-01-24T17:47:16.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.460+0000 7fbf06a00640 1 -- 172.21.15.39:0/2499332179 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00012b00 con 0x7fbf10070260 2024-01-24T17:47:16.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.460+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf101f4250 con 0x7fbf10070260 2024-01-24T17:47:16.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.461+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1004dff0 con 0x7fbf10070260 2024-01-24T17:47:16.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.462+0000 7fbf06a00640 1 -- 172.21.15.39:0/2499332179 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbf00012c60 con 0x7fbf10070260 2024-01-24T17:47:16.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.463+0000 7fbf06a00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee806a430 0x7fbee806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.463+0000 7fbf06a00640 1 -- 172.21.15.39:0/2499332179 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbf00088ab0 con 0x7fbf10070260 2024-01-24T17:47:16.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.464+0000 7fbf0de00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee806a430 0x7fbee806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.465+0000 7fbf0de00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee806a430 0x7fbee806c8f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbef8036000 tx=0x7fbef8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.468+0000 7fbf06a00640 1 -- 172.21.15.39:0/2499332179 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbf00054390 con 0x7fbf10070260 2024-01-24T17:47:16.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.676+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "ms_async_reap_threshold"} v 0) v1 -- 0x7fbf100b0d20 con 0x7fbf10070260 2024-01-24T17:47:16.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.682+0000 7fbf06a00640 1 -- 172.21.15.39:0/2499332179 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "ms_async_reap_threshold"}]=0 v21) v1 ==== 106+0+0 (secure 0 0 0) 0x7fbf000579b0 con 0x7fbf10070260 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee806a430 msgr2=0x7fbee806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee806a430 0x7fbee806c8f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbef8036000 tx=0x7fbef8030000 comp rx=0 tx=0).stop 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf10070260 msgr2=0x7fbf101f3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf10070260 0x7fbf101f3520 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000d5f0 tx=0x7fbf0000dab0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 shutdown_connections 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbf1015ac70 0x7fbf100addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbf101516c0 0x7fbf100c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbee806a430 0x7fbee806c8f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 --2- 172.21.15.39:0/2499332179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf10070260 0x7fbf101f3520 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 >> 172.21.15.39:0/2499332179 conn(0x7fbf1006a200 msgr2=0x7fbf1015ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:16.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 shutdown_connections 2024-01-24T17:47:16.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.686+0000 7fbf14a00640 1 -- 172.21.15.39:0/2499332179 wait complete. 2024-01-24T17:47:16.708 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_conn_close' 2024-01-24T17:47:16.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.926+0000 7f214be00640 1 Processor -- start 2024-01-24T17:47:16.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.927+0000 7f214be00640 1 -- start start 2024-01-24T17:47:16.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.928+0000 7f214be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214c06acb0 0x7f214c0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.928+0000 7f214be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214c0ce010 con 0x7f214c0cf8a0 2024-01-24T17:47:16.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.928+0000 7f214be00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214c0ce2b0 con 0x7f214c15cbf0 2024-01-24T17:47:16.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.928+0000 7f214be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0ce870 con 0x7f214c06acb0 2024-01-24T17:47:16.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.929+0000 7f214a000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f214c0cf8a0 0x7f214c0d0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54836/0 (socket says 172.21.15.39:54836) 2024-01-24T17:47:16.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.929+0000 7f214a000640 1 -- 172.21.15.39:0/105054984 learned_addr learned my addr 172.21.15.39:0/105054984 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:16.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.929+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105249089 0 0) 0x7f214c0ce010 con 0x7f214c0cf8a0 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.929+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c003520 con 0x7f214c0cf8a0 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.929+0000 7f214b400640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214c06acb0 0x7f214c0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583945710 0 0) 0x7f212c003520 con 0x7f214c0cf8a0 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214c06acb0 msgr2=0x7f214c0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214c06acb0 0x7f214c0d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 >> v1:172.21.15.100:6789/0 conn(0x7f214c15cbf0 legacy=0x7f214c0cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c0ce590 con 0x7f214c0cf8a0 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f213c0019f0 con 0x7f214c0cf8a0 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 >> v1:172.21.15.39:6789/0 conn(0x7f214c0cf8a0 legacy=0x7f214c0d0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f212c0039a0 0x7f212c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214c06acb0 0x7f212c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c004d00 0x7f212c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0ce870 con 0x7f212c004d00 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c00bb60 con 0x7f212c0039a0 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c00bd20 con 0x7f214c06acb0 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f214b400640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c004d00 0x7f212c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f214aa00640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f212c0039a0 0x7f212c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f214a000640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214c06acb0 0x7f212c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.930+0000 7f214b400640 1 -- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214c06acb0 msgr2=0x7f212c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.931+0000 7f214b400640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f214c06acb0 0x7f212c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.931+0000 7f214b400640 1 -- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f212c0039a0 msgr2=0x7f212c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.931+0000 7f214b400640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f212c0039a0 0x7f212c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.931+0000 7f214b400640 1 -- 172.21.15.39:0/105054984 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c003520 con 0x7f212c004d00 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.932+0000 7f214b400640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c004d00 0x7f212c005180 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f214400ba10 tx=0x7f212c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a44bfe76649b9f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.932+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144032620 con 0x7f212c004d00 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.932+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2144032780 con 0x7f212c004d00 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.932+0000 7f2149600640 1 -- 172.21.15.39:0/105054984 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144032b10 con 0x7f212c004d00 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.932+0000 7f214be00640 1 -- 172.21.15.39:0/105054984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c004d00 msgr2=0x7f212c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.932+0000 7f214be00640 1 --2- 172.21.15.39:0/105054984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f212c004d00 0x7f212c005180 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f214400ba10 tx=0x7f212c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.933+0000 7f214b400640 1 -- 172.21.15.39:0/105054984 reap_dead start 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.933+0000 7f214be00640 1 -- 172.21.15.39:0/105054984 shutdown_connections 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.933+0000 7f214be00640 1 -- 172.21.15.39:0/105054984 >> 172.21.15.39:0/105054984 conn(0x7f214c09bad0 msgr2=0x7f214c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.933+0000 7f214be00640 1 -- 172.21.15.39:0/105054984 shutdown_connections 2024-01-24T17:47:16.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.933+0000 7f214be00640 1 -- 172.21.15.39:0/105054984 wait complete. 2024-01-24T17:47:16.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.934+0000 7f214be00640 1 Processor -- start 2024-01-24T17:47:16.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.935+0000 7f214be00640 1 -- start start 2024-01-24T17:47:16.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f212c004d00 0x7f214c1f6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214c0b5e60 0x7f214c15af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214c0b7990 0x7f214c0b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c0b5bf0 con 0x7f214c0b5e60 2024-01-24T17:47:16.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c1f66b0 con 0x7f214c0b7990 2024-01-24T17:47:16.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214c1f3330 con 0x7f212c004d00 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214c0b5e60 0x7f214c15af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214c0b5e60 0x7f214c15af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49012/0 (socket says 172.21.15.39:49012) 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214a000640 1 -- 172.21.15.39:0/596804485 learned_addr learned my addr 172.21.15.39:0/596804485 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214b400640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214c0b7990 0x7f214c0b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.936+0000 7f214aa00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f212c004d00 0x7f214c1f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f214a000640 1 -- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f212c004d00 msgr2=0x7f214c1f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f214a000640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f212c004d00 0x7f214c1f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f214a000640 1 -- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214c0b7990 msgr2=0x7f214c0b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f214a000640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214c0b7990 0x7f214c0b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:16.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f214a000640 1 -- 172.21.15.39:0/596804485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c0b5630 con 0x7f214c0b5e60 2024-01-24T17:47:16.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f214a000640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214c0b5e60 0x7f214c15af20 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f213c002170 tx=0x7f213c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:16.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f2142a00640 1 -- 172.21.15.39:0/596804485 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c004d60 con 0x7f214c0b5e60 2024-01-24T17:47:16.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f2142a00640 1 -- 172.21.15.39:0/596804485 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c0070d0 con 0x7f214c0b5e60 2024-01-24T17:47:16.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f2142a00640 1 -- 172.21.15.39:0/596804485 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c007440 con 0x7f214c0b5e60 2024-01-24T17:47:16.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.937+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214c0b5910 con 0x7f214c0b5e60 2024-01-24T17:47:16.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.938+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214c0c7550 con 0x7f214c0b5e60 2024-01-24T17:47:16.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.939+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214c0c7770 con 0x7f214c0b5e60 2024-01-24T17:47:16.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.940+0000 7f2142a00640 1 -- 172.21.15.39:0/596804485 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f213c0075a0 con 0x7f214c0b5e60 2024-01-24T17:47:16.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.941+0000 7f2142a00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f212406a430 0x7f212406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:16.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.941+0000 7f2142a00640 1 -- 172.21.15.39:0/596804485 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f213c088570 con 0x7f214c0b5e60 2024-01-24T17:47:16.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.942+0000 7f214aa00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f212406a430 0x7f212406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:16.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.943+0000 7f214aa00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f212406a430 0x7f212406c8f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f21340055b0 tx=0x7f2134009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:16.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:16.946+0000 7f2142a00640 1 -- 172.21.15.39:0/596804485 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f213c053fa0 con 0x7f214c0b5e60 2024-01-24T17:47:17.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:17.216+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_conn_close"]} v 0) v1 -- 0x7f214c04dff0 con 0x7f214c0b5e60 2024-01-24T17:47:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.212+0000 7f2142a00640 1 -- 172.21.15.39:0/596804485 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_conn_close"]}]=0 v305) v1 ==== 143+0+0 (secure 0 0 0) 0x7f213c0190d0 con 0x7f214c0b5e60 2024-01-24T17:47:18.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f212406a430 msgr2=0x7f212406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:18.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f212406a430 0x7f212406c8f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f21340055b0 tx=0x7f2134009040 comp rx=0 tx=0).stop 2024-01-24T17:47:18.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214c0b5e60 msgr2=0x7f214c15af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:18.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214c0b5e60 0x7f214c15af20 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f213c002170 tx=0x7f213c00b4a0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 shutdown_connections 2024-01-24T17:47:18.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f214c0b7990 0x7f214c0b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f212406a430 0x7f212406c8f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f214c0b5e60 0x7f214c15af20 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 --2- 172.21.15.39:0/596804485 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f212c004d00 0x7f214c1f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 >> 172.21.15.39:0/596804485 conn(0x7f214c09bad0 msgr2=0x7f214c0b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:18.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 shutdown_connections 2024-01-24T17:47:18.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.215+0000 7f214be00640 1 -- 172.21.15.39:0/596804485 wait complete. 2024-01-24T17:47:18.238 INFO:tasks.cephfs_test_runner:That when a `tell` command is sent using the python CLI, ... ok 2024-01-24T17:47:18.238 INFO:tasks.cephfs_test_runner:Starting test: test_tell_session_drop (tasks.cephfs.test_sessionmap.TestSessionMap) That when a `tell` command is sent using the python CLI, 2024-01-24T17:47:18.238 INFO:tasks.cephfs_test_runner:test_tell_session_drop (tasks.cephfs.test_sessionmap.TestSessionMap) 2024-01-24T17:47:18.238 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_session_drop' 2024-01-24T17:47:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.450+0000 7fa417400640 1 Processor -- start 2024-01-24T17:47:18.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.451+0000 7fa417400640 1 -- start start 2024-01-24T17:47:18.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.452+0000 7fa417400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa4100b3e00 0x7fa4100c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:18.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.452+0000 7fa417400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4101551b0 con 0x7fa4100c8f70 2024-01-24T17:47:18.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.452+0000 7fa417400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41015e410 con 0x7fa410070540 2024-01-24T17:47:18.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.452+0000 7fa417400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41015b480 con 0x7fa4100b3e00 2024-01-24T17:47:18.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.452+0000 7fa416a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa4100c8f70 0x7fa41015e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54850/0 (socket says 172.21.15.39:54850) 2024-01-24T17:47:18.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.452+0000 7fa416a00640 1 -- 172.21.15.39:0/114769582 learned_addr learned my addr 172.21.15.39:0/114769582 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa415600640 1 --2- 172.21.15.39:0/114769582 >> v2:172.21.15.100:3300/0 conn(0x7fa4100b3e00 0x7fa4100c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231392017 0 0) 0x7fa4101551b0 con 0x7fa4100c8f70 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f0003520 con 0x7fa4100c8f70 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91138453 0 0) 0x7fa3f0003520 con 0x7fa4100c8f70 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 >> v2:172.21.15.100:3300/0 conn(0x7fa4100b3e00 msgr2=0x7fa4100c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 --2- 172.21.15.39:0/114769582 >> v2:172.21.15.100:3300/0 conn(0x7fa4100b3e00 0x7fa4100c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 >> v1:172.21.15.100:6789/0 conn(0x7fa410070540 legacy=0x7fa41015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41015b760 con 0x7fa4100c8f70 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fa408001670 con 0x7fa4100c8f70 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 >> v1:172.21.15.39:6789/0 conn(0x7fa4100c8f70 legacy=0x7fa41015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.453+0000 7fa414c00640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f00039f0 0x7fa3f0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa414c00640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3f00047c0 0x7fa3f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa414c00640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa3f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41015b480 con 0x7fa3f0005180 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f000c0b0 con 0x7fa3f00047c0 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f000c270 con 0x7fa3f00039f0 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa416a00640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa3f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa415600640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3f00047c0 0x7fa3f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa416000640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f00039f0 0x7fa3f0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa416a00640 1 -- 172.21.15.39:0/114769582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f00039f0 msgr2=0x7fa3f0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa416a00640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa3f00039f0 0x7fa3f0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa416a00640 1 -- 172.21.15.39:0/114769582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3f00047c0 msgr2=0x7fa3f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa416a00640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3f00047c0 0x7fa3f0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.454+0000 7fa416a00640 1 -- 172.21.15.39:0/114769582 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0003520 con 0x7fa3f0005180 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.455+0000 7fa416a00640 1 -- 172.21.15.39:0/114769582 reap_dead start 2024-01-24T17:47:18.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.455+0000 7fa416a00640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa3f000b980 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fa40800c0e0 tx=0x7fa3f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e11ecd03d98f45 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408005ad0 con 0x7fa3f0005180 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa408005c70 con 0x7fa3f0005180 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa414c00640 1 -- 172.21.15.39:0/114769582 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40802c810 con 0x7fa3f0005180 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa417400640 1 -- 172.21.15.39:0/114769582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 msgr2=0x7fa3f000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa417400640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa3f000b980 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fa40800c0e0 tx=0x7fa3f00057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa417400640 1 -- 172.21.15.39:0/114769582 shutdown_connections 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa417400640 1 --2- 172.21.15.39:0/114769582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa3f000b980 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa417400640 1 -- 172.21.15.39:0/114769582 >> 172.21.15.39:0/114769582 conn(0x7fa41006a340 msgr2=0x7fa4100cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:18.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.456+0000 7fa417400640 1 -- 172.21.15.39:0/114769582 shutdown_connections 2024-01-24T17:47:18.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.457+0000 7fa417400640 1 -- 172.21.15.39:0/114769582 wait complete. 2024-01-24T17:47:18.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.458+0000 7fa417400640 1 Processor -- start 2024-01-24T17:47:18.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.458+0000 7fa417400640 1 -- start start 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.459+0000 7fa417400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa410154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.459+0000 7fa417400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4101f3470 0x7fa4100c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.459+0000 7fa417400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4101e4190 0x7fa4101e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.459+0000 7fa417400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4101f2230 con 0x7fa3f0005180 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.459+0000 7fa417400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4101549d0 con 0x7fa4101f3470 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.459+0000 7fa417400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4101e4b60 con 0x7fa4101e4190 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa410154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa410154490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49026/0 (socket says 172.21.15.39:49026) 2024-01-24T17:47:18.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416000640 1 -- 172.21.15.39:0/493264044 learned_addr learned my addr 172.21.15.39:0/493264044 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa415600640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4101f3470 0x7fa4100c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416a00640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4101e4190 0x7fa4101e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416000640 1 -- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4101e4190 msgr2=0x7fa4101e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416000640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4101e4190 0x7fa4101e4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416000640 1 -- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4101f3470 msgr2=0x7fa4100c3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416000640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4101f3470 0x7fa4100c3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa416000640 1 -- 172.21.15.39:0/493264044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4100c4310 con 0x7fa3f0005180 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.460+0000 7fa415600640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4101f3470 0x7fa4100c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.461+0000 7fa416000640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa410154490 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa40000e820 tx=0x7fa40000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.461+0000 7fa40e000640 1 -- 172.21.15.39:0/493264044 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400003bd0 con 0x7fa3f0005180 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.461+0000 7fa40e000640 1 -- 172.21.15.39:0/493264044 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa400003d30 con 0x7fa3f0005180 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.461+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4100cce60 con 0x7fa3f0005180 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.461+0000 7fa40e000640 1 -- 172.21.15.39:0/493264044 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa400003ea0 con 0x7fa3f0005180 2024-01-24T17:47:18.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.461+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4100cc5c0 con 0x7fa3f0005180 2024-01-24T17:47:18.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.462+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41004dff0 con 0x7fa3f0005180 2024-01-24T17:47:18.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.463+0000 7fa40e000640 1 -- 172.21.15.39:0/493264044 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa4000052f0 con 0x7fa3f0005180 2024-01-24T17:47:18.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.464+0000 7fa40e000640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3e806a430 0x7fa3e806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:18.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.464+0000 7fa40e000640 1 -- 172.21.15.39:0/493264044 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa400088740 con 0x7fa3f0005180 2024-01-24T17:47:18.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.465+0000 7fa415600640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3e806a430 0x7fa3e806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:18.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.466+0000 7fa415600640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3e806a430 0x7fa3e806c8f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa4100b42b0 tx=0x7fa3f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.468+0000 7fa40e000640 1 -- 172.21.15.39:0/493264044 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa400054020 con 0x7fa3f0005180 2024-01-24T17:47:18.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:18.733+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_session_drop"]} v 0) v1 -- 0x7fa4100cbc10 con 0x7fa3f0005180 2024-01-24T17:47:19.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.222+0000 7fa40e000640 1 -- 172.21.15.39:0/493264044 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_session_drop"]}]=0 v306) v1 ==== 148+0+0 (secure 0 0 0) 0x7fa400001850 con 0x7fa3f0005180 2024-01-24T17:47:19.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3e806a430 msgr2=0x7fa3e806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:19.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3e806a430 0x7fa3e806c8f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa4100b42b0 tx=0x7fa3f8008040 comp rx=0 tx=0).stop 2024-01-24T17:47:19.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 msgr2=0x7fa410154490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:19.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa410154490 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa40000e820 tx=0x7fa40000ece0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 shutdown_connections 2024-01-24T17:47:19.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa4101e4190 0x7fa4101e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4101f3470 0x7fa4100c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa3e806a430 0x7fa3e806c8f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 --2- 172.21.15.39:0/493264044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3f0005180 0x7fa410154490 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 >> 172.21.15.39:0/493264044 conn(0x7fa41006a340 msgr2=0x7fa4100c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 shutdown_connections 2024-01-24T17:47:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.226+0000 7fa417400640 1 -- 172.21.15.39:0/493264044 wait complete. 2024-01-24T17:47:19.247 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-24T17:47:19.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.465+0000 7f9627e00640 1 Processor -- start 2024-01-24T17:47:19.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.465+0000 7f9627e00640 1 -- start start 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.466+0000 7f9627e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f962015aa60 0x7f96200ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.466+0000 7f9627e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96200cb280 0x7f96200cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.466+0000 7f9627e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96200ca4c0 con 0x7f962006cec0 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.466+0000 7f9627e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96200cbc50 con 0x7f96200cb280 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.466+0000 7f9627e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96200cae90 con 0x7f962015aa60 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.467+0000 7f9624a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96200cb280 0x7f96200cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.467+0000 7f9624a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f96200cb280 0x7f96200cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49030/0 (socket says 172.21.15.39:49030) 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.467+0000 7f9624a00640 1 -- 172.21.15.39:0/2498158453 learned_addr learned my addr 172.21.15.39:0/2498158453 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.467+0000 7f9625400640 1 --2- 172.21.15.39:0/2498158453 >> v2:172.21.15.100:3300/0 conn(0x7f962015aa60 0x7f96200ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.467+0000 7f9624a00640 1 -- 172.21.15.39:0/2498158453 >> v2:172.21.15.100:3300/0 conn(0x7f962015aa60 msgr2=0x7f96200ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:19.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.467+0000 7f9624a00640 1 --2- 172.21.15.39:0/2498158453 >> v2:172.21.15.100:3300/0 conn(0x7f962015aa60 0x7f96200ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.467+0000 7f9624a00640 1 -- 172.21.15.39:0/2498158453 >> v1:172.21.15.100:6789/0 conn(0x7f962006cec0 legacy=0x7f962006d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:19.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.467+0000 7f9624a00640 1 -- 172.21.15.39:0/2498158453 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962006da00 con 0x7f96200cb280 2024-01-24T17:47:19.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.468+0000 7f9624a00640 1 --2- 172.21.15.39:0/2498158453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96200cb280 0x7f96200cb710 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f962006c190 tx=0x7f960802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aec552797e5275e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:19.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f961fe00640 1 -- 172.21.15.39:0/2498158453 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608031870 con 0x7f96200cb280 2024-01-24T17:47:19.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f961fe00640 1 -- 172.21.15.39:0/2498158453 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96080319d0 con 0x7f96200cb280 2024-01-24T17:47:19.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f961fe00640 1 -- 172.21.15.39:0/2498158453 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608031d60 con 0x7f96200cb280 2024-01-24T17:47:19.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f9627e00640 1 -- 172.21.15.39:0/2498158453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96200cb280 msgr2=0x7f96200cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:19.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f9627e00640 1 --2- 172.21.15.39:0/2498158453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96200cb280 0x7f96200cb710 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f962006c190 tx=0x7f960802b910 comp rx=0 tx=0).stop 2024-01-24T17:47:19.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f9627e00640 1 -- 172.21.15.39:0/2498158453 shutdown_connections 2024-01-24T17:47:19.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f9627e00640 1 --2- 172.21.15.39:0/2498158453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96200cb280 0x7f96200cb710 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f9627e00640 1 --2- 172.21.15.39:0/2498158453 >> v2:172.21.15.100:3300/0 conn(0x7f962015aa60 0x7f96200ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.469+0000 7f9627e00640 1 -- 172.21.15.39:0/2498158453 >> 172.21.15.39:0/2498158453 conn(0x7f96200ce9e0 msgr2=0x7f96200cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:19.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.470+0000 7f9627e00640 1 -- 172.21.15.39:0/2498158453 shutdown_connections 2024-01-24T17:47:19.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.470+0000 7f9627e00640 1 -- 172.21.15.39:0/2498158453 wait complete. 2024-01-24T17:47:19.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.471+0000 7f9627e00640 1 Processor -- start 2024-01-24T17:47:19.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.471+0000 7f9627e00640 1 -- start start 2024-01-24T17:47:19.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.472+0000 7f9627e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f962006cec0 0x7f9620154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:19.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.472+0000 7f9627e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96200cb280 0x7f962015cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:19.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.472+0000 7f9627e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f962015aa60 0x7f96200b2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:19.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.472+0000 7f9627e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9620152a70 con 0x7f962006cec0 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.472+0000 7f9627e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962015d9a0 con 0x7f962015aa60 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.472+0000 7f9627e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9620154110 con 0x7f96200cb280 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f962006cec0 0x7f9620154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f962006cec0 0x7f9620154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49040/0 (socket says 172.21.15.39:49040) 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 -- 172.21.15.39:0/514706388 learned_addr learned my addr 172.21.15.39:0/514706388 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9624a00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96200cb280 0x7f962015cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625e00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f962015aa60 0x7f96200b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 -- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96200cb280 msgr2=0x7f962015cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96200cb280 0x7f962015cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 -- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f962015aa60 msgr2=0x7f96200b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f962015aa60 0x7f96200b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 -- 172.21.15.39:0/514706388 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96080094d0 con 0x7f962006cec0 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.473+0000 7f9625400640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f962006cec0 0x7f9620154bf0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f961000d800 tx=0x7f961000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:19.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.474+0000 7f961d600640 1 -- 172.21.15.39:0/514706388 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961000b6c0 con 0x7f962006cec0 2024-01-24T17:47:19.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.474+0000 7f961d600640 1 -- 172.21.15.39:0/514706388 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961000b820 con 0x7f962006cec0 2024-01-24T17:47:19.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.474+0000 7f961d600640 1 -- 172.21.15.39:0/514706388 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961000bbb0 con 0x7f962006cec0 2024-01-24T17:47:19.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.474+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96201543f0 con 0x7f962006cec0 2024-01-24T17:47:19.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.474+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962015dc80 con 0x7f962006cec0 2024-01-24T17:47:19.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.475+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962004dff0 con 0x7f962006cec0 2024-01-24T17:47:19.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.476+0000 7f961d600640 1 -- 172.21.15.39:0/514706388 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f961000bd10 con 0x7f962006cec0 2024-01-24T17:47:19.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.477+0000 7f961d600640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f95f806a430 0x7f95f806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:19.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.477+0000 7f961d600640 1 -- 172.21.15.39:0/514706388 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9610088c00 con 0x7f962006cec0 2024-01-24T17:47:19.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.477+0000 7f9624a00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f95f806a430 0x7f95f806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:19.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.479+0000 7f9624a00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f95f806a430 0x7f95f806c8f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9608036000 tx=0x7f960803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:19.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.482+0000 7f961d600640 1 -- 172.21.15.39:0/514706388 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9610054450 con 0x7f962006cec0 2024-01-24T17:47:19.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.674+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f962006c8a0 con 0x7f962006cec0 2024-01-24T17:47:19.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.681+0000 7f961d600640 1 -- 172.21.15.39:0/514706388 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9610057a70 con 0x7f962006cec0 2024-01-24T17:47:19.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f95f806a430 msgr2=0x7f95f806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:19.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f95f806a430 0x7f95f806c8f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9608036000 tx=0x7f960803c040 comp rx=0 tx=0).stop 2024-01-24T17:47:19.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f962006cec0 msgr2=0x7f9620154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:19.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f962006cec0 0x7f9620154bf0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f961000d800 tx=0x7f961000dcc0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 shutdown_connections 2024-01-24T17:47:19.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f962015aa60 0x7f96200b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96200cb280 0x7f962015cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f95f806a430 0x7f95f806c8f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 --2- 172.21.15.39:0/514706388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f962006cec0 0x7f9620154bf0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:19.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 >> 172.21.15.39:0/514706388 conn(0x7f96200ce9e0 msgr2=0x7f962006ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:19.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 shutdown_connections 2024-01-24T17:47:19.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:19.684+0000 7f9627e00640 1 -- 172.21.15.39:0/514706388 wait complete. 2024-01-24T17:47:19.704 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:47:19.705 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:47:19.826 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:47:19.946 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:47:19.946 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:47:19.946 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:20.005 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:47:20.006 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:47:20.006 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:20.062 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:47:20.062 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:47:20.062 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:20.123 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:47:20.124 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:47:20.124 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:20.179 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:47:20.179 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-24T17:47:20.179 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:47:20.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.408+0000 7fd18a200640 1 Processor -- start 2024-01-24T17:47:20.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.408+0000 7fd18a200640 1 -- start start 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.410+0000 7fd18a200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1840ca700 0x7fd18415a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.410+0000 7fd18a200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd18415a910 0x7fd18406bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.410+0000 7fd18a200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1841540f0 con 0x7fd1840ca320 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.410+0000 7fd18a200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd184159fd0 con 0x7fd1840ca700 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.410+0000 7fd18a200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18415da00 con 0x7fd18415a910 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd188e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd1840ca320 0x7fd18415d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54864/0 (socket says 172.21.15.39:54864) 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd188e00640 1 -- 172.21.15.39:0/2994937749 learned_addr learned my addr 172.21.15.39:0/2994937749 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd17bc00640 1 --2- 172.21.15.39:0/2994937749 >> v2:172.21.15.39:3300/0 conn(0x7fd1840ca700 0x7fd18415a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd189800640 1 --2- 172.21.15.39:0/2994937749 >> v2:172.21.15.100:3300/0 conn(0x7fd18415a910 0x7fd18406bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd183e00640 1 -- 172.21.15.39:0/2994937749 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15073502 0 0) 0x7fd1841540f0 con 0x7fd1840ca320 2024-01-24T17:47:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd183e00640 1 -- 172.21.15.39:0/2994937749 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd164003520 con 0x7fd1840ca320 2024-01-24T17:47:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd17bc00640 1 -- 172.21.15.39:0/2994937749 >> v2:172.21.15.100:3300/0 conn(0x7fd18415a910 msgr2=0x7fd18406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd17bc00640 1 --2- 172.21.15.39:0/2994937749 >> v2:172.21.15.100:3300/0 conn(0x7fd18415a910 0x7fd18406bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd17bc00640 1 -- 172.21.15.39:0/2994937749 >> v1:172.21.15.39:6789/0 conn(0x7fd1840ca320 legacy=0x7fd18415d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.411+0000 7fd17bc00640 1 -- 172.21.15.39:0/2994937749 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18415dce0 con 0x7fd1840ca700 2024-01-24T17:47:20.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.412+0000 7fd17bc00640 1 --2- 172.21.15.39:0/2994937749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd18415a3d0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd184070320 tx=0x7fd16c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6958af6d556930cf server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:20.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd183e00640 1 -- 172.21.15.39:0/2994937749 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c031710 con 0x7fd1840ca700 2024-01-24T17:47:20.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd183e00640 1 -- 172.21.15.39:0/2994937749 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c02f050 con 0x7fd1840ca700 2024-01-24T17:47:20.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd183e00640 1 -- 172.21.15.39:0/2994937749 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c031a70 con 0x7fd1840ca700 2024-01-24T17:47:20.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd18a200640 1 -- 172.21.15.39:0/2994937749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 msgr2=0x7fd18415a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:20.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd18a200640 1 --2- 172.21.15.39:0/2994937749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd18415a3d0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fd184070320 tx=0x7fd16c02b7b0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd18a200640 1 -- 172.21.15.39:0/2994937749 shutdown_connections 2024-01-24T17:47:20.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd18a200640 1 --2- 172.21.15.39:0/2994937749 >> v2:172.21.15.100:3300/0 conn(0x7fd18415a910 0x7fd18406bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd18a200640 1 --2- 172.21.15.39:0/2994937749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd18415a3d0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd18a200640 1 -- 172.21.15.39:0/2994937749 >> 172.21.15.39:0/2994937749 conn(0x7fd18406a340 msgr2=0x7fd1840cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:20.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd18a200640 1 -- 172.21.15.39:0/2994937749 shutdown_connections 2024-01-24T17:47:20.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.413+0000 7fd18a200640 1 -- 172.21.15.39:0/2994937749 wait complete. 2024-01-24T17:47:20.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.415+0000 7fd18a200640 1 Processor -- start 2024-01-24T17:47:20.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.415+0000 7fd18a200640 1 -- start start 2024-01-24T17:47:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.416+0000 7fd18a200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1840ca320 0x7fd18414e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.416+0000 7fd18a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd1841531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.416+0000 7fd18a200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd18415a910 0x7fd184153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.416+0000 7fd18a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd184150d20 con 0x7fd1840ca700 2024-01-24T17:47:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.416+0000 7fd18a200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18415b0b0 con 0x7fd1840ca320 2024-01-24T17:47:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.416+0000 7fd18a200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18415c890 con 0x7fd18415a910 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.416+0000 7fd17bc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd1841531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd17bc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd1841531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49064/0 (socket says 172.21.15.39:49064) 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd17bc00640 1 -- 172.21.15.39:0/3120578755 learned_addr learned my addr 172.21.15.39:0/3120578755 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd188e00640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1840ca320 0x7fd18414e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd17bc00640 1 -- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd18415a910 msgr2=0x7fd184153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd17bc00640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd18415a910 0x7fd184153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd17bc00640 1 -- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1840ca320 msgr2=0x7fd18414e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd17bc00640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1840ca320 0x7fd18414e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd17bc00640 1 -- 172.21.15.39:0/3120578755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c009400 con 0x7fd1840ca700 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd188e00640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1840ca320 0x7fd18414e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd17bc00640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd1841531c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fd16c000c00 tx=0x7fd16c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.417+0000 7fd181600640 1 -- 172.21.15.39:0/3120578755 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c033890 con 0x7fd1840ca700 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.418+0000 7fd181600640 1 -- 172.21.15.39:0/3120578755 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16c0339f0 con 0x7fd1840ca700 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.418+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd184150f30 con 0x7fd1840ca700 2024-01-24T17:47:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.418+0000 7fd181600640 1 -- 172.21.15.39:0/3120578755 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c033da0 con 0x7fd1840ca700 2024-01-24T17:47:20.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.418+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1841514f0 con 0x7fd1840ca700 2024-01-24T17:47:20.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.419+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18404dff0 con 0x7fd1840ca700 2024-01-24T17:47:20.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.420+0000 7fd181600640 1 -- 172.21.15.39:0/3120578755 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd16c03a040 con 0x7fd1840ca700 2024-01-24T17:47:20.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.421+0000 7fd181600640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:20.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.421+0000 7fd181600640 1 -- 172.21.15.39:0/3120578755 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd16c0a9cd0 con 0x7fd1840ca700 2024-01-24T17:47:20.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.421+0000 7fd188e00640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:20.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.422+0000 7fd188e00640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd174007fd0 tx=0x7fd174009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:20.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.425+0000 7fd181600640 1 -- 172.21.15.39:0/3120578755 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd16c075520 con 0x7fd1840ca700 2024-01-24T17:47:20.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.647+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd184071840 con 0x7fd1840ca700 2024-01-24T17:47:20.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.648+0000 7fd181600640 1 -- 172.21.15.39:0/3120578755 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fd16c04cde0 con 0x7fd1840ca700 2024-01-24T17:47:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:20.649 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5510,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/2035474662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2035474662},{"type":"v1","addr":"172.21.15.100:6836","nonce":2035474662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5845,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1886779141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1886779141},{"type":"v1","addr":"172.21.15.39:6833","nonce":1886779141}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":5854,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/2624477336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":2624477336},{"type":"v1","addr":"172.21.15.100:6837","nonce":2624477336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2465319794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2465319794},{"type":"v1","addr":"172.21.15.39:6835","nonce":2465319794}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[]} 2024-01-24T17:47:20.650 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 44 2024-01-24T17:47:20.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 msgr2=0x7fd15c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:20.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd174007fd0 tx=0x7fd174009040 comp rx=0 tx=0).stop 2024-01-24T17:47:20.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 msgr2=0x7fd1841531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:20.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd1841531c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fd16c000c00 tx=0x7fd16c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 shutdown_connections 2024-01-24T17:47:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd18415a910 0x7fd184153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840ca700 0x7fd1841531c0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 --2- 172.21.15.39:0/3120578755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1840ca320 0x7fd18414e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 >> 172.21.15.39:0/3120578755 conn(0x7fd18406a340 msgr2=0x7fd18406bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 shutdown_connections 2024-01-24T17:47:20.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:20.651+0000 7fd18a200640 1 -- 172.21.15.39:0/3120578755 wait complete. 2024-01-24T17:47:20.669 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-24T17:47:20.669 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-24T17:47:20.669 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-24T17:47:20.670 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:47:20.670 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-24T17:47:20.670 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-24T17:47:20.670 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-24T17:47:20.670 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:47:20.670 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-24T17:47:20.670 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-24T17:47:20.671 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-24T17:47:20.671 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:47:20.671 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-24T17:47:20.671 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-24T17:47:20.671 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-24T17:47:20.671 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:47:20.687 INFO:tasks.ceph.mds.a:Stopped 2024-01-24T17:47:20.687 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-24T17:47:20.690 INFO:tasks.ceph.mds.b:Stopped 2024-01-24T17:47:20.690 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-24T17:47:20.695 INFO:tasks.ceph.mds.d:Stopped 2024-01-24T17:47:20.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-24T17:47:20.735 INFO:tasks.ceph.mds.a:Started 2024-01-24T17:47:20.736 DEBUG:teuthology.parallel:result is None 2024-01-24T17:47:20.742 INFO:tasks.ceph.mds.b:Started 2024-01-24T17:47:20.742 DEBUG:teuthology.parallel:result is None 2024-01-24T17:47:20.748 INFO:tasks.ceph.mds.d:Started 2024-01-24T17:47:20.748 DEBUG:teuthology.parallel:result is None 2024-01-24T17:47:20.904 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:47:20.905 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:47:20.925 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:47:21.248 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:47:21.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:47:22.342 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-01-24T17:47:26.687 INFO:tasks.ceph.mds.c:Stopped 2024-01-24T17:47:26.688 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-24T17:47:26.696 INFO:tasks.ceph.mds.c:Started 2024-01-24T17:47:26.696 DEBUG:teuthology.parallel:result is None 2024-01-24T17:47:26.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-24T17:47:26.874 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:47:26.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.907+0000 7f24a0e00640 1 Processor -- start 2024-01-24T17:47:26.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.907+0000 7f24a0e00640 1 -- start start 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.908+0000 7f24a0e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f249c15cbe0 0x7f249c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f24a0e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f249c0ce510 0x7f249c0d0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f24a0e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0d1800 con 0x7f249c0d13d0 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f24a0e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249c0b8820 con 0x7f249c15cbe0 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f24a0e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249c0b5790 con 0x7f249c0ce510 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f2499e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f249c15cbe0 0x7f249c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f2499e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f249c15cbe0 0x7f249c0cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36440/0 (socket says 172.21.15.39:36440) 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f2499e00640 1 -- 172.21.15.39:0/2653142816 learned_addr learned my addr 172.21.15.39:0/2653142816 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:26.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f2499400640 1 --2- 172.21.15.39:0/2653142816 >> v2:172.21.15.100:3300/0 conn(0x7f249c0ce510 0x7f249c0d0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:26.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f2499e00640 1 -- 172.21.15.39:0/2653142816 >> v2:172.21.15.100:3300/0 conn(0x7f249c0ce510 msgr2=0x7f249c0d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:26.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f2499e00640 1 --2- 172.21.15.39:0/2653142816 >> v2:172.21.15.100:3300/0 conn(0x7f249c0ce510 0x7f249c0d0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:26.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f2499e00640 1 -- 172.21.15.39:0/2653142816 >> v1:172.21.15.100:6789/0 conn(0x7f249c0d13d0 legacy=0x7f249c160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:26.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.909+0000 7f2499e00640 1 -- 172.21.15.39:0/2653142816 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c0b5a50 con 0x7f249c15cbe0 2024-01-24T17:47:26.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.910+0000 7f2499e00640 1 --2- 172.21.15.39:0/2653142816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cdfd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f249c08bdb0 tx=0x7f248c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50537247ebc64ec server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:26.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f2498a00640 1 -- 172.21.15.39:0/2653142816 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c004230 con 0x7f249c15cbe0 2024-01-24T17:47:26.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f2498a00640 1 -- 172.21.15.39:0/2653142816 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c004390 con 0x7f249c15cbe0 2024-01-24T17:47:26.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f2498a00640 1 -- 172.21.15.39:0/2653142816 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c003810 con 0x7f249c15cbe0 2024-01-24T17:47:26.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f24a0e00640 1 -- 172.21.15.39:0/2653142816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 msgr2=0x7f249c0cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:26.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f24a0e00640 1 --2- 172.21.15.39:0/2653142816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cdfd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f249c08bdb0 tx=0x7f248c02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:47:26.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f24a0e00640 1 -- 172.21.15.39:0/2653142816 shutdown_connections 2024-01-24T17:47:26.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f24a0e00640 1 --2- 172.21.15.39:0/2653142816 >> v2:172.21.15.100:3300/0 conn(0x7f249c0ce510 0x7f249c0d0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:26.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f24a0e00640 1 --2- 172.21.15.39:0/2653142816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cdfd0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:26.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.911+0000 7f24a0e00640 1 -- 172.21.15.39:0/2653142816 >> 172.21.15.39:0/2653142816 conn(0x7f249c09bc80 msgr2=0x7f249c0d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:26.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.912+0000 7f24a0e00640 1 -- 172.21.15.39:0/2653142816 shutdown_connections 2024-01-24T17:47:26.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.912+0000 7f24a0e00640 1 -- 172.21.15.39:0/2653142816 wait complete. 2024-01-24T17:47:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.913+0000 7f24a0e00640 1 Processor -- start 2024-01-24T17:47:26.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.913+0000 7f24a0e00640 1 -- start start 2024-01-24T17:47:26.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.914+0000 7f24a0e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f249c0ce510 0x7f249c150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:26.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.914+0000 7f24a0e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249c0d13d0 0x7f249c1542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:26.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.914+0000 7f24a0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.914+0000 7f24a0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c156240 con 0x7f249c15cbe0 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.914+0000 7f24a0e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c155ef0 con 0x7f249c0ce510 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.914+0000 7f24a0e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0c7e20 con 0x7f249c0d13d0 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36446/0 (socket says 172.21.15.39:36446) 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 -- 172.21.15.39:0/1673933068 learned_addr learned my addr 172.21.15.39:0/1673933068 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f2499e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f249c0ce510 0x7f249c150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f2499400640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249c0d13d0 0x7f249c1542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 -- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249c0d13d0 msgr2=0x7f249c1542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249c0d13d0 0x7f249c1542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 -- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f249c0ce510 msgr2=0x7f249c150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f249c0ce510 0x7f249c150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 -- 172.21.15.39:0/1673933068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c009460 con 0x7f249c15cbe0 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.915+0000 7f249a800640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cc440 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f249400c780 tx=0x7f249400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.916+0000 7f2492000640 1 -- 172.21.15.39:0/1673933068 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2494003d80 con 0x7f249c15cbe0 2024-01-24T17:47:26.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.916+0000 7f2492000640 1 -- 172.21.15.39:0/1673933068 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249400ce90 con 0x7f249c15cbe0 2024-01-24T17:47:26.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.916+0000 7f2492000640 1 -- 172.21.15.39:0/1673933068 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24940052b0 con 0x7f249c15cbe0 2024-01-24T17:47:26.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.916+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c0c8100 con 0x7f249c15cbe0 2024-01-24T17:47:26.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.916+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c0c7a40 con 0x7f249c15cbe0 2024-01-24T17:47:26.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.917+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249c156380 con 0x7f249c15cbe0 2024-01-24T17:47:26.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.918+0000 7f2492000640 1 -- 172.21.15.39:0/1673933068 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2494005410 con 0x7f249c15cbe0 2024-01-24T17:47:26.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.919+0000 7f2492000640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f247406a430 0x7f247406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:26.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.919+0000 7f2492000640 1 -- 172.21.15.39:0/1673933068 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2494087b60 con 0x7f249c15cbe0 2024-01-24T17:47:26.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.920+0000 7f2499e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f247406a430 0x7f247406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:26.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.921+0000 7f2499e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f247406a430 0x7f247406c8f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f248c035000 tx=0x7f248c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:26.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:26.924+0000 7f2492000640 1 -- 172.21.15.39:0/1673933068 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f249c156380 con 0x7f249c15cbe0 2024-01-24T17:47:27.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.129+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f249c156380 con 0x7f249c15cbe0 2024-01-24T17:47:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.900+0000 7f2492000640 1 -- 172.21.15.39:0/1673933068 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v84) v1 ==== 97+0+0 (secure 0 0 0) 0x7f249c156380 con 0x7f249c15cbe0 2024-01-24T17:47:27.902 INFO:teuthology.orchestra.run.smithi039.stderr: removed all blocklist entries 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f247406a430 msgr2=0x7f247406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f247406a430 0x7f247406c8f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f248c035000 tx=0x7f248c008030 comp rx=0 tx=0).stop 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 msgr2=0x7f249c0cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cc440 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f249400c780 tx=0x7f249400cc40 comp rx=0 tx=0).stop 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 shutdown_connections 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f247406a430 0x7f247406c8f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249c15cbe0 0x7f249c0cc440 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249c0d13d0 0x7f249c1542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 --2- 172.21.15.39:0/1673933068 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f249c0ce510 0x7f249c150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 >> 172.21.15.39:0/1673933068 conn(0x7f249c09bc80 msgr2=0x7f249c15bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.904+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 shutdown_connections 2024-01-24T17:47:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:27.905+0000 7f24a0e00640 1 -- 172.21.15.39:0/1673933068 wait complete. 2024-01-24T17:47:27.925 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-24T17:47:28.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.141+0000 7f0468400640 1 Processor -- start 2024-01-24T17:47:28.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.142+0000 7f0468400640 1 -- start start 2024-01-24T17:47:28.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.143+0000 7f0468400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f04600b0010 0x7f046015bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.143+0000 7f0468400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f046015f390 0x7f04600c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.143+0000 7f0468400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0460070470 con 0x7f0460070920 2024-01-24T17:47:28.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.143+0000 7f0468400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046015ef00 con 0x7f04600b0010 2024-01-24T17:47:28.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.143+0000 7f0468400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04600710a0 con 0x7f046015f390 2024-01-24T17:47:28.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.143+0000 7f0465000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0460070920 0x7f04600b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49040/0 (socket says 172.21.15.39:49040) 2024-01-24T17:47:28.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.143+0000 7f0465000640 1 -- 172.21.15.39:0/2581491077 learned_addr learned my addr 172.21.15.39:0/2581491077 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f0465a00640 1 --2- 172.21.15.39:0/2581491077 >> v2:172.21.15.100:3300/0 conn(0x7f04600b0010 0x7f046015bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f0466400640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f046015f390 0x7f04600c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185889810 0 0) 0x7f0460070470 con 0x7f0460070920 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0440003520 con 0x7f0460070920 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290554955 0 0) 0x7f0440003520 con 0x7f0460070920 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f046015f390 msgr2=0x7f04600c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f046015f390 0x7f04600c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 >> v2:172.21.15.100:3300/0 conn(0x7f04600b0010 msgr2=0x7f046015bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 --2- 172.21.15.39:0/2581491077 >> v2:172.21.15.100:3300/0 conn(0x7f04600b0010 0x7f046015bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046015c0d0 con 0x7f0460070920 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f0448001670 con 0x7f0460070920 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 >> v1:172.21.15.39:6789/0 conn(0x7f0460070920 legacy=0x7f04600b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0440003970 0x7f0440003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f046015f390 0x7f04400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.144+0000 7f045fe00640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f0440005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046015ef00 con 0x7f0440004d00 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04600710a0 con 0x7f0440003970 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044000b980 con 0x7f046015f390 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f0466400640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f0440005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f0465a00640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0440003970 0x7f0440003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f0465000640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f046015f390 0x7f04400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f0466400640 1 -- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f046015f390 msgr2=0x7f04400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:28.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f0466400640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f046015f390 0x7f04400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f0466400640 1 -- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0440003970 msgr2=0x7f0440003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:28.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f0466400640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0440003970 0x7f0440003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.145+0000 7f0466400640 1 -- 172.21.15.39:0/2581491077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0440003520 con 0x7f0440004d00 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.146+0000 7f0466400640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f0440005180 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0458009fc0 tx=0x7f04400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9038b9cebb68f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.147+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045802cc20 con 0x7f0440004d00 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.147+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045800f040 con 0x7f0440004d00 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.147+0000 7f045fe00640 1 -- 172.21.15.39:0/2581491077 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458034440 con 0x7f0440004d00 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.147+0000 7f0468400640 1 -- 172.21.15.39:0/2581491077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 msgr2=0x7f0440005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.147+0000 7f0468400640 1 --2- 172.21.15.39:0/2581491077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f0440005180 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0458009fc0 tx=0x7f04400057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.148+0000 7f0466400640 1 -- 172.21.15.39:0/2581491077 reap_dead start 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.148+0000 7f0468400640 1 -- 172.21.15.39:0/2581491077 shutdown_connections 2024-01-24T17:47:28.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.148+0000 7f0468400640 1 -- 172.21.15.39:0/2581491077 >> 172.21.15.39:0/2581491077 conn(0x7f046006bf20 msgr2=0x7f046006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:28.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.148+0000 7f0468400640 1 -- 172.21.15.39:0/2581491077 shutdown_connections 2024-01-24T17:47:28.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.148+0000 7f0468400640 1 -- 172.21.15.39:0/2581491077 wait complete. 2024-01-24T17:47:28.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.150+0000 7f0468400640 1 Processor -- start 2024-01-24T17:47:28.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.151+0000 7f0468400640 1 -- start start 2024-01-24T17:47:28.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.151+0000 7f0468400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f04600c48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.151+0000 7f0468400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04601f35c0 0x7f04601f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0468400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0460146310 0x7f0460146790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0468400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04601f3f80 con 0x7f0440004d00 2024-01-24T17:47:28.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0468400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460147040 con 0x7f0460146310 2024-01-24T17:47:28.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0468400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0460147320 con 0x7f04601f35c0 2024-01-24T17:47:28.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0465a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f04600c48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0465a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f04600c48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36452/0 (socket says 172.21.15.39:36452) 2024-01-24T17:47:28.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0465a00640 1 -- 172.21.15.39:0/1007121921 learned_addr learned my addr 172.21.15.39:0/1007121921 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:28.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0465a00640 1 -- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04601f35c0 msgr2=0x7f04601f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0465a00640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04601f35c0 0x7f04601f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0465a00640 1 -- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0460146310 msgr2=0x7f0460146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0465a00640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0460146310 0x7f0460146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.152+0000 7f0465a00640 1 -- 172.21.15.39:0/1007121921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460147600 con 0x7f0440004d00 2024-01-24T17:47:28.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.153+0000 7f0465a00640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f04600c48f0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0450011600 tx=0x7f0450011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:28.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.153+0000 7f045d600640 1 -- 172.21.15.39:0/1007121921 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450015870 con 0x7f0440004d00 2024-01-24T17:47:28.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.153+0000 7f045d600640 1 -- 172.21.15.39:0/1007121921 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04500159d0 con 0x7f0440004d00 2024-01-24T17:47:28.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.153+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04601478e0 con 0x7f0440004d00 2024-01-24T17:47:28.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.153+0000 7f045d600640 1 -- 172.21.15.39:0/1007121921 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450015d40 con 0x7f0440004d00 2024-01-24T17:47:28.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.153+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04600c4e30 con 0x7f0440004d00 2024-01-24T17:47:28.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.154+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046004dff0 con 0x7f0440004d00 2024-01-24T17:47:28.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.155+0000 7f045d600640 1 -- 172.21.15.39:0/1007121921 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f045000f490 con 0x7f0440004d00 2024-01-24T17:47:28.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.156+0000 7f045d600640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f043806a430 0x7f043806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.156+0000 7f045d600640 1 -- 172.21.15.39:0/1007121921 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0450088e90 con 0x7f0440004d00 2024-01-24T17:47:28.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.157+0000 7f0465000640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f043806a430 0x7f043806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.158+0000 7f0465000640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f043806a430 0x7f043806c8f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0448001050 tx=0x7f0448000fe0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:28.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.161+0000 7f045d600640 1 -- 172.21.15.39:0/1007121921 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0450054730 con 0x7f0440004d00 2024-01-24T17:47:28.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.427+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f04600b06a0 con 0x7f0440004d00 2024-01-24T17:47:28.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.428+0000 7f045d600640 1 -- 172.21.15.39:0/1007121921 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0450057d50 con 0x7f0440004d00 2024-01-24T17:47:28.429 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:28.429 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:47:28.429 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_dump": [ 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.a", 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlZZEtLBAAUxf4chL3tgy3zHlZoraXlg==", 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.430 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.b", 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Fln4utEhAA52ngJFqrdfa9bx56mVhRVA==", 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.c", 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flo8iGARAAf+29RXWr4RHkILIz7em3mA==", 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:47:28.431 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.d", 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flh+08IRAARhZxy//j8c9b3Hg9uV9ajg==", 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:47:28.432 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.0", 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCQS7FlOj/pLBAAzi89tAacOfDKdlU0BUJvvA==", 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.1", 2024-01-24T17:47:28.433 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCUS7FlbRAKDRAAAuAy47MDTtvmh8qNA3ZLTg==", 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.2", 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCXS7Fl7QmRGRAAVECH7oYTdvDPSHBy9WmHwA==", 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.3", 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCaS7FlCQHIMxAA9OchsMzsKFxs110SeJ3Atw==", 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.4", 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCiS7FlDsBKBRAAXF2HafraT5nRbB5tBRUgWQ==", 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:47:28.435 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.5", 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQClS7FlfrXBHhAAy8GP3TLX/iok3OYPGTP5Mw==", 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.6", 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCoS7Fla11hNhAA/3oMP7cFtPSTdJl7eSnPTA==", 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.7", 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCsS7FlKFMtEhAAPmaFzr5XHVQaKZYkJiIarA==", 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:47:28.437 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.0", 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlTi/3LhAAc8FwXtLH2XjlFMO7wDMcSQ==", 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow r", 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.1", 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlH5s7ChAAlFfFDziTgfv4ibWO08N+4Q==", 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.2", 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlIISlOhAAoVxxez7UjXDxyHQNNPnn2g==", 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.439 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.3", 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlO/U/FBAA2pI9aGSS2cZ3AMnn+W1fQA==", 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:47:28.440 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.admin", 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCGS7FlxKo7HhAAlu037DIc1WmWkz81BG3j7Q==", 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.441 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.x", 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlRbF6HxAAoFjlyWb5oqIkC3L1iPRmHw==", 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.y", 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7Fl512kFRAAdCpU+QjRmGJmMD5uTdFmHQ==", 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:47:28.442 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:47:28.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:28.443 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:47:28.443 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:47:28.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f043806a430 msgr2=0x7f043806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f043806a430 0x7f043806c8f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0448001050 tx=0x7f0448000fe0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 msgr2=0x7f04600c48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f04600c48f0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0450011600 tx=0x7f0450011ac0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 shutdown_connections 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0460146310 0x7f0460146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04601f35c0 0x7f04601f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f043806a430 0x7f043806c8f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 --2- 172.21.15.39:0/1007121921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0440004d00 0x7f04600c48f0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 >> 172.21.15.39:0/1007121921 conn(0x7f046006bf20 msgr2=0x7f046015d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 shutdown_connections 2024-01-24T17:47:28.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.431+0000 7f0468400640 1 -- 172.21.15.39:0/1007121921 wait complete. 2024-01-24T17:47:28.452 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.668+0000 7fbe14800640 1 Processor -- start 2024-01-24T17:47:28.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.669+0000 7fbe14800640 1 -- start start 2024-01-24T17:47:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.670+0000 7fbe14800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbe0c15cbe0 0x7fbe0c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.670+0000 7fbe14800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbe0c0ce510 0x7fbe0c0d0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.670+0000 7fbe14800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c0d1800 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.670+0000 7fbe14800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0c0b8820 con 0x7fbe0c0ce510 2024-01-24T17:47:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.670+0000 7fbe14800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0c0b5790 con 0x7fbe0c15cbe0 2024-01-24T17:47:28.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.671+0000 7fbe11400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbe0c0ce510 0x7fbe0c0d0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.671+0000 7fbe11400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbe0c0ce510 0x7fbe0c0d0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36462/0 (socket says 172.21.15.39:36462) 2024-01-24T17:47:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.671+0000 7fbe11400640 1 -- 172.21.15.39:0/1037811548 learned_addr learned my addr 172.21.15.39:0/1037811548 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.671+0000 7fbe11e00640 1 --2- 172.21.15.39:0/1037811548 >> v2:172.21.15.100:3300/0 conn(0x7fbe0c15cbe0 0x7fbe0c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.671+0000 7fbe11400640 1 -- 172.21.15.39:0/1037811548 >> v2:172.21.15.100:3300/0 conn(0x7fbe0c15cbe0 msgr2=0x7fbe0c0cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.671+0000 7fbe11400640 1 --2- 172.21.15.39:0/1037811548 >> v2:172.21.15.100:3300/0 conn(0x7fbe0c15cbe0 0x7fbe0c0cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.671+0000 7fbe11400640 1 -- 172.21.15.39:0/1037811548 >> v1:172.21.15.100:6789/0 conn(0x7fbe0c0d13d0 legacy=0x7fbe0c160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.671+0000 7fbe11400640 1 -- 172.21.15.39:0/1037811548 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c0b5a50 con 0x7fbe0c0ce510 2024-01-24T17:47:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.672+0000 7fbe11400640 1 --2- 172.21.15.39:0/1037811548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0ce510 0x7fbe0c0d0e90 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c08baa0 tx=0x7fbdf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c372acd52d9a4d8a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe10a00640 1 -- 172.21.15.39:0/1037811548 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4031870 con 0x7fbe0c0ce510 2024-01-24T17:47:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe10a00640 1 -- 172.21.15.39:0/1037811548 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf40319d0 con 0x7fbe0c0ce510 2024-01-24T17:47:28.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe10a00640 1 -- 172.21.15.39:0/1037811548 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4031d60 con 0x7fbe0c0ce510 2024-01-24T17:47:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe14800640 1 -- 172.21.15.39:0/1037811548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0ce510 msgr2=0x7fbe0c0d0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe14800640 1 --2- 172.21.15.39:0/1037811548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0ce510 0x7fbe0c0d0e90 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c08baa0 tx=0x7fbdf402b910 comp rx=0 tx=0).stop 2024-01-24T17:47:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe14800640 1 -- 172.21.15.39:0/1037811548 shutdown_connections 2024-01-24T17:47:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe14800640 1 --2- 172.21.15.39:0/1037811548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0ce510 0x7fbe0c0d0e90 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe14800640 1 --2- 172.21.15.39:0/1037811548 >> v2:172.21.15.100:3300/0 conn(0x7fbe0c15cbe0 0x7fbe0c0cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.673+0000 7fbe14800640 1 -- 172.21.15.39:0/1037811548 >> 172.21.15.39:0/1037811548 conn(0x7fbe0c09bc80 msgr2=0x7fbe0c0d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:28.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.674+0000 7fbe14800640 1 -- 172.21.15.39:0/1037811548 shutdown_connections 2024-01-24T17:47:28.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.674+0000 7fbe14800640 1 -- 172.21.15.39:0/1037811548 wait complete. 2024-01-24T17:47:28.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.675+0000 7fbe14800640 1 Processor -- start 2024-01-24T17:47:28.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.675+0000 7fbe14800640 1 -- start start 2024-01-24T17:47:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.676+0000 7fbe14800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe0c0ce510 0x7fbe0c0c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.676+0000 7fbe14800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0d13d0 0x7fbe0c15b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.676+0000 7fbe14800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe0c15cbe0 0x7fbe0c156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.676+0000 7fbe14800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c158a50 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.676+0000 7fbe14800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c15a960 con 0x7fbe0c0ce510 2024-01-24T17:47:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.676+0000 7fbe14800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c0c7220 con 0x7fbe0c15cbe0 2024-01-24T17:47:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0d13d0 0x7fbe0c15b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0d13d0 0x7fbe0c15b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36474/0 (socket says 172.21.15.39:36474) 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11400640 1 -- 172.21.15.39:0/3324762049 learned_addr learned my addr 172.21.15.39:0/3324762049 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11e00640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe0c0ce510 0x7fbe0c0c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe12800640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe0c15cbe0 0x7fbe0c156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11400640 1 -- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe0c15cbe0 msgr2=0x7fbe0c156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11400640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe0c15cbe0 0x7fbe0c156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11400640 1 -- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe0c0ce510 msgr2=0x7fbe0c0c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11400640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe0c0ce510 0x7fbe0c0c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11400640 1 -- 172.21.15.39:0/3324762049 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40094d0 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.677+0000 7fbe11e00640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe0c0ce510 0x7fbe0c0c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:28.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.678+0000 7fbe11400640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0d13d0 0x7fbe0c15b050 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4000c00 tx=0x7fbdf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.678+0000 7fbe0a000640 1 -- 172.21.15.39:0/3324762049 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf403abe0 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.678+0000 7fbe0a000640 1 -- 172.21.15.39:0/3324762049 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf403ad40 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.678+0000 7fbe0a000640 1 -- 172.21.15.39:0/3324762049 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4044610 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.678+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c0c7500 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.678+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c0c7ac0 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.679+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0c04dff0 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.681+0000 7fbe0a000640 1 -- 172.21.15.39:0/3324762049 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbdf404d450 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.681+0000 7fbe0a000640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbde406a430 0x7fbde406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:28.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.681+0000 7fbe0a000640 1 -- 172.21.15.39:0/3324762049 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbdf402d040 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.682+0000 7fbe11e00640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbde406a430 0x7fbde406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:28.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.684+0000 7fbe11e00640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbde406a430 0x7fbde406c8f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009820 tx=0x7fbdfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:28.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.693+0000 7fbe0a000640 1 -- 172.21.15.39:0/3324762049 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbdf40765a0 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.889+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe0c158b90 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.892+0000 7fbe0a000640 1 -- 172.21.15.39:0/3324762049 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+12390 (secure 0 0 0) 0x7fbdf4079bc0 con 0x7fbe0c0d13d0 2024-01-24T17:47:28.892 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:28.893 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":84,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:27.892051+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:28.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbde406a430 msgr2=0x7fbde406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbde406a430 0x7fbde406c8f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009820 tx=0x7fbdfc008040 comp rx=0 tx=0).stop 2024-01-24T17:47:28.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0d13d0 msgr2=0x7fbe0c15b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:28.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0d13d0 0x7fbe0c15b050 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4000c00 tx=0x7fbdf40046c0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 shutdown_connections 2024-01-24T17:47:28.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe0c15cbe0 0x7fbe0c156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbde406a430 0x7fbde406c8f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe0c0d13d0 0x7fbe0c15b050 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 --2- 172.21.15.39:0/3324762049 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe0c0ce510 0x7fbe0c0c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:28.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.894+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 >> 172.21.15.39:0/3324762049 conn(0x7fbe0c09bc80 msgr2=0x7fbe0c06c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:28.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.895+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 shutdown_connections 2024-01-24T17:47:28.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:28.895+0000 7fbe14800640 1 -- 172.21.15.39:0/3324762049 wait complete. 2024-01-24T17:47:28.915 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:47:28.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:47:29.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-01-24T17:47:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.144+0000 7f6388a00640 1 Processor -- start 2024-01-24T17:47:29.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.144+0000 7f6388a00640 1 -- start start 2024-01-24T17:47:29.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.146+0000 7f6388a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 0x7f63840b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.146+0000 7f6388a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6384155170 con 0x7f63840ce8c0 2024-01-24T17:47:29.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.146+0000 7f6388a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638415b1e0 con 0x7f6384070460 2024-01-24T17:47:29.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.146+0000 7f6388a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63840b12b0 con 0x7f638415ac70 2024-01-24T17:47:29.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.146+0000 7f6383400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f63840ce8c0 0x7f638406fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49050/0 (socket says 172.21.15.39:49050) 2024-01-24T17:47:29.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.146+0000 7f6383400640 1 -- 172.21.15.39:0/2375932845 learned_addr learned my addr 172.21.15.39:0/2375932845 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:29.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6383e00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 0x7f63840b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940688316 0 0) 0x7f6384155170 con 0x7f63840ce8c0 2024-01-24T17:47:29.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6364003520 con 0x7f63840ce8c0 2024-01-24T17:47:29.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727726135 0 0) 0x7f6364003520 con 0x7f63840ce8c0 2024-01-24T17:47:29.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 msgr2=0x7f63840b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:29.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 0x7f63840b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 >> v1:172.21.15.100:6789/0 conn(0x7f6384070460 legacy=0x7f63840b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638415b4c0 con 0x7f63840ce8c0 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f6374001670 con 0x7f63840ce8c0 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 >> v1:172.21.15.39:6789/0 conn(0x7f63840ce8c0 legacy=0x7f638406fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63640039a0 0x7f6364003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 0x7f63640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6364004d00 0x7f6364005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636400ba30 con 0x7f6364004d00 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636400bb70 con 0x7f63640039a0 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.147+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636400bd30 con 0x7f638415ac70 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f6383e00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6364004d00 0x7f6364005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f6383400640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63640039a0 0x7f6364003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f637be00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 0x7f63640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f6383e00640 1 -- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 msgr2=0x7f63640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:29.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f6383e00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 0x7f63640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f6383e00640 1 -- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63640039a0 msgr2=0x7f6364003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:29.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f6383e00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63640039a0 0x7f6364003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f6383e00640 1 -- 172.21.15.39:0/2375932845 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364003520 con 0x7f6364004d00 2024-01-24T17:47:29.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.148+0000 7f637be00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f638415ac70 0x7f63640047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:29.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.149+0000 7f6383e00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6364004d00 0x7f6364005180 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f637c009f90 tx=0x7f63640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee757bf60c687dd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:29.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.149+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c02cbd0 con 0x7f6364004d00 2024-01-24T17:47:29.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c00f040 con 0x7f6364004d00 2024-01-24T17:47:29.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6382a00640 1 -- 172.21.15.39:0/2375932845 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c034440 con 0x7f6364004d00 2024-01-24T17:47:29.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6388a00640 1 -- 172.21.15.39:0/2375932845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6364004d00 msgr2=0x7f6364005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6388a00640 1 --2- 172.21.15.39:0/2375932845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6364004d00 0x7f6364005180 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f637c009f90 tx=0x7f63640057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6383e00640 1 -- 172.21.15.39:0/2375932845 reap_dead start 2024-01-24T17:47:29.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6388a00640 1 -- 172.21.15.39:0/2375932845 shutdown_connections 2024-01-24T17:47:29.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6388a00640 1 -- 172.21.15.39:0/2375932845 >> 172.21.15.39:0/2375932845 conn(0x7f638406a200 msgr2=0x7f63840affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:29.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6388a00640 1 -- 172.21.15.39:0/2375932845 shutdown_connections 2024-01-24T17:47:29.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.150+0000 7f6388a00640 1 -- 172.21.15.39:0/2375932845 wait complete. 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.152+0000 7f6388a00640 1 Processor -- start 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.152+0000 7f6388a00640 1 -- start start 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.153+0000 7f6388a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6364004d00 0x7f63840cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.153+0000 7f6388a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63841f52a0 0x7f63840cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.153+0000 7f6388a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63840c8030 0x7f6384159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.153+0000 7f6388a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638414a150 con 0x7f63840c8030 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.153+0000 7f6388a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63840cdf30 con 0x7f63841f52a0 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.153+0000 7f6388a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63841f5730 con 0x7f6364004d00 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63840c8030 0x7f6384159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63840c8030 0x7f6384159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36490/0 (socket says 172.21.15.39:36490) 2024-01-24T17:47:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 -- 172.21.15.39:0/2800176344 learned_addr learned my addr 172.21.15.39:0/2800176344 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383400640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6364004d00 0x7f63840cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f637be00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63841f52a0 0x7f63840cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 -- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6364004d00 msgr2=0x7f63840cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6364004d00 0x7f63840cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 -- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63841f52a0 msgr2=0x7f63840cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63841f52a0 0x7f63840cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 -- 172.21.15.39:0/2800176344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63841f5a10 con 0x7f63840c8030 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383400640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6364004d00 0x7f63840cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f637be00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63841f52a0 0x7f63840cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:29.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.154+0000 7f6383e00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63840c8030 0x7f6384159740 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f637c037040 tx=0x7f637c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.155+0000 7f637b400640 1 -- 172.21.15.39:0/2800176344 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c043070 con 0x7f63840c8030 2024-01-24T17:47:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.155+0000 7f637b400640 1 -- 172.21.15.39:0/2800176344 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c02ce30 con 0x7f63840c8030 2024-01-24T17:47:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.155+0000 7f637b400640 1 -- 172.21.15.39:0/2800176344 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c034d10 con 0x7f63840c8030 2024-01-24T17:47:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.155+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63841f5cf0 con 0x7f63840c8030 2024-01-24T17:47:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.155+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63840c84c0 con 0x7f63840c8030 2024-01-24T17:47:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.156+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63840738c0 con 0x7f63840c8030 2024-01-24T17:47:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.157+0000 7f637b400640 1 -- 172.21.15.39:0/2800176344 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f637c03e440 con 0x7f63840c8030 2024-01-24T17:47:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.158+0000 7f637b400640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f635c06a430 0x7f635c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.158+0000 7f637b400640 1 -- 172.21.15.39:0/2800176344 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f637c0a9fa0 con 0x7f63840c8030 2024-01-24T17:47:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.159+0000 7f6383400640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f635c06a430 0x7f635c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.160+0000 7f6383400640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f635c06a430 0x7f635c06c8f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6374001eb0 tx=0x7f6374001d80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:29.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.162+0000 7f637b400640 1 -- 172.21.15.39:0/2800176344 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f637c0758c0 con 0x7f63840c8030 2024-01-24T17:47:29.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.358+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63840c8650 con 0x7f63840c8030 2024-01-24T17:47:29.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.359+0000 7f637b400640 1 -- 172.21.15.39:0/2800176344 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63840c8650 con 0x7f63840c8030 2024-01-24T17:47:29.359 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:47:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f635c06a430 msgr2=0x7f635c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f635c06a430 0x7f635c06c8f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6374001eb0 tx=0x7f6374001d80 comp rx=0 tx=0).stop 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63840c8030 msgr2=0x7f6384159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63840c8030 0x7f6384159740 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f637c037040 tx=0x7f637c009750 comp rx=0 tx=0).stop 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 shutdown_connections 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f635c06a430 0x7f635c06c8f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63840c8030 0x7f6384159740 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63841f52a0 0x7f63840cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 --2- 172.21.15.39:0/2800176344 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6364004d00 0x7f63840cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.361+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 >> 172.21.15.39:0/2800176344 conn(0x7f638406a200 msgr2=0x7f63840af470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.362+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 shutdown_connections 2024-01-24T17:47:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.362+0000 7f6388a00640 1 -- 172.21.15.39:0/2800176344 wait complete. 2024-01-24T17:47:29.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T17:47:29.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.601+0000 7f2d66800640 1 Processor -- start 2024-01-24T17:47:29.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.601+0000 7f2d66800640 1 -- start start 2024-01-24T17:47:29.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.603+0000 7f2d66800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2d60070920 0x7f2d600b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.603+0000 7f2d66800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d60070470 con 0x7f2d6015f390 2024-01-24T17:47:29.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.603+0000 7f2d66800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6015ef00 con 0x7f2d600b0010 2024-01-24T17:47:29.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.603+0000 7f2d66800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d600710a0 con 0x7f2d60070920 2024-01-24T17:47:29.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.603+0000 7f2d5fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2d6015f390 0x7f2d600c9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49062/0 (socket says 172.21.15.39:49062) 2024-01-24T17:47:29.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.603+0000 7f2d5fe00640 1 -- 172.21.15.39:0/2870702190 learned_addr learned my addr 172.21.15.39:0/2870702190 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:29.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d57e00640 1 --2- 172.21.15.39:0/2870702190 >> v2:172.21.15.100:3300/0 conn(0x7f2d60070920 0x7f2d600b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725205869 0 0) 0x7f2d60070470 con 0x7f2d6015f390 2024-01-24T17:47:29.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d40003520 con 0x7f2d6015f390 2024-01-24T17:47:29.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008460426 0 0) 0x7f2d40003520 con 0x7f2d6015f390 2024-01-24T17:47:29.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 >> v2:172.21.15.100:3300/0 conn(0x7f2d60070920 msgr2=0x7f2d600b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:29.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 --2- 172.21.15.39:0/2870702190 >> v2:172.21.15.100:3300/0 conn(0x7f2d60070920 0x7f2d600b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 >> v1:172.21.15.100:6789/0 conn(0x7f2d600b0010 legacy=0x7f2d6015bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d600b0930 con 0x7f2d6015f390 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2d580019f0 con 0x7f2d6015f390 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 >> v1:172.21.15.39:6789/0 conn(0x7f2d6015f390 legacy=0x7f2d600c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d40003970 0x7f2d40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.604+0000 7f2d5ea00640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d400047c0 0x7f2d40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5ea00640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40005180 0x7f2d4000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600710a0 con 0x7f2d40003970 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4000c030 con 0x7f2d400047c0 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4000c1f0 con 0x7f2d40005180 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5f400640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d40003970 0x7f2d40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5fe00640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40005180 0x7f2d4000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d57e00640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d400047c0 0x7f2d40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5f400640 1 -- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40005180 msgr2=0x7f2d4000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:29.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5f400640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40005180 0x7f2d4000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5f400640 1 -- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d400047c0 msgr2=0x7f2d40004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:29.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5f400640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d400047c0 0x7f2d40004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5f400640 1 -- 172.21.15.39:0/2870702190 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40003520 con 0x7f2d40003970 2024-01-24T17:47:29.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5fe00640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40005180 0x7f2d4000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:29.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d57e00640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d400047c0 0x7f2d40004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:29.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.605+0000 7f2d5fe00640 1 -- 172.21.15.39:0/2870702190 reap_dead start 2024-01-24T17:47:29.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.607+0000 7f2d5f400640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d40003970 0x7f2d40003d90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f2d50001750 tx=0x7f2d400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82c6234f3184c6d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:29.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.607+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d500052d0 con 0x7f2d40003970 2024-01-24T17:47:29.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.607+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5002e050 con 0x7f2d40003970 2024-01-24T17:47:29.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.607+0000 7f2d5ea00640 1 -- 172.21.15.39:0/2870702190 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50003830 con 0x7f2d40003970 2024-01-24T17:47:29.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.607+0000 7f2d66800640 1 -- 172.21.15.39:0/2870702190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d40003970 msgr2=0x7f2d40003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.607+0000 7f2d66800640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d40003970 0x7f2d40003d90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f2d50001750 tx=0x7f2d400057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.608+0000 7f2d66800640 1 -- 172.21.15.39:0/2870702190 shutdown_connections 2024-01-24T17:47:29.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.608+0000 7f2d66800640 1 --2- 172.21.15.39:0/2870702190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d40003970 0x7f2d40003d90 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.608+0000 7f2d66800640 1 -- 172.21.15.39:0/2870702190 >> 172.21.15.39:0/2870702190 conn(0x7f2d6006bf20 msgr2=0x7f2d6006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:29.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.608+0000 7f2d66800640 1 -- 172.21.15.39:0/2870702190 shutdown_connections 2024-01-24T17:47:29.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.608+0000 7f2d66800640 1 -- 172.21.15.39:0/2870702190 wait complete. 2024-01-24T17:47:29.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.610+0000 7f2d66800640 1 Processor -- start 2024-01-24T17:47:29.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.610+0000 7f2d66800640 1 -- start start 2024-01-24T17:47:29.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d66800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40003970 0x7f2d60157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d66800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d600c4580 0x7f2d600c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d66800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d600c24b0 0x7f2d600c2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d66800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600c4f40 con 0x7f2d600c24b0 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d66800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600c39f0 con 0x7f2d600c4580 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d66800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600c71e0 con 0x7f2d40003970 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d600c24b0 0x7f2d600c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d600c24b0 0x7f2d600c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36518/0 (socket says 172.21.15.39:36518) 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5fe00640 1 -- 172.21.15.39:0/1861562661 learned_addr learned my addr 172.21.15.39:0/1861562661 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5f400640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40003970 0x7f2d60157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d57e00640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d600c4580 0x7f2d600c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5fe00640 1 -- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40003970 msgr2=0x7f2d60157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5fe00640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40003970 0x7f2d60157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5fe00640 1 -- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d600c4580 msgr2=0x7f2d600c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5fe00640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d600c4580 0x7f2d600c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5fe00640 1 -- 172.21.15.39:0/1861562661 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d600c3150 con 0x7f2d600c24b0 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.611+0000 7f2d5f400640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40003970 0x7f2d60157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.612+0000 7f2d5fe00640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d600c24b0 0x7f2d600c2930 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f2d6015b140 tx=0x7f2d5800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.612+0000 7f2d57400640 1 -- 172.21.15.39:0/1861562661 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58004d60 con 0x7f2d600c24b0 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.612+0000 7f2d57400640 1 -- 172.21.15.39:0/1861562661 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5800baa0 con 0x7f2d600c24b0 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.612+0000 7f2d57400640 1 -- 172.21.15.39:0/1861562661 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5800be50 con 0x7f2d600c24b0 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.612+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d600c3430 con 0x7f2d600c24b0 2024-01-24T17:47:29.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.612+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d600c2e70 con 0x7f2d600c24b0 2024-01-24T17:47:29.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.613+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6004dff0 con 0x7f2d600c24b0 2024-01-24T17:47:29.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.614+0000 7f2d57400640 1 -- 172.21.15.39:0/1861562661 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2d58007110 con 0x7f2d600c24b0 2024-01-24T17:47:29.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.615+0000 7f2d57400640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2d3806a430 0x7f2d3806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:29.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.616+0000 7f2d57400640 1 -- 172.21.15.39:0/1861562661 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2d58088800 con 0x7f2d600c24b0 2024-01-24T17:47:29.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.616+0000 7f2d5f400640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2d3806a430 0x7f2d3806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:29.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.617+0000 7f2d5f400640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2d3806a430 0x7f2d3806c8f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2d50033000 tx=0x7f2d50009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:29.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.620+0000 7f2d57400640 1 -- 172.21.15.39:0/1861562661 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2d58054130 con 0x7f2d600c24b0 2024-01-24T17:47:29.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.807+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2d600c7320 con 0x7f2d600c24b0 2024-01-24T17:47:29.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.808+0000 7f2d57400640 1 -- 172.21.15.39:0/1861562661 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v84) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2d58057750 con 0x7f2d600c24b0 2024-01-24T17:47:29.809 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:29.809 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-01-24T17:47:29.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:47:29.809 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-01-24T17:47:29.809 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-01-24T17:47:29.809 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:29.809 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-01-24T17:47:29.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2d3806a430 msgr2=0x7f2d3806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2d3806a430 0x7f2d3806c8f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2d50033000 tx=0x7f2d50009030 comp rx=0 tx=0).stop 2024-01-24T17:47:29.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d600c24b0 msgr2=0x7f2d600c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:29.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d600c24b0 0x7f2d600c2930 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f2d6015b140 tx=0x7f2d5800b4a0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 shutdown_connections 2024-01-24T17:47:29.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2d3806a430 0x7f2d3806c8f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d600c24b0 0x7f2d600c2930 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2d600c4580 0x7f2d600c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 --2- 172.21.15.39:0/1861562661 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2d40003970 0x7f2d60157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:29.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 >> 172.21.15.39:0/1861562661 conn(0x7f2d6006bf20 msgr2=0x7f2d600c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:29.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 shutdown_connections 2024-01-24T17:47:29.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:29.810+0000 7f2d66800640 1 -- 172.21.15.39:0/1861562661 wait complete. 2024-01-24T17:47:29.828 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T17:47:29.829 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T17:47:30.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.047+0000 7fa778a00640 1 Processor -- start 2024-01-24T17:47:30.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.048+0000 7fa778a00640 1 -- start start 2024-01-24T17:47:30.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.049+0000 7fa778a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa77415ac70 0x7fa77406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.049+0000 7fa778a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7740b12b0 con 0x7fa7741516c0 2024-01-24T17:47:30.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.049+0000 7fa778a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77415b170 con 0x7fa774070260 2024-01-24T17:47:30.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.049+0000 7fa778a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7740706d0 con 0x7fa77415ac70 2024-01-24T17:47:30.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.050+0000 7fa76be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa77415ac70 0x7fa77406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.050+0000 7fa772800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa7741516c0 0x7fa7740b3d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49070/0 (socket says 172.21.15.39:49070) 2024-01-24T17:47:30.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.050+0000 7fa772800640 1 -- 172.21.15.39:0/1389008798 learned_addr learned my addr 172.21.15.39:0/1389008798 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:30.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.050+0000 7fa771e00640 1 -- 172.21.15.39:0/1389008798 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238100594 0 0) 0x7fa7740b12b0 con 0x7fa7741516c0 2024-01-24T17:47:30.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.050+0000 7fa771e00640 1 -- 172.21.15.39:0/1389008798 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa754003520 con 0x7fa7741516c0 2024-01-24T17:47:30.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.050+0000 7fa76be00640 1 -- 172.21.15.39:0/1389008798 >> v1:172.21.15.100:6789/0 conn(0x7fa774070260 legacy=0x7fa7740ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.050+0000 7fa76be00640 1 -- 172.21.15.39:0/1389008798 >> v1:172.21.15.39:6789/0 conn(0x7fa7741516c0 legacy=0x7fa7740b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.050+0000 7fa76be00640 1 -- 172.21.15.39:0/1389008798 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77415b450 con 0x7fa77415ac70 2024-01-24T17:47:30.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.052+0000 7fa76be00640 1 --2- 172.21.15.39:0/1389008798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77415ac70 0x7fa77406fd20 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0097a0 tx=0x7fa75c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a935094e0d1a554d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:30.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.052+0000 7fa771e00640 1 -- 172.21.15.39:0/1389008798 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c030610 con 0x7fa77415ac70 2024-01-24T17:47:30.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.052+0000 7fa771e00640 1 -- 172.21.15.39:0/1389008798 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c030770 con 0x7fa77415ac70 2024-01-24T17:47:30.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.052+0000 7fa771e00640 1 -- 172.21.15.39:0/1389008798 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c030ae0 con 0x7fa77415ac70 2024-01-24T17:47:30.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.053+0000 7fa778a00640 1 -- 172.21.15.39:0/1389008798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77415ac70 msgr2=0x7fa77406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.053+0000 7fa778a00640 1 --2- 172.21.15.39:0/1389008798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77415ac70 0x7fa77406fd20 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0097a0 tx=0x7fa75c02b960 comp rx=0 tx=0).stop 2024-01-24T17:47:30.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.053+0000 7fa778a00640 1 -- 172.21.15.39:0/1389008798 shutdown_connections 2024-01-24T17:47:30.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.053+0000 7fa778a00640 1 --2- 172.21.15.39:0/1389008798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77415ac70 0x7fa77406fd20 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.053+0000 7fa778a00640 1 -- 172.21.15.39:0/1389008798 >> 172.21.15.39:0/1389008798 conn(0x7fa77406a200 msgr2=0x7fa7740affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:30.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.053+0000 7fa778a00640 1 -- 172.21.15.39:0/1389008798 shutdown_connections 2024-01-24T17:47:30.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.053+0000 7fa778a00640 1 -- 172.21.15.39:0/1389008798 wait complete. 2024-01-24T17:47:30.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.055+0000 7fa778a00640 1 Processor -- start 2024-01-24T17:47:30.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.055+0000 7fa778a00640 1 -- start start 2024-01-24T17:47:30.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.056+0000 7fa778a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa774070260 0x7fa7740c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.056+0000 7fa778a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7741516c0 0x7fa7740c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.056+0000 7fa778a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa77415ac70 0x7fa7741f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.056+0000 7fa778a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740c3030 con 0x7fa7741516c0 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.056+0000 7fa778a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77414f7e0 con 0x7fa77415ac70 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.056+0000 7fa778a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740c3dc0 con 0x7fa774070260 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7741516c0 0x7fa7740c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7741516c0 0x7fa7740c2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36534/0 (socket says 172.21.15.39:36534) 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 -- 172.21.15.39:0/3412180810 learned_addr learned my addr 172.21.15.39:0/3412180810 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa772800640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa774070260 0x7fa7740c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa773200640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa77415ac70 0x7fa7741f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 -- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa774070260 msgr2=0x7fa7740c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa774070260 0x7fa7740c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 -- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa77415ac70 msgr2=0x7fa7741f1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa77415ac70 0x7fa7741f1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 -- 172.21.15.39:0/3412180810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c009400 con 0x7fa7741516c0 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa773200640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa77415ac70 0x7fa7741f1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.057+0000 7fa76be00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7741516c0 0x7fa7740c2af0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0097a0 tx=0x7fa75c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.058+0000 7fa76aa00640 1 -- 172.21.15.39:0/3412180810 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c03f070 con 0x7fa7741516c0 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.058+0000 7fa76aa00640 1 -- 172.21.15.39:0/3412180810 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c006e10 con 0x7fa7741516c0 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.058+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7740c40a0 con 0x7fa7741516c0 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.058+0000 7fa76aa00640 1 -- 172.21.15.39:0/3412180810 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c030ea0 con 0x7fa7741516c0 2024-01-24T17:47:30.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.058+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77414fd10 con 0x7fa7741516c0 2024-01-24T17:47:30.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.059+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77404e840 con 0x7fa7741516c0 2024-01-24T17:47:30.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.060+0000 7fa76aa00640 1 -- 172.21.15.39:0/3412180810 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa75c003810 con 0x7fa7741516c0 2024-01-24T17:47:30.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.061+0000 7fa76aa00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa74c06a430 0x7fa74c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.062+0000 7fa76aa00640 1 -- 172.21.15.39:0/3412180810 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa75c0aa270 con 0x7fa7741516c0 2024-01-24T17:47:30.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.062+0000 7fa772800640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa74c06a430 0x7fa74c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.063+0000 7fa772800640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa74c06a430 0x7fa74c06c8f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa7640011e0 tx=0x7fa764009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:30.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.066+0000 7fa76aa00640 1 -- 172.21.15.39:0/3412180810 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa75c075c50 con 0x7fa7741516c0 2024-01-24T17:47:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.259+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa7740b0d20 con 0x7fa7741516c0 2024-01-24T17:47:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.443+0000 7fa76aa00640 1 -- 172.21.15.39:0/3412180810 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v85) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa75c079270 con 0x7fa7741516c0 2024-01-24T17:47:30.444 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-01-24T17:47:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.449+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa74c06a430 msgr2=0x7fa74c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.449+0000 7fa778a00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa74c06a430 0x7fa74c06c8f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa7640011e0 tx=0x7fa764009040 comp rx=0 tx=0).stop 2024-01-24T17:47:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7741516c0 msgr2=0x7fa7740c2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7741516c0 0x7fa7740c2af0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0097a0 tx=0x7fa75c0046c0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 shutdown_connections 2024-01-24T17:47:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa77415ac70 0x7fa7741f1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa74c06a430 0x7fa74c06c8f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7741516c0 0x7fa7740c2af0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 --2- 172.21.15.39:0/3412180810 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa774070260 0x7fa7740c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 >> 172.21.15.39:0/3412180810 conn(0x7fa77406a200 msgr2=0x7fa77415e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 shutdown_connections 2024-01-24T17:47:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.450+0000 7fa778a00640 1 -- 172.21.15.39:0/3412180810 wait complete. 2024-01-24T17:47:30.474 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T17:47:30.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.691+0000 7f2f07600640 1 Processor -- start 2024-01-24T17:47:30.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.691+0000 7f2f07600640 1 -- start start 2024-01-24T17:47:30.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.693+0000 7f2f07600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f000b3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.693+0000 7f2f07600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2f000cd110 0x7f2f000cd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.693+0000 7f2f07600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f000ce410 con 0x7f2f0015e220 2024-01-24T17:47:30.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.693+0000 7f2f07600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f000cdac0 con 0x7f2f000cd110 2024-01-24T17:47:30.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.693+0000 7f2f07600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000cc020 con 0x7f2f0015de40 2024-01-24T17:47:30.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.693+0000 7f2effe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2f0015e220 0x7f2f000b15d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49086/0 (socket says 172.21.15.39:49086) 2024-01-24T17:47:30.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2effe00640 1 -- 172.21.15.39:0/666406398 learned_addr learned my addr 172.21.15.39:0/666406398 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:30.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2f05600640 1 --2- 172.21.15.39:0/666406398 >> v2:172.21.15.39:3300/0 conn(0x7f2f000cd110 0x7f2f000cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2f04c00640 1 --2- 172.21.15.39:0/666406398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f000b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2eff400640 1 -- 172.21.15.39:0/666406398 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064492490 0 0) 0x7f2f000ce410 con 0x7f2f0015e220 2024-01-24T17:47:30.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2eff400640 1 -- 172.21.15.39:0/666406398 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee0003520 con 0x7f2f0015e220 2024-01-24T17:47:30.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2f05600640 1 -- 172.21.15.39:0/666406398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 msgr2=0x7f2f000b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:30.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2f05600640 1 --2- 172.21.15.39:0/666406398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f000b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2f05600640 1 -- 172.21.15.39:0/666406398 >> v1:172.21.15.39:6789/0 conn(0x7f2f0015e220 legacy=0x7f2f000b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.694+0000 7f2f05600640 1 -- 172.21.15.39:0/666406398 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f000cc300 con 0x7f2f000cd110 2024-01-24T17:47:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.696+0000 7f2f05600640 1 --2- 172.21.15.39:0/666406398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f000cd110 0x7f2f000cd580 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00097a0 tx=0x7f2ef002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d28920ad0799da server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.696+0000 7f2eff400640 1 -- 172.21.15.39:0/666406398 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0031780 con 0x7f2f000cd110 2024-01-24T17:47:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.696+0000 7f2eff400640 1 -- 172.21.15.39:0/666406398 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef00318e0 con 0x7f2f000cd110 2024-01-24T17:47:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.696+0000 7f2eff400640 1 -- 172.21.15.39:0/666406398 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0031c50 con 0x7f2f000cd110 2024-01-24T17:47:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.697+0000 7f2f07600640 1 -- 172.21.15.39:0/666406398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f000cd110 msgr2=0x7f2f000cd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.697+0000 7f2f07600640 1 --2- 172.21.15.39:0/666406398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f000cd110 0x7f2f000cd580 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00097a0 tx=0x7f2ef002b820 comp rx=0 tx=0).stop 2024-01-24T17:47:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.697+0000 7f2f07600640 1 -- 172.21.15.39:0/666406398 shutdown_connections 2024-01-24T17:47:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.697+0000 7f2f07600640 1 --2- 172.21.15.39:0/666406398 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f000cd110 0x7f2f000cd580 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.697+0000 7f2f07600640 1 --2- 172.21.15.39:0/666406398 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f000b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.697+0000 7f2f07600640 1 -- 172.21.15.39:0/666406398 >> 172.21.15.39:0/666406398 conn(0x7f2f000d0840 msgr2=0x7f2f000d0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.697+0000 7f2f07600640 1 -- 172.21.15.39:0/666406398 shutdown_connections 2024-01-24T17:47:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.697+0000 7f2f07600640 1 -- 172.21.15.39:0/666406398 wait complete. 2024-01-24T17:47:30.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.699+0000 7f2f07600640 1 Processor -- start 2024-01-24T17:47:30.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.699+0000 7f2f07600640 1 -- start start 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.700+0000 7f2f07600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f000cd110 0x7f2f0009a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.700+0000 7f2f07600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f0009aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.700+0000 7f2f07600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f0015e220 0x7f2f000c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.700+0000 7f2f07600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0014e8b0 con 0x7f2f0015e220 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.700+0000 7f2f07600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0014e5f0 con 0x7f2f000cd110 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.700+0000 7f2f07600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000c6090 con 0x7f2f0015de40 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f05600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f0015e220 0x7f2f000c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f05600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f0015e220 0x7f2f000c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36566/0 (socket says 172.21.15.39:36566) 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f05600640 1 -- 172.21.15.39:0/682934646 learned_addr learned my addr 172.21.15.39:0/682934646 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f04c00640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f000cd110 0x7f2f0009a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2effe00640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f0009aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f05600640 1 -- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 msgr2=0x7f2f0009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f05600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f0009aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f05600640 1 -- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f000cd110 msgr2=0x7f2f0009a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:30.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f05600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f000cd110 0x7f2f0009a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:30.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2f05600640 1 -- 172.21.15.39:0/682934646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef0009400 con 0x7f2f0015e220 2024-01-24T17:47:30.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.701+0000 7f2effe00640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f0009aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.702+0000 7f2f05600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f0015e220 0x7f2f000c5310 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0034040 tx=0x7f2ef00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.702+0000 7f2efcc00640 1 -- 172.21.15.39:0/682934646 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0032820 con 0x7f2f0015e220 2024-01-24T17:47:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.702+0000 7f2efcc00640 1 -- 172.21.15.39:0/682934646 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef0032980 con 0x7f2f0015e220 2024-01-24T17:47:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.702+0000 7f2efcc00640 1 -- 172.21.15.39:0/682934646 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0032cf0 con 0x7f2f0015e220 2024-01-24T17:47:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.702+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f000c6370 con 0x7f2f0015e220 2024-01-24T17:47:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.702+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f000c6930 con 0x7f2f0015e220 2024-01-24T17:47:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.703+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0014ea20 con 0x7f2f0015e220 2024-01-24T17:47:30.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.704+0000 7f2efcc00640 1 -- 172.21.15.39:0/682934646 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2ef004e440 con 0x7f2f0015e220 2024-01-24T17:47:30.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.705+0000 7f2efcc00640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2ed806a430 0x7f2ed806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.705+0000 7f2efcc00640 1 -- 172.21.15.39:0/682934646 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2ef00aa7e0 con 0x7f2f0015e220 2024-01-24T17:47:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.706+0000 7f2f04c00640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2ed806a430 0x7f2ed806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:30.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.707+0000 7f2f04c00640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2ed806a430 0x7f2ed806c8f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80074a0 tx=0x7f2ef800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:30.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.710+0000 7f2efcc00640 1 -- 172.21.15.39:0/682934646 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2ef0075fa0 con 0x7f2f0015e220 2024-01-24T17:47:30.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:30.904+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2f00067050 con 0x7f2f0015e220 2024-01-24T17:47:31.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.450+0000 7f2efcc00640 1 -- 172.21.15.39:0/682934646 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2ef00795c0 con 0x7f2f0015e220 2024-01-24T17:47:31.451 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-01-24T17:47:31.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.457+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2ed806a430 msgr2=0x7f2ed806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.457+0000 7f2f07600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2ed806a430 0x7f2ed806c8f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f2ef80074a0 tx=0x7f2ef800b040 comp rx=0 tx=0).stop 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.457+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f0015e220 msgr2=0x7f2f000c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.457+0000 7f2f07600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f0015e220 0x7f2f000c5310 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0034040 tx=0x7f2ef00046c0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.458+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 shutdown_connections 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.458+0000 7f2f07600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2ed806a430 0x7f2ed806c8f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.458+0000 7f2f07600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f0015e220 0x7f2f000c5310 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.458+0000 7f2f07600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2f0015de40 0x7f2f0009aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.458+0000 7f2f07600640 1 --2- 172.21.15.39:0/682934646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2f000cd110 0x7f2f0009a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.458+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 >> 172.21.15.39:0/682934646 conn(0x7f2f000d0840 msgr2=0x7f2f0006b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:31.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.461+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 shutdown_connections 2024-01-24T17:47:31.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.462+0000 7f2f07600640 1 -- 172.21.15.39:0/682934646 wait complete. 2024-01-24T17:47:31.494 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T17:47:31.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.679+0000 7fdb2b200640 1 Processor -- start 2024-01-24T17:47:31.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.680+0000 7fdb2b200640 1 -- start start 2024-01-24T17:47:31.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.681+0000 7fdb2b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdb240b4310 0x7fdb240b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:31.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.681+0000 7fdb2b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb240ce660 0x7fdb240af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:31.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.681+0000 7fdb2b200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdb2406acd0 0x7fdb2406c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:31.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.681+0000 7fdb2b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2406bde0 con 0x7fdb240b4310 2024-01-24T17:47:31.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.681+0000 7fdb2b200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb240b4c30 con 0x7fdb2406acd0 2024-01-24T17:47:31.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.681+0000 7fdb2b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2406b1d0 con 0x7fdb240ce660 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.681+0000 7fdb29e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdb240b4310 0x7fdb240b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb29e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdb240b4310 0x7fdb240b46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36582/0 (socket says 172.21.15.39:36582) 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb29e00640 1 -- 172.21.15.39:0/559984728 learned_addr learned my addr 172.21.15.39:0/559984728 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb2a800640 1 --2- 172.21.15.39:0/559984728 >> v2:172.21.15.100:3300/0 conn(0x7fdb2406acd0 0x7fdb2406c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb29400640 1 --2- 172.21.15.39:0/559984728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb240ce660 0x7fdb240af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb29e00640 1 -- 172.21.15.39:0/559984728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb240ce660 msgr2=0x7fdb240af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb29e00640 1 --2- 172.21.15.39:0/559984728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb240ce660 0x7fdb240af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb29e00640 1 -- 172.21.15.39:0/559984728 >> v2:172.21.15.100:3300/0 conn(0x7fdb2406acd0 msgr2=0x7fdb2406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb29e00640 1 --2- 172.21.15.39:0/559984728 >> v2:172.21.15.100:3300/0 conn(0x7fdb2406acd0 0x7fdb2406c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.682+0000 7fdb29e00640 1 -- 172.21.15.39:0/559984728 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2406b4b0 con 0x7fdb240b4310 2024-01-24T17:47:31.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.684+0000 7fdb29e00640 1 --2- 172.21.15.39:0/559984728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240b46f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fdb2408c130 tx=0x7fdb1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e40380d7b57460 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:31.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.684+0000 7fdb28a00640 1 -- 172.21.15.39:0/559984728 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14004230 con 0x7fdb240b4310 2024-01-24T17:47:31.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.684+0000 7fdb28a00640 1 -- 172.21.15.39:0/559984728 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb14004390 con 0x7fdb240b4310 2024-01-24T17:47:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.684+0000 7fdb28a00640 1 -- 172.21.15.39:0/559984728 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14003810 con 0x7fdb240b4310 2024-01-24T17:47:31.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 -- 172.21.15.39:0/559984728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 msgr2=0x7fdb240b46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 --2- 172.21.15.39:0/559984728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240b46f0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fdb2408c130 tx=0x7fdb1402b8f0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 -- 172.21.15.39:0/559984728 shutdown_connections 2024-01-24T17:47:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 --2- 172.21.15.39:0/559984728 >> v2:172.21.15.100:3300/0 conn(0x7fdb2406acd0 0x7fdb2406c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 --2- 172.21.15.39:0/559984728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb240ce660 0x7fdb240af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 --2- 172.21.15.39:0/559984728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240b46f0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 -- 172.21.15.39:0/559984728 >> 172.21.15.39:0/559984728 conn(0x7fdb240d2890 msgr2=0x7fdb240d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 -- 172.21.15.39:0/559984728 shutdown_connections 2024-01-24T17:47:31.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.685+0000 7fdb2b200640 1 -- 172.21.15.39:0/559984728 wait complete. 2024-01-24T17:47:31.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.687+0000 7fdb2b200640 1 Processor -- start 2024-01-24T17:47:31.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.687+0000 7fdb2b200640 1 -- start start 2024-01-24T17:47:31.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb2b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb2406acd0 0x7fdb24157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:31.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb2b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb2b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb240ce660 0x7fdb240cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb2b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240c9e10 con 0x7fdb240b4310 2024-01-24T17:47:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb2b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24157640 con 0x7fdb240ce660 2024-01-24T17:47:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb2b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240cc620 con 0x7fdb2406acd0 2024-01-24T17:47:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb29400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb29400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36598/0 (socket says 172.21.15.39:36598) 2024-01-24T17:47:31.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb29400640 1 -- 172.21.15.39:0/3201164149 learned_addr learned my addr 172.21.15.39:0/3201164149 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:31.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb2a800640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb240ce660 0x7fdb240cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb29400640 1 -- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb2406acd0 msgr2=0x7fdb24157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb29e00640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb2406acd0 0x7fdb24157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.688+0000 7fdb29400640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb2406acd0 0x7fdb24157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb29400640 1 -- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb240ce660 msgr2=0x7fdb240cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb29400640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb240ce660 0x7fdb240cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb29400640 1 -- 172.21.15.39:0/3201164149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb14009460 con 0x7fdb240b4310 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb29400640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240c98d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00d620 tx=0x7fdb0c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb22000640 1 -- 172.21.15.39:0/3201164149 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00b620 con 0x7fdb240b4310 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb22000640 1 -- 172.21.15.39:0/3201164149 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c00b780 con 0x7fdb240b4310 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb22000640 1 -- 172.21.15.39:0/3201164149 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00bb30 con 0x7fdb240b4310 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb240b69f0 con 0x7fdb240b4310 2024-01-24T17:47:31.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.689+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb240b6c60 con 0x7fdb240b4310 2024-01-24T17:47:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.690+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2404e840 con 0x7fdb240b4310 2024-01-24T17:47:31.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.692+0000 7fdb22000640 1 -- 172.21.15.39:0/3201164149 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdb0c00bc90 con 0x7fdb240b4310 2024-01-24T17:47:31.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.693+0000 7fdb22000640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdafc06a430 0x7fdafc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:31.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.693+0000 7fdb22000640 1 -- 172.21.15.39:0/3201164149 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fdb0c0897b0 con 0x7fdb240b4310 2024-01-24T17:47:31.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.693+0000 7fdb29e00640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdafc06a430 0x7fdafc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:31.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.695+0000 7fdb29e00640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdafc06a430 0x7fdafc06c8f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdb14035000 tx=0x7fdb14008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:31.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.699+0000 7fdb22000640 1 -- 172.21.15.39:0/3201164149 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdb0c054f70 con 0x7fdb240b4310 2024-01-24T17:47:31.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.934+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fdb2406cc80 con 0x7fdb240b4310 2024-01-24T17:47:31.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.947+0000 7fdb22000640 1 -- 172.21.15.39:0/3201164149 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v47) v1 ==== 174+0+0 (secure 0 0 0) 0x7fdb0c058590 con 0x7fdb240b4310 2024-01-24T17:47:31.949 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 12 and data pool 13 2024-01-24T17:47:31.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdafc06a430 msgr2=0x7fdafc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdafc06a430 0x7fdafc06c8f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdb14035000 tx=0x7fdb14008030 comp rx=0 tx=0).stop 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 msgr2=0x7fdb240c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240c98d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00d620 tx=0x7fdb0c00dae0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 shutdown_connections 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb240ce660 0x7fdb240cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdafc06a430 0x7fdafc06c8f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb240b4310 0x7fdb240c98d0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.952+0000 7fdb2b200640 1 --2- 172.21.15.39:0/3201164149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb2406acd0 0x7fdb24157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.953+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 >> 172.21.15.39:0/3201164149 conn(0x7fdb240d2890 msgr2=0x7fdb240b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.953+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 shutdown_connections 2024-01-24T17:47:31.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:31.953+0000 7fdb2b200640 1 -- 172.21.15.39:0/3201164149 wait complete. 2024-01-24T17:47:31.980 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:32.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.154+0000 7f3aebe00640 1 Processor -- start 2024-01-24T17:47:32.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.154+0000 7f3aebe00640 1 -- start start 2024-01-24T17:47:32.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.155+0000 7f3aebe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3aec0cdaf0 0x7f3aec06bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.155+0000 7f3aebe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aec1540f0 con 0x7f3aec0ca700 2024-01-24T17:47:32.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.155+0000 7f3aebe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aec15da00 con 0x7f3aec0ca320 2024-01-24T17:47:32.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.155+0000 7f3aebe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aec159fd0 con 0x7f3aec0cdaf0 2024-01-24T17:47:32.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3aea000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3aec0ca700 0x7f3aec15a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49088/0 (socket says 172.21.15.39:49088) 2024-01-24T17:47:32.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3aea000640 1 -- 172.21.15.39:0/3163784013 learned_addr learned my addr 172.21.15.39:0/3163784013 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:32.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3aeb400640 1 --2- 172.21.15.39:0/3163784013 >> v2:172.21.15.100:3300/0 conn(0x7f3aec0cdaf0 0x7f3aec06bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299881557 0 0) 0x7f3aec1540f0 con 0x7f3aec0ca700 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3acc005730 con 0x7f3aec0ca700 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3239675963 0 0) 0x7f3acc005730 con 0x7f3aec0ca700 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 >> v2:172.21.15.100:3300/0 conn(0x7f3aec0cdaf0 msgr2=0x7f3aec06bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3ae9600640 1 --2- 172.21.15.39:0/3163784013 >> v2:172.21.15.100:3300/0 conn(0x7f3aec0cdaf0 0x7f3aec06bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 >> v1:172.21.15.100:6789/0 conn(0x7f3aec0ca320 legacy=0x7f3aec15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.156+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec15dce0 con 0x7f3aec0ca700 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f3ad4001670 con 0x7f3aec0ca700 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 >> v1:172.21.15.39:6789/0 conn(0x7f3aec0ca700 legacy=0x7f3aec15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3ae9600640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3acc005b80 0x7f3acc005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3ae9600640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3acc0069d0 0x7f3acc006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3ae9600640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc0074b0 0x7f3acc00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec159fd0 con 0x7f3acc005b80 2024-01-24T17:47:32.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc00e380 con 0x7f3acc0069d0 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc00e540 con 0x7f3acc0074b0 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aeaa00640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3acc005b80 0x7f3acc005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aeb400640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc0074b0 0x7f3acc00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aea000640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3acc0069d0 0x7f3acc006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aeaa00640 1 -- 172.21.15.39:0/3163784013 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc0074b0 msgr2=0x7f3acc00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aeaa00640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc0074b0 0x7f3acc00db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aeaa00640 1 -- 172.21.15.39:0/3163784013 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3acc0069d0 msgr2=0x7f3acc006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aeaa00640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3acc0069d0 0x7f3acc006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aeaa00640 1 -- 172.21.15.39:0/3163784013 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc005730 con 0x7f3acc005b80 2024-01-24T17:47:32.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.157+0000 7f3aeb400640 1 -- 172.21.15.39:0/3163784013 reap_dead start 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.158+0000 7f3aeaa00640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3acc005b80 0x7f3acc005fa0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3adc007fd0 tx=0x7f3acc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9bf59011c06cf2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc0052d0 con 0x7f3acc005b80 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc02e050 con 0x7f3acc005b80 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3ae9600640 1 -- 172.21.15.39:0/3163784013 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc003640 con 0x7f3acc005b80 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3aebe00640 1 -- 172.21.15.39:0/3163784013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3acc005b80 msgr2=0x7f3acc005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3aebe00640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3acc005b80 0x7f3acc005fa0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3adc007fd0 tx=0x7f3acc007a00 comp rx=0 tx=0).stop 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3aebe00640 1 -- 172.21.15.39:0/3163784013 shutdown_connections 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3aebe00640 1 --2- 172.21.15.39:0/3163784013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3acc005b80 0x7f3acc005fa0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3aebe00640 1 -- 172.21.15.39:0/3163784013 >> 172.21.15.39:0/3163784013 conn(0x7f3aec06a340 msgr2=0x7f3aec0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3aebe00640 1 -- 172.21.15.39:0/3163784013 shutdown_connections 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.159+0000 7f3aebe00640 1 -- 172.21.15.39:0/3163784013 wait complete. 2024-01-24T17:47:32.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.161+0000 7f3aebe00640 1 Processor -- start 2024-01-24T17:47:32.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.161+0000 7f3aebe00640 1 -- start start 2024-01-24T17:47:32.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.162+0000 7f3aebe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc005b80 0x7f3aec1e70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.162+0000 7f3aebe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aec1e5860 0x7f3aec15a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.162+0000 7f3aebe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aec1473e0 0x7f3aec147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.162+0000 7f3aebe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec1e53d0 con 0x7f3aec1473e0 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.162+0000 7f3aebe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec147d80 con 0x7f3aec1e5860 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.162+0000 7f3aebe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec1e47c0 con 0x7f3acc005b80 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.162+0000 7f3aeb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aec1473e0 0x7f3aec147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeb400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aec1473e0 0x7f3aec147840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36624/0 (socket says 172.21.15.39:36624) 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeb400640 1 -- 172.21.15.39:0/3121405311 learned_addr learned my addr 172.21.15.39:0/3121405311 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aea000640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aec1e5860 0x7f3aec15a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeaa00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc005b80 0x7f3aec1e70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeb400640 1 -- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc005b80 msgr2=0x7f3aec1e70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeb400640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc005b80 0x7f3aec1e70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeb400640 1 -- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aec1e5860 msgr2=0x7f3aec15a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeb400640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aec1e5860 0x7f3aec15a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeb400640 1 -- 172.21.15.39:0/3121405311 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec1e4aa0 con 0x7f3aec1473e0 2024-01-24T17:47:32.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3aeb400640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aec1473e0 0x7f3aec147840 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3aec0cdfa0 tx=0x7f3ae400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:32.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.163+0000 7f3ae3400640 1 -- 172.21.15.39:0/3121405311 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4016620 con 0x7f3aec1473e0 2024-01-24T17:47:32.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.164+0000 7f3ae3400640 1 -- 172.21.15.39:0/3121405311 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4016780 con 0x7f3aec1473e0 2024-01-24T17:47:32.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.164+0000 7f3ae3400640 1 -- 172.21.15.39:0/3121405311 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4016b30 con 0x7f3aec1473e0 2024-01-24T17:47:32.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.164+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec1e4d80 con 0x7f3aec1473e0 2024-01-24T17:47:32.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.164+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec1e5ce0 con 0x7f3aec1473e0 2024-01-24T17:47:32.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.165+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec1e5580 con 0x7f3aec1473e0 2024-01-24T17:47:32.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.165+0000 7f3ae3400640 1 -- 172.21.15.39:0/3121405311 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3ae4016c90 con 0x7f3aec1473e0 2024-01-24T17:47:32.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.166+0000 7f3ae3400640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3ac406a460 0x7f3ac406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.166+0000 7f3ae3400640 1 -- 172.21.15.39:0/3121405311 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ae4089640 con 0x7f3aec1473e0 2024-01-24T17:47:32.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.167+0000 7f3aeaa00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3ac406a460 0x7f3ac406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.168+0000 7f3aeaa00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3ac406a460 0x7f3ac406c920 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3adc033000 tx=0x7f3adc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:32.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.170+0000 7f3ae3400640 1 -- 172.21.15.39:0/3121405311 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3aec1e5580 con 0x7f3aec1473e0 2024-01-24T17:47:32.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.356+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3aec1e5580 con 0x7f3aec1473e0 2024-01-24T17:47:32.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.358+0000 7f3ae3400640 1 -- 172.21.15.39:0/3121405311 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16117 (secure 0 0 0) 0x7f3aec1e5580 con 0x7f3aec1473e0 2024-01-24T17:47:32.359 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:32.359 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":87,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:31.936089+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3ac406a460 msgr2=0x7f3ac406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3ac406a460 0x7f3ac406c920 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3adc033000 tx=0x7f3adc009030 comp rx=0 tx=0).stop 2024-01-24T17:47:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aec1473e0 msgr2=0x7f3aec147840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aec1473e0 0x7f3aec147840 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f3aec0cdfa0 tx=0x7f3ae400f520 comp rx=0 tx=0).stop 2024-01-24T17:47:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 shutdown_connections 2024-01-24T17:47:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3ac406a460 0x7f3ac406c920 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3aec1473e0 0x7f3aec147840 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3aec1e5860 0x7f3aec15a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 --2- 172.21.15.39:0/3121405311 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3acc005b80 0x7f3aec1e70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 >> 172.21.15.39:0/3121405311 conn(0x7f3aec06a340 msgr2=0x7f3aec159360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:32.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 shutdown_connections 2024-01-24T17:47:32.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.361+0000 7f3aebe00640 1 -- 172.21.15.39:0/3121405311 wait complete. 2024-01-24T17:47:32.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:32.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.595+0000 7f093da00640 1 Processor -- start 2024-01-24T17:47:32.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.596+0000 7f093da00640 1 -- start start 2024-01-24T17:47:32.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.597+0000 7f093da00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f09380d12e0 0x7f0938168270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.597+0000 7f093da00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09380b8820 con 0x7f09380d0e90 2024-01-24T17:47:32.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.597+0000 7f093da00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09380b5790 con 0x7f093815cbe0 2024-01-24T17:47:32.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.597+0000 7f093da00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09380b5a50 con 0x7f09380d12e0 2024-01-24T17:47:32.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.598+0000 7f0937400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f09380d12e0 0x7f0938168270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.598+0000 7f092de00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f09380d0e90 0x7f0938164b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:49102/0 (socket says 172.21.15.39:49102) 2024-01-24T17:47:32.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.598+0000 7f092de00640 1 -- 172.21.15.39:0/780571643 learned_addr learned my addr 172.21.15.39:0/780571643 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:32.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.598+0000 7f0937400640 1 -- 172.21.15.39:0/780571643 >> v1:172.21.15.100:6789/0 conn(0x7f093815cbe0 legacy=0x7f09380cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.598+0000 7f0937400640 1 -- 172.21.15.39:0/780571643 >> v1:172.21.15.39:6789/0 conn(0x7f09380d0e90 legacy=0x7f0938164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.598+0000 7f0937400640 1 -- 172.21.15.39:0/780571643 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09380b5d30 con 0x7f09380d12e0 2024-01-24T17:47:32.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.599+0000 7f0937400640 1 --2- 172.21.15.39:0/780571643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d12e0 0x7f0938168270 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f09300097a0 tx=0x7f093002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa3628dceac97ce server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:32.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.600+0000 7f0936000640 1 -- 172.21.15.39:0/780571643 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930030610 con 0x7f09380d12e0 2024-01-24T17:47:32.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.600+0000 7f0936000640 1 -- 172.21.15.39:0/780571643 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0930030770 con 0x7f09380d12e0 2024-01-24T17:47:32.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.600+0000 7f0936000640 1 -- 172.21.15.39:0/780571643 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930030b00 con 0x7f09380d12e0 2024-01-24T17:47:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.600+0000 7f093da00640 1 -- 172.21.15.39:0/780571643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d12e0 msgr2=0x7f0938168270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.600+0000 7f093da00640 1 --2- 172.21.15.39:0/780571643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d12e0 0x7f0938168270 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f09300097a0 tx=0x7f093002b960 comp rx=0 tx=0).stop 2024-01-24T17:47:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.600+0000 7f093da00640 1 -- 172.21.15.39:0/780571643 shutdown_connections 2024-01-24T17:47:32.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.600+0000 7f093da00640 1 --2- 172.21.15.39:0/780571643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d12e0 0x7f0938168270 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.600+0000 7f093da00640 1 -- 172.21.15.39:0/780571643 >> 172.21.15.39:0/780571643 conn(0x7f093809bc80 msgr2=0x7f09380d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:32.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.601+0000 7f093da00640 1 -- 172.21.15.39:0/780571643 shutdown_connections 2024-01-24T17:47:32.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.601+0000 7f093da00640 1 -- 172.21.15.39:0/780571643 wait complete. 2024-01-24T17:47:32.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.602+0000 7f093da00640 1 Processor -- start 2024-01-24T17:47:32.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.602+0000 7f093da00640 1 -- start start 2024-01-24T17:47:32.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.603+0000 7f093da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d0e90 0x7f0938147be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.603+0000 7f093da00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09380d12e0 0x7f093815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.603+0000 7f093da00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f093815cbe0 0x7f093815fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.603+0000 7f093da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093815ab90 con 0x7f09380d0e90 2024-01-24T17:47:32.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.603+0000 7f093da00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09380ca9d0 con 0x7f09380d12e0 2024-01-24T17:47:32.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.603+0000 7f093da00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09380cacb0 con 0x7f093815cbe0 2024-01-24T17:47:32.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.603+0000 7f0936a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d0e90 0x7f0938147be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0936a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d0e90 0x7f0938147be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36638/0 (socket says 172.21.15.39:36638) 2024-01-24T17:47:32.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0936a00640 1 -- 172.21.15.39:0/10465884 learned_addr learned my addr 172.21.15.39:0/10465884 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:32.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f092de00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09380d12e0 0x7f093815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0937400640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f093815cbe0 0x7f093815fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0936a00640 1 -- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f093815cbe0 msgr2=0x7f093815fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0936a00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f093815cbe0 0x7f093815fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0936a00640 1 -- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09380d12e0 msgr2=0x7f093815f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0936a00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09380d12e0 0x7f093815f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0936a00640 1 -- 172.21.15.39:0/10465884 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0930009400 con 0x7f09380d0e90 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f0937400640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f093815cbe0 0x7f093815fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.604+0000 7f092de00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09380d12e0 0x7f093815f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.605+0000 7f0936a00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d0e90 0x7f0938147be0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f092800c730 tx=0x7f092800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.605+0000 7f092f400640 1 -- 172.21.15.39:0/10465884 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928003bc0 con 0x7f09380d0e90 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.605+0000 7f092f400640 1 -- 172.21.15.39:0/10465884 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0928003d40 con 0x7f09380d0e90 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.605+0000 7f092f400640 1 -- 172.21.15.39:0/10465884 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092800aa90 con 0x7f09380d0e90 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.605+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09380caf90 con 0x7f09380d0e90 2024-01-24T17:47:32.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.605+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0938148120 con 0x7f09380d0e90 2024-01-24T17:47:32.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.606+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093804dff0 con 0x7f09380d0e90 2024-01-24T17:47:32.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.607+0000 7f092f400640 1 -- 172.21.15.39:0/10465884 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f092800abf0 con 0x7f09380d0e90 2024-01-24T17:47:32.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.608+0000 7f092f400640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f091006a430 0x7f091006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.608+0000 7f092f400640 1 -- 172.21.15.39:0/10465884 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0928088790 con 0x7f09380d0e90 2024-01-24T17:47:32.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.609+0000 7f092de00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f091006a430 0x7f091006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:32.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.610+0000 7f092de00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f091006a430 0x7f091006c8f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f09380cc560 tx=0x7f0920009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:32.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.612+0000 7f092f400640 1 -- 172.21.15.39:0/10465884 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0928053d70 con 0x7f09380d0e90 2024-01-24T17:47:32.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.809+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f093815acd0 con 0x7f09380d0e90 2024-01-24T17:47:32.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.812+0000 7f092f400640 1 -- 172.21.15.39:0/10465884 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+16117 (secure 0 0 0) 0x7f0928057390 con 0x7f09380d0e90 2024-01-24T17:47:32.813 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:32.813 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":87,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:31.936089+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:32.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.814+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f091006a430 msgr2=0x7f091006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.814+0000 7f093da00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f091006a430 0x7f091006c8f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f09380cc560 tx=0x7f0920009040 comp rx=0 tx=0).stop 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.814+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d0e90 msgr2=0x7f0938147be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.814+0000 7f093da00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d0e90 0x7f0938147be0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f092800c730 tx=0x7f092800cbf0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.815+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 shutdown_connections 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.815+0000 7f093da00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f093815cbe0 0x7f093815fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.815+0000 7f093da00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09380d12e0 0x7f093815f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.815+0000 7f093da00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f091006a430 0x7f091006c8f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.815+0000 7f093da00640 1 --2- 172.21.15.39:0/10465884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09380d0e90 0x7f0938147be0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.815+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 >> 172.21.15.39:0/10465884 conn(0x7f093809bc80 msgr2=0x7f093806c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.815+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 shutdown_connections 2024-01-24T17:47:32.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:32.815+0000 7f093da00640 1 -- 172.21.15.39:0/10465884 wait complete. 2024-01-24T17:47:32.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T17:47:33.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.059+0000 7efdd0c00640 1 Processor -- start 2024-01-24T17:47:33.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.059+0000 7efdd0c00640 1 -- start start 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdd0c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efdcc0b3780 0x7efdcc15e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdd0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc0cf190 0x7efdcc0cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdd0c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdcc15e280 con 0x7efdcc15dea0 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdd0c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc0cfb60 con 0x7efdcc0b3780 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdd0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0cdee0 con 0x7efdcc0cf190 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdcaa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efdcc0b3780 0x7efdcc15e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdcaa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7efdcc0b3780 0x7efdcc15e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59538/0 (socket says 172.21.15.39:59538) 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdcaa00640 1 -- 172.21.15.39:0/228958779 learned_addr learned my addr 172.21.15.39:0/228958779 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:33.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.060+0000 7efdcb400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efdcc15dea0 0x7efdcc0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53980/0 (socket says 172.21.15.39:53980) 2024-01-24T17:47:33.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.061+0000 7efdcbe00640 1 --2- 172.21.15.39:0/228958779 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc0cf190 0x7efdcc0cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:33.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.061+0000 7efdcaa00640 1 -- 172.21.15.39:0/228958779 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc0cf190 msgr2=0x7efdcc0cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:33.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.061+0000 7efdcaa00640 1 --2- 172.21.15.39:0/228958779 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc0cf190 0x7efdcc0cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.061+0000 7efdcaa00640 1 -- 172.21.15.39:0/228958779 >> v1:172.21.15.39:6789/0 conn(0x7efdcc15dea0 legacy=0x7efdcc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:33.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.061+0000 7efdca000640 1 -- 172.21.15.39:0/228958779 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053877934 0 0) 0x7efdcc15e280 con 0x7efdcc15dea0 2024-01-24T17:47:33.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.061+0000 7efdcaa00640 1 -- 172.21.15.39:0/228958779 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc0d0700 con 0x7efdcc0b3780 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.062+0000 7efdcaa00640 1 --2- 172.21.15.39:0/228958779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0b3780 0x7efdcc15e640 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009870 tx=0x7efdb402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63e272ddc0759f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.062+0000 7efdca000640 1 -- 172.21.15.39:0/228958779 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4030840 con 0x7efdcc0b3780 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.062+0000 7efdca000640 1 -- 172.21.15.39:0/228958779 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb40309a0 con 0x7efdcc0b3780 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.062+0000 7efdca000640 1 -- 172.21.15.39:0/228958779 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4030d30 con 0x7efdcc0b3780 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.063+0000 7efdd0c00640 1 -- 172.21.15.39:0/228958779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0b3780 msgr2=0x7efdcc15e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.063+0000 7efdd0c00640 1 --2- 172.21.15.39:0/228958779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0b3780 0x7efdcc15e640 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009870 tx=0x7efdb402b980 comp rx=0 tx=0).stop 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.063+0000 7efdd0c00640 1 -- 172.21.15.39:0/228958779 shutdown_connections 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.063+0000 7efdd0c00640 1 --2- 172.21.15.39:0/228958779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0b3780 0x7efdcc15e640 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.063+0000 7efdd0c00640 1 --2- 172.21.15.39:0/228958779 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc0cf190 0x7efdcc0cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.063+0000 7efdd0c00640 1 -- 172.21.15.39:0/228958779 >> 172.21.15.39:0/228958779 conn(0x7efdcc09bb30 msgr2=0x7efdcc0d27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.063+0000 7efdd0c00640 1 -- 172.21.15.39:0/228958779 shutdown_connections 2024-01-24T17:47:33.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.063+0000 7efdd0c00640 1 -- 172.21.15.39:0/228958779 wait complete. 2024-01-24T17:47:33.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.065+0000 7efdd0c00640 1 Processor -- start 2024-01-24T17:47:33.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.065+0000 7efdd0c00640 1 -- start start 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.066+0000 7efdd0c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdcc0b3780 0x7efdcc09f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.066+0000 7efdd0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0cf190 0x7efdcc0b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.066+0000 7efdd0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc15dea0 0x7efdcc1f1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.066+0000 7efdd0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc156800 con 0x7efdcc0cf190 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.066+0000 7efdd0c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc09fda0 con 0x7efdcc0b3780 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.066+0000 7efdd0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc159ef0 con 0x7efdcc15dea0 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.066+0000 7efdcaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0cf190 0x7efdcc0b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.066+0000 7efdcaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0cf190 0x7efdcc0b01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59544/0 (socket says 172.21.15.39:59544) 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcaa00640 1 -- 172.21.15.39:0/949283290 learned_addr learned my addr 172.21.15.39:0/949283290 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcb400640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdcc0b3780 0x7efdcc09f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcbe00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc15dea0 0x7efdcc1f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcaa00640 1 -- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc15dea0 msgr2=0x7efdcc1f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:33.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcaa00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc15dea0 0x7efdcc1f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcaa00640 1 -- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdcc0b3780 msgr2=0x7efdcc09f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:33.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcaa00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdcc0b3780 0x7efdcc09f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcaa00640 1 -- 172.21.15.39:0/949283290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb40094d0 con 0x7efdcc0cf190 2024-01-24T17:47:33.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.067+0000 7efdcaa00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0cf190 0x7efdcc0b01b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7efdb4000c00 tx=0x7efdb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:33.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.068+0000 7efdc3400640 1 -- 172.21.15.39:0/949283290 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb403e040 con 0x7efdcc0cf190 2024-01-24T17:47:33.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.068+0000 7efdc3400640 1 -- 172.21.15.39:0/949283290 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb4039e20 con 0x7efdcc0cf190 2024-01-24T17:47:33.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.068+0000 7efdc3400640 1 -- 172.21.15.39:0/949283290 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4004270 con 0x7efdcc0cf190 2024-01-24T17:47:33.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.068+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc09c4a0 con 0x7efdcc0cf190 2024-01-24T17:47:33.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.068+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc09c760 con 0x7efdcc0cf190 2024-01-24T17:47:33.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.069+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc067050 con 0x7efdcc0cf190 2024-01-24T17:47:33.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.070+0000 7efdc3400640 1 -- 172.21.15.39:0/949283290 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efdb40043d0 con 0x7efdcc0cf190 2024-01-24T17:47:33.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.071+0000 7efdc3400640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efda406a430 0x7efda406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:33.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.071+0000 7efdc3400640 1 -- 172.21.15.39:0/949283290 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efdb40ab030 con 0x7efdcc0cf190 2024-01-24T17:47:33.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.072+0000 7efdcb400640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efda406a430 0x7efda406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:33.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.073+0000 7efdcb400640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efda406a430 0x7efda406c8f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7efdbc007fd0 tx=0x7efdbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:33.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.076+0000 7efdc3400640 1 -- 172.21.15.39:0/949283290 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efdb4076740 con 0x7efdcc0cf190 2024-01-24T17:47:33.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.307+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7efdcc0cffb0 con 0x7efdcc0cf190 2024-01-24T17:47:33.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.958+0000 7efdc3400640 1 -- 172.21.15.39:0/949283290 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7efdb4079d60 con 0x7efdcc0cf190 2024-01-24T17:47:33.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efda406a430 msgr2=0x7efda406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:33.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efda406a430 0x7efda406c8f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7efdbc007fd0 tx=0x7efdbc009040 comp rx=0 tx=0).stop 2024-01-24T17:47:33.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0cf190 msgr2=0x7efdcc0b01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:33.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0cf190 0x7efdcc0b01b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7efdb4000c00 tx=0x7efdb4004730 comp rx=0 tx=0).stop 2024-01-24T17:47:33.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 shutdown_connections 2024-01-24T17:47:33.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efdcc15dea0 0x7efdcc1f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efda406a430 0x7efda406c8f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efdcc0cf190 0x7efdcc0b01b0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 --2- 172.21.15.39:0/949283290 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efdcc0b3780 0x7efdcc09f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:33.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 >> 172.21.15.39:0/949283290 conn(0x7efdcc09bb30 msgr2=0x7efdcc15d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:33.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 shutdown_connections 2024-01-24T17:47:33.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:33.962+0000 7efdd0c00640 1 -- 172.21.15.39:0/949283290 wait complete. 2024-01-24T17:47:33.985 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-24T17:47:33.986 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-24T17:47:34.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.209+0000 7f60e0c00640 1 Processor -- start 2024-01-24T17:47:34.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.210+0000 7f60e0c00640 1 -- start start 2024-01-24T17:47:34.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.211+0000 7f60e0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 0x7f60dc15a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.211+0000 7f60e0c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f60dc0ca6d0 0x7f60dc15b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.211+0000 7f60e0c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60dc15b040 con 0x7f60dc06bd40 2024-01-24T17:47:34.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.211+0000 7f60e0c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60dc06c1a0 con 0x7f60dc0ca6d0 2024-01-24T17:47:34.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.211+0000 7f60e0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc06c420 con 0x7f60dc0cadc0 2024-01-24T17:47:34.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.212+0000 7f60d3e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f60dc06bd40 0x7f60dc0cb1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:53992/0 (socket says 172.21.15.39:53992) 2024-01-24T17:47:34.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.212+0000 7f60d3e00640 1 -- 172.21.15.39:0/4204934395 learned_addr learned my addr 172.21.15.39:0/4204934395 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.212+0000 7f60dbe00640 1 --2- 172.21.15.39:0/4204934395 >> v2:172.21.15.100:3300/0 conn(0x7f60dc0ca6d0 0x7f60dc15b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.212+0000 7f60db400640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 0x7f60dc15a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.212+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135391594 0 0) 0x7f60dc15b040 con 0x7f60dc06bd40 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.212+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60bc003520 con 0x7f60dc06bd40 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1998691621 0 0) 0x7f60bc003520 con 0x7f60dc06bd40 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 msgr2=0x7f60dc15a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 0x7f60dc15a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 >> v2:172.21.15.100:3300/0 conn(0x7f60dc0ca6d0 msgr2=0x7f60dc15b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 --2- 172.21.15.39:0/4204934395 >> v2:172.21.15.100:3300/0 conn(0x7f60dc0ca6d0 0x7f60dc15b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc06c700 con 0x7f60dc06bd40 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f60c4001670 con 0x7f60dc06bd40 2024-01-24T17:47:34.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60db400640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 0x7f60dc15a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 >> v1:172.21.15.39:6789/0 conn(0x7f60dc06bd40 legacy=0x7f60dc0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60bc003970 0x7f60bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 0x7f60bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc06c1a0 con 0x7f60bc0047c0 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc06c420 con 0x7f60bc003970 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc00fd90 con 0x7f60dc0cadc0 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.213+0000 7f60d3e00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.214+0000 7f60db400640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60bc003970 0x7f60bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.214+0000 7f60dbe00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 0x7f60bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.214+0000 7f60d3e00640 1 -- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 msgr2=0x7f60bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:34.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.214+0000 7f60d3e00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 0x7f60bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:34.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.214+0000 7f60d3e00640 1 -- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60bc003970 msgr2=0x7f60bc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:34.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.214+0000 7f60d3e00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60bc003970 0x7f60bc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:34.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.214+0000 7f60d3e00640 1 -- 172.21.15.39:0/4204934395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc0093a0 con 0x7f60bc0047c0 2024-01-24T17:47:34.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.214+0000 7f60dbe00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc0cadc0 0x7f60bc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:34.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.215+0000 7f60d3e00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60bc004c40 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c110 tx=0x7f60bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9698da49c2cc45 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:34.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.215+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4005ad0 con 0x7f60bc0047c0 2024-01-24T17:47:34.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.215+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c4005c50 con 0x7f60bc0047c0 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.215+0000 7f60daa00640 1 -- 172.21.15.39:0/4204934395 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c402c830 con 0x7f60bc0047c0 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.216+0000 7f60e0c00640 1 -- 172.21.15.39:0/4204934395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 msgr2=0x7f60bc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.216+0000 7f60e0c00640 1 --2- 172.21.15.39:0/4204934395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60bc004c40 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c110 tx=0x7f60bc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.216+0000 7f60dbe00640 1 -- 172.21.15.39:0/4204934395 reap_dead start 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.216+0000 7f60e0c00640 1 -- 172.21.15.39:0/4204934395 shutdown_connections 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.216+0000 7f60e0c00640 1 -- 172.21.15.39:0/4204934395 >> 172.21.15.39:0/4204934395 conn(0x7f60dc06a200 msgr2=0x7f60dc0aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.216+0000 7f60e0c00640 1 -- 172.21.15.39:0/4204934395 shutdown_connections 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.216+0000 7f60e0c00640 1 -- 172.21.15.39:0/4204934395 wait complete. 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.218+0000 7f60e0c00640 1 Processor -- start 2024-01-24T17:47:34.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.218+0000 7f60e0c00640 1 -- start start 2024-01-24T17:47:34.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60e0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60dc15e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60e0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc15c050 0x7f60dc15c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60e0c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60dc0c5e50 0x7f60dc0c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60e0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc15da40 con 0x7f60bc0047c0 2024-01-24T17:47:34.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60e0c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc153400 con 0x7f60dc0c5e50 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60e0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc158b70 con 0x7f60dc15c050 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60db400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60dc15e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60db400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60dc15e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59550/0 (socket says 172.21.15.39:59550) 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60db400640 1 -- 172.21.15.39:0/2835818357 learned_addr learned my addr 172.21.15.39:0/2835818357 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.219+0000 7f60db400640 1 -- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc15c050 msgr2=0x7f60dc15c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60d3e00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc15c050 0x7f60dc15c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60dbe00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60dc0c5e50 0x7f60dc0c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60db400640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc15c050 0x7f60dc15c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60db400640 1 -- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60dc0c5e50 msgr2=0x7f60dc0c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60db400640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60dc0c5e50 0x7f60dc0c62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60db400640 1 -- 172.21.15.39:0/2835818357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc157f60 con 0x7f60bc0047c0 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60dbe00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60dc0c5e50 0x7f60dc0c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60db400640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60dc15e2b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0116d0 tx=0x7f60cc011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:34.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60d3400640 1 -- 172.21.15.39:0/2835818357 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00f9c0 con 0x7f60bc0047c0 2024-01-24T17:47:34.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60d3400640 1 -- 172.21.15.39:0/2835818357 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc00fb20 con 0x7f60bc0047c0 2024-01-24T17:47:34.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60d3400640 1 -- 172.21.15.39:0/2835818357 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00fe90 con 0x7f60bc0047c0 2024-01-24T17:47:34.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60dc158240 con 0x7f60bc0047c0 2024-01-24T17:47:34.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.220+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60dc158800 con 0x7f60bc0047c0 2024-01-24T17:47:34.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.222+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc04dff0 con 0x7f60bc0047c0 2024-01-24T17:47:34.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.223+0000 7f60d3400640 1 -- 172.21.15.39:0/2835818357 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f60cc015460 con 0x7f60bc0047c0 2024-01-24T17:47:34.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.223+0000 7f60d3400640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f60b406a430 0x7f60b406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:34.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.224+0000 7f60d3400640 1 -- 172.21.15.39:0/2835818357 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60cc0892d0 con 0x7f60bc0047c0 2024-01-24T17:47:34.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.224+0000 7f60d3e00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f60b406a430 0x7f60b406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:34.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.225+0000 7f60d3e00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f60b406a430 0x7f60b406c8f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f60c4034000 tx=0x7f60c4007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:34.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.228+0000 7f60d3400640 1 -- 172.21.15.39:0/2835818357 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f60cc054820 con 0x7f60bc0047c0 2024-01-24T17:47:34.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:34.459+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f60dc071f30 con 0x7f60bc0047c0 2024-01-24T17:47:35.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.315+0000 7f60d3400640 1 -- 172.21.15.39:0/2835818357 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f60cc057e40 con 0x7f60bc0047c0 2024-01-24T17:47:35.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f60b406a430 msgr2=0x7f60b406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f60b406a430 0x7f60b406c8f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f60c4034000 tx=0x7f60c4007000 comp rx=0 tx=0).stop 2024-01-24T17:47:35.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 msgr2=0x7f60dc15e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60dc15e2b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0116d0 tx=0x7f60cc011b90 comp rx=0 tx=0).stop 2024-01-24T17:47:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 shutdown_connections 2024-01-24T17:47:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60dc0c5e50 0x7f60dc0c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f60dc15c050 0x7f60dc15c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f60b406a430 0x7f60b406c8f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 --2- 172.21.15.39:0/2835818357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60bc0047c0 0x7f60dc15e2b0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.318+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 >> 172.21.15.39:0/2835818357 conn(0x7f60dc06a200 msgr2=0x7f60dc06fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.319+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 shutdown_connections 2024-01-24T17:47:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.319+0000 7f60e0c00640 1 -- 172.21.15.39:0/2835818357 wait complete. 2024-01-24T17:47:35.340 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-24T17:47:35.340 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:47:35.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.561+0000 7f44eda00640 1 Processor -- start 2024-01-24T17:47:35.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.561+0000 7f44eda00640 1 -- start start 2024-01-24T17:47:35.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.563+0000 7f44eda00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e8070920 0x7f44e80b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:35.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.563+0000 7f44eda00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e8070470 con 0x7f44e815f390 2024-01-24T17:47:35.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.563+0000 7f44eda00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e815ef00 con 0x7f44e80b0010 2024-01-24T17:47:35.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.563+0000 7f44eda00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80710a0 con 0x7f44e8070920 2024-01-24T17:47:35.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.564+0000 7f44e7400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f44e815f390 0x7f44e80c9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54000/0 (socket says 172.21.15.39:54000) 2024-01-24T17:47:35.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.564+0000 7f44e7400640 1 -- 172.21.15.39:0/308499538 learned_addr learned my addr 172.21.15.39:0/308499538 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:35.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.564+0000 7f44dde00640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e8070920 0x7f44e80b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:35.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.564+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788453768 0 0) 0x7f44e8070470 con 0x7f44e815f390 2024-01-24T17:47:35.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.564+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c8003520 con 0x7f44e815f390 2024-01-24T17:47:35.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294533366 0 0) 0x7f44c8003520 con 0x7f44e815f390 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e8070920 msgr2=0x7f44e80b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e8070920 0x7f44e80b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 >> v1:172.21.15.100:6789/0 conn(0x7f44e80b0010 legacy=0x7f44e815bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80b0930 con 0x7f44e815f390 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f44e00019f0 con 0x7f44e815f390 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 >> v1:172.21.15.39:6789/0 conn(0x7f44e815f390 legacy=0x7f44e80c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44c80039a0 0x7f44c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e8070920 0x7f44c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80710a0 con 0x7f44c8004d00 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c800baf0 con 0x7f44c80039a0 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c800bcb0 con 0x7f44e8070920 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e7400640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44dde00640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e8070920 0x7f44c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:35.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.565+0000 7f44e6a00640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44c80039a0 0x7f44c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:35.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.566+0000 7f44e7400640 1 -- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e8070920 msgr2=0x7f44c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:35.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.566+0000 7f44e7400640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e8070920 0x7f44c80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.566+0000 7f44e7400640 1 -- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44c80039a0 msgr2=0x7f44c8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:35.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.566+0000 7f44e7400640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44c80039a0 0x7f44c8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.566+0000 7f44e7400640 1 -- 172.21.15.39:0/308499538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c8003520 con 0x7f44c8004d00 2024-01-24T17:47:35.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.566+0000 7f44e6a00640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44c80039a0 0x7f44c8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:35.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44e7400640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44c8005180 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f44e000a700 tx=0x7f44c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1335a364801dfde8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:35.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0034630 con 0x7f44c8004d00 2024-01-24T17:47:35.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e002f070 con 0x7f44c8004d00 2024-01-24T17:47:35.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44e6000640 1 -- 172.21.15.39:0/308499538 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0034a10 con 0x7f44c8004d00 2024-01-24T17:47:35.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44eda00640 1 -- 172.21.15.39:0/308499538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 msgr2=0x7f44c8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44eda00640 1 --2- 172.21.15.39:0/308499538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44c8005180 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f44e000a700 tx=0x7f44c80057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44e7400640 1 -- 172.21.15.39:0/308499538 reap_dead start 2024-01-24T17:47:35.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44eda00640 1 -- 172.21.15.39:0/308499538 shutdown_connections 2024-01-24T17:47:35.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.567+0000 7f44eda00640 1 -- 172.21.15.39:0/308499538 >> 172.21.15.39:0/308499538 conn(0x7f44e806bf20 msgr2=0x7f44e806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:35.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.568+0000 7f44eda00640 1 -- 172.21.15.39:0/308499538 shutdown_connections 2024-01-24T17:47:35.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.568+0000 7f44eda00640 1 -- 172.21.15.39:0/308499538 wait complete. 2024-01-24T17:47:35.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.569+0000 7f44eda00640 1 Processor -- start 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.569+0000 7f44eda00640 1 -- start start 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.570+0000 7f44eda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44e8154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.570+0000 7f44eda00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e81f25a0 0x7f44e80c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.570+0000 7f44eda00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44e81f2a20 0x7f44e80c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.570+0000 7f44eda00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80cca50 con 0x7f44c8004d00 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.570+0000 7f44eda00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80c7120 con 0x7f44e81f2a20 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.570+0000 7f44eda00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80cc120 con 0x7f44e81f25a0 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44e8154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44e8154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59558/0 (socket says 172.21.15.39:59558) 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 -- 172.21.15.39:0/15163380 learned_addr learned my addr 172.21.15.39:0/15163380 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44dde00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e81f25a0 0x7f44e80c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e7400640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44e81f2a20 0x7f44e80c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 -- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e81f25a0 msgr2=0x7f44e80c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e81f25a0 0x7f44e80c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 -- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44e81f2a20 msgr2=0x7f44e80c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44e81f2a20 0x7f44e80c81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 -- 172.21.15.39:0/15163380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80cc400 con 0x7f44c8004d00 2024-01-24T17:47:35.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.571+0000 7f44e6a00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44e8154640 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f44d800d7a0 tx=0x7f44d800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:35.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.572+0000 7f44df400640 1 -- 172.21.15.39:0/15163380 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8017070 con 0x7f44c8004d00 2024-01-24T17:47:35.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.572+0000 7f44df400640 1 -- 172.21.15.39:0/15163380 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d8005070 con 0x7f44c8004d00 2024-01-24T17:47:35.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.572+0000 7f44df400640 1 -- 172.21.15.39:0/15163380 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8005420 con 0x7f44c8004d00 2024-01-24T17:47:35.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.572+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e80cc6e0 con 0x7f44c8004d00 2024-01-24T17:47:35.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.572+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e80cd0c0 con 0x7f44c8004d00 2024-01-24T17:47:35.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.573+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e804dff0 con 0x7f44c8004d00 2024-01-24T17:47:35.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.574+0000 7f44df400640 1 -- 172.21.15.39:0/15163380 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f44d8007030 con 0x7f44c8004d00 2024-01-24T17:47:35.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.575+0000 7f44df400640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44c006a460 0x7f44c006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:35.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.575+0000 7f44df400640 1 -- 172.21.15.39:0/15163380 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44d80881e0 con 0x7f44c8004d00 2024-01-24T17:47:35.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.575+0000 7f44dde00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44c006a460 0x7f44c006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:35.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.576+0000 7f44dde00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44c006a460 0x7f44c006c920 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f44e80c9140 tx=0x7f44d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:35.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.579+0000 7f44df400640 1 -- 172.21.15.39:0/15163380 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f44d80537c0 con 0x7f44c8004d00 2024-01-24T17:47:35.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.809+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44e8070700 con 0x7f44c8004d00 2024-01-24T17:47:35.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.811+0000 7f44df400640 1 -- 172.21.15.39:0/15163380 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f44d8056de0 con 0x7f44c8004d00 2024-01-24T17:47:35.811 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:35.811 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/57362003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":57362003},{"type":"v1","addr":"172.21.15.100:6836","nonce":57362003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6139,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/3662299631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":3662299631},{"type":"v1","addr":"172.21.15.100:6837","nonce":3662299631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2829786836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2829786836},{"type":"v1","addr":"172.21.15.39:6833","nonce":2829786836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:47:31.936549+0000","modified":"2024-01-24T17:47:35.302809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7293":{"gid":7293,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3073867595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3073867595},{"type":"v1","addr":"172.21.15.39:6835","nonce":3073867595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-24T17:47:35.812 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 51 2024-01-24T17:47:35.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.813+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44c006a460 msgr2=0x7f44c006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.813+0000 7f44eda00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44c006a460 0x7f44c006c920 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f44e80c9140 tx=0x7f44d0008040 comp rx=0 tx=0).stop 2024-01-24T17:47:35.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.813+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 msgr2=0x7f44e8154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.813+0000 7f44eda00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44e8154640 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f44d800d7a0 tx=0x7f44d800dc60 comp rx=0 tx=0).stop 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.813+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 shutdown_connections 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.814+0000 7f44eda00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f44e81f2a20 0x7f44e80c81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.814+0000 7f44eda00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f44e81f25a0 0x7f44e80c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.814+0000 7f44eda00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f44c006a460 0x7f44c006c920 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.814+0000 7f44eda00640 1 --2- 172.21.15.39:0/15163380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44c8004d00 0x7f44e8154640 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.814+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 >> 172.21.15.39:0/15163380 conn(0x7f44e806bf20 msgr2=0x7f44e80af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.814+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 shutdown_connections 2024-01-24T17:47:35.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:35.814+0000 7f44eda00640 1 -- 172.21.15.39:0/15163380 wait complete. 2024-01-24T17:47:35.834 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:36.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.043+0000 7f049b400640 1 Processor -- start 2024-01-24T17:47:36.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.043+0000 7f049b400640 1 -- start start 2024-01-24T17:47:36.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.044+0000 7f049b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0494159c10 0x7f04940ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.045+0000 7f049b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f049406d430 0x7f04940710e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.045+0000 7f049b400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04940cbd10 con 0x7f04940cd230 2024-01-24T17:47:36.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.045+0000 7f049b400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04940c65a0 con 0x7f049406d430 2024-01-24T17:47:36.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.045+0000 7f049b400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0494159ff0 con 0x7f0494159c10 2024-01-24T17:47:36.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.045+0000 7f049aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f049406d430 0x7f04940710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.045+0000 7f0499600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f04940cd230 0x7f049406cd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54004/0 (socket says 172.21.15.39:54004) 2024-01-24T17:47:36.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.045+0000 7f0499600640 1 -- 172.21.15.39:0/2689312842 learned_addr learned my addr 172.21.15.39:0/2689312842 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:36.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.045+0000 7f049a000640 1 --2- 172.21.15.39:0/2689312842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0494159c10 0x7f04940ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.046+0000 7f049aa00640 1 -- 172.21.15.39:0/2689312842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0494159c10 msgr2=0x7f04940ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:36.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.046+0000 7f049aa00640 1 --2- 172.21.15.39:0/2689312842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0494159c10 0x7f04940ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.046+0000 7f049aa00640 1 -- 172.21.15.39:0/2689312842 >> v1:172.21.15.39:6789/0 conn(0x7f04940cd230 legacy=0x7f049406cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.046+0000 7f049aa00640 1 -- 172.21.15.39:0/2689312842 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04940cb3e0 con 0x7f049406d430 2024-01-24T17:47:36.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.047+0000 7f049aa00640 1 --2- 172.21.15.39:0/2689312842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f049406d430 0x7f04940710e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f04840097a0 tx=0x7f048402db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25a77f2b98b7dea server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:36.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.047+0000 7f0498c00640 1 -- 172.21.15.39:0/2689312842 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048402f910 con 0x7f049406d430 2024-01-24T17:47:36.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.047+0000 7f0498c00640 1 -- 172.21.15.39:0/2689312842 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048402fa70 con 0x7f049406d430 2024-01-24T17:47:36.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.047+0000 7f0498c00640 1 -- 172.21.15.39:0/2689312842 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048402fe00 con 0x7f049406d430 2024-01-24T17:47:36.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.048+0000 7f049b400640 1 -- 172.21.15.39:0/2689312842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f049406d430 msgr2=0x7f04940710e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.048+0000 7f049b400640 1 --2- 172.21.15.39:0/2689312842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f049406d430 0x7f04940710e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f04840097a0 tx=0x7f048402db50 comp rx=0 tx=0).stop 2024-01-24T17:47:36.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.048+0000 7f049b400640 1 -- 172.21.15.39:0/2689312842 shutdown_connections 2024-01-24T17:47:36.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.048+0000 7f049b400640 1 --2- 172.21.15.39:0/2689312842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f049406d430 0x7f04940710e0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.048+0000 7f049b400640 1 --2- 172.21.15.39:0/2689312842 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0494159c10 0x7f04940ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.048+0000 7f049b400640 1 -- 172.21.15.39:0/2689312842 >> 172.21.15.39:0/2689312842 conn(0x7f04940cf660 msgr2=0x7f04940cfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:36.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.048+0000 7f049b400640 1 -- 172.21.15.39:0/2689312842 shutdown_connections 2024-01-24T17:47:36.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.048+0000 7f049b400640 1 -- 172.21.15.39:0/2689312842 wait complete. 2024-01-24T17:47:36.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.050+0000 7f049b400640 1 Processor -- start 2024-01-24T17:47:36.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.051+0000 7f049b400640 1 -- start start 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f049406d430 0x7f0494147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049b400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940cd230 0x7f04940c3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0494159c10 0x7f04940ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049b400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049414a690 con 0x7f0494159c10 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049b400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049414a970 con 0x7f04940cd230 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049b400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049414ac50 con 0x7f049406d430 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0494159c10 0x7f04940ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0494159c10 0x7f04940ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59576/0 (socket says 172.21.15.39:59576) 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049aa00640 1 -- 172.21.15.39:0/2179885291 learned_addr learned my addr 172.21.15.39:0/2179885291 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f049a000640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f049406d430 0x7f0494147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.052+0000 7f0499600640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940cd230 0x7f04940c3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f049aa00640 1 -- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f049406d430 msgr2=0x7f0494147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f049aa00640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f049406d430 0x7f0494147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f049aa00640 1 -- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940cd230 msgr2=0x7f04940c3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f049aa00640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940cd230 0x7f04940c3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f049aa00640 1 -- 172.21.15.39:0/2179885291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0484009400 con 0x7f0494159c10 2024-01-24T17:47:36.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f049aa00640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0494159c10 0x7f04940ce870 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f04840097a0 tx=0x7f04840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f0492000640 1 -- 172.21.15.39:0/2179885291 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0484037ca0 con 0x7f0494159c10 2024-01-24T17:47:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f0492000640 1 -- 172.21.15.39:0/2179885291 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484037e00 con 0x7f0494159c10 2024-01-24T17:47:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f0492000640 1 -- 172.21.15.39:0/2179885291 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840426b0 con 0x7f0494159c10 2024-01-24T17:47:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.053+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049414af30 con 0x7f0494159c10 2024-01-24T17:47:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.054+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04940c6d20 con 0x7f0494159c10 2024-01-24T17:47:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.055+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049414b450 con 0x7f0494159c10 2024-01-24T17:47:36.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.056+0000 7f0492000640 1 -- 172.21.15.39:0/2179885291 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0484042810 con 0x7f0494159c10 2024-01-24T17:47:36.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.057+0000 7f0492000640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046c06a430 0x7f046c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.057+0000 7f0492000640 1 -- 172.21.15.39:0/2179885291 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0484033070 con 0x7f0494159c10 2024-01-24T17:47:36.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.057+0000 7f049a000640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046c06a430 0x7f046c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.058+0000 7f049a000640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046c06a430 0x7f046c06c8f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f048c006fd0 tx=0x7f048c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:36.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.061+0000 7f0492000640 1 -- 172.21.15.39:0/2179885291 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f049414b450 con 0x7f0494159c10 2024-01-24T17:47:36.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.251+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f049414b450 con 0x7f0494159c10 2024-01-24T17:47:36.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.253+0000 7f0492000640 1 -- 172.21.15.39:0/2179885291 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16104 (secure 0 0 0) 0x7f049414b450 con 0x7f0494159c10 2024-01-24T17:47:36.254 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:36.254 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:32.941651+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046c06a430 msgr2=0x7f046c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046c06a430 0x7f046c06c8f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f048c006fd0 tx=0x7f048c008040 comp rx=0 tx=0).stop 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0494159c10 msgr2=0x7f04940ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0494159c10 0x7f04940ce870 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f04840097a0 tx=0x7f04840046c0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 shutdown_connections 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f046c06a430 0x7f046c06c8f0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0494159c10 0x7f04940ce870 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940cd230 0x7f04940c3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 --2- 172.21.15.39:0/2179885291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f049406d430 0x7f0494147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 >> 172.21.15.39:0/2179885291 conn(0x7f04940cf660 msgr2=0x7f0494158aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:36.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 shutdown_connections 2024-01-24T17:47:36.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.256+0000 7f049b400640 1 -- 172.21.15.39:0/2179885291 wait complete. 2024-01-24T17:47:36.276 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-24T17:47:36.276 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:47:36.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.488+0000 7faec6000640 1 Processor -- start 2024-01-24T17:47:36.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.488+0000 7faec6000640 1 -- start start 2024-01-24T17:47:36.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faec6000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec00cf200 0x7faec00d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faec6000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7faec00b1bf0 0x7faec00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faec6000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec00ce2f0 con 0x7faec00d05d0 2024-01-24T17:47:36.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faec6000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec00cd440 con 0x7faec00b1bf0 2024-01-24T17:47:36.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faec6000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec00ce780 con 0x7faec00cf200 2024-01-24T17:47:36.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faeb7c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7faec00d05d0 0x7faec00cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54008/0 (socket says 172.21.15.39:54008) 2024-01-24T17:47:36.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faeb7c00640 1 -- 172.21.15.39:0/4141899202 learned_addr learned my addr 172.21.15.39:0/4141899202 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:36.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faec5600640 1 --2- 172.21.15.39:0/4141899202 >> v2:172.21.15.100:3300/0 conn(0x7faec00b1bf0 0x7faec00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.490+0000 7faec4c00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec00cf200 0x7faec00d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912332720 0 0) 0x7faec00ce2f0 con 0x7faec00d05d0 2024-01-24T17:47:36.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea0003520 con 0x7faec00d05d0 2024-01-24T17:47:36.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583962850 0 0) 0x7faea0003520 con 0x7faec00d05d0 2024-01-24T17:47:36.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec00cf200 msgr2=0x7faec00d3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:36.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec00cf200 0x7faec00d3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 >> v2:172.21.15.100:3300/0 conn(0x7faec00b1bf0 msgr2=0x7faec00b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:36.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 --2- 172.21.15.39:0/4141899202 >> v2:172.21.15.100:3300/0 conn(0x7faec00b1bf0 0x7faec00b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec00cd6e0 con 0x7faec00d05d0 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7faea8001670 con 0x7faec00d05d0 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 >> v1:172.21.15.39:6789/0 conn(0x7faec00d05d0 legacy=0x7faec00cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea0003970 0x7faea0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec00cf200 0x7faea00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0004d00 0x7faea0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec00cd440 con 0x7faea0003970 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec00ce780 con 0x7faea0004d00 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.491+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea000ba00 con 0x7faec00cf200 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.492+0000 7faec4c00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea0003970 0x7faea0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.492+0000 7faeb7c00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec00cf200 0x7faea00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.492+0000 7faec5600640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0004d00 0x7faea0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.492+0000 7faec4c00640 1 -- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec00cf200 msgr2=0x7faea00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.492+0000 7faec4c00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec00cf200 0x7faea00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.492+0000 7faec4c00640 1 -- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0004d00 msgr2=0x7faea0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.492+0000 7faec4c00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0004d00 0x7faea0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.492+0000 7faec4c00640 1 -- 172.21.15.39:0/4141899202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0003520 con 0x7faea0003970 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.493+0000 7faec4c00640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea0003970 0x7faea0003d90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009ab0 tx=0x7faea00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19d504cb797290 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.493+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0005990 con 0x7faea0003970 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.493+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb0005af0 con 0x7faea0003970 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.493+0000 7faebfe00640 1 -- 172.21.15.39:0/4141899202 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0005ea0 con 0x7faea0003970 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.494+0000 7faec6000640 1 -- 172.21.15.39:0/4141899202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea0003970 msgr2=0x7faea0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.494+0000 7faec6000640 1 --2- 172.21.15.39:0/4141899202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea0003970 0x7faea0003d90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009ab0 tx=0x7faea00057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.494+0000 7faec5600640 1 -- 172.21.15.39:0/4141899202 reap_dead start 2024-01-24T17:47:36.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.494+0000 7faec6000640 1 -- 172.21.15.39:0/4141899202 shutdown_connections 2024-01-24T17:47:36.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.494+0000 7faec6000640 1 -- 172.21.15.39:0/4141899202 >> 172.21.15.39:0/4141899202 conn(0x7faec00d27f0 msgr2=0x7faec00d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:36.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.494+0000 7faec6000640 1 -- 172.21.15.39:0/4141899202 shutdown_connections 2024-01-24T17:47:36.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.494+0000 7faec6000640 1 -- 172.21.15.39:0/4141899202 wait complete. 2024-01-24T17:47:36.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.496+0000 7faec6000640 1 Processor -- start 2024-01-24T17:47:36.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.496+0000 7faec6000640 1 -- start start 2024-01-24T17:47:36.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.497+0000 7faec6000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0003970 0x7faec015cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.497+0000 7faec6000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec0151960 0x7faec01f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.497+0000 7faec6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faec01f1e50 0x7faec015e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.497+0000 7faec6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec00b69a0 con 0x7faec01f1e50 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.497+0000 7faec6000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec015d5e0 con 0x7faea0003970 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.497+0000 7faec6000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec015d8c0 con 0x7faec0151960 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.497+0000 7faec5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faec01f1e50 0x7faec015e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faec01f1e50 0x7faec015e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59604/0 (socket says 172.21.15.39:59604) 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec5600640 1 -- 172.21.15.39:0/1583390716 learned_addr learned my addr 172.21.15.39:0/1583390716 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec4c00640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0003970 0x7faec015cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faeb7c00640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec0151960 0x7faec01f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec5600640 1 -- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec0151960 msgr2=0x7faec01f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec5600640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec0151960 0x7faec01f1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec5600640 1 -- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0003970 msgr2=0x7faec015cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec5600640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0003970 0x7faec015cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec5600640 1 -- 172.21.15.39:0/1583390716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec015dba0 con 0x7faec01f1e50 2024-01-24T17:47:36.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.498+0000 7faec5600640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faec01f1e50 0x7faec015e970 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7faec00d0b60 tx=0x7faeb80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:36.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.499+0000 7faebd600640 1 -- 172.21.15.39:0/1583390716 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb800e070 con 0x7faec01f1e50 2024-01-24T17:47:36.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.499+0000 7faebd600640 1 -- 172.21.15.39:0/1583390716 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8011e10 con 0x7faec01f1e50 2024-01-24T17:47:36.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.499+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec015de80 con 0x7faec01f1e50 2024-01-24T17:47:36.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.499+0000 7faebd600640 1 -- 172.21.15.39:0/1583390716 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8003990 con 0x7faec01f1e50 2024-01-24T17:47:36.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.499+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec015eeb0 con 0x7faec01f1e50 2024-01-24T17:47:36.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.500+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec00b5410 con 0x7faec01f1e50 2024-01-24T17:47:36.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.501+0000 7faebd600640 1 -- 172.21.15.39:0/1583390716 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7faeb8003af0 con 0x7faec01f1e50 2024-01-24T17:47:36.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.502+0000 7faebd600640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae9806a430 0x7fae9806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:36.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.502+0000 7faebd600640 1 -- 172.21.15.39:0/1583390716 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faeb8088ec0 con 0x7faec01f1e50 2024-01-24T17:47:36.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.503+0000 7faec4c00640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae9806a430 0x7fae9806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:36.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.504+0000 7faec4c00640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae9806a430 0x7fae9806c8f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7faeb0035000 tx=0x7faeb0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:36.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.507+0000 7faebd600640 1 -- 172.21.15.39:0/1583390716 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7faec00b5410 con 0x7faec01f1e50 2024-01-24T17:47:36.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.696+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faec00cdca0 con 0x7fae9806a430 2024-01-24T17:47:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.720+0000 7faebd600640 1 -- 172.21.15.39:0/1583390716 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287482 (secure 0 0 0) 0x7faeb0005d60 con 0x7fae9806a430 2024-01-24T17:47:36.723 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae9806a430 msgr2=0x7fae9806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae9806a430 0x7fae9806c8f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7faeb0035000 tx=0x7faeb0030000 comp rx=0 tx=0).stop 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faec01f1e50 msgr2=0x7faec015e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faec01f1e50 0x7faec015e970 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7faec00d0b60 tx=0x7faeb80115a0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 shutdown_connections 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae9806a430 0x7fae9806c8f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faec01f1e50 0x7faec015e970 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faec0151960 0x7faec01f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 --2- 172.21.15.39:0/1583390716 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faea0003970 0x7faec015cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.727+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 >> 172.21.15.39:0/1583390716 conn(0x7faec00d27f0 msgr2=0x7faec006c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:36.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.728+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 shutdown_connections 2024-01-24T17:47:36.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:36.728+0000 7faec6000640 1 -- 172.21.15.39:0/1583390716 wait complete. 2024-01-24T17:47:36.747 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-01-24T17:47:36.340685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258424,"kb_used_data":9460,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754716296,"statfs":{"total":773094113280,"available":772829487104,"internally_reserved":0,"allocated":9687040,"data_stored":4798413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.907629"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987878+0000","last_change":"2024-01-24T17:47:31.978602+0000","last_active":"2024-01-24T17:47:31.987878+0000","last_peered":"2024-01-24T17:47:31.987878+0000","last_clean":"2024-01-24T17:47:31.987878+0000","last_became_active":"2024-01-24T17:47:31.975764+0000","last_became_peered":"2024-01-24T17:47:31.975764+0000","last_unstale":"2024-01-24T17:47:31.987878+0000","last_undegraded":"2024-01-24T17:47:31.987878+0000","last_fullsized":"2024-01-24T17:47:31.987878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:50.606488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989598+0000","last_change":"2024-01-24T17:47:31.988043+0000","last_active":"2024-01-24T17:47:31.989598+0000","last_peered":"2024-01-24T17:47:31.989598+0000","last_clean":"2024-01-24T17:47:31.989598+0000","last_became_active":"2024-01-24T17:47:31.987625+0000","last_became_peered":"2024-01-24T17:47:31.987625+0000","last_unstale":"2024-01-24T17:47:31.989598+0000","last_undegraded":"2024-01-24T17:47:31.989598+0000","last_fullsized":"2024-01-24T17:47:31.989598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:36.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.973573+0000","last_change":"2024-01-24T17:47:31.969778+0000","last_active":"2024-01-24T17:47:31.973573+0000","last_peered":"2024-01-24T17:47:31.973573+0000","last_clean":"2024-01-24T17:47:31.973573+0000","last_became_active":"2024-01-24T17:47:31.969522+0000","last_became_peered":"2024-01-24T17:47:31.969522+0000","last_unstale":"2024-01-24T17:47:31.973573+0000","last_undegraded":"2024-01-24T17:47:31.973573+0000","last_fullsized":"2024-01-24T17:47:31.973573+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:39:01.873919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.985803+0000","last_change":"2024-01-24T17:47:31.976646+0000","last_active":"2024-01-24T17:47:31.985803+0000","last_peered":"2024-01-24T17:47:31.985803+0000","last_clean":"2024-01-24T17:47:31.985803+0000","last_became_active":"2024-01-24T17:47:31.976249+0000","last_became_peered":"2024-01-24T17:47:31.976249+0000","last_unstale":"2024-01-24T17:47:31.985803+0000","last_undegraded":"2024-01-24T17:47:31.985803+0000","last_fullsized":"2024-01-24T17:47:31.985803+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:01:55.725510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986743+0000","last_change":"2024-01-24T17:47:31.979278+0000","last_active":"2024-01-24T17:47:31.986743+0000","last_peered":"2024-01-24T17:47:31.986743+0000","last_clean":"2024-01-24T17:47:31.986743+0000","last_became_active":"2024-01-24T17:47:31.978982+0000","last_became_peered":"2024-01-24T17:47:31.978982+0000","last_unstale":"2024-01-24T17:47:31.986743+0000","last_undegraded":"2024-01-24T17:47:31.986743+0000","last_fullsized":"2024-01-24T17:47:31.986743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:47:24.166509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.992486+0000","last_change":"2024-01-24T17:47:31.985796+0000","last_active":"2024-01-24T17:47:31.992486+0000","last_peered":"2024-01-24T17:47:31.992486+0000","last_clean":"2024-01-24T17:47:31.992486+0000","last_became_active":"2024-01-24T17:47:31.985418+0000","last_became_peered":"2024-01-24T17:47:31.985418+0000","last_unstale":"2024-01-24T17:47:31.992486+0000","last_undegraded":"2024-01-24T17:47:31.992486+0000","last_fullsized":"2024-01-24T17:47:31.992486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:02.086314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989228+0000","last_change":"2024-01-24T17:47:31.985251+0000","last_active":"2024-01-24T17:47:31.989228+0000","last_peered":"2024-01-24T17:47:31.989228+0000","last_clean":"2024-01-24T17:47:31.989228+0000","last_became_active":"2024-01-24T17:47:31.984888+0000","last_became_peered":"2024-01-24T17:47:31.984888+0000","last_unstale":"2024-01-24T17:47:31.989228+0000","last_undegraded":"2024-01-24T17:47:31.989228+0000","last_fullsized":"2024-01-24T17:47:31.989228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:29.312975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.978282+0000","last_change":"2024-01-24T17:47:31.972718+0000","last_active":"2024-01-24T17:47:31.978282+0000","last_peered":"2024-01-24T17:47:31.978282+0000","last_clean":"2024-01-24T17:47:31.978282+0000","last_became_active":"2024-01-24T17:47:31.971566+0000","last_became_peered":"2024-01-24T17:47:31.971566+0000","last_unstale":"2024-01-24T17:47:31.978282+0000","last_undegraded":"2024-01-24T17:47:31.978282+0000","last_fullsized":"2024-01-24T17:47:31.978282+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:41.672001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988168+0000","last_change":"2024-01-24T17:47:31.979950+0000","last_active":"2024-01-24T17:47:31.988168+0000","last_peered":"2024-01-24T17:47:31.988168+0000","last_clean":"2024-01-24T17:47:31.988168+0000","last_became_active":"2024-01-24T17:47:31.979492+0000","last_became_peered":"2024-01-24T17:47:31.979492+0000","last_unstale":"2024-01-24T17:47:31.988168+0000","last_undegraded":"2024-01-24T17:47:31.988168+0000","last_fullsized":"2024-01-24T17:47:31.988168+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:29.312975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989433+0000","last_change":"2024-01-24T17:47:31.982990+0000","last_active":"2024-01-24T17:47:31.989433+0000","last_peered":"2024-01-24T17:47:31.989433+0000","last_clean":"2024-01-24T17:47:31.989433+0000","last_became_active":"2024-01-24T17:47:31.982732+0000","last_became_peered":"2024-01-24T17:47:31.982732+0000","last_unstale":"2024-01-24T17:47:31.989433+0000","last_undegraded":"2024-01-24T17:47:31.989433+0000","last_fullsized":"2024-01-24T17:47:31.989433+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:45:52.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.976356+0000","last_change":"2024-01-24T17:47:31.969482+0000","last_active":"2024-01-24T17:47:31.976356+0000","last_peered":"2024-01-24T17:47:31.976356+0000","last_clean":"2024-01-24T17:47:31.976356+0000","last_became_active":"2024-01-24T17:47:31.969131+0000","last_became_peered":"2024-01-24T17:47:31.969131+0000","last_unstale":"2024-01-24T17:47:31.976356+0000","last_undegraded":"2024-01-24T17:47:31.976356+0000","last_fullsized":"2024-01-24T17:47:31.976356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:42.446661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982536+0000","last_change":"2024-01-24T17:47:31.976914+0000","last_active":"2024-01-24T17:47:31.982536+0000","last_peered":"2024-01-24T17:47:31.982536+0000","last_clean":"2024-01-24T17:47:31.982536+0000","last_became_active":"2024-01-24T17:47:31.976624+0000","last_became_peered":"2024-01-24T17:47:31.976624+0000","last_unstale":"2024-01-24T17:47:31.982536+0000","last_undegraded":"2024-01-24T17:47:31.982536+0000","last_fullsized":"2024-01-24T17:47:31.982536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:02.086314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.981604+0000","last_change":"2024-01-24T17:47:31.970970+0000","last_active":"2024-01-24T17:47:31.981604+0000","last_peered":"2024-01-24T17:47:31.981604+0000","last_clean":"2024-01-24T17:47:31.981604+0000","last_became_active":"2024-01-24T17:47:31.970702+0000","last_became_peered":"2024-01-24T17:47:31.970702+0000","last_unstale":"2024-01-24T17:47:31.981604+0000","last_undegraded":"2024-01-24T17:47:31.981604+0000","last_fullsized":"2024-01-24T17:47:31.981604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:49.369796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987581+0000","last_change":"2024-01-24T17:47:31.985226+0000","last_active":"2024-01-24T17:47:31.987581+0000","last_peered":"2024-01-24T17:47:31.987581+0000","last_clean":"2024-01-24T17:47:31.987581+0000","last_became_active":"2024-01-24T17:47:31.982699+0000","last_became_peered":"2024-01-24T17:47:31.982699+0000","last_unstale":"2024-01-24T17:47:31.987581+0000","last_undegraded":"2024-01-24T17:47:31.987581+0000","last_fullsized":"2024-01-24T17:47:31.987581+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:27:00.612042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989090+0000","last_change":"2024-01-24T17:47:31.982126+0000","last_active":"2024-01-24T17:47:31.989090+0000","last_peered":"2024-01-24T17:47:31.989090+0000","last_clean":"2024-01-24T17:47:31.989090+0000","last_became_active":"2024-01-24T17:47:31.981499+0000","last_became_peered":"2024-01-24T17:47:31.981499+0000","last_unstale":"2024-01-24T17:47:31.989090+0000","last_undegraded":"2024-01-24T17:47:31.989090+0000","last_fullsized":"2024-01-24T17:47:31.989090+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:44:25.582344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.973161+0000","last_change":"2024-01-24T17:47:31.969027+0000","last_active":"2024-01-24T17:47:31.973161+0000","last_peered":"2024-01-24T17:47:31.973161+0000","last_clean":"2024-01-24T17:47:31.973161+0000","last_became_active":"2024-01-24T17:47:31.968140+0000","last_became_peered":"2024-01-24T17:47:31.968140+0000","last_unstale":"2024-01-24T17:47:31.973161+0000","last_undegraded":"2024-01-24T17:47:31.973161+0000","last_fullsized":"2024-01-24T17:47:31.973161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:01:55.725510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.980899+0000","last_change":"2024-01-24T17:47:31.975245+0000","last_active":"2024-01-24T17:47:31.980899+0000","last_peered":"2024-01-24T17:47:31.980899+0000","last_clean":"2024-01-24T17:47:31.980899+0000","last_became_active":"2024-01-24T17:47:31.974992+0000","last_became_peered":"2024-01-24T17:47:31.974992+0000","last_unstale":"2024-01-24T17:47:31.980899+0000","last_undegraded":"2024-01-24T17:47:31.980899+0000","last_fullsized":"2024-01-24T17:47:31.980899+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:05:57.892601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.976432+0000","last_change":"2024-01-24T17:47:31.970431+0000","last_active":"2024-01-24T17:47:31.976432+0000","last_peered":"2024-01-24T17:47:31.976432+0000","last_clean":"2024-01-24T17:47:31.976432+0000","last_became_active":"2024-01-24T17:47:31.969836+0000","last_became_peered":"2024-01-24T17:47:31.969836+0000","last_unstale":"2024-01-24T17:47:31.976432+0000","last_undegraded":"2024-01-24T17:47:31.976432+0000","last_fullsized":"2024-01-24T17:47:31.976432+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:22:35.200706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984293+0000","last_change":"2024-01-24T17:47:31.982167+0000","last_active":"2024-01-24T17:47:31.984293+0000","last_peered":"2024-01-24T17:47:31.984293+0000","last_clean":"2024-01-24T17:47:31.984293+0000","last_became_active":"2024-01-24T17:47:31.981907+0000","last_became_peered":"2024-01-24T17:47:31.981907+0000","last_unstale":"2024-01-24T17:47:31.984293+0000","last_undegraded":"2024-01-24T17:47:31.984293+0000","last_fullsized":"2024-01-24T17:47:31.984293+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:02.086314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987287+0000","last_change":"2024-01-24T17:47:31.983379+0000","last_active":"2024-01-24T17:47:31.987287+0000","last_peered":"2024-01-24T17:47:31.987287+0000","last_clean":"2024-01-24T17:47:31.987287+0000","last_became_active":"2024-01-24T17:47:31.982770+0000","last_became_peered":"2024-01-24T17:47:31.982770+0000","last_unstale":"2024-01-24T17:47:31.987287+0000","last_undegraded":"2024-01-24T17:47:31.987287+0000","last_fullsized":"2024-01-24T17:47:31.987287+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:37:29.833370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986542+0000","last_change":"2024-01-24T17:47:31.980010+0000","last_active":"2024-01-24T17:47:31.986542+0000","last_peered":"2024-01-24T17:47:31.986542+0000","last_clean":"2024-01-24T17:47:31.986542+0000","last_became_active":"2024-01-24T17:47:31.978501+0000","last_became_peered":"2024-01-24T17:47:31.978501+0000","last_unstale":"2024-01-24T17:47:31.986542+0000","last_undegraded":"2024-01-24T17:47:31.986542+0000","last_fullsized":"2024-01-24T17:47:31.986542+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:39:01.873919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986642+0000","last_change":"2024-01-24T17:47:31.980468+0000","last_active":"2024-01-24T17:47:31.986642+0000","last_peered":"2024-01-24T17:47:31.986642+0000","last_clean":"2024-01-24T17:47:31.986642+0000","last_became_active":"2024-01-24T17:47:31.980182+0000","last_became_peered":"2024-01-24T17:47:31.980182+0000","last_unstale":"2024-01-24T17:47:31.986642+0000","last_undegraded":"2024-01-24T17:47:31.986642+0000","last_fullsized":"2024-01-24T17:47:31.986642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:41.672001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.981112+0000","last_change":"2024-01-24T17:47:31.975478+0000","last_active":"2024-01-24T17:47:31.981112+0000","last_peered":"2024-01-24T17:47:31.981112+0000","last_clean":"2024-01-24T17:47:31.981112+0000","last_became_active":"2024-01-24T17:47:31.975178+0000","last_became_peered":"2024-01-24T17:47:31.975178+0000","last_unstale":"2024-01-24T17:47:31.981112+0000","last_undegraded":"2024-01-24T17:47:31.981112+0000","last_fullsized":"2024-01-24T17:47:31.981112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:25:35.181055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984509+0000","last_change":"2024-01-24T17:47:31.977142+0000","last_active":"2024-01-24T17:47:31.984509+0000","last_peered":"2024-01-24T17:47:31.984509+0000","last_clean":"2024-01-24T17:47:31.984509+0000","last_became_active":"2024-01-24T17:47:31.976847+0000","last_became_peered":"2024-01-24T17:47:31.976847+0000","last_unstale":"2024-01-24T17:47:31.984509+0000","last_undegraded":"2024-01-24T17:47:31.984509+0000","last_fullsized":"2024-01-24T17:47:31.984509+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:14.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988429+0000","last_change":"2024-01-24T17:47:31.981607+0000","last_active":"2024-01-24T17:47:31.988429+0000","last_peered":"2024-01-24T17:47:31.988429+0000","last_clean":"2024-01-24T17:47:31.988429+0000","last_became_active":"2024-01-24T17:47:31.981270+0000","last_became_peered":"2024-01-24T17:47:31.981270+0000","last_unstale":"2024-01-24T17:47:31.988429+0000","last_undegraded":"2024-01-24T17:47:31.988429+0000","last_fullsized":"2024-01-24T17:47:31.988429+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:06:51.007190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987653+0000","last_change":"2024-01-24T17:47:31.982887+0000","last_active":"2024-01-24T17:47:31.987653+0000","last_peered":"2024-01-24T17:47:31.987653+0000","last_clean":"2024-01-24T17:47:31.987653+0000","last_became_active":"2024-01-24T17:47:31.982336+0000","last_became_peered":"2024-01-24T17:47:31.982336+0000","last_unstale":"2024-01-24T17:47:31.987653+0000","last_undegraded":"2024-01-24T17:47:31.987653+0000","last_fullsized":"2024-01-24T17:47:31.987653+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:07:19.226610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987936+0000","last_change":"2024-01-24T17:47:31.981798+0000","last_active":"2024-01-24T17:47:31.987936+0000","last_peered":"2024-01-24T17:47:31.987936+0000","last_clean":"2024-01-24T17:47:31.987936+0000","last_became_active":"2024-01-24T17:47:31.981509+0000","last_became_peered":"2024-01-24T17:47:31.981509+0000","last_unstale":"2024-01-24T17:47:31.987936+0000","last_undegraded":"2024-01-24T17:47:31.987936+0000","last_fullsized":"2024-01-24T17:47:31.987936+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:18:05.792016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988031+0000","last_change":"2024-01-24T17:47:31.980086+0000","last_active":"2024-01-24T17:47:31.988031+0000","last_peered":"2024-01-24T17:47:31.988031+0000","last_clean":"2024-01-24T17:47:31.988031+0000","last_became_active":"2024-01-24T17:47:31.979832+0000","last_became_peered":"2024-01-24T17:47:31.979832+0000","last_unstale":"2024-01-24T17:47:31.988031+0000","last_undegraded":"2024-01-24T17:47:31.988031+0000","last_fullsized":"2024-01-24T17:47:31.988031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:36.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984218+0000","last_change":"2024-01-24T17:47:31.979202+0000","last_active":"2024-01-24T17:47:31.984218+0000","last_peered":"2024-01-24T17:47:31.984218+0000","last_clean":"2024-01-24T17:47:31.984218+0000","last_became_active":"2024-01-24T17:47:31.978020+0000","last_became_peered":"2024-01-24T17:47:31.978020+0000","last_unstale":"2024-01-24T17:47:31.984218+0000","last_undegraded":"2024-01-24T17:47:31.984218+0000","last_fullsized":"2024-01-24T17:47:31.984218+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:42:33.273575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988724+0000","last_change":"2024-01-24T17:47:31.986551+0000","last_active":"2024-01-24T17:47:31.988724+0000","last_peered":"2024-01-24T17:47:31.988724+0000","last_clean":"2024-01-24T17:47:31.988724+0000","last_became_active":"2024-01-24T17:47:31.986260+0000","last_became_peered":"2024-01-24T17:47:31.986260+0000","last_unstale":"2024-01-24T17:47:31.988724+0000","last_undegraded":"2024-01-24T17:47:31.988724+0000","last_fullsized":"2024-01-24T17:47:31.988724+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:50:13.325782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.992498+0000","last_change":"2024-01-24T17:47:31.986372+0000","last_active":"2024-01-24T17:47:31.992498+0000","last_peered":"2024-01-24T17:47:31.992498+0000","last_clean":"2024-01-24T17:47:31.992498+0000","last_became_active":"2024-01-24T17:47:31.984535+0000","last_became_peered":"2024-01-24T17:47:31.984535+0000","last_unstale":"2024-01-24T17:47:31.992498+0000","last_undegraded":"2024-01-24T17:47:31.992498+0000","last_fullsized":"2024-01-24T17:47:31.992498+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:42:33.273575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984514+0000","last_change":"2024-01-24T17:47:31.981085+0000","last_active":"2024-01-24T17:47:31.984514+0000","last_peered":"2024-01-24T17:47:31.984514+0000","last_clean":"2024-01-24T17:47:31.984514+0000","last_became_active":"2024-01-24T17:47:31.980820+0000","last_became_peered":"2024-01-24T17:47:31.980820+0000","last_unstale":"2024-01-24T17:47:31.984514+0000","last_undegraded":"2024-01-24T17:47:31.984514+0000","last_fullsized":"2024-01-24T17:47:31.984514+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:36:49.719214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982863+0000","last_change":"2024-01-24T17:47:31.977921+0000","last_active":"2024-01-24T17:47:31.982863+0000","last_peered":"2024-01-24T17:47:31.982863+0000","last_clean":"2024-01-24T17:47:31.982863+0000","last_became_active":"2024-01-24T17:47:31.976080+0000","last_became_peered":"2024-01-24T17:47:31.976080+0000","last_unstale":"2024-01-24T17:47:31.982863+0000","last_undegraded":"2024-01-24T17:47:31.982863+0000","last_fullsized":"2024-01-24T17:47:31.982863+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:45:52.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"87'4","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986228+0000","last_change":"2024-01-24T17:47:31.469900+0000","last_active":"2024-01-24T17:47:31.986228+0000","last_peered":"2024-01-24T17:47:31.986228+0000","last_clean":"2024-01-24T17:47:31.986228+0000","last_became_active":"2024-01-24T17:47:31.469573+0000","last_became_peered":"2024-01-24T17:47:31.469573+0000","last_unstale":"2024-01-24T17:47:31.986228+0000","last_undegraded":"2024-01-24T17:47:31.986228+0000","last_fullsized":"2024-01-24T17:47:31.986228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:43:24.898534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.985151+0000","last_change":"2024-01-24T17:47:31.975999+0000","last_active":"2024-01-24T17:47:31.985151+0000","last_peered":"2024-01-24T17:47:31.985151+0000","last_clean":"2024-01-24T17:47:31.985151+0000","last_became_active":"2024-01-24T17:47:31.975475+0000","last_became_peered":"2024-01-24T17:47:31.975475+0000","last_unstale":"2024-01-24T17:47:31.985151+0000","last_undegraded":"2024-01-24T17:47:31.985151+0000","last_fullsized":"2024-01-24T17:47:31.985151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:14.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1e","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.971733+0000","last_change":"2024-01-24T17:47:31.486123+0000","last_active":"2024-01-24T17:47:31.971733+0000","last_peered":"2024-01-24T17:47:31.971733+0000","last_clean":"2024-01-24T17:47:31.971733+0000","last_became_active":"2024-01-24T17:47:31.485142+0000","last_became_peered":"2024-01-24T17:47:31.485142+0000","last_unstale":"2024-01-24T17:47:31.971733+0000","last_undegraded":"2024-01-24T17:47:31.971733+0000","last_fullsized":"2024-01-24T17:47:31.971733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:26:18.040164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982554+0000","last_change":"2024-01-24T17:47:31.980165+0000","last_active":"2024-01-24T17:47:31.982554+0000","last_peered":"2024-01-24T17:47:31.982554+0000","last_clean":"2024-01-24T17:47:31.982554+0000","last_became_active":"2024-01-24T17:47:31.979897+0000","last_became_peered":"2024-01-24T17:47:31.979897+0000","last_unstale":"2024-01-24T17:47:31.982554+0000","last_undegraded":"2024-01-24T17:47:31.982554+0000","last_fullsized":"2024-01-24T17:47:31.982554+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:50:13.325782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1d","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.985998+0000","last_change":"2024-01-24T17:47:31.474567+0000","last_active":"2024-01-24T17:47:31.985998+0000","last_peered":"2024-01-24T17:47:31.985998+0000","last_clean":"2024-01-24T17:47:31.985998+0000","last_became_active":"2024-01-24T17:47:31.473003+0000","last_became_peered":"2024-01-24T17:47:31.473003+0000","last_unstale":"2024-01-24T17:47:31.985998+0000","last_undegraded":"2024-01-24T17:47:31.985998+0000","last_fullsized":"2024-01-24T17:47:31.985998+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:19:02.354998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988033+0000","last_change":"2024-01-24T17:47:31.980899+0000","last_active":"2024-01-24T17:47:31.988033+0000","last_peered":"2024-01-24T17:47:31.988033+0000","last_clean":"2024-01-24T17:47:31.988033+0000","last_became_active":"2024-01-24T17:47:31.979510+0000","last_became_peered":"2024-01-24T17:47:31.979510+0000","last_unstale":"2024-01-24T17:47:31.988033+0000","last_undegraded":"2024-01-24T17:47:31.988033+0000","last_fullsized":"2024-01-24T17:47:31.988033+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:44:25.582344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1c","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986121+0000","last_change":"2024-01-24T17:47:31.475836+0000","last_active":"2024-01-24T17:47:31.986121+0000","last_peered":"2024-01-24T17:47:31.986121+0000","last_clean":"2024-01-24T17:47:31.986121+0000","last_became_active":"2024-01-24T17:47:31.475494+0000","last_became_peered":"2024-01-24T17:47:31.475494+0000","last_unstale":"2024-01-24T17:47:31.986121+0000","last_undegraded":"2024-01-24T17:47:31.986121+0000","last_fullsized":"2024-01-24T17:47:31.986121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:50:12.311017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984150+0000","last_change":"2024-01-24T17:47:31.978201+0000","last_active":"2024-01-24T17:47:31.984150+0000","last_peered":"2024-01-24T17:47:31.984150+0000","last_clean":"2024-01-24T17:47:31.984150+0000","last_became_active":"2024-01-24T17:47:31.977899+0000","last_became_peered":"2024-01-24T17:47:31.977899+0000","last_unstale":"2024-01-24T17:47:31.984150+0000","last_undegraded":"2024-01-24T17:47:31.984150+0000","last_fullsized":"2024-01-24T17:47:31.984150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:18:28.650673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1b","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.970911+0000","last_change":"2024-01-24T17:47:31.480673+0000","last_active":"2024-01-24T17:47:31.970911+0000","last_peered":"2024-01-24T17:47:31.970911+0000","last_clean":"2024-01-24T17:47:31.970911+0000","last_became_active":"2024-01-24T17:47:31.480287+0000","last_became_peered":"2024-01-24T17:47:31.480287+0000","last_unstale":"2024-01-24T17:47:31.970911+0000","last_undegraded":"2024-01-24T17:47:31.970911+0000","last_fullsized":"2024-01-24T17:47:31.970911+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:28.298210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.973632+0000","last_change":"2024-01-24T17:47:31.972189+0000","last_active":"2024-01-24T17:47:31.973632+0000","last_peered":"2024-01-24T17:47:31.973632+0000","last_clean":"2024-01-24T17:47:31.973632+0000","last_became_active":"2024-01-24T17:47:31.971764+0000","last_became_peered":"2024-01-24T17:47:31.971764+0000","last_unstale":"2024-01-24T17:47:31.973632+0000","last_undegraded":"2024-01-24T17:47:31.973632+0000","last_fullsized":"2024-01-24T17:47:31.973632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:05:01.572141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1a","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984659+0000","last_change":"2024-01-24T17:47:31.480183+0000","last_active":"2024-01-24T17:47:31.984659+0000","last_peered":"2024-01-24T17:47:31.984659+0000","last_clean":"2024-01-24T17:47:31.984659+0000","last_became_active":"2024-01-24T17:47:31.479839+0000","last_became_peered":"2024-01-24T17:47:31.479839+0000","last_unstale":"2024-01-24T17:47:31.984659+0000","last_undegraded":"2024-01-24T17:47:31.984659+0000","last_fullsized":"2024-01-24T17:47:31.984659+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:45:11.903479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.985233+0000","last_change":"2024-01-24T17:47:31.977862+0000","last_active":"2024-01-24T17:47:31.985233+0000","last_peered":"2024-01-24T17:47:31.985233+0000","last_clean":"2024-01-24T17:47:31.985233+0000","last_became_active":"2024-01-24T17:47:31.977607+0000","last_became_peered":"2024-01-24T17:47:31.977607+0000","last_unstale":"2024-01-24T17:47:31.985233+0000","last_undegraded":"2024-01-24T17:47:31.985233+0000","last_fullsized":"2024-01-24T17:47:31.985233+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:45:51.512395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.955448+0000","last_change":"2024-01-24T17:47:31.471080+0000","last_active":"2024-01-24T17:47:31.955448+0000","last_peered":"2024-01-24T17:47:31.955448+0000","last_clean":"2024-01-24T17:47:31.955448+0000","last_became_active":"2024-01-24T17:47:31.470740+0000","last_became_peered":"2024-01-24T17:47:31.470740+0000","last_unstale":"2024-01-24T17:47:31.955448+0000","last_undegraded":"2024-01-24T17:47:31.955448+0000","last_fullsized":"2024-01-24T17:47:31.955448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:44:24.567578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.977579+0000","last_change":"2024-01-24T17:47:31.975277+0000","last_active":"2024-01-24T17:47:31.977579+0000","last_peered":"2024-01-24T17:47:31.977579+0000","last_clean":"2024-01-24T17:47:31.977579+0000","last_became_active":"2024-01-24T17:47:31.974407+0000","last_became_peered":"2024-01-24T17:47:31.974407+0000","last_unstale":"2024-01-24T17:47:31.977579+0000","last_undegraded":"2024-01-24T17:47:31.977579+0000","last_fullsized":"2024-01-24T17:47:31.977579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:25:35.181055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.958750+0000","last_change":"2024-01-24T17:47:31.480759+0000","last_active":"2024-01-24T17:47:31.958750+0000","last_peered":"2024-01-24T17:47:31.958750+0000","last_clean":"2024-01-24T17:47:31.958750+0000","last_became_active":"2024-01-24T17:47:31.480504+0000","last_became_peered":"2024-01-24T17:47:31.480504+0000","last_unstale":"2024-01-24T17:47:31.958750+0000","last_undegraded":"2024-01-24T17:47:31.958750+0000","last_fullsized":"2024-01-24T17:47:31.958750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:49.591723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.978534+0000","last_change":"2024-01-24T17:47:31.975701+0000","last_active":"2024-01-24T17:47:31.978534+0000","last_peered":"2024-01-24T17:47:31.978534+0000","last_clean":"2024-01-24T17:47:31.978534+0000","last_became_active":"2024-01-24T17:47:31.975449+0000","last_became_peered":"2024-01-24T17:47:31.975449+0000","last_unstale":"2024-01-24T17:47:31.978534+0000","last_undegraded":"2024-01-24T17:47:31.978534+0000","last_fullsized":"2024-01-24T17:47:31.978534+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:05:57.892601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.975101+0000","last_change":"2024-01-24T17:47:31.474134+0000","last_active":"2024-01-24T17:47:31.975101+0000","last_peered":"2024-01-24T17:47:31.975101+0000","last_clean":"2024-01-24T17:47:31.975101+0000","last_became_active":"2024-01-24T17:47:31.471905+0000","last_became_peered":"2024-01-24T17:47:31.471905+0000","last_unstale":"2024-01-24T17:47:31.975101+0000","last_undegraded":"2024-01-24T17:47:31.975101+0000","last_fullsized":"2024-01-24T17:47:31.975101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:53:42.524052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988147+0000","last_change":"2024-01-24T17:47:31.978890+0000","last_active":"2024-01-24T17:47:31.988147+0000","last_peered":"2024-01-24T17:47:31.988147+0000","last_clean":"2024-01-24T17:47:31.988147+0000","last_became_active":"2024-01-24T17:47:31.978399+0000","last_became_peered":"2024-01-24T17:47:31.978399+0000","last_unstale":"2024-01-24T17:47:31.988147+0000","last_undegraded":"2024-01-24T17:47:31.988147+0000","last_fullsized":"2024-01-24T17:47:31.988147+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:16:08.410074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.947492+0000","last_change":"2024-01-24T17:47:31.490514+0000","last_active":"2024-01-24T17:47:31.947492+0000","last_peered":"2024-01-24T17:47:31.947492+0000","last_clean":"2024-01-24T17:47:31.947492+0000","last_became_active":"2024-01-24T17:47:31.490208+0000","last_became_peered":"2024-01-24T17:47:31.490208+0000","last_unstale":"2024-01-24T17:47:31.947492+0000","last_undegraded":"2024-01-24T17:47:31.947492+0000","last_fullsized":"2024-01-24T17:47:31.947492+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:06:49.992424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987654+0000","last_change":"2024-01-24T17:47:31.980709+0000","last_active":"2024-01-24T17:47:31.987654+0000","last_peered":"2024-01-24T17:47:31.987654+0000","last_clean":"2024-01-24T17:47:31.987654+0000","last_became_active":"2024-01-24T17:47:31.980394+0000","last_became_peered":"2024-01-24T17:47:31.980394+0000","last_unstale":"2024-01-24T17:47:31.987654+0000","last_undegraded":"2024-01-24T17:47:31.987654+0000","last_fullsized":"2024-01-24T17:47:31.987654+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:50.606488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.15","version":"87'3","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988092+0000","last_change":"2024-01-24T17:47:31.469330+0000","last_active":"2024-01-24T17:47:31.988092+0000","last_peered":"2024-01-24T17:47:31.988092+0000","last_clean":"2024-01-24T17:47:31.988092+0000","last_became_active":"2024-01-24T17:47:31.468761+0000","last_became_peered":"2024-01-24T17:47:31.468761+0000","last_unstale":"2024-01-24T17:47:31.988092+0000","last_undegraded":"2024-01-24T17:47:31.988092+0000","last_fullsized":"2024-01-24T17:47:31.988092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:11:15.408613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986307+0000","last_change":"2024-01-24T17:47:31.977501+0000","last_active":"2024-01-24T17:47:31.986307+0000","last_peered":"2024-01-24T17:47:31.986307+0000","last_clean":"2024-01-24T17:47:31.986307+0000","last_became_active":"2024-01-24T17:47:31.977202+0000","last_became_peered":"2024-01-24T17:47:31.977202+0000","last_unstale":"2024-01-24T17:47:31.986307+0000","last_undegraded":"2024-01-24T17:47:31.986307+0000","last_fullsized":"2024-01-24T17:47:31.986307+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:47:58.923857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"87'6","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.990402+0000","last_change":"2024-01-24T17:47:31.470804+0000","last_active":"2024-01-24T17:47:31.990402+0000","last_peered":"2024-01-24T17:47:31.990402+0000","last_clean":"2024-01-24T17:47:31.990402+0000","last_became_active":"2024-01-24T17:47:31.470455+0000","last_became_peered":"2024-01-24T17:47:31.470455+0000","last_unstale":"2024-01-24T17:47:31.990402+0000","last_undegraded":"2024-01-24T17:47:31.990402+0000","last_fullsized":"2024-01-24T17:47:31.990402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:06:23.354739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.981438+0000","last_change":"2024-01-24T17:47:31.974806+0000","last_active":"2024-01-24T17:47:31.981438+0000","last_peered":"2024-01-24T17:47:31.981438+0000","last_clean":"2024-01-24T17:47:31.981438+0000","last_became_active":"2024-01-24T17:47:31.974394+0000","last_became_peered":"2024-01-24T17:47:31.974394+0000","last_unstale":"2024-01-24T17:47:31.981438+0000","last_undegraded":"2024-01-24T17:47:31.981438+0000","last_fullsized":"2024-01-24T17:47:31.981438+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:36.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.13","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.983348+0000","last_change":"2024-01-24T17:47:31.471680+0000","last_active":"2024-01-24T17:47:31.983348+0000","last_peered":"2024-01-24T17:47:31.983348+0000","last_clean":"2024-01-24T17:47:31.983348+0000","last_became_active":"2024-01-24T17:47:31.471405+0000","last_became_peered":"2024-01-24T17:47:31.471405+0000","last_unstale":"2024-01-24T17:47:31.983348+0000","last_undegraded":"2024-01-24T17:47:31.983348+0000","last_fullsized":"2024-01-24T17:47:31.983348+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:26:59.597277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987965+0000","last_change":"2024-01-24T17:47:31.978744+0000","last_active":"2024-01-24T17:47:31.987965+0000","last_peered":"2024-01-24T17:47:31.987965+0000","last_clean":"2024-01-24T17:47:31.987965+0000","last_became_active":"2024-01-24T17:47:31.978145+0000","last_became_peered":"2024-01-24T17:47:31.978145+0000","last_unstale":"2024-01-24T17:47:31.987965+0000","last_undegraded":"2024-01-24T17:47:31.987965+0000","last_fullsized":"2024-01-24T17:47:31.987965+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:08:04.652220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.975812+0000","last_change":"2024-01-24T17:47:31.490315+0000","last_active":"2024-01-24T17:47:31.975812+0000","last_peered":"2024-01-24T17:47:31.975812+0000","last_clean":"2024-01-24T17:47:31.975812+0000","last_became_active":"2024-01-24T17:47:31.490055+0000","last_became_peered":"2024-01-24T17:47:31.490055+0000","last_unstale":"2024-01-24T17:47:31.975812+0000","last_undegraded":"2024-01-24T17:47:31.975812+0000","last_fullsized":"2024-01-24T17:47:31.975812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:53.162650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.976931+0000","last_change":"2024-01-24T17:47:31.975363+0000","last_active":"2024-01-24T17:47:31.976931+0000","last_peered":"2024-01-24T17:47:31.976931+0000","last_clean":"2024-01-24T17:47:31.976931+0000","last_became_active":"2024-01-24T17:47:31.974933+0000","last_became_peered":"2024-01-24T17:47:31.974933+0000","last_unstale":"2024-01-24T17:47:31.976931+0000","last_undegraded":"2024-01-24T17:47:31.976931+0000","last_fullsized":"2024-01-24T17:47:31.976931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:33:35.823078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.947824+0000","last_change":"2024-01-24T17:47:31.485453+0000","last_active":"2024-01-24T17:47:31.947824+0000","last_peered":"2024-01-24T17:47:31.947824+0000","last_clean":"2024-01-24T17:47:31.947824+0000","last_became_active":"2024-01-24T17:47:31.484853+0000","last_became_peered":"2024-01-24T17:47:31.484853+0000","last_unstale":"2024-01-24T17:47:31.947824+0000","last_undegraded":"2024-01-24T17:47:31.947824+0000","last_fullsized":"2024-01-24T17:47:31.947824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:22:34.185940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989223+0000","last_change":"2024-01-24T17:47:31.986869+0000","last_active":"2024-01-24T17:47:31.989223+0000","last_peered":"2024-01-24T17:47:31.989223+0000","last_clean":"2024-01-24T17:47:31.989223+0000","last_became_active":"2024-01-24T17:47:31.986596+0000","last_became_peered":"2024-01-24T17:47:31.986596+0000","last_unstale":"2024-01-24T17:47:31.989223+0000","last_undegraded":"2024-01-24T17:47:31.989223+0000","last_fullsized":"2024-01-24T17:47:31.989223+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:06:24.369505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.10","version":"87'8","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988222+0000","last_change":"2024-01-24T17:47:31.471831+0000","last_active":"2024-01-24T17:47:31.988222+0000","last_peered":"2024-01-24T17:47:31.988222+0000","last_clean":"2024-01-24T17:47:31.988222+0000","last_became_active":"2024-01-24T17:47:31.469998+0000","last_became_peered":"2024-01-24T17:47:31.469998+0000","last_unstale":"2024-01-24T17:47:31.988222+0000","last_undegraded":"2024-01-24T17:47:31.988222+0000","last_fullsized":"2024-01-24T17:47:31.988222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:12:08.558957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982845+0000","last_change":"2024-01-24T17:47:31.977884+0000","last_active":"2024-01-24T17:47:31.982845+0000","last_peered":"2024-01-24T17:47:31.982845+0000","last_clean":"2024-01-24T17:47:31.982845+0000","last_became_active":"2024-01-24T17:47:31.977202+0000","last_became_peered":"2024-01-24T17:47:31.977202+0000","last_unstale":"2024-01-24T17:47:31.982845+0000","last_undegraded":"2024-01-24T17:47:31.982845+0000","last_fullsized":"2024-01-24T17:47:31.982845+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:37:29.833370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.974819+0000","last_change":"2024-01-24T17:47:31.475331+0000","last_active":"2024-01-24T17:47:31.974819+0000","last_peered":"2024-01-24T17:47:31.974819+0000","last_clean":"2024-01-24T17:47:31.974819+0000","last_became_active":"2024-01-24T17:47:31.475070+0000","last_became_peered":"2024-01-24T17:47:31.475070+0000","last_unstale":"2024-01-24T17:47:31.974819+0000","last_undegraded":"2024-01-24T17:47:31.974819+0000","last_fullsized":"2024-01-24T17:47:31.974819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:09:11.261877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982155+0000","last_change":"2024-01-24T17:47:31.974592+0000","last_active":"2024-01-24T17:47:31.982155+0000","last_peered":"2024-01-24T17:47:31.982155+0000","last_clean":"2024-01-24T17:47:31.982155+0000","last_became_active":"2024-01-24T17:47:31.974297+0000","last_became_peered":"2024-01-24T17:47:31.974297+0000","last_unstale":"2024-01-24T17:47:31.982155+0000","last_undegraded":"2024-01-24T17:47:31.982155+0000","last_fullsized":"2024-01-24T17:47:31.982155+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:06:41.488722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.e","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982728+0000","last_change":"2024-01-24T17:47:31.477111+0000","last_active":"2024-01-24T17:47:31.982728+0000","last_peered":"2024-01-24T17:47:31.982728+0000","last_clean":"2024-01-24T17:47:31.982728+0000","last_became_active":"2024-01-24T17:47:31.476458+0000","last_became_peered":"2024-01-24T17:47:31.476458+0000","last_unstale":"2024-01-24T17:47:31.982728+0000","last_undegraded":"2024-01-24T17:47:31.982728+0000","last_fullsized":"2024-01-24T17:47:31.982728+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:28.298210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.990383+0000","last_change":"2024-01-24T17:47:31.987367+0000","last_active":"2024-01-24T17:47:31.990383+0000","last_peered":"2024-01-24T17:47:31.990383+0000","last_clean":"2024-01-24T17:47:31.990383+0000","last_became_active":"2024-01-24T17:47:31.985568+0000","last_became_peered":"2024-01-24T17:47:31.985568+0000","last_unstale":"2024-01-24T17:47:31.990383+0000","last_undegraded":"2024-01-24T17:47:31.990383+0000","last_fullsized":"2024-01-24T17:47:31.990383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:36:49.719214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.954294+0000","last_change":"2024-01-24T17:47:31.471090+0000","last_active":"2024-01-24T17:47:31.954294+0000","last_peered":"2024-01-24T17:47:31.954294+0000","last_clean":"2024-01-24T17:47:31.954294+0000","last_became_active":"2024-01-24T17:47:31.470835+0000","last_became_peered":"2024-01-24T17:47:31.470835+0000","last_unstale":"2024-01-24T17:47:31.954294+0000","last_undegraded":"2024-01-24T17:47:31.954294+0000","last_fullsized":"2024-01-24T17:47:31.954294+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:06:23.354739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.990290+0000","last_change":"2024-01-24T17:47:31.988558+0000","last_active":"2024-01-24T17:47:31.990290+0000","last_peered":"2024-01-24T17:47:31.990290+0000","last_clean":"2024-01-24T17:47:31.990290+0000","last_became_active":"2024-01-24T17:47:31.988277+0000","last_became_peered":"2024-01-24T17:47:31.988277+0000","last_unstale":"2024-01-24T17:47:31.990290+0000","last_undegraded":"2024-01-24T17:47:31.990290+0000","last_fullsized":"2024-01-24T17:47:31.990290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:01.119477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.983760+0000","last_change":"2024-01-24T17:47:31.474153+0000","last_active":"2024-01-24T17:47:31.983760+0000","last_peered":"2024-01-24T17:47:31.983760+0000","last_clean":"2024-01-24T17:47:31.983760+0000","last_became_active":"2024-01-24T17:47:31.473899+0000","last_became_peered":"2024-01-24T17:47:31.473899+0000","last_unstale":"2024-01-24T17:47:31.983760+0000","last_undegraded":"2024-01-24T17:47:31.983760+0000","last_fullsized":"2024-01-24T17:47:31.983760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:08:03.637455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986154+0000","last_change":"2024-01-24T17:47:31.976045+0000","last_active":"2024-01-24T17:47:31.986154+0000","last_peered":"2024-01-24T17:47:31.986154+0000","last_clean":"2024-01-24T17:47:31.986154+0000","last_became_active":"2024-01-24T17:47:31.975787+0000","last_became_peered":"2024-01-24T17:47:31.975787+0000","last_unstale":"2024-01-24T17:47:31.986154+0000","last_undegraded":"2024-01-24T17:47:31.986154+0000","last_fullsized":"2024-01-24T17:47:31.986154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:11:48.804217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.975910+0000","last_change":"2024-01-24T17:47:31.475124+0000","last_active":"2024-01-24T17:47:31.975910+0000","last_peered":"2024-01-24T17:47:31.975910+0000","last_clean":"2024-01-24T17:47:31.975910+0000","last_became_active":"2024-01-24T17:47:31.474152+0000","last_became_peered":"2024-01-24T17:47:31.474152+0000","last_unstale":"2024-01-24T17:47:31.975910+0000","last_undegraded":"2024-01-24T17:47:31.975910+0000","last_fullsized":"2024-01-24T17:47:31.975910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:57:19.521381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984085+0000","last_change":"2024-01-24T17:47:31.978320+0000","last_active":"2024-01-24T17:47:31.984085+0000","last_peered":"2024-01-24T17:47:31.984085+0000","last_clean":"2024-01-24T17:47:31.984085+0000","last_became_active":"2024-01-24T17:47:31.977585+0000","last_became_peered":"2024-01-24T17:47:31.977585+0000","last_unstale":"2024-01-24T17:47:31.984085+0000","last_undegraded":"2024-01-24T17:47:31.984085+0000","last_fullsized":"2024-01-24T17:47:31.984085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:05:01.572141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.956610+0000","last_change":"2024-01-24T17:47:31.485244+0000","last_active":"2024-01-24T17:47:31.956610+0000","last_peered":"2024-01-24T17:47:31.956610+0000","last_clean":"2024-01-24T17:47:31.956610+0000","last_became_active":"2024-01-24T17:47:31.484975+0000","last_became_peered":"2024-01-24T17:47:31.484975+0000","last_unstale":"2024-01-24T17:47:31.956610+0000","last_undegraded":"2024-01-24T17:47:31.956610+0000","last_fullsized":"2024-01-24T17:47:31.956610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:44:58.760429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989632+0000","last_change":"2024-01-24T17:47:31.982104+0000","last_active":"2024-01-24T17:47:31.989632+0000","last_peered":"2024-01-24T17:47:31.989632+0000","last_clean":"2024-01-24T17:47:31.989632+0000","last_became_active":"2024-01-24T17:47:31.981840+0000","last_became_peered":"2024-01-24T17:47:31.981840+0000","last_unstale":"2024-01-24T17:47:31.989632+0000","last_undegraded":"2024-01-24T17:47:31.989632+0000","last_fullsized":"2024-01-24T17:47:31.989632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:27:00.612042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.955603+0000","last_change":"2024-01-24T17:47:31.462477+0000","last_active":"2024-01-24T17:47:31.955603+0000","last_peered":"2024-01-24T17:47:31.955603+0000","last_clean":"2024-01-24T17:47:31.955603+0000","last_became_active":"2024-01-24T17:47:31.461816+0000","last_became_peered":"2024-01-24T17:47:31.461816+0000","last_unstale":"2024-01-24T17:47:31.955603+0000","last_undegraded":"2024-01-24T17:47:31.955603+0000","last_fullsized":"2024-01-24T17:47:31.955603+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:00.104711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987672+0000","last_change":"2024-01-24T17:47:31.984830+0000","last_active":"2024-01-24T17:47:31.987672+0000","last_peered":"2024-01-24T17:47:31.987672+0000","last_clean":"2024-01-24T17:47:31.987672+0000","last_became_active":"2024-01-24T17:47:31.983625+0000","last_became_peered":"2024-01-24T17:47:31.983625+0000","last_unstale":"2024-01-24T17:47:31.987672+0000","last_undegraded":"2024-01-24T17:47:31.987672+0000","last_fullsized":"2024-01-24T17:47:31.987672+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:45:23.940696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.971903+0000","last_change":"2024-01-24T17:47:31.480515+0000","last_active":"2024-01-24T17:47:31.971903+0000","last_peered":"2024-01-24T17:47:31.971903+0000","last_clean":"2024-01-24T17:47:31.971903+0000","last_became_active":"2024-01-24T17:47:31.480244+0000","last_became_peered":"2024-01-24T17:47:31.480244+0000","last_unstale":"2024-01-24T17:47:31.971903+0000","last_undegraded":"2024-01-24T17:47:31.971903+0000","last_fullsized":"2024-01-24T17:47:31.971903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:35.480526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986338+0000","last_change":"2024-01-24T17:47:31.973621+0000","last_active":"2024-01-24T17:47:31.986338+0000","last_peered":"2024-01-24T17:47:31.986338+0000","last_clean":"2024-01-24T17:47:31.986338+0000","last_became_active":"2024-01-24T17:47:31.973367+0000","last_became_peered":"2024-01-24T17:47:31.973367+0000","last_unstale":"2024-01-24T17:47:31.986338+0000","last_undegraded":"2024-01-24T17:47:31.986338+0000","last_fullsized":"2024-01-24T17:47:31.986338+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:48:09.166883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.962666+0000","last_change":"2024-01-24T17:47:31.472561+0000","last_active":"2024-01-24T17:47:31.962666+0000","last_peered":"2024-01-24T17:47:31.962666+0000","last_clean":"2024-01-24T17:47:31.962666+0000","last_became_active":"2024-01-24T17:47:31.472096+0000","last_became_peered":"2024-01-24T17:47:31.472096+0000","last_unstale":"2024-01-24T17:47:31.962666+0000","last_undegraded":"2024-01-24T17:47:31.962666+0000","last_fullsized":"2024-01-24T17:47:31.962666+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:40:28.939958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987656+0000","last_change":"2024-01-24T17:47:31.982250+0000","last_active":"2024-01-24T17:47:31.987656+0000","last_peered":"2024-01-24T17:47:31.987656+0000","last_clean":"2024-01-24T17:47:31.987656+0000","last_became_active":"2024-01-24T17:47:31.980597+0000","last_became_peered":"2024-01-24T17:47:31.980597+0000","last_unstale":"2024-01-24T17:47:31.987656+0000","last_undegraded":"2024-01-24T17:47:31.987656+0000","last_fullsized":"2024-01-24T17:47:31.987656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:16:26.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.954368+0000","last_change":"2024-01-24T17:47:31.476349+0000","last_active":"2024-01-24T17:47:31.954368+0000","last_peered":"2024-01-24T17:47:31.954368+0000","last_clean":"2024-01-24T17:47:31.954368+0000","last_became_active":"2024-01-24T17:47:31.476061+0000","last_became_peered":"2024-01-24T17:47:31.476061+0000","last_unstale":"2024-01-24T17:47:31.954368+0000","last_undegraded":"2024-01-24T17:47:31.954368+0000","last_fullsized":"2024-01-24T17:47:31.954368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:35.480526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.968387+0000","last_change":"2024-01-24T17:47:31.964868+0000","last_active":"2024-01-24T17:47:31.968387+0000","last_peered":"2024-01-24T17:47:31.968387+0000","last_clean":"2024-01-24T17:47:31.968387+0000","last_became_active":"2024-01-24T17:47:31.964563+0000","last_became_peered":"2024-01-24T17:47:31.964563+0000","last_unstale":"2024-01-24T17:47:31.968387+0000","last_undegraded":"2024-01-24T17:47:31.968387+0000","last_fullsized":"2024-01-24T17:47:31.968387+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:18:28.650673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.947117+0000","last_change":"2024-01-24T17:47:31.485474+0000","last_active":"2024-01-24T17:47:31.947117+0000","last_peered":"2024-01-24T17:47:31.947117+0000","last_clean":"2024-01-24T17:47:31.947117+0000","last_became_active":"2024-01-24T17:47:31.485117+0000","last_became_peered":"2024-01-24T17:47:31.485117+0000","last_unstale":"2024-01-24T17:47:31.947117+0000","last_undegraded":"2024-01-24T17:47:31.947117+0000","last_fullsized":"2024-01-24T17:47:31.947117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:41.431895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.980080+0000","last_change":"2024-01-24T17:47:31.972677+0000","last_active":"2024-01-24T17:47:31.980080+0000","last_peered":"2024-01-24T17:47:31.980080+0000","last_clean":"2024-01-24T17:47:31.980080+0000","last_became_active":"2024-01-24T17:47:31.972325+0000","last_became_peered":"2024-01-24T17:47:31.972325+0000","last_unstale":"2024-01-24T17:47:31.980080+0000","last_undegraded":"2024-01-24T17:47:31.980080+0000","last_fullsized":"2024-01-24T17:47:31.980080+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:06:24.369505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.975911+0000","last_change":"2024-01-24T17:47:31.484099+0000","last_active":"2024-01-24T17:47:31.975911+0000","last_peered":"2024-01-24T17:47:31.975911+0000","last_clean":"2024-01-24T17:47:31.975911+0000","last_became_active":"2024-01-24T17:47:31.483661+0000","last_became_peered":"2024-01-24T17:47:31.483661+0000","last_unstale":"2024-01-24T17:47:31.975911+0000","last_undegraded":"2024-01-24T17:47:31.975911+0000","last_fullsized":"2024-01-24T17:47:31.975911+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:30.230951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.976971+0000","last_change":"2024-01-24T17:47:31.971659+0000","last_active":"2024-01-24T17:47:31.976971+0000","last_peered":"2024-01-24T17:47:31.976971+0000","last_clean":"2024-01-24T17:47:31.976971+0000","last_became_active":"2024-01-24T17:47:31.971348+0000","last_became_peered":"2024-01-24T17:47:31.971348+0000","last_unstale":"2024-01-24T17:47:31.976971+0000","last_undegraded":"2024-01-24T17:47:31.976971+0000","last_fullsized":"2024-01-24T17:47:31.976971+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:26:19.054930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.947277+0000","last_change":"2024-01-24T17:47:31.480055+0000","last_active":"2024-01-24T17:47:31.947277+0000","last_peered":"2024-01-24T17:47:31.947277+0000","last_clean":"2024-01-24T17:47:31.947277+0000","last_became_active":"2024-01-24T17:47:31.479420+0000","last_became_peered":"2024-01-24T17:47:31.479420+0000","last_unstale":"2024-01-24T17:47:31.947277+0000","last_undegraded":"2024-01-24T17:47:31.947277+0000","last_fullsized":"2024-01-24T17:47:31.947277+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:16:04.683723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.980916+0000","last_change":"2024-01-24T17:47:31.973486+0000","last_active":"2024-01-24T17:47:31.980916+0000","last_peered":"2024-01-24T17:47:31.980916+0000","last_clean":"2024-01-24T17:47:31.980916+0000","last_became_active":"2024-01-24T17:47:31.973216+0000","last_became_peered":"2024-01-24T17:47:31.973216+0000","last_unstale":"2024-01-24T17:47:31.980916+0000","last_undegraded":"2024-01-24T17:47:31.980916+0000","last_fullsized":"2024-01-24T17:47:31.980916+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:22:35.200706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.962877+0000","last_change":"2024-01-24T17:47:31.476802+0000","last_active":"2024-01-24T17:47:31.962877+0000","last_peered":"2024-01-24T17:47:31.962877+0000","last_clean":"2024-01-24T17:47:31.962877+0000","last_became_active":"2024-01-24T17:47:31.475610+0000","last_became_peered":"2024-01-24T17:47:31.475610+0000","last_unstale":"2024-01-24T17:47:31.962877+0000","last_undegraded":"2024-01-24T17:47:31.962877+0000","last_fullsized":"2024-01-24T17:47:31.962877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:35:53.319043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989926+0000","last_change":"2024-01-24T17:47:31.984081+0000","last_active":"2024-01-24T17:47:31.989926+0000","last_peered":"2024-01-24T17:47:31.989926+0000","last_clean":"2024-01-24T17:47:31.989926+0000","last_became_active":"2024-01-24T17:47:31.983740+0000","last_became_peered":"2024-01-24T17:47:31.983740+0000","last_unstale":"2024-01-24T17:47:31.989926+0000","last_undegraded":"2024-01-24T17:47:31.989926+0000","last_fullsized":"2024-01-24T17:47:31.989926+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:01.119477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.954107+0000","last_change":"2024-01-24T17:47:31.472320+0000","last_active":"2024-01-24T17:47:31.954107+0000","last_peered":"2024-01-24T17:47:31.954107+0000","last_clean":"2024-01-24T17:47:31.954107+0000","last_became_active":"2024-01-24T17:47:31.472035+0000","last_became_peered":"2024-01-24T17:47:31.472035+0000","last_unstale":"2024-01-24T17:47:31.954107+0000","last_undegraded":"2024-01-24T17:47:31.954107+0000","last_fullsized":"2024-01-24T17:47:31.954107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:49.591723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.972806+0000","last_change":"2024-01-24T17:47:31.968866+0000","last_active":"2024-01-24T17:47:31.972806+0000","last_peered":"2024-01-24T17:47:31.972806+0000","last_clean":"2024-01-24T17:47:31.972806+0000","last_became_active":"2024-01-24T17:47:31.968532+0000","last_became_peered":"2024-01-24T17:47:31.968532+0000","last_unstale":"2024-01-24T17:47:31.972806+0000","last_undegraded":"2024-01-24T17:47:31.972806+0000","last_fullsized":"2024-01-24T17:47:31.972806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:29.312975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.981269+0000","last_change":"2024-01-24T17:47:31.470768+0000","last_active":"2024-01-24T17:47:31.981269+0000","last_peered":"2024-01-24T17:47:31.981269+0000","last_clean":"2024-01-24T17:47:31.981269+0000","last_became_active":"2024-01-24T17:47:31.470489+0000","last_became_peered":"2024-01-24T17:47:31.470489+0000","last_unstale":"2024-01-24T17:47:31.981269+0000","last_undegraded":"2024-01-24T17:47:31.981269+0000","last_fullsized":"2024-01-24T17:47:31.981269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:25:54.025878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":225,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.466273+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:47:31.466273+0000","last_peered":"2024-01-24T17:47:31.466273+0000","last_clean":"2024-01-24T17:47:31.466273+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:47:31.466273+0000","last_undegraded":"2024-01-24T17:47:31.466273+0000","last_fullsized":"2024-01-24T17:47:31.466273+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1073152,"data_stored":457790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.075,"15min":1.0329999999999999},"min":{"1min":0.66500000000000004,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5269999999999999,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1020000000000001,"15min":1.0509999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.413,"5min":1.917,"15min":1.917},"last":1.718}]},{"osd":1,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.095,"15min":1.103},"min":{"1min":0.81200000000000006,"5min":0.621,"15min":0.621},"max":{"1min":1.4930000000000001,"5min":2.04,"15min":2.04},"last":1.292},{"interface":"front","average":{"1min":1.141,"5min":1.1870000000000001,"15min":1.1930000000000001},"min":{"1min":0.81200000000000006,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.454,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.944}]},{"osd":2,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1930000000000001,"15min":1.165},"min":{"1min":0.71099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6359999999999999,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1679999999999999,"15min":1.214},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7430000000000001,"5min":2.484,"15min":2.484},"last":1.4319999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7230000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.88400000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.9059999999999999,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.5349999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.274,"15min":1.3129999999999999},"min":{"1min":0.86499999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.681,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":2.1499999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.4139999999999999,"15min":1.431},"min":{"1min":0.73399999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.6859999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.466}]},{"osd":5,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2869999999999999,"15min":1.341},"min":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9810000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.619},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.359,"15min":1.4219999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.0030000000000001,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":2.2480000000000002}]},{"osd":6,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4590000000000001,"15min":1.522},"min":{"1min":0.755,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.853,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":2.4129999999999998},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.431,"15min":1.474},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.8100000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.923}]}]},{"osd":6,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":1056768,"data_stored":450945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.081,"15min":1.04},"min":{"1min":0.748,"5min":0.623,"15min":0.61199999999999999},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.0920000000000001,"15min":1.0349999999999999},"min":{"1min":0.79100000000000004,"5min":0.69499999999999995,"15min":0.66300000000000003},"max":{"1min":1.4390000000000001,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.98799999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.113,"15min":1.022},"min":{"1min":0.78400000000000003,"5min":0.66700000000000004,"15min":0.64200000000000002},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.173,"15min":1.131},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.9129999999999998,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1759999999999999,"15min":1.1479999999999999},"min":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.63600000000000001},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1919999999999999,"15min":1.1739999999999999},"min":{"1min":0.755,"5min":0.69099999999999995,"15min":0.56599999999999995},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.0910000000000002},"last":1.093}]},{"osd":3,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.236,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.75,"5min":2.0009999999999999,"15min":2.1669999999999998},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2090000000000001,"15min":1.2450000000000001},"min":{"1min":0.95299999999999996,"5min":0.81899999999999995,"15min":0.69799999999999995},"max":{"1min":1.625,"5min":1.758,"15min":2.46},"last":1.625}]},{"osd":4,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3089999999999999,"15min":1.2549999999999999},"min":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.877,"5min":1.9179999999999999,"15min":1.988},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.753,"5min":0.73899999999999999,"15min":0.44600000000000001},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.3049999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.339,"15min":1.3069999999999999},"min":{"1min":0.77600000000000002,"5min":0.65300000000000002,"15min":0.46800000000000003},"max":{"1min":1.694,"5min":1.8879999999999999,"15min":2.2170000000000001},"last":1.284},{"interface":"front","average":{"1min":1.49,"5min":1.4419999999999999,"15min":1.4399999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.9750000000000001,"5min":2.0030000000000001,"15min":2.4849999999999999},"last":1.712}]},{"osd":7,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3480000000000001,"15min":1.365},"min":{"1min":0.91200000000000003,"5min":0.85399999999999998,"15min":0.51400000000000001},"max":{"1min":1.911,"5min":1.978,"15min":2.2050000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4470000000000001,"15min":1.4610000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.52300000000000002},"max":{"1min":1.7649999999999999,"5min":1.9019999999999999,"15min":2.6920000000000002},"last":1.3620000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1679360,"data_stored":1049686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0349999999999999,"15min":1.089},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5009999999999999,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.0369999999999999,"15min":1.0329999999999999},"min":{"1min":0.67300000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.492,"5min":1.889,"15min":1.889},"last":1.234}]},{"osd":1,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0700000000000001,"15min":1.0720000000000001},"min":{"1min":0.77200000000000002,"5min":0.622,"15min":0.622},"max":{"1min":1.504,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.171,"15min":1.2470000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.597,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.357}]},{"osd":2,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2290000000000001,"15min":1.2310000000000001},"min":{"1min":0.875,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6319999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2649999999999999,"15min":1.3069999999999999},"min":{"1min":0.92300000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.4690000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.94499999999999995}]},{"osd":3,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.23,"15min":1.3200000000000001},"min":{"1min":0.76400000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.5629999999999999,"5min":3.036,"15min":3.036},"last":1.448},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.2789999999999999,"15min":1.323},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.718,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.26}]},{"osd":4,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.409,"15min":1.4450000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.9690000000000001,"5min":2.069,"15min":2.069},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.278,"15min":1.343},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0409999999999999,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.1220000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3149999999999999,"15min":1.3480000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.8759999999999999,"5min":2.504,"15min":2.504},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.4690000000000001,"15min":1.5109999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.1059999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.548}]},{"osd":7,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.401,"15min":1.474},"min":{"1min":0.80600000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.1309999999999998,"5min":3.161,"15min":3.161},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.5309999999999999,"15min":1.615},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.0819999999999999,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.5980000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":1056768,"data_stored":450945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.075,"15min":1.0469999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.5840000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0880000000000001,"15min":1.0940000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.4690000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.3149999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.1140000000000001,"15min":1.0629999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":2.113},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1459999999999999,"15min":1.202},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.3360000000000001,"5min":1.849,"15min":1.849},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1970000000000001,"15min":1.2090000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.718,"5min":2.105,"15min":2.105},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.266,"15min":1.3220000000000001},"min":{"1min":0.752,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.786,"5min":1.986,"15min":1.986},"last":1.149}]},{"osd":3,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1879999999999999,"15min":1.1919999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.516,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.276,"15min":1.329},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5820000000000001,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.7929999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.3,"15min":1.357},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6459999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2669999999999999,"15min":1.2949999999999999},"min":{"1min":0.65600000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.667,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.738}]},{"osd":6,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.359,"15min":1.4410000000000001},"min":{"1min":0.67800000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.917,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":2.1930000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4179999999999999,"15min":1.464},"min":{"1min":0.69499999999999995,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.74,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.518}]},{"osd":7,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.4099999999999999,"15min":1.4179999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.881,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4990000000000001,"15min":1.587},"min":{"1min":0.79300000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.8799999999999999,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":2.2719999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1081344,"data_stored":458394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.89000000000000001,"15min":0.86399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.5580000000000001,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96599999999999997,"15min":0.89500000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.6739999999999999,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.88600000000000001,"15min":0.93300000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.05,"15min":1.0589999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.657,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.087,"15min":1.105},"min":{"1min":0.56399999999999995,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.4790000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.103,"5min":1.1619999999999999,"15min":1.1579999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.7150000000000001,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.81699999999999995}]},{"osd":4,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.284,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.8660000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.274,"5min":1.3089999999999999,"15min":1.3400000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.0390000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.325}]},{"osd":5,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.387,"15min":1.421},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.1269999999999998,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.379,"15min":1.411},"min":{"1min":0.78400000000000003,"5min":0.746,"15min":0.746},"max":{"1min":1.9430000000000001,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":0.95199999999999996}]},{"osd":6,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5029999999999999,"15min":1.4930000000000001},"min":{"1min":0.86899999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0110000000000001,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.143},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.427,"15min":1.448},"min":{"1min":0.80200000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.9019999999999999,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.028}]},{"osd":7,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.49,"15min":1.5},"min":{"1min":0.93300000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.833,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.5149999999999999,"15min":1.516},"min":{"1min":1.075,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.0990000000000002,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.1020000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":1044480,"data_stored":444670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78900000000000003,"15min":0.79300000000000004},"min":{"1min":0.443,"5min":0.318,"15min":0.318},"max":{"1min":1.3420000000000001,"5min":1.667,"15min":1.667},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.66200000000000003,"5min":0.375,"15min":0.375},"max":{"1min":1.4370000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0109999999999999,"15min":1.0469999999999999},"min":{"1min":0.505,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.569,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.319},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.111,"15min":1.1950000000000001},"min":{"1min":0.76300000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.677,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.93300000000000005}]},{"osd":3,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94299999999999995,"15min":0.98299999999999998},"min":{"1min":0.51900000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.5089999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.135,"15min":1.1950000000000001},"min":{"1min":0.68000000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5680000000000001,"5min":2.996,"15min":2.996},"last":0.77100000000000002}]},{"osd":4,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.248,"15min":1.3100000000000001},"min":{"1min":0.78500000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.5,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.25},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.2589999999999999,"15min":1.3779999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.831,"5min":1.889,"15min":2.3839999999999999},"last":1.1819999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3009999999999999,"15min":1.389},"min":{"1min":0.873,"5min":0.73999999999999999,"15min":0.73899999999999999},"max":{"1min":1.851,"5min":1.851,"15min":2.3140000000000001},"last":0.998},{"interface":"front","average":{"1min":1.119,"5min":1.202,"15min":1.3},"min":{"1min":0.754,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.4750000000000001,"5min":1.889,"15min":2.0720000000000001},"last":0.97599999999999998}]},{"osd":6,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.304,"15min":1.3799999999999999},"min":{"1min":0.63200000000000001,"5min":0.623,"15min":0.623},"max":{"1min":1.615,"5min":1.9350000000000001,"15min":2.2679999999999998},"last":1.194},{"interface":"front","average":{"1min":1.298,"5min":1.3380000000000001,"15min":1.4490000000000001},"min":{"1min":0.95499999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.782,"5min":2.2530000000000001,"15min":3.7410000000000001},"last":1.046}]},{"osd":7,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.401,"15min":1.5449999999999999},"min":{"1min":0.81999999999999995,"5min":0.80800000000000005,"15min":0.80300000000000005},"max":{"1min":1.9670000000000001,"5min":2.1499999999999999,"15min":3.343},"last":1.117},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.367,"15min":1.48},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.7190000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.3460000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1650688,"data_stored":1041313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.876,"15min":0.86099999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1220000000000001,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"min":{"1min":0.442,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.3049999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.62}]},{"osd":2,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.89500000000000002,"15min":0.96499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.105,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":1.052,"15min":1.1000000000000001},"min":{"1min":0.46899999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5229999999999999,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.129,"15min":1.159},"min":{"1min":0.53000000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.597,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.0980000000000001,"15min":1.212},"min":{"1min":0.495,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.6220000000000001,"5min":3.149,"15min":3.149},"last":0.68200000000000005}]},{"osd":4,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.331,"15min":1.3580000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.696,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.28,"15min":1.3180000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5880000000000001,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.871}]},{"osd":5,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.2689999999999999,"15min":1.339},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5449999999999999,"5min":2.577,"15min":2.577},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3480000000000001,"15min":1.3720000000000001},"min":{"1min":0.89800000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0950000000000002,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":0.92400000000000004}]},{"osd":6,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.345,"15min":1.4259999999999999},"min":{"1min":0.78700000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.9219999999999999,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.3700000000000001,"15min":1.5009999999999999},"min":{"1min":0.72599999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.6719999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.143}]},{"osd":7,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.413,"15min":1.46},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0840000000000001,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.4430000000000001,"15min":1.5349999999999999},"min":{"1min":0.92500000000000004,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.6950000000000001,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":0.89700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":1044480,"data_stored":444670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.79100000000000004,"15min":0.81499999999999995},"min":{"1min":0.441,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.3420000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.93100000000000005,"15min":0.96999999999999997},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.57699999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.83699999999999997},"min":{"1min":0.46500000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2,"5min":1.331,"15min":1.331},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.99099999999999999,"15min":1.0820000000000001},"min":{"1min":0.50800000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.55,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.86299999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1100000000000001,"15min":1.1799999999999999},"min":{"1min":0.73999999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.5149999999999999,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.157,"15min":1.1850000000000001},"min":{"1min":0.67900000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.427,"5min":2.407,"15min":2.407},"last":0.95499999999999996}]},{"osd":4,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.2649999999999999,"15min":1.389},"min":{"1min":0.93400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.387,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3169999999999999,"15min":1.4550000000000001},"min":{"1min":0.91500000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.615,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.01}]},{"osd":5,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.29,"15min":1.3819999999999999},"min":{"1min":0.90800000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.6259999999999999,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2390000000000001,"15min":1.2589999999999999},"min":{"1min":0.85499999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.7310000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4139999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3300000000000001,"15min":1.3580000000000001},"min":{"1min":0.93700000000000006,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.95,"5min":2.335,"15min":2.335},"last":1.244},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.381,"15min":1.4490000000000001},"min":{"1min":1.0900000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.931,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3320000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.397,"15min":1.512},"min":{"1min":1.073,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.841,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.202},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.46,"15min":1.579},"min":{"1min":1.002,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.907,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.292}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:47:36.753 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:47:37.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.013+0000 7ff0cbe00640 1 Processor -- start 2024-01-24T17:47:37.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.013+0000 7ff0cbe00640 1 -- start start 2024-01-24T17:47:37.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.015+0000 7ff0cbe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc070540 0x7ff0cc15ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.015+0000 7ff0cbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff0cc15e910 0x7ff0cc15eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.015+0000 7ff0cbe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0cc1551b0 con 0x7ff0cc0b3e00 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.015+0000 7ff0cbe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0cc15e410 con 0x7ff0cc15e910 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.015+0000 7ff0cbe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc15b1b0 con 0x7ff0cc070540 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.015+0000 7ff0cb400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff0cc15e910 0x7ff0cc15eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.015+0000 7ff0cb400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff0cc15e910 0x7ff0cc15eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59618/0 (socket says 172.21.15.39:59618) 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.015+0000 7ff0cb400640 1 -- 172.21.15.39:0/2626637593 learned_addr learned my addr 172.21.15.39:0/2626637593 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.016+0000 7ff0caa00640 1 --2- 172.21.15.39:0/2626637593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc070540 0x7ff0cc15ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.016+0000 7ff0cb400640 1 -- 172.21.15.39:0/2626637593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc070540 msgr2=0x7ff0cc15ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.016+0000 7ff0cb400640 1 --2- 172.21.15.39:0/2626637593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc070540 0x7ff0cc15ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.016+0000 7ff0cb400640 1 -- 172.21.15.39:0/2626637593 >> v1:172.21.15.100:6789/0 conn(0x7ff0cc0b3e00 legacy=0x7ff0cc0c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:37.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.016+0000 7ff0cb400640 1 -- 172.21.15.39:0/2626637593 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0cc15b480 con 0x7ff0cc15e910 2024-01-24T17:47:37.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.017+0000 7ff0cb400640 1 --2- 172.21.15.39:0/2626637593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc15e910 0x7ff0cc15eda0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009e50 tx=0x7ff0c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824b8e4bd3aa1584 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:37.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.017+0000 7ff0ca000640 1 -- 172.21.15.39:0/2626637593 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c402cb00 con 0x7ff0cc15e910 2024-01-24T17:47:37.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.017+0000 7ff0ca000640 1 -- 172.21.15.39:0/2626637593 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c402cc60 con 0x7ff0cc15e910 2024-01-24T17:47:37.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.017+0000 7ff0ca000640 1 -- 172.21.15.39:0/2626637593 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40343f0 con 0x7ff0cc15e910 2024-01-24T17:47:37.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.018+0000 7ff0cbe00640 1 -- 172.21.15.39:0/2626637593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc15e910 msgr2=0x7ff0cc15eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.018+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/2626637593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc15e910 0x7ff0cc15eda0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009e50 tx=0x7ff0c402bf60 comp rx=0 tx=0).stop 2024-01-24T17:47:37.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.018+0000 7ff0cbe00640 1 -- 172.21.15.39:0/2626637593 shutdown_connections 2024-01-24T17:47:37.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.018+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/2626637593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc15e910 0x7ff0cc15eda0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.018+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/2626637593 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc070540 0x7ff0cc15ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.018+0000 7ff0cbe00640 1 -- 172.21.15.39:0/2626637593 >> 172.21.15.39:0/2626637593 conn(0x7ff0cc06a340 msgr2=0x7ff0cc0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:37.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.018+0000 7ff0cbe00640 1 -- 172.21.15.39:0/2626637593 shutdown_connections 2024-01-24T17:47:37.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.018+0000 7ff0cbe00640 1 -- 172.21.15.39:0/2626637593 wait complete. 2024-01-24T17:47:37.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.020+0000 7ff0cbe00640 1 Processor -- start 2024-01-24T17:47:37.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.021+0000 7ff0cbe00640 1 -- start start 2024-01-24T17:47:37.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc070540 0x7ff0cc0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0cbe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc0b3e00 0x7ff0cc0c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0cbe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0cc15e910 0x7ff0cc1534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0cbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc1e5420 con 0x7ff0cc070540 2024-01-24T17:47:37.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0cbe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc1e4e60 con 0x7ff0cc15e910 2024-01-24T17:47:37.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0cbe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc1e5140 con 0x7ff0cc0b3e00 2024-01-24T17:47:37.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc070540 0x7ff0cc0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc070540 0x7ff0cc0c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59632/0 (socket says 172.21.15.39:59632) 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.022+0000 7ff0caa00640 1 -- 172.21.15.39:0/3085274743 learned_addr learned my addr 172.21.15.39:0/3085274743 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0c3e00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc0b3e00 0x7ff0cc0c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0caa00640 1 -- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc0b3e00 msgr2=0x7ff0cc0c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0cb400640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0cc15e910 0x7ff0cc1534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0caa00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc0b3e00 0x7ff0cc0c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0caa00640 1 -- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0cc15e910 msgr2=0x7ff0cc1534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0caa00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0cc15e910 0x7ff0cc1534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0caa00640 1 -- 172.21.15.39:0/3085274743 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4009b30 con 0x7ff0cc070540 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0c3e00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc0b3e00 0x7ff0cc0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0caa00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc070540 0x7ff0cc0c7380 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00d800 tx=0x7ff0bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0c3400640 1 -- 172.21.15.39:0/3085274743 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc00b6c0 con 0x7ff0cc070540 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0c3400640 1 -- 172.21.15.39:0/3085274743 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc00b820 con 0x7ff0cc070540 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0c3400640 1 -- 172.21.15.39:0/3085274743 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc00bbd0 con 0x7ff0cc070540 2024-01-24T17:47:37.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.023+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0cc0c3090 con 0x7ff0cc070540 2024-01-24T17:47:37.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.024+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0cc1efeb0 con 0x7ff0cc070540 2024-01-24T17:47:37.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.025+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0cc04dff0 con 0x7ff0cc070540 2024-01-24T17:47:37.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.025+0000 7ff0c3400640 1 -- 172.21.15.39:0/3085274743 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff0bc00bd30 con 0x7ff0cc070540 2024-01-24T17:47:37.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.026+0000 7ff0c3400640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0a406a430 0x7ff0a406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.026+0000 7ff0c3400640 1 -- 172.21.15.39:0/3085274743 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0bc088d90 con 0x7ff0cc070540 2024-01-24T17:47:37.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.027+0000 7ff0c3e00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0a406a430 0x7ff0a406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.028+0000 7ff0c3e00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0a406a430 0x7ff0a406c8f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0c4e80 tx=0x7ff0b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:37.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.030+0000 7ff0c3400640 1 -- 172.21.15.39:0/3085274743 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff0bc054410 con 0x7ff0cc070540 2024-01-24T17:47:37.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.222+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0cc15ba40 con 0x7ff0a406a430 2024-01-24T17:47:37.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.244+0000 7ff0c3400640 1 -- 172.21.15.39:0/3085274743 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287482 (secure 0 0 0) 0x7ff0b40013e0 con 0x7ff0a406a430 2024-01-24T17:47:37.249 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:37.250 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:47:37.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.250+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0a406a430 msgr2=0x7ff0a406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.250+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0a406a430 0x7ff0a406c8f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0c4e80 tx=0x7ff0b4009040 comp rx=0 tx=0).stop 2024-01-24T17:47:37.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.250+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc070540 msgr2=0x7ff0cc0c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.250+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc070540 0x7ff0cc0c7380 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00d800 tx=0x7ff0bc00dcc0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.251+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 shutdown_connections 2024-01-24T17:47:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.251+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0cc15e910 0x7ff0cc1534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.251+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff0cc0b3e00 0x7ff0cc0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.251+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff0a406a430 0x7ff0a406c8f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.251+0000 7ff0cbe00640 1 --2- 172.21.15.39:0/3085274743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0cc070540 0x7ff0cc0c7380 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.251+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 >> 172.21.15.39:0/3085274743 conn(0x7ff0cc06a340 msgr2=0x7ff0cc0b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.251+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 shutdown_connections 2024-01-24T17:47:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.251+0000 7ff0cbe00640 1 -- 172.21.15.39:0/3085274743 wait complete. 2024-01-24T17:47:37.273 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-01-24T17:47:36.340685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258424,"kb_used_data":9460,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754716296,"statfs":{"total":773094113280,"available":772829487104,"internally_reserved":0,"allocated":9687040,"data_stored":4798413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.907629"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987878+0000","last_change":"2024-01-24T17:47:31.978602+0000","last_active":"2024-01-24T17:47:31.987878+0000","last_peered":"2024-01-24T17:47:31.987878+0000","last_clean":"2024-01-24T17:47:31.987878+0000","last_became_active":"2024-01-24T17:47:31.975764+0000","last_became_peered":"2024-01-24T17:47:31.975764+0000","last_unstale":"2024-01-24T17:47:31.987878+0000","last_undegraded":"2024-01-24T17:47:31.987878+0000","last_fullsized":"2024-01-24T17:47:31.987878+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:50.606488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989598+0000","last_change":"2024-01-24T17:47:31.988043+0000","last_active":"2024-01-24T17:47:31.989598+0000","last_peered":"2024-01-24T17:47:31.989598+0000","last_clean":"2024-01-24T17:47:31.989598+0000","last_became_active":"2024-01-24T17:47:31.987625+0000","last_became_peered":"2024-01-24T17:47:31.987625+0000","last_unstale":"2024-01-24T17:47:31.989598+0000","last_undegraded":"2024-01-24T17:47:31.989598+0000","last_fullsized":"2024-01-24T17:47:31.989598+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:36.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.973573+0000","last_change":"2024-01-24T17:47:31.969778+0000","last_active":"2024-01-24T17:47:31.973573+0000","last_peered":"2024-01-24T17:47:31.973573+0000","last_clean":"2024-01-24T17:47:31.973573+0000","last_became_active":"2024-01-24T17:47:31.969522+0000","last_became_peered":"2024-01-24T17:47:31.969522+0000","last_unstale":"2024-01-24T17:47:31.973573+0000","last_undegraded":"2024-01-24T17:47:31.973573+0000","last_fullsized":"2024-01-24T17:47:31.973573+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:39:01.873919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.985803+0000","last_change":"2024-01-24T17:47:31.976646+0000","last_active":"2024-01-24T17:47:31.985803+0000","last_peered":"2024-01-24T17:47:31.985803+0000","last_clean":"2024-01-24T17:47:31.985803+0000","last_became_active":"2024-01-24T17:47:31.976249+0000","last_became_peered":"2024-01-24T17:47:31.976249+0000","last_unstale":"2024-01-24T17:47:31.985803+0000","last_undegraded":"2024-01-24T17:47:31.985803+0000","last_fullsized":"2024-01-24T17:47:31.985803+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:01:55.725510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986743+0000","last_change":"2024-01-24T17:47:31.979278+0000","last_active":"2024-01-24T17:47:31.986743+0000","last_peered":"2024-01-24T17:47:31.986743+0000","last_clean":"2024-01-24T17:47:31.986743+0000","last_became_active":"2024-01-24T17:47:31.978982+0000","last_became_peered":"2024-01-24T17:47:31.978982+0000","last_unstale":"2024-01-24T17:47:31.986743+0000","last_undegraded":"2024-01-24T17:47:31.986743+0000","last_fullsized":"2024-01-24T17:47:31.986743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:47:24.166509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.992486+0000","last_change":"2024-01-24T17:47:31.985796+0000","last_active":"2024-01-24T17:47:31.992486+0000","last_peered":"2024-01-24T17:47:31.992486+0000","last_clean":"2024-01-24T17:47:31.992486+0000","last_became_active":"2024-01-24T17:47:31.985418+0000","last_became_peered":"2024-01-24T17:47:31.985418+0000","last_unstale":"2024-01-24T17:47:31.992486+0000","last_undegraded":"2024-01-24T17:47:31.992486+0000","last_fullsized":"2024-01-24T17:47:31.992486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:02.086314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989228+0000","last_change":"2024-01-24T17:47:31.985251+0000","last_active":"2024-01-24T17:47:31.989228+0000","last_peered":"2024-01-24T17:47:31.989228+0000","last_clean":"2024-01-24T17:47:31.989228+0000","last_became_active":"2024-01-24T17:47:31.984888+0000","last_became_peered":"2024-01-24T17:47:31.984888+0000","last_unstale":"2024-01-24T17:47:31.989228+0000","last_undegraded":"2024-01-24T17:47:31.989228+0000","last_fullsized":"2024-01-24T17:47:31.989228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:29.312975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.978282+0000","last_change":"2024-01-24T17:47:31.972718+0000","last_active":"2024-01-24T17:47:31.978282+0000","last_peered":"2024-01-24T17:47:31.978282+0000","last_clean":"2024-01-24T17:47:31.978282+0000","last_became_active":"2024-01-24T17:47:31.971566+0000","last_became_peered":"2024-01-24T17:47:31.971566+0000","last_unstale":"2024-01-24T17:47:31.978282+0000","last_undegraded":"2024-01-24T17:47:31.978282+0000","last_fullsized":"2024-01-24T17:47:31.978282+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:41.672001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988168+0000","last_change":"2024-01-24T17:47:31.979950+0000","last_active":"2024-01-24T17:47:31.988168+0000","last_peered":"2024-01-24T17:47:31.988168+0000","last_clean":"2024-01-24T17:47:31.988168+0000","last_became_active":"2024-01-24T17:47:31.979492+0000","last_became_peered":"2024-01-24T17:47:31.979492+0000","last_unstale":"2024-01-24T17:47:31.988168+0000","last_undegraded":"2024-01-24T17:47:31.988168+0000","last_fullsized":"2024-01-24T17:47:31.988168+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:29.312975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989433+0000","last_change":"2024-01-24T17:47:31.982990+0000","last_active":"2024-01-24T17:47:31.989433+0000","last_peered":"2024-01-24T17:47:31.989433+0000","last_clean":"2024-01-24T17:47:31.989433+0000","last_became_active":"2024-01-24T17:47:31.982732+0000","last_became_peered":"2024-01-24T17:47:31.982732+0000","last_unstale":"2024-01-24T17:47:31.989433+0000","last_undegraded":"2024-01-24T17:47:31.989433+0000","last_fullsized":"2024-01-24T17:47:31.989433+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:45:52.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.976356+0000","last_change":"2024-01-24T17:47:31.969482+0000","last_active":"2024-01-24T17:47:31.976356+0000","last_peered":"2024-01-24T17:47:31.976356+0000","last_clean":"2024-01-24T17:47:31.976356+0000","last_became_active":"2024-01-24T17:47:31.969131+0000","last_became_peered":"2024-01-24T17:47:31.969131+0000","last_unstale":"2024-01-24T17:47:31.976356+0000","last_undegraded":"2024-01-24T17:47:31.976356+0000","last_fullsized":"2024-01-24T17:47:31.976356+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:42.446661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982536+0000","last_change":"2024-01-24T17:47:31.976914+0000","last_active":"2024-01-24T17:47:31.982536+0000","last_peered":"2024-01-24T17:47:31.982536+0000","last_clean":"2024-01-24T17:47:31.982536+0000","last_became_active":"2024-01-24T17:47:31.976624+0000","last_became_peered":"2024-01-24T17:47:31.976624+0000","last_unstale":"2024-01-24T17:47:31.982536+0000","last_undegraded":"2024-01-24T17:47:31.982536+0000","last_fullsized":"2024-01-24T17:47:31.982536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:02.086314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.981604+0000","last_change":"2024-01-24T17:47:31.970970+0000","last_active":"2024-01-24T17:47:31.981604+0000","last_peered":"2024-01-24T17:47:31.981604+0000","last_clean":"2024-01-24T17:47:31.981604+0000","last_became_active":"2024-01-24T17:47:31.970702+0000","last_became_peered":"2024-01-24T17:47:31.970702+0000","last_unstale":"2024-01-24T17:47:31.981604+0000","last_undegraded":"2024-01-24T17:47:31.981604+0000","last_fullsized":"2024-01-24T17:47:31.981604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:49.369796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987581+0000","last_change":"2024-01-24T17:47:31.985226+0000","last_active":"2024-01-24T17:47:31.987581+0000","last_peered":"2024-01-24T17:47:31.987581+0000","last_clean":"2024-01-24T17:47:31.987581+0000","last_became_active":"2024-01-24T17:47:31.982699+0000","last_became_peered":"2024-01-24T17:47:31.982699+0000","last_unstale":"2024-01-24T17:47:31.987581+0000","last_undegraded":"2024-01-24T17:47:31.987581+0000","last_fullsized":"2024-01-24T17:47:31.987581+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:27:00.612042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989090+0000","last_change":"2024-01-24T17:47:31.982126+0000","last_active":"2024-01-24T17:47:31.989090+0000","last_peered":"2024-01-24T17:47:31.989090+0000","last_clean":"2024-01-24T17:47:31.989090+0000","last_became_active":"2024-01-24T17:47:31.981499+0000","last_became_peered":"2024-01-24T17:47:31.981499+0000","last_unstale":"2024-01-24T17:47:31.989090+0000","last_undegraded":"2024-01-24T17:47:31.989090+0000","last_fullsized":"2024-01-24T17:47:31.989090+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:44:25.582344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.973161+0000","last_change":"2024-01-24T17:47:31.969027+0000","last_active":"2024-01-24T17:47:31.973161+0000","last_peered":"2024-01-24T17:47:31.973161+0000","last_clean":"2024-01-24T17:47:31.973161+0000","last_became_active":"2024-01-24T17:47:31.968140+0000","last_became_peered":"2024-01-24T17:47:31.968140+0000","last_unstale":"2024-01-24T17:47:31.973161+0000","last_undegraded":"2024-01-24T17:47:31.973161+0000","last_fullsized":"2024-01-24T17:47:31.973161+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:01:55.725510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.980899+0000","last_change":"2024-01-24T17:47:31.975245+0000","last_active":"2024-01-24T17:47:31.980899+0000","last_peered":"2024-01-24T17:47:31.980899+0000","last_clean":"2024-01-24T17:47:31.980899+0000","last_became_active":"2024-01-24T17:47:31.974992+0000","last_became_peered":"2024-01-24T17:47:31.974992+0000","last_unstale":"2024-01-24T17:47:31.980899+0000","last_undegraded":"2024-01-24T17:47:31.980899+0000","last_fullsized":"2024-01-24T17:47:31.980899+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:05:57.892601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.976432+0000","last_change":"2024-01-24T17:47:31.970431+0000","last_active":"2024-01-24T17:47:31.976432+0000","last_peered":"2024-01-24T17:47:31.976432+0000","last_clean":"2024-01-24T17:47:31.976432+0000","last_became_active":"2024-01-24T17:47:31.969836+0000","last_became_peered":"2024-01-24T17:47:31.969836+0000","last_unstale":"2024-01-24T17:47:31.976432+0000","last_undegraded":"2024-01-24T17:47:31.976432+0000","last_fullsized":"2024-01-24T17:47:31.976432+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:22:35.200706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984293+0000","last_change":"2024-01-24T17:47:31.982167+0000","last_active":"2024-01-24T17:47:31.984293+0000","last_peered":"2024-01-24T17:47:31.984293+0000","last_clean":"2024-01-24T17:47:31.984293+0000","last_became_active":"2024-01-24T17:47:31.981907+0000","last_became_peered":"2024-01-24T17:47:31.981907+0000","last_unstale":"2024-01-24T17:47:31.984293+0000","last_undegraded":"2024-01-24T17:47:31.984293+0000","last_fullsized":"2024-01-24T17:47:31.984293+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:34:02.086314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987287+0000","last_change":"2024-01-24T17:47:31.983379+0000","last_active":"2024-01-24T17:47:31.987287+0000","last_peered":"2024-01-24T17:47:31.987287+0000","last_clean":"2024-01-24T17:47:31.987287+0000","last_became_active":"2024-01-24T17:47:31.982770+0000","last_became_peered":"2024-01-24T17:47:31.982770+0000","last_unstale":"2024-01-24T17:47:31.987287+0000","last_undegraded":"2024-01-24T17:47:31.987287+0000","last_fullsized":"2024-01-24T17:47:31.987287+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:37:29.833370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986542+0000","last_change":"2024-01-24T17:47:31.980010+0000","last_active":"2024-01-24T17:47:31.986542+0000","last_peered":"2024-01-24T17:47:31.986542+0000","last_clean":"2024-01-24T17:47:31.986542+0000","last_became_active":"2024-01-24T17:47:31.978501+0000","last_became_peered":"2024-01-24T17:47:31.978501+0000","last_unstale":"2024-01-24T17:47:31.986542+0000","last_undegraded":"2024-01-24T17:47:31.986542+0000","last_fullsized":"2024-01-24T17:47:31.986542+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:39:01.873919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986642+0000","last_change":"2024-01-24T17:47:31.980468+0000","last_active":"2024-01-24T17:47:31.986642+0000","last_peered":"2024-01-24T17:47:31.986642+0000","last_clean":"2024-01-24T17:47:31.986642+0000","last_became_active":"2024-01-24T17:47:31.980182+0000","last_became_peered":"2024-01-24T17:47:31.980182+0000","last_unstale":"2024-01-24T17:47:31.986642+0000","last_undegraded":"2024-01-24T17:47:31.986642+0000","last_fullsized":"2024-01-24T17:47:31.986642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:41.672001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.981112+0000","last_change":"2024-01-24T17:47:31.975478+0000","last_active":"2024-01-24T17:47:31.981112+0000","last_peered":"2024-01-24T17:47:31.981112+0000","last_clean":"2024-01-24T17:47:31.981112+0000","last_became_active":"2024-01-24T17:47:31.975178+0000","last_became_peered":"2024-01-24T17:47:31.975178+0000","last_unstale":"2024-01-24T17:47:31.981112+0000","last_undegraded":"2024-01-24T17:47:31.981112+0000","last_fullsized":"2024-01-24T17:47:31.981112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:25:35.181055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984509+0000","last_change":"2024-01-24T17:47:31.977142+0000","last_active":"2024-01-24T17:47:31.984509+0000","last_peered":"2024-01-24T17:47:31.984509+0000","last_clean":"2024-01-24T17:47:31.984509+0000","last_became_active":"2024-01-24T17:47:31.976847+0000","last_became_peered":"2024-01-24T17:47:31.976847+0000","last_unstale":"2024-01-24T17:47:31.984509+0000","last_undegraded":"2024-01-24T17:47:31.984509+0000","last_fullsized":"2024-01-24T17:47:31.984509+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:14.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988429+0000","last_change":"2024-01-24T17:47:31.981607+0000","last_active":"2024-01-24T17:47:31.988429+0000","last_peered":"2024-01-24T17:47:31.988429+0000","last_clean":"2024-01-24T17:47:31.988429+0000","last_became_active":"2024-01-24T17:47:31.981270+0000","last_became_peered":"2024-01-24T17:47:31.981270+0000","last_unstale":"2024-01-24T17:47:31.988429+0000","last_undegraded":"2024-01-24T17:47:31.988429+0000","last_fullsized":"2024-01-24T17:47:31.988429+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:06:51.007190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987653+0000","last_change":"2024-01-24T17:47:31.982887+0000","last_active":"2024-01-24T17:47:31.987653+0000","last_peered":"2024-01-24T17:47:31.987653+0000","last_clean":"2024-01-24T17:47:31.987653+0000","last_became_active":"2024-01-24T17:47:31.982336+0000","last_became_peered":"2024-01-24T17:47:31.982336+0000","last_unstale":"2024-01-24T17:47:31.987653+0000","last_undegraded":"2024-01-24T17:47:31.987653+0000","last_fullsized":"2024-01-24T17:47:31.987653+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:07:19.226610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987936+0000","last_change":"2024-01-24T17:47:31.981798+0000","last_active":"2024-01-24T17:47:31.987936+0000","last_peered":"2024-01-24T17:47:31.987936+0000","last_clean":"2024-01-24T17:47:31.987936+0000","last_became_active":"2024-01-24T17:47:31.981509+0000","last_became_peered":"2024-01-24T17:47:31.981509+0000","last_unstale":"2024-01-24T17:47:31.987936+0000","last_undegraded":"2024-01-24T17:47:31.987936+0000","last_fullsized":"2024-01-24T17:47:31.987936+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:18:05.792016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988031+0000","last_change":"2024-01-24T17:47:31.980086+0000","last_active":"2024-01-24T17:47:31.988031+0000","last_peered":"2024-01-24T17:47:31.988031+0000","last_clean":"2024-01-24T17:47:31.988031+0000","last_became_active":"2024-01-24T17:47:31.979832+0000","last_became_peered":"2024-01-24T17:47:31.979832+0000","last_unstale":"2024-01-24T17:47:31.988031+0000","last_undegraded":"2024-01-24T17:47:31.988031+0000","last_fullsized":"2024-01-24T17:47:31.988031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:36.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984218+0000","last_change":"2024-01-24T17:47:31.979202+0000","last_active":"2024-01-24T17:47:31.984218+0000","last_peered":"2024-01-24T17:47:31.984218+0000","last_clean":"2024-01-24T17:47:31.984218+0000","last_became_active":"2024-01-24T17:47:31.978020+0000","last_became_peered":"2024-01-24T17:47:31.978020+0000","last_unstale":"2024-01-24T17:47:31.984218+0000","last_undegraded":"2024-01-24T17:47:31.984218+0000","last_fullsized":"2024-01-24T17:47:31.984218+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:42:33.273575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988724+0000","last_change":"2024-01-24T17:47:31.986551+0000","last_active":"2024-01-24T17:47:31.988724+0000","last_peered":"2024-01-24T17:47:31.988724+0000","last_clean":"2024-01-24T17:47:31.988724+0000","last_became_active":"2024-01-24T17:47:31.986260+0000","last_became_peered":"2024-01-24T17:47:31.986260+0000","last_unstale":"2024-01-24T17:47:31.988724+0000","last_undegraded":"2024-01-24T17:47:31.988724+0000","last_fullsized":"2024-01-24T17:47:31.988724+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:50:13.325782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.992498+0000","last_change":"2024-01-24T17:47:31.986372+0000","last_active":"2024-01-24T17:47:31.992498+0000","last_peered":"2024-01-24T17:47:31.992498+0000","last_clean":"2024-01-24T17:47:31.992498+0000","last_became_active":"2024-01-24T17:47:31.984535+0000","last_became_peered":"2024-01-24T17:47:31.984535+0000","last_unstale":"2024-01-24T17:47:31.992498+0000","last_undegraded":"2024-01-24T17:47:31.992498+0000","last_fullsized":"2024-01-24T17:47:31.992498+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:42:33.273575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984514+0000","last_change":"2024-01-24T17:47:31.981085+0000","last_active":"2024-01-24T17:47:31.984514+0000","last_peered":"2024-01-24T17:47:31.984514+0000","last_clean":"2024-01-24T17:47:31.984514+0000","last_became_active":"2024-01-24T17:47:31.980820+0000","last_became_peered":"2024-01-24T17:47:31.980820+0000","last_unstale":"2024-01-24T17:47:31.984514+0000","last_undegraded":"2024-01-24T17:47:31.984514+0000","last_fullsized":"2024-01-24T17:47:31.984514+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:36:49.719214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982863+0000","last_change":"2024-01-24T17:47:31.977921+0000","last_active":"2024-01-24T17:47:31.982863+0000","last_peered":"2024-01-24T17:47:31.982863+0000","last_clean":"2024-01-24T17:47:31.982863+0000","last_became_active":"2024-01-24T17:47:31.976080+0000","last_became_peered":"2024-01-24T17:47:31.976080+0000","last_unstale":"2024-01-24T17:47:31.982863+0000","last_undegraded":"2024-01-24T17:47:31.982863+0000","last_fullsized":"2024-01-24T17:47:31.982863+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:45:52.581810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"87'4","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986228+0000","last_change":"2024-01-24T17:47:31.469900+0000","last_active":"2024-01-24T17:47:31.986228+0000","last_peered":"2024-01-24T17:47:31.986228+0000","last_clean":"2024-01-24T17:47:31.986228+0000","last_became_active":"2024-01-24T17:47:31.469573+0000","last_became_peered":"2024-01-24T17:47:31.469573+0000","last_unstale":"2024-01-24T17:47:31.986228+0000","last_undegraded":"2024-01-24T17:47:31.986228+0000","last_fullsized":"2024-01-24T17:47:31.986228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:43:24.898534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.985151+0000","last_change":"2024-01-24T17:47:31.975999+0000","last_active":"2024-01-24T17:47:31.985151+0000","last_peered":"2024-01-24T17:47:31.985151+0000","last_clean":"2024-01-24T17:47:31.985151+0000","last_became_active":"2024-01-24T17:47:31.975475+0000","last_became_peered":"2024-01-24T17:47:31.975475+0000","last_unstale":"2024-01-24T17:47:31.985151+0000","last_undegraded":"2024-01-24T17:47:31.985151+0000","last_fullsized":"2024-01-24T17:47:31.985151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:24:14.660148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1e","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.971733+0000","last_change":"2024-01-24T17:47:31.486123+0000","last_active":"2024-01-24T17:47:31.971733+0000","last_peered":"2024-01-24T17:47:31.971733+0000","last_clean":"2024-01-24T17:47:31.971733+0000","last_became_active":"2024-01-24T17:47:31.485142+0000","last_became_peered":"2024-01-24T17:47:31.485142+0000","last_unstale":"2024-01-24T17:47:31.971733+0000","last_undegraded":"2024-01-24T17:47:31.971733+0000","last_fullsized":"2024-01-24T17:47:31.971733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:26:18.040164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982554+0000","last_change":"2024-01-24T17:47:31.980165+0000","last_active":"2024-01-24T17:47:31.982554+0000","last_peered":"2024-01-24T17:47:31.982554+0000","last_clean":"2024-01-24T17:47:31.982554+0000","last_became_active":"2024-01-24T17:47:31.979897+0000","last_became_peered":"2024-01-24T17:47:31.979897+0000","last_unstale":"2024-01-24T17:47:31.982554+0000","last_undegraded":"2024-01-24T17:47:31.982554+0000","last_fullsized":"2024-01-24T17:47:31.982554+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:50:13.325782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1d","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.985998+0000","last_change":"2024-01-24T17:47:31.474567+0000","last_active":"2024-01-24T17:47:31.985998+0000","last_peered":"2024-01-24T17:47:31.985998+0000","last_clean":"2024-01-24T17:47:31.985998+0000","last_became_active":"2024-01-24T17:47:31.473003+0000","last_became_peered":"2024-01-24T17:47:31.473003+0000","last_unstale":"2024-01-24T17:47:31.985998+0000","last_undegraded":"2024-01-24T17:47:31.985998+0000","last_fullsized":"2024-01-24T17:47:31.985998+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:19:02.354998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988033+0000","last_change":"2024-01-24T17:47:31.980899+0000","last_active":"2024-01-24T17:47:31.988033+0000","last_peered":"2024-01-24T17:47:31.988033+0000","last_clean":"2024-01-24T17:47:31.988033+0000","last_became_active":"2024-01-24T17:47:31.979510+0000","last_became_peered":"2024-01-24T17:47:31.979510+0000","last_unstale":"2024-01-24T17:47:31.988033+0000","last_undegraded":"2024-01-24T17:47:31.988033+0000","last_fullsized":"2024-01-24T17:47:31.988033+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:44:25.582344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1c","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986121+0000","last_change":"2024-01-24T17:47:31.475836+0000","last_active":"2024-01-24T17:47:31.986121+0000","last_peered":"2024-01-24T17:47:31.986121+0000","last_clean":"2024-01-24T17:47:31.986121+0000","last_became_active":"2024-01-24T17:47:31.475494+0000","last_became_peered":"2024-01-24T17:47:31.475494+0000","last_unstale":"2024-01-24T17:47:31.986121+0000","last_undegraded":"2024-01-24T17:47:31.986121+0000","last_fullsized":"2024-01-24T17:47:31.986121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:50:12.311017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984150+0000","last_change":"2024-01-24T17:47:31.978201+0000","last_active":"2024-01-24T17:47:31.984150+0000","last_peered":"2024-01-24T17:47:31.984150+0000","last_clean":"2024-01-24T17:47:31.984150+0000","last_became_active":"2024-01-24T17:47:31.977899+0000","last_became_peered":"2024-01-24T17:47:31.977899+0000","last_unstale":"2024-01-24T17:47:31.984150+0000","last_undegraded":"2024-01-24T17:47:31.984150+0000","last_fullsized":"2024-01-24T17:47:31.984150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:18:28.650673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1b","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.970911+0000","last_change":"2024-01-24T17:47:31.480673+0000","last_active":"2024-01-24T17:47:31.970911+0000","last_peered":"2024-01-24T17:47:31.970911+0000","last_clean":"2024-01-24T17:47:31.970911+0000","last_became_active":"2024-01-24T17:47:31.480287+0000","last_became_peered":"2024-01-24T17:47:31.480287+0000","last_unstale":"2024-01-24T17:47:31.970911+0000","last_undegraded":"2024-01-24T17:47:31.970911+0000","last_fullsized":"2024-01-24T17:47:31.970911+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:28.298210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.973632+0000","last_change":"2024-01-24T17:47:31.972189+0000","last_active":"2024-01-24T17:47:31.973632+0000","last_peered":"2024-01-24T17:47:31.973632+0000","last_clean":"2024-01-24T17:47:31.973632+0000","last_became_active":"2024-01-24T17:47:31.971764+0000","last_became_peered":"2024-01-24T17:47:31.971764+0000","last_unstale":"2024-01-24T17:47:31.973632+0000","last_undegraded":"2024-01-24T17:47:31.973632+0000","last_fullsized":"2024-01-24T17:47:31.973632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:05:01.572141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1a","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984659+0000","last_change":"2024-01-24T17:47:31.480183+0000","last_active":"2024-01-24T17:47:31.984659+0000","last_peered":"2024-01-24T17:47:31.984659+0000","last_clean":"2024-01-24T17:47:31.984659+0000","last_became_active":"2024-01-24T17:47:31.479839+0000","last_became_peered":"2024-01-24T17:47:31.479839+0000","last_unstale":"2024-01-24T17:47:31.984659+0000","last_undegraded":"2024-01-24T17:47:31.984659+0000","last_fullsized":"2024-01-24T17:47:31.984659+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:45:11.903479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.985233+0000","last_change":"2024-01-24T17:47:31.977862+0000","last_active":"2024-01-24T17:47:31.985233+0000","last_peered":"2024-01-24T17:47:31.985233+0000","last_clean":"2024-01-24T17:47:31.985233+0000","last_became_active":"2024-01-24T17:47:31.977607+0000","last_became_peered":"2024-01-24T17:47:31.977607+0000","last_unstale":"2024-01-24T17:47:31.985233+0000","last_undegraded":"2024-01-24T17:47:31.985233+0000","last_fullsized":"2024-01-24T17:47:31.985233+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:45:51.512395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.955448+0000","last_change":"2024-01-24T17:47:31.471080+0000","last_active":"2024-01-24T17:47:31.955448+0000","last_peered":"2024-01-24T17:47:31.955448+0000","last_clean":"2024-01-24T17:47:31.955448+0000","last_became_active":"2024-01-24T17:47:31.470740+0000","last_became_peered":"2024-01-24T17:47:31.470740+0000","last_unstale":"2024-01-24T17:47:31.955448+0000","last_undegraded":"2024-01-24T17:47:31.955448+0000","last_fullsized":"2024-01-24T17:47:31.955448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:44:24.567578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.977579+0000","last_change":"2024-01-24T17:47:31.975277+0000","last_active":"2024-01-24T17:47:31.977579+0000","last_peered":"2024-01-24T17:47:31.977579+0000","last_clean":"2024-01-24T17:47:31.977579+0000","last_became_active":"2024-01-24T17:47:31.974407+0000","last_became_peered":"2024-01-24T17:47:31.974407+0000","last_unstale":"2024-01-24T17:47:31.977579+0000","last_undegraded":"2024-01-24T17:47:31.977579+0000","last_fullsized":"2024-01-24T17:47:31.977579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:25:35.181055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.958750+0000","last_change":"2024-01-24T17:47:31.480759+0000","last_active":"2024-01-24T17:47:31.958750+0000","last_peered":"2024-01-24T17:47:31.958750+0000","last_clean":"2024-01-24T17:47:31.958750+0000","last_became_active":"2024-01-24T17:47:31.480504+0000","last_became_peered":"2024-01-24T17:47:31.480504+0000","last_unstale":"2024-01-24T17:47:31.958750+0000","last_undegraded":"2024-01-24T17:47:31.958750+0000","last_fullsized":"2024-01-24T17:47:31.958750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:49.591723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.978534+0000","last_change":"2024-01-24T17:47:31.975701+0000","last_active":"2024-01-24T17:47:31.978534+0000","last_peered":"2024-01-24T17:47:31.978534+0000","last_clean":"2024-01-24T17:47:31.978534+0000","last_became_active":"2024-01-24T17:47:31.975449+0000","last_became_peered":"2024-01-24T17:47:31.975449+0000","last_unstale":"2024-01-24T17:47:31.978534+0000","last_undegraded":"2024-01-24T17:47:31.978534+0000","last_fullsized":"2024-01-24T17:47:31.978534+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:05:57.892601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.975101+0000","last_change":"2024-01-24T17:47:31.474134+0000","last_active":"2024-01-24T17:47:31.975101+0000","last_peered":"2024-01-24T17:47:31.975101+0000","last_clean":"2024-01-24T17:47:31.975101+0000","last_became_active":"2024-01-24T17:47:31.471905+0000","last_became_peered":"2024-01-24T17:47:31.471905+0000","last_unstale":"2024-01-24T17:47:31.975101+0000","last_undegraded":"2024-01-24T17:47:31.975101+0000","last_fullsized":"2024-01-24T17:47:31.975101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:53:42.524052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988147+0000","last_change":"2024-01-24T17:47:31.978890+0000","last_active":"2024-01-24T17:47:31.988147+0000","last_peered":"2024-01-24T17:47:31.988147+0000","last_clean":"2024-01-24T17:47:31.988147+0000","last_became_active":"2024-01-24T17:47:31.978399+0000","last_became_peered":"2024-01-24T17:47:31.978399+0000","last_unstale":"2024-01-24T17:47:31.988147+0000","last_undegraded":"2024-01-24T17:47:31.988147+0000","last_fullsized":"2024-01-24T17:47:31.988147+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:16:08.410074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.947492+0000","last_change":"2024-01-24T17:47:31.490514+0000","last_active":"2024-01-24T17:47:31.947492+0000","last_peered":"2024-01-24T17:47:31.947492+0000","last_clean":"2024-01-24T17:47:31.947492+0000","last_became_active":"2024-01-24T17:47:31.490208+0000","last_became_peered":"2024-01-24T17:47:31.490208+0000","last_unstale":"2024-01-24T17:47:31.947492+0000","last_undegraded":"2024-01-24T17:47:31.947492+0000","last_fullsized":"2024-01-24T17:47:31.947492+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:06:49.992424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987654+0000","last_change":"2024-01-24T17:47:31.980709+0000","last_active":"2024-01-24T17:47:31.987654+0000","last_peered":"2024-01-24T17:47:31.987654+0000","last_clean":"2024-01-24T17:47:31.987654+0000","last_became_active":"2024-01-24T17:47:31.980394+0000","last_became_peered":"2024-01-24T17:47:31.980394+0000","last_unstale":"2024-01-24T17:47:31.987654+0000","last_undegraded":"2024-01-24T17:47:31.987654+0000","last_fullsized":"2024-01-24T17:47:31.987654+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:50.606488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.15","version":"87'3","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988092+0000","last_change":"2024-01-24T17:47:31.469330+0000","last_active":"2024-01-24T17:47:31.988092+0000","last_peered":"2024-01-24T17:47:31.988092+0000","last_clean":"2024-01-24T17:47:31.988092+0000","last_became_active":"2024-01-24T17:47:31.468761+0000","last_became_peered":"2024-01-24T17:47:31.468761+0000","last_unstale":"2024-01-24T17:47:31.988092+0000","last_undegraded":"2024-01-24T17:47:31.988092+0000","last_fullsized":"2024-01-24T17:47:31.988092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:11:15.408613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986307+0000","last_change":"2024-01-24T17:47:31.977501+0000","last_active":"2024-01-24T17:47:31.986307+0000","last_peered":"2024-01-24T17:47:31.986307+0000","last_clean":"2024-01-24T17:47:31.986307+0000","last_became_active":"2024-01-24T17:47:31.977202+0000","last_became_peered":"2024-01-24T17:47:31.977202+0000","last_unstale":"2024-01-24T17:47:31.986307+0000","last_undegraded":"2024-01-24T17:47:31.986307+0000","last_fullsized":"2024-01-24T17:47:31.986307+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:47:58.923857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.14","version":"87'6","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.990402+0000","last_change":"2024-01-24T17:47:31.470804+0000","last_active":"2024-01-24T17:47:31.990402+0000","last_peered":"2024-01-24T17:47:31.990402+0000","last_clean":"2024-01-24T17:47:31.990402+0000","last_became_active":"2024-01-24T17:47:31.470455+0000","last_became_peered":"2024-01-24T17:47:31.470455+0000","last_unstale":"2024-01-24T17:47:31.990402+0000","last_undegraded":"2024-01-24T17:47:31.990402+0000","last_fullsized":"2024-01-24T17:47:31.990402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:06:23.354739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.981438+0000","last_change":"2024-01-24T17:47:31.974806+0000","last_active":"2024-01-24T17:47:31.981438+0000","last_peered":"2024-01-24T17:47:31.981438+0000","last_clean":"2024-01-24T17:47:31.981438+0000","last_became_active":"2024-01-24T17:47:31.974394+0000","last_became_peered":"2024-01-24T17:47:31.974394+0000","last_unstale":"2024-01-24T17:47:31.981438+0000","last_undegraded":"2024-01-24T17:47:31.981438+0000","last_fullsized":"2024-01-24T17:47:31.981438+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:36.495291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.13","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.983348+0000","last_change":"2024-01-24T17:47:31.471680+0000","last_active":"2024-01-24T17:47:31.983348+0000","last_peered":"2024-01-24T17:47:31.983348+0000","last_clean":"2024-01-24T17:47:31.983348+0000","last_became_active":"2024-01-24T17:47:31.471405+0000","last_became_peered":"2024-01-24T17:47:31.471405+0000","last_unstale":"2024-01-24T17:47:31.983348+0000","last_undegraded":"2024-01-24T17:47:31.983348+0000","last_fullsized":"2024-01-24T17:47:31.983348+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:26:59.597277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987965+0000","last_change":"2024-01-24T17:47:31.978744+0000","last_active":"2024-01-24T17:47:31.987965+0000","last_peered":"2024-01-24T17:47:31.987965+0000","last_clean":"2024-01-24T17:47:31.987965+0000","last_became_active":"2024-01-24T17:47:31.978145+0000","last_became_peered":"2024-01-24T17:47:31.978145+0000","last_unstale":"2024-01-24T17:47:31.987965+0000","last_undegraded":"2024-01-24T17:47:31.987965+0000","last_fullsized":"2024-01-24T17:47:31.987965+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:08:04.652220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.975812+0000","last_change":"2024-01-24T17:47:31.490315+0000","last_active":"2024-01-24T17:47:31.975812+0000","last_peered":"2024-01-24T17:47:31.975812+0000","last_clean":"2024-01-24T17:47:31.975812+0000","last_became_active":"2024-01-24T17:47:31.490055+0000","last_became_peered":"2024-01-24T17:47:31.490055+0000","last_unstale":"2024-01-24T17:47:31.975812+0000","last_undegraded":"2024-01-24T17:47:31.975812+0000","last_fullsized":"2024-01-24T17:47:31.975812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:55:53.162650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.976931+0000","last_change":"2024-01-24T17:47:31.975363+0000","last_active":"2024-01-24T17:47:31.976931+0000","last_peered":"2024-01-24T17:47:31.976931+0000","last_clean":"2024-01-24T17:47:31.976931+0000","last_became_active":"2024-01-24T17:47:31.974933+0000","last_became_peered":"2024-01-24T17:47:31.974933+0000","last_unstale":"2024-01-24T17:47:31.976931+0000","last_undegraded":"2024-01-24T17:47:31.976931+0000","last_fullsized":"2024-01-24T17:47:31.976931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:33:35.823078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.947824+0000","last_change":"2024-01-24T17:47:31.485453+0000","last_active":"2024-01-24T17:47:31.947824+0000","last_peered":"2024-01-24T17:47:31.947824+0000","last_clean":"2024-01-24T17:47:31.947824+0000","last_became_active":"2024-01-24T17:47:31.484853+0000","last_became_peered":"2024-01-24T17:47:31.484853+0000","last_unstale":"2024-01-24T17:47:31.947824+0000","last_undegraded":"2024-01-24T17:47:31.947824+0000","last_fullsized":"2024-01-24T17:47:31.947824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:22:34.185940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989223+0000","last_change":"2024-01-24T17:47:31.986869+0000","last_active":"2024-01-24T17:47:31.989223+0000","last_peered":"2024-01-24T17:47:31.989223+0000","last_clean":"2024-01-24T17:47:31.989223+0000","last_became_active":"2024-01-24T17:47:31.986596+0000","last_became_peered":"2024-01-24T17:47:31.986596+0000","last_unstale":"2024-01-24T17:47:31.989223+0000","last_undegraded":"2024-01-24T17:47:31.989223+0000","last_fullsized":"2024-01-24T17:47:31.989223+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:06:24.369505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.10","version":"87'8","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.988222+0000","last_change":"2024-01-24T17:47:31.471831+0000","last_active":"2024-01-24T17:47:31.988222+0000","last_peered":"2024-01-24T17:47:31.988222+0000","last_clean":"2024-01-24T17:47:31.988222+0000","last_became_active":"2024-01-24T17:47:31.469998+0000","last_became_peered":"2024-01-24T17:47:31.469998+0000","last_unstale":"2024-01-24T17:47:31.988222+0000","last_undegraded":"2024-01-24T17:47:31.988222+0000","last_fullsized":"2024-01-24T17:47:31.988222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:12:08.558957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982845+0000","last_change":"2024-01-24T17:47:31.977884+0000","last_active":"2024-01-24T17:47:31.982845+0000","last_peered":"2024-01-24T17:47:31.982845+0000","last_clean":"2024-01-24T17:47:31.982845+0000","last_became_active":"2024-01-24T17:47:31.977202+0000","last_became_peered":"2024-01-24T17:47:31.977202+0000","last_unstale":"2024-01-24T17:47:31.982845+0000","last_undegraded":"2024-01-24T17:47:31.982845+0000","last_fullsized":"2024-01-24T17:47:31.982845+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:37:29.833370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.974819+0000","last_change":"2024-01-24T17:47:31.475331+0000","last_active":"2024-01-24T17:47:31.974819+0000","last_peered":"2024-01-24T17:47:31.974819+0000","last_clean":"2024-01-24T17:47:31.974819+0000","last_became_active":"2024-01-24T17:47:31.475070+0000","last_became_peered":"2024-01-24T17:47:31.475070+0000","last_unstale":"2024-01-24T17:47:31.974819+0000","last_undegraded":"2024-01-24T17:47:31.974819+0000","last_fullsized":"2024-01-24T17:47:31.974819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:09:11.261877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982155+0000","last_change":"2024-01-24T17:47:31.974592+0000","last_active":"2024-01-24T17:47:31.982155+0000","last_peered":"2024-01-24T17:47:31.982155+0000","last_clean":"2024-01-24T17:47:31.982155+0000","last_became_active":"2024-01-24T17:47:31.974297+0000","last_became_peered":"2024-01-24T17:47:31.974297+0000","last_unstale":"2024-01-24T17:47:31.982155+0000","last_undegraded":"2024-01-24T17:47:31.982155+0000","last_fullsized":"2024-01-24T17:47:31.982155+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:06:41.488722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.e","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.982728+0000","last_change":"2024-01-24T17:47:31.477111+0000","last_active":"2024-01-24T17:47:31.982728+0000","last_peered":"2024-01-24T17:47:31.982728+0000","last_clean":"2024-01-24T17:47:31.982728+0000","last_became_active":"2024-01-24T17:47:31.476458+0000","last_became_peered":"2024-01-24T17:47:31.476458+0000","last_unstale":"2024-01-24T17:47:31.982728+0000","last_undegraded":"2024-01-24T17:47:31.982728+0000","last_fullsized":"2024-01-24T17:47:31.982728+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:28.298210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.990383+0000","last_change":"2024-01-24T17:47:31.987367+0000","last_active":"2024-01-24T17:47:31.990383+0000","last_peered":"2024-01-24T17:47:31.990383+0000","last_clean":"2024-01-24T17:47:31.990383+0000","last_became_active":"2024-01-24T17:47:31.985568+0000","last_became_peered":"2024-01-24T17:47:31.985568+0000","last_unstale":"2024-01-24T17:47:31.990383+0000","last_undegraded":"2024-01-24T17:47:31.990383+0000","last_fullsized":"2024-01-24T17:47:31.990383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:36:49.719214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.954294+0000","last_change":"2024-01-24T17:47:31.471090+0000","last_active":"2024-01-24T17:47:31.954294+0000","last_peered":"2024-01-24T17:47:31.954294+0000","last_clean":"2024-01-24T17:47:31.954294+0000","last_became_active":"2024-01-24T17:47:31.470835+0000","last_became_peered":"2024-01-24T17:47:31.470835+0000","last_unstale":"2024-01-24T17:47:31.954294+0000","last_undegraded":"2024-01-24T17:47:31.954294+0000","last_fullsized":"2024-01-24T17:47:31.954294+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:06:23.354739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.990290+0000","last_change":"2024-01-24T17:47:31.988558+0000","last_active":"2024-01-24T17:47:31.990290+0000","last_peered":"2024-01-24T17:47:31.990290+0000","last_clean":"2024-01-24T17:47:31.990290+0000","last_became_active":"2024-01-24T17:47:31.988277+0000","last_became_peered":"2024-01-24T17:47:31.988277+0000","last_unstale":"2024-01-24T17:47:31.990290+0000","last_undegraded":"2024-01-24T17:47:31.990290+0000","last_fullsized":"2024-01-24T17:47:31.990290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:01.119477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.b","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.983760+0000","last_change":"2024-01-24T17:47:31.474153+0000","last_active":"2024-01-24T17:47:31.983760+0000","last_peered":"2024-01-24T17:47:31.983760+0000","last_clean":"2024-01-24T17:47:31.983760+0000","last_became_active":"2024-01-24T17:47:31.473899+0000","last_became_peered":"2024-01-24T17:47:31.473899+0000","last_unstale":"2024-01-24T17:47:31.983760+0000","last_undegraded":"2024-01-24T17:47:31.983760+0000","last_fullsized":"2024-01-24T17:47:31.983760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:08:03.637455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986154+0000","last_change":"2024-01-24T17:47:31.976045+0000","last_active":"2024-01-24T17:47:31.986154+0000","last_peered":"2024-01-24T17:47:31.986154+0000","last_clean":"2024-01-24T17:47:31.986154+0000","last_became_active":"2024-01-24T17:47:31.975787+0000","last_became_peered":"2024-01-24T17:47:31.975787+0000","last_unstale":"2024-01-24T17:47:31.986154+0000","last_undegraded":"2024-01-24T17:47:31.986154+0000","last_fullsized":"2024-01-24T17:47:31.986154+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:11:48.804217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.975910+0000","last_change":"2024-01-24T17:47:31.475124+0000","last_active":"2024-01-24T17:47:31.975910+0000","last_peered":"2024-01-24T17:47:31.975910+0000","last_clean":"2024-01-24T17:47:31.975910+0000","last_became_active":"2024-01-24T17:47:31.474152+0000","last_became_peered":"2024-01-24T17:47:31.474152+0000","last_unstale":"2024-01-24T17:47:31.975910+0000","last_undegraded":"2024-01-24T17:47:31.975910+0000","last_fullsized":"2024-01-24T17:47:31.975910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:57:19.521381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.984085+0000","last_change":"2024-01-24T17:47:31.978320+0000","last_active":"2024-01-24T17:47:31.984085+0000","last_peered":"2024-01-24T17:47:31.984085+0000","last_clean":"2024-01-24T17:47:31.984085+0000","last_became_active":"2024-01-24T17:47:31.977585+0000","last_became_peered":"2024-01-24T17:47:31.977585+0000","last_unstale":"2024-01-24T17:47:31.984085+0000","last_undegraded":"2024-01-24T17:47:31.984085+0000","last_fullsized":"2024-01-24T17:47:31.984085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:05:01.572141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.956610+0000","last_change":"2024-01-24T17:47:31.485244+0000","last_active":"2024-01-24T17:47:31.956610+0000","last_peered":"2024-01-24T17:47:31.956610+0000","last_clean":"2024-01-24T17:47:31.956610+0000","last_became_active":"2024-01-24T17:47:31.484975+0000","last_became_peered":"2024-01-24T17:47:31.484975+0000","last_unstale":"2024-01-24T17:47:31.956610+0000","last_undegraded":"2024-01-24T17:47:31.956610+0000","last_fullsized":"2024-01-24T17:47:31.956610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:44:58.760429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989632+0000","last_change":"2024-01-24T17:47:31.982104+0000","last_active":"2024-01-24T17:47:31.989632+0000","last_peered":"2024-01-24T17:47:31.989632+0000","last_clean":"2024-01-24T17:47:31.989632+0000","last_became_active":"2024-01-24T17:47:31.981840+0000","last_became_peered":"2024-01-24T17:47:31.981840+0000","last_unstale":"2024-01-24T17:47:31.989632+0000","last_undegraded":"2024-01-24T17:47:31.989632+0000","last_fullsized":"2024-01-24T17:47:31.989632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:27:00.612042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.955603+0000","last_change":"2024-01-24T17:47:31.462477+0000","last_active":"2024-01-24T17:47:31.955603+0000","last_peered":"2024-01-24T17:47:31.955603+0000","last_clean":"2024-01-24T17:47:31.955603+0000","last_became_active":"2024-01-24T17:47:31.461816+0000","last_became_peered":"2024-01-24T17:47:31.461816+0000","last_unstale":"2024-01-24T17:47:31.955603+0000","last_undegraded":"2024-01-24T17:47:31.955603+0000","last_fullsized":"2024-01-24T17:47:31.955603+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:00.104711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987672+0000","last_change":"2024-01-24T17:47:31.984830+0000","last_active":"2024-01-24T17:47:31.987672+0000","last_peered":"2024-01-24T17:47:31.987672+0000","last_clean":"2024-01-24T17:47:31.987672+0000","last_became_active":"2024-01-24T17:47:31.983625+0000","last_became_peered":"2024-01-24T17:47:31.983625+0000","last_unstale":"2024-01-24T17:47:31.987672+0000","last_undegraded":"2024-01-24T17:47:31.987672+0000","last_fullsized":"2024-01-24T17:47:31.987672+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:45:23.940696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.7","version":"87'2","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.971903+0000","last_change":"2024-01-24T17:47:31.480515+0000","last_active":"2024-01-24T17:47:31.971903+0000","last_peered":"2024-01-24T17:47:31.971903+0000","last_clean":"2024-01-24T17:47:31.971903+0000","last_became_active":"2024-01-24T17:47:31.480244+0000","last_became_peered":"2024-01-24T17:47:31.480244+0000","last_unstale":"2024-01-24T17:47:31.971903+0000","last_undegraded":"2024-01-24T17:47:31.971903+0000","last_fullsized":"2024-01-24T17:47:31.971903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:35.480526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.986338+0000","last_change":"2024-01-24T17:47:31.973621+0000","last_active":"2024-01-24T17:47:31.986338+0000","last_peered":"2024-01-24T17:47:31.986338+0000","last_clean":"2024-01-24T17:47:31.986338+0000","last_became_active":"2024-01-24T17:47:31.973367+0000","last_became_peered":"2024-01-24T17:47:31.973367+0000","last_unstale":"2024-01-24T17:47:31.986338+0000","last_undegraded":"2024-01-24T17:47:31.986338+0000","last_fullsized":"2024-01-24T17:47:31.986338+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:48:09.166883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.962666+0000","last_change":"2024-01-24T17:47:31.472561+0000","last_active":"2024-01-24T17:47:31.962666+0000","last_peered":"2024-01-24T17:47:31.962666+0000","last_clean":"2024-01-24T17:47:31.962666+0000","last_became_active":"2024-01-24T17:47:31.472096+0000","last_became_peered":"2024-01-24T17:47:31.472096+0000","last_unstale":"2024-01-24T17:47:31.962666+0000","last_undegraded":"2024-01-24T17:47:31.962666+0000","last_fullsized":"2024-01-24T17:47:31.962666+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:40:28.939958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.987656+0000","last_change":"2024-01-24T17:47:31.982250+0000","last_active":"2024-01-24T17:47:31.987656+0000","last_peered":"2024-01-24T17:47:31.987656+0000","last_clean":"2024-01-24T17:47:31.987656+0000","last_became_active":"2024-01-24T17:47:31.980597+0000","last_became_peered":"2024-01-24T17:47:31.980597+0000","last_unstale":"2024-01-24T17:47:31.987656+0000","last_undegraded":"2024-01-24T17:47:31.987656+0000","last_fullsized":"2024-01-24T17:47:31.987656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:16:26.944831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.954368+0000","last_change":"2024-01-24T17:47:31.476349+0000","last_active":"2024-01-24T17:47:31.954368+0000","last_peered":"2024-01-24T17:47:31.954368+0000","last_clean":"2024-01-24T17:47:31.954368+0000","last_became_active":"2024-01-24T17:47:31.476061+0000","last_became_peered":"2024-01-24T17:47:31.476061+0000","last_unstale":"2024-01-24T17:47:31.954368+0000","last_undegraded":"2024-01-24T17:47:31.954368+0000","last_fullsized":"2024-01-24T17:47:31.954368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:54:35.480526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.968387+0000","last_change":"2024-01-24T17:47:31.964868+0000","last_active":"2024-01-24T17:47:31.968387+0000","last_peered":"2024-01-24T17:47:31.968387+0000","last_clean":"2024-01-24T17:47:31.968387+0000","last_became_active":"2024-01-24T17:47:31.964563+0000","last_became_peered":"2024-01-24T17:47:31.964563+0000","last_unstale":"2024-01-24T17:47:31.968387+0000","last_undegraded":"2024-01-24T17:47:31.968387+0000","last_fullsized":"2024-01-24T17:47:31.968387+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:18:28.650673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.947117+0000","last_change":"2024-01-24T17:47:31.485474+0000","last_active":"2024-01-24T17:47:31.947117+0000","last_peered":"2024-01-24T17:47:31.947117+0000","last_clean":"2024-01-24T17:47:31.947117+0000","last_became_active":"2024-01-24T17:47:31.485117+0000","last_became_peered":"2024-01-24T17:47:31.485117+0000","last_unstale":"2024-01-24T17:47:31.947117+0000","last_undegraded":"2024-01-24T17:47:31.947117+0000","last_fullsized":"2024-01-24T17:47:31.947117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:27:41.431895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.980080+0000","last_change":"2024-01-24T17:47:31.972677+0000","last_active":"2024-01-24T17:47:31.980080+0000","last_peered":"2024-01-24T17:47:31.980080+0000","last_clean":"2024-01-24T17:47:31.980080+0000","last_became_active":"2024-01-24T17:47:31.972325+0000","last_became_peered":"2024-01-24T17:47:31.972325+0000","last_unstale":"2024-01-24T17:47:31.980080+0000","last_undegraded":"2024-01-24T17:47:31.980080+0000","last_fullsized":"2024-01-24T17:47:31.980080+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:06:24.369505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.975911+0000","last_change":"2024-01-24T17:47:31.484099+0000","last_active":"2024-01-24T17:47:31.975911+0000","last_peered":"2024-01-24T17:47:31.975911+0000","last_clean":"2024-01-24T17:47:31.975911+0000","last_became_active":"2024-01-24T17:47:31.483661+0000","last_became_peered":"2024-01-24T17:47:31.483661+0000","last_unstale":"2024-01-24T17:47:31.975911+0000","last_undegraded":"2024-01-24T17:47:31.975911+0000","last_fullsized":"2024-01-24T17:47:31.975911+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:30.230951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.976971+0000","last_change":"2024-01-24T17:47:31.971659+0000","last_active":"2024-01-24T17:47:31.976971+0000","last_peered":"2024-01-24T17:47:31.976971+0000","last_clean":"2024-01-24T17:47:31.976971+0000","last_became_active":"2024-01-24T17:47:31.971348+0000","last_became_peered":"2024-01-24T17:47:31.971348+0000","last_unstale":"2024-01-24T17:47:31.976971+0000","last_undegraded":"2024-01-24T17:47:31.976971+0000","last_fullsized":"2024-01-24T17:47:31.976971+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:26:19.054930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.947277+0000","last_change":"2024-01-24T17:47:31.480055+0000","last_active":"2024-01-24T17:47:31.947277+0000","last_peered":"2024-01-24T17:47:31.947277+0000","last_clean":"2024-01-24T17:47:31.947277+0000","last_became_active":"2024-01-24T17:47:31.479420+0000","last_became_peered":"2024-01-24T17:47:31.479420+0000","last_unstale":"2024-01-24T17:47:31.947277+0000","last_undegraded":"2024-01-24T17:47:31.947277+0000","last_fullsized":"2024-01-24T17:47:31.947277+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:16:04.683723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.980916+0000","last_change":"2024-01-24T17:47:31.973486+0000","last_active":"2024-01-24T17:47:31.980916+0000","last_peered":"2024-01-24T17:47:31.980916+0000","last_clean":"2024-01-24T17:47:31.980916+0000","last_became_active":"2024-01-24T17:47:31.973216+0000","last_became_peered":"2024-01-24T17:47:31.973216+0000","last_unstale":"2024-01-24T17:47:31.980916+0000","last_undegraded":"2024-01-24T17:47:31.980916+0000","last_fullsized":"2024-01-24T17:47:31.980916+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:22:35.200706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.962877+0000","last_change":"2024-01-24T17:47:31.476802+0000","last_active":"2024-01-24T17:47:31.962877+0000","last_peered":"2024-01-24T17:47:31.962877+0000","last_clean":"2024-01-24T17:47:31.962877+0000","last_became_active":"2024-01-24T17:47:31.475610+0000","last_became_peered":"2024-01-24T17:47:31.475610+0000","last_unstale":"2024-01-24T17:47:31.962877+0000","last_undegraded":"2024-01-24T17:47:31.962877+0000","last_fullsized":"2024-01-24T17:47:31.962877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:35:53.319043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.989926+0000","last_change":"2024-01-24T17:47:31.984081+0000","last_active":"2024-01-24T17:47:31.989926+0000","last_peered":"2024-01-24T17:47:31.989926+0000","last_clean":"2024-01-24T17:47:31.989926+0000","last_became_active":"2024-01-24T17:47:31.983740+0000","last_became_peered":"2024-01-24T17:47:31.983740+0000","last_unstale":"2024-01-24T17:47:31.989926+0000","last_undegraded":"2024-01-24T17:47:31.989926+0000","last_fullsized":"2024-01-24T17:47:31.989926+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:13:01.119477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.954107+0000","last_change":"2024-01-24T17:47:31.472320+0000","last_active":"2024-01-24T17:47:31.954107+0000","last_peered":"2024-01-24T17:47:31.954107+0000","last_clean":"2024-01-24T17:47:31.954107+0000","last_became_active":"2024-01-24T17:47:31.472035+0000","last_became_peered":"2024-01-24T17:47:31.472035+0000","last_unstale":"2024-01-24T17:47:31.954107+0000","last_undegraded":"2024-01-24T17:47:31.954107+0000","last_fullsized":"2024-01-24T17:47:31.954107+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:47:49.591723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.972806+0000","last_change":"2024-01-24T17:47:31.968866+0000","last_active":"2024-01-24T17:47:31.972806+0000","last_peered":"2024-01-24T17:47:31.972806+0000","last_clean":"2024-01-24T17:47:31.972806+0000","last_became_active":"2024-01-24T17:47:31.968532+0000","last_became_peered":"2024-01-24T17:47:31.968532+0000","last_unstale":"2024-01-24T17:47:31.972806+0000","last_undegraded":"2024-01-24T17:47:31.972806+0000","last_fullsized":"2024-01-24T17:47:31.972806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:31.428975+0000","last_clean_scrub_stamp":"2024-01-24T17:47:31.428975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:39:29.312975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.d","version":"87'1","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.981269+0000","last_change":"2024-01-24T17:47:31.470768+0000","last_active":"2024-01-24T17:47:31.981269+0000","last_peered":"2024-01-24T17:47:31.981269+0000","last_clean":"2024-01-24T17:47:31.981269+0000","last_became_active":"2024-01-24T17:47:31.470489+0000","last_became_peered":"2024-01-24T17:47:31.470489+0000","last_unstale":"2024-01-24T17:47:31.981269+0000","last_undegraded":"2024-01-24T17:47:31.981269+0000","last_fullsized":"2024-01-24T17:47:31.981269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:47:30.414210+0000","last_clean_scrub_stamp":"2024-01-24T17:47:30.414210+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:25:54.025878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":225,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-24T17:47:31.466273+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:47:31.466273+0000","last_peered":"2024-01-24T17:47:31.466273+0000","last_clean":"2024-01-24T17:47:31.466273+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:47:31.466273+0000","last_undegraded":"2024-01-24T17:47:31.466273+0000","last_fullsized":"2024-01-24T17:47:31.466273+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1073152,"data_stored":457790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.075,"15min":1.0329999999999999},"min":{"1min":0.66500000000000004,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.5269999999999999,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1020000000000001,"15min":1.0509999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.413,"5min":1.917,"15min":1.917},"last":1.718}]},{"osd":1,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.095,"15min":1.103},"min":{"1min":0.81200000000000006,"5min":0.621,"15min":0.621},"max":{"1min":1.4930000000000001,"5min":2.04,"15min":2.04},"last":1.292},{"interface":"front","average":{"1min":1.141,"5min":1.1870000000000001,"15min":1.1930000000000001},"min":{"1min":0.81200000000000006,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.454,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.944}]},{"osd":2,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1930000000000001,"15min":1.165},"min":{"1min":0.71099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6359999999999999,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.1679999999999999,"15min":1.214},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7430000000000001,"5min":2.484,"15min":2.484},"last":1.4319999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.2410000000000001,"15min":1.2509999999999999},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7230000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3,"15min":1.3160000000000001},"min":{"1min":0.88400000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.9059999999999999,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.5349999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.274,"15min":1.3129999999999999},"min":{"1min":0.86499999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.681,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":2.1499999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.4139999999999999,"15min":1.431},"min":{"1min":0.73399999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.6859999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":1.466}]},{"osd":5,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2869999999999999,"15min":1.341},"min":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9810000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.619},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.359,"15min":1.4219999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.0030000000000001,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":2.2480000000000002}]},{"osd":6,"last update":"Wed Jan 24 17:46:29 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4590000000000001,"15min":1.522},"min":{"1min":0.755,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.853,"5min":2.6960000000000002,"15min":2.6960000000000002},"last":2.4129999999999998},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.431,"15min":1.474},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.8100000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":1.923}]}]},{"osd":6,"up_from":13,"seq":55834574924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":1056768,"data_stored":450945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.081,"15min":1.04},"min":{"1min":0.748,"5min":0.623,"15min":0.61199999999999999},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.0920000000000001,"15min":1.0349999999999999},"min":{"1min":0.79100000000000004,"5min":0.69499999999999995,"15min":0.66300000000000003},"max":{"1min":1.4390000000000001,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.98799999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.113,"15min":1.022},"min":{"1min":0.78400000000000003,"5min":0.66700000000000004,"15min":0.64200000000000002},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.173,"15min":1.131},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.9129999999999998,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1759999999999999,"15min":1.1479999999999999},"min":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.63600000000000001},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1919999999999999,"15min":1.1739999999999999},"min":{"1min":0.755,"5min":0.69099999999999995,"15min":0.56599999999999995},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.0910000000000002},"last":1.093}]},{"osd":3,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.236,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.75,"5min":2.0009999999999999,"15min":2.1669999999999998},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2090000000000001,"15min":1.2450000000000001},"min":{"1min":0.95299999999999996,"5min":0.81899999999999995,"15min":0.69799999999999995},"max":{"1min":1.625,"5min":1.758,"15min":2.46},"last":1.625}]},{"osd":4,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3089999999999999,"15min":1.2549999999999999},"min":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.877,"5min":1.9179999999999999,"15min":1.988},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.753,"5min":0.73899999999999999,"15min":0.44600000000000001},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.3049999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.339,"15min":1.3069999999999999},"min":{"1min":0.77600000000000002,"5min":0.65300000000000002,"15min":0.46800000000000003},"max":{"1min":1.694,"5min":1.8879999999999999,"15min":2.2170000000000001},"last":1.284},{"interface":"front","average":{"1min":1.49,"5min":1.4419999999999999,"15min":1.4399999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.9750000000000001,"5min":2.0030000000000001,"15min":2.4849999999999999},"last":1.712}]},{"osd":7,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3480000000000001,"15min":1.365},"min":{"1min":0.91200000000000003,"5min":0.85399999999999998,"15min":0.51400000000000001},"max":{"1min":1.911,"5min":1.978,"15min":2.2050000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4470000000000001,"15min":1.4610000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.52300000000000002},"max":{"1min":1.7649999999999999,"5min":1.9019999999999999,"15min":2.6920000000000002},"last":1.3620000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1679360,"data_stored":1049686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0349999999999999,"15min":1.089},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5009999999999999,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.0369999999999999,"15min":1.0329999999999999},"min":{"1min":0.67300000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.492,"5min":1.889,"15min":1.889},"last":1.234}]},{"osd":1,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0700000000000001,"15min":1.0720000000000001},"min":{"1min":0.77200000000000002,"5min":0.622,"15min":0.622},"max":{"1min":1.504,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.171,"15min":1.2470000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.597,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.357}]},{"osd":2,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2290000000000001,"15min":1.2310000000000001},"min":{"1min":0.875,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6319999999999999,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.2649999999999999,"15min":1.3069999999999999},"min":{"1min":0.92300000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.4690000000000001,"5min":3.1349999999999998,"15min":3.1349999999999998},"last":0.94499999999999995}]},{"osd":3,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.23,"15min":1.3200000000000001},"min":{"1min":0.76400000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.5629999999999999,"5min":3.036,"15min":3.036},"last":1.448},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.2789999999999999,"15min":1.323},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":1.718,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.26}]},{"osd":4,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.409,"15min":1.4450000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.9690000000000001,"5min":2.069,"15min":2.069},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.278,"15min":1.343},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.0409999999999999,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":1.1220000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.3149999999999999,"15min":1.3480000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.8759999999999999,"5min":2.504,"15min":2.504},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.4690000000000001,"15min":1.5109999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.1059999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.548}]},{"osd":7,"last update":"Wed Jan 24 17:46:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.401,"15min":1.474},"min":{"1min":0.80600000000000005,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.1309999999999998,"5min":3.161,"15min":3.161},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.5309999999999999,"15min":1.615},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.0819999999999999,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.5980000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":1056768,"data_stored":450945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.075,"15min":1.0469999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.5840000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0880000000000001,"15min":1.0940000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.4690000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.3149999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.1140000000000001,"15min":1.0629999999999999},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":2.113},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1459999999999999,"15min":1.202},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.3360000000000001,"5min":1.849,"15min":1.849},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.1970000000000001,"15min":1.2090000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.718,"5min":2.105,"15min":2.105},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.266,"15min":1.3220000000000001},"min":{"1min":0.752,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.786,"5min":1.986,"15min":1.986},"last":1.149}]},{"osd":3,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.1879999999999999,"15min":1.1919999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.516,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.276,"15min":1.329},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5820000000000001,"5min":2.9409999999999998,"15min":2.9409999999999998},"last":1.7929999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.3,"15min":1.357},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6459999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2669999999999999,"15min":1.2949999999999999},"min":{"1min":0.65600000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.667,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.738}]},{"osd":6,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.359,"15min":1.4410000000000001},"min":{"1min":0.67800000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.917,"5min":3.0840000000000001,"15min":3.0840000000000001},"last":2.1930000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.4179999999999999,"15min":1.464},"min":{"1min":0.69499999999999995,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.74,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.518}]},{"osd":7,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.4099999999999999,"15min":1.4179999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.881,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.4990000000000001,"15min":1.587},"min":{"1min":0.79300000000000004,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.8799999999999999,"5min":4.3250000000000002,"15min":4.3250000000000002},"last":2.2719999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574927,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1081344,"data_stored":458394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.89000000000000001,"15min":0.86399999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.5580000000000001,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.96599999999999997,"15min":0.89500000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.6739999999999999,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.88600000000000001,"15min":0.93300000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.208,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.05,"15min":1.0589999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.657,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.087,"15min":1.105},"min":{"1min":0.56399999999999995,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.4790000000000001,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.103,"5min":1.1619999999999999,"15min":1.1579999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.7150000000000001,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.81699999999999995}]},{"osd":4,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.284,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.8660000000000001,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.274,"5min":1.3089999999999999,"15min":1.3400000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":2.0390000000000001,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.325}]},{"osd":5,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.387,"15min":1.421},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":2.1269999999999998,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.379,"15min":1.411},"min":{"1min":0.78400000000000003,"5min":0.746,"15min":0.746},"max":{"1min":1.9430000000000001,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":0.95199999999999996}]},{"osd":6,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5029999999999999,"15min":1.4930000000000001},"min":{"1min":0.86899999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0110000000000001,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":1.143},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.427,"15min":1.448},"min":{"1min":0.80200000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.9019999999999999,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.028}]},{"osd":7,"last update":"Wed Jan 24 17:46:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.49,"15min":1.5},"min":{"1min":0.93300000000000005,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.833,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.5149999999999999,"15min":1.516},"min":{"1min":1.075,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.0990000000000002,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.1020000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":1044480,"data_stored":444670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78900000000000003,"15min":0.79300000000000004},"min":{"1min":0.443,"5min":0.318,"15min":0.318},"max":{"1min":1.3420000000000001,"5min":1.667,"15min":1.667},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.66200000000000003,"5min":0.375,"15min":0.375},"max":{"1min":1.4370000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0109999999999999,"15min":1.0469999999999999},"min":{"1min":0.505,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.569,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.319},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.111,"15min":1.1950000000000001},"min":{"1min":0.76300000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.677,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.93300000000000005}]},{"osd":3,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94299999999999995,"15min":0.98299999999999998},"min":{"1min":0.51900000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.5089999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.129,"5min":1.135,"15min":1.1950000000000001},"min":{"1min":0.68000000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5680000000000001,"5min":2.996,"15min":2.996},"last":0.77100000000000002}]},{"osd":4,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.248,"15min":1.3100000000000001},"min":{"1min":0.78500000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.5,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.25},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.2589999999999999,"15min":1.3779999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.831,"5min":1.889,"15min":2.3839999999999999},"last":1.1819999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3009999999999999,"15min":1.389},"min":{"1min":0.873,"5min":0.73999999999999999,"15min":0.73899999999999999},"max":{"1min":1.851,"5min":1.851,"15min":2.3140000000000001},"last":0.998},{"interface":"front","average":{"1min":1.119,"5min":1.202,"15min":1.3},"min":{"1min":0.754,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.4750000000000001,"5min":1.889,"15min":2.0720000000000001},"last":0.97599999999999998}]},{"osd":6,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.304,"15min":1.3799999999999999},"min":{"1min":0.63200000000000001,"5min":0.623,"15min":0.623},"max":{"1min":1.615,"5min":1.9350000000000001,"15min":2.2679999999999998},"last":1.194},{"interface":"front","average":{"1min":1.298,"5min":1.3380000000000001,"15min":1.4490000000000001},"min":{"1min":0.95499999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.782,"5min":2.2530000000000001,"15min":3.7410000000000001},"last":1.046}]},{"osd":7,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.401,"15min":1.5449999999999999},"min":{"1min":0.81999999999999995,"5min":0.80800000000000005,"15min":0.80300000000000005},"max":{"1min":1.9670000000000001,"5min":2.1499999999999999,"15min":3.343},"last":1.117},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.367,"15min":1.48},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.7190000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.3460000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32728,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339112,"statfs":{"total":96636764160,"available":96603250688,"internally_reserved":0,"allocated":1650688,"data_stored":1041313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.876,"15min":0.86099999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1220000000000001,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90200000000000002,"15min":0.94399999999999995},"min":{"1min":0.442,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.3049999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.62}]},{"osd":2,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.89500000000000002,"15min":0.96499999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.105,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":1.052,"15min":1.1000000000000001},"min":{"1min":0.46899999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5229999999999999,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.129,"15min":1.159},"min":{"1min":0.53000000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.597,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.0980000000000001,"15min":1.212},"min":{"1min":0.495,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.6220000000000001,"5min":3.149,"15min":3.149},"last":0.68200000000000005}]},{"osd":4,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.331,"15min":1.3580000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.696,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.28,"15min":1.3180000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.5880000000000001,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.871}]},{"osd":5,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.2689999999999999,"15min":1.339},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5449999999999999,"5min":2.577,"15min":2.577},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3480000000000001,"15min":1.3720000000000001},"min":{"1min":0.89800000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0950000000000002,"5min":2.9809999999999999,"15min":2.9809999999999999},"last":0.92400000000000004}]},{"osd":6,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.345,"15min":1.4259999999999999},"min":{"1min":0.78700000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.9219999999999999,"5min":3.2850000000000001,"15min":3.2850000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.3700000000000001,"15min":1.5009999999999999},"min":{"1min":0.72599999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.6719999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.143}]},{"osd":7,"last update":"Wed Jan 24 17:46:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.413,"15min":1.46},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.0840000000000001,"5min":2.8439999999999999,"15min":2.8439999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.4430000000000001,"15min":1.5349999999999999},"min":{"1min":0.92500000000000004,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.6950000000000001,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":0.89700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":1044480,"data_stored":444670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.79100000000000004,"15min":0.81499999999999995},"min":{"1min":0.441,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.3420000000000001,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.93100000000000005,"15min":0.96999999999999997},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.57699999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.83699999999999997},"min":{"1min":0.46500000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2,"5min":1.331,"15min":1.331},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.99099999999999999,"15min":1.0820000000000001},"min":{"1min":0.50800000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.55,"5min":2.0499999999999998,"15min":2.0499999999999998},"last":0.86299999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1100000000000001,"15min":1.1799999999999999},"min":{"1min":0.73999999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.5149999999999999,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.157,"15min":1.1850000000000001},"min":{"1min":0.67900000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.427,"5min":2.407,"15min":2.407},"last":0.95499999999999996}]},{"osd":4,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.2649999999999999,"15min":1.389},"min":{"1min":0.93400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.387,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.3169999999999999,"15min":1.4550000000000001},"min":{"1min":0.91500000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.615,"5min":3.3740000000000001,"15min":3.3740000000000001},"last":1.01}]},{"osd":5,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.29,"15min":1.3819999999999999},"min":{"1min":0.90800000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.6259999999999999,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.2390000000000001,"15min":1.2589999999999999},"min":{"1min":0.85499999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.7310000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4139999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3300000000000001,"15min":1.3580000000000001},"min":{"1min":0.93700000000000006,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.95,"5min":2.335,"15min":2.335},"last":1.244},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.381,"15min":1.4490000000000001},"min":{"1min":1.0900000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.931,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3320000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:46:36 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.397,"15min":1.512},"min":{"1min":1.073,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.841,"5min":3.0230000000000001,"15min":3.0230000000000001},"last":1.202},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.46,"15min":1.579},"min":{"1min":1.002,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.907,"5min":3.1600000000000001,"15min":3.1600000000000001},"last":1.292}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:47:37.277 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-24T17:47:37.278 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-24T17:47:37.414 INFO:tasks.ceph.mds.c.smithi039.stderr:2024-01-24T17:47:37.413+0000 7fa70fc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:47:37.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.549+0000 7fbda8a00640 1 Processor -- start 2024-01-24T17:47:37.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.549+0000 7fbda8a00640 1 -- start start 2024-01-24T17:47:37.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.551+0000 7fbda8a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbda406ef50 0x7fbda406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.551+0000 7fbda8a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 0x7fbda4069de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.551+0000 7fbda8a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda40cb5a0 con 0x7fbda415cb10 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.551+0000 7fbda8a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda40cdfd0 con 0x7fbda406ef50 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.551+0000 7fbda8a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda4160700 con 0x7fbda40cdb80 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.551+0000 7fbda1e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbda406ef50 0x7fbda406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.551+0000 7fbda1e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbda406ef50 0x7fbda406b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59634/0 (socket says 172.21.15.39:59634) 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.551+0000 7fbda1e00640 1 -- 172.21.15.39:0/2918313848 learned_addr learned my addr 172.21.15.39:0/2918313848 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.552+0000 7fbda3200640 1 --2- 172.21.15.39:0/2918313848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 0x7fbda4069de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.552+0000 7fbda1e00640 1 -- 172.21.15.39:0/2918313848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 msgr2=0x7fbda4069de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.552+0000 7fbda1e00640 1 --2- 172.21.15.39:0/2918313848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 0x7fbda4069de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.552+0000 7fbda1e00640 1 -- 172.21.15.39:0/2918313848 >> v1:172.21.15.100:6789/0 conn(0x7fbda415cb10 legacy=0x7fbda406e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.552+0000 7fbda1e00640 1 -- 172.21.15.39:0/2918313848 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda41609e0 con 0x7fbda406ef50 2024-01-24T17:47:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.552+0000 7fbda3200640 1 --2- 172.21.15.39:0/2918313848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 0x7fbda4069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:37.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.553+0000 7fbda1e00640 1 --2- 172.21.15.39:0/2918313848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda406b7c0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009870 tx=0x7fbd8c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe475257ffc5f4b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:37.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.553+0000 7fbda1400640 1 -- 172.21.15.39:0/2918313848 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c031640 con 0x7fbda406ef50 2024-01-24T17:47:37.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.554+0000 7fbda1400640 1 -- 172.21.15.39:0/2918313848 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c0317a0 con 0x7fbda406ef50 2024-01-24T17:47:37.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.554+0000 7fbda1400640 1 -- 172.21.15.39:0/2918313848 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c031b30 con 0x7fbda406ef50 2024-01-24T17:47:37.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.554+0000 7fbda8a00640 1 -- 172.21.15.39:0/2918313848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 msgr2=0x7fbda406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.554+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2918313848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda406b7c0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009870 tx=0x7fbd8c02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.554+0000 7fbda8a00640 1 -- 172.21.15.39:0/2918313848 shutdown_connections 2024-01-24T17:47:37.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.554+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2918313848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda406b7c0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.554+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2918313848 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 0x7fbda4069de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.554+0000 7fbda8a00640 1 -- 172.21.15.39:0/2918313848 >> 172.21.15.39:0/2918313848 conn(0x7fbda40cf5a0 msgr2=0x7fbda40cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:37.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.555+0000 7fbda8a00640 1 -- 172.21.15.39:0/2918313848 shutdown_connections 2024-01-24T17:47:37.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.555+0000 7fbda8a00640 1 -- 172.21.15.39:0/2918313848 wait complete. 2024-01-24T17:47:37.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.556+0000 7fbda8a00640 1 Processor -- start 2024-01-24T17:47:37.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.556+0000 7fbda8a00640 1 -- start start 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.557+0000 7fbda8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda40b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.557+0000 7fbda8a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 0x7fbda415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.557+0000 7fbda8a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbda415cb10 0x7fbda415d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.557+0000 7fbda8a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda4148820 con 0x7fbda406ef50 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.557+0000 7fbda8a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda415f9d0 con 0x7fbda415cb10 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.557+0000 7fbda8a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda415fcb0 con 0x7fbda40cdb80 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda40b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda40b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59646/0 (socket says 172.21.15.39:59646) 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 -- 172.21.15.39:0/2780093833 learned_addr learned my addr 172.21.15.39:0/2780093833 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda3200640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbda415cb10 0x7fbda415d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 -- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 msgr2=0x7fbda415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 0x7fbda415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 -- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbda415cb10 msgr2=0x7fbda415d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbda415cb10 0x7fbda415d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 -- 172.21.15.39:0/2780093833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c0094d0 con 0x7fbda406ef50 2024-01-24T17:47:37.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.558+0000 7fbda2800640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda40b4820 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400c7c0 tx=0x7fbd9400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:37.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.559+0000 7fbd9aa00640 1 -- 172.21.15.39:0/2780093833 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9400aa50 con 0x7fbda406ef50 2024-01-24T17:47:37.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.559+0000 7fbd9aa00640 1 -- 172.21.15.39:0/2780093833 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9400abb0 con 0x7fbda406ef50 2024-01-24T17:47:37.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.559+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda415ff90 con 0x7fbda406ef50 2024-01-24T17:47:37.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.559+0000 7fbd9aa00640 1 -- 172.21.15.39:0/2780093833 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94012420 con 0x7fbda406ef50 2024-01-24T17:47:37.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.559+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda40c8560 con 0x7fbda406ef50 2024-01-24T17:47:37.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.560+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda406bd00 con 0x7fbda406ef50 2024-01-24T17:47:37.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.561+0000 7fbd9aa00640 1 -- 172.21.15.39:0/2780093833 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbd94012580 con 0x7fbda406ef50 2024-01-24T17:47:37.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.562+0000 7fbd9aa00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd7c06a430 0x7fbd7c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:37.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.562+0000 7fbd9aa00640 1 -- 172.21.15.39:0/2780093833 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd94088910 con 0x7fbda406ef50 2024-01-24T17:47:37.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.563+0000 7fbda1e00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd7c06a430 0x7fbd7c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:37.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.564+0000 7fbda1e00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd7c06a430 0x7fbd7c06c8f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c036000 tx=0x7fbd8c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.567+0000 7fbd9aa00640 1 -- 172.21.15.39:0/2780093833 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbd94053ef0 con 0x7fbda406ef50 2024-01-24T17:47:37.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.838+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbda404e840 con 0x7fbda406ef50 2024-01-24T17:47:37.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.846+0000 7fbd9aa00640 1 -- 172.21.15.39:0/2780093833 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7fbd940160e0 con 0x7fbda406ef50 2024-01-24T17:47:37.847 INFO:teuthology.orchestra.run.smithi039.stderr:updated caps for client.0 2024-01-24T17:47:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.848+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd7c06a430 msgr2=0x7fbd7c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.848+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd7c06a430 0x7fbd7c06c8f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c036000 tx=0x7fbd8c030000 comp rx=0 tx=0).stop 2024-01-24T17:47:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.848+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 msgr2=0x7fbda40b4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.848+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda40b4820 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400c7c0 tx=0x7fbd9400cc80 comp rx=0 tx=0).stop 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.849+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 shutdown_connections 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.849+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbda415cb10 0x7fbda415d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.849+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbda40cdb80 0x7fbda415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.849+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbd7c06a430 0x7fbd7c06c8f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.849+0000 7fbda8a00640 1 --2- 172.21.15.39:0/2780093833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbda406ef50 0x7fbda40b4820 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.849+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 >> 172.21.15.39:0/2780093833 conn(0x7fbda40cf5a0 msgr2=0x7fbda40abea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.849+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 shutdown_connections 2024-01-24T17:47:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:37.849+0000 7fbda8a00640 1 -- 172.21.15.39:0/2780093833 wait complete. 2024-01-24T17:47:37.871 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:47:38.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.088+0000 7fd2dfc00640 1 Processor -- start 2024-01-24T17:47:38.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.088+0000 7fd2dfc00640 1 -- start start 2024-01-24T17:47:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.090+0000 7fd2dfc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd2d815cf10 0x7fd2d8070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.090+0000 7fd2dfc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2d80b0720 0x7fd2d80cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.090+0000 7fd2dfc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d80ccf90 con 0x7fd2d8070980 2024-01-24T17:47:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.090+0000 7fd2dfc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d80c9bd0 con 0x7fd2d80b0720 2024-01-24T17:47:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.090+0000 7fd2dfc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d80c9eb0 con 0x7fd2d815cf10 2024-01-24T17:47:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.090+0000 7fd2ddc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2d80b0720 0x7fd2d80cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.091+0000 7fd2ddc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd2d80b0720 0x7fd2d80cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59654/0 (socket says 172.21.15.39:59654) 2024-01-24T17:47:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.091+0000 7fd2ddc00640 1 -- 172.21.15.39:0/223593750 learned_addr learned my addr 172.21.15.39:0/223593750 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.091+0000 7fd2dd200640 1 --2- 172.21.15.39:0/223593750 >> v2:172.21.15.100:3300/0 conn(0x7fd2d815cf10 0x7fd2d8070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.091+0000 7fd2ddc00640 1 -- 172.21.15.39:0/223593750 >> v2:172.21.15.100:3300/0 conn(0x7fd2d815cf10 msgr2=0x7fd2d8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.091+0000 7fd2ddc00640 1 --2- 172.21.15.39:0/223593750 >> v2:172.21.15.100:3300/0 conn(0x7fd2d815cf10 0x7fd2d8070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.091+0000 7fd2ddc00640 1 -- 172.21.15.39:0/223593750 >> v1:172.21.15.100:6789/0 conn(0x7fd2d8070980 legacy=0x7fd2d80b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.091+0000 7fd2ddc00640 1 -- 172.21.15.39:0/223593750 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d80cc940 con 0x7fd2d80b0720 2024-01-24T17:47:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.093+0000 7fd2ddc00640 1 --2- 172.21.15.39:0/223593750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d80b0720 0x7fd2d80cd420 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd2d806e420 tx=0x7fd2d002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688c4e26048dbc78 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.093+0000 7fd2d7400640 1 -- 172.21.15.39:0/223593750 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d002ca80 con 0x7fd2d80b0720 2024-01-24T17:47:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.093+0000 7fd2d7400640 1 -- 172.21.15.39:0/223593750 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d002cbe0 con 0x7fd2d80b0720 2024-01-24T17:47:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.093+0000 7fd2d7400640 1 -- 172.21.15.39:0/223593750 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00343f0 con 0x7fd2d80b0720 2024-01-24T17:47:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.094+0000 7fd2dfc00640 1 -- 172.21.15.39:0/223593750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d80b0720 msgr2=0x7fd2d80cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.094+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/223593750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d80b0720 0x7fd2d80cd420 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd2d806e420 tx=0x7fd2d002bee0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.094+0000 7fd2dfc00640 1 -- 172.21.15.39:0/223593750 shutdown_connections 2024-01-24T17:47:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.094+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/223593750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d80b0720 0x7fd2d80cd420 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.094+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/223593750 >> v2:172.21.15.100:3300/0 conn(0x7fd2d815cf10 0x7fd2d8070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.094+0000 7fd2dfc00640 1 -- 172.21.15.39:0/223593750 >> 172.21.15.39:0/223593750 conn(0x7fd2d806bf20 msgr2=0x7fd2d806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.094+0000 7fd2dfc00640 1 -- 172.21.15.39:0/223593750 shutdown_connections 2024-01-24T17:47:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.094+0000 7fd2dfc00640 1 -- 172.21.15.39:0/223593750 wait complete. 2024-01-24T17:47:38.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.096+0000 7fd2dfc00640 1 Processor -- start 2024-01-24T17:47:38.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.096+0000 7fd2dfc00640 1 -- start start 2024-01-24T17:47:38.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2dfc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2d8070980 0x7fd2d81f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:38.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2dfc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d80b0720 0x7fd2d815e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:38.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2dfc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d815cf10 0x7fd2d815e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:38.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2dfc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80cf2b0 con 0x7fd2d815cf10 2024-01-24T17:47:38.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2dfc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d815c620 con 0x7fd2d80b0720 2024-01-24T17:47:38.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2dfc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80c5cc0 con 0x7fd2d8070980 2024-01-24T17:47:38.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2ddc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d815cf10 0x7fd2d815e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:38.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2ddc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d815cf10 0x7fd2d815e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59660/0 (socket says 172.21.15.39:59660) 2024-01-24T17:47:38.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2ddc00640 1 -- 172.21.15.39:0/4243242529 learned_addr learned my addr 172.21.15.39:0/4243242529 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:38.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2d7e00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d80b0720 0x7fd2d815e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:38.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.097+0000 7fd2dd200640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2d8070980 0x7fd2d81f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2ddc00640 1 -- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2d8070980 msgr2=0x7fd2d81f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2ddc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2d8070980 0x7fd2d81f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2ddc00640 1 -- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d80b0720 msgr2=0x7fd2d815e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2ddc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d80b0720 0x7fd2d815e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2ddc00640 1 -- 172.21.15.39:0/4243242529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0009b30 con 0x7fd2d815cf10 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2dd200640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2d8070980 0x7fd2d81f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2ddc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d815cf10 0x7fd2d815e880 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0036040 tx=0x7fd2d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2d4c00640 1 -- 172.21.15.39:0/4243242529 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0009510 con 0x7fd2d815cf10 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2d4c00640 1 -- 172.21.15.39:0/4243242529 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d003e450 con 0x7fd2d815cf10 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2d4c00640 1 -- 172.21.15.39:0/4243242529 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d003e800 con 0x7fd2d815cf10 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.098+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d80c5fa0 con 0x7fd2d815cf10 2024-01-24T17:47:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.099+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d80c6560 con 0x7fd2d815cf10 2024-01-24T17:47:38.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.100+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d80c6750 con 0x7fd2d815cf10 2024-01-24T17:47:38.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.101+0000 7fd2d4c00640 1 -- 172.21.15.39:0/4243242529 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd2d002cd50 con 0x7fd2d815cf10 2024-01-24T17:47:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.102+0000 7fd2d4c00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd2b006a460 0x7fd2b006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.102+0000 7fd2d4c00640 1 -- 172.21.15.39:0/4243242529 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2d00ab370 con 0x7fd2d815cf10 2024-01-24T17:47:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.102+0000 7fd2dd200640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd2b006a460 0x7fd2b006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:38.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.103+0000 7fd2dd200640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd2b006a460 0x7fd2b006c920 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8006fd0 tx=0x7fd2c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.106+0000 7fd2d4c00640 1 -- 172.21.15.39:0/4243242529 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd2d00769f0 con 0x7fd2d815cf10 2024-01-24T17:47:38.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.342+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2d804dff0 con 0x7fd2d815cf10 2024-01-24T17:47:38.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.345+0000 7fd2d4c00640 1 -- 172.21.15.39:0/4243242529 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4736 (secure 0 0 0) 0x7fd2d007a010 con 0x7fd2d815cf10 2024-01-24T17:47:38.346 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:38.346 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/57362003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":57362003},{"type":"v1","addr":"172.21.15.100:6836","nonce":57362003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6139,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/3662299631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":3662299631},{"type":"v1","addr":"172.21.15.100:6837","nonce":3662299631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2829786836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2829786836},{"type":"v1","addr":"172.21.15.39:6833","nonce":2829786836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:47:31.936549+0000","modified":"2024-01-24T17:47:35.302809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7293":{"gid":7293,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3073867595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3073867595},{"type":"v1","addr":"172.21.15.39:6835","nonce":3073867595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-24T17:47:38.346 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 51 2024-01-24T17:47:38.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd2b006a460 msgr2=0x7fd2b006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:38.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd2b006a460 0x7fd2b006c920 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8006fd0 tx=0x7fd2c8008040 comp rx=0 tx=0).stop 2024-01-24T17:47:38.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d815cf10 msgr2=0x7fd2d815e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:38.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d815cf10 0x7fd2d815e880 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0036040 tx=0x7fd2d0004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:38.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 shutdown_connections 2024-01-24T17:47:38.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd2b006a460 0x7fd2b006c920 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2d815cf10 0x7fd2d815e880 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d80b0720 0x7fd2d815e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 --2- 172.21.15.39:0/4243242529 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2d8070980 0x7fd2d81f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:38.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.348+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 >> 172.21.15.39:0/4243242529 conn(0x7fd2d806bf20 msgr2=0x7fd2d80c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:38.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.349+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 shutdown_connections 2024-01-24T17:47:38.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:38.349+0000 7fd2dfc00640 1 -- 172.21.15.39:0/4243242529 wait complete. 2024-01-24T17:47:38.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:47:31.936549+0000', 'modified': '2024-01-24T17:47:35.302809+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7293': {'gid': 7293, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6835/3073867595', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6834', 'nonce': 3073867595}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 3073867595}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:47:38.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T17:47:38.773 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-24T17:47:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.003+0000 7f9eb0a00640 1 Processor -- start 2024-01-24T17:47:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.003+0000 7f9eb0a00640 1 -- start start 2024-01-24T17:47:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.004+0000 7f9eb0a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac15cbf0 0x7f9eac0cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.004+0000 7f9eb0a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9eac06acb0 0x7f9eac0cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.004+0000 7f9eb0a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eac0ce010 con 0x7f9eac0b1570 2024-01-24T17:47:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.004+0000 7f9eb0a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eac0ce870 con 0x7f9eac06acb0 2024-01-24T17:47:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.005+0000 7f9eb0a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac0ce2b0 con 0x7f9eac15cbf0 2024-01-24T17:47:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.005+0000 7f9ea9e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9eac06acb0 0x7f9eac0cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.005+0000 7f9ea9e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9eac06acb0 0x7f9eac0cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59662/0 (socket says 172.21.15.39:59662) 2024-01-24T17:47:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.005+0000 7f9ea9e00640 1 -- 172.21.15.39:0/1005740043 learned_addr learned my addr 172.21.15.39:0/1005740043 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.005+0000 7f9eaa800640 1 --2- 172.21.15.39:0/1005740043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac15cbf0 0x7f9eac0cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.006+0000 7f9ea9e00640 1 -- 172.21.15.39:0/1005740043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac15cbf0 msgr2=0x7f9eac0cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.006+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/1005740043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac15cbf0 0x7f9eac0cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.006+0000 7f9ea9e00640 1 -- 172.21.15.39:0/1005740043 >> v1:172.21.15.100:6789/0 conn(0x7f9eac0b1570 legacy=0x7f9eac1605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.006+0000 7f9ea9e00640 1 -- 172.21.15.39:0/1005740043 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac0ce590 con 0x7f9eac06acb0 2024-01-24T17:47:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.006+0000 7f9eaa800640 1 --2- 172.21.15.39:0/1005740043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac15cbf0 0x7f9eac0cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:47:39.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.007+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/1005740043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac06acb0 0x7f9eac0cf6d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009870 tx=0x7f9e9402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725ee827a03d0f7a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.008+0000 7f9ea9400640 1 -- 172.21.15.39:0/1005740043 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e940317c0 con 0x7f9eac06acb0 2024-01-24T17:47:39.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.008+0000 7f9ea9400640 1 -- 172.21.15.39:0/1005740043 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e94031920 con 0x7f9eac06acb0 2024-01-24T17:47:39.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.008+0000 7f9ea9400640 1 -- 172.21.15.39:0/1005740043 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94031cb0 con 0x7f9eac06acb0 2024-01-24T17:47:39.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.008+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1005740043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac06acb0 msgr2=0x7f9eac0cf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.008+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1005740043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac06acb0 0x7f9eac0cf6d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009870 tx=0x7f9e9402b860 comp rx=0 tx=0).stop 2024-01-24T17:47:39.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.009+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1005740043 shutdown_connections 2024-01-24T17:47:39.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.009+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1005740043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac06acb0 0x7f9eac0cf6d0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.009+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1005740043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac15cbf0 0x7f9eac0cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.009+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1005740043 >> 172.21.15.39:0/1005740043 conn(0x7f9eac09bad0 msgr2=0x7f9eac0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:39.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.009+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1005740043 shutdown_connections 2024-01-24T17:47:39.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.009+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1005740043 wait complete. 2024-01-24T17:47:39.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.010+0000 7f9eb0a00640 1 Processor -- start 2024-01-24T17:47:39.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.011+0000 7f9eb0a00640 1 -- start start 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9eb0a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac06acb0 0x7f9eac0ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9eb0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac0b1570 0x7f9eac160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9eb0a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9eac15cbf0 0x7f9eac0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9eb0a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac0c9940 con 0x7f9eac0b1570 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9eb0a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac160a00 con 0x7f9eac15cbf0 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9eb0a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac160ce0 con 0x7f9eac06acb0 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9ea9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac0b1570 0x7f9eac160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9ea9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac0b1570 0x7f9eac160000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59676/0 (socket says 172.21.15.39:59676) 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9ea9e00640 1 -- 172.21.15.39:0/1935702559 learned_addr learned my addr 172.21.15.39:0/1935702559 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9eaa800640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac06acb0 0x7f9eac0ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.012+0000 7f9eab200640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9eac15cbf0 0x7f9eac0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9ea9e00640 1 -- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac06acb0 msgr2=0x7f9eac0ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac06acb0 0x7f9eac0ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9ea9e00640 1 -- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9eac15cbf0 msgr2=0x7f9eac0d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9eac15cbf0 0x7f9eac0d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9ea9e00640 1 -- 172.21.15.39:0/1935702559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940094d0 con 0x7f9eac0b1570 2024-01-24T17:47:39.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9eaa800640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac06acb0 0x7f9eac0ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:47:39.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9eab200640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9eac15cbf0 0x7f9eac0d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:47:39.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac0b1570 0x7f9eac160000 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f9e9402b8a0 tx=0x7f9e940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9ea2a00640 1 -- 172.21.15.39:0/1935702559 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94032860 con 0x7f9eac0b1570 2024-01-24T17:47:39.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9ea2a00640 1 -- 172.21.15.39:0/1935702559 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e940329e0 con 0x7f9eac0b1570 2024-01-24T17:47:39.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.013+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1935702559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac0c9b90 con 0x7f9eac0b1570 2024-01-24T17:47:39.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.014+0000 7f9ea2a00640 1 -- 172.21.15.39:0/1935702559 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94032d90 con 0x7f9eac0b1570 2024-01-24T17:47:39.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.014+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1935702559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac0ca8d0 con 0x7f9eac0b1570 2024-01-24T17:47:39.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.016+0000 7f9ea2a00640 1 -- 172.21.15.39:0/1935702559 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9e9404e450 con 0x7f9eac0b1570 2024-01-24T17:47:39.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.017+0000 7f9ea2a00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e8406a430 0x7f9e8406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.017+0000 7f9ea2a00640 1 -- 172.21.15.39:0/1935702559 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e940aab00 con 0x7f9eac0b1570 2024-01-24T17:47:39.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.017+0000 7f9eaa800640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e8406a430 0x7f9e8406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.018+0000 7f9eb3fa2740 1 Processor -- start 2024-01-24T17:47:39.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.018+0000 7f9eb3fa2740 1 -- start start 2024-01-24T17:47:39.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eaa800640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e8406a430 0x7f9e8406c8f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0b3320 tx=0x7f9e9c003a50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df7d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df4d150 0x55fb4df7a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 0x55fb4df7a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb4df286c0 con 0x55fb4df4d150 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb4debd070 con 0x55fb4df50d50 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb4deba800 con 0x55fb4df7d1d0 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9ea9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df4d150 0x55fb4df7a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9ea9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df4d150 0x55fb4df7a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59682/0 (socket says 172.21.15.39:59682) 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9ea9e00640 1 -- 172.21.15.39:0/2939580321 learned_addr learned my addr 172.21.15.39:0/2939580321 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eaa800640 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df7d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9eab200640 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 0x55fb4df7a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9ea9e00640 1 -- 172.21.15.39:0/2939580321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 msgr2=0x55fb4df7d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df7d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9ea9e00640 1 -- 172.21.15.39:0/2939580321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 msgr2=0x55fb4df7a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.019+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 0x55fb4df7a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.020+0000 7f9ea9e00640 1 -- 172.21.15.39:0/2939580321 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb4de7d780 con 0x55fb4df4d150 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.020+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df4d150 0x55fb4df7a440 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f9e940a8f90 tx=0x7f9e94079ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e505875b07e06e5c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.020+0000 7f9e9b400640 1 -- 172.21.15.39:0/2939580321 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9407dbd0 con 0x55fb4df4d150 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.020+0000 7f9e9b400640 1 -- 172.21.15.39:0/2939580321 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e94081d80 con 0x55fb4df4d150 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.020+0000 7f9e9b400640 1 -- 172.21.15.39:0/2939580321 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e940764b0 con 0x55fb4df4d150 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2939580321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df4d150 msgr2=0x55fb4df7a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df4d150 0x55fb4df7a440 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f9e940a8f90 tx=0x7f9e94079ca0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2939580321 shutdown_connections 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 0x55fb4df7a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df4d150 0x55fb4df7a440 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2939580321 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df7d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2939580321 >> 172.21.15.39:0/2939580321 conn(0x55fb4df4f460 msgr2=0x55fb4df97fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2939580321 shutdown_connections 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.021+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2939580321 wait complete. 2024-01-24T17:47:39.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.022+0000 7f9eb3fa2740 1 -- start start 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.022+0000 7f9eb3fa2740 1 Processor -- start 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4df9ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df984f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df6a0d0 0x55fb4df6a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb4deb60f0 con 0x55fb4df50d50 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb4de578c0 con 0x55fb4df6a0d0 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb4de57560 con 0x55fb4df7d1d0 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eb3fa2740 10 client.0 _ll_register_callbacks cb 0x55fb4deafa30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eaa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4df9ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eaa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4df9ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59696/0 (socket says 172.21.15.39:59696) 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eaa800640 1 -- 172.21.15.39:0/597316014 learned_addr learned my addr 172.21.15.39:0/597316014 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.023+0000 7f9eab200640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df6a0d0 0x55fb4df6a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9eaa800640 1 -- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 msgr2=0x55fb4df984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9eaa800640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9eaa800640 1 -- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df6a0d0 msgr2=0x55fb4df6a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9eaa800640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df6a0d0 0x55fb4df6a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9eaa800640 1 -- 172.21.15.39:0/597316014 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac0ca8d0 con 0x55fb4df50d50 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9eab200640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df6a0d0 0x55fb4df6a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9eaa800640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4df9ab70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0169d0 tx=0x7f9e9c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192a3479df0df4bb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9e93e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9e93e00640 1 -- 172.21.15.39:0/597316014 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c016e90 con 0x55fb4df50d50 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9e93e00640 1 -- 172.21.15.39:0/597316014 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e9c014b60 con 0x55fb4df50d50 2024-01-24T17:47:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.024+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/597316014 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb4dee0640 con 0x55fb4df50d50 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9e93e00640 1 -- 172.21.15.39:0/597316014 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e9c014d20 con 0x55fb4df50d50 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9eb3fa2740 10 client.7428 fetch_fsmap learned FSMap version 51 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/597316014 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55fb4deb4770 con 0x55fb4df50d50 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9e93e00640 1 -- 172.21.15.39:0/597316014 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55fb4dee0640 con 0x55fb4df50d50 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9e93e00640 1 -- 172.21.15.39:0/597316014 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 51) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9e9c022420 con 0x55fb4df50d50 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9eb3fa2740 10 client.7428 fetch_fsmap finished waiting for FSMap version 51 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9eb3fa2740 10 client.7428 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9eb3fa2740 20 client.7428 populate_metadata read hostname 'smithi039' 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55fb4df963f0 0x55fb4df967d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9eb3fa2740 4 client.7428 mds_command: new command op to 7293 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/597316014 --> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb4dea8940 con 0x55fb4df963f0 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.025+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55fb4df963f0 0x55fb4df967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.026+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55fb4df963f0 0x55fb4df967d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbacdf79b657db8f server_cookie=38cf302e718c7683 in_seq=0 out_seq=0 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.026+0000 7f9e93e00640 10 client.7428 ms_handle_connect on v2:172.21.15.39:6834/3073867595 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9e93e00640 1 -- 172.21.15.39:0/597316014 <== mds.0 v2:172.21.15.39:6834/3073867595 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55fb4dea8940 con 0x55fb4df963f0 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9e93e00640 10 client.7428 handle_command_reply: tid=0 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 client.7428 shutdown 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/597316014 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 msgr2=0x55fb4df9ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4df9ab70 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0169d0 tx=0x7f9e9c004ab0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/597316014 shutdown_connections 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55fb4df963f0 0x55fb4df967d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df6a0d0 0x55fb4df6a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df7d1d0 0x55fb4df984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/597316014 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4df9ab70 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9e93e00640 0 client.7428 ms_handle_reset on v2:172.21.15.39:6834/3073867595 2024-01-24T17:47:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/597316014 >> 172.21.15.39:0/597316014 conn(0x55fb4df4d150 msgr2=0x55fb4df80d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:39.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/597316014 shutdown_connections 2024-01-24T17:47:39.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/597316014 wait complete. 2024-01-24T17:47:39.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.027+0000 7f9eb3fa2740 20 client.7428 trim_cache size 0 max 16384 2024-01-24T17:47:39.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.044+0000 7f9eb3fa2740 1 Processor -- start 2024-01-24T17:47:39.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.044+0000 7f9eb3fa2740 1 -- start start 2024-01-24T17:47:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 0x55fb4df9b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df9ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df7d1d0 0x55fb4df981c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb4dc1df20 con 0x55fb4df7d1d0 2024-01-24T17:47:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb4df96d20 con 0x55fb4df50d50 2024-01-24T17:47:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb4df77de0 con 0x55fb4df6a0d0 2024-01-24T17:47:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eab200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df7d1d0 0x55fb4df981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eab200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df7d1d0 0x55fb4df981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59710/0 (socket says 172.21.15.39:59710) 2024-01-24T17:47:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eab200640 1 -- 172.21.15.39:0/3382918372 learned_addr learned my addr 172.21.15.39:0/3382918372 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df9ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eaa800640 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 0x55fb4df9b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eab200640 1 -- 172.21.15.39:0/3382918372 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 msgr2=0x55fb4df9ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eab200640 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df9ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eab200640 1 -- 172.21.15.39:0/3382918372 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 msgr2=0x55fb4df9b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eab200640 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 0x55fb4df9b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.045+0000 7f9eab200640 1 -- 172.21.15.39:0/3382918372 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb4deea2f0 con 0x55fb4df7d1d0 2024-01-24T17:47:39.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.046+0000 7f9eab200640 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df7d1d0 0x55fb4df981c0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4004e40 tx=0x7f9ea4005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e12ddfc990e6bb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.046+0000 7f9e9be00640 1 -- 172.21.15.39:0/3382918372 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4011be0 con 0x55fb4df7d1d0 2024-01-24T17:47:39.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.046+0000 7f9e9be00640 1 -- 172.21.15.39:0/3382918372 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea4011d40 con 0x55fb4df7d1d0 2024-01-24T17:47:39.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.046+0000 7f9e9be00640 1 -- 172.21.15.39:0/3382918372 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40155f0 con 0x55fb4df7d1d0 2024-01-24T17:47:39.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/3382918372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df7d1d0 msgr2=0x55fb4df981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df7d1d0 0x55fb4df981c0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4004e40 tx=0x7f9ea4005300 comp rx=0 tx=0).stop 2024-01-24T17:47:39.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/3382918372 shutdown_connections 2024-01-24T17:47:39.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df7d1d0 0x55fb4df981c0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df9ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/3382918372 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df50d50 0x55fb4df9b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/3382918372 >> 172.21.15.39:0/3382918372 conn(0x55fb4df4d150 msgr2=0x55fb4df80880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/3382918372 shutdown_connections 2024-01-24T17:47:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.047+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/3382918372 wait complete. 2024-01-24T17:47:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.048+0000 7f9eb3fa2740 1 -- start start 2024-01-24T17:47:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.048+0000 7f9eb3fa2740 1 Processor -- start 2024-01-24T17:47:39.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4e001200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df99d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eb3fa2740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df7d1d0 0x55fb4dfb7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb4df97bd0 con 0x55fb4df50d50 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb4dfb8a20 con 0x55fb4df7d1d0 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eb3fa2740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb4df998e0 con 0x55fb4df6a0d0 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eb3fa2740 10 client.0 _ll_register_callbacks cb 0x55fb4df7da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eaa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4e001200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eaa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4e001200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59724/0 (socket says 172.21.15.39:59724) 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.049+0000 7f9eaa800640 1 -- 172.21.15.39:0/2936240243 learned_addr learned my addr 172.21.15.39:0/2936240243 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eab200640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df7d1d0 0x55fb4dfb7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df99d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eaa800640 1 -- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 msgr2=0x55fb4df99d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eaa800640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df99d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eaa800640 1 -- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df7d1d0 msgr2=0x55fb4dfb7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eaa800640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df7d1d0 0x55fb4dfb7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eaa800640 1 -- 172.21.15.39:0/2936240243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea4004aa0 con 0x55fb4df50d50 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df99d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eab200640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df7d1d0 0x55fb4dfb7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eaa800640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4e001200 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00a010 tx=0x7f9e9c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400ae52a166c046e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9e93e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9e93e00640 1 -- 172.21.15.39:0/2936240243 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c014d70 con 0x55fb4df50d50 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9e93e00640 1 -- 172.21.15.39:0/2936240243 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e9c03cab0 con 0x55fb4df50d50 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.050+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2936240243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e94081d80 con 0x55fb4df50d50 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9e93e00640 1 -- 172.21.15.39:0/2936240243 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e9c03cc70 con 0x55fb4df50d50 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9eb3fa2740 10 client.7434 fetch_fsmap learned FSMap version 51 2024-01-24T17:47:39.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2936240243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55fb4de56ad0 con 0x55fb4df50d50 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9e93e00640 1 -- 172.21.15.39:0/2936240243 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e94081d80 con 0x55fb4df50d50 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9e93e00640 1 -- 172.21.15.39:0/2936240243 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 51) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9e9c039020 con 0x55fb4df50d50 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9eb3fa2740 10 client.7434 fetch_fsmap finished waiting for FSMap version 51 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9eb3fa2740 10 client.7434 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9eb3fa2740 20 client.7434 populate_metadata read hostname 'smithi039' 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55fb4df80880 0x55fb4df80c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9eb3fa2740 4 client.7434 mds_command: new command op to 7293 tid=0[{"prefix": "status"}] 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2936240243 --> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fb4deaf170 con 0x55fb4df80880 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.051+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55fb4df80880 0x55fb4df80c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9ea9e00640 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55fb4df80880 0x55fb4df80c60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0269b648cb4247 server_cookie=c8e12b4eca430ebb in_seq=0 out_seq=0 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9e93e00640 10 client.7434 ms_handle_connect on v2:172.21.15.39:6834/3073867595 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9e93e00640 1 -- 172.21.15.39:0/2936240243 <== mds.0 v2:172.21.15.39:6834/3073867595 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55fb4deaf170 con 0x55fb4df80880 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9e93e00640 10 client.7434 handle_command_reply: tid=0 2024-01-24T17:47:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9eb3fa2740 1 client.7434 shutdown 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2936240243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 msgr2=0x55fb4e001200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4e001200 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00a010 tx=0x7f9e9c014000 comp rx=0 tx=0).stop 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2936240243 shutdown_connections 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55fb4df80880 0x55fb4df80c60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55fb4df7d1d0 0x55fb4dfb7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55fb4df6a0d0 0x55fb4df99d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9eb3fa2740 1 --2- 172.21.15.39:0/2936240243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb4df50d50 0x55fb4e001200 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.052+0000 7f9e93e00640 0 client.7434 ms_handle_reset on v2:172.21.15.39:6834/3073867595 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.053+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2936240243 >> 172.21.15.39:0/2936240243 conn(0x55fb4df4d150 msgr2=0x55fb4df98190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.053+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2936240243 shutdown_connections 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.053+0000 7f9eb3fa2740 1 -- 172.21.15.39:0/2936240243 wait complete. 2024-01-24T17:47:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.053+0000 7f9eb3fa2740 20 client.7434 trim_cache size 0 max 16384 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 7293, 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 7.1108854030000002, 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 51, 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 88, 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 88, 2024-01-24T17:47:39.059 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 12.17566598 2024-01-24T17:47:39.060 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.053+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e8406a430 msgr2=0x7f9e8406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.053+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e8406a430 0x7f9e8406c8f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0b3320 tx=0x7f9e9c003a50 comp rx=0 tx=0).stop 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.053+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1935702559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac0b1570 msgr2=0x7f9eac160000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.053+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac0b1570 0x7f9eac160000 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f9e9402b8a0 tx=0x7f9e940046c0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.054+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1935702559 shutdown_connections 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.054+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9eac15cbf0 0x7f9eac0d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.054+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9e8406a430 0x7f9e8406c8f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.054+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9eac0b1570 0x7f9eac160000 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.054+0000 7f9eb0a00640 1 --2- 172.21.15.39:0/1935702559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9eac06acb0 0x7f9eac0ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.054+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1935702559 >> 172.21.15.39:0/1935702559 conn(0x7f9eac09bad0 msgr2=0x7f9eac0afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.054+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1935702559 shutdown_connections 2024-01-24T17:47:39.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.054+0000 7f9eb0a00640 1 -- 172.21.15.39:0/1935702559 wait complete. 2024-01-24T17:47:39.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:47:39.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.286+0000 7fcd47e00640 1 Processor -- start 2024-01-24T17:47:39.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.286+0000 7fcd47e00640 1 -- start start 2024-01-24T17:47:39.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd47e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcd48070540 0x7fcd4815ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd47e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcd480b3e00 0x7fcd480c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd47e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd481551b0 con 0x7fcd480c8f70 2024-01-24T17:47:39.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd47e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd4815e410 con 0x7fcd48070540 2024-01-24T17:47:39.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd47e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd4815b1b0 con 0x7fcd480b3e00 2024-01-24T17:47:39.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd46a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcd48070540 0x7fcd4815ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd46a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcd48070540 0x7fcd4815ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59730/0 (socket says 172.21.15.39:59730) 2024-01-24T17:47:39.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd46a00640 1 -- 172.21.15.39:0/3856063413 learned_addr learned my addr 172.21.15.39:0/3856063413 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:39.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.288+0000 7fcd46000640 1 --2- 172.21.15.39:0/3856063413 >> v2:172.21.15.100:3300/0 conn(0x7fcd480b3e00 0x7fcd480c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.289+0000 7fcd46a00640 1 -- 172.21.15.39:0/3856063413 >> v2:172.21.15.100:3300/0 conn(0x7fcd480b3e00 msgr2=0x7fcd480c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:39.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.289+0000 7fcd46a00640 1 --2- 172.21.15.39:0/3856063413 >> v2:172.21.15.100:3300/0 conn(0x7fcd480b3e00 0x7fcd480c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.289+0000 7fcd46a00640 1 -- 172.21.15.39:0/3856063413 >> v1:172.21.15.100:6789/0 conn(0x7fcd480c8f70 legacy=0x7fcd4815e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.289+0000 7fcd46a00640 1 -- 172.21.15.39:0/3856063413 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4815b480 con 0x7fcd48070540 2024-01-24T17:47:39.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.291+0000 7fcd46a00640 1 --2- 172.21.15.39:0/3856063413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd4815ac70 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009860 tx=0x7fcd3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e944f861b90df57 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.291+0000 7fcd45600640 1 -- 172.21.15.39:0/3856063413 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38030840 con 0x7fcd48070540 2024-01-24T17:47:39.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.291+0000 7fcd45600640 1 -- 172.21.15.39:0/3856063413 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd380309a0 con 0x7fcd48070540 2024-01-24T17:47:39.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.291+0000 7fcd45600640 1 -- 172.21.15.39:0/3856063413 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38030d30 con 0x7fcd48070540 2024-01-24T17:47:39.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.292+0000 7fcd47e00640 1 -- 172.21.15.39:0/3856063413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 msgr2=0x7fcd4815ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.292+0000 7fcd47e00640 1 --2- 172.21.15.39:0/3856063413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd4815ac70 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009860 tx=0x7fcd3802b980 comp rx=0 tx=0).stop 2024-01-24T17:47:39.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.292+0000 7fcd47e00640 1 -- 172.21.15.39:0/3856063413 shutdown_connections 2024-01-24T17:47:39.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.292+0000 7fcd47e00640 1 --2- 172.21.15.39:0/3856063413 >> v2:172.21.15.100:3300/0 conn(0x7fcd480b3e00 0x7fcd480c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.292+0000 7fcd47e00640 1 --2- 172.21.15.39:0/3856063413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd4815ac70 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.292+0000 7fcd47e00640 1 -- 172.21.15.39:0/3856063413 >> 172.21.15.39:0/3856063413 conn(0x7fcd4806a340 msgr2=0x7fcd480cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:39.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.292+0000 7fcd47e00640 1 -- 172.21.15.39:0/3856063413 shutdown_connections 2024-01-24T17:47:39.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.292+0000 7fcd47e00640 1 -- 172.21.15.39:0/3856063413 wait complete. 2024-01-24T17:47:39.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.294+0000 7fcd47e00640 1 Processor -- start 2024-01-24T17:47:39.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.295+0000 7fcd47e00640 1 -- start start 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd480b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd480b3e00 0x7fcd48148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd47e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd480c8f70 0x7fcd48148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd47e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480b2100 con 0x7fcd48070540 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd47e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480b2520 con 0x7fcd480c8f70 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd47e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480b1480 con 0x7fcd480b3e00 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd46a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd480b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd46a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd480b0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59746/0 (socket says 172.21.15.39:59746) 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd46a00640 1 -- 172.21.15.39:0/537439234 learned_addr learned my addr 172.21.15.39:0/537439234 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd46a00640 1 -- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd480b3e00 msgr2=0x7fcd48148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd46a00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd480b3e00 0x7fcd48148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd46a00640 1 -- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd480c8f70 msgr2=0x7fcd48148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd46a00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd480c8f70 0x7fcd48148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.296+0000 7fcd46a00640 1 -- 172.21.15.39:0/537439234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd38009470 con 0x7fcd48070540 2024-01-24T17:47:39.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.297+0000 7fcd46a00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd480b0ba0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fcd38000c00 tx=0x7fcd38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.297+0000 7fcd3ea00640 1 -- 172.21.15.39:0/537439234 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38039c90 con 0x7fcd48070540 2024-01-24T17:47:39.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.297+0000 7fcd3ea00640 1 -- 172.21.15.39:0/537439234 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd38039df0 con 0x7fcd48070540 2024-01-24T17:47:39.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.297+0000 7fcd3ea00640 1 -- 172.21.15.39:0/537439234 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38043640 con 0x7fcd48070540 2024-01-24T17:47:39.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.297+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd480b1760 con 0x7fcd48070540 2024-01-24T17:47:39.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.297+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd480b1d20 con 0x7fcd48070540 2024-01-24T17:47:39.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.298+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd480b2240 con 0x7fcd48070540 2024-01-24T17:47:39.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.299+0000 7fcd3ea00640 1 -- 172.21.15.39:0/537439234 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcd380437a0 con 0x7fcd48070540 2024-01-24T17:47:39.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.300+0000 7fcd3ea00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcd2006a430 0x7fcd2006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:39.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.300+0000 7fcd3ea00640 1 -- 172.21.15.39:0/537439234 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd3803e070 con 0x7fcd48070540 2024-01-24T17:47:39.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.301+0000 7fcd46000640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcd2006a430 0x7fcd2006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:39.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.302+0000 7fcd46000640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcd2006a430 0x7fcd2006c8f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fcd480b42b0 tx=0x7fcd300095a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:39.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.307+0000 7fcd3ea00640 1 -- 172.21.15.39:0/537439234 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcd38076f70 con 0x7fcd48070540 2024-01-24T17:47:39.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.544+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcd4804dff0 con 0x7fcd48070540 2024-01-24T17:47:39.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.544+0000 7fcd3ea00640 1 -- 172.21.15.39:0/537439234 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcd3807a590 con 0x7fcd48070540 2024-01-24T17:47:39.545 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcd2006a430 msgr2=0x7fcd2006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcd2006a430 0x7fcd2006c8f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fcd480b42b0 tx=0x7fcd300095a0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 msgr2=0x7fcd480b0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd480b0ba0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fcd38000c00 tx=0x7fcd38004730 comp rx=0 tx=0).stop 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 shutdown_connections 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcd480c8f70 0x7fcd48148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcd480b3e00 0x7fcd48148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcd2006a430 0x7fcd2006c8f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 --2- 172.21.15.39:0/537439234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd48070540 0x7fcd480b0ba0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:39.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 >> 172.21.15.39:0/537439234 conn(0x7fcd4806a340 msgr2=0x7fcd4815cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:39.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 shutdown_connections 2024-01-24T17:47:39.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:39.547+0000 7fcd47e00640 1 -- 172.21.15.39:0/537439234 wait complete. 2024-01-24T17:47:39.569 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:47:39.569 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:47:39.569 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:47:39.569 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:47:39.569 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:47:39.569 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:47:39.570 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:47:39.570 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:47:39.570 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:47:39.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:39.570 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:47:39.623 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:47:39.624 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42272sec preferred_lft 42272sec 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:39.625 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:39.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:39.627 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:47:39.718 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:47:39.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:39.723 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:47:39.780 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:47:39.780 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:47:39.780 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:47:39.780 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:47:39.780 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:47:39.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:39.781 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:47:39.781 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:47:39.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:47:39.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:47:39.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:47:39.781 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:47:39.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:39.968 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:47:40.022 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:47:40.022 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:47:40.023 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:47:40.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:40.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:47:40.113 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:47:40.113 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:47:40.113 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:47:40.169 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:47:40.170 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:47:40.170 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:47:40.221 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:47:40.318 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:47:40.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:47:40.413 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:47:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:47:40.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:40.418 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:47:40.474 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:47:40.474 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:47:40.481 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:47:40.563 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:47:40.563 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:47:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:47:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:47:40.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:40.649 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:47:40.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:47:40.674+0000 7f9535e43480 -1 init, newargv = 0x5646725a61d0 newargc=15 2024-01-24T17:47:40.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[66933]: starting ceph client 2024-01-24T17:47:40.684 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[66933]: starting fuse 2024-01-24T17:47:40.695 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:47:40.695 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:47:40.696 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:47:40.697 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:47:40.698 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:47:40.698 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:47:40.698 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:47:40.698 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:47:40.901 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.66933.asok 2024-01-24T17:47:41.082 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.66933.asok 2024-01-24T17:47:41.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:41.083 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.66933.asok status 2024-01-24T17:47:41.330 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:47:41.330 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:47:41.330 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "66933", 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:47:41.331 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 7446, 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 7446 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 840203626 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:47:41.332 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 840203626 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.7446 192.168.0.1:0/840203626", 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/840203626", 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 51, 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 88, 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:47:41.333 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:47:41.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:41.346 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:47:41.346 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:47:41.402 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:47:41.403 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:47:41.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:41.403 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:47:41.502 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:47:41.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.721+0000 7f63dfe00640 1 Processor -- start 2024-01-24T17:47:41.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.721+0000 7f63dfe00640 1 -- start start 2024-01-24T17:47:41.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63dfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63e0071e10 0x7f63e006fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:41.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e00b0570 0x7f63e00cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:41.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63dfe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e01658b0 con 0x7f63e00c1500 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63dfe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e0161250 con 0x7f63e0071e10 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63dfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0164ca0 con 0x7f63e00b0570 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63dea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63e0071e10 0x7f63e006fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63d7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f63e00c1500 0x7f63e00afe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54010/0 (socket says 172.21.15.39:54010) 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63dea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63e0071e10 0x7f63e006fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59752/0 (socket says 172.21.15.39:59752) 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.723+0000 7f63d7e00640 1 -- 172.21.15.39:0/2518888440 learned_addr learned my addr 172.21.15.39:0/2518888440 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.724+0000 7f63dea00640 1 -- 172.21.15.39:0/2518888440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e00b0570 msgr2=0x7f63e00cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.724+0000 7f63dea00640 1 --2- 172.21.15.39:0/2518888440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e00b0570 0x7f63e00cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.724+0000 7f63dea00640 1 -- 172.21.15.39:0/2518888440 >> v1:172.21.15.39:6789/0 conn(0x7f63e00c1500 legacy=0x7f63e00afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:41.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.724+0000 7f63dea00640 1 -- 172.21.15.39:0/2518888440 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e0164f80 con 0x7f63e0071e10 2024-01-24T17:47:41.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.725+0000 7f63dea00640 1 --2- 172.21.15.39:0/2518888440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e0071e10 0x7f63e006fd70 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009790 tx=0x7f63d002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f375f169bac0c63 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:41.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.725+0000 7f63de000640 1 -- 172.21.15.39:0/2518888440 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0031780 con 0x7f63e0071e10 2024-01-24T17:47:41.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63de000640 1 -- 172.21.15.39:0/2518888440 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d00318e0 con 0x7f63e0071e10 2024-01-24T17:47:41.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63de000640 1 -- 172.21.15.39:0/2518888440 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0031c70 con 0x7f63e0071e10 2024-01-24T17:47:41.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63dfe00640 1 -- 172.21.15.39:0/2518888440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e0071e10 msgr2=0x7f63e006fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:41.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63dfe00640 1 --2- 172.21.15.39:0/2518888440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e0071e10 0x7f63e006fd70 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009790 tx=0x7f63d002b820 comp rx=0 tx=0).stop 2024-01-24T17:47:41.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63dfe00640 1 -- 172.21.15.39:0/2518888440 shutdown_connections 2024-01-24T17:47:41.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63dfe00640 1 --2- 172.21.15.39:0/2518888440 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e00b0570 0x7f63e00cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63dfe00640 1 --2- 172.21.15.39:0/2518888440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e0071e10 0x7f63e006fd70 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63dfe00640 1 -- 172.21.15.39:0/2518888440 >> 172.21.15.39:0/2518888440 conn(0x7f63e006a340 msgr2=0x7f63e00cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:41.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63dfe00640 1 -- 172.21.15.39:0/2518888440 shutdown_connections 2024-01-24T17:47:41.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.726+0000 7f63dfe00640 1 -- 172.21.15.39:0/2518888440 wait complete. 2024-01-24T17:47:41.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.728+0000 7f63dfe00640 1 Processor -- start 2024-01-24T17:47:41.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.728+0000 7f63dfe00640 1 -- start start 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.729+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e0071e10 0x7f63e00c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.729+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e00b0570 0x7f63e014d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.729+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e00c1500 0x7f63e00c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.729+0000 7f63dfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c19a0 con 0x7f63e00c1500 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.729+0000 7f63dfe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c1f60 con 0x7f63e00b0570 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.729+0000 7f63dfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e014ddb0 con 0x7f63e0071e10 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e00c1500 0x7f63e00c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63df400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e00c1500 0x7f63e00c54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59760/0 (socket says 172.21.15.39:59760) 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63df400640 1 -- 172.21.15.39:0/1434108816 learned_addr learned my addr 172.21.15.39:0/1434108816 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63dea00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e0071e10 0x7f63e00c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63d7e00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e00b0570 0x7f63e014d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:41.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63df400640 1 -- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e0071e10 msgr2=0x7f63e00c23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:41.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63df400640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e0071e10 0x7f63e00c23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63df400640 1 -- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e00b0570 msgr2=0x7f63e014d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:41.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63df400640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e00b0570 0x7f63e014d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.730+0000 7f63df400640 1 -- 172.21.15.39:0/1434108816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00093a0 con 0x7f63e00c1500 2024-01-24T17:47:41.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.731+0000 7f63df400640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e00c1500 0x7f63e00c54b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f63d800b9d0 tx=0x7f63d800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:41.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.731+0000 7f63d6a00640 1 -- 172.21.15.39:0/1434108816 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d800cbd0 con 0x7f63e00c1500 2024-01-24T17:47:41.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.731+0000 7f63d6a00640 1 -- 172.21.15.39:0/1434108816 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d800cd50 con 0x7f63e00c1500 2024-01-24T17:47:41.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.731+0000 7f63d6a00640 1 -- 172.21.15.39:0/1434108816 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8014680 con 0x7f63e00c1500 2024-01-24T17:47:41.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.731+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e00c59f0 con 0x7f63e00c1500 2024-01-24T17:47:41.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.731+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e00c5f70 con 0x7f63e00c1500 2024-01-24T17:47:41.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.732+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e00c20f0 con 0x7f63e00c1500 2024-01-24T17:47:41.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.733+0000 7f63d6a00640 1 -- 172.21.15.39:0/1434108816 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f63d801d3b0 con 0x7f63e00c1500 2024-01-24T17:47:41.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.734+0000 7f63d6a00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63b806a430 0x7f63b806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:41.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.734+0000 7f63d6a00640 1 -- 172.21.15.39:0/1434108816 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63d80888e0 con 0x7f63e00c1500 2024-01-24T17:47:41.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.735+0000 7f63dea00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63b806a430 0x7f63b806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:41.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.736+0000 7f63dea00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63b806a430 0x7f63b806c8f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f63d0036000 tx=0x7f63d003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:41.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.739+0000 7f63d6a00640 1 -- 172.21.15.39:0/1434108816 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f63d8053ec0 con 0x7f63e00c1500 2024-01-24T17:47:41.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.972+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f63e004dff0 con 0x7f63e00c1500 2024-01-24T17:47:41.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.973+0000 7f63d6a00640 1 -- 172.21.15.39:0/1434108816 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) v1 ==== 53+0+73 (secure 0 0 0) 0x7f63d8019020 con 0x7f63e00c1500 2024-01-24T17:47:41.974 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:47:41.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63b806a430 msgr2=0x7f63b806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63b806a430 0x7f63b806c8f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f63d0036000 tx=0x7f63d003d040 comp rx=0 tx=0).stop 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e00c1500 msgr2=0x7f63e00c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e00c1500 0x7f63e00c54b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f63d800b9d0 tx=0x7f63d800be90 comp rx=0 tx=0).stop 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 shutdown_connections 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f63b806a430 0x7f63b806c8f0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63e00c1500 0x7f63e00c54b0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f63e00b0570 0x7f63e014d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 --2- 172.21.15.39:0/1434108816 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f63e0071e10 0x7f63e00c23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 >> 172.21.15.39:0/1434108816 conn(0x7f63e006a340 msgr2=0x7f63e00ce7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 shutdown_connections 2024-01-24T17:47:41.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:41.976+0000 7f63dfe00640 1 -- 172.21.15.39:0/1434108816 wait complete. 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:47:41.995 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:47:41.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:41.996 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:47:42.052 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:47:42.052 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:47:42.052 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:47:42.052 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:42.053 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42267sec preferred_lft 42267sec 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:47:42.054 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:47:42.055 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:47:42.055 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:47:42.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:42.056 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:47:42.149 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:47:42.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:42.153 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:47:42.203 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:47:42.203 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:47:42.203 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:47:42.204 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:47:42.204 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:47:42.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:42.204 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:47:42.204 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:47:42.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:47:42.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:47:42.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:47:42.204 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:47:42.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:42.383 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:47:42.430 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:47:42.431 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:47:42.431 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:47:42.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:42.432 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:47:42.516 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-24T17:47:42.516 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:47:42.516 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:47:42.566 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:47:42.566 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:47:42.567 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:47:42.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:47:42.708 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:47:42.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:47:42.797 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:47:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:47:42.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:42.801 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:47:42.855 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:47:42.855 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-24T17:47:42.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:47:42.939 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:47:42.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:47:43.011 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:47:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:47:43.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:43.013 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:47:43.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:47:43.049+0000 7f892a1f6480 -1 init, newargv = 0x55b26ec5e150 newargc=15 2024-01-24T17:47:43.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[47918]: starting ceph client 2024-01-24T17:47:43.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[47918]: starting fuse 2024-01-24T17:47:44.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:47:44.256 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:47:44.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:47:44.348 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:47:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:47:44.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:44.352 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:47:44.403 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:47:44.403 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:47:44.404 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2024-01-24T17:47:44.405 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:47:44.584 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.47918.asok 2024-01-24T17:47:44.765 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.47918.asok 2024-01-24T17:47:44.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:44.765 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.47918.asok status 2024-01-24T17:47:45.002 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:47:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:47:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:47:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:47:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2024-01-24T17:47:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:47:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:47:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "47918", 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 6244, 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6244 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:47:45.003 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3072915373 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 3072915373 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.6244 192.168.0.1:0/3072915373", 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/3072915373", 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 51, 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 88, 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:47:45.004 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:47:45.005 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:47:45.005 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:47:45.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:45.016 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:47:45.016 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:47:45.069 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:47:45.069 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:47:45.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:45.070 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:47:45.162 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:47:45.163 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:45.225 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:45.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.444+0000 7f42fca00640 1 Processor -- start 2024-01-24T17:47:45.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.444+0000 7f42fca00640 1 -- start start 2024-01-24T17:47:45.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.445+0000 7f42fca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f8070540 0x7f42f815ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.445+0000 7f42fca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f42f80b3e00 0x7f42f80c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.446+0000 7f42fca00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f81551b0 con 0x7f42f80c8f70 2024-01-24T17:47:45.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.446+0000 7f42fca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f815e410 con 0x7f42f80b3e00 2024-01-24T17:47:45.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.446+0000 7f42fca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f815b1b0 con 0x7f42f8070540 2024-01-24T17:47:45.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.446+0000 7f42f5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f42f80b3e00 0x7f42f80c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.446+0000 7f42f5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f42f80b3e00 0x7f42f80c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46498/0 (socket says 172.21.15.39:46498) 2024-01-24T17:47:45.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.446+0000 7f42f5e00640 1 -- 172.21.15.39:0/292915938 learned_addr learned my addr 172.21.15.39:0/292915938 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:45.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.446+0000 7f42f6800640 1 --2- 172.21.15.39:0/292915938 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f8070540 0x7f42f815ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.447+0000 7f42f5e00640 1 -- 172.21.15.39:0/292915938 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f8070540 msgr2=0x7f42f815ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:45.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.447+0000 7f42f5e00640 1 --2- 172.21.15.39:0/292915938 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f8070540 0x7f42f815ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.447+0000 7f42f5e00640 1 -- 172.21.15.39:0/292915938 >> v1:172.21.15.100:6789/0 conn(0x7f42f80c8f70 legacy=0x7f42f815e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.447+0000 7f42f5e00640 1 -- 172.21.15.39:0/292915938 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f815b480 con 0x7f42f80b3e00 2024-01-24T17:47:45.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.448+0000 7f42f5e00640 1 --2- 172.21.15.39:0/292915938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f80b3e00 0x7f42f80c8a30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009870 tx=0x7f42e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29f511b28910737 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:45.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.448+0000 7f42f5400640 1 -- 172.21.15.39:0/292915938 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0030840 con 0x7f42f80b3e00 2024-01-24T17:47:45.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.448+0000 7f42f5400640 1 -- 172.21.15.39:0/292915938 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e00309a0 con 0x7f42f80b3e00 2024-01-24T17:47:45.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.448+0000 7f42f5400640 1 -- 172.21.15.39:0/292915938 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0030d30 con 0x7f42f80b3e00 2024-01-24T17:47:45.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.449+0000 7f42fca00640 1 -- 172.21.15.39:0/292915938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f80b3e00 msgr2=0x7f42f80c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.449+0000 7f42fca00640 1 --2- 172.21.15.39:0/292915938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f80b3e00 0x7f42f80c8a30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009870 tx=0x7f42e002b980 comp rx=0 tx=0).stop 2024-01-24T17:47:45.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.449+0000 7f42fca00640 1 -- 172.21.15.39:0/292915938 shutdown_connections 2024-01-24T17:47:45.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.449+0000 7f42fca00640 1 --2- 172.21.15.39:0/292915938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f80b3e00 0x7f42f80c8a30 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.449+0000 7f42fca00640 1 --2- 172.21.15.39:0/292915938 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f8070540 0x7f42f815ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.449+0000 7f42fca00640 1 -- 172.21.15.39:0/292915938 >> 172.21.15.39:0/292915938 conn(0x7f42f806a340 msgr2=0x7f42f80cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:45.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.449+0000 7f42fca00640 1 -- 172.21.15.39:0/292915938 shutdown_connections 2024-01-24T17:47:45.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.449+0000 7f42fca00640 1 -- 172.21.15.39:0/292915938 wait complete. 2024-01-24T17:47:45.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.451+0000 7f42fca00640 1 Processor -- start 2024-01-24T17:47:45.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.451+0000 7f42fca00640 1 -- start start 2024-01-24T17:47:45.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.452+0000 7f42fca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f8070540 0x7f42f814f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.452+0000 7f42fca00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42f80b3e00 0x7f42f81f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.452+0000 7f42fca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f80c8f70 0x7f42f80cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.452+0000 7f42fca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8153230 con 0x7f42f8070540 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.452+0000 7f42fca00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f81f2730 con 0x7f42f80b3e00 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.452+0000 7f42fca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f81f67f0 con 0x7f42f80c8f70 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.452+0000 7f42f6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f8070540 0x7f42f814f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f8070540 0x7f42f814f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46508/0 (socket says 172.21.15.39:46508) 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f6800640 1 -- 172.21.15.39:0/4049404297 learned_addr learned my addr 172.21.15.39:0/4049404297 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f7200640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f80c8f70 0x7f42f80cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f5e00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42f80b3e00 0x7f42f81f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f6800640 1 -- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f80c8f70 msgr2=0x7f42f80cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f6800640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f80c8f70 0x7f42f80cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f6800640 1 -- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42f80b3e00 msgr2=0x7f42f81f5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f6800640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42f80b3e00 0x7f42f81f5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f6800640 1 -- 172.21.15.39:0/4049404297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e00094d0 con 0x7f42f8070540 2024-01-24T17:47:45.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.453+0000 7f42f6800640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f8070540 0x7f42f814f210 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f42e800d800 tx=0x7f42e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:45.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.454+0000 7f42eea00640 1 -- 172.21.15.39:0/4049404297 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e800b6c0 con 0x7f42f8070540 2024-01-24T17:47:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.454+0000 7f42eea00640 1 -- 172.21.15.39:0/4049404297 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e800b820 con 0x7f42f8070540 2024-01-24T17:47:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.454+0000 7f42eea00640 1 -- 172.21.15.39:0/4049404297 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e800bbd0 con 0x7f42f8070540 2024-01-24T17:47:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.454+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f81f6ad0 con 0x7f42f8070540 2024-01-24T17:47:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.454+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f8152ec0 con 0x7f42f8070540 2024-01-24T17:47:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.455+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f804dff0 con 0x7f42f8070540 2024-01-24T17:47:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.456+0000 7f42eea00640 1 -- 172.21.15.39:0/4049404297 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f42e800bd30 con 0x7f42f8070540 2024-01-24T17:47:45.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.457+0000 7f42eea00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42d006a460 0x7f42d006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.457+0000 7f42eea00640 1 -- 172.21.15.39:0/4049404297 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42e8088e30 con 0x7f42f8070540 2024-01-24T17:47:45.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.457+0000 7f42f5e00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42d006a460 0x7f42d006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.459+0000 7f42f5e00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42d006a460 0x7f42d006c920 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f42e0036000 tx=0x7f42e0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:45.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.461+0000 7f42eea00640 1 -- 172.21.15.39:0/4049404297 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f42e8054410 con 0x7f42f8070540 2024-01-24T17:47:45.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.655+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42f80b0770 con 0x7f42f8070540 2024-01-24T17:47:45.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.658+0000 7f42eea00640 1 -- 172.21.15.39:0/4049404297 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16104 (secure 0 0 0) 0x7f42e8057a30 con 0x7f42f8070540 2024-01-24T17:47:45.659 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:45.660 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:32.941651+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42d006a460 msgr2=0x7f42d006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42d006a460 0x7f42d006c920 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f42e0036000 tx=0x7f42e0008030 comp rx=0 tx=0).stop 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f8070540 msgr2=0x7f42f814f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f8070540 0x7f42f814f210 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f42e800d800 tx=0x7f42e800dcc0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 shutdown_connections 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42f80c8f70 0x7f42f80cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f42f80b3e00 0x7f42f81f5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f42d006a460 0x7f42d006c920 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 --2- 172.21.15.39:0/4049404297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42f8070540 0x7f42f814f210 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 >> 172.21.15.39:0/4049404297 conn(0x7f42f806a340 msgr2=0x7f42f815a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:45.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 shutdown_connections 2024-01-24T17:47:45.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.661+0000 7f42fca00640 1 -- 172.21.15.39:0/4049404297 wait complete. 2024-01-24T17:47:45.680 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:47:30.315286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:47:30.960101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:47:45.680 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:47:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.899+0000 7fdb72c00640 1 Processor -- start 2024-01-24T17:47:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.900+0000 7fdb72c00640 1 -- start start 2024-01-24T17:47:45.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.901+0000 7fdb72c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c070460 0x7fdb6c0b1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.901+0000 7fdb72c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdb6c15ac70 0x7fdb6c0b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.901+0000 7fdb72c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c155170 con 0x7fdb6c0ce8c0 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.901+0000 7fdb72c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c0b12b0 con 0x7fdb6c15ac70 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.901+0000 7fdb72c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c15b1e0 con 0x7fdb6c070460 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb72200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fdb6c15ac70 0x7fdb6c0b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb71800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fdb6c0ce8c0 0x7fdb6c06fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:43322/0 (socket says 172.21.15.39:43322) 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb71800640 1 -- 172.21.15.39:0/1511913256 learned_addr learned my addr 172.21.15.39:0/1511913256 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb70e00640 1 --2- 172.21.15.39:0/1511913256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c070460 0x7fdb6c0b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb72200640 1 -- 172.21.15.39:0/1511913256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c070460 msgr2=0x7fdb6c0b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb72200640 1 --2- 172.21.15.39:0/1511913256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c070460 0x7fdb6c0b1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb72200640 1 -- 172.21.15.39:0/1511913256 >> v1:172.21.15.39:6789/0 conn(0x7fdb6c0ce8c0 legacy=0x7fdb6c06fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb72200640 1 -- 172.21.15.39:0/1511913256 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c15b4c0 con 0x7fdb6c15ac70 2024-01-24T17:47:45.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.902+0000 7fdb6be00640 1 -- 172.21.15.39:0/1511913256 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856649917 0 0) 0x7fdb6c155170 con 0x7fdb6c0ce8c0 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.903+0000 7fdb72200640 1 --2- 172.21.15.39:0/1511913256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c15ac70 0x7fdb6c0b1c40 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009d90 tx=0x7fdb6402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cbdb59a82d3186 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb6be00640 1 -- 172.21.15.39:0/1511913256 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6402cc50 con 0x7fdb6c15ac70 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb6be00640 1 -- 172.21.15.39:0/1511913256 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6402cdb0 con 0x7fdb6c15ac70 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb6be00640 1 -- 172.21.15.39:0/1511913256 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb64034660 con 0x7fdb6c15ac70 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb72c00640 1 -- 172.21.15.39:0/1511913256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c15ac70 msgr2=0x7fdb6c0b1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb72c00640 1 --2- 172.21.15.39:0/1511913256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c15ac70 0x7fdb6c0b1c40 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009d90 tx=0x7fdb6402bea0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb72c00640 1 -- 172.21.15.39:0/1511913256 shutdown_connections 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb72c00640 1 --2- 172.21.15.39:0/1511913256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c15ac70 0x7fdb6c0b1c40 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb72c00640 1 --2- 172.21.15.39:0/1511913256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c070460 0x7fdb6c0b1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb72c00640 1 -- 172.21.15.39:0/1511913256 >> 172.21.15.39:0/1511913256 conn(0x7fdb6c06a200 msgr2=0x7fdb6c0affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb72c00640 1 -- 172.21.15.39:0/1511913256 shutdown_connections 2024-01-24T17:47:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.904+0000 7fdb72c00640 1 -- 172.21.15.39:0/1511913256 wait complete. 2024-01-24T17:47:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.906+0000 7fdb72c00640 1 Processor -- start 2024-01-24T17:47:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.906+0000 7fdb72c00640 1 -- start start 2024-01-24T17:47:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.907+0000 7fdb72c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c070460 0x7fdb6c1f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.907+0000 7fdb72c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c0ce8c0 0x7fdb6c1f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.907+0000 7fdb72c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6c15ac70 0x7fdb6c0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.907+0000 7fdb72c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0ccef0 con 0x7fdb6c070460 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.907+0000 7fdb72c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c1f1f60 con 0x7fdb6c15ac70 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.907+0000 7fdb72c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0c7030 con 0x7fdb6c0ce8c0 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c070460 0x7fdb6c1f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c070460 0x7fdb6c1f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46536/0 (socket says 172.21.15.39:46536) 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 -- 172.21.15.39:0/328125083 learned_addr learned my addr 172.21.15.39:0/328125083 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb72200640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6c15ac70 0x7fdb6c0c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb70e00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c0ce8c0 0x7fdb6c1f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 -- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c0ce8c0 msgr2=0x7fdb6c1f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c0ce8c0 0x7fdb6c1f3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 -- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6c15ac70 msgr2=0x7fdb6c0c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6c15ac70 0x7fdb6c0c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 -- 172.21.15.39:0/328125083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64009a70 con 0x7fdb6c070460 2024-01-24T17:47:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.908+0000 7fdb71800640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c070460 0x7fdb6c1f1a20 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00c4d0 tx=0x7fdb5c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.909+0000 7fdb69600640 1 -- 172.21.15.39:0/328125083 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c0074e0 con 0x7fdb6c070460 2024-01-24T17:47:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.909+0000 7fdb69600640 1 -- 172.21.15.39:0/328125083 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c007660 con 0x7fdb6c070460 2024-01-24T17:47:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.909+0000 7fdb69600640 1 -- 172.21.15.39:0/328125083 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c004a20 con 0x7fdb6c070460 2024-01-24T17:47:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.909+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c0cc370 con 0x7fdb6c070460 2024-01-24T17:47:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.909+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c0cc930 con 0x7fdb6c070460 2024-01-24T17:47:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.910+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c0738c0 con 0x7fdb6c070460 2024-01-24T17:47:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.911+0000 7fdb69600640 1 -- 172.21.15.39:0/328125083 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdb5c004b80 con 0x7fdb6c070460 2024-01-24T17:47:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.912+0000 7fdb69600640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb4406a430 0x7fdb4406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.912+0000 7fdb69600640 1 -- 172.21.15.39:0/328125083 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb5c087b50 con 0x7fdb6c070460 2024-01-24T17:47:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.913+0000 7fdb70e00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb4406a430 0x7fdb4406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:45.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.914+0000 7fdb70e00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb4406a430 0x7fdb4406c8f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0adbe0 tx=0x7fdb54008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:45.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:45.916+0000 7fdb69600640 1 -- 172.21.15.39:0/328125083 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdb5c053130 con 0x7fdb6c070460 2024-01-24T17:47:46.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.114+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb6c04e840 con 0x7fdb6c070460 2024-01-24T17:47:46.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.115+0000 7fdb69600640 1 -- 172.21.15.39:0/328125083 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb5c018020 con 0x7fdb6c070460 2024-01-24T17:47:46.115 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:47:46.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb4406a430 msgr2=0x7fdb4406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb4406a430 0x7fdb4406c8f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0adbe0 tx=0x7fdb54008040 comp rx=0 tx=0).stop 2024-01-24T17:47:46.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c070460 msgr2=0x7fdb6c1f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c070460 0x7fdb6c1f1a20 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c00c4d0 tx=0x7fdb5c00c990 comp rx=0 tx=0).stop 2024-01-24T17:47:46.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 shutdown_connections 2024-01-24T17:47:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdb6c15ac70 0x7fdb6c0c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdb6c0ce8c0 0x7fdb6c1f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdb4406a430 0x7fdb4406c8f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 --2- 172.21.15.39:0/328125083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb6c070460 0x7fdb6c1f1a20 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.117+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 >> 172.21.15.39:0/328125083 conn(0x7fdb6c06a200 msgr2=0x7fdb6c072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.118+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 shutdown_connections 2024-01-24T17:47:46.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.118+0000 7fdb72c00640 1 -- 172.21.15.39:0/328125083 wait complete. 2024-01-24T17:47:46.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:47:46.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.363+0000 7f309cc00640 1 Processor -- start 2024-01-24T17:47:46.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.363+0000 7f309cc00640 1 -- start start 2024-01-24T17:47:46.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.364+0000 7f309cc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f30980ca5f0 0x7f30980ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.364+0000 7f309cc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309806c000 con 0x7f3098159950 2024-01-24T17:47:46.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.365+0000 7f309cc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309806d010 con 0x7f30980cb990 2024-01-24T17:47:46.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.365+0000 7f309cc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30980cae00 con 0x7f30980ca5f0 2024-01-24T17:47:46.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.365+0000 7f3095e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3098159950 0x7f30980cb280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:43328/0 (socket says 172.21.15.39:43328) 2024-01-24T17:47:46.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.365+0000 7f3095e00640 1 -- 172.21.15.39:0/2245693945 learned_addr learned my addr 172.21.15.39:0/2245693945 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.365+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644963740 0 0) 0x7f309806c000 con 0x7f3098159950 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.365+0000 7f3096800640 1 --2- 172.21.15.39:0/2245693945 >> v2:172.21.15.100:3300/0 conn(0x7f30980ca5f0 0x7f30980ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.365+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3078005730 con 0x7f3098159950 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.365+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 658384125 0 0) 0x7f3078005730 con 0x7f3098159950 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 >> v2:172.21.15.100:3300/0 conn(0x7f30980ca5f0 msgr2=0x7f30980ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 --2- 172.21.15.39:0/2245693945 >> v2:172.21.15.100:3300/0 conn(0x7f30980ca5f0 0x7f30980ad9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 >> v1:172.21.15.100:6789/0 conn(0x7f30980cb990 legacy=0x7f30980c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309806c250 con 0x7f3098159950 2024-01-24T17:47:46.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f3088001670 con 0x7f3098159950 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 >> v1:172.21.15.39:6789/0 conn(0x7f3098159950 legacy=0x7f30980cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f3078005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30780069d0 0x7f3078006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30780073e0 0x7f307800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30980cae00 con 0x7f3078005bd0 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307800e2f0 con 0x7f30780069d0 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307800e4b0 con 0x7f30780073e0 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3095e00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f3078005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3096800640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30780073e0 0x7f307800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3095400640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30780069d0 0x7f3078006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3095e00640 1 -- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30780073e0 msgr2=0x7f307800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3095e00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30780073e0 0x7f307800db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3095e00640 1 -- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30780069d0 msgr2=0x7f3078006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:46.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.366+0000 7f3095e00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30780069d0 0x7f3078006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.367+0000 7f3095e00640 1 -- 172.21.15.39:0/2245693945 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078005730 con 0x7f3078005bd0 2024-01-24T17:47:46.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.367+0000 7f3096800640 1 -- 172.21.15.39:0/2245693945 reap_dead start 2024-01-24T17:47:46.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.367+0000 7f3095400640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f30780069d0 0x7f3078006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:46.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f3095e00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f3078005fd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f308800c190 tx=0x7f3078007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c90129b1b2c6968 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3088005ad0 con 0x7f3078005bd0 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3088005c30 con 0x7f3078005bd0 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f3094a00640 1 -- 172.21.15.39:0/2245693945 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308802c7d0 con 0x7f3078005bd0 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f309cc00640 1 -- 172.21.15.39:0/2245693945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 msgr2=0x7f3078005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f309cc00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f3078005fd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f308800c190 tx=0x7f3078007a00 comp rx=0 tx=0).stop 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f309cc00640 1 -- 172.21.15.39:0/2245693945 shutdown_connections 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f309cc00640 1 --2- 172.21.15.39:0/2245693945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f3078005fd0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.368+0000 7f309cc00640 1 -- 172.21.15.39:0/2245693945 >> 172.21.15.39:0/2245693945 conn(0x7f30980ce9e0 msgr2=0x7f30980cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:46.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.369+0000 7f309cc00640 1 -- 172.21.15.39:0/2245693945 shutdown_connections 2024-01-24T17:47:46.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.369+0000 7f309cc00640 1 -- 172.21.15.39:0/2245693945 wait complete. 2024-01-24T17:47:46.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.370+0000 7f309cc00640 1 Processor -- start 2024-01-24T17:47:46.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.371+0000 7f309cc00640 1 -- start start 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.371+0000 7f309cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f30980ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.371+0000 7f309cc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3098158b20 0x7f30980ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.371+0000 7f309cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30980c2140 0x7f30980c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f309cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30981f22f0 con 0x7f3078005bd0 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f309cc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30980c2e90 con 0x7f3098158b20 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f309cc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30980c3170 con 0x7f30980c2140 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f3095e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f30980ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f3095e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f30980ac300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46550/0 (socket says 172.21.15.39:46550) 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f3095e00640 1 -- 172.21.15.39:0/3052680709 learned_addr learned my addr 172.21.15.39:0/3052680709 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f3095e00640 1 -- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30980c2140 msgr2=0x7f30980c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f3095e00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30980c2140 0x7f30980c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f3095e00640 1 -- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3098158b20 msgr2=0x7f30980ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f3095e00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3098158b20 0x7f30980ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.372+0000 7f3095e00640 1 -- 172.21.15.39:0/3052680709 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30980c3450 con 0x7f3078005bd0 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.373+0000 7f3095e00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f30980ac300 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f308802c370 tx=0x7f308802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.373+0000 7f308e000640 1 -- 172.21.15.39:0/3052680709 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308803c040 con 0x7f3078005bd0 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.373+0000 7f308e000640 1 -- 172.21.15.39:0/3052680709 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308802cca0 con 0x7f3078005bd0 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.373+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30980c3730 con 0x7f3078005bd0 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.373+0000 7f308e000640 1 -- 172.21.15.39:0/3052680709 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30880072b0 con 0x7f3078005bd0 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.373+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30980c2b00 con 0x7f3078005bd0 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.374+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309804dff0 con 0x7f3078005bd0 2024-01-24T17:47:46.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.375+0000 7f308e000640 1 -- 172.21.15.39:0/3052680709 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3088032070 con 0x7f3078005bd0 2024-01-24T17:47:46.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.376+0000 7f308e000640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307006a430 0x7f307006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.376+0000 7f308e000640 1 -- 172.21.15.39:0/3052680709 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30880a9b20 con 0x7f3078005bd0 2024-01-24T17:47:46.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.376+0000 7f3095400640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307006a430 0x7f307006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.378+0000 7f3095400640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307006a430 0x7f307006c8f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f30980afb40 tx=0x7f3080009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:46.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.380+0000 7f308e000640 1 -- 172.21.15.39:0/3052680709 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f30880751b0 con 0x7f3078005bd0 2024-01-24T17:47:46.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.566+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f309806dbb0 con 0x7f3078005bd0 2024-01-24T17:47:46.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.567+0000 7f308e000640 1 -- 172.21.15.39:0/3052680709 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f30880460a0 con 0x7f3078005bd0 2024-01-24T17:47:46.567 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307006a430 msgr2=0x7f307006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307006a430 0x7f307006c8f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f30980afb40 tx=0x7f3080009040 comp rx=0 tx=0).stop 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 msgr2=0x7f30980ac300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f30980ac300 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f308802c370 tx=0x7f308802e070 comp rx=0 tx=0).stop 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 shutdown_connections 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f30980c2140 0x7f30980c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3098158b20 0x7f30980ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f307006a430 0x7f307006c8f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 --2- 172.21.15.39:0/3052680709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3078005bd0 0x7f30980ac300 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 >> 172.21.15.39:0/3052680709 conn(0x7f30980ce9e0 msgr2=0x7f30980ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.569+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 shutdown_connections 2024-01-24T17:47:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.570+0000 7f309cc00640 1 -- 172.21.15.39:0/3052680709 wait complete. 2024-01-24T17:47:46.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:47:46.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.800+0000 7fb0ee200640 1 Processor -- start 2024-01-24T17:47:46.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.801+0000 7fb0ee200640 1 -- start start 2024-01-24T17:47:46.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.802+0000 7fb0ee200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb0e8071ab0 0x7fb0e80cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.802+0000 7fb0ee200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0e8160ea0 0x7fb0e806ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.802+0000 7fb0ee200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e815cfa0 con 0x7fb0e80cb7c0 2024-01-24T17:47:46.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.802+0000 7fb0ee200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e8071e90 con 0x7fb0e8160ea0 2024-01-24T17:47:46.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.802+0000 7fb0ee200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e81614c0 con 0x7fb0e8071ab0 2024-01-24T17:47:46.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.803+0000 7fb0e7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0e8160ea0 0x7fb0e806ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.803+0000 7fb0e7400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb0e8071ab0 0x7fb0e80cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.803+0000 7fb0e7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0e8160ea0 0x7fb0e806ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46560/0 (socket says 172.21.15.39:46560) 2024-01-24T17:47:46.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.803+0000 7fb0e7e00640 1 -- 172.21.15.39:0/1224183137 learned_addr learned my addr 172.21.15.39:0/1224183137 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:46.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.803+0000 7fb0e7e00640 1 -- 172.21.15.39:0/1224183137 >> v2:172.21.15.100:3300/0 conn(0x7fb0e8071ab0 msgr2=0x7fb0e80cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:46.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.803+0000 7fb0e7e00640 1 --2- 172.21.15.39:0/1224183137 >> v2:172.21.15.100:3300/0 conn(0x7fb0e8071ab0 0x7fb0e80cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.803+0000 7fb0e7e00640 1 -- 172.21.15.39:0/1224183137 >> v1:172.21.15.100:6789/0 conn(0x7fb0e80cb7c0 legacy=0x7fb0e8160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.803+0000 7fb0e7e00640 1 -- 172.21.15.39:0/1224183137 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e80cbc20 con 0x7fb0e8160ea0 2024-01-24T17:47:46.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.805+0000 7fb0e7e00640 1 --2- 172.21.15.39:0/1224183137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e806ffa0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80ce520 tx=0x7fb0e002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c235467ce67b6d2f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:46.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.805+0000 7fb0e6a00640 1 -- 172.21.15.39:0/1224183137 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e002cb10 con 0x7fb0e8160ea0 2024-01-24T17:47:46.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.805+0000 7fb0e6a00640 1 -- 172.21.15.39:0/1224183137 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e002cc70 con 0x7fb0e8160ea0 2024-01-24T17:47:46.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.805+0000 7fb0e6a00640 1 -- 172.21.15.39:0/1224183137 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e00343f0 con 0x7fb0e8160ea0 2024-01-24T17:47:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.805+0000 7fb0ee200640 1 -- 172.21.15.39:0/1224183137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 msgr2=0x7fb0e806ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.805+0000 7fb0ee200640 1 --2- 172.21.15.39:0/1224183137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e806ffa0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80ce520 tx=0x7fb0e002bf70 comp rx=0 tx=0).stop 2024-01-24T17:47:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.806+0000 7fb0ee200640 1 -- 172.21.15.39:0/1224183137 shutdown_connections 2024-01-24T17:47:46.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.806+0000 7fb0ee200640 1 --2- 172.21.15.39:0/1224183137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e806ffa0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.806+0000 7fb0ee200640 1 --2- 172.21.15.39:0/1224183137 >> v2:172.21.15.100:3300/0 conn(0x7fb0e8071ab0 0x7fb0e80cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.806+0000 7fb0ee200640 1 -- 172.21.15.39:0/1224183137 >> 172.21.15.39:0/1224183137 conn(0x7fb0e80ce9e0 msgr2=0x7fb0e80cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:46.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.806+0000 7fb0ee200640 1 -- 172.21.15.39:0/1224183137 shutdown_connections 2024-01-24T17:47:46.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.806+0000 7fb0ee200640 1 -- 172.21.15.39:0/1224183137 wait complete. 2024-01-24T17:47:46.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.807+0000 7fb0ee200640 1 Processor -- start 2024-01-24T17:47:46.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.808+0000 7fb0ee200640 1 -- start start 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.808+0000 7fb0ee200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0e8071ab0 0x7fb0e815a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.808+0000 7fb0ee200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0e80cb7c0 0x7fb0e815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.808+0000 7fb0ee200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e815ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.808+0000 7fb0ee200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80b0e80 con 0x7fb0e8160ea0 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.808+0000 7fb0ee200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81f3070 con 0x7fb0e8071ab0 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.808+0000 7fb0ee200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81f23f0 con 0x7fb0e80cb7c0 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e815ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e815ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46564/0 (socket says 172.21.15.39:46564) 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7e00640 1 -- 172.21.15.39:0/793225164 learned_addr learned my addr 172.21.15.39:0/793225164 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:46.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7400640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0e8071ab0 0x7fb0e815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0de800640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0e80cb7c0 0x7fb0e815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7e00640 1 -- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0e80cb7c0 msgr2=0x7fb0e815fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7e00640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0e80cb7c0 0x7fb0e815fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7e00640 1 -- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0e8071ab0 msgr2=0x7fb0e815a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7e00640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0e8071ab0 0x7fb0e815a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7e00640 1 -- 172.21.15.39:0/793225164 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0009b30 con 0x7fb0e8160ea0 2024-01-24T17:47:46.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.809+0000 7fb0e7400640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0e8071ab0 0x7fb0e815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:46.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.810+0000 7fb0e7e00640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e815ae60 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0036040 tx=0x7fb0e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:46.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.810+0000 7fb0dfe00640 1 -- 172.21.15.39:0/793225164 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0009510 con 0x7fb0e8160ea0 2024-01-24T17:47:46.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.810+0000 7fb0dfe00640 1 -- 172.21.15.39:0/793225164 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e003e450 con 0x7fb0e8160ea0 2024-01-24T17:47:46.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.810+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e81f26d0 con 0x7fb0e8160ea0 2024-01-24T17:47:46.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.810+0000 7fb0dfe00640 1 -- 172.21.15.39:0/793225164 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e003e800 con 0x7fb0e8160ea0 2024-01-24T17:47:46.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.810+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e81f2c90 con 0x7fb0e8160ea0 2024-01-24T17:47:46.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.811+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e804dff0 con 0x7fb0e8160ea0 2024-01-24T17:47:46.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.812+0000 7fb0dfe00640 1 -- 172.21.15.39:0/793225164 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb0e003e960 con 0x7fb0e8160ea0 2024-01-24T17:47:46.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.813+0000 7fb0dfe00640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0c006a460 0x7fb0c006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:46.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.813+0000 7fb0dfe00640 1 -- 172.21.15.39:0/793225164 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0e00ab640 con 0x7fb0e8160ea0 2024-01-24T17:47:46.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.814+0000 7fb0e7400640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0c006a460 0x7fb0c006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:46.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.815+0000 7fb0e7400640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0c006a460 0x7fb0c006c920 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80c8b70 tx=0x7fb0d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:46.817+0000 7fb0dfe00640 1 -- 172.21.15.39:0/793225164 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb0e0076c20 con 0x7fb0e8160ea0 2024-01-24T17:47:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.009+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0e806ca50 con 0x7fb0e8160ea0 2024-01-24T17:47:47.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.010+0000 7fb0dfe00640 1 -- 172.21.15.39:0/793225164 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb0e007a240 con 0x7fb0e8160ea0 2024-01-24T17:47:47.011 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0c006a460 msgr2=0x7fb0c006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0c006a460 0x7fb0c006c920 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb0e80c8b70 tx=0x7fb0d8008040 comp rx=0 tx=0).stop 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 msgr2=0x7fb0e815ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e815ae60 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0036040 tx=0x7fb0e0004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 shutdown_connections 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0c006a460 0x7fb0c006c920 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0e8160ea0 0x7fb0e815ae60 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0e80cb7c0 0x7fb0e815fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.012+0000 7fb0ee200640 1 --2- 172.21.15.39:0/793225164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0e8071ab0 0x7fb0e815a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.013+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 >> 172.21.15.39:0/793225164 conn(0x7fb0e80ce9e0 msgr2=0x7fb0e815c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.013+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 shutdown_connections 2024-01-24T17:47:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.013+0000 7fb0ee200640 1 -- 172.21.15.39:0/793225164 wait complete. 2024-01-24T17:47:47.033 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:47:47.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.252+0000 7f6895200640 1 Processor -- start 2024-01-24T17:47:47.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.252+0000 7f6895200640 1 -- start start 2024-01-24T17:47:47.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.253+0000 7f6895200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f68900b0010 0x7f689015bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.253+0000 7f6895200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6890070920 0x7f68900b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.253+0000 7f6895200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f689015f390 0x7f68900cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.253+0000 7f6895200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6890070470 con 0x7f689015f390 2024-01-24T17:47:47.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.253+0000 7f6895200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689015ef00 con 0x7f68900b0010 2024-01-24T17:47:47.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.253+0000 7f6895200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68900710a0 con 0x7f6890070920 2024-01-24T17:47:47.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f689015f390 0x7f68900cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f689015f390 0x7f68900cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46574/0 (socket says 172.21.15.39:46574) 2024-01-24T17:47:47.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688e800640 1 -- 172.21.15.39:0/3324397493 learned_addr learned my addr 172.21.15.39:0/3324397493 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:47.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688de00640 1 --2- 172.21.15.39:0/3324397493 >> v2:172.21.15.100:3300/0 conn(0x7f68900b0010 0x7f689015bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688d400640 1 --2- 172.21.15.39:0/3324397493 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6890070920 0x7f68900b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688e800640 1 -- 172.21.15.39:0/3324397493 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6890070920 msgr2=0x7f68900b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:47.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688e800640 1 --2- 172.21.15.39:0/3324397493 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6890070920 0x7f68900b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688e800640 1 -- 172.21.15.39:0/3324397493 >> v2:172.21.15.100:3300/0 conn(0x7f68900b0010 msgr2=0x7f689015bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:47.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688e800640 1 --2- 172.21.15.39:0/3324397493 >> v2:172.21.15.100:3300/0 conn(0x7f68900b0010 0x7f689015bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.254+0000 7f688e800640 1 -- 172.21.15.39:0/3324397493 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689015c0d0 con 0x7f689015f390 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.255+0000 7f688e800640 1 --2- 172.21.15.39:0/3324397493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f689015f390 0x7f68900cafa0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f68900b1010 tx=0x7f688802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41b696b96e14182 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.255+0000 7f688ca00640 1 -- 172.21.15.39:0/3324397493 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688802cb00 con 0x7f689015f390 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f688ca00640 1 -- 172.21.15.39:0/3324397493 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688802cc60 con 0x7f689015f390 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f688ca00640 1 -- 172.21.15.39:0/3324397493 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68880343f0 con 0x7f689015f390 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 -- 172.21.15.39:0/3324397493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f689015f390 msgr2=0x7f68900cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 --2- 172.21.15.39:0/3324397493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f689015f390 0x7f68900cafa0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f68900b1010 tx=0x7f688802bf60 comp rx=0 tx=0).stop 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 -- 172.21.15.39:0/3324397493 shutdown_connections 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 --2- 172.21.15.39:0/3324397493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f689015f390 0x7f68900cafa0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 --2- 172.21.15.39:0/3324397493 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6890070920 0x7f68900b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 --2- 172.21.15.39:0/3324397493 >> v2:172.21.15.100:3300/0 conn(0x7f68900b0010 0x7f689015bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 -- 172.21.15.39:0/3324397493 >> 172.21.15.39:0/3324397493 conn(0x7f689006bf20 msgr2=0x7f689006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 -- 172.21.15.39:0/3324397493 shutdown_connections 2024-01-24T17:47:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.256+0000 7f6895200640 1 -- 172.21.15.39:0/3324397493 wait complete. 2024-01-24T17:47:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.258+0000 7f6895200640 1 Processor -- start 2024-01-24T17:47:47.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.258+0000 7f6895200640 1 -- start start 2024-01-24T17:47:47.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f6895200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6890070920 0x7f6890157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f6895200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68900b0010 0x7f68901efe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f6895200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f689015f390 0x7f68901542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f6895200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68900c7c30 con 0x7f6890070920 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f6895200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68900c5bd0 con 0x7f689015f390 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f6895200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68900c7010 con 0x7f68900b0010 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f688de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6890070920 0x7f6890157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f688de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6890070920 0x7f6890157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46588/0 (socket says 172.21.15.39:46588) 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.259+0000 7f688de00640 1 -- 172.21.15.39:0/2773292766 learned_addr learned my addr 172.21.15.39:0/2773292766 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.260+0000 7f688d400640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68900b0010 0x7f68901efe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.260+0000 7f688e800640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f689015f390 0x7f68901542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.260+0000 7f688de00640 1 -- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68900b0010 msgr2=0x7f68901efe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.260+0000 7f688de00640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68900b0010 0x7f68901efe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.260+0000 7f688de00640 1 -- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f689015f390 msgr2=0x7f68901542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.260+0000 7f688de00640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f689015f390 0x7f68901542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.260+0000 7f688de00640 1 -- 172.21.15.39:0/2773292766 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6888009b30 con 0x7f6890070920 2024-01-24T17:47:47.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.260+0000 7f688de00640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6890070920 0x7f6890157220 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f688000d800 tx=0x7f688000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:47.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.261+0000 7f6886000640 1 -- 172.21.15.39:0/2773292766 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688000b6c0 con 0x7f6890070920 2024-01-24T17:47:47.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.261+0000 7f6886000640 1 -- 172.21.15.39:0/2773292766 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688000b820 con 0x7f6890070920 2024-01-24T17:47:47.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.261+0000 7f6886000640 1 -- 172.21.15.39:0/2773292766 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688000bbb0 con 0x7f6890070920 2024-01-24T17:47:47.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.261+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6890146dc0 con 0x7f6890070920 2024-01-24T17:47:47.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.261+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68900c7570 con 0x7f6890070920 2024-01-24T17:47:47.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.262+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689004dff0 con 0x7f6890070920 2024-01-24T17:47:47.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.263+0000 7f6886000640 1 -- 172.21.15.39:0/2773292766 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f688000bd10 con 0x7f6890070920 2024-01-24T17:47:47.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.264+0000 7f6886000640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f686806a430 0x7f686806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.264+0000 7f6886000640 1 -- 172.21.15.39:0/2773292766 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6880088e70 con 0x7f6890070920 2024-01-24T17:47:47.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.265+0000 7f688d400640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f686806a430 0x7f686806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.266+0000 7f688d400640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f686806a430 0x7f686806c8f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6890151350 tx=0x7f6878008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:47.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.269+0000 7f6886000640 1 -- 172.21.15.39:0/2773292766 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6880054450 con 0x7f6890070920 2024-01-24T17:47:47.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.468+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6890151380 con 0x7f6890070920 2024-01-24T17:47:47.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.469+0000 7f6886000640 1 -- 172.21.15.39:0/2773292766 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v88) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6880057a70 con 0x7f6890070920 2024-01-24T17:47:47.470 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:47:47.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.472+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f686806a430 msgr2=0x7f686806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.472+0000 7f6895200640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f686806a430 0x7f686806c8f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6890151350 tx=0x7f6878008040 comp rx=0 tx=0).stop 2024-01-24T17:47:47.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.472+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6890070920 msgr2=0x7f6890157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.472+0000 7f6895200640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6890070920 0x7f6890157220 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f688000d800 tx=0x7f688000dcc0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.473+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 shutdown_connections 2024-01-24T17:47:47.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.473+0000 7f6895200640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f689015f390 0x7f68901542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.473+0000 7f6895200640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f68900b0010 0x7f68901efe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.473+0000 7f6895200640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f686806a430 0x7f686806c8f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.473+0000 7f6895200640 1 --2- 172.21.15.39:0/2773292766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6890070920 0x7f6890157220 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.473+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 >> 172.21.15.39:0/2773292766 conn(0x7f689006bf20 msgr2=0x7f68900af6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:47.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.473+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 shutdown_connections 2024-01-24T17:47:47.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.473+0000 7f6895200640 1 -- 172.21.15.39:0/2773292766 wait complete. 2024-01-24T17:47:47.492 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:47:47.492 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:47.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:47.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.798+0000 7f0559200640 1 Processor -- start 2024-01-24T17:47:47.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.798+0000 7f0559200640 1 -- start start 2024-01-24T17:47:47.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.800+0000 7f0559200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f05540b0010 0x7f055415bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.800+0000 7f0559200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f055415f390 0x7f05540c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.800+0000 7f0559200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0554070470 con 0x7f0554070920 2024-01-24T17:47:47.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.800+0000 7f0559200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055415ef00 con 0x7f055415f390 2024-01-24T17:47:47.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.800+0000 7f0559200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05540710a0 con 0x7f05540b0010 2024-01-24T17:47:47.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.800+0000 7f0552800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f055415f390 0x7f05540c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.800+0000 7f0552800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f055415f390 0x7f05540c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46598/0 (socket says 172.21.15.39:46598) 2024-01-24T17:47:47.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.800+0000 7f0552800640 1 -- 172.21.15.39:0/3756575392 learned_addr learned my addr 172.21.15.39:0/3756575392 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:47.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.801+0000 7f0551e00640 1 --2- 172.21.15.39:0/3756575392 >> v2:172.21.15.100:3300/0 conn(0x7f05540b0010 0x7f055415bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.801+0000 7f0552800640 1 -- 172.21.15.39:0/3756575392 >> v2:172.21.15.100:3300/0 conn(0x7f05540b0010 msgr2=0x7f055415bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:47.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.801+0000 7f0552800640 1 --2- 172.21.15.39:0/3756575392 >> v2:172.21.15.100:3300/0 conn(0x7f05540b0010 0x7f055415bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.801+0000 7f0552800640 1 -- 172.21.15.39:0/3756575392 >> v1:172.21.15.100:6789/0 conn(0x7f0554070920 legacy=0x7f05540b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:47.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.801+0000 7f0552800640 1 -- 172.21.15.39:0/3756575392 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055415c0d0 con 0x7f055415f390 2024-01-24T17:47:47.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.802+0000 7f0552800640 1 --2- 172.21.15.39:0/3756575392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540c9450 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f05540b1010 tx=0x7f054c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5fc0f6a96cec9a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:47.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.803+0000 7f0551400640 1 -- 172.21.15.39:0/3756575392 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c02cb20 con 0x7f055415f390 2024-01-24T17:47:47.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.803+0000 7f0551400640 1 -- 172.21.15.39:0/3756575392 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054c02cc80 con 0x7f055415f390 2024-01-24T17:47:47.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.803+0000 7f0551400640 1 -- 172.21.15.39:0/3756575392 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c0343f0 con 0x7f055415f390 2024-01-24T17:47:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.803+0000 7f0559200640 1 -- 172.21.15.39:0/3756575392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 msgr2=0x7f05540c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.803+0000 7f0559200640 1 --2- 172.21.15.39:0/3756575392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540c9450 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f05540b1010 tx=0x7f054c02bf80 comp rx=0 tx=0).stop 2024-01-24T17:47:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.804+0000 7f0559200640 1 -- 172.21.15.39:0/3756575392 shutdown_connections 2024-01-24T17:47:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.804+0000 7f0559200640 1 --2- 172.21.15.39:0/3756575392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540c9450 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.804+0000 7f0559200640 1 --2- 172.21.15.39:0/3756575392 >> v2:172.21.15.100:3300/0 conn(0x7f05540b0010 0x7f055415bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.804+0000 7f0559200640 1 -- 172.21.15.39:0/3756575392 >> 172.21.15.39:0/3756575392 conn(0x7f055406bf20 msgr2=0x7f055406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:47.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.804+0000 7f0559200640 1 -- 172.21.15.39:0/3756575392 shutdown_connections 2024-01-24T17:47:47.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.804+0000 7f0559200640 1 -- 172.21.15.39:0/3756575392 wait complete. 2024-01-24T17:47:47.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.805+0000 7f0559200640 1 Processor -- start 2024-01-24T17:47:47.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.805+0000 7f0559200640 1 -- start start 2024-01-24T17:47:47.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.806+0000 7f0559200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0554070920 0x7f05540c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.806+0000 7f0559200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05540b0010 0x7f05540ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.806+0000 7f0559200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540ad770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.806+0000 7f0559200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05540c88a0 con 0x7f055415f390 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.806+0000 7f0559200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05540c7070 con 0x7f05540b0010 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.806+0000 7f0559200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05540c7350 con 0x7f0554070920 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46602/0 (socket says 172.21.15.39:46602) 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 -- 172.21.15.39:0/2862960374 learned_addr learned my addr 172.21.15.39:0/2862960374 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0551e00640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0554070920 0x7f05540c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f054be00640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05540b0010 0x7f05540ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 -- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0554070920 msgr2=0x7f05540c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0554070920 0x7f05540c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 -- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05540b0010 msgr2=0x7f05540ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05540b0010 0x7f05540ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 -- 172.21.15.39:0/2862960374 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c009b30 con 0x7f055415f390 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f054be00640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05540b0010 0x7f05540ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.807+0000 7f0552800640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540ad770 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f054c009f40 tx=0x7f054c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:47.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.808+0000 7f054a000640 1 -- 172.21.15.39:0/2862960374 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c040670 con 0x7f055415f390 2024-01-24T17:47:47.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.808+0000 7f054a000640 1 -- 172.21.15.39:0/2862960374 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054c0407d0 con 0x7f055415f390 2024-01-24T17:47:47.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.808+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05540c7630 con 0x7f055415f390 2024-01-24T17:47:47.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.808+0000 7f054a000640 1 -- 172.21.15.39:0/2862960374 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c040bc0 con 0x7f055415f390 2024-01-24T17:47:47.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.808+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05540c7bf0 con 0x7f055415f390 2024-01-24T17:47:47.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.809+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05540b0920 con 0x7f055415f390 2024-01-24T17:47:47.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.810+0000 7f054a000640 1 -- 172.21.15.39:0/2862960374 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f054c040d20 con 0x7f055415f390 2024-01-24T17:47:47.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.811+0000 7f054a000640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f052c06a430 0x7f052c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:47.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.811+0000 7f054a000640 1 -- 172.21.15.39:0/2862960374 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f054c0ab910 con 0x7f055415f390 2024-01-24T17:47:47.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.812+0000 7f0551e00640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f052c06a430 0x7f052c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:47.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.813+0000 7f0551e00640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f052c06a430 0x7f052c06c8f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0544006fd0 tx=0x7f0544008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:47.815+0000 7f054a000640 1 -- 172.21.15.39:0/2862960374 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f054c076f00 con 0x7f055415f390 2024-01-24T17:47:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.009+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f055404dff0 con 0x7f055415f390 2024-01-24T17:47:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.011+0000 7f054a000640 1 -- 172.21.15.39:0/2862960374 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16104 (secure 0 0 0) 0x7f054c009510 con 0x7f055415f390 2024-01-24T17:47:48.012 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:48.012 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:32.941651+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:48.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.013+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f052c06a430 msgr2=0x7f052c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.013+0000 7f0559200640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f052c06a430 0x7f052c06c8f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0544006fd0 tx=0x7f0544008040 comp rx=0 tx=0).stop 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 msgr2=0x7f05540ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540ad770 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f054c009f40 tx=0x7f054c004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 shutdown_connections 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f052c06a430 0x7f052c06c8f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f055415f390 0x7f05540ad770 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f05540b0010 0x7f05540ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 --2- 172.21.15.39:0/2862960374 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0554070920 0x7f05540c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 >> 172.21.15.39:0/2862960374 conn(0x7f055406bf20 msgr2=0x7f055415d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 shutdown_connections 2024-01-24T17:47:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.014+0000 7f0559200640 1 -- 172.21.15.39:0/2862960374 wait complete. 2024-01-24T17:47:48.033 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:47:30.315286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:47:30.960101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:47:48.033 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:47:48.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.254+0000 7fc79d000640 1 Processor -- start 2024-01-24T17:47:48.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.255+0000 7fc79d000640 1 -- start start 2024-01-24T17:47:48.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.256+0000 7fc79d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7981610c0 0x7fc79806cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.256+0000 7fc79d000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc7980b1db0 0x7fc7980b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.256+0000 7fc79d000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7981614a0 con 0x7fc7980c1590 2024-01-24T17:47:48.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.256+0000 7fc79d000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7980b28e0 con 0x7fc7980b1db0 2024-01-24T17:47:48.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.256+0000 7fc79d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79806d4a0 con 0x7fc7981610c0 2024-01-24T17:47:48.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.256+0000 7fc796800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc7980c1590 0x7fc7980cc890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:43344/0 (socket says 172.21.15.39:43344) 2024-01-24T17:47:48.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.256+0000 7fc796800640 1 -- 172.21.15.39:0/2040731438 learned_addr learned my addr 172.21.15.39:0/2040731438 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:48.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc78fe00640 1 --2- 172.21.15.39:0/2040731438 >> v2:172.21.15.100:3300/0 conn(0x7fc7980b1db0 0x7fc7980b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795e00640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7981610c0 0x7fc79806cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177612356 0 0) 0x7fc7981614a0 con 0x7fc7980c1590 2024-01-24T17:47:48.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778003520 con 0x7fc7980c1590 2024-01-24T17:47:48.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765410303 0 0) 0x7fc778003520 con 0x7fc7980c1590 2024-01-24T17:47:48.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7981610c0 msgr2=0x7fc79806cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:48.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7981610c0 0x7fc79806cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 >> v2:172.21.15.100:3300/0 conn(0x7fc7980b1db0 msgr2=0x7fc7980b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 --2- 172.21.15.39:0/2040731438 >> v2:172.21.15.100:3300/0 conn(0x7fc7980b1db0 0x7fc7980b21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79806c1a0 con 0x7fc7980c1590 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc7900019f0 con 0x7fc7980c1590 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.257+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 >> v1:172.21.15.39:6789/0 conn(0x7fc7980c1590 legacy=0x7fc7980cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795400640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc778003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795400640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7981610c0 0x7fc7780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795400640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc778004d00 0x7fc778005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7980b28e0 con 0x7fc7780039a0 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79806d4a0 con 0x7fc778004d00 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77800b980 con 0x7fc7981610c0 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795e00640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc778003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795e00640 1 -- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7981610c0 msgr2=0x7fc7780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:48.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795e00640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7981610c0 0x7fc7780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795e00640 1 -- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc778004d00 msgr2=0x7fc778005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:48.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795e00640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc778004d00 0x7fc778005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.258+0000 7fc795e00640 1 -- 172.21.15.39:0/2040731438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc778003520 con 0x7fc7780039a0 2024-01-24T17:47:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.259+0000 7fc795e00640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc778003dc0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc788009ac0 tx=0x7fc7780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6e44be431132b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78802c710 con 0x7fc7780039a0 2024-01-24T17:47:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78802c870 con 0x7fc7780039a0 2024-01-24T17:47:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc795400640 1 -- 172.21.15.39:0/2040731438 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78802cc00 con 0x7fc7780039a0 2024-01-24T17:47:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc79d000640 1 -- 172.21.15.39:0/2040731438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 msgr2=0x7fc778003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc79d000640 1 --2- 172.21.15.39:0/2040731438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc778003dc0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc788009ac0 tx=0x7fc7780057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc796800640 1 -- 172.21.15.39:0/2040731438 reap_dead start 2024-01-24T17:47:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc79d000640 1 -- 172.21.15.39:0/2040731438 shutdown_connections 2024-01-24T17:47:48.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc79d000640 1 -- 172.21.15.39:0/2040731438 >> 172.21.15.39:0/2040731438 conn(0x7fc7980ce9e0 msgr2=0x7fc7980cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:48.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc79d000640 1 -- 172.21.15.39:0/2040731438 shutdown_connections 2024-01-24T17:47:48.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.260+0000 7fc79d000640 1 -- 172.21.15.39:0/2040731438 wait complete. 2024-01-24T17:47:48.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.262+0000 7fc79d000640 1 Processor -- start 2024-01-24T17:47:48.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.262+0000 7fc79d000640 1 -- start start 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.263+0000 7fc79d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc79815a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.263+0000 7fc79d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7980c64f0 0x7fc798161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.263+0000 7fc79d000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7981623f0 0x7fc7981631b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.263+0000 7fc79d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7980ae020 con 0x7fc7780039a0 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.263+0000 7fc79d000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79815aa50 con 0x7fc7981623f0 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.263+0000 7fc79d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7980ae270 con 0x7fc7980c64f0 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc79815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc79815a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46624/0 (socket says 172.21.15.39:46624) 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 -- 172.21.15.39:0/2983807509 learned_addr learned my addr 172.21.15.39:0/2983807509 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 -- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7980c64f0 msgr2=0x7fc798161eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7980c64f0 0x7fc798161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 -- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7981623f0 msgr2=0x7fc7981631b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7981623f0 0x7fc7981631b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 -- 172.21.15.39:0/2983807509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7980ae550 con 0x7fc7780039a0 2024-01-24T17:47:48.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.264+0000 7fc795e00640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc79815a510 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc788037040 tx=0x7fc788004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:48.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.265+0000 7fc78e000640 1 -- 172.21.15.39:0/2983807509 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788003810 con 0x7fc7780039a0 2024-01-24T17:47:48.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.265+0000 7fc78e000640 1 -- 172.21.15.39:0/2983807509 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78802de90 con 0x7fc7780039a0 2024-01-24T17:47:48.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.265+0000 7fc78e000640 1 -- 172.21.15.39:0/2983807509 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788003b90 con 0x7fc7780039a0 2024-01-24T17:47:48.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.265+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7980ae830 con 0x7fc7780039a0 2024-01-24T17:47:48.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.265+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7980b0b70 con 0x7fc7780039a0 2024-01-24T17:47:48.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.266+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79804dff0 con 0x7fc7780039a0 2024-01-24T17:47:48.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.267+0000 7fc78e000640 1 -- 172.21.15.39:0/2983807509 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc788003cf0 con 0x7fc7780039a0 2024-01-24T17:47:48.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.268+0000 7fc78e000640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc77006a460 0x7fc77006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.268+0000 7fc78e000640 1 -- 172.21.15.39:0/2983807509 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7880aa8a0 con 0x7fc7780039a0 2024-01-24T17:47:48.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.268+0000 7fc78fe00640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc77006a460 0x7fc77006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.269+0000 7fc78fe00640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc77006a460 0x7fc77006c920 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc7800093d0 tx=0x7fc780009300 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:48.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.272+0000 7fc78e000640 1 -- 172.21.15.39:0/2983807509 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc788075fc0 con 0x7fc7780039a0 2024-01-24T17:47:48.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.464+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7980ae020 con 0x7fc7780039a0 2024-01-24T17:47:48.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.465+0000 7fc78e000640 1 -- 172.21.15.39:0/2983807509 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7880795e0 con 0x7fc7780039a0 2024-01-24T17:47:48.466 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:47:48.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc77006a460 msgr2=0x7fc77006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc77006a460 0x7fc77006c920 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc7800093d0 tx=0x7fc780009300 comp rx=0 tx=0).stop 2024-01-24T17:47:48.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 msgr2=0x7fc79815a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc79815a510 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc788037040 tx=0x7fc788004730 comp rx=0 tx=0).stop 2024-01-24T17:47:48.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 shutdown_connections 2024-01-24T17:47:48.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc7981623f0 0x7fc7981631b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7980c64f0 0x7fc798161eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc77006a460 0x7fc77006c920 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 --2- 172.21.15.39:0/2983807509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7780039a0 0x7fc79815a510 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.468+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 >> 172.21.15.39:0/2983807509 conn(0x7fc7980ce9e0 msgr2=0x7fc7980ab340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:48.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.469+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 shutdown_connections 2024-01-24T17:47:48.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.469+0000 7fc79d000640 1 -- 172.21.15.39:0/2983807509 wait complete. 2024-01-24T17:47:48.487 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:47:48.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.711+0000 7f025e400640 1 Processor -- start 2024-01-24T17:47:48.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.711+0000 7f025e400640 1 -- start start 2024-01-24T17:47:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.713+0000 7f025e400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f025806a2e0 0x7f02580cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.713+0000 7f025e400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025806e550 con 0x7f025806e170 2024-01-24T17:47:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.713+0000 7f025e400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025815dbb0 con 0x7f02580ae5b0 2024-01-24T17:47:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.713+0000 7f025e400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02580ce630 con 0x7f025806a2e0 2024-01-24T17:47:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.714+0000 7f025d000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f025806e170 0x7f0258069bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:43350/0 (socket says 172.21.15.39:43350) 2024-01-24T17:47:48.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.714+0000 7f025d000640 1 -- 172.21.15.39:0/3764950531 learned_addr learned my addr 172.21.15.39:0/3764950531 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:48.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.714+0000 7f0257e00640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f025806a2e0 0x7f02580cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.714+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382469750 0 0) 0x7f025806e550 con 0x7f025806e170 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.714+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0238003520 con 0x7f025806e170 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.714+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1826065915 0 0) 0x7f0238003520 con 0x7f025806e170 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f025806a2e0 msgr2=0x7f02580cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f025806a2e0 0x7f02580cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 >> v1:172.21.15.100:6789/0 conn(0x7f02580ae5b0 legacy=0x7f02580cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025815de50 con 0x7f025806e170 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f0248001670 con 0x7f025806e170 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 >> v1:172.21.15.39:6789/0 conn(0x7f025806e170 legacy=0x7f0258069bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02380039a0 0x7f0238003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f025806a2e0 0x7f02380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0238004d00 0x7f0238005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580ce630 con 0x7f02380039a0 2024-01-24T17:47:48.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023800ba60 con 0x7f0238004d00 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023800bc20 con 0x7f025806a2e0 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f025d000640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02380039a0 0x7f0238003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f0257e00640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f025806a2e0 0x7f02380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f025da00640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0238004d00 0x7f0238005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f025d000640 1 -- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f025806a2e0 msgr2=0x7f02380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f025d000640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f025806a2e0 0x7f02380047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.715+0000 7f025d000640 1 -- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0238004d00 msgr2=0x7f0238005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.716+0000 7f025d000640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0238004d00 0x7f0238005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.716+0000 7f025d000640 1 -- 172.21.15.39:0/3764950531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238003520 con 0x7f02380039a0 2024-01-24T17:47:48.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.716+0000 7f025da00640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0238004d00 0x7f0238005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:48.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.717+0000 7f025d000640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02380039a0 0x7f0238003dc0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f0248001eb0 tx=0x7f02380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5fc43aa919ff66 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:48.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.717+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248005ad0 con 0x7f02380039a0 2024-01-24T17:47:48.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.717+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0248005c30 con 0x7f02380039a0 2024-01-24T17:47:48.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.717+0000 7f0257400640 1 -- 172.21.15.39:0/3764950531 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024802c7c0 con 0x7f02380039a0 2024-01-24T17:47:48.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.718+0000 7f025e400640 1 -- 172.21.15.39:0/3764950531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02380039a0 msgr2=0x7f0238003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.718+0000 7f025e400640 1 --2- 172.21.15.39:0/3764950531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02380039a0 0x7f0238003dc0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f0248001eb0 tx=0x7f02380057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.718+0000 7f025da00640 1 -- 172.21.15.39:0/3764950531 reap_dead start 2024-01-24T17:47:48.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.718+0000 7f025e400640 1 -- 172.21.15.39:0/3764950531 shutdown_connections 2024-01-24T17:47:48.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.718+0000 7f025e400640 1 -- 172.21.15.39:0/3764950531 >> 172.21.15.39:0/3764950531 conn(0x7f02580cf4a0 msgr2=0x7f02580cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:48.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.718+0000 7f025e400640 1 -- 172.21.15.39:0/3764950531 shutdown_connections 2024-01-24T17:47:48.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.718+0000 7f025e400640 1 -- 172.21.15.39:0/3764950531 wait complete. 2024-01-24T17:47:48.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.720+0000 7f025e400640 1 Processor -- start 2024-01-24T17:47:48.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.720+0000 7f025e400640 1 -- start start 2024-01-24T17:47:48.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f025e400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f02380039a0 0x7f02580b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f025e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02580c6b10 0x7f02581475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f025e400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f02581495e0 0x7f02580c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f025e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580c6f90 con 0x7f02580c6b10 2024-01-24T17:47:48.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f025e400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580c8780 con 0x7f02581495e0 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f025e400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0258149e90 con 0x7f02380039a0 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f0257e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02580c6b10 0x7f02581475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f0257e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02580c6b10 0x7f02581475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46652/0 (socket says 172.21.15.39:46652) 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f0257e00640 1 -- 172.21.15.39:0/3250014200 learned_addr learned my addr 172.21.15.39:0/3250014200 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f025da00640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f02581495e0 0x7f02580c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.721+0000 7f025d000640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f02380039a0 0x7f02580b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0257e00640 1 -- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f02380039a0 msgr2=0x7f02580b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0257e00640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f02380039a0 0x7f02580b2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0257e00640 1 -- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f02581495e0 msgr2=0x7f02580c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0257e00640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f02581495e0 0x7f02580c9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0257e00640 1 -- 172.21.15.39:0/3250014200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025814a170 con 0x7f02580c6b10 2024-01-24T17:47:48.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0257e00640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02580c6b10 0x7f02581475b0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f0240010890 tx=0x7f0240010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:48.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0254c00640 1 -- 172.21.15.39:0/3250014200 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240009610 con 0x7f02580c6b10 2024-01-24T17:47:48.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0254c00640 1 -- 172.21.15.39:0/3250014200 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024000eb60 con 0x7f02580c6b10 2024-01-24T17:47:48.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025814a450 con 0x7f02580c6b10 2024-01-24T17:47:48.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f0254c00640 1 -- 172.21.15.39:0/3250014200 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024001c400 con 0x7f02580c6b10 2024-01-24T17:47:48.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.722+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0258149a60 con 0x7f02580c6b10 2024-01-24T17:47:48.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.723+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025804e840 con 0x7f02580c6b10 2024-01-24T17:47:48.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.724+0000 7f0254c00640 1 -- 172.21.15.39:0/3250014200 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f024001c560 con 0x7f02580c6b10 2024-01-24T17:47:48.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.725+0000 7f0254c00640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f023006a460 0x7f023006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:48.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.726+0000 7f0254c00640 1 -- 172.21.15.39:0/3250014200 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0240088df0 con 0x7f02580c6b10 2024-01-24T17:47:48.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.726+0000 7f025d000640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f023006a460 0x7f023006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:48.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.727+0000 7f025d000640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f023006a460 0x7f023006c920 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0248034000 tx=0x7f0248009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:48.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.730+0000 7f0254c00640 1 -- 172.21.15.39:0/3250014200 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f02400543f0 con 0x7f02580c6b10 2024-01-24T17:47:48.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.923+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f02580c8530 con 0x7f02580c6b10 2024-01-24T17:47:48.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.924+0000 7f0254c00640 1 -- 172.21.15.39:0/3250014200 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0240057a10 con 0x7f02580c6b10 2024-01-24T17:47:48.925 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:47:48.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f023006a460 msgr2=0x7f023006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f023006a460 0x7f023006c920 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0248034000 tx=0x7f0248009030 comp rx=0 tx=0).stop 2024-01-24T17:47:48.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02580c6b10 msgr2=0x7f02581475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:48.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02580c6b10 0x7f02581475b0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f0240010890 tx=0x7f0240010d50 comp rx=0 tx=0).stop 2024-01-24T17:47:48.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 shutdown_connections 2024-01-24T17:47:48.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f02581495e0 0x7f02580c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f023006a460 0x7f023006c920 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02580c6b10 0x7f02581475b0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 --2- 172.21.15.39:0/3250014200 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f02380039a0 0x7f02580b2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:48.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 >> 172.21.15.39:0/3250014200 conn(0x7f02580cf4a0 msgr2=0x7f02580ac5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:48.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 shutdown_connections 2024-01-24T17:47:48.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:48.927+0000 7f025e400640 1 -- 172.21.15.39:0/3250014200 wait complete. 2024-01-24T17:47:48.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:47:49.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.172+0000 7f2c7dc00640 1 Processor -- start 2024-01-24T17:47:49.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.172+0000 7f2c7dc00640 1 -- start start 2024-01-24T17:47:49.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.173+0000 7f2c7dc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2c78071e40 0x7f2c78070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.173+0000 7f2c7dc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2c780cb250 0x7f2c780cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.173+0000 7f2c7dc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c780cbf90 con 0x7f2c78159530 2024-01-24T17:47:49.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.173+0000 7f2c7dc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c780cc270 con 0x7f2c780cb250 2024-01-24T17:47:49.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.173+0000 7f2c7dc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c780cc550 con 0x7f2c78071e40 2024-01-24T17:47:49.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.173+0000 7f2c77e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2c78159530 0x7f2c780cab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:43358/0 (socket says 172.21.15.39:43358) 2024-01-24T17:47:49.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.173+0000 7f2c77e00640 1 -- 172.21.15.39:0/2556698546 learned_addr learned my addr 172.21.15.39:0/2556698546 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:49.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.173+0000 7f2c7d200640 1 --2- 172.21.15.39:0/2556698546 >> v2:172.21.15.39:3300/0 conn(0x7f2c780cb250 0x7f2c780cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.174+0000 7f2c77400640 1 --2- 172.21.15.39:0/2556698546 >> v2:172.21.15.100:3300/0 conn(0x7f2c78071e40 0x7f2c78070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.174+0000 7f2c76a00640 1 -- 172.21.15.39:0/2556698546 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66873978 0 0) 0x7f2c780cbf90 con 0x7f2c78159530 2024-01-24T17:47:49.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.174+0000 7f2c7d200640 1 -- 172.21.15.39:0/2556698546 >> v2:172.21.15.100:3300/0 conn(0x7f2c78071e40 msgr2=0x7f2c78070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:49.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.174+0000 7f2c7d200640 1 --2- 172.21.15.39:0/2556698546 >> v2:172.21.15.100:3300/0 conn(0x7f2c78071e40 0x7f2c78070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.174+0000 7f2c7d200640 1 -- 172.21.15.39:0/2556698546 >> v1:172.21.15.39:6789/0 conn(0x7f2c78159530 legacy=0x7f2c780cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.174+0000 7f2c7d200640 1 -- 172.21.15.39:0/2556698546 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c780cc830 con 0x7f2c780cb250 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.175+0000 7f2c7d200640 1 --2- 172.21.15.39:0/2556698546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c780cb250 0x7f2c780cd030 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2c7806e080 tx=0x7f2c7002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b226188bfd958065 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.175+0000 7f2c76a00640 1 -- 172.21.15.39:0/2556698546 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7002cc80 con 0x7f2c780cb250 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.175+0000 7f2c76a00640 1 -- 172.21.15.39:0/2556698546 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7002cde0 con 0x7f2c780cb250 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.175+0000 7f2c76a00640 1 -- 172.21.15.39:0/2556698546 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70034660 con 0x7f2c780cb250 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.176+0000 7f2c7dc00640 1 -- 172.21.15.39:0/2556698546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c780cb250 msgr2=0x7f2c780cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.176+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/2556698546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c780cb250 0x7f2c780cd030 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2c7806e080 tx=0x7f2c7002bed0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.176+0000 7f2c7dc00640 1 -- 172.21.15.39:0/2556698546 shutdown_connections 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.176+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/2556698546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c780cb250 0x7f2c780cd030 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.176+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/2556698546 >> v2:172.21.15.100:3300/0 conn(0x7f2c78071e40 0x7f2c78070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.176+0000 7f2c7dc00640 1 -- 172.21.15.39:0/2556698546 >> 172.21.15.39:0/2556698546 conn(0x7f2c7806a200 msgr2=0x7f2c780affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.176+0000 7f2c7dc00640 1 -- 172.21.15.39:0/2556698546 shutdown_connections 2024-01-24T17:47:49.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.176+0000 7f2c7dc00640 1 -- 172.21.15.39:0/2556698546 wait complete. 2024-01-24T17:47:49.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.178+0000 7f2c7dc00640 1 Processor -- start 2024-01-24T17:47:49.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.178+0000 7f2c7dc00640 1 -- start start 2024-01-24T17:47:49.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.179+0000 7f2c7dc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c780cb250 0x7f2c781500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.179+0000 7f2c7dc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c78159530 0x7f2c7814e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.179+0000 7f2c7dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c78159a90 0x7f2c7814f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.179+0000 7f2c7dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78152e70 con 0x7f2c78159a90 2024-01-24T17:47:49.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.179+0000 7f2c7dc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78151850 con 0x7f2c780cb250 2024-01-24T17:47:49.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.179+0000 7f2c7dc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c781507b0 con 0x7f2c78159530 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.179+0000 7f2c7d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c78159a90 0x7f2c7814f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c7d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c78159a90 0x7f2c7814f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46660/0 (socket says 172.21.15.39:46660) 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c7d200640 1 -- 172.21.15.39:0/551090021 learned_addr learned my addr 172.21.15.39:0/551090021 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c77e00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c780cb250 0x7f2c781500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c77400640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c78159530 0x7f2c7814e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c7d200640 1 -- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c78159530 msgr2=0x7f2c7814e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c7d200640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c78159530 0x7f2c7814e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c7d200640 1 -- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c780cb250 msgr2=0x7f2c781500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c7d200640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c780cb250 0x7f2c781500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c7d200640 1 -- 172.21.15.39:0/551090021 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c70009a70 con 0x7f2c78159a90 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.180+0000 7f2c7d200640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c78159a90 0x7f2c7814f4d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2c70037040 tx=0x7f2c70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.181+0000 7f2c6fe00640 1 -- 172.21.15.39:0/551090021 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70040650 con 0x7f2c78159a90 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.181+0000 7f2c6fe00640 1 -- 172.21.15.39:0/551090021 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c700407b0 con 0x7f2c78159a90 2024-01-24T17:47:49.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.181+0000 7f2c6fe00640 1 -- 172.21.15.39:0/551090021 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70040b60 con 0x7f2c78159a90 2024-01-24T17:47:49.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.181+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c78150a90 con 0x7f2c78159a90 2024-01-24T17:47:49.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.181+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c78151050 con 0x7f2c78159a90 2024-01-24T17:47:49.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.182+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c780738c0 con 0x7f2c78159a90 2024-01-24T17:47:49.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.183+0000 7f2c6fe00640 1 -- 172.21.15.39:0/551090021 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2c70040cc0 con 0x7f2c78159a90 2024-01-24T17:47:49.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.184+0000 7f2c6fe00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c5006a460 0x7f2c5006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.184+0000 7f2c6fe00640 1 -- 172.21.15.39:0/551090021 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c700ab4a0 con 0x7f2c78159a90 2024-01-24T17:47:49.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.185+0000 7f2c77e00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c5006a460 0x7f2c5006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.186+0000 7f2c77e00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c5006a460 0x7f2c5006c920 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2c68000f60 tx=0x7f2c68007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:49.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.189+0000 7f2c6fe00640 1 -- 172.21.15.39:0/551090021 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2c70076a90 con 0x7f2c78159a90 2024-01-24T17:47:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.384+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2c7804e840 con 0x7f2c78159a90 2024-01-24T17:47:49.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.385+0000 7f2c6fe00640 1 -- 172.21.15.39:0/551090021 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2c70009450 con 0x7f2c78159a90 2024-01-24T17:47:49.386 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c5006a460 msgr2=0x7f2c5006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c5006a460 0x7f2c5006c920 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2c68000f60 tx=0x7f2c68007c00 comp rx=0 tx=0).stop 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c78159a90 msgr2=0x7f2c7814f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c78159a90 0x7f2c7814f4d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2c70037040 tx=0x7f2c70004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 shutdown_connections 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c5006a460 0x7f2c5006c920 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c78159a90 0x7f2c7814f4d0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2c78159530 0x7f2c7814e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 --2- 172.21.15.39:0/551090021 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c780cb250 0x7f2c781500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 >> 172.21.15.39:0/551090021 conn(0x7f2c7806a200 msgr2=0x7f2c781555b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:49.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 shutdown_connections 2024-01-24T17:47:49.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.388+0000 7f2c7dc00640 1 -- 172.21.15.39:0/551090021 wait complete. 2024-01-24T17:47:49.408 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:47:49.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.625+0000 7f475e400640 1 Processor -- start 2024-01-24T17:47:49.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.625+0000 7f475e400640 1 -- start start 2024-01-24T17:47:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.627+0000 7f475e400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47580cf8a0 0x7f47580d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.627+0000 7f475e400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f475806acb0 0x7f47580d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.627+0000 7f475e400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47580ce010 con 0x7f475815cbf0 2024-01-24T17:47:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.627+0000 7f475e400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47580ce870 con 0x7f475806acb0 2024-01-24T17:47:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.627+0000 7f475e400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47580ce2b0 con 0x7f47580cf8a0 2024-01-24T17:47:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.627+0000 7f475da00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f475806acb0 0x7f47580d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.628+0000 7f4757e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47580cf8a0 0x7f47580d0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.628+0000 7f475da00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f475806acb0 0x7f47580d0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:37616/0 (socket says 172.21.15.39:37616) 2024-01-24T17:47:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.628+0000 7f475da00640 1 -- 172.21.15.39:0/83145435 learned_addr learned my addr 172.21.15.39:0/83145435 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:49.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.628+0000 7f4757e00640 1 -- 172.21.15.39:0/83145435 >> v2:172.21.15.100:3300/0 conn(0x7f475806acb0 msgr2=0x7f47580d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:49.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.628+0000 7f4757e00640 1 --2- 172.21.15.39:0/83145435 >> v2:172.21.15.100:3300/0 conn(0x7f475806acb0 0x7f47580d0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.628+0000 7f4757e00640 1 -- 172.21.15.39:0/83145435 >> v1:172.21.15.100:6789/0 conn(0x7f475815cbf0 legacy=0x7f47580cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.628+0000 7f4757e00640 1 -- 172.21.15.39:0/83145435 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47580ce590 con 0x7f47580cf8a0 2024-01-24T17:47:49.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.629+0000 7f475da00640 1 --2- 172.21.15.39:0/83145435 >> v2:172.21.15.100:3300/0 conn(0x7f475806acb0 0x7f47580d0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:47:49.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.630+0000 7f4757e00640 1 --2- 172.21.15.39:0/83145435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47580cf8a0 0x7f47580d0a90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f47400097a0 tx=0x7f474002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af20d15a6aa0947d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:49.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.630+0000 7f4757400640 1 -- 172.21.15.39:0/83145435 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740031780 con 0x7f47580cf8a0 2024-01-24T17:47:49.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.630+0000 7f4757400640 1 -- 172.21.15.39:0/83145435 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47400318e0 con 0x7f47580cf8a0 2024-01-24T17:47:49.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.630+0000 7f4757400640 1 -- 172.21.15.39:0/83145435 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740031c70 con 0x7f47580cf8a0 2024-01-24T17:47:49.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.631+0000 7f475e400640 1 -- 172.21.15.39:0/83145435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47580cf8a0 msgr2=0x7f47580d0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.631+0000 7f475e400640 1 --2- 172.21.15.39:0/83145435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47580cf8a0 0x7f47580d0a90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f47400097a0 tx=0x7f474002b820 comp rx=0 tx=0).stop 2024-01-24T17:47:49.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.631+0000 7f475e400640 1 -- 172.21.15.39:0/83145435 shutdown_connections 2024-01-24T17:47:49.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.631+0000 7f475e400640 1 --2- 172.21.15.39:0/83145435 >> v2:172.21.15.100:3300/0 conn(0x7f475806acb0 0x7f47580d0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.631+0000 7f475e400640 1 --2- 172.21.15.39:0/83145435 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47580cf8a0 0x7f47580d0a90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.631+0000 7f475e400640 1 -- 172.21.15.39:0/83145435 >> 172.21.15.39:0/83145435 conn(0x7f475809bad0 msgr2=0x7f47580d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:49.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.631+0000 7f475e400640 1 -- 172.21.15.39:0/83145435 shutdown_connections 2024-01-24T17:47:49.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.631+0000 7f475e400640 1 -- 172.21.15.39:0/83145435 wait complete. 2024-01-24T17:47:49.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.633+0000 7f475e400640 1 Processor -- start 2024-01-24T17:47:49.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.633+0000 7f475e400640 1 -- start start 2024-01-24T17:47:49.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f475806acb0 0x7f4758158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475e400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47580cf8a0 0x7f47580d1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475e400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475815cbf0 0x7f47580cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580cb3f0 con 0x7f475806acb0 2024-01-24T17:47:49.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475e400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580ca7e0 con 0x7f47580cf8a0 2024-01-24T17:47:49.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475e400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47580caac0 con 0x7f475815cbf0 2024-01-24T17:47:49.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f475806acb0 0x7f4758158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f475806acb0 0x7f4758158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46674/0 (socket says 172.21.15.39:46674) 2024-01-24T17:47:49.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475d000640 1 -- 172.21.15.39:0/959100283 learned_addr learned my addr 172.21.15.39:0/959100283 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:49.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f4757e00640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47580cf8a0 0x7f47580d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.634+0000 7f475da00640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475815cbf0 0x7f47580cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f475d000640 1 -- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475815cbf0 msgr2=0x7f47580cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f475d000640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475815cbf0 0x7f47580cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f475d000640 1 -- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47580cf8a0 msgr2=0x7f47580d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f475d000640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47580cf8a0 0x7f47580d1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f475d000640 1 -- 172.21.15.39:0/959100283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4750009b30 con 0x7f475806acb0 2024-01-24T17:47:49.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f4757e00640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47580cf8a0 0x7f47580d1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:47:49.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f475d000640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f475806acb0 0x7f4758158de0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f474800c3a0 tx=0x7f474800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:49.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f4754c00640 1 -- 172.21.15.39:0/959100283 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47480052d0 con 0x7f475806acb0 2024-01-24T17:47:49.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f4754c00640 1 -- 172.21.15.39:0/959100283 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748005450 con 0x7f475806acb0 2024-01-24T17:47:49.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f4754c00640 1 -- 172.21.15.39:0/959100283 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748003880 con 0x7f475806acb0 2024-01-24T17:47:49.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.635+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4740009400 con 0x7f475806acb0 2024-01-24T17:47:49.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.636+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47580cb080 con 0x7f475806acb0 2024-01-24T17:47:49.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.637+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475804dff0 con 0x7f475806acb0 2024-01-24T17:47:49.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.638+0000 7f4754c00640 1 -- 172.21.15.39:0/959100283 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f47480039e0 con 0x7f475806acb0 2024-01-24T17:47:49.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.639+0000 7f4754c00640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f473006a460 0x7f473006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:49.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.639+0000 7f4754c00640 1 -- 172.21.15.39:0/959100283 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4748087e30 con 0x7f475806acb0 2024-01-24T17:47:49.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.639+0000 7f4757e00640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f473006a460 0x7f473006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:49.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.641+0000 7f4757e00640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f473006a460 0x7f473006c920 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4740036000 tx=0x7f474003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:49.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.644+0000 7f4754c00640 1 -- 172.21.15.39:0/959100283 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f47480534b0 con 0x7f475806acb0 2024-01-24T17:47:49.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.852+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f47580d0000 con 0x7f475806acb0 2024-01-24T17:47:49.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.853+0000 7f4754c00640 1 -- 172.21.15.39:0/959100283 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v88) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4748056ad0 con 0x7f475806acb0 2024-01-24T17:47:49.854 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:47:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f473006a460 msgr2=0x7f473006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f473006a460 0x7f473006c920 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4740036000 tx=0x7f474003d040 comp rx=0 tx=0).stop 2024-01-24T17:47:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f475806acb0 msgr2=0x7f4758158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f475806acb0 0x7f4758158de0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f474800c3a0 tx=0x7f474800c860 comp rx=0 tx=0).stop 2024-01-24T17:47:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 shutdown_connections 2024-01-24T17:47:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f475815cbf0 0x7f47580cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47580cf8a0 0x7f47580d1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f473006a460 0x7f473006c920 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 --2- 172.21.15.39:0/959100283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f475806acb0 0x7f4758158de0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:49.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.856+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 >> 172.21.15.39:0/959100283 conn(0x7f475809bad0 msgr2=0x7f47580af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:49.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.857+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 shutdown_connections 2024-01-24T17:47:49.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:49.857+0000 7f475e400640 1 -- 172.21.15.39:0/959100283 wait complete. 2024-01-24T17:47:49.876 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:47:49.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:49.876 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:47:49.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[66933]: fuse finished with error 0 and tester_r 0 2024-01-24T17:47:49.974 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:47:51.383 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:47:51.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:47:56.076 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:47:56.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:47:56.077 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:47:56.137 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:47:56.138 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:56.196 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:56.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.424+0000 7f3445c00640 1 Processor -- start 2024-01-24T17:47:56.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.425+0000 7f3445c00640 1 -- start start 2024-01-24T17:47:56.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.426+0000 7f3445c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f34400b22b0 0x7f34400b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.426+0000 7f3445c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f344006afd0 0x7f3440160a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.426+0000 7f3445c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 0x7f3440160f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.426+0000 7f3445c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400b2bd0 con 0x7f344006afd0 2024-01-24T17:47:56.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.426+0000 7f3445c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3440161580 con 0x7f34400b22b0 2024-01-24T17:47:56.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.426+0000 7f3445c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3440161860 con 0x7f344006b460 2024-01-24T17:47:56.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.426+0000 7f3437e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f344006afd0 0x7f3440160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.426+0000 7f3437e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f344006afd0 0x7f3440160a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37346/0 (socket says 172.21.15.39:37346) 2024-01-24T17:47:56.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.427+0000 7f3437e00640 1 -- 172.21.15.39:0/403177389 learned_addr learned my addr 172.21.15.39:0/403177389 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:56.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.427+0000 7f3437e00640 1 -- 172.21.15.39:0/403177389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 msgr2=0x7f3440160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:56.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.427+0000 7f343ea00640 1 --2- 172.21.15.39:0/403177389 >> v2:172.21.15.100:3300/0 conn(0x7f34400b22b0 0x7f34400b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.427+0000 7f3437e00640 1 --2- 172.21.15.39:0/403177389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 0x7f3440160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.427+0000 7f3437e00640 1 -- 172.21.15.39:0/403177389 >> v2:172.21.15.100:3300/0 conn(0x7f34400b22b0 msgr2=0x7f34400b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:56.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.427+0000 7f3437e00640 1 --2- 172.21.15.39:0/403177389 >> v2:172.21.15.100:3300/0 conn(0x7f34400b22b0 0x7f34400b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.427+0000 7f3437e00640 1 -- 172.21.15.39:0/403177389 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34400cb5f0 con 0x7f344006afd0 2024-01-24T17:47:56.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.427+0000 7f343ea00640 1 --2- 172.21.15.39:0/403177389 >> v2:172.21.15.100:3300/0 conn(0x7f34400b22b0 0x7f34400b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.428+0000 7f3437e00640 1 --2- 172.21.15.39:0/403177389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344006afd0 0x7f3440160a10 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f34400ce050 tx=0x7f342802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5c7072004542e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.429+0000 7f343e000640 1 -- 172.21.15.39:0/403177389 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428031870 con 0x7f344006afd0 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.429+0000 7f343e000640 1 -- 172.21.15.39:0/403177389 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34280319d0 con 0x7f344006afd0 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.429+0000 7f343e000640 1 -- 172.21.15.39:0/403177389 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428031d60 con 0x7f344006afd0 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 -- 172.21.15.39:0/403177389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344006afd0 msgr2=0x7f3440160a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 --2- 172.21.15.39:0/403177389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344006afd0 0x7f3440160a10 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f34400ce050 tx=0x7f342802b910 comp rx=0 tx=0).stop 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 -- 172.21.15.39:0/403177389 shutdown_connections 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 --2- 172.21.15.39:0/403177389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 0x7f3440160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 --2- 172.21.15.39:0/403177389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344006afd0 0x7f3440160a10 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 --2- 172.21.15.39:0/403177389 >> v2:172.21.15.100:3300/0 conn(0x7f34400b22b0 0x7f34400b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 -- 172.21.15.39:0/403177389 >> 172.21.15.39:0/403177389 conn(0x7f34400cd2b0 msgr2=0x7f34400cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:56.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 -- 172.21.15.39:0/403177389 shutdown_connections 2024-01-24T17:47:56.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.430+0000 7f3445c00640 1 -- 172.21.15.39:0/403177389 wait complete. 2024-01-24T17:47:56.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.432+0000 7f3445c00640 1 Processor -- start 2024-01-24T17:47:56.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.432+0000 7f3445c00640 1 -- start start 2024-01-24T17:47:56.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f3445c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f344006afd0 0x7f3440152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f3445c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 0x7f34400b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f3445c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34400b22b0 0x7f34400adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f3445c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440153450 con 0x7f34400b22b0 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f3445c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34400b1670 con 0x7f344006afd0 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f3445c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34400acf20 con 0x7f344006b460 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f343f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34400b22b0 0x7f34400adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f343f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34400b22b0 0x7f34400adfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37354/0 (socket says 172.21.15.39:37354) 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f343f400640 1 -- 172.21.15.39:0/250502564 learned_addr learned my addr 172.21.15.39:0/250502564 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f343ea00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f344006afd0 0x7f3440152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.433+0000 7f3437e00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 0x7f34400b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f343f400640 1 -- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 msgr2=0x7f34400b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f343f400640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 0x7f34400b1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f343f400640 1 -- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f344006afd0 msgr2=0x7f3440152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f343f400640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f344006afd0 0x7f3440152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f343f400640 1 -- 172.21.15.39:0/250502564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34280094d0 con 0x7f34400b22b0 2024-01-24T17:47:56.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f343f400640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34400b22b0 0x7f34400adfc0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f343800b970 tx=0x7f343800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:56.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f3436a00640 1 -- 172.21.15.39:0/250502564 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343800cb40 con 0x7f34400b22b0 2024-01-24T17:47:56.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f3436a00640 1 -- 172.21.15.39:0/250502564 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343800ccc0 con 0x7f34400b22b0 2024-01-24T17:47:56.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34400ad200 con 0x7f34400b22b0 2024-01-24T17:47:56.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.434+0000 7f3436a00640 1 -- 172.21.15.39:0/250502564 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438014680 con 0x7f34400b22b0 2024-01-24T17:47:56.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.435+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34400ad7c0 con 0x7f34400b22b0 2024-01-24T17:47:56.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.436+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344006cd60 con 0x7f34400b22b0 2024-01-24T17:47:56.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.437+0000 7f3436a00640 1 -- 172.21.15.39:0/250502564 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f343801d3b0 con 0x7f34400b22b0 2024-01-24T17:47:56.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.438+0000 7f3436a00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f341806a460 0x7f341806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.438+0000 7f3436a00640 1 -- 172.21.15.39:0/250502564 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3438088900 con 0x7f34400b22b0 2024-01-24T17:47:56.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.438+0000 7f343ea00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f341806a460 0x7f341806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.439+0000 7f343ea00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f341806a460 0x7f341806c920 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f343000a950 tx=0x7f3430008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:56.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.443+0000 7f3436a00640 1 -- 172.21.15.39:0/250502564 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3438053e50 con 0x7f34400b22b0 2024-01-24T17:47:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.640+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3440067050 con 0x7f34400b22b0 2024-01-24T17:47:56.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.643+0000 7f3436a00640 1 -- 172.21.15.39:0/250502564 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16104 (secure 0 0 0) 0x7f3438019020 con 0x7f34400b22b0 2024-01-24T17:47:56.645 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:56.645 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:32.941651+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:56.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f341806a460 msgr2=0x7f341806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f341806a460 0x7f341806c920 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f343000a950 tx=0x7f3430008040 comp rx=0 tx=0).stop 2024-01-24T17:47:56.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34400b22b0 msgr2=0x7f34400adfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34400b22b0 0x7f34400adfc0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f343800b970 tx=0x7f343800be30 comp rx=0 tx=0).stop 2024-01-24T17:47:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 shutdown_connections 2024-01-24T17:47:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f341806a460 0x7f341806c920 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34400b22b0 0x7f34400adfc0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f344006b460 0x7f34400b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 --2- 172.21.15.39:0/250502564 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f344006afd0 0x7f3440152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 >> 172.21.15.39:0/250502564 conn(0x7f34400cd2b0 msgr2=0x7f34400cf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.645+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 shutdown_connections 2024-01-24T17:47:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.646+0000 7f3445c00640 1 -- 172.21.15.39:0/250502564 wait complete. 2024-01-24T17:47:56.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:47:30.315286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:47:30.960101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:47:56.667 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:47:56.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.878+0000 7f313b400640 1 Processor -- start 2024-01-24T17:47:56.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.878+0000 7f313b400640 1 -- start start 2024-01-24T17:47:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.880+0000 7f313b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 0x7f3134167f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.880+0000 7f313b400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31340aff70 con 0x7f313406b260 2024-01-24T17:47:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.880+0000 7f313b400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31340b0810 con 0x7f31340ce7b0 2024-01-24T17:47:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.880+0000 7f313b400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3134160620 con 0x7f31340ad2d0 2024-01-24T17:47:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.880+0000 7f3139600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f313406b260 0x7f3134164800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:45794/0 (socket says 172.21.15.39:45794) 2024-01-24T17:47:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.880+0000 7f3139600640 1 -- 172.21.15.39:0/2089340080 learned_addr learned my addr 172.21.15.39:0/2089340080 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f313aa00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 0x7f3134167f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691663225 0 0) 0x7f31340aff70 con 0x7f313406b260 2024-01-24T17:47:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3114003520 con 0x7f313406b260 2024-01-24T17:47:56.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3050583903 0 0) 0x7f3114003520 con 0x7f313406b260 2024-01-24T17:47:56.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 msgr2=0x7f3134167f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 0x7f3134167f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 >> v1:172.21.15.100:6789/0 conn(0x7f31340ce7b0 legacy=0x7f31340ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31340ad760 con 0x7f313406b260 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f311c001670 con 0x7f313406b260 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.881+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 >> v1:172.21.15.39:6789/0 conn(0x7f313406b260 legacy=0x7f3134164800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f3138c00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 0x7f31140039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f3138c00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3114003f20 0x7f31140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f3138c00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3114004d00 0x7f3114005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311400ba30 con 0x7f3114004d00 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311400bba0 con 0x7f3114003f20 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311400bd60 con 0x7f31340ad2d0 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f313aa00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3114004d00 0x7f3114005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f313a000640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 0x7f31140039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f313aa00640 1 -- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 msgr2=0x7f31140039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:56.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f313aa00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 0x7f31140039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f3139600640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3114003f20 0x7f31140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f313aa00640 1 -- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3114003f20 msgr2=0x7f31140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f313aa00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3114003f20 0x7f31140047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f313aa00640 1 -- 172.21.15.39:0/2089340080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3114003520 con 0x7f3114004d00 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.882+0000 7f313a000640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31340ad2d0 0x7f31140039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f313aa00640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3114004d00 0x7f3114005180 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f312c009fc0 tx=0x7f31140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f92cbd7dee4994 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c02cbf0 con 0x7f3114004d00 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c00f040 con 0x7f3114004d00 2024-01-24T17:47:56.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f3138c00640 1 -- 172.21.15.39:0/2089340080 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c034440 con 0x7f3114004d00 2024-01-24T17:47:56.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f313b400640 1 -- 172.21.15.39:0/2089340080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3114004d00 msgr2=0x7f3114005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f313b400640 1 --2- 172.21.15.39:0/2089340080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3114004d00 0x7f3114005180 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f312c009fc0 tx=0x7f31140057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f313aa00640 1 -- 172.21.15.39:0/2089340080 reap_dead start 2024-01-24T17:47:56.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f313b400640 1 -- 172.21.15.39:0/2089340080 shutdown_connections 2024-01-24T17:47:56.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.884+0000 7f313b400640 1 -- 172.21.15.39:0/2089340080 >> 172.21.15.39:0/2089340080 conn(0x7f31340d0640 msgr2=0x7f31340d0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:56.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.885+0000 7f313b400640 1 -- 172.21.15.39:0/2089340080 shutdown_connections 2024-01-24T17:47:56.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.885+0000 7f313b400640 1 -- 172.21.15.39:0/2089340080 wait complete. 2024-01-24T17:47:56.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.886+0000 7f313b400640 1 Processor -- start 2024-01-24T17:47:56.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.887+0000 7f313b400640 1 -- start start 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.887+0000 7f313b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3114004d00 0x7f3134155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.887+0000 7f313b400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31340c81d0 0x7f3134156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3134157550 0x7f313415f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313b400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31340c8720 con 0x7f3134157550 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313b400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31340c8a00 con 0x7f31340c81d0 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313b400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3134156410 con 0x7f3114004d00 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3134157550 0x7f313415f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3134157550 0x7f313415f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37380/0 (socket says 172.21.15.39:37380) 2024-01-24T17:47:56.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313aa00640 1 -- 172.21.15.39:0/1072613318 learned_addr learned my addr 172.21.15.39:0/1072613318 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f3139600640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31340c81d0 0x7f3134156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313a000640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3114004d00 0x7f3134155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313aa00640 1 -- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3114004d00 msgr2=0x7f3134155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313aa00640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3114004d00 0x7f3134155b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313aa00640 1 -- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31340c81d0 msgr2=0x7f3134156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313aa00640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31340c81d0 0x7f3134156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f313aa00640 1 -- 172.21.15.39:0/1072613318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31341566f0 con 0x7f3134157550 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.888+0000 7f3139600640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31340c81d0 0x7f3134156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.889+0000 7f313aa00640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3134157550 0x7f313415f710 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f312c002230 tx=0x7f312c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.889+0000 7f3132000640 1 -- 172.21.15.39:0/1072613318 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c043070 con 0x7f3134157550 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.889+0000 7f3132000640 1 -- 172.21.15.39:0/1072613318 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c02de80 con 0x7f3134157550 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.889+0000 7f3132000640 1 -- 172.21.15.39:0/1072613318 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c034d10 con 0x7f3134157550 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.889+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3134156070 con 0x7f3134157550 2024-01-24T17:47:56.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.889+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31340c8ce0 con 0x7f3134157550 2024-01-24T17:47:56.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.890+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313404e840 con 0x7f3134157550 2024-01-24T17:47:56.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.891+0000 7f3132000640 1 -- 172.21.15.39:0/1072613318 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f312c03e440 con 0x7f3134157550 2024-01-24T17:47:56.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.893+0000 7f3132000640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f310c06a430 0x7f310c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:56.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.893+0000 7f3132000640 1 -- 172.21.15.39:0/1072613318 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f312c0aa470 con 0x7f3134157550 2024-01-24T17:47:56.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.893+0000 7f313a000640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f310c06a430 0x7f310c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:56.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.895+0000 7f313a000640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f310c06a430 0x7f310c06c8f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3124001630 tx=0x7f3124009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:56.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:56.897+0000 7f3132000640 1 -- 172.21.15.39:0/1072613318 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f312c075af0 con 0x7f3134157550 2024-01-24T17:47:57.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.094+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f313415bab0 con 0x7f3134157550 2024-01-24T17:47:57.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.094+0000 7f3132000640 1 -- 172.21.15.39:0/1072613318 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f312c079110 con 0x7f3134157550 2024-01-24T17:47:57.096 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:47:57.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f310c06a430 msgr2=0x7f310c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f310c06a430 0x7f310c06c8f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3124001630 tx=0x7f3124009040 comp rx=0 tx=0).stop 2024-01-24T17:47:57.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3134157550 msgr2=0x7f313415f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3134157550 0x7f313415f710 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f312c002230 tx=0x7f312c009750 comp rx=0 tx=0).stop 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 shutdown_connections 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f310c06a430 0x7f310c06c8f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3134157550 0x7f313415f710 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f31340c81d0 0x7f3134156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 --2- 172.21.15.39:0/1072613318 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3114004d00 0x7f3134155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.097+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 >> 172.21.15.39:0/1072613318 conn(0x7f31340d0640 msgr2=0x7f313406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.098+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 shutdown_connections 2024-01-24T17:47:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.098+0000 7f313b400640 1 -- 172.21.15.39:0/1072613318 wait complete. 2024-01-24T17:47:57.118 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:47:57.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.345+0000 7fa9b8200640 1 Processor -- start 2024-01-24T17:47:57.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.345+0000 7fa9b8200640 1 -- start start 2024-01-24T17:47:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.346+0000 7fa9b8200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa9b00cb990 0x7fa9b00ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.346+0000 7fa9b8200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00ca4c0 con 0x7fa9b015aa60 2024-01-24T17:47:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.346+0000 7fa9b8200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b006d9e0 con 0x7fa9b006cec0 2024-01-24T17:47:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.346+0000 7fa9b8200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b00cae90 con 0x7fa9b00cb990 2024-01-24T17:47:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.346+0000 7fa9b4e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa9b00cb990 0x7fa9b00ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.346+0000 7fa9b4e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa9b00cb990 0x7fa9b00ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37392/0 (socket says 172.21.15.39:37392) 2024-01-24T17:47:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.346+0000 7fa9b4e00640 1 -- 172.21.15.39:0/472544197 learned_addr learned my addr 172.21.15.39:0/472544197 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:57.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.346+0000 7fa9b5800640 1 --1- 172.21.15.39:0/472544197 >> v1:172.21.15.39:6789/0 conn(0x7fa9b015aa60 0x7fa9b00cb280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:45796/0 (socket says 172.21.15.39:45796) 2024-01-24T17:47:57.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.347+0000 7fa9b4e00640 1 -- 172.21.15.39:0/472544197 >> v1:172.21.15.100:6789/0 conn(0x7fa9b006cec0 legacy=0x7fa9b006d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:57.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.347+0000 7fa9b4e00640 1 -- 172.21.15.39:0/472544197 >> v1:172.21.15.39:6789/0 conn(0x7fa9b015aa60 legacy=0x7fa9b00cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.347+0000 7fa9b4e00640 1 -- 172.21.15.39:0/472544197 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b006dc90 con 0x7fa9b00cb990 2024-01-24T17:47:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.348+0000 7fa9b4e00640 1 --2- 172.21.15.39:0/472544197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b00cb990 0x7fa9b00ca950 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa998009870 tx=0x7fa99802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f0a4827dbfa07c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.348+0000 7fa9afe00640 1 -- 172.21.15.39:0/472544197 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998030860 con 0x7fa9b00cb990 2024-01-24T17:47:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.348+0000 7fa9afe00640 1 -- 172.21.15.39:0/472544197 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9980309c0 con 0x7fa9b00cb990 2024-01-24T17:47:57.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.348+0000 7fa9afe00640 1 -- 172.21.15.39:0/472544197 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998030d50 con 0x7fa9b00cb990 2024-01-24T17:47:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.349+0000 7fa9b8200640 1 -- 172.21.15.39:0/472544197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b00cb990 msgr2=0x7fa9b00ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.349+0000 7fa9b8200640 1 --2- 172.21.15.39:0/472544197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b00cb990 0x7fa9b00ca950 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa998009870 tx=0x7fa99802b9a0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.349+0000 7fa9b8200640 1 -- 172.21.15.39:0/472544197 shutdown_connections 2024-01-24T17:47:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.349+0000 7fa9b8200640 1 --2- 172.21.15.39:0/472544197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b00cb990 0x7fa9b00ca950 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.349+0000 7fa9b8200640 1 -- 172.21.15.39:0/472544197 >> 172.21.15.39:0/472544197 conn(0x7fa9b00ce9e0 msgr2=0x7fa9b00cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.349+0000 7fa9b8200640 1 -- 172.21.15.39:0/472544197 shutdown_connections 2024-01-24T17:47:57.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.349+0000 7fa9b8200640 1 -- 172.21.15.39:0/472544197 wait complete. 2024-01-24T17:47:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.351+0000 7fa9b8200640 1 Processor -- start 2024-01-24T17:47:57.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.351+0000 7fa9b8200640 1 -- start start 2024-01-24T17:47:57.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b8200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b006cec0 0x7fa9b0155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b8200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b00cb990 0x7fa9b00ac570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b8200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b015aa60 0x7fa9b00cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b8200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b01564e0 con 0x7fa9b006cec0 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b8200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b0154f00 con 0x7fa9b015aa60 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b8200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b01551e0 con 0x7fa9b00cb990 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b006cec0 0x7fa9b0155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b006cec0 0x7fa9b0155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37396/0 (socket says 172.21.15.39:37396) 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b5800640 1 -- 172.21.15.39:0/1914478291 learned_addr learned my addr 172.21.15.39:0/1914478291 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.352+0000 7fa9b6200640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b015aa60 0x7fa9b00cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9b4e00640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b00cb990 0x7fa9b00ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9b5800640 1 -- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b00cb990 msgr2=0x7fa9b00ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9b5800640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b00cb990 0x7fa9b00ac570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9b5800640 1 -- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b015aa60 msgr2=0x7fa9b00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9b5800640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b015aa60 0x7fa9b00cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9b5800640 1 -- 172.21.15.39:0/1914478291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980094d0 con 0x7fa9b006cec0 2024-01-24T17:47:57.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9b5800640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b006cec0 0x7fa9b0155fa0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c4d0 tx=0x7fa9a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:57.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9ad600640 1 -- 172.21.15.39:0/1914478291 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0007480 con 0x7fa9b006cec0 2024-01-24T17:47:57.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9ad600640 1 -- 172.21.15.39:0/1914478291 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a00075e0 con 0x7fa9b006cec0 2024-01-24T17:47:57.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9ad600640 1 -- 172.21.15.39:0/1914478291 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00049e0 con 0x7fa9b006cec0 2024-01-24T17:47:57.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.353+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b01554c0 con 0x7fa9b006cec0 2024-01-24T17:47:57.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.354+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b0155a80 con 0x7fa9b006cec0 2024-01-24T17:47:57.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.355+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b006c210 con 0x7fa9b006cec0 2024-01-24T17:47:57.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.355+0000 7fa9ad600640 1 -- 172.21.15.39:0/1914478291 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa9a0004b40 con 0x7fa9b006cec0 2024-01-24T17:47:57.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.356+0000 7fa9ad600640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa98806a430 0x7fa98806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.356+0000 7fa9ad600640 1 -- 172.21.15.39:0/1914478291 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9a0087930 con 0x7fa9b006cec0 2024-01-24T17:47:57.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.357+0000 7fa9b4e00640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa98806a430 0x7fa98806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.358+0000 7fa9b4e00640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa98806a430 0x7fa98806c8f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa998036000 tx=0x7fa99803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:57.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.361+0000 7fa9ad600640 1 -- 172.21.15.39:0/1914478291 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa9a0053060 con 0x7fa9b006cec0 2024-01-24T17:47:57.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.559+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa9b004dff0 con 0x7fa9b006cec0 2024-01-24T17:47:57.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.560+0000 7fa9ad600640 1 -- 172.21.15.39:0/1914478291 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa9a00180a0 con 0x7fa9b006cec0 2024-01-24T17:47:57.562 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:47:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa98806a430 msgr2=0x7fa98806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa98806a430 0x7fa98806c8f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa998036000 tx=0x7fa99803b040 comp rx=0 tx=0).stop 2024-01-24T17:47:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b006cec0 msgr2=0x7fa9b0155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b006cec0 0x7fa9b0155fa0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c4d0 tx=0x7fa9a000c990 comp rx=0 tx=0).stop 2024-01-24T17:47:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 shutdown_connections 2024-01-24T17:47:57.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9b015aa60 0x7fa9b00cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9b00cb990 0x7fa9b00ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa98806a430 0x7fa98806c8f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 --2- 172.21.15.39:0/1914478291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b006cec0 0x7fa9b0155fa0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 >> 172.21.15.39:0/1914478291 conn(0x7fa9b00ce9e0 msgr2=0x7fa9b00af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.563+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 shutdown_connections 2024-01-24T17:47:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.564+0000 7fa9b8200640 1 -- 172.21.15.39:0/1914478291 wait complete. 2024-01-24T17:47:57.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:47:57.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.800+0000 7fd89fe00640 1 Processor -- start 2024-01-24T17:47:57.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.800+0000 7fd89fe00640 1 -- start start 2024-01-24T17:47:57.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.802+0000 7fd89fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd8a0159530 0x7fd8a0070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.802+0000 7fd89fe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a00cbf90 con 0x7fd8a0071e40 2024-01-24T17:47:57.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.802+0000 7fd89fe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a00cc270 con 0x7fd8a00cb250 2024-01-24T17:47:57.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.802+0000 7fd89fe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a00cc550 con 0x7fd8a0159530 2024-01-24T17:47:57.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.802+0000 7fd89e000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd8a0071e40 0x7fd8a00cab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:45798/0 (socket says 172.21.15.39:45798) 2024-01-24T17:47:57.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.802+0000 7fd89e000640 1 -- 172.21.15.39:0/3505431340 learned_addr learned my addr 172.21.15.39:0/3505431340 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:57.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.802+0000 7fd89ea00640 1 --2- 172.21.15.39:0/3505431340 >> v2:172.21.15.100:3300/0 conn(0x7fd8a0159530 0x7fd8a0070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960197783 0 0) 0x7fd8a00cbf90 con 0x7fd8a0071e40 2024-01-24T17:47:57.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd880003520 con 0x7fd8a0071e40 2024-01-24T17:47:57.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 499610359 0 0) 0x7fd880003520 con 0x7fd8a0071e40 2024-01-24T17:47:57.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 >> v2:172.21.15.100:3300/0 conn(0x7fd8a0159530 msgr2=0x7fd8a0070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:57.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 --2- 172.21.15.39:0/3505431340 >> v2:172.21.15.100:3300/0 conn(0x7fd8a0159530 0x7fd8a0070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 >> v1:172.21.15.100:6789/0 conn(0x7fd8a00cb250 legacy=0x7fd8a00c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00cc830 con 0x7fd8a0071e40 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fd888001670 con 0x7fd8a0071e40 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 >> v1:172.21.15.39:6789/0 conn(0x7fd8a0071e40 legacy=0x7fd8a00cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd880003970 0x7fd880003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8800047c0 0x7fd880004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880005180 0x7fd88000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00cc550 con 0x7fd880003970 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88000c020 con 0x7fd880005180 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.803+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88000c1e0 con 0x7fd8800047c0 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89ea00640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd880003970 0x7fd880003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89e000640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8800047c0 0x7fd880004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89f400640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880005180 0x7fd88000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89ea00640 1 -- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8800047c0 msgr2=0x7fd880004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:57.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89ea00640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8800047c0 0x7fd880004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89ea00640 1 -- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880005180 msgr2=0x7fd88000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:57.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89ea00640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880005180 0x7fd88000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89ea00640 1 -- 172.21.15.39:0/3505431340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd880003520 con 0x7fd880003970 2024-01-24T17:47:57.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89f400640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880005180 0x7fd88000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:57.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.804+0000 7fd89f400640 1 -- 172.21.15.39:0/3505431340 reap_dead start 2024-01-24T17:47:57.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.805+0000 7fd89ea00640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd880003970 0x7fd880003d90 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fd890009ac0 tx=0x7fd8800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8adb054a3706bf5d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:57.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89002c6b0 con 0x7fd880003970 2024-01-24T17:47:57.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89002c810 con 0x7fd880003970 2024-01-24T17:47:57.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89d600640 1 -- 172.21.15.39:0/3505431340 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89002cba0 con 0x7fd880003970 2024-01-24T17:47:57.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89fe00640 1 -- 172.21.15.39:0/3505431340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd880003970 msgr2=0x7fd880003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89fe00640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd880003970 0x7fd880003d90 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fd890009ac0 tx=0x7fd8800057f0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89fe00640 1 -- 172.21.15.39:0/3505431340 shutdown_connections 2024-01-24T17:47:57.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89fe00640 1 --2- 172.21.15.39:0/3505431340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd880003970 0x7fd880003d90 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89fe00640 1 -- 172.21.15.39:0/3505431340 >> 172.21.15.39:0/3505431340 conn(0x7fd8a006a200 msgr2=0x7fd8a00affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:57.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89fe00640 1 -- 172.21.15.39:0/3505431340 shutdown_connections 2024-01-24T17:47:57.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.806+0000 7fd89fe00640 1 -- 172.21.15.39:0/3505431340 wait complete. 2024-01-24T17:47:57.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.808+0000 7fd89fe00640 1 Processor -- start 2024-01-24T17:47:57.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.808+0000 7fd89fe00640 1 -- start start 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.809+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880003970 0x7fd8a00c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.809+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8a00c9850 0x7fd8a00c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.809+0000 7fd89fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a0156f60 0x7fd8a01573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.809+0000 7fd89fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00ca1f0 con 0x7fd8a0156f60 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.809+0000 7fd89fe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a0154740 con 0x7fd880003970 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.809+0000 7fd89fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a0157920 con 0x7fd8a00c9850 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a0156f60 0x7fd8a01573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a0156f60 0x7fd8a01573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37410/0 (socket says 172.21.15.39:37410) 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89f400640 1 -- 172.21.15.39:0/1630342179 learned_addr learned my addr 172.21.15.39:0/1630342179 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89ea00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880003970 0x7fd8a00c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89e000640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8a00c9850 0x7fd8a00c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89f400640 1 -- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8a00c9850 msgr2=0x7fd8a00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89f400640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8a00c9850 0x7fd8a00c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89f400640 1 -- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880003970 msgr2=0x7fd8a00c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89f400640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880003970 0x7fd8a00c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89f400640 1 -- 172.21.15.39:0/1630342179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a0154990 con 0x7fd8a0156f60 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.810+0000 7fd89e000640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8a00c9850 0x7fd8a00c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.811+0000 7fd89f400640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a0156f60 0x7fd8a01573e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00b0b50 tx=0x7fd89800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.811+0000 7fd896a00640 1 -- 172.21.15.39:0/1630342179 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898003d80 con 0x7fd8a0156f60 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.811+0000 7fd896a00640 1 -- 172.21.15.39:0/1630342179 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898005070 con 0x7fd8a0156f60 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.811+0000 7fd896a00640 1 -- 172.21.15.39:0/1630342179 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898005420 con 0x7fd8a0156f60 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.811+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a0154c70 con 0x7fd8a0156f60 2024-01-24T17:47:57.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.811+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a0155230 con 0x7fd8a0156f60 2024-01-24T17:47:57.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.812+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a00738c0 con 0x7fd8a0156f60 2024-01-24T17:47:57.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.813+0000 7fd896a00640 1 -- 172.21.15.39:0/1630342179 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd898005580 con 0x7fd8a0156f60 2024-01-24T17:47:57.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.814+0000 7fd896a00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87806a430 0x7fd87806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:57.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.814+0000 7fd896a00640 1 -- 172.21.15.39:0/1630342179 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8980881f0 con 0x7fd8a0156f60 2024-01-24T17:47:57.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.814+0000 7fd89ea00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87806a430 0x7fd87806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:57.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.816+0000 7fd89ea00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87806a430 0x7fd87806c8f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd890036000 tx=0x7fd890030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:57.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:57.818+0000 7fd896a00640 1 -- 172.21.15.39:0/1630342179 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd8980537d0 con 0x7fd8a0156f60 2024-01-24T17:47:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.007+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8a0155450 con 0x7fd8a0156f60 2024-01-24T17:47:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.008+0000 7fd896a00640 1 -- 172.21.15.39:0/1630342179 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd898005830 con 0x7fd8a0156f60 2024-01-24T17:47:58.009 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:47:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.010+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87806a430 msgr2=0x7fd87806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.010+0000 7fd89fe00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87806a430 0x7fd87806c8f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd890036000 tx=0x7fd890030000 comp rx=0 tx=0).stop 2024-01-24T17:47:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.010+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a0156f60 msgr2=0x7fd8a01573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.010+0000 7fd89fe00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a0156f60 0x7fd8a01573e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00b0b50 tx=0x7fd89800d9d0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.011+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 shutdown_connections 2024-01-24T17:47:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.011+0000 7fd89fe00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd87806a430 0x7fd87806c8f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.011+0000 7fd89fe00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8a0156f60 0x7fd8a01573e0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.011+0000 7fd89fe00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd8a00c9850 0x7fd8a00c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.011+0000 7fd89fe00640 1 --2- 172.21.15.39:0/1630342179 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd880003970 0x7fd8a00c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.011+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 >> 172.21.15.39:0/1630342179 conn(0x7fd8a006a200 msgr2=0x7fd8a00adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:58.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.011+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 shutdown_connections 2024-01-24T17:47:58.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.011+0000 7fd89fe00640 1 -- 172.21.15.39:0/1630342179 wait complete. 2024-01-24T17:47:58.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:47:58.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.245+0000 7f1e2d600640 1 Processor -- start 2024-01-24T17:47:58.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.246+0000 7f1e2d600640 1 -- start start 2024-01-24T17:47:58.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.247+0000 7f1e2d600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1e2815cbf0 0x7f1e280cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.247+0000 7f1e2d600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1e280b1570 0x7f1e280cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.247+0000 7f1e2d600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e280ce010 con 0x7f1e2806acb0 2024-01-24T17:47:58.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.247+0000 7f1e2d600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e280ce870 con 0x7f1e280b1570 2024-01-24T17:47:58.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.247+0000 7f1e2d600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e280ce2b0 con 0x7f1e2815cbf0 2024-01-24T17:47:58.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.248+0000 7f1e2cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1e280b1570 0x7f1e280cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.248+0000 7f1e2cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1e280b1570 0x7f1e280cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37418/0 (socket says 172.21.15.39:37418) 2024-01-24T17:47:58.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.248+0000 7f1e2cc00640 1 -- 172.21.15.39:0/213714092 learned_addr learned my addr 172.21.15.39:0/213714092 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:58.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.248+0000 7f1e27e00640 1 --2- 172.21.15.39:0/213714092 >> v2:172.21.15.100:3300/0 conn(0x7f1e2815cbf0 0x7f1e280cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.248+0000 7f1e2cc00640 1 -- 172.21.15.39:0/213714092 >> v2:172.21.15.100:3300/0 conn(0x7f1e2815cbf0 msgr2=0x7f1e280cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:58.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.248+0000 7f1e2cc00640 1 --2- 172.21.15.39:0/213714092 >> v2:172.21.15.100:3300/0 conn(0x7f1e2815cbf0 0x7f1e280cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.248+0000 7f1e2cc00640 1 -- 172.21.15.39:0/213714092 >> v1:172.21.15.100:6789/0 conn(0x7f1e2806acb0 legacy=0x7f1e281605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.249+0000 7f1e2cc00640 1 -- 172.21.15.39:0/213714092 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280ce590 con 0x7f1e280b1570 2024-01-24T17:47:58.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.250+0000 7f1e2cc00640 1 --2- 172.21.15.39:0/213714092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e280b1570 0x7f1e280cf6d0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f1e280d03f0 tx=0x7f1e2002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efed602752d5ca29 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:58.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.250+0000 7f1e27400640 1 -- 172.21.15.39:0/213714092 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2002ca80 con 0x7f1e280b1570 2024-01-24T17:47:58.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.250+0000 7f1e27400640 1 -- 172.21.15.39:0/213714092 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2002cbe0 con 0x7f1e280b1570 2024-01-24T17:47:58.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.250+0000 7f1e27400640 1 -- 172.21.15.39:0/213714092 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e200343f0 con 0x7f1e280b1570 2024-01-24T17:47:58.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.251+0000 7f1e2d600640 1 -- 172.21.15.39:0/213714092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e280b1570 msgr2=0x7f1e280cf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.251+0000 7f1e2d600640 1 --2- 172.21.15.39:0/213714092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e280b1570 0x7f1e280cf6d0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f1e280d03f0 tx=0x7f1e2002bee0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.251+0000 7f1e2d600640 1 -- 172.21.15.39:0/213714092 shutdown_connections 2024-01-24T17:47:58.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.251+0000 7f1e2d600640 1 --2- 172.21.15.39:0/213714092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e280b1570 0x7f1e280cf6d0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.251+0000 7f1e2d600640 1 --2- 172.21.15.39:0/213714092 >> v2:172.21.15.100:3300/0 conn(0x7f1e2815cbf0 0x7f1e280cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.251+0000 7f1e2d600640 1 -- 172.21.15.39:0/213714092 >> 172.21.15.39:0/213714092 conn(0x7f1e2809bad0 msgr2=0x7f1e280d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:58.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.251+0000 7f1e2d600640 1 -- 172.21.15.39:0/213714092 shutdown_connections 2024-01-24T17:47:58.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.251+0000 7f1e2d600640 1 -- 172.21.15.39:0/213714092 wait complete. 2024-01-24T17:47:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.253+0000 7f1e2d600640 1 Processor -- start 2024-01-24T17:47:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.253+0000 7f1e2d600640 1 -- start start 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.254+0000 7f1e2d600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e2806acb0 0x7f1e281f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.254+0000 7f1e2d600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e280b1570 0x7f1e280cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.254+0000 7f1e2d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e2815cbf0 0x7f1e281508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.254+0000 7f1e2d600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2815d100 con 0x7f1e2815cbf0 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.254+0000 7f1e2d600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2815e6c0 con 0x7f1e280b1570 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.254+0000 7f1e2d600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2815e9a0 con 0x7f1e2806acb0 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e2815cbf0 0x7f1e281508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e2815cbf0 0x7f1e281508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37428/0 (socket says 172.21.15.39:37428) 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 -- 172.21.15.39:0/3409503204 learned_addr learned my addr 172.21.15.39:0/3409503204 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e1fe00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e280b1570 0x7f1e280cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e27e00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e2806acb0 0x7f1e281f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 -- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e2806acb0 msgr2=0x7f1e281f1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e2806acb0 0x7f1e281f1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 -- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e280b1570 msgr2=0x7f1e280cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e280b1570 0x7f1e280cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 -- 172.21.15.39:0/3409503204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e20009b30 con 0x7f1e2815cbf0 2024-01-24T17:47:58.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e1fe00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e280b1570 0x7f1e280cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:58.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.255+0000 7f1e2cc00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e2815cbf0 0x7f1e281508c0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f1e20036040 tx=0x7f1e20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.256+0000 7f1e24c00640 1 -- 172.21.15.39:0/3409503204 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e20009510 con 0x7f1e2815cbf0 2024-01-24T17:47:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.256+0000 7f1e24c00640 1 -- 172.21.15.39:0/3409503204 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2003e450 con 0x7f1e2815cbf0 2024-01-24T17:47:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.256+0000 7f1e24c00640 1 -- 172.21.15.39:0/3409503204 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2003e800 con 0x7f1e2815cbf0 2024-01-24T17:47:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.256+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2815ec80 con 0x7f1e2815cbf0 2024-01-24T17:47:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.256+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e280cf940 con 0x7f1e2815cbf0 2024-01-24T17:47:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.257+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e280cfd10 con 0x7f1e2815cbf0 2024-01-24T17:47:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.258+0000 7f1e24c00640 1 -- 172.21.15.39:0/3409503204 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1e2002cd50 con 0x7f1e2815cbf0 2024-01-24T17:47:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.259+0000 7f1e24c00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e0006a460 0x7f1e0006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.259+0000 7f1e24c00640 1 -- 172.21.15.39:0/3409503204 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e200ab370 con 0x7f1e2815cbf0 2024-01-24T17:47:58.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.259+0000 7f1e27e00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e0006a460 0x7f1e0006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.261+0000 7f1e27e00640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e0006a460 0x7f1e0006c920 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1e2806b450 tx=0x7f1e18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:58.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.263+0000 7f1e24c00640 1 -- 172.21.15.39:0/3409503204 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1e280cfd10 con 0x7f1e2815cbf0 2024-01-24T17:47:58.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.463+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1e280cfd10 con 0x7f1e2815cbf0 2024-01-24T17:47:58.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.466+0000 7f1e24c00640 1 -- 172.21.15.39:0/3409503204 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v88) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1e280cfd10 con 0x7f1e2815cbf0 2024-01-24T17:47:58.468 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:47:58.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e0006a460 msgr2=0x7f1e0006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e0006a460 0x7f1e0006c920 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1e2806b450 tx=0x7f1e18008040 comp rx=0 tx=0).stop 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e2815cbf0 msgr2=0x7f1e281508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e2815cbf0 0x7f1e281508c0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f1e20036040 tx=0x7f1e20004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 shutdown_connections 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1e0006a460 0x7f1e0006c920 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e2815cbf0 0x7f1e281508c0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e280b1570 0x7f1e280cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 --2- 172.21.15.39:0/3409503204 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1e2806acb0 0x7f1e281f1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 >> 172.21.15.39:0/3409503204 conn(0x7f1e2809bad0 msgr2=0x7f1e280aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 shutdown_connections 2024-01-24T17:47:58.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.469+0000 7f1e2d600640 1 -- 172.21.15.39:0/3409503204 wait complete. 2024-01-24T17:47:58.488 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:47:58.488 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:47:58.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:47:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.764+0000 7f8964a00640 1 Processor -- start 2024-01-24T17:47:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.765+0000 7f8964a00640 1 -- start start 2024-01-24T17:47:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.766+0000 7f8964a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f896015cc40 0x7f89600b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.766+0000 7f8964a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89600d0610 0x7f89600d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.766+0000 7f8964a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600d1760 con 0x7f89600d1350 2024-01-24T17:47:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f8964a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896006b620 con 0x7f89600d0610 2024-01-24T17:47:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f8964a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89600d3810 con 0x7f896015cc40 2024-01-24T17:47:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f895de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89600d0610 0x7f89600d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f895de00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f89600d0610 0x7f89600d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37436/0 (socket says 172.21.15.39:37436) 2024-01-24T17:47:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f895de00640 1 -- 172.21.15.39:0/1034443100 learned_addr learned my addr 172.21.15.39:0/1034443100 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f895e800640 1 --2- 172.21.15.39:0/1034443100 >> v2:172.21.15.100:3300/0 conn(0x7f896015cc40 0x7f89600b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f895de00640 1 -- 172.21.15.39:0/1034443100 >> v2:172.21.15.100:3300/0 conn(0x7f896015cc40 msgr2=0x7f89600b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f895de00640 1 --2- 172.21.15.39:0/1034443100 >> v2:172.21.15.100:3300/0 conn(0x7f896015cc40 0x7f89600b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.768+0000 7f895de00640 1 -- 172.21.15.39:0/1034443100 >> v1:172.21.15.39:6789/0 conn(0x7f89600d1350 legacy=0x7f89600cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.768+0000 7f895de00640 1 -- 172.21.15.39:0/1034443100 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89600d3530 con 0x7f89600d0610 2024-01-24T17:47:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.767+0000 7f895d400640 1 -- 172.21.15.39:0/1034443100 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844951169 0 0) 0x7f89600d1760 con 0x7f89600d1350 2024-01-24T17:47:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.768+0000 7f895e800640 1 --2- 172.21.15.39:0/1034443100 >> v2:172.21.15.100:3300/0 conn(0x7f896015cc40 0x7f89600b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:47:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.769+0000 7f895de00640 1 --2- 172.21.15.39:0/1034443100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89600d0610 0x7f89600d0e10 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f896006a9f0 tx=0x7f894802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657346335040140c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:58.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.769+0000 7f895d400640 1 -- 172.21.15.39:0/1034443100 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948031870 con 0x7f89600d0610 2024-01-24T17:47:58.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.769+0000 7f895d400640 1 -- 172.21.15.39:0/1034443100 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89480319d0 con 0x7f89600d0610 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.769+0000 7f895d400640 1 -- 172.21.15.39:0/1034443100 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948031d60 con 0x7f89600d0610 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.769+0000 7f8964a00640 1 -- 172.21.15.39:0/1034443100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89600d0610 msgr2=0x7f89600d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.769+0000 7f8964a00640 1 --2- 172.21.15.39:0/1034443100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89600d0610 0x7f89600d0e10 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f896006a9f0 tx=0x7f894802b910 comp rx=0 tx=0).stop 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.770+0000 7f8964a00640 1 -- 172.21.15.39:0/1034443100 shutdown_connections 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.770+0000 7f8964a00640 1 --2- 172.21.15.39:0/1034443100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89600d0610 0x7f89600d0e10 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.770+0000 7f8964a00640 1 --2- 172.21.15.39:0/1034443100 >> v2:172.21.15.100:3300/0 conn(0x7f896015cc40 0x7f89600b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.770+0000 7f8964a00640 1 -- 172.21.15.39:0/1034443100 >> 172.21.15.39:0/1034443100 conn(0x7f89600d2830 msgr2=0x7f89600d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.770+0000 7f8964a00640 1 -- 172.21.15.39:0/1034443100 shutdown_connections 2024-01-24T17:47:58.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.770+0000 7f8964a00640 1 -- 172.21.15.39:0/1034443100 wait complete. 2024-01-24T17:47:58.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.771+0000 7f8964a00640 1 Processor -- start 2024-01-24T17:47:58.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.772+0000 7f8964a00640 1 -- start start 2024-01-24T17:47:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.772+0000 7f8964a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89600d0610 0x7f89601f5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.772+0000 7f8964a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89600d1350 0x7f89600cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.772+0000 7f8964a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f896015cc40 0x7f89601f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.772+0000 7f8964a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896015f700 con 0x7f896015cc40 2024-01-24T17:47:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.772+0000 7f8964a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89601f5e80 con 0x7f89600d0610 2024-01-24T17:47:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.772+0000 7f8964a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89601f6160 con 0x7f89600d1350 2024-01-24T17:47:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f896015cc40 0x7f89601f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f896015cc40 0x7f89601f1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37446/0 (socket says 172.21.15.39:37446) 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895f200640 1 -- 172.21.15.39:0/2340547512 learned_addr learned my addr 172.21.15.39:0/2340547512 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895e800640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89600d0610 0x7f89601f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895de00640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89600d1350 0x7f89600cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895f200640 1 -- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89600d1350 msgr2=0x7f89600cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895f200640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89600d1350 0x7f89600cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895f200640 1 -- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89600d0610 msgr2=0x7f89601f5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895f200640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89600d0610 0x7f89601f5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.773+0000 7f895f200640 1 -- 172.21.15.39:0/2340547512 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480094d0 con 0x7f896015cc40 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.774+0000 7f895f200640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f896015cc40 0x7f89601f1880 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f895800c780 tx=0x7f895800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.774+0000 7f8956a00640 1 -- 172.21.15.39:0/2340547512 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958003b70 con 0x7f896015cc40 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.774+0000 7f8956a00640 1 -- 172.21.15.39:0/2340547512 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958003cd0 con 0x7f896015cc40 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.774+0000 7f8956a00640 1 -- 172.21.15.39:0/2340547512 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958005270 con 0x7f896015cc40 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.774+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89601f6440 con 0x7f896015cc40 2024-01-24T17:47:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.774+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8960157fe0 con 0x7f896015cc40 2024-01-24T17:47:58.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.775+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89601581d0 con 0x7f896015cc40 2024-01-24T17:47:58.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.776+0000 7f8956a00640 1 -- 172.21.15.39:0/2340547512 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f89580053d0 con 0x7f896015cc40 2024-01-24T17:47:58.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.777+0000 7f8956a00640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f893806a430 0x7f893806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:58.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.778+0000 7f8956a00640 1 -- 172.21.15.39:0/2340547512 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8958088650 con 0x7f896015cc40 2024-01-24T17:47:58.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.778+0000 7f895e800640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f893806a430 0x7f893806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.779+0000 7f895e800640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f893806a430 0x7f893806c8f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8950006fd0 tx=0x7f8950008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:58.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.782+0000 7f8956a00640 1 -- 172.21.15.39:0/2340547512 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8958053cd0 con 0x7f896015cc40 2024-01-24T17:47:58.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.981+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8960067050 con 0x7f896015cc40 2024-01-24T17:47:58.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.985+0000 7f8956a00640 1 -- 172.21.15.39:0/2340547512 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16104 (secure 0 0 0) 0x7f89580572f0 con 0x7f896015cc40 2024-01-24T17:47:58.987 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:47:58.987 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:32.941651+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:47:58.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.987+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f893806a430 msgr2=0x7f893806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.987+0000 7f8964a00640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f893806a430 0x7f893806c8f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8950006fd0 tx=0x7f8950008040 comp rx=0 tx=0).stop 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.987+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f896015cc40 msgr2=0x7f89601f1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.987+0000 7f8964a00640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f896015cc40 0x7f89601f1880 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f895800c780 tx=0x7f895800cb80 comp rx=0 tx=0).stop 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.988+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 shutdown_connections 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.988+0000 7f8964a00640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f893806a430 0x7f893806c8f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.988+0000 7f8964a00640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f896015cc40 0x7f89601f1880 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.988+0000 7f8964a00640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f89600d1350 0x7f89600cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.988+0000 7f8964a00640 1 --2- 172.21.15.39:0/2340547512 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f89600d0610 0x7f89601f5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.988+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 >> 172.21.15.39:0/2340547512 conn(0x7f89600d2830 msgr2=0x7f896015d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.988+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 shutdown_connections 2024-01-24T17:47:58.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:58.988+0000 7f8964a00640 1 -- 172.21.15.39:0/2340547512 wait complete. 2024-01-24T17:47:59.009 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:47:30.315286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:47:30.960101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:47:59.009 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:47:59.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.219+0000 7fafb5000640 1 Processor -- start 2024-01-24T17:47:59.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.219+0000 7fafb5000640 1 -- start start 2024-01-24T17:47:59.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.221+0000 7fafb5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fafb00ce8c0 0x7fafb006fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.221+0000 7fafb5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00b2710 0x7fafb0070290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.221+0000 7fafb5000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb0155170 con 0x7fafb015ac70 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.221+0000 7fafb5000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb00b12b0 con 0x7fafb00ce8c0 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.221+0000 7fafb5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb015b1e0 con 0x7fafb00b2710 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.221+0000 7fafaf400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fafb00ce8c0 0x7fafb006fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.221+0000 7fafaf400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fafb00ce8c0 0x7fafb006fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37462/0 (socket says 172.21.15.39:37462) 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.221+0000 7fafaf400640 1 -- 172.21.15.39:0/426901004 learned_addr learned my addr 172.21.15.39:0/426901004 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.222+0000 7fafafe00640 1 --2- 172.21.15.39:0/426901004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00b2710 0x7fafb0070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.222+0000 7fafaf400640 1 -- 172.21.15.39:0/426901004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00b2710 msgr2=0x7fafb0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.222+0000 7fafaf400640 1 --2- 172.21.15.39:0/426901004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00b2710 0x7fafb0070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.222+0000 7fafaf400640 1 -- 172.21.15.39:0/426901004 >> v1:172.21.15.39:6789/0 conn(0x7fafb015ac70 legacy=0x7fafb015e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.222+0000 7fafaf400640 1 -- 172.21.15.39:0/426901004 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb015b4c0 con 0x7fafb00ce8c0 2024-01-24T17:47:59.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.222+0000 7fafafe00640 1 --2- 172.21.15.39:0/426901004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00b2710 0x7fafb0070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:59.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.222+0000 7fafae000640 1 -- 172.21.15.39:0/426901004 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28365295 0 0) 0x7fafb0155170 con 0x7fafb015ac70 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.223+0000 7fafaf400640 1 --2- 172.21.15.39:0/426901004 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00ce8c0 0x7fafb006fd50 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009e50 tx=0x7fafa002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f56e4d5c63e107 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.223+0000 7fafae000640 1 -- 172.21.15.39:0/426901004 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa002caf0 con 0x7fafb00ce8c0 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.223+0000 7fafae000640 1 -- 172.21.15.39:0/426901004 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa002cc50 con 0x7fafb00ce8c0 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.223+0000 7fafae000640 1 -- 172.21.15.39:0/426901004 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa00343f0 con 0x7fafb00ce8c0 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.224+0000 7fafb5000640 1 -- 172.21.15.39:0/426901004 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00ce8c0 msgr2=0x7fafb006fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.224+0000 7fafb5000640 1 --2- 172.21.15.39:0/426901004 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00ce8c0 0x7fafb006fd50 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009e50 tx=0x7fafa002bf50 comp rx=0 tx=0).stop 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.224+0000 7fafb5000640 1 -- 172.21.15.39:0/426901004 shutdown_connections 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.224+0000 7fafb5000640 1 --2- 172.21.15.39:0/426901004 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00b2710 0x7fafb0070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.224+0000 7fafb5000640 1 --2- 172.21.15.39:0/426901004 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00ce8c0 0x7fafb006fd50 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.224+0000 7fafb5000640 1 -- 172.21.15.39:0/426901004 >> 172.21.15.39:0/426901004 conn(0x7fafb006a200 msgr2=0x7fafb00affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:59.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.224+0000 7fafb5000640 1 -- 172.21.15.39:0/426901004 shutdown_connections 2024-01-24T17:47:59.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.224+0000 7fafb5000640 1 -- 172.21.15.39:0/426901004 wait complete. 2024-01-24T17:47:59.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.226+0000 7fafb5000640 1 Processor -- start 2024-01-24T17:47:59.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.226+0000 7fafb5000640 1 -- start start 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafb5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00b2710 0x7fafb0145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafb5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00ce8c0 0x7fafb00c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafb5000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fafb015ac70 0x7fafb0143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafb5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb0144c40 con 0x7fafb00b2710 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafb5000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb0144f20 con 0x7fafb015ac70 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafb5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb0145200 con 0x7fafb00ce8c0 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafaf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00b2710 0x7fafb0145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafaf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00b2710 0x7fafb0145ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37468/0 (socket says 172.21.15.39:37468) 2024-01-24T17:47:59.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.227+0000 7fafaf400640 1 -- 172.21.15.39:0/202569256 learned_addr learned my addr 172.21.15.39:0/202569256 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:59.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafafe00640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fafb015ac70 0x7fafb0143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafaf400640 1 -- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00ce8c0 msgr2=0x7fafb00c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafaea00640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00ce8c0 0x7fafb00c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafaf400640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00ce8c0 0x7fafb00c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafaf400640 1 -- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fafb015ac70 msgr2=0x7fafb0143eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafaf400640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fafb015ac70 0x7fafb0143eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafaf400640 1 -- 172.21.15.39:0/202569256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa0009b30 con 0x7fafb00b2710 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafafe00640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fafb015ac70 0x7fafb0143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafaf400640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00b2710 0x7fafb0145ce0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009e50 tx=0x7fafa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafa7400640 1 -- 172.21.15.39:0/202569256 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa003e450 con 0x7fafb00b2710 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafa7400640 1 -- 172.21.15.39:0/202569256 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa003e5b0 con 0x7fafb00b2710 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafa7400640 1 -- 172.21.15.39:0/202569256 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa003e740 con 0x7fafb00b2710 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.228+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb01454e0 con 0x7fafb00b2710 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.229+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb00c4980 con 0x7fafb00b2710 2024-01-24T17:47:59.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.230+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb00738c0 con 0x7fafb00b2710 2024-01-24T17:47:59.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.231+0000 7fafa7400640 1 -- 172.21.15.39:0/202569256 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fafa003e8a0 con 0x7fafb00b2710 2024-01-24T17:47:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.232+0000 7fafa7400640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf8806a430 0x7faf8806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.232+0000 7fafa7400640 1 -- 172.21.15.39:0/202569256 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafa00ab610 con 0x7fafb00b2710 2024-01-24T17:47:59.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.232+0000 7fafaea00640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf8806a430 0x7faf8806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.233+0000 7fafaea00640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf8806a430 0x7faf8806c8f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fafb00ce580 tx=0x7faf98009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:59.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.236+0000 7fafa7400640 1 -- 172.21.15.39:0/202569256 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fafa0076ae0 con 0x7fafb00b2710 2024-01-24T17:47:59.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.430+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fafb0145340 con 0x7fafb00b2710 2024-01-24T17:47:59.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.431+0000 7fafa7400640 1 -- 172.21.15.39:0/202569256 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fafb0145340 con 0x7fafb00b2710 2024-01-24T17:47:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf8806a430 msgr2=0x7faf8806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf8806a430 0x7faf8806c8f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fafb00ce580 tx=0x7faf98009040 comp rx=0 tx=0).stop 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00b2710 msgr2=0x7fafb0145ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00b2710 0x7fafb0145ce0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009e50 tx=0x7fafa0004a40 comp rx=0 tx=0).stop 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 shutdown_connections 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fafb015ac70 0x7fafb0143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fafb00ce8c0 0x7fafb00c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faf8806a430 0x7faf8806c8f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 --2- 172.21.15.39:0/202569256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafb00b2710 0x7fafb0145ce0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 >> 172.21.15.39:0/202569256 conn(0x7fafb006a200 msgr2=0x7fafb006bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:59.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 shutdown_connections 2024-01-24T17:47:59.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.434+0000 7fafb5000640 1 -- 172.21.15.39:0/202569256 wait complete. 2024-01-24T17:47:59.455 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:47:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.676+0000 7ffb77400640 1 Processor -- start 2024-01-24T17:47:59.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.676+0000 7ffb77400640 1 -- start start 2024-01-24T17:47:59.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.678+0000 7ffb77400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb700c8bd0 0x7ffb7015d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.678+0000 7ffb77400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffb700ce350 0x7ffb700ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.678+0000 7ffb77400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700c9030 con 0x7ffb7015d450 2024-01-24T17:47:59.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.678+0000 7ffb77400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb700ced20 con 0x7ffb700ce350 2024-01-24T17:47:59.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.678+0000 7ffb77400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700c9290 con 0x7ffb700c8bd0 2024-01-24T17:47:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.679+0000 7ffb75600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb700c8bd0 0x7ffb7015d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.679+0000 7ffb76000640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ffb7015d450 0x7ffb700cbf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:40794/0 (socket says 172.21.15.39:40794) 2024-01-24T17:47:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.679+0000 7ffb76000640 1 -- 172.21.15.39:0/3355179807 learned_addr learned my addr 172.21.15.39:0/3355179807 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:47:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.679+0000 7ffb76a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ffb700ce350 0x7ffb700ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.679+0000 7ffb76a00640 1 -- 172.21.15.39:0/3355179807 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb700c8bd0 msgr2=0x7ffb7015d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:47:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.679+0000 7ffb76a00640 1 --2- 172.21.15.39:0/3355179807 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb700c8bd0 0x7ffb7015d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.679+0000 7ffb76a00640 1 -- 172.21.15.39:0/3355179807 >> v1:172.21.15.100:6789/0 conn(0x7ffb7015d450 legacy=0x7ffb700cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.679+0000 7ffb76a00640 1 -- 172.21.15.39:0/3355179807 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700c9570 con 0x7ffb700ce350 2024-01-24T17:47:59.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.681+0000 7ffb76a00640 1 --2- 172.21.15.39:0/3355179807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ce7e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009d90 tx=0x7ffb6802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f0f2578dac92d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:59.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.681+0000 7ffb74c00640 1 -- 172.21.15.39:0/3355179807 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6802cbc0 con 0x7ffb700ce350 2024-01-24T17:47:59.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.681+0000 7ffb74c00640 1 -- 172.21.15.39:0/3355179807 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6802cd20 con 0x7ffb700ce350 2024-01-24T17:47:59.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.681+0000 7ffb74c00640 1 -- 172.21.15.39:0/3355179807 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68034660 con 0x7ffb700ce350 2024-01-24T17:47:59.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.681+0000 7ffb77400640 1 -- 172.21.15.39:0/3355179807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 msgr2=0x7ffb700ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.681+0000 7ffb77400640 1 --2- 172.21.15.39:0/3355179807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ce7e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009d90 tx=0x7ffb6802be10 comp rx=0 tx=0).stop 2024-01-24T17:47:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.682+0000 7ffb77400640 1 -- 172.21.15.39:0/3355179807 shutdown_connections 2024-01-24T17:47:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.682+0000 7ffb77400640 1 --2- 172.21.15.39:0/3355179807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ce7e0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.682+0000 7ffb77400640 1 --2- 172.21.15.39:0/3355179807 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb700c8bd0 0x7ffb7015d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.682+0000 7ffb77400640 1 -- 172.21.15.39:0/3355179807 >> 172.21.15.39:0/3355179807 conn(0x7ffb7006a200 msgr2=0x7ffb700affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.682+0000 7ffb77400640 1 -- 172.21.15.39:0/3355179807 shutdown_connections 2024-01-24T17:47:59.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.682+0000 7ffb77400640 1 -- 172.21.15.39:0/3355179807 wait complete. 2024-01-24T17:47:59.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.684+0000 7ffb77400640 1 Processor -- start 2024-01-24T17:47:59.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.684+0000 7ffb77400640 1 -- start start 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb77400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb700c8bd0 0x7ffb70154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb77400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb77400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb7015d450 0x7ffb700c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb77400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700c5ca0 con 0x7ffb700ce350 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb77400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb70159cb0 con 0x7ffb700c8bd0 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb77400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700acd20 con 0x7ffb7015d450 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb75600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb75600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37480/0 (socket says 172.21.15.39:37480) 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.686+0000 7ffb75600640 1 -- 172.21.15.39:0/1244023876 learned_addr learned my addr 172.21.15.39:0/1244023876 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb75600640 1 -- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb7015d450 msgr2=0x7ffb700c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb75600640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb7015d450 0x7ffb700c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb75600640 1 -- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb700c8bd0 msgr2=0x7ffb70154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb75600640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb700c8bd0 0x7ffb70154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb75600640 1 -- 172.21.15.39:0/1244023876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb68009a70 con 0x7ffb700ce350 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb75600640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ac2d0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800e9b0 tx=0x7ffb5800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb6e000640 1 -- 172.21.15.39:0/1244023876 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58009670 con 0x7ffb700ce350 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb6e000640 1 -- 172.21.15.39:0/1244023876 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5800c9a0 con 0x7ffb700ce350 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb6e000640 1 -- 172.21.15.39:0/1244023876 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5800cd50 con 0x7ffb700ce350 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb700ad000 con 0x7ffb700ce350 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.687+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb701549b0 con 0x7ffb700ce350 2024-01-24T17:47:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.689+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb700738c0 con 0x7ffb700ce350 2024-01-24T17:47:59.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.689+0000 7ffb6e000640 1 -- 172.21.15.39:0/1244023876 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ffb58013470 con 0x7ffb700ce350 2024-01-24T17:47:59.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.691+0000 7ffb6e000640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffb4806a430 0x7ffb4806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:47:59.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.692+0000 7ffb6e000640 1 -- 172.21.15.39:0/1244023876 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb58088bf0 con 0x7ffb700ce350 2024-01-24T17:47:59.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.692+0000 7ffb76000640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffb4806a430 0x7ffb4806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:47:59.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.693+0000 7ffb76000640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffb4806a430 0x7ffb4806c8f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffb60001550 tx=0x7ffb60007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:47:59.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.697+0000 7ffb6e000640 1 -- 172.21.15.39:0/1244023876 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ffb580541d0 con 0x7ffb700ce350 2024-01-24T17:47:59.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.889+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffb700b0870 con 0x7ffb700ce350 2024-01-24T17:47:59.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.890+0000 7ffb6e000640 1 -- 172.21.15.39:0/1244023876 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb58017020 con 0x7ffb700ce350 2024-01-24T17:47:59.892 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:47:59.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffb4806a430 msgr2=0x7ffb4806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffb4806a430 0x7ffb4806c8f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffb60001550 tx=0x7ffb60007a40 comp rx=0 tx=0).stop 2024-01-24T17:47:59.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 msgr2=0x7ffb700ac2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:47:59.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ac2d0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800e9b0 tx=0x7ffb5800ee70 comp rx=0 tx=0).stop 2024-01-24T17:47:59.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 shutdown_connections 2024-01-24T17:47:59.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ffb7015d450 0x7ffb700c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ffb4806a430 0x7ffb4806c8f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb700ce350 0x7ffb700ac2d0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 --2- 172.21.15.39:0/1244023876 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ffb700c8bd0 0x7ffb70154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:47:59.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 >> 172.21.15.39:0/1244023876 conn(0x7ffb7006a200 msgr2=0x7ffb7006bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:47:59.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.893+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 shutdown_connections 2024-01-24T17:47:59.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:47:59.894+0000 7ffb77400640 1 -- 172.21.15.39:0/1244023876 wait complete. 2024-01-24T17:47:59.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:48:00.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.135+0000 7f1effe00640 1 Processor -- start 2024-01-24T17:48:00.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.135+0000 7f1effe00640 1 -- start start 2024-01-24T17:48:00.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.137+0000 7f1effe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1f0015d450 0x7f1f000c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.137+0000 7f1effe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.137+0000 7f1effe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f000c9290 con 0x7f1f000ce350 2024-01-24T17:48:00.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.137+0000 7f1effe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f000c9570 con 0x7f1f0015d450 2024-01-24T17:48:00.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.137+0000 7f1effe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0006fed0 con 0x7f1f000cbf90 2024-01-24T17:48:00.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efe000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efea00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1f0015d450 0x7f1f000c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efea00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1f0015d450 0x7f1f000c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:58106/0 (socket says 172.21.15.39:58106) 2024-01-24T17:48:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efe000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:59066/0 (socket says 172.21.15.39:59066) 2024-01-24T17:48:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efea00640 1 -- 172.21.15.39:0/215350219 learned_addr learned my addr 172.21.15.39:0/215350219 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efe000640 1 -- 172.21.15.39:0/215350219 >> v2:172.21.15.100:3300/0 conn(0x7f1f0015d450 msgr2=0x7f1f000c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:00.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efe000640 1 --2- 172.21.15.39:0/215350219 >> v2:172.21.15.100:3300/0 conn(0x7f1f0015d450 0x7f1f000c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efe000640 1 -- 172.21.15.39:0/215350219 >> v1:172.21.15.100:6789/0 conn(0x7f1f000ce350 legacy=0x7f1f000706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efe000640 1 -- 172.21.15.39:0/215350219 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000701b0 con 0x7f1f000cbf90 2024-01-24T17:48:00.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.138+0000 7f1efea00640 1 --2- 172.21.15.39:0/215350219 >> v2:172.21.15.100:3300/0 conn(0x7f1f0015d450 0x7f1f000c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:48:00.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.140+0000 7f1efe000640 1 --2- 172.21.15.39:0/215350219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015d830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1ee80097a0 tx=0x7f1ee802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ced77aa98e119627 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:00.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.140+0000 7f1efd600640 1 -- 172.21.15.39:0/215350219 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8031780 con 0x7f1f000cbf90 2024-01-24T17:48:00.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.140+0000 7f1efd600640 1 -- 172.21.15.39:0/215350219 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee80318e0 con 0x7f1f000cbf90 2024-01-24T17:48:00.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.140+0000 7f1efd600640 1 -- 172.21.15.39:0/215350219 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8031c70 con 0x7f1f000cbf90 2024-01-24T17:48:00.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.141+0000 7f1effe00640 1 -- 172.21.15.39:0/215350219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 msgr2=0x7f1f0015d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.141+0000 7f1effe00640 1 --2- 172.21.15.39:0/215350219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015d830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1ee80097a0 tx=0x7f1ee802b820 comp rx=0 tx=0).stop 2024-01-24T17:48:00.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.141+0000 7f1effe00640 1 -- 172.21.15.39:0/215350219 shutdown_connections 2024-01-24T17:48:00.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.141+0000 7f1effe00640 1 --2- 172.21.15.39:0/215350219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015d830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.141+0000 7f1effe00640 1 --2- 172.21.15.39:0/215350219 >> v2:172.21.15.100:3300/0 conn(0x7f1f0015d450 0x7f1f000c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.141+0000 7f1effe00640 1 -- 172.21.15.39:0/215350219 >> 172.21.15.39:0/215350219 conn(0x7f1f0006a200 msgr2=0x7f1f000affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:00.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.141+0000 7f1effe00640 1 -- 172.21.15.39:0/215350219 shutdown_connections 2024-01-24T17:48:00.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.141+0000 7f1effe00640 1 -- 172.21.15.39:0/215350219 wait complete. 2024-01-24T17:48:00.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.143+0000 7f1effe00640 1 Processor -- start 2024-01-24T17:48:00.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.143+0000 7f1effe00640 1 -- start start 2024-01-24T17:48:00.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1effe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1effe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f000ce350 0x7f1f001584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1effe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f0015d450 0x7f1f001f5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1effe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000c5790 con 0x7f1f0015d450 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1effe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000aef90 con 0x7f1f000ce350 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1effe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000af270 con 0x7f1f000cbf90 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1eff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f0015d450 0x7f1f001f5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1eff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f0015d450 0x7f1f001f5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37496/0 (socket says 172.21.15.39:37496) 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.144+0000 7f1eff400640 1 -- 172.21.15.39:0/2364498300 learned_addr learned my addr 172.21.15.39:0/2364498300 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1efe000640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f000ce350 0x7f1f001584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1efea00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1eff400640 1 -- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 msgr2=0x7f1f0015a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1eff400640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1eff400640 1 -- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f000ce350 msgr2=0x7f1f001584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1eff400640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f000ce350 0x7f1f001584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1eff400640 1 -- 172.21.15.39:0/2364498300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00094d0 con 0x7f1f0015d450 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1efe000640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f000ce350 0x7f1f001584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1efea00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1eff400640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f0015d450 0x7f1f001f5e30 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c7a0 tx=0x7f1ef800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.145+0000 7f1ef6a00640 1 -- 172.21.15.39:0/2364498300 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8003b70 con 0x7f1f0015d450 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.146+0000 7f1ef6a00640 1 -- 172.21.15.39:0/2364498300 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8003cf0 con 0x7f1f0015d450 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.146+0000 7f1ef6a00640 1 -- 172.21.15.39:0/2364498300 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8003ea0 con 0x7f1f0015d450 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.146+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee8009400 con 0x7f1f0015d450 2024-01-24T17:48:00.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.146+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f000af830 con 0x7f1f0015d450 2024-01-24T17:48:00.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.147+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f000738c0 con 0x7f1f0015d450 2024-01-24T17:48:00.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.148+0000 7f1ef6a00640 1 -- 172.21.15.39:0/2364498300 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1ef800aa90 con 0x7f1f0015d450 2024-01-24T17:48:00.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.149+0000 7f1ef6a00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1ed806a430 0x7f1ed806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.149+0000 7f1ef6a00640 1 -- 172.21.15.39:0/2364498300 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ef8088890 con 0x7f1f0015d450 2024-01-24T17:48:00.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.149+0000 7f1efea00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1ed806a430 0x7f1ed806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.150+0000 7f1efea00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1ed806a430 0x7f1ed806c8f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009bb0 tx=0x7f1ef0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:00.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.153+0000 7f1ef6a00640 1 -- 172.21.15.39:0/2364498300 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1ef8053d50 con 0x7f1f0015d450 2024-01-24T17:48:00.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.349+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1f000ca980 con 0x7f1f0015d450 2024-01-24T17:48:00.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.350+0000 7f1ef6a00640 1 -- 172.21.15.39:0/2364498300 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ef8057370 con 0x7f1f0015d450 2024-01-24T17:48:00.352 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:48:00.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1ed806a430 msgr2=0x7f1ed806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1ed806a430 0x7f1ed806c8f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009bb0 tx=0x7f1ef0008040 comp rx=0 tx=0).stop 2024-01-24T17:48:00.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f0015d450 msgr2=0x7f1f001f5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f0015d450 0x7f1f001f5e30 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c7a0 tx=0x7f1ef800cc60 comp rx=0 tx=0).stop 2024-01-24T17:48:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 shutdown_connections 2024-01-24T17:48:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1ed806a430 0x7f1ed806c8f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f0015d450 0x7f1f001f5e30 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f000ce350 0x7f1f001584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 --2- 172.21.15.39:0/2364498300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1f000cbf90 0x7f1f0015a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.353+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 >> 172.21.15.39:0/2364498300 conn(0x7f1f0006a200 msgr2=0x7f1f000cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.354+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 shutdown_connections 2024-01-24T17:48:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.354+0000 7f1effe00640 1 -- 172.21.15.39:0/2364498300 wait complete. 2024-01-24T17:48:00.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:48:00.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.594+0000 7f04d1e00640 1 Processor -- start 2024-01-24T17:48:00.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.594+0000 7f04d1e00640 1 -- start start 2024-01-24T17:48:00.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.595+0000 7f04d1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15cf10 0x7f04cc070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.595+0000 7f04d1e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0ccf90 con 0x7f04cc0b0720 2024-01-24T17:48:00.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.596+0000 7f04d1e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04cc0c9bd0 con 0x7f04cc070980 2024-01-24T17:48:00.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.596+0000 7f04d1e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0c9eb0 con 0x7f04cc15cf10 2024-01-24T17:48:00.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.596+0000 7f04cb400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f04cc0b0720 0x7f04cc0cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:45832/0 (socket says 172.21.15.39:45832) 2024-01-24T17:48:00.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.596+0000 7f04cb400640 1 -- 172.21.15.39:0/106715989 learned_addr learned my addr 172.21.15.39:0/106715989 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:00.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.596+0000 7f04caa00640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15cf10 0x7f04cc070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.596+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41655518 0 0) 0x7f04cc0ccf90 con 0x7f04cc0b0720 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ac003520 con 0x7f04cc0b0720 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1094739229 0 0) 0x7f04ac003520 con 0x7f04cc0b0720 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15cf10 msgr2=0x7f04cc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15cf10 0x7f04cc070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 >> v1:172.21.15.100:6789/0 conn(0x7f04cc070980 legacy=0x7f04cc0b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0cc940 con 0x7f04cc0b0720 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f04c40019f0 con 0x7f04cc0b0720 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 >> v1:172.21.15.39:6789/0 conn(0x7f04cc0b0720 legacy=0x7f04cc0cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04ac0039a0 0x7f04ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15cf10 0x7f04ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0c9eb0 con 0x7f04ac004d00 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00bab0 con 0x7f04ac0039a0 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00bc70 con 0x7f04cc15cf10 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.597+0000 7f04cb400640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.598+0000 7f04c1e00640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15cf10 0x7f04ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.598+0000 7f04caa00640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04ac0039a0 0x7f04ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.598+0000 7f04cb400640 1 -- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15cf10 msgr2=0x7f04ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.598+0000 7f04cb400640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15cf10 0x7f04ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.598+0000 7f04cb400640 1 -- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04ac0039a0 msgr2=0x7f04ac003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.598+0000 7f04cb400640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04ac0039a0 0x7f04ac003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.598+0000 7f04cb400640 1 -- 172.21.15.39:0/106715989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac003520 con 0x7f04ac004d00 2024-01-24T17:48:00.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.599+0000 7f04cb400640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04ac005180 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f04c400a830 tx=0x7f04ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7e54348d9cfa91 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:00.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.599+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c40072d0 con 0x7f04ac004d00 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.599+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c402f070 con 0x7f04ac004d00 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.599+0000 7f04ca000640 1 -- 172.21.15.39:0/106715989 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c40076b0 con 0x7f04ac004d00 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.600+0000 7f04d1e00640 1 -- 172.21.15.39:0/106715989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 msgr2=0x7f04ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.600+0000 7f04d1e00640 1 --2- 172.21.15.39:0/106715989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04ac005180 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f04c400a830 tx=0x7f04ac0057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.600+0000 7f04cb400640 1 -- 172.21.15.39:0/106715989 reap_dead start 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.600+0000 7f04d1e00640 1 -- 172.21.15.39:0/106715989 shutdown_connections 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.600+0000 7f04d1e00640 1 -- 172.21.15.39:0/106715989 >> 172.21.15.39:0/106715989 conn(0x7f04cc06bf20 msgr2=0x7f04cc06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.600+0000 7f04d1e00640 1 -- 172.21.15.39:0/106715989 shutdown_connections 2024-01-24T17:48:00.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.600+0000 7f04d1e00640 1 -- 172.21.15.39:0/106715989 wait complete. 2024-01-24T17:48:00.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.602+0000 7f04d1e00640 1 Processor -- start 2024-01-24T17:48:00.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.602+0000 7f04d1e00640 1 -- start start 2024-01-24T17:48:00.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04d1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04cc1f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04d1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15f060 0x7f04cc15f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04d1e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04cc0b25c0 0x7f04cc0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04d1e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0b1520 con 0x7f04ac004d00 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04d1e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0b1800 con 0x7f04cc0b25c0 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04d1e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04cc0b1ae0 con 0x7f04cc15f060 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04cc1f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04caa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04cc1f3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37508/0 (socket says 172.21.15.39:37508) 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.603+0000 7f04caa00640 1 -- 172.21.15.39:0/4214370880 learned_addr learned my addr 172.21.15.39:0/4214370880 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.604+0000 7f04c1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15f060 0x7f04cc15f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.604+0000 7f04cb400640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04cc0b25c0 0x7f04cc0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.604+0000 7f04caa00640 1 -- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15f060 msgr2=0x7f04cc15f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.604+0000 7f04caa00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15f060 0x7f04cc15f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.604+0000 7f04caa00640 1 -- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04cc0b25c0 msgr2=0x7f04cc0b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.604+0000 7f04caa00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04cc0b25c0 0x7f04cc0b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.604+0000 7f04caa00640 1 -- 172.21.15.39:0/4214370880 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc0b1dc0 con 0x7f04ac004d00 2024-01-24T17:48:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.604+0000 7f04caa00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04cc1f3490 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f04bc011600 tx=0x7f04bc011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.605+0000 7f04c3e00640 1 -- 172.21.15.39:0/4214370880 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00f920 con 0x7f04ac004d00 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.605+0000 7f04c3e00640 1 -- 172.21.15.39:0/4214370880 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04bc00fa80 con 0x7f04ac004d00 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.605+0000 7f04c3e00640 1 -- 172.21.15.39:0/4214370880 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00fdf0 con 0x7f04ac004d00 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.605+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04cc0b20a0 con 0x7f04ac004d00 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.605+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc0b2f80 con 0x7f04ac004d00 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.606+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04cc04dff0 con 0x7f04ac004d00 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.607+0000 7f04c3e00640 1 -- 172.21.15.39:0/4214370880 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f04bc015460 con 0x7f04ac004d00 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.608+0000 7f04c3e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04a406a460 0x7f04a406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:00.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.608+0000 7f04c3e00640 1 -- 172.21.15.39:0/4214370880 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04bc088e80 con 0x7f04ac004d00 2024-01-24T17:48:00.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.608+0000 7f04c1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04a406a460 0x7f04a406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:00.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.610+0000 7f04c1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04a406a460 0x7f04a406c920 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0b06c0 tx=0x7f04b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.612+0000 7f04c3e00640 1 -- 172.21.15.39:0/4214370880 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f04bc0543d0 con 0x7f04ac004d00 2024-01-24T17:48:00.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.822+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f04cc0cc4e0 con 0x7f04ac004d00 2024-01-24T17:48:00.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.823+0000 7f04c3e00640 1 -- 172.21.15.39:0/4214370880 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v88) v1 ==== 80+0+0 (secure 0 0 0) 0x7f04bc0579f0 con 0x7f04ac004d00 2024-01-24T17:48:00.824 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04a406a460 msgr2=0x7f04a406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04a406a460 0x7f04a406c920 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0b06c0 tx=0x7f04b4009040 comp rx=0 tx=0).stop 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 msgr2=0x7f04cc1f3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04cc1f3490 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f04bc011600 tx=0x7f04bc011ac0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 shutdown_connections 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04cc0b25c0 0x7f04cc0b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f04cc15f060 0x7f04cc15f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f04a406a460 0x7f04a406c920 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 --2- 172.21.15.39:0/4214370880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04ac004d00 0x7f04cc1f3490 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 >> 172.21.15.39:0/4214370880 conn(0x7f04cc06bf20 msgr2=0x7f04cc0c20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 shutdown_connections 2024-01-24T17:48:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:00.826+0000 7f04d1e00640 1 -- 172.21.15.39:0/4214370880 wait complete. 2024-01-24T17:48:00.845 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:48:00.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:00.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:48:00.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[47918]: fuse finished with error 0 and tester_r 0 2024-01-24T17:48:00.948 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:07.050 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:48:07.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:07.050 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:48:07.111 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:07.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.338+0000 7f5f39e00640 1 Processor -- start 2024-01-24T17:48:07.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.338+0000 7f5f39e00640 1 -- start start 2024-01-24T17:48:07.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f39e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5f340d0e90 0x7f5f340d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f39e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f34164b40 0x7f5f34166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f39e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340d1820 con 0x7f5f3415cbe0 2024-01-24T17:48:07.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f39e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f340b8820 con 0x7f5f34164b40 2024-01-24T17:48:07.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f39e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f340b5790 con 0x7f5f340d0e90 2024-01-24T17:48:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f33400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f34164b40 0x7f5f34166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f33400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f5f34164b40 0x7f5f34166f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44166/0 (socket says 172.21.15.39:44166) 2024-01-24T17:48:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f33400640 1 -- 172.21.15.39:0/4289103644 learned_addr learned my addr 172.21.15.39:0/4289103644 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.340+0000 7f5f32000640 1 --2- 172.21.15.39:0/4289103644 >> v2:172.21.15.100:3300/0 conn(0x7f5f340d0e90 0x7f5f340d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.341+0000 7f5f33400640 1 -- 172.21.15.39:0/4289103644 >> v2:172.21.15.100:3300/0 conn(0x7f5f340d0e90 msgr2=0x7f5f340d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.341+0000 7f5f33400640 1 --2- 172.21.15.39:0/4289103644 >> v2:172.21.15.100:3300/0 conn(0x7f5f340d0e90 0x7f5f340d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.341+0000 7f5f33400640 1 -- 172.21.15.39:0/4289103644 >> v1:172.21.15.39:6789/0 conn(0x7f5f3415cbe0 legacy=0x7f5f340cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.341+0000 7f5f33400640 1 -- 172.21.15.39:0/4289103644 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340b5a50 con 0x7f5f34164b40 2024-01-24T17:48:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.342+0000 7f5f33400640 1 --2- 172.21.15.39:0/4289103644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f34164b40 0x7f5f34166f30 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5f3408bdb0 tx=0x7f5f2c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bd4e0607a8fb0e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.342+0000 7f5f31600640 1 -- 172.21.15.39:0/4289103644 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c02cb10 con 0x7f5f34164b40 2024-01-24T17:48:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.342+0000 7f5f31600640 1 -- 172.21.15.39:0/4289103644 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02cc70 con 0x7f5f34164b40 2024-01-24T17:48:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.342+0000 7f5f31600640 1 -- 172.21.15.39:0/4289103644 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0343f0 con 0x7f5f34164b40 2024-01-24T17:48:07.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.343+0000 7f5f39e00640 1 -- 172.21.15.39:0/4289103644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f34164b40 msgr2=0x7f5f34166f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.343+0000 7f5f39e00640 1 --2- 172.21.15.39:0/4289103644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f34164b40 0x7f5f34166f30 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5f3408bdb0 tx=0x7f5f2c02bf70 comp rx=0 tx=0).stop 2024-01-24T17:48:07.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.343+0000 7f5f39e00640 1 -- 172.21.15.39:0/4289103644 shutdown_connections 2024-01-24T17:48:07.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.343+0000 7f5f39e00640 1 --2- 172.21.15.39:0/4289103644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f34164b40 0x7f5f34166f30 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.343+0000 7f5f39e00640 1 --2- 172.21.15.39:0/4289103644 >> v2:172.21.15.100:3300/0 conn(0x7f5f340d0e90 0x7f5f340d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.343+0000 7f5f39e00640 1 -- 172.21.15.39:0/4289103644 >> 172.21.15.39:0/4289103644 conn(0x7f5f3409bc80 msgr2=0x7f5f340d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:07.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.343+0000 7f5f39e00640 1 -- 172.21.15.39:0/4289103644 shutdown_connections 2024-01-24T17:48:07.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.343+0000 7f5f39e00640 1 -- 172.21.15.39:0/4289103644 wait complete. 2024-01-24T17:48:07.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.346+0000 7f5f39e00640 1 Processor -- start 2024-01-24T17:48:07.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.346+0000 7f5f39e00640 1 -- start start 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.347+0000 7f5f39e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f340d0e90 0x7f5f3415f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.347+0000 7f5f39e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f3415cbe0 0x7f5f340cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.347+0000 7f5f39e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f34164b40 0x7f5f340cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.347+0000 7f5f39e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f34156aa0 con 0x7f5f3415cbe0 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.347+0000 7f5f39e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f34141360 con 0x7f5f34164b40 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.347+0000 7f5f39e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340cc200 con 0x7f5f340d0e90 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f3415cbe0 0x7f5f340cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f3415cbe0 0x7f5f340cb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44180/0 (socket says 172.21.15.39:44180) 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32000640 1 -- 172.21.15.39:0/1161472011 learned_addr learned my addr 172.21.15.39:0/1161472011 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:07.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32a00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f340d0e90 0x7f5f3415f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32000640 1 -- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f340d0e90 msgr2=0x7f5f3415f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32000640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f340d0e90 0x7f5f3415f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32000640 1 -- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f34164b40 msgr2=0x7f5f340cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32000640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f34164b40 0x7f5f340cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.348+0000 7f5f32000640 1 -- 172.21.15.39:0/1161472011 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c009b30 con 0x7f5f3415cbe0 2024-01-24T17:48:07.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.349+0000 7f5f32000640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f3415cbe0 0x7f5f340cb780 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400d620 tx=0x7f5f2400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.349+0000 7f5f2b400640 1 -- 172.21.15.39:0/1161472011 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2400b620 con 0x7f5f3415cbe0 2024-01-24T17:48:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.349+0000 7f5f2b400640 1 -- 172.21.15.39:0/1161472011 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2400b780 con 0x7f5f3415cbe0 2024-01-24T17:48:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.349+0000 7f5f2b400640 1 -- 172.21.15.39:0/1161472011 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2400bb30 con 0x7f5f3415cbe0 2024-01-24T17:48:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.349+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f34140a30 con 0x7f5f3415cbe0 2024-01-24T17:48:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.349+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f34140ff0 con 0x7f5f3415cbe0 2024-01-24T17:48:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.350+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f34141510 con 0x7f5f3415cbe0 2024-01-24T17:48:07.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.351+0000 7f5f2b400640 1 -- 172.21.15.39:0/1161472011 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5f2400bc90 con 0x7f5f3415cbe0 2024-01-24T17:48:07.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.352+0000 7f5f2b400640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f0c06a430 0x7f5f0c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.352+0000 7f5f2b400640 1 -- 172.21.15.39:0/1161472011 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f24088ce0 con 0x7f5f3415cbe0 2024-01-24T17:48:07.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.353+0000 7f5f32a00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f0c06a430 0x7f5f0c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.354+0000 7f5f32a00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f0c06a430 0x7f5f0c06c8f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000fd0 tx=0x7f5f1c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.357+0000 7f5f2b400640 1 -- 172.21.15.39:0/1161472011 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5f34141510 con 0x7f5f3415cbe0 2024-01-24T17:48:07.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.594+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f341414d0 con 0x7f5f3415cbe0 2024-01-24T17:48:07.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.596+0000 7f5f2b400640 1 -- 172.21.15.39:0/1161472011 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f5f24054360 con 0x7f5f3415cbe0 2024-01-24T17:48:07.598 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:07.598 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/57362003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":57362003},{"type":"v1","addr":"172.21.15.100:6836","nonce":57362003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6139,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/3662299631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":3662299631},{"type":"v1","addr":"172.21.15.100:6837","nonce":3662299631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2829786836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2829786836},{"type":"v1","addr":"172.21.15.39:6833","nonce":2829786836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:47:31.936549+0000","modified":"2024-01-24T17:47:35.302809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7293":{"gid":7293,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3073867595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3073867595},{"type":"v1","addr":"172.21.15.39:6835","nonce":3073867595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-24T17:48:07.598 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 51 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.598+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f0c06a430 msgr2=0x7f5f0c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.598+0000 7f5f39e00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f0c06a430 0x7f5f0c06c8f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000fd0 tx=0x7f5f1c009040 comp rx=0 tx=0).stop 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.598+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f3415cbe0 msgr2=0x7f5f340cb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.598+0000 7f5f39e00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f3415cbe0 0x7f5f340cb780 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400d620 tx=0x7f5f2400dae0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.599+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 shutdown_connections 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.599+0000 7f5f39e00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5f34164b40 0x7f5f340cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.599+0000 7f5f39e00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5f0c06a430 0x7f5f0c06c8f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.599+0000 7f5f39e00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f3415cbe0 0x7f5f340cb780 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.599+0000 7f5f39e00640 1 --2- 172.21.15.39:0/1161472011 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f5f340d0e90 0x7f5f3415f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.599+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 >> 172.21.15.39:0/1161472011 conn(0x7f5f3409bc80 msgr2=0x7f5f34157ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:07.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.599+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 shutdown_connections 2024-01-24T17:48:07.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.599+0000 7f5f39e00640 1 -- 172.21.15.39:0/1161472011 wait complete. 2024-01-24T17:48:07.621 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 session ls 2024-01-24T17:48:07.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.836+0000 7f8da9e00640 1 Processor -- start 2024-01-24T17:48:07.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.836+0000 7f8da9e00640 1 -- start start 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8da40ce8c0 0x7f8da406fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da415ac70 0x7f8da4070290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8da40ceca0 0x7f8da415e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da40b12b0 con 0x7f8da40ceca0 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da4155170 con 0x7f8da40ce8c0 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da415b1e0 con 0x7f8da415ac70 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8da40ceca0 0x7f8da415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8da40ceca0 0x7f8da415e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44184/0 (socket says 172.21.15.39:44184) 2024-01-24T17:48:07.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.838+0000 7f8da9400640 1 -- 172.21.15.39:0/4161240116 learned_addr learned my addr 172.21.15.39:0/4161240116 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:07.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.839+0000 7f8d9be00640 1 --2- 172.21.15.39:0/4161240116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da415ac70 0x7f8da4070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.839+0000 7f8da8a00640 1 --2- 172.21.15.39:0/4161240116 >> v2:172.21.15.100:3300/0 conn(0x7f8da40ce8c0 0x7f8da406fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.839+0000 7f8da9400640 1 -- 172.21.15.39:0/4161240116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da415ac70 msgr2=0x7f8da4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.839+0000 7f8da9400640 1 --2- 172.21.15.39:0/4161240116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da415ac70 0x7f8da4070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.839+0000 7f8da9400640 1 -- 172.21.15.39:0/4161240116 >> v2:172.21.15.100:3300/0 conn(0x7f8da40ce8c0 msgr2=0x7f8da406fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.839+0000 7f8da9400640 1 --2- 172.21.15.39:0/4161240116 >> v2:172.21.15.100:3300/0 conn(0x7f8da40ce8c0 0x7f8da406fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.839+0000 7f8da9400640 1 -- 172.21.15.39:0/4161240116 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da415b4c0 con 0x7f8da40ceca0 2024-01-24T17:48:07.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.839+0000 7f8da8a00640 1 --2- 172.21.15.39:0/4161240116 >> v2:172.21.15.100:3300/0 conn(0x7f8da40ce8c0 0x7f8da406fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:07.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.840+0000 7f8da9400640 1 --2- 172.21.15.39:0/4161240116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da415e260 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f8da406dfa0 tx=0x7f8d9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbde9bc0b80f96b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.840+0000 7f8da3e00640 1 -- 172.21.15.39:0/4161240116 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c02ca70 con 0x7f8da40ceca0 2024-01-24T17:48:07.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.840+0000 7f8da3e00640 1 -- 172.21.15.39:0/4161240116 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9c02cbd0 con 0x7f8da40ceca0 2024-01-24T17:48:07.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.840+0000 7f8da3e00640 1 -- 172.21.15.39:0/4161240116 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0343f0 con 0x7f8da40ceca0 2024-01-24T17:48:07.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 -- 172.21.15.39:0/4161240116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 msgr2=0x7f8da415e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 --2- 172.21.15.39:0/4161240116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da415e260 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f8da406dfa0 tx=0x7f8d9c02bed0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 -- 172.21.15.39:0/4161240116 shutdown_connections 2024-01-24T17:48:07.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 --2- 172.21.15.39:0/4161240116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da415e260 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 --2- 172.21.15.39:0/4161240116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da415ac70 0x7f8da4070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 --2- 172.21.15.39:0/4161240116 >> v2:172.21.15.100:3300/0 conn(0x7f8da40ce8c0 0x7f8da406fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 -- 172.21.15.39:0/4161240116 >> 172.21.15.39:0/4161240116 conn(0x7f8da406a200 msgr2=0x7f8da40affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:07.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 -- 172.21.15.39:0/4161240116 shutdown_connections 2024-01-24T17:48:07.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.841+0000 7f8da9e00640 1 -- 172.21.15.39:0/4161240116 wait complete. 2024-01-24T17:48:07.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.843+0000 7f8da9e00640 1 Processor -- start 2024-01-24T17:48:07.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.843+0000 7f8da9e00640 1 -- start start 2024-01-24T17:48:07.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8da9e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da40ce8c0 0x7f8da41ed560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8da9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da40c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8da9e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8da415ac70 0x7f8da40c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8da9e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da40c7cd0 con 0x7f8da40ceca0 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8da9e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da40c7050 con 0x7f8da415ac70 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8da9e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da40c7330 con 0x7f8da40ce8c0 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8d9be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da40c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8d9be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da40c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44198/0 (socket says 172.21.15.39:44198) 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.844+0000 7f8d9be00640 1 -- 172.21.15.39:0/2677906983 learned_addr learned my addr 172.21.15.39:0/2677906983 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8da8a00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da40ce8c0 0x7f8da41ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8da9400640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8da415ac70 0x7f8da40c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8d9be00640 1 -- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da40ce8c0 msgr2=0x7f8da41ed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8d9be00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da40ce8c0 0x7f8da41ed560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8d9be00640 1 -- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8da415ac70 msgr2=0x7f8da40c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8d9be00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8da415ac70 0x7f8da40c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8d9be00640 1 -- 172.21.15.39:0/2677906983 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c009b30 con 0x7f8da40ceca0 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8da8a00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da40ce8c0 0x7f8da41ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:07.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8da9400640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8da415ac70 0x7f8da40c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:07.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8d9be00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da40c64e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00c7f0 tx=0x7f8d8c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.845+0000 7f8da1600640 1 -- 172.21.15.39:0/2677906983 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c004220 con 0x7f8da40ceca0 2024-01-24T17:48:07.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.846+0000 7f8da1600640 1 -- 172.21.15.39:0/2677906983 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c0043a0 con 0x7f8da40ceca0 2024-01-24T17:48:07.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.846+0000 7f8da1600640 1 -- 172.21.15.39:0/2677906983 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c007860 con 0x7f8da40ceca0 2024-01-24T17:48:07.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.846+0000 7f8da9e00640 1 -- 172.21.15.39:0/2677906983 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da40c7610 con 0x7f8da40ceca0 2024-01-24T17:48:07.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.846+0000 7f8da9e00640 1 -- 172.21.15.39:0/2677906983 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da41f04c0 con 0x7f8da40ceca0 2024-01-24T17:48:07.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.848+0000 7f8da1600640 1 -- 172.21.15.39:0/2677906983 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8d8c007bd0 con 0x7f8da40ceca0 2024-01-24T17:48:07.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.849+0000 7f8da1600640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d7c06a430 0x7f8d7c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.849+0000 7f8da1600640 1 -- 172.21.15.39:0/2677906983 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d8c088970 con 0x7f8da40ceca0 2024-01-24T17:48:07.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.850+0000 7f8dadf95740 1 Processor -- start 2024-01-24T17:48:07.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.850+0000 7f8da8a00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d7c06a430 0x7f8d7c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.850+0000 7f8dadf95740 1 -- start start 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c59b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8660 0x55590c5cb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c59b950 0x55590c5cb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8dadf95740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590c50b210 con 0x55590c5c8660 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8dadf95740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590c5089a0 con 0x55590c59b950 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8dadf95740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55590c504290 con 0x55590c5c8280 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8da8a00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d7c06a430 0x7f8d7c06c8f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400b840 tx=0x7f8d94008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8d9be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8660 0x55590c5cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8d9be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8660 0x55590c5cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44214/0 (socket says 172.21.15.39:44214) 2024-01-24T17:48:07.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.851+0000 7f8d9be00640 1 -- 172.21.15.39:0/1340073298 learned_addr learned my addr 172.21.15.39:0/1340073298 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:07.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8da9400640 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c59b950 0x55590c5cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8da8a00640 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c59b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d9be00640 1 -- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 msgr2=0x55590c59b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d9be00640 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c59b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d9be00640 1 -- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c59b950 msgr2=0x55590c5cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d9be00640 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c59b950 0x55590c5cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d9be00640 1 -- 172.21.15.39:0/1340073298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55590c4c0f20 con 0x55590c5c8660 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8da9400640 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c59b950 0x55590c5cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d9be00640 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8660 0x55590c5cb010 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c05f630 tx=0x7f8d8c0575a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11fdeaa622b1432 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d99600640 1 -- 172.21.15.39:0/1340073298 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c05b9b0 con 0x55590c5c8660 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d99600640 1 -- 172.21.15.39:0/1340073298 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c05fb60 con 0x55590c5c8660 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.852+0000 7f8d99600640 1 -- 172.21.15.39:0/1340073298 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c06f080 con 0x55590c5c8660 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 -- 172.21.15.39:0/1340073298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8660 msgr2=0x55590c5cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8660 0x55590c5cb010 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c05f630 tx=0x7f8d8c0575a0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 -- 172.21.15.39:0/1340073298 shutdown_connections 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c59b950 0x55590c5cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8660 0x55590c5cb010 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1340073298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c59b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 -- 172.21.15.39:0/1340073298 >> 172.21.15.39:0/1340073298 conn(0x55590c59d720 msgr2=0x55590c5e6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:07.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 -- 172.21.15.39:0/1340073298 shutdown_connections 2024-01-24T17:48:07.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.853+0000 7f8dadf95740 1 -- 172.21.15.39:0/1340073298 wait complete. 2024-01-24T17:48:07.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.854+0000 7f8dadf95740 1 -- start start 2024-01-24T17:48:07.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.854+0000 7f8dadf95740 1 Processor -- start 2024-01-24T17:48:07.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8280 0x55590c5c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8660 0x55590c5c0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5f5e10 0x55590c5ce070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8dadf95740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590c4a5890 con 0x55590c5c8280 2024-01-24T17:48:07.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8dadf95740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590c4a5530 con 0x55590c5f5e10 2024-01-24T17:48:07.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8dadf95740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55590c201f90 con 0x55590c5c8660 2024-01-24T17:48:07.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8dadf95740 10 client.0 _ll_register_callbacks cb 0x55590c4fdbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:48:07.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8da8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8280 0x55590c5c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8da8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8280 0x55590c5c3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44228/0 (socket says 172.21.15.39:44228) 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.855+0000 7f8da8a00640 1 -- 172.21.15.39:0/1216804710 learned_addr learned my addr 172.21.15.39:0/1216804710 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8da9400640 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5f5e10 0x55590c5ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8d9be00640 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8660 0x55590c5c0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8da8a00640 1 -- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8660 msgr2=0x55590c5c0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8da8a00640 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8660 0x55590c5c0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8da8a00640 1 -- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5f5e10 msgr2=0x55590c5ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8da8a00640 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5f5e10 0x55590c5ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8da8a00640 1 -- 172.21.15.39:0/1216804710 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da41f04c0 con 0x55590c5c8280 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8da9400640 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5f5e10 0x55590c5ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8da8a00640 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8280 0x55590c5c3d60 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f8d940126f0 tx=0x7f8d94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6297875e0a8169 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.856+0000 7f8d92000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8d92000640 1 -- 172.21.15.39:0/1216804710 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9400b050 con 0x55590c5c8280 2024-01-24T17:48:07.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 1 -- 172.21.15.39:0/1216804710 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55590c52f5a0 con 0x55590c5c8280 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8d92000640 1 -- 172.21.15.39:0/1216804710 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d94013800 con 0x55590c5c8280 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8d92000640 1 -- 172.21.15.39:0/1216804710 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d940139c0 con 0x55590c5c8280 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 10 client.7605 fetch_fsmap learned FSMap version 51 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 1 -- 172.21.15.39:0/1216804710 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55590c502910 con 0x55590c5c8280 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8d92000640 1 -- 172.21.15.39:0/1216804710 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55590c52f5a0 con 0x55590c5c8280 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8d92000640 1 -- 172.21.15.39:0/1216804710 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 51) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8d94013b20 con 0x55590c5c8280 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 10 client.7605 fetch_fsmap finished waiting for FSMap version 51 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 10 client.7605 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 20 client.7605 populate_metadata read hostname 'smithi039' 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55590c5c8ac0 0x55590c5c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 4 client.7605 mds_command: new command op to 7293 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.857+0000 7f8dadf95740 1 -- 172.21.15.39:0/1216804710 --> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55590c4f7f40 con 0x55590c5c8ac0 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.858+0000 7f8d9be00640 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55590c5c8ac0 0x55590c5c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.858+0000 7f8d9be00640 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55590c5c8ac0 0x55590c5c8ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7311f53f73dc2f8 server_cookie=80bff799f3336e7a in_seq=0 out_seq=0 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.858+0000 7f8d92000640 10 client.7605 ms_handle_connect on v2:172.21.15.39:6834/3073867595 2024-01-24T17:48:07.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8d92000640 1 -- 172.21.15.39:0/1216804710 <== mds.0 v2:172.21.15.39:6834/3073867595 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55590c4f7f40 con 0x55590c5c8ac0 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8d92000640 10 client.7605 handle_command_reply: tid=0 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 client.7605 shutdown 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 -- 172.21.15.39:0/1216804710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8280 msgr2=0x55590c5c3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8280 0x55590c5c3d60 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f8d940126f0 tx=0x7f8d94004730 comp rx=0 tx=0).stop 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 -- 172.21.15.39:0/1216804710 shutdown_connections 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55590c5c8ac0 0x55590c5c8ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5f5e10 0x55590c5ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8660 0x55590c5c0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1216804710 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5c8280 0x55590c5c3d60 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 -- 172.21.15.39:0/1216804710 >> 172.21.15.39:0/1216804710 conn(0x55590c59b950 msgr2=0x55590c59f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8d92000640 0 client.7605 ms_handle_reset on v2:172.21.15.39:6834/3073867595 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 -- 172.21.15.39:0/1216804710 shutdown_connections 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.859+0000 7f8dadf95740 1 -- 172.21.15.39:0/1216804710 wait complete. 2024-01-24T17:48:07.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.860+0000 7f8dadf95740 20 client.7605 trim_cache size 0 max 16384 2024-01-24T17:48:07.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.876+0000 7f8dadf95740 1 Processor -- start 2024-01-24T17:48:07.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.876+0000 7f8dadf95740 1 -- start start 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c5bfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 0x55590c59e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c59ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8dadf95740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590c5c00c0 con 0x55590c5f5e10 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8dadf95740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590c59f3b0 con 0x55590c5c8660 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8dadf95740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55590c5f5c70 con 0x55590c5c8280 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8da9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c59ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8da9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c59ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44236/0 (socket says 172.21.15.39:44236) 2024-01-24T17:48:07.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.877+0000 7f8da9400640 1 -- 172.21.15.39:0/3513982094 learned_addr learned my addr 172.21.15.39:0/3513982094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:07.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.878+0000 7f8d9be00640 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 0x55590c59e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.878+0000 7f8da8a00640 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c5bfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.878+0000 7f8da9400640 1 -- 172.21.15.39:0/3513982094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 msgr2=0x55590c5bfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.878+0000 7f8da9400640 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c5bfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.878+0000 7f8da9400640 1 -- 172.21.15.39:0/3513982094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 msgr2=0x55590c59e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.878+0000 7f8da9400640 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 0x55590c59e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.878+0000 7f8da9400640 1 -- 172.21.15.39:0/3513982094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55590c5390a0 con 0x55590c5f5e10 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.878+0000 7f8da9400640 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c59ee70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c009f40 tx=0x7f8d9c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725035841693c1a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8d9a000640 1 -- 172.21.15.39:0/3513982094 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c048610 con 0x55590c5f5e10 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8d9a000640 1 -- 172.21.15.39:0/3513982094 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9c048770 con 0x55590c5f5e10 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8d9a000640 1 -- 172.21.15.39:0/3513982094 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c048b00 con 0x55590c5f5e10 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8dadf95740 1 -- 172.21.15.39:0/3513982094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 msgr2=0x55590c59ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8dadf95740 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c59ee70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c009f40 tx=0x7f8d9c02e000 comp rx=0 tx=0).stop 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8dadf95740 1 -- 172.21.15.39:0/3513982094 shutdown_connections 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8dadf95740 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c59ee70 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8dadf95740 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 0x55590c59e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8dadf95740 1 --2- 172.21.15.39:0/3513982094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c5bfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.879+0000 7f8dadf95740 1 -- 172.21.15.39:0/3513982094 >> 172.21.15.39:0/3513982094 conn(0x55590c59b950 msgr2=0x55590c5c0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.880+0000 7f8dadf95740 1 -- 172.21.15.39:0/3513982094 shutdown_connections 2024-01-24T17:48:07.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.880+0000 7f8dadf95740 1 -- 172.21.15.39:0/3513982094 wait complete. 2024-01-24T17:48:07.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.880+0000 7f8dadf95740 1 -- start start 2024-01-24T17:48:07.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.880+0000 7f8dadf95740 1 Processor -- start 2024-01-24T17:48:07.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.881+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c59e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.881+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 0x55590c59ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.881+0000 7f8dadf95740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c5e9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.881+0000 7f8dadf95740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590c59fd70 con 0x55590c5f5e10 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.881+0000 7f8dadf95740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590c5cc5d0 con 0x55590c5c8660 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8dadf95740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55590c5c6b40 con 0x55590c5c8280 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8dadf95740 10 client.0 _ll_register_callbacks cb 0x7f8d9400b530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c5e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c5e9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44240/0 (socket says 172.21.15.39:44240) 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 -- 172.21.15.39:0/1746412932 learned_addr learned my addr 172.21.15.39:0/1746412932 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da8a00640 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c59e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8d9be00640 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 0x55590c59ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 -- 172.21.15.39:0/1746412932 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 msgr2=0x55590c59e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c59e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 -- 172.21.15.39:0/1746412932 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 msgr2=0x55590c59ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:07.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 0x55590c59ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 -- 172.21.15.39:0/1746412932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da415b4c0 con 0x55590c5f5e10 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.882+0000 7f8da9400640 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c5e9900 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c02c4b0 tx=0x7f8d9c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db61c64f72c17300 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8d92000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8d92000640 1 -- 172.21.15.39:0/1746412932 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c048ea0 con 0x55590c5f5e10 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8d92000640 1 -- 172.21.15.39:0/1746412932 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d9c052cf0 con 0x55590c5f5e10 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8dadf95740 1 -- 172.21.15.39:0/1746412932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d8c05fb60 con 0x55590c5f5e10 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8d92000640 1 -- 172.21.15.39:0/1746412932 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d9c02ca30 con 0x55590c5f5e10 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8dadf95740 10 client.7611 fetch_fsmap learned FSMap version 51 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8dadf95740 1 -- 172.21.15.39:0/1746412932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55590c4a4c40 con 0x55590c5f5e10 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8d92000640 1 -- 172.21.15.39:0/1746412932 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c05fb60 con 0x55590c5f5e10 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8d92000640 1 -- 172.21.15.39:0/1746412932 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 51) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8d9c051640 con 0x55590c5f5e10 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8dadf95740 10 client.7611 fetch_fsmap finished waiting for FSMap version 51 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.883+0000 7f8dadf95740 10 client.7611 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.884+0000 7f8dadf95740 20 client.7611 populate_metadata read hostname 'smithi039' 2024-01-24T17:48:07.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.884+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55590c652e80 0x55590c5e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:07.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.884+0000 7f8dadf95740 4 client.7611 mds_command: new command op to 7293 tid=0[{"prefix": "session ls"}] 2024-01-24T17:48:07.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.884+0000 7f8dadf95740 1 -- 172.21.15.39:0/1746412932 --> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] -- command(tid 0: {"prefix": "session ls"}) v1 -- 0x55590c4fd310 con 0x55590c652e80 2024-01-24T17:48:07.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.884+0000 7f8da8a00640 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55590c652e80 0x55590c5e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.884+0000 7f8da8a00640 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55590c652e80 0x55590c5e9e40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b4b0fdaccd10a8c server_cookie=5f9a1a18fde7bee3 in_seq=0 out_seq=0 2024-01-24T17:48:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.884+0000 7f8d92000640 10 client.7611 ms_handle_connect on v2:172.21.15.39:6834/3073867595 2024-01-24T17:48:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8d92000640 1 -- 172.21.15.39:0/1746412932 <== mds.0 v2:172.21.15.39:6834/3073867595 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55590c4fd310 con 0x55590c652e80 2024-01-24T17:48:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8d92000640 10 client.7611 handle_command_reply: tid=0 2024-01-24T17:48:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 client.7611 shutdown 2024-01-24T17:48:07.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 -- 172.21.15.39:0/1746412932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 msgr2=0x55590c5e9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c5e9900 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c02c4b0 tx=0x7f8d9c004ab0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 -- 172.21.15.39:0/1746412932 shutdown_connections 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.39:6834/3073867595,v1:172.21.15.39:6835/3073867595] conn(0x55590c652e80 0x55590c5e9e40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55590c5f5e10 0x55590c5e9900 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55590c5c8660 0x55590c59ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8d92000640 0 client.7611 ms_handle_reset on v2:172.21.15.39:6834/3073867595 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 --2- 172.21.15.39:0/1746412932 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55590c5c8280 0x55590c59e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 -- 172.21.15.39:0/1746412932 >> 172.21.15.39:0/1746412932 conn(0x55590c59b950 msgr2=0x55590c5c7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 -- 172.21.15.39:0/1746412932 shutdown_connections 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 1 -- 172.21.15.39:0/1746412932 wait complete. 2024-01-24T17:48:07.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.885+0000 7f8dadf95740 20 client.7611 trim_cache size 0 max 16384 2024-01-24T17:48:07.891 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-01-24T17:48:07.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 -- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d7c06a430 msgr2=0x7f8d7c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d7c06a430 0x7f8d7c06c8f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400b840 tx=0x7f8d94008040 comp rx=0 tx=0).stop 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 -- 172.21.15.39:0/2677906983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 msgr2=0x7f8da40c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da40c64e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00c7f0 tx=0x7f8d8c00ccb0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 -- 172.21.15.39:0/2677906983 shutdown_connections 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8da415ac70 0x7f8da40c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f8d7c06a430 0x7f8d7c06c8f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8da40ceca0 0x7f8da40c64e0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 --2- 172.21.15.39:0/2677906983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8da40ce8c0 0x7f8da41ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 -- 172.21.15.39:0/2677906983 >> 172.21.15.39:0/2677906983 conn(0x7f8da406a200 msgr2=0x7f8da415d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 -- 172.21.15.39:0/2677906983 shutdown_connections 2024-01-24T17:48:07.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:07.886+0000 7f8da9e00640 1 -- 172.21.15.39:0/2677906983 wait complete. 2024-01-24T17:48:07.907 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-01-24T17:48:08.144 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-01-24T17:48:08.145 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-01-24T17:48:08.145 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:48:08.145 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:48:08.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:48:08.387 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:48:08.387 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:48:08.387 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:08.447 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:48:08.447 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:48:08.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:08.447 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:48:08.500 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-24T17:48:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:48:08.501 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:48:08.501 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:08.558 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:48:08.558 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:48:08.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:08.559 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:48:08.610 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-24T17:48:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:48:08.612 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:48:08.612 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:08.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:48:08.674 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:48:08.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:08.675 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-24T17:48:08.728 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-24T17:48:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:48:08.730 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:48:08.730 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:08.787 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:48:08.787 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:48:08.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:08.787 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-24T17:48:08.836 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-24T17:48:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:48:08.837 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:09.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.065+0000 7f3167e00640 1 Processor -- start 2024-01-24T17:48:09.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.065+0000 7f3167e00640 1 -- start start 2024-01-24T17:48:09.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.066+0000 7f3167e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3168070540 0x7f316815ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.066+0000 7f3167e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f31680b3e00 0x7f31680c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.066+0000 7f3167e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31681551b0 con 0x7f31680c8f70 2024-01-24T17:48:09.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.066+0000 7f3167e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316815e410 con 0x7f31680b3e00 2024-01-24T17:48:09.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.066+0000 7f3167e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316815b1b0 con 0x7f3168070540 2024-01-24T17:48:09.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.066+0000 7f3166000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f31680b3e00 0x7f31680c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.067+0000 7f3166000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f31680b3e00 0x7f31680c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44242/0 (socket says 172.21.15.39:44242) 2024-01-24T17:48:09.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.067+0000 7f3166000640 1 -- 172.21.15.39:0/3333263595 learned_addr learned my addr 172.21.15.39:0/3333263595 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:09.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.067+0000 7f3166a00640 1 --2- 172.21.15.39:0/3333263595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3168070540 0x7f316815ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.067+0000 7f3166000640 1 -- 172.21.15.39:0/3333263595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3168070540 msgr2=0x7f316815ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:09.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.067+0000 7f3166000640 1 --2- 172.21.15.39:0/3333263595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3168070540 0x7f316815ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.067+0000 7f3166000640 1 -- 172.21.15.39:0/3333263595 >> v1:172.21.15.100:6789/0 conn(0x7f31680c8f70 legacy=0x7f316815e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.067+0000 7f3166000640 1 -- 172.21.15.39:0/3333263595 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316815b480 con 0x7f31680b3e00 2024-01-24T17:48:09.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.068+0000 7f3166000640 1 --2- 172.21.15.39:0/3333263595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680b3e00 0x7f31680c8a30 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f3150009870 tx=0x7f315002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ab19c2efa705fb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:09.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.069+0000 7f3165600640 1 -- 172.21.15.39:0/3333263595 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150030840 con 0x7f31680b3e00 2024-01-24T17:48:09.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.069+0000 7f3165600640 1 -- 172.21.15.39:0/3333263595 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31500309a0 con 0x7f31680b3e00 2024-01-24T17:48:09.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.069+0000 7f3165600640 1 -- 172.21.15.39:0/3333263595 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150030d30 con 0x7f31680b3e00 2024-01-24T17:48:09.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.069+0000 7f3167e00640 1 -- 172.21.15.39:0/3333263595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680b3e00 msgr2=0x7f31680c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.069+0000 7f3167e00640 1 --2- 172.21.15.39:0/3333263595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680b3e00 0x7f31680c8a30 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f3150009870 tx=0x7f315002b980 comp rx=0 tx=0).stop 2024-01-24T17:48:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.070+0000 7f3167e00640 1 -- 172.21.15.39:0/3333263595 shutdown_connections 2024-01-24T17:48:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.070+0000 7f3167e00640 1 --2- 172.21.15.39:0/3333263595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680b3e00 0x7f31680c8a30 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.070+0000 7f3167e00640 1 --2- 172.21.15.39:0/3333263595 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3168070540 0x7f316815ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.070+0000 7f3167e00640 1 -- 172.21.15.39:0/3333263595 >> 172.21.15.39:0/3333263595 conn(0x7f316806a340 msgr2=0x7f31680cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.070+0000 7f3167e00640 1 -- 172.21.15.39:0/3333263595 shutdown_connections 2024-01-24T17:48:09.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.070+0000 7f3167e00640 1 -- 172.21.15.39:0/3333263595 wait complete. 2024-01-24T17:48:09.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.072+0000 7f3167e00640 1 Processor -- start 2024-01-24T17:48:09.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.072+0000 7f3167e00640 1 -- start start 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3168070540 0x7f31680c2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31680b3e00 0x7f31680c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680c8f70 0x7f31680b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31681f5650 con 0x7f31680c8f70 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31681f61d0 con 0x7f3168070540 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31680c8480 con 0x7f31680b3e00 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680c8f70 0x7f31680b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680c8f70 0x7f31680b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44244/0 (socket says 172.21.15.39:44244) 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3167400640 1 -- 172.21.15.39:0/798246498 learned_addr learned my addr 172.21.15.39:0/798246498 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.073+0000 7f3166a00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3168070540 0x7f31680c2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.074+0000 7f3166000640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31680b3e00 0x7f31680c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.074+0000 7f3167400640 1 -- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31680b3e00 msgr2=0x7f31680c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.074+0000 7f3167400640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31680b3e00 0x7f31680c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.074+0000 7f3167400640 1 -- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3168070540 msgr2=0x7f31680c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.074+0000 7f3167400640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3168070540 0x7f31680c2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.074+0000 7f3167400640 1 -- 172.21.15.39:0/798246498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31500094d0 con 0x7f31680c8f70 2024-01-24T17:48:09.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.074+0000 7f3166a00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3168070540 0x7f31680c2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:48:09.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.074+0000 7f3167400640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680c8f70 0x7f31680b2910 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f316000c6b0 tx=0x7f316000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.075+0000 7f315ea00640 1 -- 172.21.15.39:0/798246498 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160003b70 con 0x7f31680c8f70 2024-01-24T17:48:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.075+0000 7f315ea00640 1 -- 172.21.15.39:0/798246498 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3160003cd0 con 0x7f31680c8f70 2024-01-24T17:48:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.075+0000 7f315ea00640 1 -- 172.21.15.39:0/798246498 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3160005310 con 0x7f31680c8f70 2024-01-24T17:48:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.075+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31681f58a0 con 0x7f31680c8f70 2024-01-24T17:48:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.075+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31681f5e60 con 0x7f31680c8f70 2024-01-24T17:48:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.076+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31681f6360 con 0x7f31680c8f70 2024-01-24T17:48:09.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.077+0000 7f315ea00640 1 -- 172.21.15.39:0/798246498 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3160005470 con 0x7f31680c8f70 2024-01-24T17:48:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.078+0000 7f315ea00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f314006a430 0x7f314006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.078+0000 7f315ea00640 1 -- 172.21.15.39:0/798246498 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3160088700 con 0x7f31680c8f70 2024-01-24T17:48:09.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.078+0000 7f3166a00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f314006a430 0x7f314006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.079+0000 7f3166a00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f314006a430 0x7f314006c8f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3158007a50 tx=0x7f3158008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:09.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.082+0000 7f315ea00640 1 -- 172.21.15.39:0/798246498 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3160053c50 con 0x7f31680c8f70 2024-01-24T17:48:09.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.313+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f316804dff0 con 0x7f31680c8f70 2024-01-24T17:48:09.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.314+0000 7f315ea00640 1 -- 172.21.15.39:0/798246498 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f3160057270 con 0x7f31680c8f70 2024-01-24T17:48:09.315 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:09.316 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/57362003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":57362003},{"type":"v1","addr":"172.21.15.100:6836","nonce":57362003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6139,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/3662299631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":3662299631},{"type":"v1","addr":"172.21.15.100:6837","nonce":3662299631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2829786836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2829786836},{"type":"v1","addr":"172.21.15.39:6833","nonce":2829786836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:47:31.936549+0000","modified":"2024-01-24T17:47:35.302809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7293":{"gid":7293,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3073867595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3073867595},{"type":"v1","addr":"172.21.15.39:6835","nonce":3073867595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-24T17:48:09.316 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 51 2024-01-24T17:48:09.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.316+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f314006a430 msgr2=0x7f314006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.316+0000 7f3167e00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f314006a430 0x7f314006c8f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3158007a50 tx=0x7f3158008040 comp rx=0 tx=0).stop 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.316+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680c8f70 msgr2=0x7f31680b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.316+0000 7f3167e00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680c8f70 0x7f31680b2910 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f316000c6b0 tx=0x7f316000cb70 comp rx=0 tx=0).stop 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.318+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 shutdown_connections 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.318+0000 7f3167e00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f314006a430 0x7f314006c8f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.318+0000 7f3167e00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31680c8f70 0x7f31680b2910 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.318+0000 7f3167e00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f31680b3e00 0x7f31680c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.318+0000 7f3167e00640 1 --2- 172.21.15.39:0/798246498 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3168070540 0x7f31680c2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.318+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 >> 172.21.15.39:0/798246498 conn(0x7f316806a340 msgr2=0x7f31680af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.318+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 shutdown_connections 2024-01-24T17:48:09.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.318+0000 7f3167e00640 1 -- 172.21.15.39:0/798246498 wait complete. 2024-01-24T17:48:09.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:09.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.549+0000 7fcae0800640 1 Processor -- start 2024-01-24T17:48:09.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.549+0000 7fcae0800640 1 -- start start 2024-01-24T17:48:09.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcae0800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fcad80b0010 0x7fcad815bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcae0800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fcad815f390 0x7fcad80c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcae0800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad8070470 con 0x7fcad8070920 2024-01-24T17:48:09.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcae0800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad815ef00 con 0x7fcad815f390 2024-01-24T17:48:09.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcae0800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad80710a0 con 0x7fcad80b0010 2024-01-24T17:48:09.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcadd400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fcad8070920 0x7fcad80b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35652/0 (socket says 172.21.15.39:35652) 2024-01-24T17:48:09.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcadd400640 1 -- 172.21.15.39:0/640563877 learned_addr learned my addr 172.21.15.39:0/640563877 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:09.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcade800640 1 --2- 172.21.15.39:0/640563877 >> v2:172.21.15.39:3300/0 conn(0x7fcad815f390 0x7fcad80c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.551+0000 7fcadde00640 1 --2- 172.21.15.39:0/640563877 >> v2:172.21.15.100:3300/0 conn(0x7fcad80b0010 0x7fcad815bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.552+0000 7fcadca00640 1 -- 172.21.15.39:0/640563877 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049959990 0 0) 0x7fcad8070470 con 0x7fcad8070920 2024-01-24T17:48:09.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.552+0000 7fcade800640 1 -- 172.21.15.39:0/640563877 >> v2:172.21.15.100:3300/0 conn(0x7fcad80b0010 msgr2=0x7fcad815bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:09.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.552+0000 7fcade800640 1 --2- 172.21.15.39:0/640563877 >> v2:172.21.15.100:3300/0 conn(0x7fcad80b0010 0x7fcad815bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.552+0000 7fcade800640 1 -- 172.21.15.39:0/640563877 >> v1:172.21.15.39:6789/0 conn(0x7fcad8070920 legacy=0x7fcad80b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.552+0000 7fcade800640 1 -- 172.21.15.39:0/640563877 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad815c0d0 con 0x7fcad815f390 2024-01-24T17:48:09.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.553+0000 7fcade800640 1 --2- 172.21.15.39:0/640563877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80c9450 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fcad80b1010 tx=0x7fcad002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2fce5aa66e0d26 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:09.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.554+0000 7fcadca00640 1 -- 172.21.15.39:0/640563877 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad002cd10 con 0x7fcad815f390 2024-01-24T17:48:09.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.554+0000 7fcadca00640 1 -- 172.21.15.39:0/640563877 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad002ce70 con 0x7fcad815f390 2024-01-24T17:48:09.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.554+0000 7fcadca00640 1 -- 172.21.15.39:0/640563877 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0034640 con 0x7fcad815f390 2024-01-24T17:48:09.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.554+0000 7fcae0800640 1 -- 172.21.15.39:0/640563877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 msgr2=0x7fcad80c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.554+0000 7fcae0800640 1 --2- 172.21.15.39:0/640563877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80c9450 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fcad80b1010 tx=0x7fcad002bf60 comp rx=0 tx=0).stop 2024-01-24T17:48:09.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.555+0000 7fcae0800640 1 -- 172.21.15.39:0/640563877 shutdown_connections 2024-01-24T17:48:09.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.555+0000 7fcae0800640 1 --2- 172.21.15.39:0/640563877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80c9450 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.555+0000 7fcae0800640 1 --2- 172.21.15.39:0/640563877 >> v2:172.21.15.100:3300/0 conn(0x7fcad80b0010 0x7fcad815bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.555+0000 7fcae0800640 1 -- 172.21.15.39:0/640563877 >> 172.21.15.39:0/640563877 conn(0x7fcad806bf20 msgr2=0x7fcad806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:09.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.555+0000 7fcae0800640 1 -- 172.21.15.39:0/640563877 shutdown_connections 2024-01-24T17:48:09.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.555+0000 7fcae0800640 1 -- 172.21.15.39:0/640563877 wait complete. 2024-01-24T17:48:09.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.556+0000 7fcae0800640 1 Processor -- start 2024-01-24T17:48:09.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.557+0000 7fcae0800640 1 -- start start 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.557+0000 7fcae0800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcad8070920 0x7fcad8156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcae0800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcad80b0010 0x7fcad80ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcae0800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcae0800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8157520 con 0x7fcad815f390 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcae0800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80ac6a0 con 0x7fcad80b0010 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcae0800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80aeba0 con 0x7fcad8070920 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcade800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcade800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44256/0 (socket says 172.21.15.39:44256) 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcade800640 1 -- 172.21.15.39:0/1909237577 learned_addr learned my addr 172.21.15.39:0/1909237577 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcade800640 1 -- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcad8070920 msgr2=0x7fcad8156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcade800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcad8070920 0x7fcad8156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcade800640 1 -- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcad80b0010 msgr2=0x7fcad80ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcadd400640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcad80b0010 0x7fcad80ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcade800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcad80b0010 0x7fcad80ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.558+0000 7fcade800640 1 -- 172.21.15.39:0/1909237577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad0009a70 con 0x7fcad815f390 2024-01-24T17:48:09.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.559+0000 7fcade800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80af030 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcad002c570 tx=0x7fcad0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:09.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.559+0000 7fcad6000640 1 -- 172.21.15.39:0/1909237577 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad003e610 con 0x7fcad815f390 2024-01-24T17:48:09.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.559+0000 7fcad6000640 1 -- 172.21.15.39:0/1909237577 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad003e770 con 0x7fcad815f390 2024-01-24T17:48:09.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.559+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad80af570 con 0x7fcad815f390 2024-01-24T17:48:09.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.559+0000 7fcad6000640 1 -- 172.21.15.39:0/1909237577 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad003eae0 con 0x7fcad815f390 2024-01-24T17:48:09.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.559+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad80ae270 con 0x7fcad815f390 2024-01-24T17:48:09.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.560+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad804dff0 con 0x7fcad815f390 2024-01-24T17:48:09.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.561+0000 7fcad6000640 1 -- 172.21.15.39:0/1909237577 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fcad003b050 con 0x7fcad815f390 2024-01-24T17:48:09.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.562+0000 7fcad6000640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab006a430 0x7fcab006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:09.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.562+0000 7fcad6000640 1 -- 172.21.15.39:0/1909237577 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcad00ab110 con 0x7fcad815f390 2024-01-24T17:48:09.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.563+0000 7fcadde00640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab006a430 0x7fcab006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:09.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.564+0000 7fcadde00640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab006a430 0x7fcab006c8f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fcac80098f0 tx=0x7fcac8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:09.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.567+0000 7fcad6000640 1 -- 172.21.15.39:0/1909237577 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fcad0076780 con 0x7fcad815f390 2024-01-24T17:48:09.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.767+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcad80aece0 con 0x7fcad815f390 2024-01-24T17:48:09.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.769+0000 7fcad6000640 1 -- 172.21.15.39:0/1909237577 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16104 (secure 0 0 0) 0x7fcad0079da0 con 0x7fcad815f390 2024-01-24T17:48:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:09.771 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:32.941651+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab006a430 msgr2=0x7fcab006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab006a430 0x7fcab006c8f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fcac80098f0 tx=0x7fcac8008040 comp rx=0 tx=0).stop 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 msgr2=0x7fcad80af030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80af030 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcad002c570 tx=0x7fcad0004a40 comp rx=0 tx=0).stop 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 shutdown_connections 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcab006a430 0x7fcab006c8f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcad815f390 0x7fcad80af030 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcad80b0010 0x7fcad80ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 --2- 172.21.15.39:0/1909237577 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fcad8070920 0x7fcad8156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:09.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.772+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 >> 172.21.15.39:0/1909237577 conn(0x7fcad806bf20 msgr2=0x7fcad815d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:09.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.773+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 shutdown_connections 2024-01-24T17:48:09.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:09.773+0000 7fcae0800640 1 -- 172.21.15.39:0/1909237577 wait complete. 2024-01-24T17:48:09.794 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:47:30.315286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:47:30.960101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:48:09.794 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:48:10.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.019+0000 7f1da2400640 1 Processor -- start 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.020+0000 7f1da2400640 1 -- start start 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.021+0000 7f1da2400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1d9c0cb990 0x7f1d9c160790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.021+0000 7f1da2400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c160cd0 0x7f1d9c06ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.021+0000 7f1da2400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c15cfa0 con 0x7f1d9c071ab0 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.021+0000 7f1da2400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c071e90 con 0x7f1d9c0cb990 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.021+0000 7f1da2400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c1614c0 con 0x7f1d9c160cd0 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.021+0000 7f1d9b400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1d9c071ab0 0x7f1d9c0cb280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35656/0 (socket says 172.21.15.39:35656) 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.021+0000 7f1d9b400640 1 -- 172.21.15.39:0/3212274694 learned_addr learned my addr 172.21.15.39:0/3212274694 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:10.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9aa00640 1 --2- 172.21.15.39:0/3212274694 >> v2:172.21.15.100:3300/0 conn(0x7f1d9c0cb990 0x7f1d9c160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9be00640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c160cd0 0x7f1d9c06ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497880772 0 0) 0x7f1d9c15cfa0 con 0x7f1d9c071ab0 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d7c003520 con 0x7f1d9c071ab0 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1525067066 0 0) 0x7f1d7c003520 con 0x7f1d9c071ab0 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c160cd0 msgr2=0x7f1d9c06ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c160cd0 0x7f1d9c06ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 >> v2:172.21.15.100:3300/0 conn(0x7f1d9c0cb990 msgr2=0x7f1d9c160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 --2- 172.21.15.39:0/3212274694 >> v2:172.21.15.100:3300/0 conn(0x7f1d9c0cb990 0x7f1d9c160790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c0cd6b0 con 0x7f1d9c071ab0 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1d8c001670 con 0x7f1d9c071ab0 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 >> v1:172.21.15.39:6789/0 conn(0x7f1d9c071ab0 legacy=0x7f1d9c0cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d7c003970 0x7f1d7c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.022+0000 7f1d9a000640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9a000640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c160cd0 0x7f1d7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c071e90 con 0x7f1d7c0047c0 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c1614c0 con 0x7f1d7c003970 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c00b980 con 0x7f1d9c160cd0 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9aa00640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9b400640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d7c003970 0x7f1d7c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9aa00640 1 -- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c160cd0 msgr2=0x7f1d7c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9aa00640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c160cd0 0x7f1d7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9aa00640 1 -- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d7c003970 msgr2=0x7f1d7c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9aa00640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d7c003970 0x7f1d7c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.023+0000 7f1d9aa00640 1 -- 172.21.15.39:0/3212274694 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c003520 con 0x7f1d7c0047c0 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.024+0000 7f1d9aa00640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d7c004c40 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009ac0 tx=0x7f1d7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88e815445637105 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8402c6d0 con 0x7f1d7c0047c0 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8402c830 con 0x7f1d7c0047c0 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1d9a000640 1 -- 172.21.15.39:0/3212274694 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8402cbc0 con 0x7f1d7c0047c0 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1da2400640 1 -- 172.21.15.39:0/3212274694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 msgr2=0x7f1d7c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1da2400640 1 --2- 172.21.15.39:0/3212274694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d7c004c40 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009ac0 tx=0x7f1d7c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1d9be00640 1 -- 172.21.15.39:0/3212274694 reap_dead start 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1da2400640 1 -- 172.21.15.39:0/3212274694 shutdown_connections 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1da2400640 1 -- 172.21.15.39:0/3212274694 >> 172.21.15.39:0/3212274694 conn(0x7f1d9c0ce9e0 msgr2=0x7f1d9c0cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1da2400640 1 -- 172.21.15.39:0/3212274694 shutdown_connections 2024-01-24T17:48:10.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.025+0000 7f1da2400640 1 -- 172.21.15.39:0/3212274694 wait complete. 2024-01-24T17:48:10.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.027+0000 7f1da2400640 1 Processor -- start 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.027+0000 7f1da2400640 1 -- start start 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.028+0000 7f1da2400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d9c0c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.028+0000 7f1da2400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c0c2b00 0x7f1d9c097a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.028+0000 7f1da2400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d9c0af780 0x7f1d9c0afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.028+0000 7f1da2400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c0c3c80 con 0x7f1d7c0047c0 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.028+0000 7f1da2400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c097f40 con 0x7f1d9c0af780 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.028+0000 7f1da2400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c0b0140 con 0x7f1d9c0c2b00 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d9c0c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d9c0c4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44280/0 (socket says 172.21.15.39:44280) 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 -- 172.21.15.39:0/3971952503 learned_addr learned my addr 172.21.15.39:0/3971952503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9aa00640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c0c2b00 0x7f1d9c097a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 -- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c0c2b00 msgr2=0x7f1d9c097a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c0c2b00 0x7f1d9c097a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9be00640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d9c0af780 0x7f1d9c0afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 -- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d9c0af780 msgr2=0x7f1d9c0afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d9c0af780 0x7f1d9c0afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 -- 172.21.15.39:0/3971952503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c0af360 con 0x7f1d7c0047c0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.029+0000 7f1d9b400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d9c0c4400 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00d810 tx=0x7f1d8c00dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.030+0000 7f1d93400640 1 -- 172.21.15.39:0/3971952503 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0074e0 con 0x7f1d7c0047c0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.030+0000 7f1d93400640 1 -- 172.21.15.39:0/3971952503 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c007640 con 0x7f1d7c0047c0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.030+0000 7f1d93400640 1 -- 172.21.15.39:0/3971952503 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c004a20 con 0x7f1d7c0047c0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.030+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c15d3e0 con 0x7f1d7c0047c0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.030+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c0c3350 con 0x7f1d7c0047c0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.031+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c160860 con 0x7f1d7c0047c0 2024-01-24T17:48:10.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.032+0000 7f1d93400640 1 -- 172.21.15.39:0/3971952503 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1d8c004b80 con 0x7f1d7c0047c0 2024-01-24T17:48:10.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.033+0000 7f1d93400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d7406a460 0x7f1d7406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.033+0000 7f1d93400640 1 -- 172.21.15.39:0/3971952503 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d8c088220 con 0x7f1d7c0047c0 2024-01-24T17:48:10.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.034+0000 7f1d9aa00640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d7406a460 0x7f1d7406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.035+0000 7f1d9aa00640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d7406a460 0x7f1d7406c920 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1d84036000 tx=0x7f1d84030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.037+0000 7f1d93400640 1 -- 172.21.15.39:0/3971952503 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1d8c0538d0 con 0x7f1d7c0047c0 2024-01-24T17:48:10.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.230+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1d9c04dff0 con 0x7f1d7c0047c0 2024-01-24T17:48:10.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.231+0000 7f1d93400640 1 -- 172.21.15.39:0/3971952503 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1d8c056ef0 con 0x7f1d7c0047c0 2024-01-24T17:48:10.232 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:48:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d7406a460 msgr2=0x7f1d7406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d7406a460 0x7f1d7406c920 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1d84036000 tx=0x7f1d84030000 comp rx=0 tx=0).stop 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 msgr2=0x7f1d9c0c4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d9c0c4400 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00d810 tx=0x7f1d8c00dcd0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 shutdown_connections 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1d9c0af780 0x7f1d9c0afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1d9c0c2b00 0x7f1d9c097a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1d7406a460 0x7f1d7406c920 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 --2- 172.21.15.39:0/3971952503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d7c0047c0 0x7f1d9c0c4400 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 >> 172.21.15.39:0/3971952503 conn(0x7f1d9c0ce9e0 msgr2=0x7f1d9c15c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 shutdown_connections 2024-01-24T17:48:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.234+0000 7f1da2400640 1 -- 172.21.15.39:0/3971952503 wait complete. 2024-01-24T17:48:10.253 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:48:10.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.468+0000 7f3dbe600640 1 Processor -- start 2024-01-24T17:48:10.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.469+0000 7f3dbe600640 1 -- start start 2024-01-24T17:48:10.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.470+0000 7f3dbe600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3db80cb990 0x7f3db80ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.470+0000 7f3dbe600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3db806cec0 0x7f3db806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.470+0000 7f3dbe600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db80ca4c0 con 0x7f3db815aa60 2024-01-24T17:48:10.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.470+0000 7f3dbe600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db80cae90 con 0x7f3db80cb990 2024-01-24T17:48:10.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.470+0000 7f3dbe600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db806dc90 con 0x7f3db806cec0 2024-01-24T17:48:10.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.471+0000 7f3db7400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3db815aa60 0x7f3db80cb280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35670/0 (socket says 172.21.15.39:35670) 2024-01-24T17:48:10.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.471+0000 7f3db7400640 1 -- 172.21.15.39:0/3132240956 learned_addr learned my addr 172.21.15.39:0/3132240956 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:10.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.471+0000 7f3db6a00640 1 --2- 172.21.15.39:0/3132240956 >> v2:172.21.15.100:3300/0 conn(0x7f3db80cb990 0x7f3db80ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.471+0000 7f3db7e00640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3db806cec0 0x7f3db806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.471+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139807274 0 0) 0x7f3db80ca4c0 con 0x7f3db815aa60 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.471+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d98003520 con 0x7f3db815aa60 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223654591 0 0) 0x7f3d98003520 con 0x7f3db815aa60 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3db806cec0 msgr2=0x7f3db806d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3db806cec0 0x7f3db806d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 >> v2:172.21.15.100:3300/0 conn(0x7f3db80cb990 msgr2=0x7f3db80ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 --2- 172.21.15.39:0/3132240956 >> v2:172.21.15.100:3300/0 conn(0x7f3db80cb990 0x7f3db80ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db80c9e70 con 0x7f3db815aa60 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f3da80019f0 con 0x7f3db815aa60 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 >> v1:172.21.15.39:6789/0 conn(0x7f3db815aa60 legacy=0x7f3db80cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d98003970 0x7f3d98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d980047c0 0x7f3d98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3db806cec0 0x7f3d98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db80cae90 con 0x7f3d980047c0 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db806dc90 con 0x7f3d98003970 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9800b980 con 0x7f3db806cec0 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6a00640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d980047c0 0x7f3d98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db7400640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d98003970 0x7f3d98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db7e00640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3db806cec0 0x7f3d98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6a00640 1 -- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3db806cec0 msgr2=0x7f3d98005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.472+0000 7f3db6a00640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3db806cec0 0x7f3d98005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.473+0000 7f3db6a00640 1 -- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d98003970 msgr2=0x7f3d98003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.473+0000 7f3db6a00640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d98003970 0x7f3d98003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.473+0000 7f3db6a00640 1 -- 172.21.15.39:0/3132240956 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d98003520 con 0x7f3d980047c0 2024-01-24T17:48:10.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.473+0000 7f3db7400640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d98003970 0x7f3d98003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:10.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.474+0000 7f3db6a00640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d980047c0 0x7f3d98004c40 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009ac0 tx=0x7f3d980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214107c5e3c6f4ff server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.474+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da002c710 con 0x7f3d980047c0 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.474+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da002c870 con 0x7f3d980047c0 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.474+0000 7f3db6000640 1 -- 172.21.15.39:0/3132240956 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da002cc00 con 0x7f3d980047c0 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.474+0000 7f3dbe600640 1 -- 172.21.15.39:0/3132240956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d980047c0 msgr2=0x7f3d98004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.474+0000 7f3dbe600640 1 --2- 172.21.15.39:0/3132240956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d980047c0 0x7f3d98004c40 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009ac0 tx=0x7f3d980057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.474+0000 7f3db7e00640 1 -- 172.21.15.39:0/3132240956 reap_dead start 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.475+0000 7f3dbe600640 1 -- 172.21.15.39:0/3132240956 shutdown_connections 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.475+0000 7f3dbe600640 1 -- 172.21.15.39:0/3132240956 >> 172.21.15.39:0/3132240956 conn(0x7f3db80ce9e0 msgr2=0x7f3db80cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.475+0000 7f3dbe600640 1 -- 172.21.15.39:0/3132240956 shutdown_connections 2024-01-24T17:48:10.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.475+0000 7f3dbe600640 1 -- 172.21.15.39:0/3132240956 wait complete. 2024-01-24T17:48:10.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.476+0000 7f3dbe600640 1 Processor -- start 2024-01-24T17:48:10.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.477+0000 7f3dbe600640 1 -- start start 2024-01-24T17:48:10.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3dbe600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980047c0 0x7f3db80cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3dbe600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db8097b50 0x7f3db81f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3dbe600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3db815e3c0 0x7f3db815e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3dbe600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db815ed80 con 0x7f3db8097b50 2024-01-24T17:48:10.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3dbe600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db81f55e0 con 0x7f3db815e3c0 2024-01-24T17:48:10.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3dbe600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db815b190 con 0x7f3d980047c0 2024-01-24T17:48:10.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db8097b50 0x7f3db81f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db8097b50 0x7f3db81f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44304/0 (socket says 172.21.15.39:44304) 2024-01-24T17:48:10.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db6a00640 1 -- 172.21.15.39:0/2517676298 learned_addr learned my addr 172.21.15.39:0/2517676298 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db6a00640 1 -- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980047c0 msgr2=0x7f3db80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db6a00640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980047c0 0x7f3db80cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db6a00640 1 -- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3db815e3c0 msgr2=0x7f3db815e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db7e00640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3db815e3c0 0x7f3db815e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db6a00640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3db815e3c0 0x7f3db815e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.478+0000 7f3db6a00640 1 -- 172.21.15.39:0/2517676298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db81f57f0 con 0x7f3db8097b50 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.479+0000 7f3db6a00640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db8097b50 0x7f3db81f50a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3da002c2b0 tx=0x7f3da0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.479+0000 7f3daf400640 1 -- 172.21.15.39:0/2517676298 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0003810 con 0x7f3db8097b50 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.479+0000 7f3daf400640 1 -- 172.21.15.39:0/2517676298 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da002de90 con 0x7f3db8097b50 2024-01-24T17:48:10.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.479+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db81f5ad0 con 0x7f3db8097b50 2024-01-24T17:48:10.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.479+0000 7f3daf400640 1 -- 172.21.15.39:0/2517676298 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0003b70 con 0x7f3db8097b50 2024-01-24T17:48:10.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.479+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db81f6090 con 0x7f3db8097b50 2024-01-24T17:48:10.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.480+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db81f6220 con 0x7f3db8097b50 2024-01-24T17:48:10.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.481+0000 7f3daf400640 1 -- 172.21.15.39:0/2517676298 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3da0003cd0 con 0x7f3db8097b50 2024-01-24T17:48:10.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.482+0000 7f3daf400640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3d9006a460 0x7f3d9006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.482+0000 7f3daf400640 1 -- 172.21.15.39:0/2517676298 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3da00aab70 con 0x7f3db8097b50 2024-01-24T17:48:10.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.483+0000 7f3db7400640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3d9006a460 0x7f3d9006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.484+0000 7f3db7400640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3d9006a460 0x7f3d9006c920 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3da80012e0 tx=0x7f3da8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.486+0000 7f3daf400640 1 -- 172.21.15.39:0/2517676298 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3da0076150 con 0x7f3db8097b50 2024-01-24T17:48:10.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.683+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3db80ca120 con 0x7f3db8097b50 2024-01-24T17:48:10.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.684+0000 7f3daf400640 1 -- 172.21.15.39:0/2517676298 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3da0079770 con 0x7f3db8097b50 2024-01-24T17:48:10.684 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:48:10.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3d9006a460 msgr2=0x7f3d9006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3d9006a460 0x7f3d9006c920 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3da80012e0 tx=0x7f3da8009040 comp rx=0 tx=0).stop 2024-01-24T17:48:10.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db8097b50 msgr2=0x7f3db81f50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db8097b50 0x7f3db81f50a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3da002c2b0 tx=0x7f3da0004730 comp rx=0 tx=0).stop 2024-01-24T17:48:10.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 shutdown_connections 2024-01-24T17:48:10.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3db815e3c0 0x7f3db815e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3d9006a460 0x7f3d9006c920 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3db8097b50 0x7f3db81f50a0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 --2- 172.21.15.39:0/2517676298 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3d980047c0 0x7f3db80cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.686+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 >> 172.21.15.39:0/2517676298 conn(0x7f3db80ce9e0 msgr2=0x7f3db80af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:10.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.687+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 shutdown_connections 2024-01-24T17:48:10.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.687+0000 7f3dbe600640 1 -- 172.21.15.39:0/2517676298 wait complete. 2024-01-24T17:48:10.708 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:48:10.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.931+0000 7f29b7000640 1 Processor -- start 2024-01-24T17:48:10.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.931+0000 7f29b7000640 1 -- start start 2024-01-24T17:48:10.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.932+0000 7f29b7000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b015ca30 0x7f29b00d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.932+0000 7f29b7000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f29b00d0e90 0x7f29b01606b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.933+0000 7f29b7000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b00d37a0 con 0x7f29b0160bf0 2024-01-24T17:48:10.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.933+0000 7f29b7000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b01610a0 con 0x7f29b00d0e90 2024-01-24T17:48:10.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.933+0000 7f29b7000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b00d13c0 con 0x7f29b015ca30 2024-01-24T17:48:10.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.933+0000 7f29b5000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f29b0160bf0 0x7f29b00cd570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35672/0 (socket says 172.21.15.39:35672) 2024-01-24T17:48:10.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.933+0000 7f29b5000640 1 -- 172.21.15.39:0/3377136575 learned_addr learned my addr 172.21.15.39:0/3377136575 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:10.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.933+0000 7f29afe00640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b015ca30 0x7f29b00d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.933+0000 7f29a7e00640 1 --2- 172.21.15.39:0/3377136575 >> v2:172.21.15.100:3300/0 conn(0x7f29b00d0e90 0x7f29b01606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30148160 0 0) 0x7f29b00d37a0 con 0x7f29b0160bf0 2024-01-24T17:48:10.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2990003520 con 0x7f29b0160bf0 2024-01-24T17:48:10.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3327642015 0 0) 0x7f2990003520 con 0x7f29b0160bf0 2024-01-24T17:48:10.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b015ca30 msgr2=0x7f29b00d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b015ca30 0x7f29b00d3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 >> v2:172.21.15.100:3300/0 conn(0x7f29b00d0e90 msgr2=0x7f29b01606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 --2- 172.21.15.39:0/3377136575 >> v2:172.21.15.100:3300/0 conn(0x7f29b00d0e90 0x7f29b01606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b00d16a0 con 0x7f29b0160bf0 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f29a80019f0 con 0x7f29b0160bf0 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 >> v1:172.21.15.39:6789/0 conn(0x7f29b0160bf0 legacy=0x7f29b00cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f2990003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29900047c0 0x7f2990004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b015ca30 0x7f2990005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b01610a0 con 0x7f2990003970 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b00d13c0 con 0x7f29900047c0 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299000b980 con 0x7f29b015ca30 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.934+0000 7f29afe00640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f2990003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.935+0000 7f29a7e00640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29900047c0 0x7f2990004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.935+0000 7f29b5000640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b015ca30 0x7f2990005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.935+0000 7f29afe00640 1 -- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b015ca30 msgr2=0x7f2990005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.935+0000 7f29afe00640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b015ca30 0x7f2990005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.935+0000 7f29afe00640 1 -- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29900047c0 msgr2=0x7f2990004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:10.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.935+0000 7f29afe00640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29900047c0 0x7f2990004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.935+0000 7f29afe00640 1 -- 172.21.15.39:0/3377136575 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2990003520 con 0x7f2990003970 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.936+0000 7f29afe00640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f2990003d90 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009ac0 tx=0x7f29900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f89b5bef606ba4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.936+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a002c6b0 con 0x7f2990003970 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.936+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a002c810 con 0x7f2990003970 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.936+0000 7f29af400640 1 -- 172.21.15.39:0/3377136575 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a002cba0 con 0x7f2990003970 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.937+0000 7f29b7000640 1 -- 172.21.15.39:0/3377136575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 msgr2=0x7f2990003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.937+0000 7f29b7000640 1 --2- 172.21.15.39:0/3377136575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f2990003d90 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009ac0 tx=0x7f29900057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.937+0000 7f29b5000640 1 -- 172.21.15.39:0/3377136575 reap_dead start 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.937+0000 7f29b7000640 1 -- 172.21.15.39:0/3377136575 shutdown_connections 2024-01-24T17:48:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.937+0000 7f29b7000640 1 -- 172.21.15.39:0/3377136575 >> 172.21.15.39:0/3377136575 conn(0x7f29b009bc80 msgr2=0x7f29b00d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:10.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.937+0000 7f29b7000640 1 -- 172.21.15.39:0/3377136575 shutdown_connections 2024-01-24T17:48:10.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.937+0000 7f29b7000640 1 -- 172.21.15.39:0/3377136575 wait complete. 2024-01-24T17:48:10.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.939+0000 7f29b7000640 1 Processor -- start 2024-01-24T17:48:10.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.939+0000 7f29b7000640 1 -- start start 2024-01-24T17:48:10.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29b7000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f29b015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29b7000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b00b5cf0 0x7f29b0159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29b7000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29b00cb510 0x7f29b00cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29b7000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b015d790 con 0x7f2990003970 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29b7000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b00cbed0 con 0x7f29b00cb510 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29b7000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b00b6240 con 0x7f29b00b5cf0 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f29b015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f29b015ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44314/0 (socket says 172.21.15.39:44314) 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.940+0000 7f29afe00640 1 -- 172.21.15.39:0/3254352917 learned_addr learned my addr 172.21.15.39:0/3254352917 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.941+0000 7f29a7e00640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b00b5cf0 0x7f29b0159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.941+0000 7f29b5000640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29b00cb510 0x7f29b00cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.941+0000 7f29afe00640 1 -- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b00b5cf0 msgr2=0x7f29b0159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.941+0000 7f29afe00640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b00b5cf0 0x7f29b0159ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.941+0000 7f29afe00640 1 -- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29b00cb510 msgr2=0x7f29b00cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.941+0000 7f29afe00640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29b00cb510 0x7f29b00cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.941+0000 7f29afe00640 1 -- 172.21.15.39:0/3254352917 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b015d9a0 con 0x7f2990003970 2024-01-24T17:48:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.941+0000 7f29afe00640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f29b015ce10 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009ac0 tx=0x7f29a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.942+0000 7f29acc00640 1 -- 172.21.15.39:0/3254352917 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0003810 con 0x7f2990003970 2024-01-24T17:48:10.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.942+0000 7f29acc00640 1 -- 172.21.15.39:0/3254352917 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0003970 con 0x7f2990003970 2024-01-24T17:48:10.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.942+0000 7f29acc00640 1 -- 172.21.15.39:0/3254352917 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0003d00 con 0x7f2990003970 2024-01-24T17:48:10.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.942+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b015dc80 con 0x7f2990003970 2024-01-24T17:48:10.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.942+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b015d350 con 0x7f2990003970 2024-01-24T17:48:10.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.943+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b00af7d0 con 0x7f2990003970 2024-01-24T17:48:10.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.944+0000 7f29acc00640 1 -- 172.21.15.39:0/3254352917 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f29a0040450 con 0x7f2990003970 2024-01-24T17:48:10.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.945+0000 7f29acc00640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f298806a460 0x7f298806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:10.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.945+0000 7f29acc00640 1 -- 172.21.15.39:0/3254352917 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29a00aa8a0 con 0x7f2990003970 2024-01-24T17:48:10.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.945+0000 7f29a7e00640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f298806a460 0x7f298806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.947+0000 7f29a7e00640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f298806a460 0x7f298806c920 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f29b015b650 tx=0x7f2998008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:10.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:10.949+0000 7f29acc00640 1 -- 172.21.15.39:0/3254352917 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f29b00af7d0 con 0x7f2990003970 2024-01-24T17:48:11.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.142+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f29b00af7d0 con 0x7f2990003970 2024-01-24T17:48:11.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.142+0000 7f29acc00640 1 -- 172.21.15.39:0/3254352917 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f29b00af7d0 con 0x7f2990003970 2024-01-24T17:48:11.143 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:48:11.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f298806a460 msgr2=0x7f298806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f298806a460 0x7f298806c920 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f29b015b650 tx=0x7f2998008040 comp rx=0 tx=0).stop 2024-01-24T17:48:11.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 msgr2=0x7f29b015ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f29b015ce10 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009ac0 tx=0x7f29a0004730 comp rx=0 tx=0).stop 2024-01-24T17:48:11.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 shutdown_connections 2024-01-24T17:48:11.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f29b00cb510 0x7f29b00cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f29b00b5cf0 0x7f29b0159ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f298806a460 0x7f298806c920 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 --2- 172.21.15.39:0/3254352917 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2990003970 0x7f29b015ce10 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 >> 172.21.15.39:0/3254352917 conn(0x7f29b009bc80 msgr2=0x7f29b015aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:11.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.145+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 shutdown_connections 2024-01-24T17:48:11.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.146+0000 7f29b7000640 1 -- 172.21.15.39:0/3254352917 wait complete. 2024-01-24T17:48:11.164 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:11.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.382+0000 7fe216200640 1 Processor -- start 2024-01-24T17:48:11.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.383+0000 7fe216200640 1 -- start start 2024-01-24T17:48:11.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.384+0000 7fe216200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe2100d12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.384+0000 7fe216200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe210164b40 0x7fe210166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.384+0000 7fe216200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100d17f0 con 0x7fe21015cbe0 2024-01-24T17:48:11.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.384+0000 7fe216200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2100b8820 con 0x7fe210164b40 2024-01-24T17:48:11.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.385+0000 7fe216200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100b5790 con 0x7fe2100d0e90 2024-01-24T17:48:11.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.385+0000 7fe20fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe210164b40 0x7fe210166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.385+0000 7fe207e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe2100d12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.385+0000 7fe20fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe210164b40 0x7fe210166f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:51832/0 (socket says 172.21.15.39:51832) 2024-01-24T17:48:11.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.385+0000 7fe20fe00640 1 -- 172.21.15.39:0/3336923699 learned_addr learned my addr 172.21.15.39:0/3336923699 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:11.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.385+0000 7fe20f400640 1 --1- 172.21.15.39:0/3336923699 >> v1:172.21.15.100:6789/0 conn(0x7fe21015cbe0 0x7fe2100cdfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:60610/0 (socket says 172.21.15.39:60610) 2024-01-24T17:48:11.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.386+0000 7fe207e00640 1 -- 172.21.15.39:0/3336923699 >> v2:172.21.15.100:3300/0 conn(0x7fe210164b40 msgr2=0x7fe210166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:11.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.386+0000 7fe207e00640 1 --2- 172.21.15.39:0/3336923699 >> v2:172.21.15.100:3300/0 conn(0x7fe210164b40 0x7fe210166f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.386+0000 7fe207e00640 1 -- 172.21.15.39:0/3336923699 >> v1:172.21.15.100:6789/0 conn(0x7fe21015cbe0 legacy=0x7fe2100cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.386+0000 7fe207e00640 1 -- 172.21.15.39:0/3336923699 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100b5a50 con 0x7fe2100d0e90 2024-01-24T17:48:11.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.386+0000 7fe20fe00640 1 --2- 172.21.15.39:0/3336923699 >> v2:172.21.15.100:3300/0 conn(0x7fe210164b40 0x7fe210166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:48:11.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.386+0000 7fe20ea00640 1 -- 172.21.15.39:0/3336923699 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873365877 0 0) 0x7fe2100d17f0 con 0x7fe21015cbe0 2024-01-24T17:48:11.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.388+0000 7fe207e00640 1 --2- 172.21.15.39:0/3336923699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe2100d12b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80097a0 tx=0x7fe1f802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e0c12d8dd23e4cb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:11.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.388+0000 7fe20ea00640 1 -- 172.21.15.39:0/3336923699 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f802d040 con 0x7fe2100d0e90 2024-01-24T17:48:11.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.388+0000 7fe20ea00640 1 -- 172.21.15.39:0/3336923699 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f8031810 con 0x7fe2100d0e90 2024-01-24T17:48:11.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.388+0000 7fe20ea00640 1 -- 172.21.15.39:0/3336923699 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8031ba0 con 0x7fe2100d0e90 2024-01-24T17:48:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.389+0000 7fe216200640 1 -- 172.21.15.39:0/3336923699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 msgr2=0x7fe2100d12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.389+0000 7fe216200640 1 --2- 172.21.15.39:0/3336923699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe2100d12b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80097a0 tx=0x7fe1f802b8b0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.389+0000 7fe216200640 1 -- 172.21.15.39:0/3336923699 shutdown_connections 2024-01-24T17:48:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.389+0000 7fe216200640 1 --2- 172.21.15.39:0/3336923699 >> v2:172.21.15.100:3300/0 conn(0x7fe210164b40 0x7fe210166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.389+0000 7fe216200640 1 --2- 172.21.15.39:0/3336923699 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe2100d12b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.389+0000 7fe216200640 1 -- 172.21.15.39:0/3336923699 >> 172.21.15.39:0/3336923699 conn(0x7fe21009bc80 msgr2=0x7fe2100d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.389+0000 7fe216200640 1 -- 172.21.15.39:0/3336923699 shutdown_connections 2024-01-24T17:48:11.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.389+0000 7fe216200640 1 -- 172.21.15.39:0/3336923699 wait complete. 2024-01-24T17:48:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.392+0000 7fe216200640 1 Processor -- start 2024-01-24T17:48:11.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.392+0000 7fe216200640 1 -- start start 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.393+0000 7fe216200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe210160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.393+0000 7fe216200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe21015cbe0 0x7fe21015e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.393+0000 7fe216200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe210164b40 0x7fe210157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.393+0000 7fe216200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21015d250 con 0x7fe21015cbe0 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.393+0000 7fe216200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21015ef80 con 0x7fe210164b40 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.393+0000 7fe216200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21015f260 con 0x7fe2100d0e90 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe207e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe21015cbe0 0x7fe21015e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe207e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe21015cbe0 0x7fe21015e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44330/0 (socket says 172.21.15.39:44330) 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe207e00640 1 -- 172.21.15.39:0/2982977718 learned_addr learned my addr 172.21.15.39:0/2982977718 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe20f400640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe210160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe20fe00640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe210164b40 0x7fe210157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe207e00640 1 -- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 msgr2=0x7fe210160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe207e00640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe210160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe207e00640 1 -- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe210164b40 msgr2=0x7fe210157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe207e00640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe210164b40 0x7fe210157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe207e00640 1 -- 172.21.15.39:0/2982977718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe208009b30 con 0x7fe21015cbe0 2024-01-24T17:48:11.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.394+0000 7fe20f400640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe210160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:48:11.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.395+0000 7fe207e00640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe21015cbe0 0x7fe21015e730 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80098c0 tx=0x7fe1f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:11.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.395+0000 7fe207400640 1 -- 172.21.15.39:0/2982977718 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8006e10 con 0x7fe21015cbe0 2024-01-24T17:48:11.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.395+0000 7fe207400640 1 -- 172.21.15.39:0/2982977718 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f803ad60 con 0x7fe21015cbe0 2024-01-24T17:48:11.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.395+0000 7fe207400640 1 -- 172.21.15.39:0/2982977718 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8044620 con 0x7fe21015cbe0 2024-01-24T17:48:11.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.395+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f8009400 con 0x7fe21015cbe0 2024-01-24T17:48:11.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.395+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21015f820 con 0x7fe21015cbe0 2024-01-24T17:48:11.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.396+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2100b4630 con 0x7fe21015cbe0 2024-01-24T17:48:11.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.397+0000 7fe207400640 1 -- 172.21.15.39:0/2982977718 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe1f804d440 con 0x7fe21015cbe0 2024-01-24T17:48:11.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.398+0000 7fe207400640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1e806a430 0x7fe1e806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.398+0000 7fe207400640 1 -- 172.21.15.39:0/2982977718 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1f80aad80 con 0x7fe21015cbe0 2024-01-24T17:48:11.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.399+0000 7fe20f400640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1e806a430 0x7fe1e806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.400+0000 7fe20f400640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1e806a430 0x7fe1e806c8f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe200000fb0 tx=0x7fe200007be0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:11.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.404+0000 7fe207400640 1 -- 172.21.15.39:0/2982977718 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe1f8076370 con 0x7fe21015cbe0 2024-01-24T17:48:11.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.642+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe21006b580 con 0x7fe21015cbe0 2024-01-24T17:48:11.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.643+0000 7fe207400640 1 -- 172.21.15.39:0/2982977718 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4736 (secure 0 0 0) 0x7fe1f8079990 con 0x7fe21015cbe0 2024-01-24T17:48:11.644 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:11.644 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/57362003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":57362003},{"type":"v1","addr":"172.21.15.100:6836","nonce":57362003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6139,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/3662299631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":3662299631},{"type":"v1","addr":"172.21.15.100:6837","nonce":3662299631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2829786836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2829786836},{"type":"v1","addr":"172.21.15.39:6833","nonce":2829786836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:47:31.936549+0000","modified":"2024-01-24T17:47:35.302809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7293":{"gid":7293,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3073867595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3073867595},{"type":"v1","addr":"172.21.15.39:6835","nonce":3073867595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-24T17:48:11.644 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 51 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1e806a430 msgr2=0x7fe1e806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1e806a430 0x7fe1e806c8f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe200000fb0 tx=0x7fe200007be0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe21015cbe0 msgr2=0x7fe21015e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe21015cbe0 0x7fe21015e730 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fe1f80098c0 tx=0x7fe1f80046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 shutdown_connections 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe210164b40 0x7fe210157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe1e806a430 0x7fe1e806c8f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe21015cbe0 0x7fe21015e730 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 --2- 172.21.15.39:0/2982977718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe2100d0e90 0x7fe210160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 >> 172.21.15.39:0/2982977718 conn(0x7fe21009bc80 msgr2=0x7fe21006ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:11.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 shutdown_connections 2024-01-24T17:48:11.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.646+0000 7fe216200640 1 -- 172.21.15.39:0/2982977718 wait complete. 2024-01-24T17:48:11.666 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:11.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.890+0000 7f7e1d800640 1 Processor -- start 2024-01-24T17:48:11.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.890+0000 7f7e1d800640 1 -- start start 2024-01-24T17:48:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.892+0000 7f7e1d800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7e18071c00 0x7f7e180cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.892+0000 7f7e1d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e180b3f20 0x7f7e180ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.892+0000 7f7e1d800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e180ccd10 con 0x7f7e18159d00 2024-01-24T17:48:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.892+0000 7f7e1d800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e180ce230 con 0x7f7e18071c00 2024-01-24T17:48:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.892+0000 7f7e1d800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180ccff0 con 0x7f7e180b3f20 2024-01-24T17:48:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.892+0000 7f7e16a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7e18159d00 0x7f7e180714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35686/0 (socket says 172.21.15.39:35686) 2024-01-24T17:48:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.892+0000 7f7e16a00640 1 -- 172.21.15.39:0/3838102556 learned_addr learned my addr 172.21.15.39:0/3838102556 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:11.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/3838102556 >> v2:172.21.15.100:3300/0 conn(0x7f7e18071c00 0x7f7e180cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e17400640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e180b3f20 0x7f7e180ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436785602 0 0) 0x7f7e180ccd10 con 0x7f7e18159d00 2024-01-24T17:48:11.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df8003520 con 0x7f7e18159d00 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 115095890 0 0) 0x7f7df8003520 con 0x7f7e18159d00 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e180b3f20 msgr2=0x7f7e180ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e180b3f20 0x7f7e180ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 >> v2:172.21.15.100:3300/0 conn(0x7f7e18071c00 msgr2=0x7f7e180cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 --2- 172.21.15.39:0/3838102556 >> v2:172.21.15.100:3300/0 conn(0x7f7e18071c00 0x7f7e180cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e180ad720 con 0x7f7e18159d00 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7e08001670 con 0x7f7e18159d00 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 >> v1:172.21.15.39:6789/0 conn(0x7f7e18159d00 legacy=0x7f7e180714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7df8003970 0x7f7df8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7df8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e180b3f20 0x7f7df8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180ce230 con 0x7f7df80047c0 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180ccff0 con 0x7f7df8003970 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.893+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df800b980 con 0x7f7e180b3f20 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.894+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7df8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.894+0000 7f7e17400640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e180b3f20 0x7f7df8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.894+0000 7f7e16a00640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7df8003970 0x7f7df8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.894+0000 7f7e0fe00640 1 -- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e180b3f20 msgr2=0x7f7df8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.894+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e180b3f20 0x7f7df8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.894+0000 7f7e0fe00640 1 -- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7df8003970 msgr2=0x7f7df8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.894+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7df8003970 0x7f7df8003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.894+0000 7f7e0fe00640 1 -- 172.21.15.39:0/3838102556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df8003520 con 0x7f7df80047c0 2024-01-24T17:48:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.895+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7df8004c40 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009ac0 tx=0x7f7df80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fda984f112bdd1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0002c710 con 0x7f7df80047c0 2024-01-24T17:48:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0002c870 con 0x7f7df80047c0 2024-01-24T17:48:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e16000640 1 -- 172.21.15.39:0/3838102556 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0002cc00 con 0x7f7df80047c0 2024-01-24T17:48:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e1d800640 1 -- 172.21.15.39:0/3838102556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 msgr2=0x7f7df8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e1d800640 1 --2- 172.21.15.39:0/3838102556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7df8004c40 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009ac0 tx=0x7f7df80057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e17400640 1 -- 172.21.15.39:0/3838102556 reap_dead start 2024-01-24T17:48:11.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e1d800640 1 -- 172.21.15.39:0/3838102556 shutdown_connections 2024-01-24T17:48:11.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e1d800640 1 -- 172.21.15.39:0/3838102556 >> 172.21.15.39:0/3838102556 conn(0x7f7e1806bf20 msgr2=0x7f7e1806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:11.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e1d800640 1 -- 172.21.15.39:0/3838102556 shutdown_connections 2024-01-24T17:48:11.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.896+0000 7f7e1d800640 1 -- 172.21.15.39:0/3838102556 wait complete. 2024-01-24T17:48:11.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.898+0000 7f7e1d800640 1 Processor -- start 2024-01-24T17:48:11.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.898+0000 7f7e1d800640 1 -- start start 2024-01-24T17:48:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.899+0000 7f7e1d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7e1814de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.899+0000 7f7e1d800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e1815cd50 0x7f7e1814d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.899+0000 7f7e1d800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e1814ada0 0x7f7e180c9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.899+0000 7f7e1d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1815ad00 con 0x7f7df80047c0 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.899+0000 7f7e1d800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1815d1b0 con 0x7f7e1814ada0 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.899+0000 7f7e1d800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1814da80 con 0x7f7e1815cd50 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7e1814de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7e1814de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44346/0 (socket says 172.21.15.39:44346) 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 -- 172.21.15.39:0/2908284299 learned_addr learned my addr 172.21.15.39:0/2908284299 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e17400640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e1814ada0 0x7f7e180c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e1815cd50 0x7f7e1814d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 -- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e1815cd50 msgr2=0x7f7e1814d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e1815cd50 0x7f7e1814d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 -- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e1814ada0 msgr2=0x7f7e180c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e1814ada0 0x7f7e180c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:11.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 -- 172.21.15.39:0/2908284299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1815af10 con 0x7f7df80047c0 2024-01-24T17:48:11.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e1815cd50 0x7f7e1814d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:11.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e17400640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e1814ada0 0x7f7e180c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:11.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.900+0000 7f7e16a00640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7e1814de20 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f7e08000ff0 tx=0x7f7e0800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:11.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.901+0000 7f7e0ea00640 1 -- 172.21.15.39:0/2908284299 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e080074e0 con 0x7f7df80047c0 2024-01-24T17:48:11.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.901+0000 7f7e0ea00640 1 -- 172.21.15.39:0/2908284299 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e08007640 con 0x7f7df80047c0 2024-01-24T17:48:11.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.901+0000 7f7e0ea00640 1 -- 172.21.15.39:0/2908284299 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08004a20 con 0x7f7df80047c0 2024-01-24T17:48:11.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.901+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1815b1f0 con 0x7f7df80047c0 2024-01-24T17:48:11.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.901+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1815b4d0 con 0x7f7df80047c0 2024-01-24T17:48:11.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.902+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1814b800 con 0x7f7df80047c0 2024-01-24T17:48:11.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.903+0000 7f7e0ea00640 1 -- 172.21.15.39:0/2908284299 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7e08004b80 con 0x7f7df80047c0 2024-01-24T17:48:11.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.904+0000 7f7e0ea00640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7df006a460 0x7f7df006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:11.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.904+0000 7f7e0ea00640 1 -- 172.21.15.39:0/2908284299 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e080881c0 con 0x7f7df80047c0 2024-01-24T17:48:11.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.905+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7df006a460 0x7f7df006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:11.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.906+0000 7f7e0fe00640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7df006a460 0x7f7df006c920 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7e00036000 tx=0x7f7e00030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:11.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:11.909+0000 7f7e0ea00640 1 -- 172.21.15.39:0/2908284299 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7e1814b800 con 0x7f7df80047c0 2024-01-24T17:48:12.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.111+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e1814b800 con 0x7f7df80047c0 2024-01-24T17:48:12.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.113+0000 7f7e0ea00640 1 -- 172.21.15.39:0/2908284299 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16104 (secure 0 0 0) 0x7f7e1814b800 con 0x7f7df80047c0 2024-01-24T17:48:12.114 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:12.114 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:32.941651+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7df006a460 msgr2=0x7f7df006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7df006a460 0x7f7df006c920 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7e00036000 tx=0x7f7e00030000 comp rx=0 tx=0).stop 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 msgr2=0x7f7e1814de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7e1814de20 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f7e08000ff0 tx=0x7f7e0800dc00 comp rx=0 tx=0).stop 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 shutdown_connections 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7e1814ada0 0x7f7e180c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7e1815cd50 0x7f7e1814d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7df006a460 0x7f7df006c920 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.115+0000 7f7e1d800640 1 --2- 172.21.15.39:0/2908284299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7df80047c0 0x7f7e1814de20 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.116+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 >> 172.21.15.39:0/2908284299 conn(0x7f7e1806bf20 msgr2=0x7f7e180aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:12.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.116+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 shutdown_connections 2024-01-24T17:48:12.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.116+0000 7f7e1d800640 1 -- 172.21.15.39:0/2908284299 wait complete. 2024-01-24T17:48:12.137 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-24T17:48:12.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:12.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.359+0000 7f3b44c00640 1 Processor -- start 2024-01-24T17:48:12.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.359+0000 7f3b44c00640 1 -- start start 2024-01-24T17:48:12.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.361+0000 7f3b44c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b40159d00 0x7f3b400cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.361+0000 7f3b44c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b400ccd10 con 0x7f3b40071c00 2024-01-24T17:48:12.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.361+0000 7f3b44c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b400ccff0 con 0x7f3b400b3f20 2024-01-24T17:48:12.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.361+0000 7f3b44c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400ce230 con 0x7f3b40159d00 2024-01-24T17:48:12.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.361+0000 7f3b3e800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3b40071c00 0x7f3b400ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35698/0 (socket says 172.21.15.39:35698) 2024-01-24T17:48:12.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.361+0000 7f3b3e800640 1 -- 172.21.15.39:0/3867475386 learned_addr learned my addr 172.21.15.39:0/3867475386 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3de00640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b40159d00 0x7f3b400cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704927813 0 0) 0x7f3b400ccd10 con 0x7f3b40071c00 2024-01-24T17:48:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b20005730 con 0x7f3b40071c00 2024-01-24T17:48:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2656628752 0 0) 0x7f3b20005730 con 0x7f3b40071c00 2024-01-24T17:48:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b40159d00 msgr2=0x7f3b400cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b40159d00 0x7f3b400cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 >> v1:172.21.15.100:6789/0 conn(0x7f3b400b3f20 legacy=0x7f3b400714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400ad720 con 0x7f3b40071c00 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f3b380019f0 con 0x7f3b40071c00 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.362+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 >> v1:172.21.15.39:6789/0 conn(0x7f3b40071c00 legacy=0x7f3b400ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b3d400640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b20005bb0 0x7f3b20005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b3d400640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b200069d0 0x7f3b20006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b3d400640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b40159d00 0x7f3b20007440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400ce230 con 0x7f3b200069d0 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2000dd20 con 0x7f3b20005bb0 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2000dee0 con 0x7f3b40159d00 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b35200640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b200069d0 0x7f3b20006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b3e800640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b40159d00 0x7f3b20007440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b3de00640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b20005bb0 0x7f3b20005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b35200640 1 -- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b40159d00 msgr2=0x7f3b20007440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b35200640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b40159d00 0x7f3b20007440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b35200640 1 -- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b20005bb0 msgr2=0x7f3b20005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b35200640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b20005bb0 0x7f3b20005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.363+0000 7f3b35200640 1 -- 172.21.15.39:0/3867475386 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20005730 con 0x7f3b200069d0 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.365+0000 7f3b35200640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b200069d0 0x7f3b20006e50 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f3b28001750 tx=0x7f3b20007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac232a4f40d36598 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.365+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b280052d0 con 0x7f3b200069d0 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.365+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2802e050 con 0x7f3b200069d0 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.365+0000 7f3b3d400640 1 -- 172.21.15.39:0/3867475386 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b280035f0 con 0x7f3b200069d0 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.366+0000 7f3b44c00640 1 -- 172.21.15.39:0/3867475386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b200069d0 msgr2=0x7f3b20006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.366+0000 7f3b44c00640 1 --2- 172.21.15.39:0/3867475386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b200069d0 0x7f3b20006e50 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f3b28001750 tx=0x7f3b20007a00 comp rx=0 tx=0).stop 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.366+0000 7f3b3e800640 1 -- 172.21.15.39:0/3867475386 reap_dead start 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.366+0000 7f3b44c00640 1 -- 172.21.15.39:0/3867475386 shutdown_connections 2024-01-24T17:48:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.366+0000 7f3b44c00640 1 -- 172.21.15.39:0/3867475386 >> 172.21.15.39:0/3867475386 conn(0x7f3b4006bf20 msgr2=0x7f3b4006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:12.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.366+0000 7f3b44c00640 1 -- 172.21.15.39:0/3867475386 shutdown_connections 2024-01-24T17:48:12.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.366+0000 7f3b44c00640 1 -- 172.21.15.39:0/3867475386 wait complete. 2024-01-24T17:48:12.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.368+0000 7f3b44c00640 1 Processor -- start 2024-01-24T17:48:12.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.368+0000 7f3b44c00640 1 -- start start 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b44c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b200069d0 0x7f3b4014d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b44c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b400c7ce0 0x7f3b400c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b44c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b400c5660 0x7f3b400c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b44c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400c86a0 con 0x7f3b400c7ce0 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b44c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400c6fa0 con 0x7f3b400c5660 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b44c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400c6390 con 0x7f3b200069d0 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b35200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b400c7ce0 0x7f3b400c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b35200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b400c7ce0 0x7f3b400c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44362/0 (socket says 172.21.15.39:44362) 2024-01-24T17:48:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b35200640 1 -- 172.21.15.39:0/3834999209 learned_addr learned my addr 172.21.15.39:0/3834999209 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b3e800640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b400c5660 0x7f3b400c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.369+0000 7f3b3de00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b200069d0 0x7f3b4014d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b35200640 1 -- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b200069d0 msgr2=0x7f3b4014d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b35200640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b200069d0 0x7f3b4014d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b35200640 1 -- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b400c5660 msgr2=0x7f3b400c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b35200640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b400c5660 0x7f3b400c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b35200640 1 -- 172.21.15.39:0/3834999209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400c6670 con 0x7f3b400c7ce0 2024-01-24T17:48:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b3e800640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b400c5660 0x7f3b400c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b35200640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b400c7ce0 0x7f3b400c8160 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f3b28008fd0 tx=0x7f3b28001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b36a00640 1 -- 172.21.15.39:0/3834999209 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b280076c0 con 0x7f3b400c7ce0 2024-01-24T17:48:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b36a00640 1 -- 172.21.15.39:0/3834999209 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b28007a30 con 0x7f3b400c7ce0 2024-01-24T17:48:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b36a00640 1 -- 172.21.15.39:0/3834999209 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28006be0 con 0x7f3b400c7ce0 2024-01-24T17:48:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b400c6950 con 0x7f3b400c7ce0 2024-01-24T17:48:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.370+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4014db10 con 0x7f3b400c7ce0 2024-01-24T17:48:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.371+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b400c7150 con 0x7f3b400c7ce0 2024-01-24T17:48:12.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.373+0000 7f3b36a00640 1 -- 172.21.15.39:0/3834999209 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3b28006d40 con 0x7f3b400c7ce0 2024-01-24T17:48:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.373+0000 7f3b36a00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3b1806a430 0x7f3b1806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.374+0000 7f3b36a00640 1 -- 172.21.15.39:0/3834999209 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b280aa6f0 con 0x7f3b400c7ce0 2024-01-24T17:48:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.374+0000 7f3b3de00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3b1806a430 0x7f3b1806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.375+0000 7f3b3de00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3b1806a430 0x7f3b1806c8f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009950 tx=0x7f3b30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:12.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.379+0000 7f3b36a00640 1 -- 172.21.15.39:0/3834999209 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3b400c7150 con 0x7f3b400c7ce0 2024-01-24T17:48:12.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.610+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b400c7150 con 0x7f3b400c7ce0 2024-01-24T17:48:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.611+0000 7f3b36a00640 1 -- 172.21.15.39:0/3834999209 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f3b400c7150 con 0x7f3b400c7ce0 2024-01-24T17:48:12.612 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:12.612 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/57362003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":57362003},{"type":"v1","addr":"172.21.15.100:6836","nonce":57362003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6139,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/3662299631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":3662299631},{"type":"v1","addr":"172.21.15.100:6837","nonce":3662299631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2829786836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2829786836},{"type":"v1","addr":"172.21.15.39:6833","nonce":2829786836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:47:31.936549+0000","modified":"2024-01-24T17:47:35.302809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7293":{"gid":7293,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3073867595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3073867595},{"type":"v1","addr":"172.21.15.39:6835","nonce":3073867595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-24T17:48:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 51 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.613+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3b1806a430 msgr2=0x7f3b1806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.613+0000 7f3b44c00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3b1806a430 0x7f3b1806c8f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009950 tx=0x7f3b30008040 comp rx=0 tx=0).stop 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.613+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b400c7ce0 msgr2=0x7f3b400c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.613+0000 7f3b44c00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b400c7ce0 0x7f3b400c8160 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f3b28008fd0 tx=0x7f3b28001120 comp rx=0 tx=0).stop 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.614+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 shutdown_connections 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.614+0000 7f3b44c00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3b400c5660 0x7f3b400c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.614+0000 7f3b44c00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3b1806a430 0x7f3b1806c8f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.614+0000 7f3b44c00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b400c7ce0 0x7f3b400c8160 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.614+0000 7f3b44c00640 1 --2- 172.21.15.39:0/3834999209 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3b200069d0 0x7f3b4014d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.614+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 >> 172.21.15.39:0/3834999209 conn(0x7f3b4006bf20 msgr2=0x7f3b4015a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.614+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 shutdown_connections 2024-01-24T17:48:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.614+0000 7f3b44c00640 1 -- 172.21.15.39:0/3834999209 wait complete. 2024-01-24T17:48:12.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:12.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.846+0000 7f6853000640 1 Processor -- start 2024-01-24T17:48:12.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.846+0000 7f6853000640 1 -- start start 2024-01-24T17:48:12.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.847+0000 7f6853000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f684c15cbe0 0x7f684c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.847+0000 7f6853000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684c0b8820 con 0x7f684c0d15a0 2024-01-24T17:48:12.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.847+0000 7f6853000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684c0b5790 con 0x7f684c0ce510 2024-01-24T17:48:12.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.847+0000 7f6853000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684c0b5a50 con 0x7f684c15cbe0 2024-01-24T17:48:12.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.848+0000 7f6851000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f684c0d15a0 0x7f684c166b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35708/0 (socket says 172.21.15.39:35708) 2024-01-24T17:48:12.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.848+0000 7f6851000640 1 -- 172.21.15.39:0/1964037272 learned_addr learned my addr 172.21.15.39:0/1964037272 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:12.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.848+0000 7f684be00640 1 --2- 172.21.15.39:0/1964037272 >> v2:172.21.15.100:3300/0 conn(0x7f684c15cbe0 0x7f684c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.848+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43224266 0 0) 0x7f684c0b8820 con 0x7f684c0d15a0 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.848+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f682c003520 con 0x7f684c0d15a0 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415564313 0 0) 0x7f682c003520 con 0x7f684c0d15a0 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 >> v2:172.21.15.100:3300/0 conn(0x7f684c15cbe0 msgr2=0x7f684c0cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 --2- 172.21.15.39:0/1964037272 >> v2:172.21.15.100:3300/0 conn(0x7f684c15cbe0 0x7f684c0cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 >> v1:172.21.15.100:6789/0 conn(0x7f684c0ce510 legacy=0x7f684c0d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c0b5d30 con 0x7f684c0d15a0 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f6844001670 con 0x7f684c0d15a0 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 >> v1:172.21.15.39:6789/0 conn(0x7f684c0d15a0 legacy=0x7f684c166b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f682c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f682c0047c0 0x7f682c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f682c005180 0x7f682c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0b5a50 con 0x7f682c003970 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c00c020 con 0x7f682c0047c0 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682c00c1e0 con 0x7f682c005180 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684be00640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f682c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684b400640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f682c0047c0 0x7f682c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684be00640 1 -- 172.21.15.39:0/1964037272 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f682c005180 msgr2=0x7f682c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.849+0000 7f684be00640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f682c005180 0x7f682c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.850+0000 7f684be00640 1 -- 172.21.15.39:0/1964037272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f682c0047c0 msgr2=0x7f682c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.850+0000 7f684be00640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f682c0047c0 0x7f682c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.850+0000 7f684be00640 1 -- 172.21.15.39:0/1964037272 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c003520 con 0x7f682c003970 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.850+0000 7f6851000640 1 -- 172.21.15.39:0/1964037272 reap_dead start 2024-01-24T17:48:12.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.850+0000 7f684b400640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f682c0047c0 0x7f682c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:12.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.851+0000 7f684be00640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f682c003d90 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f6834009ab0 tx=0x7f682c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744b2258104cb4f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:12.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.851+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834005990 con 0x7f682c003970 2024-01-24T17:48:12.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.851+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6834005af0 con 0x7f682c003970 2024-01-24T17:48:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.851+0000 7f684aa00640 1 -- 172.21.15.39:0/1964037272 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834005ea0 con 0x7f682c003970 2024-01-24T17:48:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.851+0000 7f6853000640 1 -- 172.21.15.39:0/1964037272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 msgr2=0x7f682c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.851+0000 7f6853000640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f682c003d90 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f6834009ab0 tx=0x7f682c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.852+0000 7f6853000640 1 -- 172.21.15.39:0/1964037272 shutdown_connections 2024-01-24T17:48:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.852+0000 7f6853000640 1 --2- 172.21.15.39:0/1964037272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f682c003d90 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.852+0000 7f6853000640 1 -- 172.21.15.39:0/1964037272 >> 172.21.15.39:0/1964037272 conn(0x7f684c09bc80 msgr2=0x7f684c0d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.852+0000 7f6853000640 1 -- 172.21.15.39:0/1964037272 shutdown_connections 2024-01-24T17:48:12.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.852+0000 7f6853000640 1 -- 172.21.15.39:0/1964037272 wait complete. 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.853+0000 7f6853000640 1 Processor -- start 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.854+0000 7f6853000640 1 -- start start 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.854+0000 7f6853000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f684c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.854+0000 7f6853000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f684c0a00d0 0x7f684c0cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.854+0000 7f6853000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f684c0cb670 0x7f684c09cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.854+0000 7f6853000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c15b710 con 0x7f682c003970 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.854+0000 7f6853000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0cbaf0 con 0x7f684c0a00d0 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.854+0000 7f6853000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c09f030 con 0x7f684c0cb670 2024-01-24T17:48:12.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f684c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f684c15b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:44370/0 (socket says 172.21.15.39:44370) 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684be00640 1 -- 172.21.15.39:0/2483695604 learned_addr learned my addr 172.21.15.39:0/2483695604 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684be00640 1 -- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f684c0cb670 msgr2=0x7f684c09cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684b400640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f684c0a00d0 0x7f684c0cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684be00640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f684c0cb670 0x7f684c09cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684be00640 1 -- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f684c0a00d0 msgr2=0x7f684c0cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684be00640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f684c0a00d0 0x7f684c0cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.855+0000 7f684be00640 1 -- 172.21.15.39:0/2483695604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c09f310 con 0x7f682c003970 2024-01-24T17:48:12.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.856+0000 7f684be00640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f684c15b1d0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f6834036040 tx=0x7f6834004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:12.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.856+0000 7f6843e00640 1 -- 172.21.15.39:0/2483695604 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834005e70 con 0x7f682c003970 2024-01-24T17:48:12.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.856+0000 7f6843e00640 1 -- 172.21.15.39:0/2483695604 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6834006030 con 0x7f682c003970 2024-01-24T17:48:12.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.856+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684c09f5f0 con 0x7f682c003970 2024-01-24T17:48:12.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.856+0000 7f6843e00640 1 -- 172.21.15.39:0/2483695604 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68340063a0 con 0x7f682c003970 2024-01-24T17:48:12.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.856+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c09fbb0 con 0x7f682c003970 2024-01-24T17:48:12.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.857+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684c04dff0 con 0x7f682c003970 2024-01-24T17:48:12.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.858+0000 7f6843e00640 1 -- 172.21.15.39:0/2483695604 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6834006500 con 0x7f682c003970 2024-01-24T17:48:12.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.859+0000 7f6843e00640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f682406a460 0x7f682406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:12.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.859+0000 7f6843e00640 1 -- 172.21.15.39:0/2483695604 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68340aafe0 con 0x7f682c003970 2024-01-24T17:48:12.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.860+0000 7f684b400640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f682406a460 0x7f682406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:12.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.861+0000 7f684b400640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f682406a460 0x7f682406c920 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f683c000f80 tx=0x7f683c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:12.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:12.864+0000 7f6843e00640 1 -- 172.21.15.39:0/2483695604 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6834076600 con 0x7f682c003970 2024-01-24T17:48:13.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.097+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f684c0b4dc0 con 0x7f682c003970 2024-01-24T17:48:13.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.098+0000 7f6843e00640 1 -- 172.21.15.39:0/2483695604 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4736 (secure 0 0 0) 0x7f6834079c20 con 0x7f682c003970 2024-01-24T17:48:13.099 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:13.099 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":51,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/57362003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":57362003},{"type":"v1","addr":"172.21.15.100:6836","nonce":57362003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6139,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/3662299631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":3662299631},{"type":"v1","addr":"172.21.15.100:6837","nonce":3662299631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2829786836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2829786836},{"type":"v1","addr":"172.21.15.39:6833","nonce":2829786836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:47:31.936549+0000","modified":"2024-01-24T17:47:35.302809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7293":{"gid":7293,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3073867595","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3073867595},{"type":"v1","addr":"172.21.15.39:6835","nonce":3073867595}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-24T17:48:13.100 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 51 2024-01-24T17:48:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f682406a460 msgr2=0x7f682406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f682406a460 0x7f682406c920 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f683c000f80 tx=0x7f683c009040 comp rx=0 tx=0).stop 2024-01-24T17:48:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 msgr2=0x7f684c15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f684c15b1d0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f6834036040 tx=0x7f6834004730 comp rx=0 tx=0).stop 2024-01-24T17:48:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 shutdown_connections 2024-01-24T17:48:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f684c0cb670 0x7f684c09cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f684c0a00d0 0x7f684c0cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f682406a460 0x7f682406c920 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 --2- 172.21.15.39:0/2483695604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f682c003970 0x7f684c15b1d0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 >> 172.21.15.39:0/2483695604 conn(0x7f684c09bc80 msgr2=0x7f684c0b78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:13.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 shutdown_connections 2024-01-24T17:48:13.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.101+0000 7f6853000640 1 -- 172.21.15.39:0/2483695604 wait complete. 2024-01-24T17:48:13.122 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:13.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.343+0000 7f4188c00640 1 Processor -- start 2024-01-24T17:48:13.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.343+0000 7f4188c00640 1 -- start start 2024-01-24T17:48:13.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.344+0000 7f4188c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f41840cdaf0 0x7f418406bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.344+0000 7f4188c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41841540f0 con 0x7f41840ca700 2024-01-24T17:48:13.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.344+0000 7f4188c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418415da00 con 0x7f41840ca320 2024-01-24T17:48:13.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.344+0000 7f4188c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4184159fd0 con 0x7f41840cdaf0 2024-01-24T17:48:13.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f417be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f41840ca700 0x7f418415a3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35170/0 (socket says 172.21.15.39:35170) 2024-01-24T17:48:13.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f417be00640 1 -- 172.21.15.39:0/30979253 learned_addr learned my addr 172.21.15.39:0/30979253 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:13.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4183e00640 1 --2- 172.21.15.39:0/30979253 >> v2:172.21.15.100:3300/0 conn(0x7f41840cdaf0 0x7f418406bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207779201 0 0) 0x7f41841540f0 con 0x7f41840ca700 2024-01-24T17:48:13.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4164003520 con 0x7f41840ca700 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202641902 0 0) 0x7f4164003520 con 0x7f41840ca700 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 >> v2:172.21.15.100:3300/0 conn(0x7f41840cdaf0 msgr2=0x7f418406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4182a00640 1 --2- 172.21.15.39:0/30979253 >> v2:172.21.15.100:3300/0 conn(0x7f41840cdaf0 0x7f418406bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 >> v1:172.21.15.100:6789/0 conn(0x7f41840ca320 legacy=0x7f418415d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418415dce0 con 0x7f41840ca700 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.345+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f416c001670 con 0x7f41840ca700 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 >> v1:172.21.15.39:6789/0 conn(0x7f41840ca700 legacy=0x7f418415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4182a00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4164003970 0x7f4164003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4182a00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41640047c0 0x7f4164004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4182a00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640052a0 0x7f416400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4184159fd0 con 0x7f41640047c0 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416400c170 con 0x7f41640052a0 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416400c330 con 0x7f4164003970 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f417be00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41640047c0 0x7f4164004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4183400640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4164003970 0x7f4164003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f4183e00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640052a0 0x7f416400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f417be00640 1 -- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4164003970 msgr2=0x7f4164003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f417be00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4164003970 0x7f4164003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f417be00640 1 -- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640052a0 msgr2=0x7f416400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f417be00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640052a0 0x7f416400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.346+0000 7f417be00640 1 -- 172.21.15.39:0/30979253 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4164003520 con 0x7f41640047c0 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.347+0000 7f4183e00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640052a0 0x7f416400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.347+0000 7f4183e00640 1 -- 172.21.15.39:0/30979253 reap_dead start 2024-01-24T17:48:13.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.348+0000 7f417be00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41640047c0 0x7f4164004c40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f416c00c210 tx=0x7f41640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e14213d5f072ad server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:13.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.348+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c005ad0 con 0x7f41640047c0 2024-01-24T17:48:13.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.348+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c005c30 con 0x7f41640047c0 2024-01-24T17:48:13.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.348+0000 7f4182a00640 1 -- 172.21.15.39:0/30979253 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c02ca50 con 0x7f41640047c0 2024-01-24T17:48:13.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.348+0000 7f4188c00640 1 -- 172.21.15.39:0/30979253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41640047c0 msgr2=0x7f4164004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.348+0000 7f4188c00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41640047c0 0x7f4164004c40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f416c00c210 tx=0x7f41640057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.349+0000 7f4188c00640 1 -- 172.21.15.39:0/30979253 shutdown_connections 2024-01-24T17:48:13.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.349+0000 7f4188c00640 1 --2- 172.21.15.39:0/30979253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41640047c0 0x7f4164004c40 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.349+0000 7f4188c00640 1 -- 172.21.15.39:0/30979253 >> 172.21.15.39:0/30979253 conn(0x7f418406a340 msgr2=0x7f41840cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:13.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.349+0000 7f4188c00640 1 -- 172.21.15.39:0/30979253 shutdown_connections 2024-01-24T17:48:13.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.349+0000 7f4188c00640 1 -- 172.21.15.39:0/30979253 wait complete. 2024-01-24T17:48:13.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.350+0000 7f4188c00640 1 Processor -- start 2024-01-24T17:48:13.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.351+0000 7f4188c00640 1 -- start start 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.351+0000 7f4188c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640047c0 0x7f418414d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4188c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41841f5740 0x7f41841f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4188c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41840c2860 0x7f41840c2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4188c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840c3220 con 0x7f41840c2860 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4188c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41840a9290 con 0x7f41640047c0 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4188c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4184154f50 con 0x7f41841f5740 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41840c2860 0x7f41840c2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41840c2860 0x7f41840c2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52106/0 (socket says 172.21.15.39:52106) 2024-01-24T17:48:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183e00640 1 -- 172.21.15.39:0/2004065691 learned_addr learned my addr 172.21.15.39:0/2004065691 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183400640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640047c0 0x7f418414d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f417be00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41841f5740 0x7f41841f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183e00640 1 -- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41841f5740 msgr2=0x7f41841f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183e00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41841f5740 0x7f41841f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183e00640 1 -- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640047c0 msgr2=0x7f418414d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183e00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640047c0 0x7f418414d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f4183e00640 1 -- 172.21.15.39:0/2004065691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41841f62b0 con 0x7f41840c2860 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.352+0000 7f417be00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41841f5740 0x7f41841f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.353+0000 7f4183e00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41840c2860 0x7f41840c2ce0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f418406e200 tx=0x7f417c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.353+0000 7f417b400640 1 -- 172.21.15.39:0/2004065691 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c016620 con 0x7f41840c2860 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.353+0000 7f417b400640 1 -- 172.21.15.39:0/2004065691 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c016780 con 0x7f41840c2860 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.353+0000 7f417b400640 1 -- 172.21.15.39:0/2004065691 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c016b10 con 0x7f41840c2860 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.353+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41841f6590 con 0x7f41840c2860 2024-01-24T17:48:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.353+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41840a8f20 con 0x7f41840c2860 2024-01-24T17:48:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.354+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41840a9400 con 0x7f41840c2860 2024-01-24T17:48:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.355+0000 7f417b400640 1 -- 172.21.15.39:0/2004065691 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f417c016c70 con 0x7f41840c2860 2024-01-24T17:48:13.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.357+0000 7f417b400640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f415c06a430 0x7f415c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.357+0000 7f417b400640 1 -- 172.21.15.39:0/2004065691 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f417c0897a0 con 0x7f41840c2860 2024-01-24T17:48:13.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.357+0000 7f4183400640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f415c06a430 0x7f415c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.359+0000 7f4183400640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f415c06a430 0x7f415c06c8f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4174001680 tx=0x7f4174009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:13.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.361+0000 7f417b400640 1 -- 172.21.15.39:0/2004065691 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f417c054e30 con 0x7f41840c2860 2024-01-24T17:48:13.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.560+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f418404dff0 con 0x7f41840c2860 2024-01-24T17:48:13.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.563+0000 7f417b400640 1 -- 172.21.15.39:0/2004065691 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16104 (secure 0 0 0) 0x7f417c058450 con 0x7f41840c2860 2024-01-24T17:48:13.564 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:13.564 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":88,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:47:32.941651+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:47:30.315286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-24T17:47:30.960101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f415c06a430 msgr2=0x7f415c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f415c06a430 0x7f415c06c8f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4174001680 tx=0x7f4174009040 comp rx=0 tx=0).stop 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41840c2860 msgr2=0x7f41840c2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41840c2860 0x7f41840c2ce0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f418406e200 tx=0x7f417c00f520 comp rx=0 tx=0).stop 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 shutdown_connections 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f415c06a430 0x7f415c06c8f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41840c2860 0x7f41840c2ce0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f41841f5740 0x7f41841f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 --2- 172.21.15.39:0/2004065691 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f41640047c0 0x7f418414d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 >> 172.21.15.39:0/2004065691 conn(0x7f418406a340 msgr2=0x7f418406bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 shutdown_connections 2024-01-24T17:48:13.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.566+0000 7f4188c00640 1 -- 172.21.15.39:0/2004065691 wait complete. 2024-01-24T17:48:13.587 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-24T17:48:13.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.810+0000 7faa5e000640 1 Processor -- start 2024-01-24T17:48:13.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.810+0000 7faa5e000640 1 -- start start 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.811+0000 7faa5e000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faa5815a9a0 0x7faa5806bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.811+0000 7faa5e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa580ca6d0 0x7faa5815b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.811+0000 7faa5e000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5815b040 con 0x7faa580cadc0 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.811+0000 7faa5e000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa5806c420 con 0x7faa5815a9a0 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.811+0000 7faa5e000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5806c700 con 0x7faa580ca6d0 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa57e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faa5815a9a0 0x7faa5806bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa57e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faa5815a9a0 0x7faa5806bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52116/0 (socket says 172.21.15.39:52116) 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa57e00640 1 -- 172.21.15.39:0/496152172 learned_addr learned my addr 172.21.15.39:0/496152172 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa5d600640 1 --2- 172.21.15.39:0/496152172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa580ca6d0 0x7faa5815b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa57e00640 1 -- 172.21.15.39:0/496152172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa580ca6d0 msgr2=0x7faa5815b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:13.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa57e00640 1 --2- 172.21.15.39:0/496152172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa580ca6d0 0x7faa5815b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa57e00640 1 -- 172.21.15.39:0/496152172 >> v1:172.21.15.39:6789/0 conn(0x7faa580cadc0 legacy=0x7faa580cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa57e00640 1 -- 172.21.15.39:0/496152172 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5806c9e0 con 0x7faa5815a9a0 2024-01-24T17:48:13.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.812+0000 7faa5d600640 1 --2- 172.21.15.39:0/496152172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa580ca6d0 0x7faa5815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:13.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.814+0000 7faa57e00640 1 --2- 172.21.15.39:0/496152172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa5815a9a0 0x7faa5806bd40 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7faa40009870 tx=0x7faa4002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d922a0bb7ddfe4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:13.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.814+0000 7faa57400640 1 -- 172.21.15.39:0/496152172 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40031860 con 0x7faa5815a9a0 2024-01-24T17:48:13.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.814+0000 7faa57400640 1 -- 172.21.15.39:0/496152172 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa400319c0 con 0x7faa5815a9a0 2024-01-24T17:48:13.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.814+0000 7faa57400640 1 -- 172.21.15.39:0/496152172 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40031d50 con 0x7faa5815a9a0 2024-01-24T17:48:13.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.815+0000 7faa5e000640 1 -- 172.21.15.39:0/496152172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa5815a9a0 msgr2=0x7faa5806bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.815+0000 7faa5e000640 1 --2- 172.21.15.39:0/496152172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa5815a9a0 0x7faa5806bd40 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7faa40009870 tx=0x7faa4002b900 comp rx=0 tx=0).stop 2024-01-24T17:48:13.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.815+0000 7faa5e000640 1 -- 172.21.15.39:0/496152172 shutdown_connections 2024-01-24T17:48:13.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.815+0000 7faa5e000640 1 --2- 172.21.15.39:0/496152172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa5815a9a0 0x7faa5806bd40 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.815+0000 7faa5e000640 1 --2- 172.21.15.39:0/496152172 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa580ca6d0 0x7faa5815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.815+0000 7faa5e000640 1 -- 172.21.15.39:0/496152172 >> 172.21.15.39:0/496152172 conn(0x7faa5806a200 msgr2=0x7faa580aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:13.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.815+0000 7faa5e000640 1 -- 172.21.15.39:0/496152172 shutdown_connections 2024-01-24T17:48:13.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.815+0000 7faa5e000640 1 -- 172.21.15.39:0/496152172 wait complete. 2024-01-24T17:48:13.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.817+0000 7faa5e000640 1 Processor -- start 2024-01-24T17:48:13.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.817+0000 7faa5e000640 1 -- start start 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa5e000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa580ca6d0 0x7faa580a74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa5e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa580cadc0 0x7faa5814bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa5e000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa5815a9a0 0x7faa580b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa5e000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa580a79f0 con 0x7faa580cadc0 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa5e000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa580c5d30 con 0x7faa580ca6d0 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa5e000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa580c8190 con 0x7faa5815a9a0 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa57e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa580cadc0 0x7faa5814bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa57e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa580cadc0 0x7faa5814bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52122/0 (socket says 172.21.15.39:52122) 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa57e00640 1 -- 172.21.15.39:0/2632326030 learned_addr learned my addr 172.21.15.39:0/2632326030 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa5cc00640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa580ca6d0 0x7faa580a74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa5d600640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa5815a9a0 0x7faa580b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa57e00640 1 -- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa5815a9a0 msgr2=0x7faa580b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.818+0000 7faa57e00640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa5815a9a0 0x7faa580b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa57e00640 1 -- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa580ca6d0 msgr2=0x7faa580a74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa57e00640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa580ca6d0 0x7faa580a74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa57e00640 1 -- 172.21.15.39:0/2632326030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400094d0 con 0x7faa580cadc0 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa5cc00640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa580ca6d0 0x7faa580a74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa57e00640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa580cadc0 0x7faa5814bb40 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7faa4002b940 tx=0x7faa400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa54c00640 1 -- 172.21.15.39:0/2632326030 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa40031830 con 0x7faa580cadc0 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa54c00640 1 -- 172.21.15.39:0/2632326030 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa400443f0 con 0x7faa580cadc0 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa54c00640 1 -- 172.21.15.39:0/2632326030 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa400447a0 con 0x7faa580cadc0 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa580c8430 con 0x7faa580cadc0 2024-01-24T17:48:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.819+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa580c59c0 con 0x7faa580cadc0 2024-01-24T17:48:13.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.821+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5804dff0 con 0x7faa580cadc0 2024-01-24T17:48:13.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.821+0000 7faa54c00640 1 -- 172.21.15.39:0/2632326030 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7faa400303b0 con 0x7faa580cadc0 2024-01-24T17:48:13.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.822+0000 7faa54c00640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faa3006a460 0x7faa3006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:13.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.822+0000 7faa54c00640 1 -- 172.21.15.39:0/2632326030 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa400ab190 con 0x7faa580cadc0 2024-01-24T17:48:13.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.823+0000 7faa5cc00640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faa3006a460 0x7faa3006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:13.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.824+0000 7faa5cc00640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faa3006a460 0x7faa3006c920 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7faa48000fd0 tx=0x7faa48009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:13.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:13.827+0000 7faa54c00640 1 -- 172.21.15.39:0/2632326030 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7faa40076770 con 0x7faa580cadc0 2024-01-24T17:48:14.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.059+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7faa580c5e70 con 0x7faa580cadc0 2024-01-24T17:48:14.061 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:48:14.061+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:48:14.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.079+0000 7faa54c00640 1 -- 172.21.15.39:0/2632326030 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) v1 ==== 161+0+0 (secure 0 0 0) 0x7faa40079d90 con 0x7faa580cadc0 2024-01-24T17:48:14.079 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-24T17:48:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faa3006a460 msgr2=0x7faa3006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faa3006a460 0x7faa3006c920 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7faa48000fd0 tx=0x7faa48009040 comp rx=0 tx=0).stop 2024-01-24T17:48:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa580cadc0 msgr2=0x7faa5814bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa580cadc0 0x7faa5814bb40 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7faa4002b940 tx=0x7faa400046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 shutdown_connections 2024-01-24T17:48:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faa5815a9a0 0x7faa580b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7faa3006a460 0x7faa3006c920 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa580cadc0 0x7faa5814bb40 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 --2- 172.21.15.39:0/2632326030 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa580ca6d0 0x7faa580a74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.081+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 >> 172.21.15.39:0/2632326030 conn(0x7faa5806a200 msgr2=0x7faa580adaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:14.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.082+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 shutdown_connections 2024-01-24T17:48:14.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.082+0000 7faa5e000640 1 -- 172.21.15.39:0/2632326030 wait complete. 2024-01-24T17:48:14.095 INFO:tasks.ceph.mds.c.smithi039.stderr: -1735> 2024-01-24T17:47:37.413+0000 7fa70fc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:48:14.102 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-24T17:48:14.161 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:48:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.298+0000 7fe637c00640 1 Processor -- start 2024-01-24T17:48:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.299+0000 7fe637c00640 1 -- start start 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.300+0000 7fe637c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe6300b0010 0x7fe63015bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.300+0000 7fe637c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe630070920 0x7fe6300b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.300+0000 7fe637c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe630070470 con 0x7fe63015f390 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.300+0000 7fe637c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63015ef00 con 0x7fe630070920 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.300+0000 7fe637c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6300710a0 con 0x7fe6300b0010 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.300+0000 7fe627c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe630070920 0x7fe6300b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.300+0000 7fe635c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe63015f390 0x7fe6300cafa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35208/0 (socket says 172.21.15.39:35208) 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.300+0000 7fe635c00640 1 -- 172.21.15.39:0/187553898 learned_addr learned my addr 172.21.15.39:0/187553898 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.301+0000 7fe635200640 1 --2- 172.21.15.39:0/187553898 >> v2:172.21.15.100:3300/0 conn(0x7fe6300b0010 0x7fe63015bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:14.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.301+0000 7fe627c00640 1 -- 172.21.15.39:0/187553898 >> v2:172.21.15.100:3300/0 conn(0x7fe6300b0010 msgr2=0x7fe63015bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:14.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.301+0000 7fe627c00640 1 --2- 172.21.15.39:0/187553898 >> v2:172.21.15.100:3300/0 conn(0x7fe6300b0010 0x7fe63015bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.301+0000 7fe627c00640 1 -- 172.21.15.39:0/187553898 >> v1:172.21.15.39:6789/0 conn(0x7fe63015f390 legacy=0x7fe6300cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:14.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.301+0000 7fe627c00640 1 -- 172.21.15.39:0/187553898 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63015c0d0 con 0x7fe630070920 2024-01-24T17:48:14.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.301+0000 7fe62fe00640 1 -- 172.21.15.39:0/187553898 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880735743 0 0) 0x7fe630070470 con 0x7fe63015f390 2024-01-24T17:48:14.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.302+0000 7fe627c00640 1 --2- 172.21.15.39:0/187553898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe630070920 0x7fe6300b03f0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fe6300b1010 tx=0x7fe61802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69cec774e5d83b99 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.302+0000 7fe62fe00640 1 -- 172.21.15.39:0/187553898 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61802cd00 con 0x7fe630070920 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.302+0000 7fe62fe00640 1 -- 172.21.15.39:0/187553898 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61802ce60 con 0x7fe630070920 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.302+0000 7fe62fe00640 1 -- 172.21.15.39:0/187553898 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618034660 con 0x7fe630070920 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.303+0000 7fe637c00640 1 -- 172.21.15.39:0/187553898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe630070920 msgr2=0x7fe6300b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.303+0000 7fe637c00640 1 --2- 172.21.15.39:0/187553898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe630070920 0x7fe6300b03f0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fe6300b1010 tx=0x7fe61802bf50 comp rx=0 tx=0).stop 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.303+0000 7fe637c00640 1 -- 172.21.15.39:0/187553898 shutdown_connections 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.303+0000 7fe637c00640 1 --2- 172.21.15.39:0/187553898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe630070920 0x7fe6300b03f0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.303+0000 7fe637c00640 1 --2- 172.21.15.39:0/187553898 >> v2:172.21.15.100:3300/0 conn(0x7fe6300b0010 0x7fe63015bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.303+0000 7fe637c00640 1 -- 172.21.15.39:0/187553898 >> 172.21.15.39:0/187553898 conn(0x7fe63006bf20 msgr2=0x7fe63006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.303+0000 7fe637c00640 1 -- 172.21.15.39:0/187553898 shutdown_connections 2024-01-24T17:48:14.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.303+0000 7fe637c00640 1 -- 172.21.15.39:0/187553898 wait complete. 2024-01-24T17:48:14.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.305+0000 7fe637c00640 1 Processor -- start 2024-01-24T17:48:14.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.305+0000 7fe637c00640 1 -- start start 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe637c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe630070920 0x7fe630154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe637c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6300b0010 0x7fe6300cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe637c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63015f390 0x7fe6300cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe637c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe630155260 con 0x7fe63015f390 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe637c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300c7510 con 0x7fe6300b0010 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe637c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300ad390 con 0x7fe630070920 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe635c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63015f390 0x7fe6300cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe635c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63015f390 0x7fe6300cd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52160/0 (socket says 172.21.15.39:52160) 2024-01-24T17:48:14.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.306+0000 7fe635c00640 1 -- 172.21.15.39:0/3660074949 learned_addr learned my addr 172.21.15.39:0/3660074949 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe635200640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe630070920 0x7fe630154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe627c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6300b0010 0x7fe6300cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe635c00640 1 -- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe630070920 msgr2=0x7fe630154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe635c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe630070920 0x7fe630154d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe635c00640 1 -- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6300b0010 msgr2=0x7fe6300cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe635c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6300b0010 0x7fe6300cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe635c00640 1 -- 172.21.15.39:0/3660074949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe618009a70 con 0x7fe63015f390 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe627c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6300b0010 0x7fe6300cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe635c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63015f390 0x7fe6300cd800 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe62800c3e0 tx=0x7fe62800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe62d600640 1 -- 172.21.15.39:0/3660074949 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6280074e0 con 0x7fe63015f390 2024-01-24T17:48:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe62d600640 1 -- 172.21.15.39:0/3660074949 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe628007640 con 0x7fe63015f390 2024-01-24T17:48:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe62d600640 1 -- 172.21.15.39:0/3660074949 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6280049e0 con 0x7fe63015f390 2024-01-24T17:48:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6300c6990 con 0x7fe63015f390 2024-01-24T17:48:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.307+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6300c6f50 con 0x7fe63015f390 2024-01-24T17:48:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.308+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6300ad560 con 0x7fe63015f390 2024-01-24T17:48:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.309+0000 7fe62d600640 1 -- 172.21.15.39:0/3660074949 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe628004b40 con 0x7fe63015f390 2024-01-24T17:48:14.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.310+0000 7fe62d600640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe60806a430 0x7fe60806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:14.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.310+0000 7fe62d600640 1 -- 172.21.15.39:0/3660074949 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe628088700 con 0x7fe63015f390 2024-01-24T17:48:14.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.311+0000 7fe635200640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe60806a430 0x7fe60806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:14.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.312+0000 7fe635200640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe60806a430 0x7fe60806c8f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe620006fd0 tx=0x7fe620008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:14.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.315+0000 7fe62d600640 1 -- 172.21.15.39:0/3660074949 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe628057710 con 0x7fe63015f390 2024-01-24T17:48:14.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:14.551+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe63004dff0 con 0x7fe63015f390 2024-01-24T17:48:15.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.084+0000 7fe62d600640 1 -- 172.21.15.39:0/3660074949 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe6280572a0 con 0x7fe63015f390 2024-01-24T17:48:15.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe60806a430 msgr2=0x7fe60806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:15.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe60806a430 0x7fe60806c8f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe620006fd0 tx=0x7fe620008040 comp rx=0 tx=0).stop 2024-01-24T17:48:15.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63015f390 msgr2=0x7fe6300cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:15.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63015f390 0x7fe6300cd800 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe62800c3e0 tx=0x7fe62800c8a0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 shutdown_connections 2024-01-24T17:48:15.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe60806a430 0x7fe60806c8f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe63015f390 0x7fe6300cd800 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6300b0010 0x7fe6300cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 --2- 172.21.15.39:0/3660074949 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe630070920 0x7fe630154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.087+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 >> 172.21.15.39:0/3660074949 conn(0x7fe63006bf20 msgr2=0x7fe63015a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:15.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.088+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 shutdown_connections 2024-01-24T17:48:15.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.088+0000 7fe637c00640 1 -- 172.21.15.39:0/3660074949 wait complete. 2024-01-24T17:48:15.109 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-24T17:48:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.322+0000 7fd24b000640 1 Processor -- start 2024-01-24T17:48:15.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.322+0000 7fd24b000640 1 -- start start 2024-01-24T17:48:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.324+0000 7fd24b000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd24415cdb0 0x7fd2440d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.324+0000 7fd24b000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440b2070 con 0x7fd2440d0650 2024-01-24T17:48:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.324+0000 7fd24b000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440b01b0 con 0x7fd2440c63c0 2024-01-24T17:48:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.324+0000 7fd24b000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24415d190 con 0x7fd24415cdb0 2024-01-24T17:48:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.324+0000 7fd249200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd2440d0650 0x7fd2440b1250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35210/0 (socket says 172.21.15.39:35210) 2024-01-24T17:48:15.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.324+0000 7fd249200640 1 -- 172.21.15.39:0/3555847390 learned_addr learned my addr 172.21.15.39:0/3555847390 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:15.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd249c00640 1 --2- 172.21.15.39:0/3555847390 >> v2:172.21.15.100:3300/0 conn(0x7fd24415cdb0 0x7fd2440d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:15.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566177048 0 0) 0x7fd22c001110 con 0x7fd2440d0650 2024-01-24T17:48:15.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd224003520 con 0x7fd2440d0650 2024-01-24T17:48:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4287104031 0 0) 0x7fd224003520 con 0x7fd2440d0650 2024-01-24T17:48:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 >> v2:172.21.15.100:3300/0 conn(0x7fd24415cdb0 msgr2=0x7fd2440d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 --2- 172.21.15.39:0/3555847390 >> v2:172.21.15.100:3300/0 conn(0x7fd24415cdb0 0x7fd2440d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 >> v1:172.21.15.100:6789/0 conn(0x7fd2440c63c0 legacy=0x7fd2440b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:15.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2440b0490 con 0x7fd2440d0650 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fd22c001800 con 0x7fd2440d0650 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.325+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 >> v1:172.21.15.39:6789/0 conn(0x7fd2440d0650 legacy=0x7fd2440b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd243e00640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2240039c0 0x7fd224003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd243e00640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240047c0 0x7fd224004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd243e00640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd224005180 0x7fd22400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400c050 con 0x7fd2240047c0 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400c1c0 con 0x7fd2240039c0 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22400c380 con 0x7fd224005180 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd249200640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240047c0 0x7fd224004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd249c00640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2240039c0 0x7fd224003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd249200640 1 -- 172.21.15.39:0/3555847390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd224005180 msgr2=0x7fd22400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd249200640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd224005180 0x7fd22400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd249200640 1 -- 172.21.15.39:0/3555847390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2240039c0 msgr2=0x7fd224003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:15.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd249200640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2240039c0 0x7fd224003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd249200640 1 -- 172.21.15.39:0/3555847390 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd224003520 con 0x7fd2240047c0 2024-01-24T17:48:15.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.326+0000 7fd24a600640 1 -- 172.21.15.39:0/3555847390 reap_dead start 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.327+0000 7fd249200640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240047c0 0x7fd224004c40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fd22c007fc0 tx=0x7fd2240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf0abe20587bb42 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.328+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c005ad0 con 0x7fd2240047c0 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.328+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c005c50 con 0x7fd2240047c0 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.328+0000 7fd243e00640 1 -- 172.21.15.39:0/3555847390 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c02c7e0 con 0x7fd2240047c0 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.328+0000 7fd24b000640 1 -- 172.21.15.39:0/3555847390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240047c0 msgr2=0x7fd224004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.328+0000 7fd24b000640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240047c0 0x7fd224004c40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fd22c007fc0 tx=0x7fd2240057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.328+0000 7fd24b000640 1 -- 172.21.15.39:0/3555847390 shutdown_connections 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.328+0000 7fd24b000640 1 --2- 172.21.15.39:0/3555847390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2240047c0 0x7fd224004c40 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.328+0000 7fd24b000640 1 -- 172.21.15.39:0/3555847390 >> 172.21.15.39:0/3555847390 conn(0x7fd2440d2900 msgr2=0x7fd2440d2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.329+0000 7fd24b000640 1 -- 172.21.15.39:0/3555847390 shutdown_connections 2024-01-24T17:48:15.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.329+0000 7fd24b000640 1 -- 172.21.15.39:0/3555847390 wait complete. 2024-01-24T17:48:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.330+0000 7fd24b000640 1 Processor -- start 2024-01-24T17:48:15.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.330+0000 7fd24b000640 1 -- start start 2024-01-24T17:48:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.331+0000 7fd24b000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2240047c0 0x7fd2440a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:15.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.331+0000 7fd24b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2440c7e80 0x7fd244156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.331+0000 7fd24b000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2440c6cd0 0x7fd2440c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.331+0000 7fd24b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2440c7690 con 0x7fd2440c7e80 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.331+0000 7fd24b000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2440ccc00 con 0x7fd2440c6cd0 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.331+0000 7fd24b000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd244157420 con 0x7fd2240047c0 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2440c7e80 0x7fd244156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2440c7e80 0x7fd244156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52172/0 (socket says 172.21.15.39:52172) 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249200640 1 -- 172.21.15.39:0/3218011405 learned_addr learned my addr 172.21.15.39:0/3218011405 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249c00640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2240047c0 0x7fd2440a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd24a600640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2440c6cd0 0x7fd2440c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249200640 1 -- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2240047c0 msgr2=0x7fd2440a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249200640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2240047c0 0x7fd2440a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249200640 1 -- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2440c6cd0 msgr2=0x7fd2440c7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:15.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249200640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2440c6cd0 0x7fd2440c7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:15.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.332+0000 7fd249200640 1 -- 172.21.15.39:0/3218011405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2440cbe40 con 0x7fd2440c7e80 2024-01-24T17:48:15.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.333+0000 7fd249200640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2440c7e80 0x7fd244156b20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00c1d0 tx=0x7fd22c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:15.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.333+0000 7fd241600640 1 -- 172.21.15.39:0/3218011405 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c03c040 con 0x7fd2440c7e80 2024-01-24T17:48:15.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.333+0000 7fd241600640 1 -- 172.21.15.39:0/3218011405 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c02cbf0 con 0x7fd2440c7e80 2024-01-24T17:48:15.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.333+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2440cc120 con 0x7fd2440c7e80 2024-01-24T17:48:15.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.333+0000 7fd241600640 1 -- 172.21.15.39:0/3218011405 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c007070 con 0x7fd2440c7e80 2024-01-24T17:48:15.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.333+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2440cc6e0 con 0x7fd2440c7e80 2024-01-24T17:48:15.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.334+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24404e840 con 0x7fd2440c7e80 2024-01-24T17:48:15.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.335+0000 7fd241600640 1 -- 172.21.15.39:0/3218011405 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd22c032070 con 0x7fd2440c7e80 2024-01-24T17:48:15.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.337+0000 7fd241600640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21c06a430 0x7fd21c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:15.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.337+0000 7fd241600640 1 -- 172.21.15.39:0/3218011405 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd22c0a9c40 con 0x7fd2440c7e80 2024-01-24T17:48:15.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.337+0000 7fd249c00640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21c06a430 0x7fd21c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:15.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.338+0000 7fd249c00640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21c06a430 0x7fd21c06c8f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd234009a80 tx=0x7fd234008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:15.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.341+0000 7fd241600640 1 -- 172.21.15.39:0/3218011405 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd22c0752e0 con 0x7fd2440c7e80 2024-01-24T17:48:15.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:15.530+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2440b2070 con 0x7fd2440c7e80 2024-01-24T17:48:16.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.333+0000 7fd241600640 1 -- 172.21.15.39:0/3218011405 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd2440b2070 con 0x7fd2440c7e80 2024-01-24T17:48:16.335 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-01-24T17:48:16.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21c06a430 msgr2=0x7fd21c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21c06a430 0x7fd21c06c8f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd234009a80 tx=0x7fd234008040 comp rx=0 tx=0).stop 2024-01-24T17:48:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2440c7e80 msgr2=0x7fd244156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2440c7e80 0x7fd244156b20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00c1d0 tx=0x7fd22c02e070 comp rx=0 tx=0).stop 2024-01-24T17:48:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 shutdown_connections 2024-01-24T17:48:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2440c6cd0 0x7fd2440c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd21c06a430 0x7fd21c06c8f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd2440c7e80 0x7fd244156b20 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 --2- 172.21.15.39:0/3218011405 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd2240047c0 0x7fd2440a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.338+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 >> 172.21.15.39:0/3218011405 conn(0x7fd2440d2900 msgr2=0x7fd2440af8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.339+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 shutdown_connections 2024-01-24T17:48:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.339+0000 7fd24b000640 1 -- 172.21.15.39:0/3218011405 wait complete. 2024-01-24T17:48:16.361 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-24T17:48:16.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.574+0000 7fbfd7e00640 1 Processor -- start 2024-01-24T17:48:16.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.574+0000 7fbfd7e00640 1 -- start start 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.575+0000 7fbfd7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbfd81516c0 0x7fbfd80b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.575+0000 7fbfd7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbfd815ac70 0x7fbfd806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfd7e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd80b12b0 con 0x7fbfd8070260 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfd7e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd80706d0 con 0x7fbfd815ac70 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfd7e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd815b170 con 0x7fbfd81516c0 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfcfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbfd815ac70 0x7fbfd806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfcfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbfd815ac70 0x7fbfd806fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52180/0 (socket says 172.21.15.39:52180) 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfcfe00640 1 -- 172.21.15.39:0/2238365532 learned_addr learned my addr 172.21.15.39:0/2238365532 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfd6a00640 1 --2- 172.21.15.39:0/2238365532 >> v2:172.21.15.100:3300/0 conn(0x7fbfd81516c0 0x7fbfd80b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:16.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfcfe00640 1 -- 172.21.15.39:0/2238365532 >> v2:172.21.15.100:3300/0 conn(0x7fbfd81516c0 msgr2=0x7fbfd80b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfcfe00640 1 --2- 172.21.15.39:0/2238365532 >> v2:172.21.15.100:3300/0 conn(0x7fbfd81516c0 0x7fbfd80b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfcfe00640 1 -- 172.21.15.39:0/2238365532 >> v1:172.21.15.100:6789/0 conn(0x7fbfd8070260 legacy=0x7fbfd80ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.576+0000 7fbfcfe00640 1 -- 172.21.15.39:0/2238365532 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd815b450 con 0x7fbfd815ac70 2024-01-24T17:48:16.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.578+0000 7fbfcfe00640 1 --2- 172.21.15.39:0/2238365532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd815ac70 0x7fbfd806fd20 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbfd806df70 tx=0x7fbfc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853afb0c318609ac server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:16.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.578+0000 7fbfd6000640 1 -- 172.21.15.39:0/2238365532 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0031870 con 0x7fbfd815ac70 2024-01-24T17:48:16.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.578+0000 7fbfd6000640 1 -- 172.21.15.39:0/2238365532 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc00319d0 con 0x7fbfd815ac70 2024-01-24T17:48:16.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.578+0000 7fbfd6000640 1 -- 172.21.15.39:0/2238365532 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0031d60 con 0x7fbfd815ac70 2024-01-24T17:48:16.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.578+0000 7fbfd7e00640 1 -- 172.21.15.39:0/2238365532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd815ac70 msgr2=0x7fbfd806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:16.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.578+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/2238365532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd815ac70 0x7fbfd806fd20 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbfd806df70 tx=0x7fbfc002b910 comp rx=0 tx=0).stop 2024-01-24T17:48:16.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.579+0000 7fbfd7e00640 1 -- 172.21.15.39:0/2238365532 shutdown_connections 2024-01-24T17:48:16.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.579+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/2238365532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd815ac70 0x7fbfd806fd20 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.579+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/2238365532 >> v2:172.21.15.100:3300/0 conn(0x7fbfd81516c0 0x7fbfd80b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.579+0000 7fbfd7e00640 1 -- 172.21.15.39:0/2238365532 >> 172.21.15.39:0/2238365532 conn(0x7fbfd806a200 msgr2=0x7fbfd80affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:16.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.579+0000 7fbfd7e00640 1 -- 172.21.15.39:0/2238365532 shutdown_connections 2024-01-24T17:48:16.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.579+0000 7fbfd7e00640 1 -- 172.21.15.39:0/2238365532 wait complete. 2024-01-24T17:48:16.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.580+0000 7fbfd7e00640 1 Processor -- start 2024-01-24T17:48:16.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.581+0000 7fbfd7e00640 1 -- start start 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd8070260 0x7fbfd814c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd81516c0 0x7fbfd81e6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfd815ac70 0x7fbfd81e4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80c2fa0 con 0x7fbfd81516c0 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd7e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80c31f0 con 0x7fbfd8070260 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd7e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80c34d0 con 0x7fbfd815ac70 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfcfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd81516c0 0x7fbfd81e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfcfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd81516c0 0x7fbfd81e6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52182/0 (socket says 172.21.15.39:52182) 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfcfe00640 1 -- 172.21.15.39:0/3667643734 learned_addr learned my addr 172.21.15.39:0/3667643734 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd6a00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd8070260 0x7fbfd814c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:16.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfcfe00640 1 -- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfd815ac70 msgr2=0x7fbfd81e4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd7400640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfd815ac70 0x7fbfd81e4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfcfe00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfd815ac70 0x7fbfd81e4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfcfe00640 1 -- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd8070260 msgr2=0x7fbfd814c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfcfe00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd8070260 0x7fbfd814c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfcfe00640 1 -- 172.21.15.39:0/3667643734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc00094d0 con 0x7fbfd81516c0 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd6a00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd8070260 0x7fbfd814c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.582+0000 7fbfd7400640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfd815ac70 0x7fbfd81e4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.583+0000 7fbfcfe00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd81516c0 0x7fbfd81e6100 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fbfc002b950 tx=0x7fbfc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.583+0000 7fbfcea00640 1 -- 172.21.15.39:0/3667643734 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc003ad30 con 0x7fbfd81516c0 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.583+0000 7fbfcea00640 1 -- 172.21.15.39:0/3667643734 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc00443f0 con 0x7fbfd81516c0 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.583+0000 7fbfcea00640 1 -- 172.21.15.39:0/3667643734 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc00447a0 con 0x7fbfd81516c0 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.583+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd80c37b0 con 0x7fbfd81516c0 2024-01-24T17:48:16.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.583+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd80c3d70 con 0x7fbfd81516c0 2024-01-24T17:48:16.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.584+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd804e840 con 0x7fbfd81516c0 2024-01-24T17:48:16.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.585+0000 7fbfcea00640 1 -- 172.21.15.39:0/3667643734 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbfc004d450 con 0x7fbfd81516c0 2024-01-24T17:48:16.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.587+0000 7fbfcea00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfb006a460 0x7fbfb006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:16.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.587+0000 7fbfcea00640 1 -- 172.21.15.39:0/3667643734 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fbfc002d040 con 0x7fbfd81516c0 2024-01-24T17:48:16.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.587+0000 7fbfd6a00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfb006a460 0x7fbfb006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:16.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.589+0000 7fbfd6a00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfb006a460 0x7fbfb006c920 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8006fd0 tx=0x7fbfc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:16.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.591+0000 7fbfcea00640 1 -- 172.21.15.39:0/3667643734 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbfc00764f0 con 0x7fbfd81516c0 2024-01-24T17:48:16.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:16.794+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbfd80b0d20 con 0x7fbfd81516c0 2024-01-24T17:48:17.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.343+0000 7fbfcea00640 1 -- 172.21.15.39:0/3667643734 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbfc00350d0 con 0x7fbfd81516c0 2024-01-24T17:48:17.344 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-01-24T17:48:17.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfb006a460 msgr2=0x7fbfb006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:17.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfb006a460 0x7fbfb006c920 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8006fd0 tx=0x7fbfc8008040 comp rx=0 tx=0).stop 2024-01-24T17:48:17.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd81516c0 msgr2=0x7fbfd81e6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:17.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd81516c0 0x7fbfd81e6100 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fbfc002b950 tx=0x7fbfc00046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 shutdown_connections 2024-01-24T17:48:17.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbfd815ac70 0x7fbfd81e4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbfb006a460 0x7fbfb006c920 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbfd81516c0 0x7fbfd81e6100 secure :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fbfc002b950 tx=0x7fbfc00046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 --2- 172.21.15.39:0/3667643734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfd8070260 0x7fbfd814c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.348+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 >> 172.21.15.39:0/3667643734 conn(0x7fbfd806a200 msgr2=0x7fbfd8159290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:17.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.351+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 shutdown_connections 2024-01-24T17:48:17.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.351+0000 7fbfd7e00640 1 -- 172.21.15.39:0/3667643734 wait complete. 2024-01-24T17:48:17.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-24T17:48:17.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.571+0000 7fea97e00640 1 Processor -- start 2024-01-24T17:48:17.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.571+0000 7fea97e00640 1 -- start start 2024-01-24T17:48:17.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.573+0000 7fea97e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea981516c0 0x7fea980b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:17.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.573+0000 7fea97e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fea98070430 0x7fea980ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:17.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.573+0000 7fea97e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea980b12b0 con 0x7fea9815ac70 2024-01-24T17:48:17.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.573+0000 7fea97e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9815b170 con 0x7fea98070430 2024-01-24T17:48:17.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.573+0000 7fea97e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9815b450 con 0x7fea981516c0 2024-01-24T17:48:17.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.573+0000 7fea96a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea981516c0 0x7fea980b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:17.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.574+0000 7fea96a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea981516c0 0x7fea980b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:51166/0 (socket says 172.21.15.39:51166) 2024-01-24T17:48:17.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.574+0000 7fea96a00640 1 -- 172.21.15.39:0/2682016901 learned_addr learned my addr 172.21.15.39:0/2682016901 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:17.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.574+0000 7fea97400640 1 --2- 172.21.15.39:0/2682016901 >> v2:172.21.15.100:3300/0 conn(0x7fea98070430 0x7fea980ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:17.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.574+0000 7fea96a00640 1 -- 172.21.15.39:0/2682016901 >> v2:172.21.15.100:3300/0 conn(0x7fea98070430 msgr2=0x7fea980ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:17.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.574+0000 7fea96a00640 1 --2- 172.21.15.39:0/2682016901 >> v2:172.21.15.100:3300/0 conn(0x7fea98070430 0x7fea980ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.574+0000 7fea96a00640 1 -- 172.21.15.39:0/2682016901 >> v1:172.21.15.100:6789/0 conn(0x7fea9815ac70 legacy=0x7fea9806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:17.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.574+0000 7fea96a00640 1 -- 172.21.15.39:0/2682016901 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9815b730 con 0x7fea981516c0 2024-01-24T17:48:17.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.576+0000 7fea96a00640 1 --2- 172.21.15.39:0/2682016901 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea981516c0 0x7fea980b3d80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fea88009870 tx=0x7fea8802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4b8db36f59fdd2d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:17.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.576+0000 7fea96000640 1 -- 172.21.15.39:0/2682016901 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88030840 con 0x7fea981516c0 2024-01-24T17:48:17.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.576+0000 7fea96000640 1 -- 172.21.15.39:0/2682016901 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea880309a0 con 0x7fea981516c0 2024-01-24T17:48:17.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.576+0000 7fea96000640 1 -- 172.21.15.39:0/2682016901 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88030d30 con 0x7fea981516c0 2024-01-24T17:48:17.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.577+0000 7fea97e00640 1 -- 172.21.15.39:0/2682016901 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea981516c0 msgr2=0x7fea980b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:17.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.577+0000 7fea97e00640 1 --2- 172.21.15.39:0/2682016901 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea981516c0 0x7fea980b3d80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fea88009870 tx=0x7fea8802b980 comp rx=0 tx=0).stop 2024-01-24T17:48:17.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.577+0000 7fea97e00640 1 -- 172.21.15.39:0/2682016901 shutdown_connections 2024-01-24T17:48:17.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.577+0000 7fea97e00640 1 --2- 172.21.15.39:0/2682016901 >> v2:172.21.15.100:3300/0 conn(0x7fea98070430 0x7fea980ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.577+0000 7fea97e00640 1 --2- 172.21.15.39:0/2682016901 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea981516c0 0x7fea980b3d80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.577+0000 7fea97e00640 1 -- 172.21.15.39:0/2682016901 >> 172.21.15.39:0/2682016901 conn(0x7fea9806a200 msgr2=0x7fea980affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:17.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.577+0000 7fea97e00640 1 -- 172.21.15.39:0/2682016901 shutdown_connections 2024-01-24T17:48:17.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.577+0000 7fea97e00640 1 -- 172.21.15.39:0/2682016901 wait complete. 2024-01-24T17:48:17.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.579+0000 7fea97e00640 1 Processor -- start 2024-01-24T17:48:17.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.580+0000 7fea97e00640 1 -- start start 2024-01-24T17:48:17.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea97e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea98070430 0x7fea98154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:17.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea97e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea981516c0 0x7fea9813f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:17.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea97e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea9815ac70 0x7fea980b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:17.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea97e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98140690 con 0x7fea981516c0 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea97e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98153ba0 con 0x7fea98070430 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea97e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea981400d0 con 0x7fea9815ac70 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea981516c0 0x7fea9813f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea8fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea981516c0 0x7fea9813f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52188/0 (socket says 172.21.15.39:52188) 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea8fe00640 1 -- 172.21.15.39:0/121496706 learned_addr learned my addr 172.21.15.39:0/121496706 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.581+0000 7fea97400640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea9815ac70 0x7fea980b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.582+0000 7fea96a00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea98070430 0x7fea98154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.582+0000 7fea8fe00640 1 -- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea9815ac70 msgr2=0x7fea980b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.582+0000 7fea8fe00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea9815ac70 0x7fea980b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.582+0000 7fea8fe00640 1 -- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea98070430 msgr2=0x7fea98154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.582+0000 7fea8fe00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea98070430 0x7fea98154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.582+0000 7fea8fe00640 1 -- 172.21.15.39:0/121496706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880094d0 con 0x7fea981516c0 2024-01-24T17:48:17.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.582+0000 7fea8fe00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea981516c0 0x7fea9813f6a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fea8000c450 tx=0x7fea8000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:17.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.582+0000 7fea8ea00640 1 -- 172.21.15.39:0/121496706 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea800052d0 con 0x7fea981516c0 2024-01-24T17:48:17.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.583+0000 7fea8ea00640 1 -- 172.21.15.39:0/121496706 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80005430 con 0x7fea981516c0 2024-01-24T17:48:17.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.583+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea981403b0 con 0x7fea981516c0 2024-01-24T17:48:17.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.583+0000 7fea8ea00640 1 -- 172.21.15.39:0/121496706 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80012870 con 0x7fea981516c0 2024-01-24T17:48:17.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.583+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea98154120 con 0x7fea981516c0 2024-01-24T17:48:17.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.584+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9804dff0 con 0x7fea981516c0 2024-01-24T17:48:17.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.585+0000 7fea8ea00640 1 -- 172.21.15.39:0/121496706 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fea800129d0 con 0x7fea981516c0 2024-01-24T17:48:17.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.586+0000 7fea8ea00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a460 0x7fea7006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:17.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.586+0000 7fea8ea00640 1 -- 172.21.15.39:0/121496706 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fea80088050 con 0x7fea981516c0 2024-01-24T17:48:17.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.587+0000 7fea96a00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a460 0x7fea7006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:17.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.588+0000 7fea96a00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a460 0x7fea7006c920 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fea88036000 tx=0x7fea88008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:17.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.591+0000 7fea8ea00640 1 -- 172.21.15.39:0/121496706 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fea800539d0 con 0x7fea981516c0 2024-01-24T17:48:17.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.786+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fea980b0d20 con 0x7fea981516c0 2024-01-24T17:48:17.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.796+0000 7fea8ea00640 1 -- 172.21.15.39:0/121496706 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7fea80056ff0 con 0x7fea981516c0 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a460 msgr2=0x7fea7006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a460 0x7fea7006c920 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fea88036000 tx=0x7fea88008030 comp rx=0 tx=0).stop 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea981516c0 msgr2=0x7fea9813f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea981516c0 0x7fea9813f6a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fea8000c450 tx=0x7fea8000c910 comp rx=0 tx=0).stop 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 shutdown_connections 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fea9815ac70 0x7fea980b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fea7006a460 0x7fea7006c920 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea981516c0 0x7fea9813f6a0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 --2- 172.21.15.39:0/121496706 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fea98070430 0x7fea98154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 >> 172.21.15.39:0/121496706 conn(0x7fea9806a200 msgr2=0x7fea980ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 shutdown_connections 2024-01-24T17:48:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:17.799+0000 7fea97e00640 1 -- 172.21.15.39:0/121496706 wait complete. 2024-01-24T17:48:17.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_session_drop' 2024-01-24T17:48:18.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.042+0000 7fd1c0c00640 1 Processor -- start 2024-01-24T17:48:18.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.042+0000 7fd1c0c00640 1 -- start start 2024-01-24T17:48:18.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.043+0000 7fd1c0c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd1bc0cadc0 0x7fd1bc15a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.043+0000 7fd1c0c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1bc06bd40 0x7fd1bc0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.044+0000 7fd1c0c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0cac10 con 0x7fd1bc0cb1a0 2024-01-24T17:48:18.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.044+0000 7fd1c0c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc15b040 con 0x7fd1bc06bd40 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.044+0000 7fd1c0c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc06c1a0 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.044+0000 7fd1baa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1bc06bd40 0x7fd1bc0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.044+0000 7fd1baa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1bc06bd40 0x7fd1bc0ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52204/0 (socket says 172.21.15.39:52204) 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.044+0000 7fd1baa00640 1 -- 172.21.15.39:0/1079097321 learned_addr learned my addr 172.21.15.39:0/1079097321 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.044+0000 7fd1bb400640 1 --2- 172.21.15.39:0/1079097321 >> v2:172.21.15.100:3300/0 conn(0x7fd1bc0cadc0 0x7fd1bc15a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.045+0000 7fd1baa00640 1 -- 172.21.15.39:0/1079097321 >> v2:172.21.15.100:3300/0 conn(0x7fd1bc0cadc0 msgr2=0x7fd1bc15a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.045+0000 7fd1baa00640 1 --2- 172.21.15.39:0/1079097321 >> v2:172.21.15.100:3300/0 conn(0x7fd1bc0cadc0 0x7fd1bc15a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.045+0000 7fd1baa00640 1 -- 172.21.15.39:0/1079097321 >> v1:172.21.15.100:6789/0 conn(0x7fd1bc0cb1a0 legacy=0x7fd1bc15b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:18.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.045+0000 7fd1baa00640 1 -- 172.21.15.39:0/1079097321 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc15bbe0 con 0x7fd1bc06bd40 2024-01-24T17:48:18.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.046+0000 7fd1baa00640 1 --2- 172.21.15.39:0/1079097321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc06bd40 0x7fd1bc0ca6d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0cbaf0 tx=0x7fd1a402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b04e25ecf427704 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:18.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.046+0000 7fd1ba000640 1 -- 172.21.15.39:0/1079097321 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a40317e0 con 0x7fd1bc06bd40 2024-01-24T17:48:18.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.046+0000 7fd1ba000640 1 -- 172.21.15.39:0/1079097321 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4031940 con 0x7fd1bc06bd40 2024-01-24T17:48:18.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.046+0000 7fd1ba000640 1 -- 172.21.15.39:0/1079097321 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4031cd0 con 0x7fd1bc06bd40 2024-01-24T17:48:18.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.047+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1079097321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc06bd40 msgr2=0x7fd1bc0ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.047+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1079097321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc06bd40 0x7fd1bc0ca6d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0cbaf0 tx=0x7fd1a402b880 comp rx=0 tx=0).stop 2024-01-24T17:48:18.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.047+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1079097321 shutdown_connections 2024-01-24T17:48:18.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.047+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1079097321 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc06bd40 0x7fd1bc0ca6d0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.047+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1079097321 >> v2:172.21.15.100:3300/0 conn(0x7fd1bc0cadc0 0x7fd1bc15a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.047+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1079097321 >> 172.21.15.39:0/1079097321 conn(0x7fd1bc06a200 msgr2=0x7fd1bc0aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:18.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.047+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1079097321 shutdown_connections 2024-01-24T17:48:18.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.047+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1079097321 wait complete. 2024-01-24T17:48:18.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.049+0000 7fd1c0c00640 1 Processor -- start 2024-01-24T17:48:18.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.050+0000 7fd1c0c00640 1 -- start start 2024-01-24T17:48:18.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1c0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1bc06bd40 0x7fd1bc156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1c0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc0cadc0 0x7fd1bc146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1c0c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1bc0cb1a0 0x7fd1bc0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1c0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0c7e80 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1c0c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0b3720 con 0x7fd1bc0cb1a0 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1c0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0b39b0 con 0x7fd1bc06bd40 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1baa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc0cadc0 0x7fd1bc146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1baa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc0cadc0 0x7fd1bc146150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52206/0 (socket says 172.21.15.39:52206) 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.051+0000 7fd1baa00640 1 -- 172.21.15.39:0/1103314565 learned_addr learned my addr 172.21.15.39:0/1103314565 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.052+0000 7fd1bb400640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1bc06bd40 0x7fd1bc156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.052+0000 7fd1baa00640 1 -- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1bc06bd40 msgr2=0x7fd1bc156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.052+0000 7fd1baa00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1bc06bd40 0x7fd1bc156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.052+0000 7fd1baa00640 1 -- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1bc0cb1a0 msgr2=0x7fd1bc0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.052+0000 7fd1bbe00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1bc0cb1a0 0x7fd1bc0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.052+0000 7fd1baa00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1bc0cb1a0 0x7fd1bc0c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.052+0000 7fd1baa00640 1 -- 172.21.15.39:0/1103314565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40094d0 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.052+0000 7fd1baa00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc0cadc0 0x7fd1bc146150 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fd1a402b8c0 tx=0x7fd1a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.053+0000 7fd1b3400640 1 -- 172.21.15.39:0/1103314565 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4032880 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.053+0000 7fd1b3400640 1 -- 172.21.15.39:0/1103314565 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a40329e0 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.053+0000 7fd1b3400640 1 -- 172.21.15.39:0/1103314565 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4032d90 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.053+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc0b3c90 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.053+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc146b90 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.054+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc0716e0 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.055+0000 7fd1b3400640 1 -- 172.21.15.39:0/1103314565 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd1a40453f0 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.056+0000 7fd1b3400640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd19406a430 0x7fd19406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.056+0000 7fd1b3400640 1 -- 172.21.15.39:0/1103314565 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd1a40aa7c0 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.057+0000 7fd1bb400640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd19406a430 0x7fd19406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.058+0000 7fd1bb400640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd19406a430 0x7fd19406c8f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009f80 tx=0x7fd1ac00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:18.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.060+0000 7fd1b3400640 1 -- 172.21.15.39:0/1103314565 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd1a4076130 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.331+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_session_drop"]} v 0) v1 -- 0x7fd1bc04dff0 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.349+0000 7fd1b3400640 1 -- 172.21.15.39:0/1103314565 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_tell_session_drop"]}]=0 v356) v1 ==== 145+0+0 (secure 0 0 0) 0x7fd1a4079750 con 0x7fd1bc0cadc0 2024-01-24T17:48:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd19406a430 msgr2=0x7fd19406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd19406a430 0x7fd19406c8f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009f80 tx=0x7fd1ac00b040 comp rx=0 tx=0).stop 2024-01-24T17:48:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc0cadc0 msgr2=0x7fd1bc146150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc0cadc0 0x7fd1bc146150 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fd1a402b8c0 tx=0x7fd1a40046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 shutdown_connections 2024-01-24T17:48:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd1bc0cb1a0 0x7fd1bc0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd19406a430 0x7fd19406c8f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1bc0cadc0 0x7fd1bc146150 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 --2- 172.21.15.39:0/1103314565 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1bc06bd40 0x7fd1bc156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 >> 172.21.15.39:0/1103314565 conn(0x7fd1bc06a200 msgr2=0x7fd1bc06fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 shutdown_connections 2024-01-24T17:48:18.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.352+0000 7fd1c0c00640 1 -- 172.21.15.39:0/1103314565 wait complete. 2024-01-24T17:48:18.374 INFO:tasks.cephfs_test_runner:That when a `tell` command is sent using the python CLI, ... ok 2024-01-24T17:48:18.374 INFO:tasks.cephfs_test_runner:Starting test: test_version_splitting (tasks.cephfs.test_sessionmap.TestSessionMap) That when many sessions are updated, they are correctly 2024-01-24T17:48:18.374 INFO:tasks.cephfs_test_runner:test_version_splitting (tasks.cephfs.test_sessionmap.TestSessionMap) 2024-01-24T17:48:18.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_version_splitting' 2024-01-24T17:48:18.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.604+0000 7fc533000640 1 Processor -- start 2024-01-24T17:48:18.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.604+0000 7fc533000640 1 -- start start 2024-01-24T17:48:18.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.606+0000 7fc533000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc52c0afdf0 0x7fc52c0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.607+0000 7fc533000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c165ef0 con 0x7fc52c0afa10 2024-01-24T17:48:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.607+0000 7fc52be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc52c0afa10 0x7fc52c0cbe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35224/0 (socket says 172.21.15.39:35224) 2024-01-24T17:48:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.607+0000 7fc52be00640 1 -- 172.21.15.39:0/2666028335 learned_addr learned my addr 172.21.15.39:0/2666028335 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.607+0000 7fc533000640 1 -- 172.21.15.39:0/2666028335 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c0cf650 con 0x7fc52c0cf220 2024-01-24T17:48:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.607+0000 7fc533000640 1 -- 172.21.15.39:0/2666028335 --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52c0ce6a0 con 0x7fc52c0afdf0 2024-01-24T17:48:18.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.607+0000 7fc52b400640 1 --2- 172.21.15.39:0/2666028335 >> v2:172.21.15.100:3300/0 conn(0x7fc52c0afdf0 0x7fc52c0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.608+0000 7fc52aa00640 1 -- 172.21.15.39:0/2666028335 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320803496 0 0) 0x7fc52c165ef0 con 0x7fc52c0afa10 2024-01-24T17:48:18.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.608+0000 7fc52aa00640 1 -- 172.21.15.39:0/2666028335 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50c003520 con 0x7fc52c0afa10 2024-01-24T17:48:18.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.608+0000 7fc52b400640 1 -- 172.21.15.39:0/2666028335 >> v1:172.21.15.100:6789/0 conn(0x7fc52c0cf220 legacy=0x7fc52c165030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.608+0000 7fc52b400640 1 -- 172.21.15.39:0/2666028335 >> v1:172.21.15.39:6789/0 conn(0x7fc52c0afa10 legacy=0x7fc52c0cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.608+0000 7fc52b400640 1 -- 172.21.15.39:0/2666028335 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c165c10 con 0x7fc52c0afdf0 2024-01-24T17:48:18.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.610+0000 7fc52b400640 1 --2- 172.21.15.39:0/2666028335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 0x7fc52c0cece0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc5140097a0 tx=0x7fc51402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a648744324729c1f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.611+0000 7fc52aa00640 1 -- 172.21.15.39:0/2666028335 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514030610 con 0x7fc52c0afdf0 2024-01-24T17:48:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.611+0000 7fc52aa00640 1 -- 172.21.15.39:0/2666028335 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc514030770 con 0x7fc52c0afdf0 2024-01-24T17:48:18.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.611+0000 7fc52aa00640 1 -- 172.21.15.39:0/2666028335 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc514030b00 con 0x7fc52c0afdf0 2024-01-24T17:48:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.611+0000 7fc533000640 1 -- 172.21.15.39:0/2666028335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 msgr2=0x7fc52c0cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.611+0000 7fc533000640 1 --2- 172.21.15.39:0/2666028335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 0x7fc52c0cece0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc5140097a0 tx=0x7fc51402b960 comp rx=0 tx=0).stop 2024-01-24T17:48:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.612+0000 7fc533000640 1 -- 172.21.15.39:0/2666028335 shutdown_connections 2024-01-24T17:48:18.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.612+0000 7fc533000640 1 --2- 172.21.15.39:0/2666028335 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 0x7fc52c0cece0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.612+0000 7fc533000640 1 -- 172.21.15.39:0/2666028335 >> 172.21.15.39:0/2666028335 conn(0x7fc52c0cd2b0 msgr2=0x7fc52c0cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.612+0000 7fc533000640 1 -- 172.21.15.39:0/2666028335 shutdown_connections 2024-01-24T17:48:18.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.612+0000 7fc533000640 1 -- 172.21.15.39:0/2666028335 wait complete. 2024-01-24T17:48:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.614+0000 7fc533000640 1 Processor -- start 2024-01-24T17:48:18.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.614+0000 7fc533000640 1 -- start start 2024-01-24T17:48:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc533000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52c0afa10 0x7fc52c0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc533000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 0x7fc52c15ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc533000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc52c0cf220 0x7fc52c0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc533000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0c85e0 con 0x7fc52c0cf220 2024-01-24T17:48:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc533000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0c79a0 con 0x7fc52c0afdf0 2024-01-24T17:48:18.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc533000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0c72b0 con 0x7fc52c0afa10 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc531000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc52c0cf220 0x7fc52c0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc531000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc52c0cf220 0x7fc52c0c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52218/0 (socket says 172.21.15.39:52218) 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc531000640 1 -- 172.21.15.39:0/3778552723 learned_addr learned my addr 172.21.15.39:0/3778552723 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc52b400640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 0x7fc52c15ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc52be00640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52c0afa10 0x7fc52c0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc531000640 1 -- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52c0afa10 msgr2=0x7fc52c0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.615+0000 7fc531000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52c0afa10 0x7fc52c0c6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc531000640 1 -- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 msgr2=0x7fc52c15ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc531000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 0x7fc52c15ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc531000640 1 -- 172.21.15.39:0/3778552723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc514009400 con 0x7fc52c0cf220 2024-01-24T17:48:18.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc531000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc52c0cf220 0x7fc52c0c8a00 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00c390 tx=0x7fc51c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc523e00640 1 -- 172.21.15.39:0/3778552723 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c0052d0 con 0x7fc52c0cf220 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc523e00640 1 -- 172.21.15.39:0/3778552723 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c005430 con 0x7fc52c0cf220 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc523e00640 1 -- 172.21.15.39:0/3778552723 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c003810 con 0x7fc52c0cf220 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52c0c7c40 con 0x7fc52c0cf220 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.616+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52c0c8200 con 0x7fc52c0cf220 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.618+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52c0c8720 con 0x7fc52c0cf220 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.618+0000 7fc523e00640 1 -- 172.21.15.39:0/3778552723 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc51c003970 con 0x7fc52c0cf220 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.619+0000 7fc523e00640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50406a430 0x7fc50406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:18.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.619+0000 7fc523e00640 1 -- 172.21.15.39:0/3778552723 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc51c087aa0 con 0x7fc52c0cf220 2024-01-24T17:48:18.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.620+0000 7fc52be00640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50406a430 0x7fc50406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:18.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.621+0000 7fc52be00640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50406a430 0x7fc50406c8f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc524001560 tx=0x7fc524009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:18.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.624+0000 7fc523e00640 1 -- 172.21.15.39:0/3778552723 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc51c053420 con 0x7fc52c0cf220 2024-01-24T17:48:18.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:18.889+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_version_splitting"]} v 0) v1 -- 0x7fc52c067050 con 0x7fc52c0cf220 2024-01-24T17:48:19.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.355+0000 7fc523e00640 1 -- 172.21.15.39:0/3778552723 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_version_splitting"]}]=0 v357) v1 ==== 148+0+0 (secure 0 0 0) 0x7fc51c056a40 con 0x7fc52c0cf220 2024-01-24T17:48:19.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.358+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50406a430 msgr2=0x7fc50406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:19.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.358+0000 7fc533000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50406a430 0x7fc50406c8f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc524001560 tx=0x7fc524009040 comp rx=0 tx=0).stop 2024-01-24T17:48:19.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.358+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc52c0cf220 msgr2=0x7fc52c0c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:19.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.358+0000 7fc533000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc52c0cf220 0x7fc52c0c8a00 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00c390 tx=0x7fc51c00c850 comp rx=0 tx=0).stop 2024-01-24T17:48:19.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.359+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 shutdown_connections 2024-01-24T17:48:19.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.359+0000 7fc533000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc50406a430 0x7fc50406c8f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.359+0000 7fc533000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc52c0cf220 0x7fc52c0c8a00 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.359+0000 7fc533000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc52c0afdf0 0x7fc52c15ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.359+0000 7fc533000640 1 --2- 172.21.15.39:0/3778552723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc52c0afa10 0x7fc52c0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.359+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 >> 172.21.15.39:0/3778552723 conn(0x7fc52c0cd2b0 msgr2=0x7fc52c0cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:19.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.359+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 shutdown_connections 2024-01-24T17:48:19.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.359+0000 7fc533000640 1 -- 172.21.15.39:0/3778552723 wait complete. 2024-01-24T17:48:19.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-24T17:48:19.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.598+0000 7f49c5200640 1 Processor -- start 2024-01-24T17:48:19.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.598+0000 7f49c5200640 1 -- start start 2024-01-24T17:48:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.600+0000 7f49c5200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49c0158e30 0x7f49c00ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.600+0000 7f49c5200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f49c00ada90 0x7f49c00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.600+0000 7f49c5200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00705b0 con 0x7f49c00cd2c0 2024-01-24T17:48:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.600+0000 7f49c5200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c00cbc50 con 0x7f49c0158e30 2024-01-24T17:48:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.600+0000 7f49c5200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c00cb970 con 0x7f49c00ada90 2024-01-24T17:48:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.600+0000 7f49bf400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49c0158e30 0x7f49c00ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.600+0000 7f49bf400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49c0158e30 0x7f49c00ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52226/0 (socket says 172.21.15.39:52226) 2024-01-24T17:48:19.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.600+0000 7f49bf400640 1 -- 172.21.15.39:0/1512605454 learned_addr learned my addr 172.21.15.39:0/1512605454 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:19.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.601+0000 7f49b7e00640 1 --2- 172.21.15.39:0/1512605454 >> v2:172.21.15.100:3300/0 conn(0x7f49c00ada90 0x7f49c00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:19.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.601+0000 7f49bf400640 1 -- 172.21.15.39:0/1512605454 >> v2:172.21.15.100:3300/0 conn(0x7f49c00ada90 msgr2=0x7f49c00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:19.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.601+0000 7f49bf400640 1 --2- 172.21.15.39:0/1512605454 >> v2:172.21.15.100:3300/0 conn(0x7f49c00ada90 0x7f49c00712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.601+0000 7f49bf400640 1 -- 172.21.15.39:0/1512605454 >> v1:172.21.15.100:6789/0 conn(0x7f49c00cd2c0 legacy=0x7f49c015c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:19.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.601+0000 7f49bf400640 1 -- 172.21.15.39:0/1512605454 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00ca220 con 0x7f49c0158e30 2024-01-24T17:48:19.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.601+0000 7f49b7e00640 1 --2- 172.21.15.39:0/1512605454 >> v2:172.21.15.100:3300/0 conn(0x7f49c00ada90 0x7f49c00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.602+0000 7f49bf400640 1 --2- 172.21.15.39:0/1512605454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c0158e30 0x7f49c00ccd80 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f49c00b4510 tx=0x7f49b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ab86be358c3aa6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49bea00640 1 -- 172.21.15.39:0/1512605454 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b0031870 con 0x7f49c0158e30 2024-01-24T17:48:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49bea00640 1 -- 172.21.15.39:0/1512605454 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b00319d0 con 0x7f49c0158e30 2024-01-24T17:48:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49bea00640 1 -- 172.21.15.39:0/1512605454 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b0031d60 con 0x7f49c0158e30 2024-01-24T17:48:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49c5200640 1 -- 172.21.15.39:0/1512605454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c0158e30 msgr2=0x7f49c00ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49c5200640 1 --2- 172.21.15.39:0/1512605454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c0158e30 0x7f49c00ccd80 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f49c00b4510 tx=0x7f49b002b910 comp rx=0 tx=0).stop 2024-01-24T17:48:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49c5200640 1 -- 172.21.15.39:0/1512605454 shutdown_connections 2024-01-24T17:48:19.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49c5200640 1 --2- 172.21.15.39:0/1512605454 >> v2:172.21.15.100:3300/0 conn(0x7f49c00ada90 0x7f49c00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49c5200640 1 --2- 172.21.15.39:0/1512605454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c0158e30 0x7f49c00ccd80 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49c5200640 1 -- 172.21.15.39:0/1512605454 >> 172.21.15.39:0/1512605454 conn(0x7f49c006a200 msgr2=0x7f49c00affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:19.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.603+0000 7f49c5200640 1 -- 172.21.15.39:0/1512605454 shutdown_connections 2024-01-24T17:48:19.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.604+0000 7f49c5200640 1 -- 172.21.15.39:0/1512605454 wait complete. 2024-01-24T17:48:19.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.605+0000 7f49c5200640 1 Processor -- start 2024-01-24T17:48:19.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.605+0000 7f49c5200640 1 -- start start 2024-01-24T17:48:19.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.606+0000 7f49c5200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49c00ada90 0x7f49c00c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:19.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.606+0000 7f49c5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c00cd2c0 0x7f49c01526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:19.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.606+0000 7f49c5200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49c0158e30 0x7f49c0153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.606+0000 7f49c5200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c01f6340 con 0x7f49c00cd2c0 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.606+0000 7f49c5200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c01f67d0 con 0x7f49c0158e30 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.606+0000 7f49c5200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00c83d0 con 0x7f49c00ada90 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49b7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c00cd2c0 0x7f49c01526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49b7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c00cd2c0 0x7f49c01526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52242/0 (socket says 172.21.15.39:52242) 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49b7e00640 1 -- 172.21.15.39:0/2339800861 learned_addr learned my addr 172.21.15.39:0/2339800861 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49bf400640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49c00ada90 0x7f49c00c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49bfe00640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49c0158e30 0x7f49c0153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49b7e00640 1 -- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49c00ada90 msgr2=0x7f49c00c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49b7e00640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49c00ada90 0x7f49c00c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49b7e00640 1 -- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49c0158e30 msgr2=0x7f49c0153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49b7e00640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49c0158e30 0x7f49c0153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49b7e00640 1 -- 172.21.15.39:0/2339800861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00094d0 con 0x7f49c00cd2c0 2024-01-24T17:48:19.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.607+0000 7f49bfe00640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49c0158e30 0x7f49c0153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:19.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.608+0000 7f49b7e00640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c00cd2c0 0x7f49c01526c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c7f0 tx=0x7f49a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:19.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.608+0000 7f49b7400640 1 -- 172.21.15.39:0/2339800861 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8007a30 con 0x7f49c00cd2c0 2024-01-24T17:48:19.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.608+0000 7f49b7400640 1 -- 172.21.15.39:0/2339800861 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a8007b90 con 0x7f49c00cd2c0 2024-01-24T17:48:19.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.608+0000 7f49b7400640 1 -- 172.21.15.39:0/2339800861 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a800ce40 con 0x7f49c00cd2c0 2024-01-24T17:48:19.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.608+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c01f5a50 con 0x7f49c00cd2c0 2024-01-24T17:48:19.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.608+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c01f5fd0 con 0x7f49c00cd2c0 2024-01-24T17:48:19.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.609+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c00738c0 con 0x7f49c00cd2c0 2024-01-24T17:48:19.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.610+0000 7f49b7400640 1 -- 172.21.15.39:0/2339800861 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f49a80113f0 con 0x7f49c00cd2c0 2024-01-24T17:48:19.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.611+0000 7f49b7400640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f499806a430 0x7f499806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:19.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.611+0000 7f49b7400640 1 -- 172.21.15.39:0/2339800861 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f49a8088770 con 0x7f49c00cd2c0 2024-01-24T17:48:19.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.611+0000 7f49bf400640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f499806a430 0x7f499806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:19.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.613+0000 7f49bf400640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f499806a430 0x7f499806c8f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f49b0036000 tx=0x7f49b003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:19.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.616+0000 7f49b7400640 1 -- 172.21.15.39:0/2339800861 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f49a80540d0 con 0x7f49c00cd2c0 2024-01-24T17:48:19.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.810+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f49c004e840 con 0x7f49c00cd2c0 2024-01-24T17:48:19.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.819+0000 7f49b7400640 1 -- 172.21.15.39:0/2339800861 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f49a80576f0 con 0x7f49c00cd2c0 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.822+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f499806a430 msgr2=0x7f499806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.822+0000 7f49c5200640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f499806a430 0x7f499806c8f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f49b0036000 tx=0x7f49b003c040 comp rx=0 tx=0).stop 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.822+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c00cd2c0 msgr2=0x7f49c01526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.822+0000 7f49c5200640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c00cd2c0 0x7f49c01526c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c7f0 tx=0x7f49a800cbf0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.822+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 shutdown_connections 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.822+0000 7f49c5200640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49c0158e30 0x7f49c0153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.822+0000 7f49c5200640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f499806a430 0x7f499806c8f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.823+0000 7f49c5200640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49c00cd2c0 0x7f49c01526c0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.823+0000 7f49c5200640 1 --2- 172.21.15.39:0/2339800861 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49c00ada90 0x7f49c00c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.823+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 >> 172.21.15.39:0/2339800861 conn(0x7f49c006a200 msgr2=0x7f49c00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.823+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 shutdown_connections 2024-01-24T17:48:19.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:19.823+0000 7f49c5200640 1 -- 172.21.15.39:0/2339800861 wait complete. 2024-01-24T17:48:19.841 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:48:19.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:48:19.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:48:20.084 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:48:20.084 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:48:20.084 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:20.144 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:48:20.145 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:48:20.145 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:20.202 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:48:20.202 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:48:20.203 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:20.262 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:48:20.263 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:48:20.263 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:20.318 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:48:20.318 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-24T17:48:20.318 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:20.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.537+0000 7f4e13000640 1 Processor -- start 2024-01-24T17:48:20.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.537+0000 7f4e13000640 1 -- start start 2024-01-24T17:48:20.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.538+0000 7f4e13000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e0c15cbe0 0x7f4e0c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:20.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.538+0000 7f4e13000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0b8820 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.538+0000 7f4e13000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0b5790 con 0x7f4e0c0ce510 2024-01-24T17:48:20.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.538+0000 7f4e13000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0c0b5a50 con 0x7f4e0c15cbe0 2024-01-24T17:48:20.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.539+0000 7f4e0be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e0c15cbe0 0x7f4e0c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:20.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.539+0000 7f4e0be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4e0c15cbe0 0x7f4e0c0cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52244/0 (socket says 172.21.15.39:52244) 2024-01-24T17:48:20.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.539+0000 7f4e0be00640 1 -- 172.21.15.39:0/1630941019 learned_addr learned my addr 172.21.15.39:0/1630941019 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:20.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.539+0000 7f4e0be00640 1 -- 172.21.15.39:0/1630941019 >> v1:172.21.15.100:6789/0 conn(0x7f4e0c0ce510 legacy=0x7f4e0c0d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:20.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.539+0000 7f4e0be00640 1 -- 172.21.15.39:0/1630941019 >> v1:172.21.15.39:6789/0 conn(0x7f4e0c0d15a0 legacy=0x7f4e0c166b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:20.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.539+0000 7f4e0be00640 1 -- 172.21.15.39:0/1630941019 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c0b5d30 con 0x7f4e0c15cbe0 2024-01-24T17:48:20.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.540+0000 7f4e0be00640 1 --2- 172.21.15.39:0/1630941019 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c15cbe0 0x7f4e0c0cdfd0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009ea0 tx=0x7f4dfc02bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36457d253f21184f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:20.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e0aa00640 1 -- 172.21.15.39:0/1630941019 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc02cb50 con 0x7f4e0c15cbe0 2024-01-24T17:48:20.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e0aa00640 1 -- 172.21.15.39:0/1630941019 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc02ccb0 con 0x7f4e0c15cbe0 2024-01-24T17:48:20.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e0aa00640 1 -- 172.21.15.39:0/1630941019 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc034610 con 0x7f4e0c15cbe0 2024-01-24T17:48:20.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e13000640 1 -- 172.21.15.39:0/1630941019 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c15cbe0 msgr2=0x7f4e0c0cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:20.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e13000640 1 --2- 172.21.15.39:0/1630941019 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c15cbe0 0x7f4e0c0cdfd0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009ea0 tx=0x7f4dfc02bfb0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e13000640 1 -- 172.21.15.39:0/1630941019 shutdown_connections 2024-01-24T17:48:20.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e13000640 1 --2- 172.21.15.39:0/1630941019 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c15cbe0 0x7f4e0c0cdfd0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e13000640 1 -- 172.21.15.39:0/1630941019 >> 172.21.15.39:0/1630941019 conn(0x7f4e0c09bc80 msgr2=0x7f4e0c0d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:20.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e13000640 1 -- 172.21.15.39:0/1630941019 shutdown_connections 2024-01-24T17:48:20.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.541+0000 7f4e13000640 1 -- 172.21.15.39:0/1630941019 wait complete. 2024-01-24T17:48:20.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.543+0000 7f4e13000640 1 Processor -- start 2024-01-24T17:48:20.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.543+0000 7f4e13000640 1 -- start start 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.544+0000 7f4e13000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e0c0ce510 0x7f4e0c1e6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.544+0000 7f4e13000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c0d15a0 0x7f4e0c15fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.544+0000 7f4e13000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e0c15cbe0 0x7f4e0c1e70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.544+0000 7f4e13000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c1e8960 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.544+0000 7f4e13000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c15dd30 con 0x7f4e0c15cbe0 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.544+0000 7f4e13000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c15d140 con 0x7f4e0c0ce510 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c0d15a0 0x7f4e0c15fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c0d15a0 0x7f4e0c15fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52250/0 (socket says 172.21.15.39:52250) 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 -- 172.21.15.39:0/1832562810 learned_addr learned my addr 172.21.15.39:0/1832562810 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:20.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e11000640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e0c15cbe0 0x7f4e0c1e70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:20.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 -- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e0c0ce510 msgr2=0x7f4e0c1e6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:20.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e0c0ce510 0x7f4e0c1e6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 -- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e0c15cbe0 msgr2=0x7f4e0c1e70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:20.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e0c15cbe0 0x7f4e0c1e70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 -- 172.21.15.39:0/1832562810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc009b30 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.545+0000 7f4e0b400640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c0d15a0 0x7f4e0c15fc70 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f4df400b3e0 tx=0x7f4df400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:20.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.546+0000 7f4e03e00640 1 -- 172.21.15.39:0/1832562810 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4011830 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.546+0000 7f4e03e00640 1 -- 172.21.15.39:0/1832562810 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df40119b0 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.546+0000 7f4e03e00640 1 -- 172.21.15.39:0/1832562810 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4011d60 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.546+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c15d400 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.546+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c15d9c0 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.547+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0c15db50 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.548+0000 7f4e03e00640 1 -- 172.21.15.39:0/1832562810 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4df401a400 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.549+0000 7f4e03e00640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4de406a430 0x7f4de406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:20.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.549+0000 7f4e03e00640 1 -- 172.21.15.39:0/1832562810 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4df40887d0 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.550+0000 7f4e0be00640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4de406a430 0x7f4de406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:20.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.551+0000 7f4e0be00640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4de406a430 0x7f4de406c8f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc035000 tx=0x7f4dfc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:20.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.553+0000 7f4e03e00640 1 -- 172.21.15.39:0/1832562810 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4df40540b0 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.788+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e0c0b5fe0 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.789+0000 7f4e03e00640 1 -- 172.21.15.39:0/1832562810 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+3529 (secure 0 0 0) 0x7f4df4015090 con 0x7f4e0c0d15a0 2024-01-24T17:48:20.790 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:20.790 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/57362003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":57362003},{"type":"v1","addr":"172.21.15.100:6836","nonce":57362003}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":6139,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/3662299631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":3662299631},{"type":"v1","addr":"172.21.15.100:6837","nonce":3662299631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/2829786836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2829786836},{"type":"v1","addr":"172.21.15.39:6833","nonce":2829786836}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":45},{"gid":7686,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2871246995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2871246995},{"type":"v1","addr":"172.21.15.39:6835","nonce":2871246995}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":53}],"filesystems":[]} 2024-01-24T17:48:20.790 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 53 2024-01-24T17:48:20.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4de406a430 msgr2=0x7f4de406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:20.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4de406a430 0x7f4de406c8f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc035000 tx=0x7f4dfc02e000 comp rx=0 tx=0).stop 2024-01-24T17:48:20.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c0d15a0 msgr2=0x7f4e0c15fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:20.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c0d15a0 0x7f4e0c15fc70 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f4df400b3e0 tx=0x7f4df400b8a0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 shutdown_connections 2024-01-24T17:48:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4e0c15cbe0 0x7f4e0c1e70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4de406a430 0x7f4de406c8f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e0c0d15a0 0x7f4e0c15fc70 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 --2- 172.21.15.39:0/1832562810 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4e0c0ce510 0x7f4e0c1e6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 >> 172.21.15.39:0/1832562810 conn(0x7f4e0c09bc80 msgr2=0x7f4e0c06bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.792+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 shutdown_connections 2024-01-24T17:48:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:20.793+0000 7f4e13000640 1 -- 172.21.15.39:0/1832562810 wait complete. 2024-01-24T17:48:20.812 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-24T17:48:20.812 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-24T17:48:20.812 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-24T17:48:20.812 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:20.813 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-24T17:48:20.813 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-24T17:48:20.813 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-24T17:48:20.813 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:20.813 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-24T17:48:20.813 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-24T17:48:20.813 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-24T17:48:20.814 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:20.814 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-24T17:48:20.814 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-24T17:48:20.814 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-24T17:48:20.814 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:20.830 INFO:tasks.ceph.mds.c:Stopped 2024-01-24T17:48:20.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-24T17:48:20.833 INFO:tasks.ceph.mds.d:Stopped 2024-01-24T17:48:20.833 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-24T17:48:20.834 INFO:tasks.ceph.mds.a:Stopped 2024-01-24T17:48:20.834 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-24T17:48:20.842 INFO:tasks.ceph.mds.b:Stopped 2024-01-24T17:48:20.842 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-24T17:48:20.881 INFO:tasks.ceph.mds.c:Started 2024-01-24T17:48:20.881 DEBUG:teuthology.parallel:result is None 2024-01-24T17:48:20.887 INFO:tasks.ceph.mds.a:Started 2024-01-24T17:48:20.887 DEBUG:teuthology.parallel:result is None 2024-01-24T17:48:20.889 INFO:tasks.ceph.mds.d:Started 2024-01-24T17:48:20.889 DEBUG:teuthology.parallel:result is None 2024-01-24T17:48:20.895 INFO:tasks.ceph.mds.b:Started 2024-01-24T17:48:20.895 DEBUG:teuthology.parallel:result is None 2024-01-24T17:48:20.896 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-24T17:48:21.043 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:48:21.046 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:48:21.046 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:48:21.062 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:48:21.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.089+0000 7f93f5200640 1 Processor -- start 2024-01-24T17:48:21.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.089+0000 7f93f5200640 1 -- start start 2024-01-24T17:48:21.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.090+0000 7f93f5200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f93f00d0e90 0x7f93f00d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:21.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.090+0000 7f93f5200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f93f0164b40 0x7f93f0166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:21.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.090+0000 7f93f5200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f00d1820 con 0x7f93f015cbe0 2024-01-24T17:48:21.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.090+0000 7f93f5200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f00b8820 con 0x7f93f0164b40 2024-01-24T17:48:21.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.090+0000 7f93f5200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f00b5790 con 0x7f93f00d0e90 2024-01-24T17:48:21.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.090+0000 7f93ede00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f93f015cbe0 0x7f93f00cdfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35266/0 (socket says 172.21.15.39:35266) 2024-01-24T17:48:21.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.090+0000 7f93ede00640 1 -- 172.21.15.39:0/3031022242 learned_addr learned my addr 172.21.15.39:0/3031022242 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:21.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.091+0000 7f93ed400640 1 --2- 172.21.15.39:0/3031022242 >> v2:172.21.15.100:3300/0 conn(0x7f93f00d0e90 0x7f93f00d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:21.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.091+0000 7f93ee800640 1 --2- 172.21.15.39:0/3031022242 >> v2:172.21.15.39:3300/0 conn(0x7f93f0164b40 0x7f93f0166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:21.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.091+0000 7f93eca00640 1 -- 172.21.15.39:0/3031022242 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121675110 0 0) 0x7f93e0001290 con 0x7f93f015cbe0 2024-01-24T17:48:21.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.091+0000 7f93eca00640 1 -- 172.21.15.39:0/3031022242 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d0003520 con 0x7f93f015cbe0 2024-01-24T17:48:21.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.091+0000 7f93ed400640 1 -- 172.21.15.39:0/3031022242 >> v2:172.21.15.39:3300/0 conn(0x7f93f0164b40 msgr2=0x7f93f0166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:21.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.091+0000 7f93ed400640 1 --2- 172.21.15.39:0/3031022242 >> v2:172.21.15.39:3300/0 conn(0x7f93f0164b40 0x7f93f0166f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:21.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.091+0000 7f93ed400640 1 -- 172.21.15.39:0/3031022242 >> v1:172.21.15.39:6789/0 conn(0x7f93f015cbe0 legacy=0x7f93f00cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:21.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.091+0000 7f93ed400640 1 -- 172.21.15.39:0/3031022242 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f00b5a50 con 0x7f93f00d0e90 2024-01-24T17:48:21.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.093+0000 7f93ed400640 1 --2- 172.21.15.39:0/3031022242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 0x7f93f00d12e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f93f008bdb0 tx=0x7f93d802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a554633e79306258 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:21.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.093+0000 7f93eca00640 1 -- 172.21.15.39:0/3031022242 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8031830 con 0x7f93f00d0e90 2024-01-24T17:48:21.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.093+0000 7f93eca00640 1 -- 172.21.15.39:0/3031022242 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d8031990 con 0x7f93f00d0e90 2024-01-24T17:48:21.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.093+0000 7f93eca00640 1 -- 172.21.15.39:0/3031022242 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8031d20 con 0x7f93f00d0e90 2024-01-24T17:48:21.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.093+0000 7f93f5200640 1 -- 172.21.15.39:0/3031022242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 msgr2=0x7f93f00d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:21.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.093+0000 7f93f5200640 1 --2- 172.21.15.39:0/3031022242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 0x7f93f00d12e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f93f008bdb0 tx=0x7f93d802b8d0 comp rx=0 tx=0).stop 2024-01-24T17:48:21.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.094+0000 7f93f5200640 1 -- 172.21.15.39:0/3031022242 shutdown_connections 2024-01-24T17:48:21.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.094+0000 7f93f5200640 1 --2- 172.21.15.39:0/3031022242 >> v2:172.21.15.39:3300/0 conn(0x7f93f0164b40 0x7f93f0166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:21.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.094+0000 7f93f5200640 1 --2- 172.21.15.39:0/3031022242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 0x7f93f00d12e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:21.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.094+0000 7f93f5200640 1 -- 172.21.15.39:0/3031022242 >> 172.21.15.39:0/3031022242 conn(0x7f93f009bc80 msgr2=0x7f93f00d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:21.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.094+0000 7f93f5200640 1 -- 172.21.15.39:0/3031022242 shutdown_connections 2024-01-24T17:48:21.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.094+0000 7f93f5200640 1 -- 172.21.15.39:0/3031022242 wait complete. 2024-01-24T17:48:21.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.095+0000 7f93f5200640 1 Processor -- start 2024-01-24T17:48:21.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.096+0000 7f93f5200640 1 -- start start 2024-01-24T17:48:21.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.096+0000 7f93f5200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 0x7f93f014a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:21.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.096+0000 7f93f5200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93f015cbe0 0x7f93f00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:21.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.096+0000 7f93f5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93f0164b40 0x7f93f00c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:21.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.096+0000 7f93f5200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f01552c0 con 0x7f93f0164b40 2024-01-24T17:48:21.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.096+0000 7f93f5200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f01518c0 con 0x7f93f00d0e90 2024-01-24T17:48:21.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.096+0000 7f93f5200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f00cc830 con 0x7f93f015cbe0 2024-01-24T17:48:21.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ee800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93f0164b40 0x7f93f00c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:21.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ee800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93f0164b40 0x7f93f00c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52306/0 (socket says 172.21.15.39:52306) 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ee800640 1 -- 172.21.15.39:0/3654758176 learned_addr learned my addr 172.21.15.39:0/3654758176 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ede00640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 0x7f93f014a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ed400640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93f015cbe0 0x7f93f00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ee800640 1 -- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93f015cbe0 msgr2=0x7f93f00cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ee800640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93f015cbe0 0x7f93f00cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ee800640 1 -- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 msgr2=0x7f93f014a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ee800640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 0x7f93f014a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ee800640 1 -- 172.21.15.39:0/3654758176 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8009400 con 0x7f93f0164b40 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.097+0000 7f93ede00640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 0x7f93f014a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.098+0000 7f93ee800640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93f0164b40 0x7f93f00c7d20 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009d30 tx=0x7f93e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.098+0000 7f93e6000640 1 -- 172.21.15.39:0/3654758176 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e8009ba0 con 0x7f93f0164b40 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.098+0000 7f93e6000640 1 -- 172.21.15.39:0/3654758176 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e800f040 con 0x7f93f0164b40 2024-01-24T17:48:21.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.098+0000 7f93e6000640 1 -- 172.21.15.39:0/3654758176 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e80146c0 con 0x7f93f0164b40 2024-01-24T17:48:21.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.098+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f00cad50 con 0x7f93f0164b40 2024-01-24T17:48:21.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.098+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f00cb310 con 0x7f93f0164b40 2024-01-24T17:48:21.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.099+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f0155400 con 0x7f93f0164b40 2024-01-24T17:48:21.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.100+0000 7f93e6000640 1 -- 172.21.15.39:0/3654758176 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f93e8014820 con 0x7f93f0164b40 2024-01-24T17:48:21.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.101+0000 7f93e6000640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f93c806a430 0x7f93c806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:21.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.101+0000 7f93e6000640 1 -- 172.21.15.39:0/3654758176 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f93e80890a0 con 0x7f93f0164b40 2024-01-24T17:48:21.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.102+0000 7f93ede00640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f93c806a430 0x7f93c806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:21.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.102+0000 7f93ede00640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f93c806a430 0x7f93c806c8f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f93e0007fd0 tx=0x7f93e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:21.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.106+0000 7f93e6000640 1 -- 172.21.15.39:0/3654758176 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f93f0155400 con 0x7f93f0164b40 2024-01-24T17:48:21.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:21.313+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f93f0155400 con 0x7f93f0164b40 2024-01-24T17:48:21.526 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:48:21.534 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:48:22.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.086+0000 7f93e6000640 1 -- 172.21.15.39:0/3654758176 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v96) v1 ==== 97+0+0 (secure 0 0 0) 0x7f93f0155400 con 0x7f93f0164b40 2024-01-24T17:48:22.087 INFO:teuthology.orchestra.run.smithi039.stderr: removed all blocklist entries 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f93c806a430 msgr2=0x7f93c806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f93c806a430 0x7f93c806c8f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f93e0007fd0 tx=0x7f93e0009040 comp rx=0 tx=0).stop 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93f0164b40 msgr2=0x7f93f00c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93f0164b40 0x7f93f00c7d20 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009d30 tx=0x7f93e800c490 comp rx=0 tx=0).stop 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 shutdown_connections 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f93c806a430 0x7f93c806c8f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93f0164b40 0x7f93f00c7d20 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f93f015cbe0 0x7f93f00cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 --2- 172.21.15.39:0/3654758176 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f93f00d0e90 0x7f93f014a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 >> 172.21.15.39:0/3654758176 conn(0x7f93f009bc80 msgr2=0x7f93f0157c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:22.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 shutdown_connections 2024-01-24T17:48:22.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.090+0000 7f93f5200640 1 -- 172.21.15.39:0/3654758176 wait complete. 2024-01-24T17:48:22.114 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-24T17:48:22.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.336+0000 7ff75b000640 1 Processor -- start 2024-01-24T17:48:22.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.336+0000 7ff75b000640 1 -- start start 2024-01-24T17:48:22.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff75b000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff754158b80 0x7ff7540cab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff75b000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7540cb0b0 0x7ff75406bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff75b000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff75406c280 0x7ff75406c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff75b000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75406cbd0 con 0x7ff7540cb0b0 2024-01-24T17:48:22.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff75b000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75415c910 con 0x7ff75406c280 2024-01-24T17:48:22.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff75b000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75415d1b0 con 0x7ff754158b80 2024-01-24T17:48:22.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7540cb0b0 0x7ff75406bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7540cb0b0 0x7ff75406bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52318/0 (socket says 172.21.15.39:52318) 2024-01-24T17:48:22.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759200640 1 -- 172.21.15.39:0/2633255088 learned_addr learned my addr 172.21.15.39:0/2633255088 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff75a600640 1 --2- 172.21.15.39:0/2633255088 >> v2:172.21.15.100:3300/0 conn(0x7ff75406c280 0x7ff75406c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759c00640 1 --2- 172.21.15.39:0/2633255088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff754158b80 0x7ff7540cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759200640 1 -- 172.21.15.39:0/2633255088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff754158b80 msgr2=0x7ff7540cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759200640 1 --2- 172.21.15.39:0/2633255088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff754158b80 0x7ff7540cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759200640 1 -- 172.21.15.39:0/2633255088 >> v2:172.21.15.100:3300/0 conn(0x7ff75406c280 msgr2=0x7ff75406c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759200640 1 --2- 172.21.15.39:0/2633255088 >> v2:172.21.15.100:3300/0 conn(0x7ff75406c280 0x7ff75406c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff759200640 1 -- 172.21.15.39:0/2633255088 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7540cb510 con 0x7ff7540cb0b0 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.338+0000 7ff75a600640 1 --2- 172.21.15.39:0/2633255088 >> v2:172.21.15.100:3300/0 conn(0x7ff75406c280 0x7ff75406c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:22.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.340+0000 7ff759200640 1 --2- 172.21.15.39:0/2633255088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7540cb0b0 0x7ff75406bd40 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff7540c8960 tx=0x7ff73c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78da37f199b68484 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:22.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.340+0000 7ff753e00640 1 -- 172.21.15.39:0/2633255088 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c02c950 con 0x7ff7540cb0b0 2024-01-24T17:48:22.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.340+0000 7ff753e00640 1 -- 172.21.15.39:0/2633255088 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73c02cab0 con 0x7ff7540cb0b0 2024-01-24T17:48:22.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.340+0000 7ff753e00640 1 -- 172.21.15.39:0/2633255088 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff73c02ce40 con 0x7ff7540cb0b0 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.340+0000 7ff75b000640 1 -- 172.21.15.39:0/2633255088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7540cb0b0 msgr2=0x7ff75406bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.340+0000 7ff75b000640 1 --2- 172.21.15.39:0/2633255088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7540cb0b0 0x7ff75406bd40 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff7540c8960 tx=0x7ff73c02bdb0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.341+0000 7ff75b000640 1 -- 172.21.15.39:0/2633255088 shutdown_connections 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.341+0000 7ff75b000640 1 --2- 172.21.15.39:0/2633255088 >> v2:172.21.15.100:3300/0 conn(0x7ff75406c280 0x7ff75406c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.341+0000 7ff75b000640 1 --2- 172.21.15.39:0/2633255088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7540cb0b0 0x7ff75406bd40 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.341+0000 7ff75b000640 1 --2- 172.21.15.39:0/2633255088 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff754158b80 0x7ff7540cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.341+0000 7ff75b000640 1 -- 172.21.15.39:0/2633255088 >> 172.21.15.39:0/2633255088 conn(0x7ff75406a200 msgr2=0x7ff7540aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.341+0000 7ff75b000640 1 -- 172.21.15.39:0/2633255088 shutdown_connections 2024-01-24T17:48:22.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.341+0000 7ff75b000640 1 -- 172.21.15.39:0/2633255088 wait complete. 2024-01-24T17:48:22.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.342+0000 7ff75b000640 1 Processor -- start 2024-01-24T17:48:22.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.343+0000 7ff75b000640 1 -- start start 2024-01-24T17:48:22.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.343+0000 7ff75b000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75406c280 0x7ff7541526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75b000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7540cb0b0 0x7ff7540aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff754158b80 0x7ff7540af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff754151f80 con 0x7ff754158b80 2024-01-24T17:48:22.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75b000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7540c5fc0 con 0x7ff75406c280 2024-01-24T17:48:22.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75b000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7540c4b90 con 0x7ff7540cb0b0 2024-01-24T17:48:22.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff754158b80 0x7ff7540af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff754158b80 0x7ff7540af510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52328/0 (socket says 172.21.15.39:52328) 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75a600640 1 -- 172.21.15.39:0/1588978600 learned_addr learned my addr 172.21.15.39:0/1588978600 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff759c00640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75406c280 0x7ff7541526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff759200640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7540cb0b0 0x7ff7540aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75a600640 1 -- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7540cb0b0 msgr2=0x7ff7540aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75a600640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7540cb0b0 0x7ff7540aefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75a600640 1 -- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75406c280 msgr2=0x7ff7541526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75a600640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75406c280 0x7ff7541526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.344+0000 7ff75a600640 1 -- 172.21.15.39:0/1588978600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c009b30 con 0x7ff754158b80 2024-01-24T17:48:22.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.345+0000 7ff75a600640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff754158b80 0x7ff7540af510 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00c7f0 tx=0x7ff74c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:22.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.345+0000 7ff751600640 1 -- 172.21.15.39:0/1588978600 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c007820 con 0x7ff754158b80 2024-01-24T17:48:22.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.345+0000 7ff751600640 1 -- 172.21.15.39:0/1588978600 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c007980 con 0x7ff754158b80 2024-01-24T17:48:22.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.345+0000 7ff751600640 1 -- 172.21.15.39:0/1588978600 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c007d70 con 0x7ff754158b80 2024-01-24T17:48:22.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.345+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7540c5620 con 0x7ff754158b80 2024-01-24T17:48:22.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.345+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7540c5be0 con 0x7ff754158b80 2024-01-24T17:48:22.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.346+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7540c6100 con 0x7ff754158b80 2024-01-24T17:48:22.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.347+0000 7ff751600640 1 -- 172.21.15.39:0/1588978600 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff74c0113f0 con 0x7ff754158b80 2024-01-24T17:48:22.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.349+0000 7ff751600640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72c06a430 0x7ff72c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.349+0000 7ff751600640 1 -- 172.21.15.39:0/1588978600 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff74c088a40 con 0x7ff754158b80 2024-01-24T17:48:22.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.349+0000 7ff759c00640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72c06a430 0x7ff72c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.350+0000 7ff759c00640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72c06a430 0x7ff72c06c8f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff74400b720 tx=0x7ff744008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:22.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.353+0000 7ff751600640 1 -- 172.21.15.39:0/1588978600 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff7540c6100 con 0x7ff754158b80 2024-01-24T17:48:22.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.624+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff7540c6100 con 0x7ff754158b80 2024-01-24T17:48:22.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.625+0000 7ff751600640 1 -- 172.21.15.39:0/1588978600 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff7540c6100 con 0x7ff754158b80 2024-01-24T17:48:22.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_dump": [ 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.a", 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlZZEtLBAAUxf4chL3tgy3zHlZoraXlg==", 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.b", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Fln4utEhAA52ngJFqrdfa9bx56mVhRVA==", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.c", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flo8iGARAAf+29RXWr4RHkILIz7em3mA==", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:48:22.627 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mds.d", 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7Flh+08IRAARhZxy//j8c9b3Hg9uV9ajg==", 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.628 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.0", 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCQS7FlOj/pLBAAzi89tAacOfDKdlU0BUJvvA==", 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.1", 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCUS7FlbRAKDRAAAuAy47MDTtvmh8qNA3ZLTg==", 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.629 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.2", 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCXS7Fl7QmRGRAAVECH7oYTdvDPSHBy9WmHwA==", 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.3", 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCaS7FlCQHIMxAA9OchsMzsKFxs110SeJ3Atw==", 2024-01-24T17:48:22.630 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.4", 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCiS7FlDsBKBRAAXF2HafraT5nRbB5tBRUgWQ==", 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.5", 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQClS7FlfrXBHhAAy8GP3TLX/iok3OYPGTP5Mw==", 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.6", 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCoS7Fla11hNhAA/3oMP7cFtPSTdJl7eSnPTA==", 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:48:22.632 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "osd.7", 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCsS7FlKFMtEhAAPmaFzr5XHVQaKZYkJiIarA==", 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow profile osd", 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile osd", 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.0", 2024-01-24T17:48:22.633 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlTi/3LhAAc8FwXtLH2XjlFMO7wDMcSQ==", 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow r", 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.1", 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlH5s7ChAAlFfFDziTgfv4ibWO08N+4Q==", 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:48:22.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.2", 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCIS7FlIISlOhAAoVxxez7UjXDxyHQNNPnn2g==", 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.3", 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCJS7FlO/U/FBAA2pI9aGSS2cZ3AMnn+W1fQA==", 2024-01-24T17:48:22.635 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow", 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow r", 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow rw", 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow rwx" 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "client.admin", 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCGS7FlxKo7HhAAlu037DIc1WmWkz81BG3j7Q==", 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": "allow *", 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow *", 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.636 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.x", 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7FlRbF6HxAAoFjlyWb5oqIkC3L1iPRmHw==", 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": "mgr.y", 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "AQCHS7Fl512kFRAAdCpU+QjRmGJmMD5uTdFmHQ==", 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": { 2024-01-24T17:48:22.637 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": "allow *", 2024-01-24T17:48:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "mon": "allow profile mgr", 2024-01-24T17:48:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": "allow *" 2024-01-24T17:48:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:22.638 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:48:22.638 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72c06a430 msgr2=0x7ff72c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72c06a430 0x7ff72c06c8f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff74400b720 tx=0x7ff744008040 comp rx=0 tx=0).stop 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff754158b80 msgr2=0x7ff7540af510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff754158b80 0x7ff7540af510 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00c7f0 tx=0x7ff74c00ccb0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 shutdown_connections 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff72c06a430 0x7ff72c06c8f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff754158b80 0x7ff7540af510 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff7540cb0b0 0x7ff7540aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 --2- 172.21.15.39:0/1588978600 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75406c280 0x7ff7541526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.627+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 >> 172.21.15.39:0/1588978600 conn(0x7ff75406a200 msgr2=0x7ff7540c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.628+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 shutdown_connections 2024-01-24T17:48:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.628+0000 7ff75b000640 1 -- 172.21.15.39:0/1588978600 wait complete. 2024-01-24T17:48:22.648 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:22.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.856+0000 7febe5000640 1 Processor -- start 2024-01-24T17:48:22.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.857+0000 7febe5000640 1 -- start start 2024-01-24T17:48:22.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.858+0000 7febe5000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7febe00b3e00 0x7febe00c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.858+0000 7febe5000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe01551b0 con 0x7febe00c8f70 2024-01-24T17:48:22.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.858+0000 7febe5000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe015e410 con 0x7febe0070540 2024-01-24T17:48:22.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.858+0000 7febe5000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe015b480 con 0x7febe00b3e00 2024-01-24T17:48:22.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.858+0000 7febdfe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7febe00c8f70 0x7febe015e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:35270/0 (socket says 172.21.15.39:35270) 2024-01-24T17:48:22.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdfe00640 1 -- 172.21.15.39:0/418076833 learned_addr learned my addr 172.21.15.39:0/418076833 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:22.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febd7e00640 1 --2- 172.21.15.39:0/418076833 >> v2:172.21.15.100:3300/0 conn(0x7febe00b3e00 0x7febe00c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991533752 0 0) 0x7febe01551b0 con 0x7febe00c8f70 2024-01-24T17:48:22.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc0003520 con 0x7febe00c8f70 2024-01-24T17:48:22.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580641823 0 0) 0x7febc0003520 con 0x7febe00c8f70 2024-01-24T17:48:22.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 >> v2:172.21.15.100:3300/0 conn(0x7febe00b3e00 msgr2=0x7febe00c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:22.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdea00640 1 --2- 172.21.15.39:0/418076833 >> v2:172.21.15.100:3300/0 conn(0x7febe00b3e00 0x7febe00c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 >> v1:172.21.15.100:6789/0 conn(0x7febe0070540 legacy=0x7febe015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe015b760 con 0x7febe00c8f70 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.859+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7febd80019f0 con 0x7febe00c8f70 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 >> v1:172.21.15.39:6789/0 conn(0x7febe00c8f70 legacy=0x7febe015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdea00640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febc00039a0 0x7febc0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdea00640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00047c0 0x7febc0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdea00640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febc0005180 0x7febc000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe015b480 con 0x7febc00039a0 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc000c050 con 0x7febc0005180 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc000c210 con 0x7febc00047c0 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdf400640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febc00039a0 0x7febc0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdf400640 1 -- 172.21.15.39:0/418076833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00047c0 msgr2=0x7febc0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdf400640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00047c0 0x7febc0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdf400640 1 -- 172.21.15.39:0/418076833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febc0005180 msgr2=0x7febc000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdf400640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febc0005180 0x7febc000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdfe00640 1 -- 172.21.15.39:0/418076833 reap_dead start 2024-01-24T17:48:22.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.860+0000 7febdf400640 1 -- 172.21.15.39:0/418076833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc0003520 con 0x7febc00039a0 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.862+0000 7febdf400640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febc00039a0 0x7febc0003dc0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7febd0001770 tx=0x7febc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e3813bf6e617e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.862+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd00052d0 con 0x7febc00039a0 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.862+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd002e050 con 0x7febc00039a0 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.862+0000 7febdea00640 1 -- 172.21.15.39:0/418076833 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0003640 con 0x7febc00039a0 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.862+0000 7febe5000640 1 -- 172.21.15.39:0/418076833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febc00039a0 msgr2=0x7febc0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.862+0000 7febe5000640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febc00039a0 0x7febc0003dc0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7febd0001770 tx=0x7febc00057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.863+0000 7febe5000640 1 -- 172.21.15.39:0/418076833 shutdown_connections 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.863+0000 7febe5000640 1 --2- 172.21.15.39:0/418076833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febc00039a0 0x7febc0003dc0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.863+0000 7febe5000640 1 -- 172.21.15.39:0/418076833 >> 172.21.15.39:0/418076833 conn(0x7febe006a340 msgr2=0x7febe00cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.863+0000 7febe5000640 1 -- 172.21.15.39:0/418076833 shutdown_connections 2024-01-24T17:48:22.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.863+0000 7febe5000640 1 -- 172.21.15.39:0/418076833 wait complete. 2024-01-24T17:48:22.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.865+0000 7febe5000640 1 Processor -- start 2024-01-24T17:48:22.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.865+0000 7febe5000640 1 -- start start 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febe5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00039a0 0x7febe0152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febe5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febe0152740 0x7febe00c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febe5000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00c44a0 0x7febe00c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febe5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe01e52b0 con 0x7febe0152740 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febe5000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00c83b0 con 0x7febe00c44a0 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febe5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe0152ba0 con 0x7febc00039a0 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febd7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febe0152740 0x7febe00c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febd7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febe0152740 0x7febe00c3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52344/0 (socket says 172.21.15.39:52344) 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febd7e00640 1 -- 172.21.15.39:0/2841587925 learned_addr learned my addr 172.21.15.39:0/2841587925 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febdf400640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00039a0 0x7febe0152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febdfe00640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00c44a0 0x7febe00c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febd7e00640 1 -- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00039a0 msgr2=0x7febe0152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febd7e00640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00039a0 0x7febe0152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febd7e00640 1 -- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00c44a0 msgr2=0x7febe00c4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.866+0000 7febd7e00640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00c44a0 0x7febe00c4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.867+0000 7febd7e00640 1 -- 172.21.15.39:0/2841587925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe00c7640 con 0x7febe0152740 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.867+0000 7febdf400640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00039a0 0x7febe0152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.867+0000 7febd7e00640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febe0152740 0x7febe00c3f60 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7febc800e7d0 tx=0x7febc800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.867+0000 7febd7400640 1 -- 172.21.15.39:0/2841587925 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc800ee80 con 0x7febe0152740 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.867+0000 7febd7400640 1 -- 172.21.15.39:0/2841587925 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc800c940 con 0x7febe0152740 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.867+0000 7febd7400640 1 -- 172.21.15.39:0/2841587925 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc800ccf0 con 0x7febe0152740 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.867+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe00c32c0 con 0x7febe0152740 2024-01-24T17:48:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.867+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe00c7bd0 con 0x7febe0152740 2024-01-24T17:48:22.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.869+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe004dff0 con 0x7febe0152740 2024-01-24T17:48:22.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.869+0000 7febd7400640 1 -- 172.21.15.39:0/2841587925 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7febc8013470 con 0x7febe0152740 2024-01-24T17:48:22.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.870+0000 7febd7400640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febb806a430 0x7febb806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:22.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.871+0000 7febd7400640 1 -- 172.21.15.39:0/2841587925 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7febc8088450 con 0x7febe0152740 2024-01-24T17:48:22.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.871+0000 7febdf400640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febb806a430 0x7febb806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:22.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.872+0000 7febdf400640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febb806a430 0x7febb806c8f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7febd0033000 tx=0x7febd0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:22.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:22.875+0000 7febd7400640 1 -- 172.21.15.39:0/2841587925 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7febc8053e20 con 0x7febe0152740 2024-01-24T17:48:23.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.073+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febe00cbc10 con 0x7febe0152740 2024-01-24T17:48:23.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.075+0000 7febd7400640 1 -- 172.21.15.39:0/2841587925 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+12390 (secure 0 0 0) 0x7febc8057440 con 0x7febe0152740 2024-01-24T17:48:23.076 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:23.077 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":96,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:48:22.078334+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:23.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.078+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febb806a430 msgr2=0x7febb806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.078+0000 7febe5000640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febb806a430 0x7febb806c8f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7febd0033000 tx=0x7febd0009030 comp rx=0 tx=0).stop 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.078+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febe0152740 msgr2=0x7febe00c3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.078+0000 7febe5000640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febe0152740 0x7febe00c3f60 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7febc800e7d0 tx=0x7febc800ec90 comp rx=0 tx=0).stop 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.078+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 shutdown_connections 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.078+0000 7febe5000640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7febe00c44a0 0x7febe00c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.078+0000 7febe5000640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7febb806a430 0x7febb806c8f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.078+0000 7febe5000640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febe0152740 0x7febe00c3f60 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.079+0000 7febe5000640 1 --2- 172.21.15.39:0/2841587925 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7febc00039a0 0x7febe0152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.079+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 >> 172.21.15.39:0/2841587925 conn(0x7febe006a340 msgr2=0x7febe00ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.079+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 shutdown_connections 2024-01-24T17:48:23.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.079+0000 7febe5000640 1 -- 172.21.15.39:0/2841587925 wait complete. 2024-01-24T17:48:23.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:48:23.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:48:23.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.326+0000 7f8286a00640 1 Processor -- start 2024-01-24T17:48:23.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.327+0000 7f8286a00640 1 -- start start 2024-01-24T17:48:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f8286a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82800cb990 0x7f82800ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f8286a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f828006cec0 0x7f828006d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f8286a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800ca4c0 con 0x7f828015aa60 2024-01-24T17:48:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f8286a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82800cae90 con 0x7f828006cec0 2024-01-24T17:48:23.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f8286a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828006dc90 con 0x7f82800cb990 2024-01-24T17:48:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f827f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82800cb990 0x7f82800ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f8284a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f828006cec0 0x7f828006d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f8284a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f828006cec0 0x7f828006d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:55944/0 (socket says 172.21.15.39:55944) 2024-01-24T17:48:23.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.328+0000 7f8284a00640 1 -- 172.21.15.39:0/1738164937 learned_addr learned my addr 172.21.15.39:0/1738164937 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.329+0000 7f8284a00640 1 -- 172.21.15.39:0/1738164937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82800cb990 msgr2=0x7f82800ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.329+0000 7f8284a00640 1 --2- 172.21.15.39:0/1738164937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82800cb990 0x7f82800ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.329+0000 7f8284a00640 1 -- 172.21.15.39:0/1738164937 >> v1:172.21.15.100:6789/0 conn(0x7f828015aa60 legacy=0x7f82800cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.329+0000 7f8284a00640 1 -- 172.21.15.39:0/1738164937 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82800c9e70 con 0x7f828006cec0 2024-01-24T17:48:23.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.330+0000 7f8284a00640 1 --2- 172.21.15.39:0/1738164937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f828006cec0 0x7f828006d310 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8278009870 tx=0x7f827802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f8cffc6a5a9f58 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.331+0000 7f827ea00640 1 -- 172.21.15.39:0/1738164937 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278030840 con 0x7f828006cec0 2024-01-24T17:48:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.331+0000 7f827ea00640 1 -- 172.21.15.39:0/1738164937 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82780309a0 con 0x7f828006cec0 2024-01-24T17:48:23.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.331+0000 7f827ea00640 1 -- 172.21.15.39:0/1738164937 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278030d30 con 0x7f828006cec0 2024-01-24T17:48:23.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.332+0000 7f8286a00640 1 -- 172.21.15.39:0/1738164937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f828006cec0 msgr2=0x7f828006d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.332+0000 7f8286a00640 1 --2- 172.21.15.39:0/1738164937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f828006cec0 0x7f828006d310 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8278009870 tx=0x7f827802b980 comp rx=0 tx=0).stop 2024-01-24T17:48:23.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.332+0000 7f8286a00640 1 -- 172.21.15.39:0/1738164937 shutdown_connections 2024-01-24T17:48:23.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.332+0000 7f8286a00640 1 --2- 172.21.15.39:0/1738164937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f828006cec0 0x7f828006d310 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.332+0000 7f8286a00640 1 --2- 172.21.15.39:0/1738164937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f82800cb990 0x7f82800ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.332+0000 7f8286a00640 1 -- 172.21.15.39:0/1738164937 >> 172.21.15.39:0/1738164937 conn(0x7f82800ce9e0 msgr2=0x7f82800cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:23.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.332+0000 7f8286a00640 1 -- 172.21.15.39:0/1738164937 shutdown_connections 2024-01-24T17:48:23.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.332+0000 7f8286a00640 1 -- 172.21.15.39:0/1738164937 wait complete. 2024-01-24T17:48:23.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.334+0000 7f8286a00640 1 Processor -- start 2024-01-24T17:48:23.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.334+0000 7f8286a00640 1 -- start start 2024-01-24T17:48:23.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.335+0000 7f8286a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f828006cec0 0x7f82800cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.335+0000 7f8286a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82800cb990 0x7f82801f5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.335+0000 7f8286a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f828015aa60 0x7f82801f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.335+0000 7f8286a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800af000 con 0x7f828015aa60 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.335+0000 7f8286a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828015d5b0 con 0x7f82800cb990 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.335+0000 7f8286a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828015d2d0 con 0x7f828006cec0 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f828015aa60 0x7f82801f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f828015aa60 0x7f82801f60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49550/0 (socket says 172.21.15.39:49550) 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 -- 172.21.15.39:0/2063210755 learned_addr learned my addr 172.21.15.39:0/2063210755 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f827fe00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f828006cec0 0x7f82800cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f827f400640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82800cb990 0x7f82801f5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 -- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f828006cec0 msgr2=0x7f82800cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f828006cec0 0x7f82800cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 -- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82800cb990 msgr2=0x7f82801f5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82800cb990 0x7f82801f5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 -- 172.21.15.39:0/2063210755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82780094d0 con 0x7f828015aa60 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.336+0000 7f8284a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f828015aa60 0x7f82801f60a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f8278009870 tx=0x7f8278004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.337+0000 7f8277e00640 1 -- 172.21.15.39:0/2063210755 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278039a90 con 0x7f828015aa60 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.337+0000 7f8277e00640 1 -- 172.21.15.39:0/2063210755 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8278039c10 con 0x7f828015aa60 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.337+0000 7f8277e00640 1 -- 172.21.15.39:0/2063210755 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278043410 con 0x7f828015aa60 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.337+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82800aed20 con 0x7f828015aa60 2024-01-24T17:48:23.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.337+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828015cfb0 con 0x7f828015aa60 2024-01-24T17:48:23.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.338+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828004dff0 con 0x7f828015aa60 2024-01-24T17:48:23.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.339+0000 7f8277e00640 1 -- 172.21.15.39:0/2063210755 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f827802f390 con 0x7f828015aa60 2024-01-24T17:48:23.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.340+0000 7f8277e00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825806a430 0x7f825806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.340+0000 7f8277e00640 1 -- 172.21.15.39:0/2063210755 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f82780aa720 con 0x7f828015aa60 2024-01-24T17:48:23.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.341+0000 7f827fe00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825806a430 0x7f825806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.342+0000 7f827fe00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825806a430 0x7f825806c8f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f82700012e0 tx=0x7f8270009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:23.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.344+0000 7f8277e00640 1 -- 172.21.15.39:0/2063210755 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f8278076170 con 0x7f828015aa60 2024-01-24T17:48:23.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.541+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f82800c5290 con 0x7f828015aa60 2024-01-24T17:48:23.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.542+0000 7f8277e00640 1 -- 172.21.15.39:0/2063210755 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8278079790 con 0x7f828015aa60 2024-01-24T17:48:23.543 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825806a430 msgr2=0x7f825806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825806a430 0x7f825806c8f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f82700012e0 tx=0x7f8270009040 comp rx=0 tx=0).stop 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f828015aa60 msgr2=0x7f82801f60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f828015aa60 0x7f82801f60a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f8278009870 tx=0x7f8278004730 comp rx=0 tx=0).stop 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 shutdown_connections 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f825806a430 0x7f825806c8f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f828015aa60 0x7f82801f60a0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f82800cb990 0x7f82801f5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 --2- 172.21.15.39:0/2063210755 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f828006cec0 0x7f82800cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 >> 172.21.15.39:0/2063210755 conn(0x7f82800ce9e0 msgr2=0x7f828006ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:23.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 shutdown_connections 2024-01-24T17:48:23.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.545+0000 7f8286a00640 1 -- 172.21.15.39:0/2063210755 wait complete. 2024-01-24T17:48:23.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T17:48:23.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.780+0000 7f2ca8e00640 1 Processor -- start 2024-01-24T17:48:23.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.780+0000 7f2ca8e00640 1 -- start start 2024-01-24T17:48:23.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.782+0000 7f2ca8e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b0720 0x7f2ca416c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.782+0000 7f2ca8e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca40cce00 con 0x7f2ca4161270 2024-01-24T17:48:23.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.782+0000 7f2ca8e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca40cb8a0 con 0x7f2ca40cb420 2024-01-24T17:48:23.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.782+0000 7f2ca8e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40cbe60 con 0x7f2ca40b0720 2024-01-24T17:48:23.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.782+0000 7f2ca1e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2ca4161270 0x7f2ca40cad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39368/0 (socket says 172.21.15.39:39368) 2024-01-24T17:48:23.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.782+0000 7f2ca1e00640 1 -- 172.21.15.39:0/361703116 learned_addr learned my addr 172.21.15.39:0/361703116 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:23.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.782+0000 7f2ca2800640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b0720 0x7f2ca416c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.782+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243290157 0 0) 0x7f2ca40cce00 con 0x7f2ca4161270 2024-01-24T17:48:23.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c84003520 con 0x7f2ca4161270 2024-01-24T17:48:23.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 498225895 0 0) 0x7f2c84003520 con 0x7f2ca4161270 2024-01-24T17:48:23.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b0720 msgr2=0x7f2ca416c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:23.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b0720 0x7f2ca416c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 >> v1:172.21.15.100:6789/0 conn(0x7f2ca40cb420 legacy=0x7f2ca40b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40cbb80 con 0x7f2ca4161270 2024-01-24T17:48:23.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2c94001670 con 0x7f2ca4161270 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 >> v1:172.21.15.39:6789/0 conn(0x7f2ca4161270 legacy=0x7f2ca40cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c840039a0 0x7f2c84003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b0720 0x7f2c840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c84004d00 0x7f2c84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40cbe60 con 0x7f2c840039a0 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400bb90 con 0x7f2c84004d00 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400bd50 con 0x7f2ca40b0720 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.783+0000 7f2ca1e00640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c840039a0 0x7f2c84003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.784+0000 7f2c99200640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b0720 0x7f2c840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.784+0000 7f2ca2800640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c84004d00 0x7f2c84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.784+0000 7f2ca1e00640 1 -- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b0720 msgr2=0x7f2c840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.784+0000 7f2ca1e00640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b0720 0x7f2c840047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.784+0000 7f2ca1e00640 1 -- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c84004d00 msgr2=0x7f2c84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:23.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.784+0000 7f2ca1e00640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c84004d00 0x7f2c84005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.784+0000 7f2ca1e00640 1 -- 172.21.15.39:0/361703116 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c84003520 con 0x7f2c840039a0 2024-01-24T17:48:23.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.785+0000 7f2ca1e00640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c840039a0 0x7f2c84003dc0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400c240 tx=0x7f2c840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e228ad24306ed72 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:23.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.785+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94005ad0 con 0x7f2c840039a0 2024-01-24T17:48:23.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.785+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c94005c30 con 0x7f2c840039a0 2024-01-24T17:48:23.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.785+0000 7f2ca1400640 1 -- 172.21.15.39:0/361703116 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9402c8b0 con 0x7f2c840039a0 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.786+0000 7f2ca8e00640 1 -- 172.21.15.39:0/361703116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c840039a0 msgr2=0x7f2c84003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.786+0000 7f2ca8e00640 1 --2- 172.21.15.39:0/361703116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2c840039a0 0x7f2c84003dc0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400c240 tx=0x7f2c840057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.786+0000 7f2ca2800640 1 -- 172.21.15.39:0/361703116 reap_dead start 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.786+0000 7f2ca8e00640 1 -- 172.21.15.39:0/361703116 shutdown_connections 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.786+0000 7f2ca8e00640 1 -- 172.21.15.39:0/361703116 >> 172.21.15.39:0/361703116 conn(0x7f2ca406bf20 msgr2=0x7f2ca406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.786+0000 7f2ca8e00640 1 -- 172.21.15.39:0/361703116 shutdown_connections 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.786+0000 7f2ca8e00640 1 -- 172.21.15.39:0/361703116 wait complete. 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.788+0000 7f2ca8e00640 1 Processor -- start 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.788+0000 7f2ca8e00640 1 -- start start 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2ca8e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c840039a0 0x7f2ca4158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2ca8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca40c6ba0 0x7f2ca40c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2ca8e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b2fc0 0x7f2ca40b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2ca8e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40b3980 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2ca8e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40b0b00 con 0x7f2c840039a0 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2ca8e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40c2e20 con 0x7f2ca40b2fc0 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2c99200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca40c6ba0 0x7f2ca40c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2c99200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca40c6ba0 0x7f2ca40c5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49562/0 (socket says 172.21.15.39:49562) 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.789+0000 7f2c99200640 1 -- 172.21.15.39:0/1462991043 learned_addr learned my addr 172.21.15.39:0/1462991043 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2ca1e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c840039a0 0x7f2ca4158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2ca2800640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b2fc0 0x7f2ca40b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c99200640 1 -- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b2fc0 msgr2=0x7f2ca40b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c99200640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b2fc0 0x7f2ca40b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c99200640 1 -- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c840039a0 msgr2=0x7f2ca4158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c99200640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c840039a0 0x7f2ca4158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c99200640 1 -- 172.21.15.39:0/1462991043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40c5e90 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2ca2800640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b2fc0 0x7f2ca40b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c99200640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca40c6ba0 0x7f2ca40c5660 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00d570 tx=0x7f2c8c00da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:23.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c9aa00640 1 -- 172.21.15.39:0/1462991043 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c005480 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c9aa00640 1 -- 172.21.15.39:0/1462991043 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c003610 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2c9aa00640 1 -- 172.21.15.39:0/1462991043 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c003980 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.790+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca40c6170 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.791+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca40c6730 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.792+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca40c9250 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.793+0000 7f2c9aa00640 1 -- 172.21.15.39:0/1462991043 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2c8c003ae0 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.793+0000 7f2c9aa00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c7c06a430 0x7f2c7c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:23.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.794+0000 7f2c9aa00640 1 -- 172.21.15.39:0/1462991043 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2c8c0884f0 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.794+0000 7f2ca1e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c7c06a430 0x7f2c7c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:23.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.795+0000 7f2ca1e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c7c06a430 0x7f2c7c06c8f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2c94034000 tx=0x7f2c94009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:23.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.798+0000 7f2c9aa00640 1 -- 172.21.15.39:0/1462991043 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2c8c053e20 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.996+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2ca404dff0 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.997+0000 7f2c9aa00640 1 -- 172.21.15.39:0/1462991043 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v96) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2c8c001230 con 0x7f2ca40c6ba0 2024-01-24T17:48:23.998 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:23.998 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-01-24T17:48:23.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:48:23.998 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-01-24T17:48:23.998 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-01-24T17:48:23.998 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:23.998 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-01-24T17:48:24.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.999+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c7c06a430 msgr2=0x7f2c7c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.999+0000 7f2ca8e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c7c06a430 0x7f2c7c06c8f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2c94034000 tx=0x7f2c94009030 comp rx=0 tx=0).stop 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.999+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca40c6ba0 msgr2=0x7f2ca40c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:23.999+0000 7f2ca8e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca40c6ba0 0x7f2ca40c5660 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00d570 tx=0x7f2c8c00da30 comp rx=0 tx=0).stop 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.000+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 shutdown_connections 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.000+0000 7f2ca8e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2ca40b2fc0 0x7f2ca40b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.000+0000 7f2ca8e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2c7c06a430 0x7f2c7c06c8f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.000+0000 7f2ca8e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ca40c6ba0 0x7f2ca40c5660 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.000+0000 7f2ca8e00640 1 --2- 172.21.15.39:0/1462991043 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2c840039a0 0x7f2ca4158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.000+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 >> 172.21.15.39:0/1462991043 conn(0x7f2ca406bf20 msgr2=0x7f2ca40ca840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.000+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 shutdown_connections 2024-01-24T17:48:24.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.000+0000 7f2ca8e00640 1 -- 172.21.15.39:0/1462991043 wait complete. 2024-01-24T17:48:24.021 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T17:48:24.021 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T17:48:24.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.250+0000 7fc45b200640 1 Processor -- start 2024-01-24T17:48:24.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.250+0000 7fc45b200640 1 -- start start 2024-01-24T17:48:24.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.252+0000 7fc45b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc45c15ac70 0x7fc45c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:24.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.252+0000 7fc45b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c0b12b0 con 0x7fc45c070260 2024-01-24T17:48:24.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.252+0000 7fc45b200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c15b170 con 0x7fc45c1516c0 2024-01-24T17:48:24.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.252+0000 7fc45b200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45c0706d0 con 0x7fc45c15ac70 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.252+0000 7fc453e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc45c15ac70 0x7fc45c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.252+0000 7fc45a800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc45c070260 0x7fc45c0ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39378/0 (socket says 172.21.15.39:39378) 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.252+0000 7fc45a800640 1 -- 172.21.15.39:0/3992460963 learned_addr learned my addr 172.21.15.39:0/3992460963 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.253+0000 7fc459400640 1 -- 172.21.15.39:0/3992460963 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293229833 0 0) 0x7fc45c0b12b0 con 0x7fc45c070260 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.253+0000 7fc459400640 1 -- 172.21.15.39:0/3992460963 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c003520 con 0x7fc45c070260 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.253+0000 7fc453e00640 1 -- 172.21.15.39:0/3992460963 >> v1:172.21.15.100:6789/0 conn(0x7fc45c1516c0 legacy=0x7fc45c0b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.253+0000 7fc453e00640 1 -- 172.21.15.39:0/3992460963 >> v1:172.21.15.39:6789/0 conn(0x7fc45c070260 legacy=0x7fc45c0ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.253+0000 7fc453e00640 1 -- 172.21.15.39:0/3992460963 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c15b450 con 0x7fc45c15ac70 2024-01-24T17:48:24.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.254+0000 7fc453e00640 1 --2- 172.21.15.39:0/3992460963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c06fd20 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fc4440097a0 tx=0x7fc44402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8856bba4cb19c271 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:24.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.254+0000 7fc459400640 1 -- 172.21.15.39:0/3992460963 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444030610 con 0x7fc45c15ac70 2024-01-24T17:48:24.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc459400640 1 -- 172.21.15.39:0/3992460963 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc444030770 con 0x7fc45c15ac70 2024-01-24T17:48:24.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc459400640 1 -- 172.21.15.39:0/3992460963 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444030b00 con 0x7fc45c15ac70 2024-01-24T17:48:24.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc45b200640 1 -- 172.21.15.39:0/3992460963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 msgr2=0x7fc45c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:24.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc45b200640 1 --2- 172.21.15.39:0/3992460963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c06fd20 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fc4440097a0 tx=0x7fc44402b960 comp rx=0 tx=0).stop 2024-01-24T17:48:24.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc45b200640 1 -- 172.21.15.39:0/3992460963 shutdown_connections 2024-01-24T17:48:24.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc45b200640 1 --2- 172.21.15.39:0/3992460963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c06fd20 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:24.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc45b200640 1 -- 172.21.15.39:0/3992460963 >> 172.21.15.39:0/3992460963 conn(0x7fc45c06a200 msgr2=0x7fc45c0affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:24.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc45b200640 1 -- 172.21.15.39:0/3992460963 shutdown_connections 2024-01-24T17:48:24.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.255+0000 7fc45b200640 1 -- 172.21.15.39:0/3992460963 wait complete. 2024-01-24T17:48:24.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.257+0000 7fc45b200640 1 Processor -- start 2024-01-24T17:48:24.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.257+0000 7fc45b200640 1 -- start start 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.258+0000 7fc45b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc45c070260 0x7fc45c0cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.258+0000 7fc45b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc45c1516c0 0x7fc45c0cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.258+0000 7fc45b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c0acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.258+0000 7fc45b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0aed80 con 0x7fc45c15ac70 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.258+0000 7fc45b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0c3d80 con 0x7fc45c1516c0 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.258+0000 7fc45b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0cc530 con 0x7fc45c070260 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc45a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c0acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc45a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c0acb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49578/0 (socket says 172.21.15.39:49578) 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc45a800640 1 -- 172.21.15.39:0/1293579308 learned_addr learned my addr 172.21.15.39:0/1293579308 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc459e00640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc45c070260 0x7fc45c0cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:24.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc453e00640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc45c1516c0 0x7fc45c0cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:24.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc45a800640 1 -- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc45c070260 msgr2=0x7fc45c0cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:24.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc45a800640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc45c070260 0x7fc45c0cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:24.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc45a800640 1 -- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc45c1516c0 msgr2=0x7fc45c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:24.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc45a800640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc45c1516c0 0x7fc45c0cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:24.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.259+0000 7fc45a800640 1 -- 172.21.15.39:0/1293579308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444009400 con 0x7fc45c15ac70 2024-01-24T17:48:24.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.260+0000 7fc45a800640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c0acb80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fc45400cb00 tx=0x7fc45400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:24.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.260+0000 7fc452000640 1 -- 172.21.15.39:0/1293579308 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45400aca0 con 0x7fc45c15ac70 2024-01-24T17:48:24.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.260+0000 7fc452000640 1 -- 172.21.15.39:0/1293579308 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45400ae00 con 0x7fc45c15ac70 2024-01-24T17:48:24.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.260+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c0cdf80 con 0x7fc45c15ac70 2024-01-24T17:48:24.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.260+0000 7fc452000640 1 -- 172.21.15.39:0/1293579308 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454012620 con 0x7fc45c15ac70 2024-01-24T17:48:24.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.260+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c0ae220 con 0x7fc45c15ac70 2024-01-24T17:48:24.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.261+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c0c3ed0 con 0x7fc45c15ac70 2024-01-24T17:48:24.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.262+0000 7fc452000640 1 -- 172.21.15.39:0/1293579308 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc454007070 con 0x7fc45c15ac70 2024-01-24T17:48:24.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.263+0000 7fc452000640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc43406a460 0x7fc43406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.263+0000 7fc452000640 1 -- 172.21.15.39:0/1293579308 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc4540885e0 con 0x7fc45c15ac70 2024-01-24T17:48:24.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.264+0000 7fc459e00640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc43406a460 0x7fc43406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:24.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.265+0000 7fc459e00640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc43406a460 0x7fc43406c920 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc44c000f60 tx=0x7fc44c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:24.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.270+0000 7fc452000640 1 -- 172.21.15.39:0/1293579308 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc45c0c3ed0 con 0x7fc45c15ac70 2024-01-24T17:48:24.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:24.467+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc45c0c3ed0 con 0x7fc45c15ac70 2024-01-24T17:48:25.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.379+0000 7fc452000640 1 -- 172.21.15.39:0/1293579308 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v97) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc45c0c3ed0 con 0x7fc45c15ac70 2024-01-24T17:48:25.380 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-01-24T17:48:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc43406a460 msgr2=0x7fc43406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc43406a460 0x7fc43406c920 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc44c000f60 tx=0x7fc44c009040 comp rx=0 tx=0).stop 2024-01-24T17:48:25.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 msgr2=0x7fc45c0acb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c0acb80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fc45400cb00 tx=0x7fc45400a060 comp rx=0 tx=0).stop 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 shutdown_connections 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc43406a460 0x7fc43406c920 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc45c15ac70 0x7fc45c0acb80 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc45c1516c0 0x7fc45c0cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 --2- 172.21.15.39:0/1293579308 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc45c070260 0x7fc45c0cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.383+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 >> 172.21.15.39:0/1293579308 conn(0x7fc45c06a200 msgr2=0x7fc45c0b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.384+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 shutdown_connections 2024-01-24T17:48:25.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.384+0000 7fc45b200640 1 -- 172.21.15.39:0/1293579308 wait complete. 2024-01-24T17:48:25.415 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T17:48:25.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.639+0000 7f494aa00640 1 Processor -- start 2024-01-24T17:48:25.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.639+0000 7f494aa00640 1 -- start start 2024-01-24T17:48:25.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.641+0000 7f494aa00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f4944159400 0x7f49440cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:25.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.641+0000 7f494aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f494415a7f0 0x7f49440cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:25.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.641+0000 7f494aa00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4944070db0 con 0x7f49440cdbf0 2024-01-24T17:48:25.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.641+0000 7f494aa00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49440c9bb0 con 0x7f494415a7f0 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.641+0000 7f494aa00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49440c9030 con 0x7f4944159400 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.641+0000 7f494a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f494415a7f0 0x7f49440cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.641+0000 7f4948c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f49440cdbf0 0x7f49440ca8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39384/0 (socket says 172.21.15.39:39384) 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.641+0000 7f4948c00640 1 -- 172.21.15.39:0/3333089209 learned_addr learned my addr 172.21.15.39:0/3333089209 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.642+0000 7f4949600640 1 --2- 172.21.15.39:0/3333089209 >> v2:172.21.15.100:3300/0 conn(0x7f4944159400 0x7f49440cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.642+0000 7f494a000640 1 -- 172.21.15.39:0/3333089209 >> v2:172.21.15.100:3300/0 conn(0x7f4944159400 msgr2=0x7f49440cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.642+0000 7f494a000640 1 --2- 172.21.15.39:0/3333089209 >> v2:172.21.15.100:3300/0 conn(0x7f4944159400 0x7f49440cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.642+0000 7f494a000640 1 -- 172.21.15.39:0/3333089209 >> v1:172.21.15.39:6789/0 conn(0x7f49440cdbf0 legacy=0x7f49440ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.642+0000 7f494a000640 1 -- 172.21.15.39:0/3333089209 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49440c9310 con 0x7f494415a7f0 2024-01-24T17:48:25.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.643+0000 7f494a000640 1 --2- 172.21.15.39:0/3333089209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f494415a7f0 0x7f49440cafb0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f493c009d90 tx=0x7f493c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba088f0627b677b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:25.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.643+0000 7f4943e00640 1 -- 172.21.15.39:0/3333089209 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c02cc60 con 0x7f494415a7f0 2024-01-24T17:48:25.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.643+0000 7f4943e00640 1 -- 172.21.15.39:0/3333089209 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c02cdc0 con 0x7f494415a7f0 2024-01-24T17:48:25.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.643+0000 7f4943e00640 1 -- 172.21.15.39:0/3333089209 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c034660 con 0x7f494415a7f0 2024-01-24T17:48:25.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.644+0000 7f494aa00640 1 -- 172.21.15.39:0/3333089209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f494415a7f0 msgr2=0x7f49440cafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:25.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.644+0000 7f494aa00640 1 --2- 172.21.15.39:0/3333089209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f494415a7f0 0x7f49440cafb0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f493c009d90 tx=0x7f493c02beb0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.644+0000 7f494aa00640 1 -- 172.21.15.39:0/3333089209 shutdown_connections 2024-01-24T17:48:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.644+0000 7f494aa00640 1 --2- 172.21.15.39:0/3333089209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f494415a7f0 0x7f49440cafb0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.644+0000 7f494aa00640 1 --2- 172.21.15.39:0/3333089209 >> v2:172.21.15.100:3300/0 conn(0x7f4944159400 0x7f49440cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.644+0000 7f494aa00640 1 -- 172.21.15.39:0/3333089209 >> 172.21.15.39:0/3333089209 conn(0x7f494406a200 msgr2=0x7f49440affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.644+0000 7f494aa00640 1 -- 172.21.15.39:0/3333089209 shutdown_connections 2024-01-24T17:48:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.644+0000 7f494aa00640 1 -- 172.21.15.39:0/3333089209 wait complete. 2024-01-24T17:48:25.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.647+0000 7f494aa00640 1 Processor -- start 2024-01-24T17:48:25.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.647+0000 7f494aa00640 1 -- start start 2024-01-24T17:48:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.648+0000 7f494aa00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49440cdbf0 0x7f4944155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.648+0000 7f494aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4944159400 0x7f49440aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.648+0000 7f494aa00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f494415a7f0 0x7f4944154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.648+0000 7f494aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4944155ab0 con 0x7f4944159400 2024-01-24T17:48:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.648+0000 7f494aa00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494414d410 con 0x7f494415a7f0 2024-01-24T17:48:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.648+0000 7f494aa00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49440ae850 con 0x7f49440cdbf0 2024-01-24T17:48:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4948c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4944159400 0x7f49440aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4948c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4944159400 0x7f49440aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49600/0 (socket says 172.21.15.39:49600) 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4948c00640 1 -- 172.21.15.39:0/3540797145 learned_addr learned my addr 172.21.15.39:0/3540797145 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4949600640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49440cdbf0 0x7f4944155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f494a000640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f494415a7f0 0x7f4944154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4948c00640 1 -- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49440cdbf0 msgr2=0x7f4944155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4948c00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49440cdbf0 0x7f4944155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4948c00640 1 -- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f494415a7f0 msgr2=0x7f4944154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4948c00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f494415a7f0 0x7f4944154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.649+0000 7f4948c00640 1 -- 172.21.15.39:0/3540797145 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c009a70 con 0x7f4944159400 2024-01-24T17:48:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.650+0000 7f4948c00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4944159400 0x7f49440aece0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f492c00c4a0 tx=0x7f492c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:25.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.650+0000 7f4941600640 1 -- 172.21.15.39:0/3540797145 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c005ce0 con 0x7f4944159400 2024-01-24T17:48:25.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.650+0000 7f4941600640 1 -- 172.21.15.39:0/3540797145 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c005e40 con 0x7f4944159400 2024-01-24T17:48:25.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.650+0000 7f4941600640 1 -- 172.21.15.39:0/3540797145 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c007270 con 0x7f4944159400 2024-01-24T17:48:25.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.650+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49440ad960 con 0x7f4944159400 2024-01-24T17:48:25.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.650+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49440adf20 con 0x7f4944159400 2024-01-24T17:48:25.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.651+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49440c5340 con 0x7f4944159400 2024-01-24T17:48:25.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.652+0000 7f4941600640 1 -- 172.21.15.39:0/3540797145 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f492c0073d0 con 0x7f4944159400 2024-01-24T17:48:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.653+0000 7f4941600640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f491c06a460 0x7f491c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.653+0000 7f4941600640 1 -- 172.21.15.39:0/3540797145 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f492c087ba0 con 0x7f4944159400 2024-01-24T17:48:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.654+0000 7f4949600640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f491c06a460 0x7f491c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:25.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.655+0000 7f4949600640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f491c06a460 0x7f491c06c920 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4934006fd0 tx=0x7f4934008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:25.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.659+0000 7f4941600640 1 -- 172.21.15.39:0/3540797145 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f492c053410 con 0x7f4944159400 2024-01-24T17:48:25.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:25.852+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f494404e840 con 0x7f4944159400 2024-01-24T17:48:26.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.406+0000 7f4941600640 1 -- 172.21.15.39:0/3540797145 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v98) v1 ==== 170+0+0 (secure 0 0 0) 0x7f492c0170d0 con 0x7f4944159400 2024-01-24T17:48:26.408 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-01-24T17:48:26.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.411+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f491c06a460 msgr2=0x7f491c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:26.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.411+0000 7f494aa00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f491c06a460 0x7f491c06c920 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4934006fd0 tx=0x7f4934008040 comp rx=0 tx=0).stop 2024-01-24T17:48:26.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.411+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4944159400 msgr2=0x7f49440aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:26.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.411+0000 7f494aa00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4944159400 0x7f49440aece0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f492c00c4a0 tx=0x7f492c00c960 comp rx=0 tx=0).stop 2024-01-24T17:48:26.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.412+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 shutdown_connections 2024-01-24T17:48:26.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.412+0000 7f494aa00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f494415a7f0 0x7f4944154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.412+0000 7f494aa00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f491c06a460 0x7f491c06c920 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.412+0000 7f494aa00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4944159400 0x7f49440aece0 secure :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f492c00c4a0 tx=0x7f492c00c960 comp rx=0 tx=0).stop 2024-01-24T17:48:26.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.412+0000 7f494aa00640 1 --2- 172.21.15.39:0/3540797145 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49440cdbf0 0x7f4944155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.412+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 >> 172.21.15.39:0/3540797145 conn(0x7f494406a200 msgr2=0x7f4944156860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:26.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.414+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 shutdown_connections 2024-01-24T17:48:26.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.414+0000 7f494aa00640 1 -- 172.21.15.39:0/3540797145 wait complete. 2024-01-24T17:48:26.443 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T17:48:26.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.629+0000 7f7afd000640 1 Processor -- start 2024-01-24T17:48:26.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.629+0000 7f7afd000640 1 -- start start 2024-01-24T17:48:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7afd000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af815aa60 0x7f7af80ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7afd000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7af806d5d0 0x7f7af80cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7afd000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af80cbc20 con 0x7f7af80cb280 2024-01-24T17:48:26.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7afd000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af80cae90 con 0x7f7af806d5d0 2024-01-24T17:48:26.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7afd000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80ca4c0 con 0x7f7af815aa60 2024-01-24T17:48:26.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7aefe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7af80cb280 0x7f7af806cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39400/0 (socket says 172.21.15.39:39400) 2024-01-24T17:48:26.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7aefe00640 1 -- 172.21.15.39:0/2138558012 learned_addr learned my addr 172.21.15.39:0/2138558012 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:26.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7af6800640 1 --2- 172.21.15.39:0/2138558012 >> v2:172.21.15.39:3300/0 conn(0x7f7af806d5d0 0x7f7af80cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:26.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.631+0000 7f7af5e00640 1 --2- 172.21.15.39:0/2138558012 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af815aa60 0x7f7af80ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:26.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.632+0000 7f7af5400640 1 -- 172.21.15.39:0/2138558012 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525188869 0 0) 0x7f7af80cbc20 con 0x7f7af80cb280 2024-01-24T17:48:26.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.632+0000 7f7af5400640 1 -- 172.21.15.39:0/2138558012 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad8003520 con 0x7f7af80cb280 2024-01-24T17:48:26.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.632+0000 7f7af6800640 1 -- 172.21.15.39:0/2138558012 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af815aa60 msgr2=0x7f7af80ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:26.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.632+0000 7f7af6800640 1 --2- 172.21.15.39:0/2138558012 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af815aa60 0x7f7af80ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.632+0000 7f7af6800640 1 -- 172.21.15.39:0/2138558012 >> v1:172.21.15.39:6789/0 conn(0x7f7af80cb280 legacy=0x7f7af806cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:26.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.632+0000 7f7af6800640 1 -- 172.21.15.39:0/2138558012 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af806da40 con 0x7f7af806d5d0 2024-01-24T17:48:26.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.632+0000 7f7af5e00640 1 --2- 172.21.15.39:0/2138558012 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af815aa60 0x7f7af80ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:48:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.633+0000 7f7af6800640 1 --2- 172.21.15.39:0/2138558012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af806d5d0 0x7f7af80cb6e0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009d90 tx=0x7f7af002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db5beb81722f18e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7af5400640 1 -- 172.21.15.39:0/2138558012 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af002cc50 con 0x7f7af806d5d0 2024-01-24T17:48:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7af5400640 1 -- 172.21.15.39:0/2138558012 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af002cdb0 con 0x7f7af806d5d0 2024-01-24T17:48:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7af5400640 1 -- 172.21.15.39:0/2138558012 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0034660 con 0x7f7af806d5d0 2024-01-24T17:48:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7afd000640 1 -- 172.21.15.39:0/2138558012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af806d5d0 msgr2=0x7f7af80cb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7afd000640 1 --2- 172.21.15.39:0/2138558012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af806d5d0 0x7f7af80cb6e0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009d90 tx=0x7f7af002bea0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7afd000640 1 -- 172.21.15.39:0/2138558012 shutdown_connections 2024-01-24T17:48:26.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7afd000640 1 --2- 172.21.15.39:0/2138558012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af806d5d0 0x7f7af80cb6e0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7afd000640 1 --2- 172.21.15.39:0/2138558012 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af815aa60 0x7f7af80ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.634+0000 7f7afd000640 1 -- 172.21.15.39:0/2138558012 >> 172.21.15.39:0/2138558012 conn(0x7f7af80ce9e0 msgr2=0x7f7af80cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:26.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.635+0000 7f7afd000640 1 -- 172.21.15.39:0/2138558012 shutdown_connections 2024-01-24T17:48:26.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.635+0000 7f7afd000640 1 -- 172.21.15.39:0/2138558012 wait complete. 2024-01-24T17:48:26.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.636+0000 7f7afd000640 1 Processor -- start 2024-01-24T17:48:26.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.637+0000 7f7afd000640 1 -- start start 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.637+0000 7f7afd000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af806d5d0 0x7f7af81f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.637+0000 7f7afd000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af80cb280 0x7f7af8159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.637+0000 7f7afd000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7af815aa60 0x7f7af81ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.637+0000 7f7afd000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80af370 con 0x7f7af80cb280 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7afd000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81f24d0 con 0x7f7af815aa60 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7afd000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81f2750 con 0x7f7af806d5d0 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af80cb280 0x7f7af8159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af80cb280 0x7f7af8159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49612/0 (socket says 172.21.15.39:49612) 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7aefe00640 1 -- 172.21.15.39:0/239563020 learned_addr learned my addr 172.21.15.39:0/239563020 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7af5e00640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af806d5d0 0x7f7af81f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:26.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7af6800640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7af815aa60 0x7f7af81ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:26.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7aefe00640 1 -- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af806d5d0 msgr2=0x7f7af81f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:26.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7aefe00640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af806d5d0 0x7f7af81f64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7aefe00640 1 -- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7af815aa60 msgr2=0x7f7af81ef7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:26.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7aefe00640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7af815aa60 0x7f7af81ef7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7aefe00640 1 -- 172.21.15.39:0/239563020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af0009a70 con 0x7f7af80cb280 2024-01-24T17:48:26.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.638+0000 7f7af6800640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7af815aa60 0x7f7af81ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:26.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.639+0000 7f7aefe00640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af80cb280 0x7f7af8159ea0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000c500 tx=0x7f7ae000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:26.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.639+0000 7f7aee000640 1 -- 172.21.15.39:0/239563020 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0005ce0 con 0x7f7af80cb280 2024-01-24T17:48:26.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.639+0000 7f7aee000640 1 -- 172.21.15.39:0/239563020 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae0005e40 con 0x7f7af80cb280 2024-01-24T17:48:26.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.639+0000 7f7aee000640 1 -- 172.21.15.39:0/239563020 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0007330 con 0x7f7af80cb280 2024-01-24T17:48:26.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.639+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af81f2a30 con 0x7f7af80cb280 2024-01-24T17:48:26.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.639+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af81f2ff0 con 0x7f7af80cb280 2024-01-24T17:48:26.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.640+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af804dff0 con 0x7f7af80cb280 2024-01-24T17:48:26.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.641+0000 7f7aee000640 1 -- 172.21.15.39:0/239563020 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7ae0007550 con 0x7f7af80cb280 2024-01-24T17:48:26.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.642+0000 7f7aee000640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ad006a430 0x7f7ad006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:26.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.642+0000 7f7aee000640 1 -- 172.21.15.39:0/239563020 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7ae0087ec0 con 0x7f7af80cb280 2024-01-24T17:48:26.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.643+0000 7f7af5e00640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ad006a430 0x7f7ad006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:26.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.644+0000 7f7af5e00640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ad006a430 0x7f7ad006c8f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8006fd0 tx=0x7f7ae8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:26.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.648+0000 7f7aee000640 1 -- 172.21.15.39:0/239563020 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7ae0053660 con 0x7f7af80cb280 2024-01-24T17:48:26.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.877+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7af806ff40 con 0x7f7af80cb280 2024-01-24T17:48:26.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.888+0000 7f7aee000640 1 -- 172.21.15.39:0/239563020 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v55) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7ae0056c80 con 0x7f7af80cb280 2024-01-24T17:48:26.888 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 14 and data pool 15 2024-01-24T17:48:26.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.892+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ad006a430 msgr2=0x7f7ad006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:26.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.892+0000 7f7afd000640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ad006a430 0x7f7ad006c8f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8006fd0 tx=0x7f7ae8008040 comp rx=0 tx=0).stop 2024-01-24T17:48:26.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.892+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af80cb280 msgr2=0x7f7af8159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:26.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.892+0000 7f7afd000640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af80cb280 0x7f7af8159ea0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000c500 tx=0x7f7ae000c9c0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.893+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 shutdown_connections 2024-01-24T17:48:26.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.893+0000 7f7afd000640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7af815aa60 0x7f7af81ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.893+0000 7f7afd000640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7ad006a430 0x7f7ad006c8f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.893+0000 7f7afd000640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7af80cb280 0x7f7af8159ea0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.893+0000 7f7afd000640 1 --2- 172.21.15.39:0/239563020 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7af806d5d0 0x7f7af81f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:26.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.893+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 >> 172.21.15.39:0/239563020 conn(0x7f7af80ce9e0 msgr2=0x7f7af8070160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:26.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.894+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 shutdown_connections 2024-01-24T17:48:26.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:26.894+0000 7f7afd000640 1 -- 172.21.15.39:0/239563020 wait complete. 2024-01-24T17:48:26.920 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:27.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.097+0000 7f2acb200640 1 Processor -- start 2024-01-24T17:48:27.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.097+0000 7f2acb200640 1 -- start start 2024-01-24T17:48:27.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.098+0000 7f2acb200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2acc0b3e00 0x7f2acc0c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.098+0000 7f2acb200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 0x7f2acc15e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.098+0000 7f2acb200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc1551b0 con 0x7f2acc070540 2024-01-24T17:48:27.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.098+0000 7f2acb200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc15e410 con 0x7f2acc0b3e00 2024-01-24T17:48:27.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.098+0000 7f2acb200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc15b480 con 0x7f2acc0c8f70 2024-01-24T17:48:27.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2ac9e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2acc070540 0x7f2acc15ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39402/0 (socket says 172.21.15.39:39402) 2024-01-24T17:48:27.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2ac9e00640 1 -- 172.21.15.39:0/1839950586 learned_addr learned my addr 172.21.15.39:0/1839950586 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:27.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2ac1200640 1 --2- 172.21.15.39:0/1839950586 >> v2:172.21.15.39:3300/0 conn(0x7f2acc0b3e00 0x7f2acc0c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2aca800640 1 --2- 172.21.15.39:0/1839950586 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 0x7f2acc15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2ac1200640 1 -- 172.21.15.39:0/1839950586 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 msgr2=0x7f2acc15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:27.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2ac1200640 1 --2- 172.21.15.39:0/1839950586 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 0x7f2acc15e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2ac9400640 1 -- 172.21.15.39:0/1839950586 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233398882 0 0) 0x7f2acc1551b0 con 0x7f2acc070540 2024-01-24T17:48:27.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2ac1200640 1 -- 172.21.15.39:0/1839950586 >> v1:172.21.15.39:6789/0 conn(0x7f2acc070540 legacy=0x7f2acc15ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.099+0000 7f2ac1200640 1 -- 172.21.15.39:0/1839950586 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc15b760 con 0x7f2acc0b3e00 2024-01-24T17:48:27.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.100+0000 7f2ac1200640 1 --2- 172.21.15.39:0/1839950586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc0c8a30 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40097d0 tx=0x7f2ab402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5967d90e7c188bff server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:27.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2ac9400640 1 -- 172.21.15.39:0/1839950586 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4031800 con 0x7f2acc0b3e00 2024-01-24T17:48:27.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2ac9400640 1 -- 172.21.15.39:0/1839950586 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab4031960 con 0x7f2acc0b3e00 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2ac9400640 1 -- 172.21.15.39:0/1839950586 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4031cf0 con 0x7f2acc0b3e00 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2acb200640 1 -- 172.21.15.39:0/1839950586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 msgr2=0x7f2acc0c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2acb200640 1 --2- 172.21.15.39:0/1839950586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc0c8a30 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40097d0 tx=0x7f2ab402b8a0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2acb200640 1 -- 172.21.15.39:0/1839950586 shutdown_connections 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2acb200640 1 --2- 172.21.15.39:0/1839950586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc0c8a30 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2acb200640 1 --2- 172.21.15.39:0/1839950586 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 0x7f2acc15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.101+0000 7f2acb200640 1 -- 172.21.15.39:0/1839950586 >> 172.21.15.39:0/1839950586 conn(0x7f2acc06a340 msgr2=0x7f2acc0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.102+0000 7f2acb200640 1 -- 172.21.15.39:0/1839950586 shutdown_connections 2024-01-24T17:48:27.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.102+0000 7f2acb200640 1 -- 172.21.15.39:0/1839950586 wait complete. 2024-01-24T17:48:27.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.103+0000 7f2acb200640 1 Processor -- start 2024-01-24T17:48:27.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.103+0000 7f2acb200640 1 -- start start 2024-01-24T17:48:27.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2acb200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2acc070540 0x7f2acc0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2acb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc1599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2acb200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 0x7f2acc149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2acb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0c5790 con 0x7f2acc0b3e00 2024-01-24T17:48:27.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2acb200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0cdf50 con 0x7f2acc070540 2024-01-24T17:48:27.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2acb200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc0cc3d0 con 0x7f2acc0c8f70 2024-01-24T17:48:27.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2ac1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc1599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2ac1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc1599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49628/0 (socket says 172.21.15.39:49628) 2024-01-24T17:48:27.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.104+0000 7f2ac1200640 1 -- 172.21.15.39:0/3820364149 learned_addr learned my addr 172.21.15.39:0/3820364149 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:27.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.105+0000 7f2aca800640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 0x7f2acc149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.105+0000 7f2ac9e00640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2acc070540 0x7f2acc0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.105+0000 7f2ac1200640 1 -- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 msgr2=0x7f2acc149320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.105+0000 7f2ac1200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 0x7f2acc149320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.105+0000 7f2ac1200640 1 -- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2acc070540 msgr2=0x7f2acc0cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.105+0000 7f2ac1200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2acc070540 0x7f2acc0cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.105+0000 7f2ac1200640 1 -- 172.21.15.39:0/3820364149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4009400 con 0x7f2acc0b3e00 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.105+0000 7f2ac1200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc1599e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f2ab402beb0 tx=0x7f2ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.106+0000 7f2ac2a00640 1 -- 172.21.15.39:0/3820364149 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab403bd00 con 0x7f2acc0b3e00 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.106+0000 7f2ac2a00640 1 -- 172.21.15.39:0/3820364149 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab403be60 con 0x7f2acc0b3e00 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.106+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc0cc6b0 con 0x7f2acc0b3e00 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.106+0000 7f2ac2a00640 1 -- 172.21.15.39:0/3820364149 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4045630 con 0x7f2acc0b3e00 2024-01-24T17:48:27.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.106+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc0ccc70 con 0x7f2acc0b3e00 2024-01-24T17:48:27.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.107+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc04dff0 con 0x7f2acc0b3e00 2024-01-24T17:48:27.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.108+0000 7f2ac2a00640 1 -- 172.21.15.39:0/3820364149 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2ab4045790 con 0x7f2acc0b3e00 2024-01-24T17:48:27.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.109+0000 7f2ac2a00640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aa406a430 0x7f2aa406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.109+0000 7f2ac2a00640 1 -- 172.21.15.39:0/3820364149 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ab402e070 con 0x7f2acc0b3e00 2024-01-24T17:48:27.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.109+0000 7f2ac9e00640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aa406a430 0x7f2aa406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.110+0000 7f2ac9e00640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aa406a430 0x7f2aa406c8f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0010e0 tx=0x7f2abc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:27.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.113+0000 7f2ac2a00640 1 -- 172.21.15.39:0/3820364149 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2ab4076650 con 0x7f2acc0b3e00 2024-01-24T17:48:27.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.304+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2acc0b3980 con 0x7f2acc0b3e00 2024-01-24T17:48:27.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.307+0000 7f2ac2a00640 1 -- 172.21.15.39:0/3820364149 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16053 (secure 0 0 0) 0x7f2ab4035020 con 0x7f2acc0b3e00 2024-01-24T17:48:27.308 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:27.308 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":99,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:48:26.879450+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aa406a430 msgr2=0x7f2aa406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aa406a430 0x7f2aa406c8f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0010e0 tx=0x7f2abc009040 comp rx=0 tx=0).stop 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 msgr2=0x7f2acc1599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc1599e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f2ab402beb0 tx=0x7f2ab40046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 shutdown_connections 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2acc0c8f70 0x7f2acc149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2aa406a430 0x7f2aa406c8f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2acc0b3e00 0x7f2acc1599e0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 --2- 172.21.15.39:0/3820364149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2acc070540 0x7f2acc0cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 >> 172.21.15.39:0/3820364149 conn(0x7f2acc06a340 msgr2=0x7f2acc0b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:27.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 shutdown_connections 2024-01-24T17:48:27.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.310+0000 7f2acb200640 1 -- 172.21.15.39:0/3820364149 wait complete. 2024-01-24T17:48:27.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:27.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.543+0000 7fadd3200640 1 Processor -- start 2024-01-24T17:48:27.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.543+0000 7fadd3200640 1 -- start start 2024-01-24T17:48:27.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.544+0000 7fadd3200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fadd40b3e00 0x7fadd40c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.544+0000 7fadd3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fadd40c8f70 0x7fadd415e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.544+0000 7fadd3200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd41551b0 con 0x7fadd4070540 2024-01-24T17:48:27.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.544+0000 7fadd3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd415e410 con 0x7fadd40c8f70 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.544+0000 7fadd3200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd415b480 con 0x7fadd40b3e00 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd1e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fadd4070540 0x7fadd415ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39408/0 (socket says 172.21.15.39:39408) 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd1e00640 1 -- 172.21.15.39:0/2831541754 learned_addr learned my addr 172.21.15.39:0/2831541754 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd2800640 1 --2- 172.21.15.39:0/2831541754 >> v2:172.21.15.39:3300/0 conn(0x7fadd40c8f70 0x7fadd415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd0a00640 1 -- 172.21.15.39:0/2831541754 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926067984 0 0) 0x7fadd41551b0 con 0x7fadd4070540 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd0a00640 1 -- 172.21.15.39:0/2831541754 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb4003520 con 0x7fadd4070540 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd2800640 1 -- 172.21.15.39:0/2831541754 >> v2:172.21.15.100:3300/0 conn(0x7fadd40b3e00 msgr2=0x7fadd40c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd1400640 1 --2- 172.21.15.39:0/2831541754 >> v2:172.21.15.100:3300/0 conn(0x7fadd40b3e00 0x7fadd40c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd2800640 1 --2- 172.21.15.39:0/2831541754 >> v2:172.21.15.100:3300/0 conn(0x7fadd40b3e00 0x7fadd40c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd2800640 1 -- 172.21.15.39:0/2831541754 >> v1:172.21.15.39:6789/0 conn(0x7fadd4070540 legacy=0x7fadd415ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd2800640 1 -- 172.21.15.39:0/2831541754 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd415b760 con 0x7fadd40c8f70 2024-01-24T17:48:27.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.545+0000 7fadd1400640 1 --2- 172.21.15.39:0/2831541754 >> v2:172.21.15.100:3300/0 conn(0x7fadd40b3e00 0x7fadd40c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.547+0000 7fadd2800640 1 --2- 172.21.15.39:0/2831541754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40c8f70 0x7fadd415e910 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009d90 tx=0x7fadcc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2339f951d9576d7d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.547+0000 7fadd0a00640 1 -- 172.21.15.39:0/2831541754 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc02cc60 con 0x7fadd40c8f70 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.547+0000 7fadd0a00640 1 -- 172.21.15.39:0/2831541754 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc02cdc0 con 0x7fadd40c8f70 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.547+0000 7fadd0a00640 1 -- 172.21.15.39:0/2831541754 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc034660 con 0x7fadd40c8f70 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.547+0000 7fadd3200640 1 -- 172.21.15.39:0/2831541754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40c8f70 msgr2=0x7fadd415e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.547+0000 7fadd3200640 1 --2- 172.21.15.39:0/2831541754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40c8f70 0x7fadd415e910 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009d90 tx=0x7fadcc02beb0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.548+0000 7fadd3200640 1 -- 172.21.15.39:0/2831541754 shutdown_connections 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.548+0000 7fadd3200640 1 --2- 172.21.15.39:0/2831541754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40c8f70 0x7fadd415e910 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.548+0000 7fadd3200640 1 --2- 172.21.15.39:0/2831541754 >> v2:172.21.15.100:3300/0 conn(0x7fadd40b3e00 0x7fadd40c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.548+0000 7fadd3200640 1 -- 172.21.15.39:0/2831541754 >> 172.21.15.39:0/2831541754 conn(0x7fadd406a340 msgr2=0x7fadd40cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:27.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.548+0000 7fadd3200640 1 -- 172.21.15.39:0/2831541754 shutdown_connections 2024-01-24T17:48:27.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.548+0000 7fadd3200640 1 -- 172.21.15.39:0/2831541754 wait complete. 2024-01-24T17:48:27.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.549+0000 7fadd3200640 1 Processor -- start 2024-01-24T17:48:27.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.549+0000 7fadd3200640 1 -- start start 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.550+0000 7fadd3200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd4070540 0x7fadd40c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.550+0000 7fadd3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40b3e00 0x7fadd41f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.550+0000 7fadd3200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd40c8f70 0x7fadd4159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.550+0000 7fadd3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40c2c40 con 0x7fadd40b3e00 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.550+0000 7fadd3200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40c2f20 con 0x7fadd40c8f70 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.550+0000 7fadd3200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd40c3200 con 0x7fadd4070540 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40b3e00 0x7fadd41f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40b3e00 0x7fadd41f4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49654/0 (socket says 172.21.15.39:49654) 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 -- 172.21.15.39:0/2304343473 learned_addr learned my addr 172.21.15.39:0/2304343473 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1e00640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd4070540 0x7fadd40c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 -- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd4070540 msgr2=0x7fadd40c3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd2800640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd40c8f70 0x7fadd4159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd4070540 0x7fadd40c3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 -- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd40c8f70 msgr2=0x7fadd4159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd40c8f70 0x7fadd4159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 -- 172.21.15.39:0/2304343473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc009a70 con 0x7fadd40b3e00 2024-01-24T17:48:27.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1e00640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd4070540 0x7fadd40c3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:27.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.551+0000 7fadd1400640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40b3e00 0x7fadd41f4bb0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00c7f0 tx=0x7fadbc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:27.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.552+0000 7fadca000640 1 -- 172.21.15.39:0/2304343473 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00cea0 con 0x7fadd40b3e00 2024-01-24T17:48:27.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.552+0000 7fadca000640 1 -- 172.21.15.39:0/2304343473 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc004230 con 0x7fadd40b3e00 2024-01-24T17:48:27.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.552+0000 7fadca000640 1 -- 172.21.15.39:0/2304343473 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc007830 con 0x7fadd40b3e00 2024-01-24T17:48:27.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.552+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd40c34e0 con 0x7fadd40b3e00 2024-01-24T17:48:27.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.552+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd41f50f0 con 0x7fadd40b3e00 2024-01-24T17:48:27.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.553+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd404dff0 con 0x7fadd40b3e00 2024-01-24T17:48:27.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.554+0000 7fadca000640 1 -- 172.21.15.39:0/2304343473 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fadbc007990 con 0x7fadd40b3e00 2024-01-24T17:48:27.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.555+0000 7fadca000640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadac06a460 0x7fadac06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.555+0000 7fadca000640 1 -- 172.21.15.39:0/2304343473 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadbc088900 con 0x7fadd40b3e00 2024-01-24T17:48:27.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.556+0000 7fadd1e00640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadac06a460 0x7fadac06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.557+0000 7fadd1e00640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadac06a460 0x7fadac06c920 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fadc4007fd0 tx=0x7fadc4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:27.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.559+0000 7fadca000640 1 -- 172.21.15.39:0/2304343473 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fadbc053e50 con 0x7fadd40b3e00 2024-01-24T17:48:27.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.746+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadd40cbc10 con 0x7fadd40b3e00 2024-01-24T17:48:27.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.748+0000 7fadca000640 1 -- 172.21.15.39:0/2304343473 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16053 (secure 0 0 0) 0x7fadbc01e090 con 0x7fadd40b3e00 2024-01-24T17:48:27.749 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:27.749 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":99,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:48:26.879450+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:27.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadac06a460 msgr2=0x7fadac06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadac06a460 0x7fadac06c920 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fadc4007fd0 tx=0x7fadc4009040 comp rx=0 tx=0).stop 2024-01-24T17:48:27.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40b3e00 msgr2=0x7fadd41f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40b3e00 0x7fadd41f4bb0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00c7f0 tx=0x7fadbc00ccb0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 shutdown_connections 2024-01-24T17:48:27.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fadd40c8f70 0x7fadd4159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadac06a460 0x7fadac06c920 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd40b3e00 0x7fadd41f4bb0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 --2- 172.21.15.39:0/2304343473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fadd4070540 0x7fadd40c3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 >> 172.21.15.39:0/2304343473 conn(0x7fadd406a340 msgr2=0x7fadd40adec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:27.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 shutdown_connections 2024-01-24T17:48:27.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.750+0000 7fadd3200640 1 -- 172.21.15.39:0/2304343473 wait complete. 2024-01-24T17:48:27.770 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T17:48:27.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.987+0000 7f0fd2000640 1 Processor -- start 2024-01-24T17:48:27.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.987+0000 7f0fd2000640 1 -- start start 2024-01-24T17:48:27.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.989+0000 7f0fd2000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0fcc15a9a0 0x7f0fcc06bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:27.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.989+0000 7f0fd2000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fcc15b040 con 0x7f0fcc0cadc0 2024-01-24T17:48:27.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.989+0000 7f0fd2000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fcc15bbe0 con 0x7f0fcc0ca6d0 2024-01-24T17:48:27.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.989+0000 7f0fd2000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fcc06c420 con 0x7f0fcc15a9a0 2024-01-24T17:48:27.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.989+0000 7f0fcbe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0fcc15a9a0 0x7f0fcc06bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:27.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.989+0000 7f0fd0c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0fcc0cadc0 0x7f0fcc0cb1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39416/0 (socket says 172.21.15.39:39416) 2024-01-24T17:48:27.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.989+0000 7f0fd0c00640 1 -- 172.21.15.39:0/2911299026 learned_addr learned my addr 172.21.15.39:0/2911299026 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:27.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.990+0000 7f0fcbe00640 1 -- 172.21.15.39:0/2911299026 >> v1:172.21.15.100:6789/0 conn(0x7f0fcc0ca6d0 legacy=0x7f0fcc15b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:27.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.990+0000 7f0fcbe00640 1 -- 172.21.15.39:0/2911299026 >> v1:172.21.15.39:6789/0 conn(0x7f0fcc0cadc0 legacy=0x7f0fcc0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.990+0000 7f0fcb400640 1 -- 172.21.15.39:0/2911299026 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183394442 0 0) 0x7f0fcc15b040 con 0x7f0fcc0cadc0 2024-01-24T17:48:27.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.990+0000 7f0fcbe00640 1 -- 172.21.15.39:0/2911299026 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc06c700 con 0x7f0fcc15a9a0 2024-01-24T17:48:27.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.991+0000 7f0fcbe00640 1 --2- 172.21.15.39:0/2911299026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc15a9a0 0x7f0fcc06bd40 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40097a0 tx=0x7f0fb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5373a91e4f30f66 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:27.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.992+0000 7f0fcb400640 1 -- 172.21.15.39:0/2911299026 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4031830 con 0x7f0fcc15a9a0 2024-01-24T17:48:27.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.992+0000 7f0fcb400640 1 -- 172.21.15.39:0/2911299026 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb4031990 con 0x7f0fcc15a9a0 2024-01-24T17:48:27.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.992+0000 7f0fcb400640 1 -- 172.21.15.39:0/2911299026 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4031d20 con 0x7f0fcc15a9a0 2024-01-24T17:48:27.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.992+0000 7f0fd2000640 1 -- 172.21.15.39:0/2911299026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc15a9a0 msgr2=0x7f0fcc06bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:27.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.992+0000 7f0fd2000640 1 --2- 172.21.15.39:0/2911299026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc15a9a0 0x7f0fcc06bd40 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40097a0 tx=0x7f0fb402b8d0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.992+0000 7f0fd2000640 1 -- 172.21.15.39:0/2911299026 shutdown_connections 2024-01-24T17:48:27.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.992+0000 7f0fd2000640 1 --2- 172.21.15.39:0/2911299026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc15a9a0 0x7f0fcc06bd40 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:27.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.992+0000 7f0fd2000640 1 -- 172.21.15.39:0/2911299026 >> 172.21.15.39:0/2911299026 conn(0x7f0fcc06a200 msgr2=0x7f0fcc0aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:27.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.993+0000 7f0fd2000640 1 -- 172.21.15.39:0/2911299026 shutdown_connections 2024-01-24T17:48:27.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.993+0000 7f0fd2000640 1 -- 172.21.15.39:0/2911299026 wait complete. 2024-01-24T17:48:27.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.994+0000 7f0fd2000640 1 Processor -- start 2024-01-24T17:48:27.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.994+0000 7f0fd2000640 1 -- start start 2024-01-24T17:48:28.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.995+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc0ca6d0 0x7f0fcc157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:28.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.995+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc0cadc0 0x7f0fcc0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:28.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.995+0000 7f0fd2000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc15a9a0 0x7f0fcc1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:28.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.995+0000 7f0fd2000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0b3310 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.995+0000 7f0fd2000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc159130 con 0x7f0fcc0ca6d0 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.995+0000 7f0fd2000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc0b2700 con 0x7f0fcc15a9a0 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.995+0000 7f0fcbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc0cadc0 0x7f0fcc0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fcbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc0cadc0 0x7f0fcc0b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49684/0 (socket says 172.21.15.39:49684) 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fcbe00640 1 -- 172.21.15.39:0/2748780041 learned_addr learned my addr 172.21.15.39:0/2748780041 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fd0c00640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc0ca6d0 0x7f0fcc157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fd1600640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc15a9a0 0x7f0fcc1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fcbe00640 1 -- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc15a9a0 msgr2=0x7f0fcc1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fcbe00640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc15a9a0 0x7f0fcc1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fcbe00640 1 -- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc0ca6d0 msgr2=0x7f0fcc157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fcbe00640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc0ca6d0 0x7f0fcc157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fcbe00640 1 -- 172.21.15.39:0/2748780041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4009400 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fd1600640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc15a9a0 0x7f0fcc1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.996+0000 7f0fcbe00640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc0cadc0 0x7f0fcc0b2020 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40097a0 tx=0x7f0fb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.997+0000 7f0fc8c00640 1 -- 172.21.15.39:0/2748780041 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4004030 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.997+0000 7f0fc8c00640 1 -- 172.21.15.39:0/2748780041 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb40041b0 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.997+0000 7f0fc8c00640 1 -- 172.21.15.39:0/2748780041 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb403b7e0 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.997+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc0b29e0 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.997+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc0b2fa0 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.998+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc04dff0 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:27.999+0000 7f0fc8c00640 1 -- 172.21.15.39:0/2748780041 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0fb4043070 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.000+0000 7f0fc8c00640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0fa406a430 0x7f0fa406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:28.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.000+0000 7f0fc8c00640 1 -- 172.21.15.39:0/2748780041 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fb40aa7c0 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.000+0000 7f0fd0c00640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0fa406a430 0x7f0fa406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:28.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.001+0000 7f0fd0c00640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0fa406a430 0x7f0fa406c8f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0010e0 tx=0x7f0fbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:28.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.005+0000 7f0fc8c00640 1 -- 172.21.15.39:0/2748780041 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0fb4075d10 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.235+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0fcc0b01d0 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.910+0000 7f0fc8c00640 1 -- 172.21.15.39:0/2748780041 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v58) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0fb40350b0 con 0x7f0fcc0cadc0 2024-01-24T17:48:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0fa406a430 msgr2=0x7f0fa406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0fa406a430 0x7f0fa406c8f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0010e0 tx=0x7f0fbc009040 comp rx=0 tx=0).stop 2024-01-24T17:48:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc0cadc0 msgr2=0x7f0fcc0b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc0cadc0 0x7f0fcc0b2020 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40097a0 tx=0x7f0fb40046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 shutdown_connections 2024-01-24T17:48:28.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0fcc15a9a0 0x7f0fcc1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0fa406a430 0x7f0fa406c8f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0fcc0cadc0 0x7f0fcc0b2020 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 --2- 172.21.15.39:0/2748780041 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0fcc0ca6d0 0x7f0fcc157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.913+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 >> 172.21.15.39:0/2748780041 conn(0x7f0fcc06a200 msgr2=0x7f0fcc15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.914+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 shutdown_connections 2024-01-24T17:48:28.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:28.914+0000 7f0fd2000640 1 -- 172.21.15.39:0/2748780041 wait complete. 2024-01-24T17:48:28.935 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-24T17:48:28.935 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-24T17:48:29.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.153+0000 7fc025200640 1 Processor -- start 2024-01-24T17:48:29.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.153+0000 7fc025200640 1 -- start start 2024-01-24T17:48:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.154+0000 7fc025200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc020159740 0x7fc02015db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.154+0000 7fc025200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200ccbe0 0x7fc0200c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.154+0000 7fc025200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02015e0d0 con 0x7fc02006bd40 2024-01-24T17:48:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.154+0000 7fc025200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02015d140 con 0x7fc020159740 2024-01-24T17:48:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.154+0000 7fc025200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02015d700 con 0x7fc0200ccbe0 2024-01-24T17:48:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc02006bd40 0x7fc020070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39426/0 (socket says 172.21.15.39:39426) 2024-01-24T17:48:29.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01fe00640 1 -- 172.21.15.39:0/2949189540 learned_addr learned my addr 172.21.15.39:0/2949189540 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:29.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01f400640 1 --2- 172.21.15.39:0/2949189540 >> v2:172.21.15.100:3300/0 conn(0x7fc020159740 0x7fc02015db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01ea00640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200ccbe0 0x7fc0200c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191020285 0 0) 0x7fc018001110 con 0x7fc02006bd40 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc000003520 con 0x7fc02006bd40 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694474971 0 0) 0x7fc000003520 con 0x7fc02006bd40 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200ccbe0 msgr2=0x7fc0200c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200ccbe0 0x7fc0200c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 >> v2:172.21.15.100:3300/0 conn(0x7fc020159740 msgr2=0x7fc02015db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 --2- 172.21.15.39:0/2949189540 >> v2:172.21.15.100:3300/0 conn(0x7fc020159740 0x7fc02015db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0200ca500 con 0x7fc02006bd40 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.155+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc018001800 con 0x7fc02006bd40 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 >> v1:172.21.15.39:6789/0 conn(0x7fc02006bd40 legacy=0x7fc020070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01e000640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc000003970 0x7fc000003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01e000640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01e000640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200ccbe0 0x7fc000005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02015d140 con 0x7fc0000047c0 2024-01-24T17:48:29.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02015d700 con 0x7fc000003970 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00000db90 con 0x7fc0200ccbe0 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01ea00640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01f400640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc000003970 0x7fc000003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01fe00640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200ccbe0 0x7fc000005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01ea00640 1 -- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200ccbe0 msgr2=0x7fc000005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01ea00640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200ccbe0 0x7fc000005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01ea00640 1 -- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc000003970 msgr2=0x7fc000003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01ea00640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc000003970 0x7fc000003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01ea00640 1 -- 172.21.15.39:0/2949189540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc000003520 con 0x7fc0000047c0 2024-01-24T17:48:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.156+0000 7fc01f400640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc000003970 0x7fc000003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:29.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.157+0000 7fc01ea00640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fc008009ac0 tx=0x7fc0000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc608546581c684c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:29.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.158+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00802c6b0 con 0x7fc0000047c0 2024-01-24T17:48:29.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.158+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00802c810 con 0x7fc0000047c0 2024-01-24T17:48:29.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.158+0000 7fc01e000640 1 -- 172.21.15.39:0/2949189540 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00802cba0 con 0x7fc0000047c0 2024-01-24T17:48:29.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.158+0000 7fc025200640 1 -- 172.21.15.39:0/2949189540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 msgr2=0x7fc000004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:29.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.158+0000 7fc025200640 1 --2- 172.21.15.39:0/2949189540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc000004c40 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fc008009ac0 tx=0x7fc0000057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:29.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.158+0000 7fc01fe00640 1 -- 172.21.15.39:0/2949189540 reap_dead start 2024-01-24T17:48:29.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.158+0000 7fc025200640 1 -- 172.21.15.39:0/2949189540 shutdown_connections 2024-01-24T17:48:29.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.158+0000 7fc025200640 1 -- 172.21.15.39:0/2949189540 >> 172.21.15.39:0/2949189540 conn(0x7fc02006a200 msgr2=0x7fc0200aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:29.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.159+0000 7fc025200640 1 -- 172.21.15.39:0/2949189540 shutdown_connections 2024-01-24T17:48:29.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.159+0000 7fc025200640 1 -- 172.21.15.39:0/2949189540 wait complete. 2024-01-24T17:48:29.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.160+0000 7fc025200640 1 Processor -- start 2024-01-24T17:48:29.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.161+0000 7fc025200640 1 -- start start 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.161+0000 7fc025200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc02014c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.161+0000 7fc025200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0201f33d0 0x7fc0200ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc025200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200abe00 0x7fc0200c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc025200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200c7190 con 0x7fc0000047c0 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc025200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02014cce0 con 0x7fc0201f33d0 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc025200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200ac380 con 0x7fc0200abe00 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc01f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc02014c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc01f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc02014c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49708/0 (socket says 172.21.15.39:49708) 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc01f400640 1 -- 172.21.15.39:0/2259775355 learned_addr learned my addr 172.21.15.39:0/2259775355 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:29.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc01f400640 1 -- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200abe00 msgr2=0x7fc0200c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:29.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc01f400640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200abe00 0x7fc0200c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:29.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc01f400640 1 -- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0201f33d0 msgr2=0x7fc0200ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:29.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc01f400640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0201f33d0 0x7fc0200ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:29.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.162+0000 7fc01f400640 1 -- 172.21.15.39:0/2259775355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0200ac660 con 0x7fc0000047c0 2024-01-24T17:48:29.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.163+0000 7fc01f400640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc02014c7a0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fc01000f3d0 tx=0x7fc01000f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:29.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.163+0000 7fc017400640 1 -- 172.21.15.39:0/2259775355 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01001a070 con 0x7fc0000047c0 2024-01-24T17:48:29.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.163+0000 7fc017400640 1 -- 172.21.15.39:0/2259775355 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc010003990 con 0x7fc0000047c0 2024-01-24T17:48:29.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.163+0000 7fc017400640 1 -- 172.21.15.39:0/2259775355 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010003d00 con 0x7fc0000047c0 2024-01-24T17:48:29.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.163+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0200ac940 con 0x7fc0000047c0 2024-01-24T17:48:29.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.163+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0200c6cb0 con 0x7fc0000047c0 2024-01-24T17:48:29.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.164+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0200c7400 con 0x7fc0000047c0 2024-01-24T17:48:29.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.165+0000 7fc017400640 1 -- 172.21.15.39:0/2259775355 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc010004020 con 0x7fc0000047c0 2024-01-24T17:48:29.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.166+0000 7fc017400640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbff806a430 0x7fbff806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:29.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.167+0000 7fc017400640 1 -- 172.21.15.39:0/2259775355 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc010089480 con 0x7fc0000047c0 2024-01-24T17:48:29.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.167+0000 7fc01ea00640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbff806a430 0x7fbff806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:29.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.168+0000 7fc01ea00640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbff806a430 0x7fbff806c8f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc008036000 tx=0x7fc00802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:29.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.171+0000 7fc017400640 1 -- 172.21.15.39:0/2259775355 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc010054a60 con 0x7fc0000047c0 2024-01-24T17:48:29.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:29.400+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc0200c6ea0 con 0x7fc0000047c0 2024-01-24T17:48:30.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.331+0000 7fc017400640 1 -- 172.21.15.39:0/2259775355 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc010003e60 con 0x7fc0000047c0 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbff806a430 msgr2=0x7fbff806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbff806a430 0x7fbff806c8f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc008036000 tx=0x7fc00802e000 comp rx=0 tx=0).stop 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 msgr2=0x7fc02014c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc02014c7a0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fc01000f3d0 tx=0x7fc01000f890 comp rx=0 tx=0).stop 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 shutdown_connections 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc0200abe00 0x7fc0200c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc0201f33d0 0x7fc0200ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbff806a430 0x7fbff806c8f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 --2- 172.21.15.39:0/2259775355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc0000047c0 0x7fc02014c7a0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 >> 172.21.15.39:0/2259775355 conn(0x7fc02006a200 msgr2=0x7fc020071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 shutdown_connections 2024-01-24T17:48:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.336+0000 7fc025200640 1 -- 172.21.15.39:0/2259775355 wait complete. 2024-01-24T17:48:30.361 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-24T17:48:30.361 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:30.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.590+0000 7f6b06800640 1 Processor -- start 2024-01-24T17:48:30.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.590+0000 7f6b06800640 1 -- start start 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.592+0000 7f6b06800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6b00159d00 0x7f6b000cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.592+0000 7f6b06800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b000b3f20 0x7f6b000714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.592+0000 7f6b06800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b000ccd10 con 0x7f6b00071a30 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.592+0000 7f6b06800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b000ce230 con 0x7f6b00159d00 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.592+0000 7f6b06800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000ccff0 con 0x7f6b000b3f20 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.592+0000 7f6affe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f6b00071a30 0x7f6b000c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39436/0 (socket says 172.21.15.39:39436) 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.592+0000 7f6affe00640 1 -- 172.21.15.39:0/3675483021 learned_addr learned my addr 172.21.15.39:0/3675483021 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.592+0000 7f6aff400640 1 --2- 172.21.15.39:0/3675483021 >> v2:172.21.15.39:3300/0 conn(0x7f6b00159d00 0x7f6b000cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:30.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.593+0000 7f6af6800640 1 --2- 172.21.15.39:0/3675483021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b000b3f20 0x7f6b000714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:30.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.593+0000 7f6afea00640 1 -- 172.21.15.39:0/3675483021 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792894294 0 0) 0x7f6b000ccd10 con 0x7f6b00071a30 2024-01-24T17:48:30.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.593+0000 7f6aff400640 1 -- 172.21.15.39:0/3675483021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b000b3f20 msgr2=0x7f6b000714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:30.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.593+0000 7f6aff400640 1 --2- 172.21.15.39:0/3675483021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b000b3f20 0x7f6b000714f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.593+0000 7f6aff400640 1 -- 172.21.15.39:0/3675483021 >> v1:172.21.15.39:6789/0 conn(0x7f6b00071a30 legacy=0x7f6b000c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:30.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.593+0000 7f6aff400640 1 -- 172.21.15.39:0/3675483021 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000ad720 con 0x7f6b00159d00 2024-01-24T17:48:30.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.594+0000 7f6aff400640 1 --2- 172.21.15.39:0/3675483021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00159d00 0x7f6b000cdcf0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097a0 tx=0x7f6af002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436c63eb7477b055 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.594+0000 7f6afea00640 1 -- 172.21.15.39:0/3675483021 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0031830 con 0x7f6b00159d00 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.594+0000 7f6afea00640 1 -- 172.21.15.39:0/3675483021 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af0031990 con 0x7f6b00159d00 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.594+0000 7f6afea00640 1 -- 172.21.15.39:0/3675483021 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0031d20 con 0x7f6b00159d00 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.595+0000 7f6b06800640 1 -- 172.21.15.39:0/3675483021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00159d00 msgr2=0x7f6b000cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.595+0000 7f6b06800640 1 --2- 172.21.15.39:0/3675483021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00159d00 0x7f6b000cdcf0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097a0 tx=0x7f6af002b8d0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.595+0000 7f6b06800640 1 -- 172.21.15.39:0/3675483021 shutdown_connections 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.595+0000 7f6b06800640 1 --2- 172.21.15.39:0/3675483021 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b000b3f20 0x7f6b000714f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.595+0000 7f6b06800640 1 --2- 172.21.15.39:0/3675483021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00159d00 0x7f6b000cdcf0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.595+0000 7f6b06800640 1 -- 172.21.15.39:0/3675483021 >> 172.21.15.39:0/3675483021 conn(0x7f6b0006bf20 msgr2=0x7f6b0006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.595+0000 7f6b06800640 1 -- 172.21.15.39:0/3675483021 shutdown_connections 2024-01-24T17:48:30.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.595+0000 7f6b06800640 1 -- 172.21.15.39:0/3675483021 wait complete. 2024-01-24T17:48:30.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.597+0000 7f6b06800640 1 Processor -- start 2024-01-24T17:48:30.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.597+0000 7f6b06800640 1 -- start start 2024-01-24T17:48:30.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6b06800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00071a30 0x7f6b001edb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:30.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6b06800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b000b3f20 0x7f6b000c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:30.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6b06800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b00159d00 0x7f6b000c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:30.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6b06800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000c6f80 con 0x7f6b00071a30 2024-01-24T17:48:30.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6b06800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000c6370 con 0x7f6b000b3f20 2024-01-24T17:48:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6b06800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000c6650 con 0x7f6b00159d00 2024-01-24T17:48:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6aff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00071a30 0x7f6b001edb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6aff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00071a30 0x7f6b001edb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49728/0 (socket says 172.21.15.39:49728) 2024-01-24T17:48:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6aff400640 1 -- 172.21.15.39:0/82169413 learned_addr learned my addr 172.21.15.39:0/82169413 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.598+0000 7f6affe00640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b00159d00 0x7f6b000c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6af6800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b000b3f20 0x7f6b000c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:30.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6aff400640 1 -- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b00159d00 msgr2=0x7f6b000c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:30.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6aff400640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b00159d00 0x7f6b000c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6aff400640 1 -- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b000b3f20 msgr2=0x7f6b000c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:30.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6aff400640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b000b3f20 0x7f6b000c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6aff400640 1 -- 172.21.15.39:0/82169413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0009400 con 0x7f6b00071a30 2024-01-24T17:48:30.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6af6800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b000b3f20 0x7f6b000c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:30.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6aff400640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00071a30 0x7f6b001edb80 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097a0 tx=0x7f6af00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:30.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6af7e00640 1 -- 172.21.15.39:0/82169413 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0004230 con 0x7f6b00071a30 2024-01-24T17:48:30.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6af7e00640 1 -- 172.21.15.39:0/82169413 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af00043b0 con 0x7f6b00071a30 2024-01-24T17:48:30.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6af7e00640 1 -- 172.21.15.39:0/82169413 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0004520 con 0x7f6b00071a30 2024-01-24T17:48:30.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.599+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b000c6930 con 0x7f6b00071a30 2024-01-24T17:48:30.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.600+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b001ee0c0 con 0x7f6b00071a30 2024-01-24T17:48:30.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.601+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0004dff0 con 0x7f6b00071a30 2024-01-24T17:48:30.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.601+0000 7f6af7e00640 1 -- 172.21.15.39:0/82169413 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6af003a8a0 con 0x7f6b00071a30 2024-01-24T17:48:30.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.602+0000 7f6af7e00640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ad806a430 0x7f6ad806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:30.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.602+0000 7f6af7e00640 1 -- 172.21.15.39:0/82169413 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6af00aafe0 con 0x7f6b00071a30 2024-01-24T17:48:30.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.603+0000 7f6af6800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ad806a430 0x7f6ad806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:30.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.604+0000 7f6af6800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ad806a430 0x7f6ad806c8f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6b00070060 tx=0x7f6ae8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:30.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.607+0000 7f6af7e00640 1 -- 172.21.15.39:0/82169413 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6af00766b0 con 0x7f6b00071a30 2024-01-24T17:48:30.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.839+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b0006c510 con 0x7f6b00071a30 2024-01-24T17:48:30.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.841+0000 7f6af7e00640 1 -- 172.21.15.39:0/82169413 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f6af0079cd0 con 0x7f6b00071a30 2024-01-24T17:48:30.842 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:30.842 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5717,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1968583805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1968583805},{"type":"v1","addr":"172.21.15.100:6837","nonce":1968583805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":6436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/872967797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":872967797},{"type":"v1","addr":"172.21.15.39:6833","nonce":872967797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3116066709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3116066709},{"type":"v1","addr":"172.21.15.100:6836","nonce":3116066709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:48:30.324664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7761":{"gid":7761,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3495904204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3495904204},{"type":"v1","addr":"172.21.15.39:6835","nonce":3495904204}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:48:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 59 2024-01-24T17:48:30.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ad806a430 msgr2=0x7f6ad806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:30.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ad806a430 0x7f6ad806c8f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6b00070060 tx=0x7f6ae8008040 comp rx=0 tx=0).stop 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00071a30 msgr2=0x7f6b001edb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00071a30 0x7f6b001edb80 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097a0 tx=0x7f6af00046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 shutdown_connections 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6b00159d00 0x7f6b000c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6b000b3f20 0x7f6b000c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6ad806a430 0x7f6ad806c8f0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 --2- 172.21.15.39:0/82169413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b00071a30 0x7f6b001edb80 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 >> 172.21.15.39:0/82169413 conn(0x7f6b0006bf20 msgr2=0x7f6b000cbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:30.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 shutdown_connections 2024-01-24T17:48:30.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:30.844+0000 7f6b06800640 1 -- 172.21.15.39:0/82169413 wait complete. 2024-01-24T17:48:30.864 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:31.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.083+0000 7ff6b5400640 1 Processor -- start 2024-01-24T17:48:31.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.084+0000 7ff6b5400640 1 -- start start 2024-01-24T17:48:31.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.085+0000 7ff6b5400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6b015cbe0 0x7ff6b00cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.085+0000 7ff6b5400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff6b00ce510 0x7ff6b00d0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.085+0000 7ff6b5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 0x7ff6b0160820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.085+0000 7ff6b5400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00d1840 con 0x7ff6b015cbe0 2024-01-24T17:48:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.085+0000 7ff6b5400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00b8820 con 0x7ff6b00ce510 2024-01-24T17:48:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.085+0000 7ff6b5400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b00b5790 con 0x7ff6b00d13d0 2024-01-24T17:48:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.086+0000 7ff6ae800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6b015cbe0 0x7ff6b00cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.086+0000 7ff6ae800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6b015cbe0 0x7ff6b00cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49732/0 (socket says 172.21.15.39:49732) 2024-01-24T17:48:31.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.086+0000 7ff6ae800640 1 -- 172.21.15.39:0/327900369 learned_addr learned my addr 172.21.15.39:0/327900369 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.086+0000 7ff6af200640 1 --2- 172.21.15.39:0/327900369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 0x7ff6b0160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.086+0000 7ff6ade00640 1 --2- 172.21.15.39:0/327900369 >> v2:172.21.15.100:3300/0 conn(0x7ff6b00ce510 0x7ff6b00d0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.086+0000 7ff6ae800640 1 -- 172.21.15.39:0/327900369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 msgr2=0x7ff6b0160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.086+0000 7ff6ae800640 1 --2- 172.21.15.39:0/327900369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 0x7ff6b0160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.086+0000 7ff6ae800640 1 -- 172.21.15.39:0/327900369 >> v2:172.21.15.100:3300/0 conn(0x7ff6b00ce510 msgr2=0x7ff6b00d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.087+0000 7ff6ae800640 1 --2- 172.21.15.39:0/327900369 >> v2:172.21.15.100:3300/0 conn(0x7ff6b00ce510 0x7ff6b00d0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.087+0000 7ff6ae800640 1 -- 172.21.15.39:0/327900369 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b00b5a50 con 0x7ff6b015cbe0 2024-01-24T17:48:31.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.088+0000 7ff6ae800640 1 --2- 172.21.15.39:0/327900369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b015cbe0 0x7ff6b00cdfd0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7ff6b008bdb0 tx=0x7ff6a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7011132721e9901 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:31.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.089+0000 7ff6ad400640 1 -- 172.21.15.39:0/327900369 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0004230 con 0x7ff6b015cbe0 2024-01-24T17:48:31.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.089+0000 7ff6ad400640 1 -- 172.21.15.39:0/327900369 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a0004390 con 0x7ff6b015cbe0 2024-01-24T17:48:31.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.089+0000 7ff6ad400640 1 -- 172.21.15.39:0/327900369 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0003810 con 0x7ff6b015cbe0 2024-01-24T17:48:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.089+0000 7ff6b5400640 1 -- 172.21.15.39:0/327900369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b015cbe0 msgr2=0x7ff6b00cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.089+0000 7ff6b5400640 1 --2- 172.21.15.39:0/327900369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b015cbe0 0x7ff6b00cdfd0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7ff6b008bdb0 tx=0x7ff6a002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.090+0000 7ff6b5400640 1 -- 172.21.15.39:0/327900369 shutdown_connections 2024-01-24T17:48:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.090+0000 7ff6b5400640 1 --2- 172.21.15.39:0/327900369 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 0x7ff6b0160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.090+0000 7ff6b5400640 1 --2- 172.21.15.39:0/327900369 >> v2:172.21.15.100:3300/0 conn(0x7ff6b00ce510 0x7ff6b00d0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.090+0000 7ff6b5400640 1 --2- 172.21.15.39:0/327900369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b015cbe0 0x7ff6b00cdfd0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.090+0000 7ff6b5400640 1 -- 172.21.15.39:0/327900369 >> 172.21.15.39:0/327900369 conn(0x7ff6b009bc80 msgr2=0x7ff6b00d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.090+0000 7ff6b5400640 1 -- 172.21.15.39:0/327900369 shutdown_connections 2024-01-24T17:48:31.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.090+0000 7ff6b5400640 1 -- 172.21.15.39:0/327900369 wait complete. 2024-01-24T17:48:31.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.092+0000 7ff6b5400640 1 Processor -- start 2024-01-24T17:48:31.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.092+0000 7ff6b5400640 1 -- start start 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6b5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b00ce510 0x7ff6b014d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6b5400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 0x7ff6b00c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6b5400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6b015cbe0 0x7ff6b006c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6b5400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b014db10 con 0x7ff6b00ce510 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6b5400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b014ab30 con 0x7ff6b015cbe0 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6b5400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b014add0 con 0x7ff6b00d13d0 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6ae800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b00ce510 0x7ff6b014d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6ae800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b00ce510 0x7ff6b014d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49738/0 (socket says 172.21.15.39:49738) 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6ae800640 1 -- 172.21.15.39:0/355768007 learned_addr learned my addr 172.21.15.39:0/355768007 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6ae800640 1 -- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 msgr2=0x7ff6b00c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6ae800640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 0x7ff6b00c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6ae800640 1 -- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6b015cbe0 msgr2=0x7ff6b006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6ae800640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6b015cbe0 0x7ff6b006c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.094+0000 7ff6ae800640 1 -- 172.21.15.39:0/355768007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a0009460 con 0x7ff6b00ce510 2024-01-24T17:48:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.095+0000 7ff6ae800640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b00ce510 0x7ff6b014d5d0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0000c00 tx=0x7ff6a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:31.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.095+0000 7ff6a6a00640 1 -- 172.21.15.39:0/355768007 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0003e50 con 0x7ff6b00ce510 2024-01-24T17:48:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.095+0000 7ff6a6a00640 1 -- 172.21.15.39:0/355768007 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a0004500 con 0x7ff6b00ce510 2024-01-24T17:48:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.095+0000 7ff6a6a00640 1 -- 172.21.15.39:0/355768007 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0006280 con 0x7ff6b00ce510 2024-01-24T17:48:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.095+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b014b0b0 con 0x7ff6b00ce510 2024-01-24T17:48:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.095+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b014b670 con 0x7ff6b00ce510 2024-01-24T17:48:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.096+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b004dff0 con 0x7ff6b00ce510 2024-01-24T17:48:31.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.097+0000 7ff6a6a00640 1 -- 172.21.15.39:0/355768007 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff6a002f050 con 0x7ff6b00ce510 2024-01-24T17:48:31.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.098+0000 7ff6a6a00640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff68806a430 0x7ff68806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.098+0000 7ff6a6a00640 1 -- 172.21.15.39:0/355768007 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6a00aac70 con 0x7ff6b00ce510 2024-01-24T17:48:31.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.099+0000 7ff6ade00640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff68806a430 0x7ff68806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.100+0000 7ff6ade00640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff68806a430 0x7ff68806c8f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff6b009c6a0 tx=0x7ff698008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:31.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.105+0000 7ff6a6a00640 1 -- 172.21.15.39:0/355768007 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff6a00761c0 con 0x7ff6b00ce510 2024-01-24T17:48:31.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.312+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6b00d0c70 con 0x7ff6b00ce510 2024-01-24T17:48:31.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.315+0000 7ff6a6a00640 1 -- 172.21.15.39:0/355768007 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16042 (secure 0 0 0) 0x7ff6a003f090 con 0x7ff6b00ce510 2024-01-24T17:48:31.316 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:31.316 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":100,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:48:27.885313+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff68806a430 msgr2=0x7ff68806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff68806a430 0x7ff68806c8f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff6b009c6a0 tx=0x7ff698008040 comp rx=0 tx=0).stop 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b00ce510 msgr2=0x7ff6b014d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b00ce510 0x7ff6b014d5d0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0000c00 tx=0x7ff6a00046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 shutdown_connections 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6b015cbe0 0x7ff6b006c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6b00d13d0 0x7ff6b00c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff68806a430 0x7ff68806c8f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 --2- 172.21.15.39:0/355768007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6b00ce510 0x7ff6b014d5d0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 >> 172.21.15.39:0/355768007 conn(0x7ff6b009bc80 msgr2=0x7ff6b00d2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:31.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 shutdown_connections 2024-01-24T17:48:31.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.318+0000 7ff6b5400640 1 -- 172.21.15.39:0/355768007 wait complete. 2024-01-24T17:48:31.343 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-24T17:48:31.343 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:48:31.378 INFO:tasks.ceph.mds.a.smithi039.stderr:2024-01-24T17:48:31.377+0000 7f76f8c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:48:31.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.566+0000 7f6592800640 1 Processor -- start 2024-01-24T17:48:31.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.566+0000 7f6592800640 1 -- start start 2024-01-24T17:48:31.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.567+0000 7f6592800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.567+0000 7f6592800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f658c0d15a0 0x7f658c166b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.567+0000 7f6592800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c0b8820 con 0x7f658c0ce510 2024-01-24T17:48:31.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.567+0000 7f6592800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658c0b5790 con 0x7f658c0d15a0 2024-01-24T17:48:31.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.567+0000 7f6592800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0b5a50 con 0x7f658c15cbe0 2024-01-24T17:48:31.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.568+0000 7f658be00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f658c0d15a0 0x7f658c166b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.568+0000 7f658b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.568+0000 7f658b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c0cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:39268/0 (socket says 172.21.15.39:39268) 2024-01-24T17:48:31.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.568+0000 7f658b400640 1 -- 172.21.15.39:0/3267423527 learned_addr learned my addr 172.21.15.39:0/3267423527 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:31.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.569+0000 7f658be00640 1 -- 172.21.15.39:0/3267423527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 msgr2=0x7f658c0cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:31.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.569+0000 7f658be00640 1 --2- 172.21.15.39:0/3267423527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c0cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.569+0000 7f658be00640 1 -- 172.21.15.39:0/3267423527 >> v1:172.21.15.100:6789/0 conn(0x7f658c0ce510 legacy=0x7f658c0d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.569+0000 7f658be00640 1 -- 172.21.15.39:0/3267423527 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0b5d30 con 0x7f658c0d15a0 2024-01-24T17:48:31.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.569+0000 7f658a000640 1 -- 172.21.15.39:0/3267423527 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785163699 0 0) 0x7f658c0b8820 con 0x7f658c0ce510 2024-01-24T17:48:31.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.570+0000 7f658be00640 1 --2- 172.21.15.39:0/3267423527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 0x7f658c166b50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6584009d90 tx=0x7f658402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43e53ac2ed55cff6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:31.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.571+0000 7f658a000640 1 -- 172.21.15.39:0/3267423527 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658402cc50 con 0x7f658c0d15a0 2024-01-24T17:48:31.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.571+0000 7f658a000640 1 -- 172.21.15.39:0/3267423527 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658402cdb0 con 0x7f658c0d15a0 2024-01-24T17:48:31.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.571+0000 7f658a000640 1 -- 172.21.15.39:0/3267423527 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584034660 con 0x7f658c0d15a0 2024-01-24T17:48:31.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.572+0000 7f6592800640 1 -- 172.21.15.39:0/3267423527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 msgr2=0x7f658c166b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.572+0000 7f6592800640 1 --2- 172.21.15.39:0/3267423527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 0x7f658c166b50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6584009d90 tx=0x7f658402bea0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.572+0000 7f6592800640 1 -- 172.21.15.39:0/3267423527 shutdown_connections 2024-01-24T17:48:31.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.572+0000 7f6592800640 1 --2- 172.21.15.39:0/3267423527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 0x7f658c166b50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.572+0000 7f6592800640 1 --2- 172.21.15.39:0/3267423527 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c0cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.572+0000 7f6592800640 1 -- 172.21.15.39:0/3267423527 >> 172.21.15.39:0/3267423527 conn(0x7f658c09bc80 msgr2=0x7f658c0d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:31.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.572+0000 7f6592800640 1 -- 172.21.15.39:0/3267423527 shutdown_connections 2024-01-24T17:48:31.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.572+0000 7f6592800640 1 -- 172.21.15.39:0/3267423527 wait complete. 2024-01-24T17:48:31.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.574+0000 7f6592800640 1 Processor -- start 2024-01-24T17:48:31.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.574+0000 7f6592800640 1 -- start start 2024-01-24T17:48:31.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f6592800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f658c0ce510 0x7f658c148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f6592800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 0x7f658c154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f6592800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f6592800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c157af0 con 0x7f658c0ce510 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f6592800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c1571c0 con 0x7f658c0d15a0 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f6592800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c1574a0 con 0x7f658c15cbe0 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f658b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f658c0ce510 0x7f658c148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f658b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f658c0ce510 0x7f658c148b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49746/0 (socket says 172.21.15.39:49746) 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.575+0000 7f658b400640 1 -- 172.21.15.39:0/714757552 learned_addr learned my addr 172.21.15.39:0/714757552 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658be00640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658aa00640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 0x7f658c154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658b400640 1 -- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 msgr2=0x7f658c154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658b400640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658b400640 1 -- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 msgr2=0x7f658c154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658b400640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 0x7f658c154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658b400640 1 -- 172.21.15.39:0/714757552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584009a70 con 0x7f658c0ce510 2024-01-24T17:48:31.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658aa00640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 0x7f658c154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:31.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f658b400640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f658c0ce510 0x7f658c148b20 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f657c00e7d0 tx=0x7f657c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:31.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f6583400640 1 -- 172.21.15.39:0/714757552 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c009560 con 0x7f658c0ce510 2024-01-24T17:48:31.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f6583400640 1 -- 172.21.15.39:0/714757552 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657c00c940 con 0x7f658c0ce510 2024-01-24T17:48:31.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.576+0000 7f6583400640 1 -- 172.21.15.39:0/714757552 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c00eee0 con 0x7f658c0ce510 2024-01-24T17:48:31.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.577+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c157780 con 0x7f658c0ce510 2024-01-24T17:48:31.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.577+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c1587d0 con 0x7f658c0ce510 2024-01-24T17:48:31.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.578+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658c04dff0 con 0x7f658c0ce510 2024-01-24T17:48:31.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.579+0000 7f6583400640 1 -- 172.21.15.39:0/714757552 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f657c00ccf0 con 0x7f658c0ce510 2024-01-24T17:48:31.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.580+0000 7f6583400640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f656406a430 0x7f656406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:31.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.580+0000 7f6583400640 1 -- 172.21.15.39:0/714757552 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f657c088d10 con 0x7f658c0ce510 2024-01-24T17:48:31.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.581+0000 7f658aa00640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f656406a430 0x7f656406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:31.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.582+0000 7f658aa00640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f656406a430 0x7f656406c8f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f658c1570f0 tx=0x7f6574009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:31.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.584+0000 7f6583400640 1 -- 172.21.15.39:0/714757552 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f657c054390 con 0x7f658c0ce510 2024-01-24T17:48:31.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.777+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f658c0b62f0 con 0x7f656406a430 2024-01-24T17:48:31.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.801+0000 7f6583400640 1 -- 172.21.15.39:0/714757552 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287090 (secure 0 0 0) 0x7f65740013d0 con 0x7f656406a430 2024-01-24T17:48:31.805 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:31.807 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f656406a430 msgr2=0x7f656406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f656406a430 0x7f656406c8f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f658c1570f0 tx=0x7f6574009040 comp rx=0 tx=0).stop 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f658c0ce510 msgr2=0x7f658c148b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f658c0ce510 0x7f658c148b20 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f657c00e7d0 tx=0x7f657c00ec90 comp rx=0 tx=0).stop 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 shutdown_connections 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f658c15cbe0 0x7f658c154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f658c0d15a0 0x7f658c154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f656406a430 0x7f656406c8f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 --2- 172.21.15.39:0/714757552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f658c0ce510 0x7f658c148b20 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 >> 172.21.15.39:0/714757552 conn(0x7f658c09bc80 msgr2=0x7f658c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:31.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 shutdown_connections 2024-01-24T17:48:31.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:31.806+0000 7f6592800640 1 -- 172.21.15.39:0/714757552 wait complete. 2024-01-24T17:48:31.827 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-01-24T17:48:30.359933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259612,"kb_used_data":10672,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754715108,"statfs":{"total":773094113280,"available":772828270592,"internally_reserved":0,"allocated":10928128,"data_stored":5350742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.979674"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929630+0000","last_change":"2024-01-24T17:48:26.919046+0000","last_active":"2024-01-24T17:48:26.929630+0000","last_peered":"2024-01-24T17:48:26.929630+0000","last_clean":"2024-01-24T17:48:26.929630+0000","last_became_active":"2024-01-24T17:48:26.917400+0000","last_became_peered":"2024-01-24T17:48:26.917400+0000","last_unstale":"2024-01-24T17:48:26.929630+0000","last_undegraded":"2024-01-24T17:48:26.929630+0000","last_fullsized":"2024-01-24T17:48:26.929630+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:27:29.531410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.919147+0000","last_change":"2024-01-24T17:48:26.913733+0000","last_active":"2024-01-24T17:48:26.919147+0000","last_peered":"2024-01-24T17:48:26.919147+0000","last_clean":"2024-01-24T17:48:26.919147+0000","last_became_active":"2024-01-24T17:48:26.913390+0000","last_became_peered":"2024-01-24T17:48:26.913390+0000","last_unstale":"2024-01-24T17:48:26.919147+0000","last_undegraded":"2024-01-24T17:48:26.919147+0000","last_fullsized":"2024-01-24T17:48:26.919147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:40:28.505137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.931746+0000","last_change":"2024-01-24T17:48:26.925051+0000","last_active":"2024-01-24T17:48:26.931746+0000","last_peered":"2024-01-24T17:48:26.931746+0000","last_clean":"2024-01-24T17:48:26.931746+0000","last_became_active":"2024-01-24T17:48:26.924187+0000","last_became_peered":"2024-01-24T17:48:26.924187+0000","last_unstale":"2024-01-24T17:48:26.931746+0000","last_undegraded":"2024-01-24T17:48:26.931746+0000","last_fullsized":"2024-01-24T17:48:26.931746+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:16:26.055063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930228+0000","last_change":"2024-01-24T17:48:26.918982+0000","last_active":"2024-01-24T17:48:26.930228+0000","last_peered":"2024-01-24T17:48:26.930228+0000","last_clean":"2024-01-24T17:48:26.930228+0000","last_became_active":"2024-01-24T17:48:26.918660+0000","last_became_peered":"2024-01-24T17:48:26.918660+0000","last_unstale":"2024-01-24T17:48:26.930228+0000","last_undegraded":"2024-01-24T17:48:26.930228+0000","last_fullsized":"2024-01-24T17:48:26.930228+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:33.271424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.926293+0000","last_change":"2024-01-24T17:48:26.923046+0000","last_active":"2024-01-24T17:48:26.926293+0000","last_peered":"2024-01-24T17:48:26.926293+0000","last_clean":"2024-01-24T17:48:26.926293+0000","last_became_active":"2024-01-24T17:48:26.922781+0000","last_became_peered":"2024-01-24T17:48:26.922781+0000","last_unstale":"2024-01-24T17:48:26.926293+0000","last_undegraded":"2024-01-24T17:48:26.926293+0000","last_fullsized":"2024-01-24T17:48:26.926293+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:33:54.874293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.926283+0000","last_change":"2024-01-24T17:48:26.921781+0000","last_active":"2024-01-24T17:48:26.926283+0000","last_peered":"2024-01-24T17:48:26.926283+0000","last_clean":"2024-01-24T17:48:26.926283+0000","last_became_active":"2024-01-24T17:48:26.921087+0000","last_became_peered":"2024-01-24T17:48:26.921087+0000","last_unstale":"2024-01-24T17:48:26.926283+0000","last_undegraded":"2024-01-24T17:48:26.926283+0000","last_fullsized":"2024-01-24T17:48:26.926283+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:00:45.740090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.919364+0000","last_change":"2024-01-24T17:48:26.915775+0000","last_active":"2024-01-24T17:48:26.919364+0000","last_peered":"2024-01-24T17:48:26.919364+0000","last_clean":"2024-01-24T17:48:26.919364+0000","last_became_active":"2024-01-24T17:48:26.915490+0000","last_became_peered":"2024-01-24T17:48:26.915490+0000","last_unstale":"2024-01-24T17:48:26.919364+0000","last_undegraded":"2024-01-24T17:48:26.919364+0000","last_fullsized":"2024-01-24T17:48:26.919364+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:32:22.150580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930305+0000","last_change":"2024-01-24T17:48:26.920702+0000","last_active":"2024-01-24T17:48:26.930305+0000","last_peered":"2024-01-24T17:48:26.930305+0000","last_clean":"2024-01-24T17:48:26.930305+0000","last_became_active":"2024-01-24T17:48:26.920434+0000","last_became_peered":"2024-01-24T17:48:26.920434+0000","last_unstale":"2024-01-24T17:48:26.930305+0000","last_undegraded":"2024-01-24T17:48:26.930305+0000","last_fullsized":"2024-01-24T17:48:26.930305+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:51:40.896415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929286+0000","last_change":"2024-01-24T17:48:26.925409+0000","last_active":"2024-01-24T17:48:26.929286+0000","last_peered":"2024-01-24T17:48:26.929286+0000","last_clean":"2024-01-24T17:48:26.929286+0000","last_became_active":"2024-01-24T17:48:26.925233+0000","last_became_peered":"2024-01-24T17:48:26.925233+0000","last_unstale":"2024-01-24T17:48:26.929286+0000","last_undegraded":"2024-01-24T17:48:26.929286+0000","last_fullsized":"2024-01-24T17:48:26.929286+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:30:28.625862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.921359+0000","last_change":"2024-01-24T17:48:26.914375+0000","last_active":"2024-01-24T17:48:26.921359+0000","last_peered":"2024-01-24T17:48:26.921359+0000","last_clean":"2024-01-24T17:48:26.921359+0000","last_became_active":"2024-01-24T17:48:26.914123+0000","last_became_peered":"2024-01-24T17:48:26.914123+0000","last_unstale":"2024-01-24T17:48:26.921359+0000","last_undegraded":"2024-01-24T17:48:26.921359+0000","last_fullsized":"2024-01-24T17:48:26.921359+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:18:48.467429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.918930+0000","last_change":"2024-01-24T17:48:26.913663+0000","last_active":"2024-01-24T17:48:26.918930+0000","last_peered":"2024-01-24T17:48:26.918930+0000","last_clean":"2024-01-24T17:48:26.918930+0000","last_became_active":"2024-01-24T17:48:26.913375+0000","last_became_peered":"2024-01-24T17:48:26.913375+0000","last_unstale":"2024-01-24T17:48:26.918930+0000","last_undegraded":"2024-01-24T17:48:26.918930+0000","last_fullsized":"2024-01-24T17:48:26.918930+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:01.025561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929731+0000","last_change":"2024-01-24T17:48:26.921337+0000","last_active":"2024-01-24T17:48:26.929731+0000","last_peered":"2024-01-24T17:48:26.929731+0000","last_clean":"2024-01-24T17:48:26.929731+0000","last_became_active":"2024-01-24T17:48:26.921068+0000","last_became_peered":"2024-01-24T17:48:26.921068+0000","last_unstale":"2024-01-24T17:48:26.929731+0000","last_undegraded":"2024-01-24T17:48:26.929731+0000","last_fullsized":"2024-01-24T17:48:26.929731+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:50.103242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.940117+0000","last_change":"2024-01-24T17:48:26.932900+0000","last_active":"2024-01-24T17:48:26.940117+0000","last_peered":"2024-01-24T17:48:26.940117+0000","last_clean":"2024-01-24T17:48:26.940117+0000","last_became_active":"2024-01-24T17:48:26.930460+0000","last_became_peered":"2024-01-24T17:48:26.930460+0000","last_unstale":"2024-01-24T17:48:26.940117+0000","last_undegraded":"2024-01-24T17:48:26.940117+0000","last_fullsized":"2024-01-24T17:48:26.940117+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:38:44.025239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.935362+0000","last_change":"2024-01-24T17:48:26.926123+0000","last_active":"2024-01-24T17:48:26.935362+0000","last_peered":"2024-01-24T17:48:26.935362+0000","last_clean":"2024-01-24T17:48:26.935362+0000","last_became_active":"2024-01-24T17:48:26.925865+0000","last_became_peered":"2024-01-24T17:48:26.925865+0000","last_unstale":"2024-01-24T17:48:26.935362+0000","last_undegraded":"2024-01-24T17:48:26.935362+0000","last_fullsized":"2024-01-24T17:48:26.935362+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:57:05.790265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.935504+0000","last_change":"2024-01-24T17:48:26.928443+0000","last_active":"2024-01-24T17:48:26.935504+0000","last_peered":"2024-01-24T17:48:26.935504+0000","last_clean":"2024-01-24T17:48:26.935504+0000","last_became_active":"2024-01-24T17:48:26.927549+0000","last_became_peered":"2024-01-24T17:48:26.927549+0000","last_unstale":"2024-01-24T17:48:26.935504+0000","last_undegraded":"2024-01-24T17:48:26.935504+0000","last_fullsized":"2024-01-24T17:48:26.935504+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:51:23.333640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.924180+0000","last_change":"2024-01-24T17:48:26.918229+0000","last_active":"2024-01-24T17:48:26.924180+0000","last_peered":"2024-01-24T17:48:26.924180+0000","last_clean":"2024-01-24T17:48:26.924180+0000","last_became_active":"2024-01-24T17:48:26.917618+0000","last_became_peered":"2024-01-24T17:48:26.917618+0000","last_unstale":"2024-01-24T17:48:26.924180+0000","last_undegraded":"2024-01-24T17:48:26.924180+0000","last_fullsized":"2024-01-24T17:48:26.924180+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:15:06.378822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933724+0000","last_change":"2024-01-24T17:48:26.930098+0000","last_active":"2024-01-24T17:48:26.933724+0000","last_peered":"2024-01-24T17:48:26.933724+0000","last_clean":"2024-01-24T17:48:26.933724+0000","last_became_active":"2024-01-24T17:48:26.929843+0000","last_became_peered":"2024-01-24T17:48:26.929843+0000","last_unstale":"2024-01-24T17:48:26.933724+0000","last_undegraded":"2024-01-24T17:48:26.933724+0000","last_fullsized":"2024-01-24T17:48:26.933724+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:33:37.315009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.935532+0000","last_change":"2024-01-24T17:48:26.928528+0000","last_active":"2024-01-24T17:48:26.935532+0000","last_peered":"2024-01-24T17:48:26.935532+0000","last_clean":"2024-01-24T17:48:26.935532+0000","last_became_active":"2024-01-24T17:48:26.927985+0000","last_became_peered":"2024-01-24T17:48:26.927985+0000","last_unstale":"2024-01-24T17:48:26.935532+0000","last_undegraded":"2024-01-24T17:48:26.935532+0000","last_fullsized":"2024-01-24T17:48:26.935532+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:16:11.808000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933318+0000","last_change":"2024-01-24T17:48:26.926495+0000","last_active":"2024-01-24T17:48:26.933318+0000","last_peered":"2024-01-24T17:48:26.933318+0000","last_clean":"2024-01-24T17:48:26.933318+0000","last_became_active":"2024-01-24T17:48:26.921079+0000","last_became_peered":"2024-01-24T17:48:26.921079+0000","last_unstale":"2024-01-24T17:48:26.933318+0000","last_undegraded":"2024-01-24T17:48:26.933318+0000","last_fullsized":"2024-01-24T17:48:26.933318+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:50:11.805736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.925056+0000","last_change":"2024-01-24T17:48:26.917987+0000","last_active":"2024-01-24T17:48:26.925056+0000","last_peered":"2024-01-24T17:48:26.925056+0000","last_clean":"2024-01-24T17:48:26.925056+0000","last_became_active":"2024-01-24T17:48:26.917105+0000","last_became_peered":"2024-01-24T17:48:26.917105+0000","last_unstale":"2024-01-24T17:48:26.925056+0000","last_undegraded":"2024-01-24T17:48:26.925056+0000","last_fullsized":"2024-01-24T17:48:26.925056+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:19:08.635766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930071+0000","last_change":"2024-01-24T17:48:26.924914+0000","last_active":"2024-01-24T17:48:26.930071+0000","last_peered":"2024-01-24T17:48:26.930071+0000","last_clean":"2024-01-24T17:48:26.930071+0000","last_became_active":"2024-01-24T17:48:26.924548+0000","last_became_peered":"2024-01-24T17:48:26.924548+0000","last_unstale":"2024-01-24T17:48:26.930071+0000","last_undegraded":"2024-01-24T17:48:26.930071+0000","last_fullsized":"2024-01-24T17:48:26.930071+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:19:54.838909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.931875+0000","last_change":"2024-01-24T17:48:26.927897+0000","last_active":"2024-01-24T17:48:26.931875+0000","last_peered":"2024-01-24T17:48:26.931875+0000","last_clean":"2024-01-24T17:48:26.931875+0000","last_became_active":"2024-01-24T17:48:26.927529+0000","last_became_peered":"2024-01-24T17:48:26.927529+0000","last_unstale":"2024-01-24T17:48:26.931875+0000","last_undegraded":"2024-01-24T17:48:26.931875+0000","last_fullsized":"2024-01-24T17:48:26.931875+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:19:08.635766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929585+0000","last_change":"2024-01-24T17:48:26.917605+0000","last_active":"2024-01-24T17:48:26.929585+0000","last_peered":"2024-01-24T17:48:26.929585+0000","last_clean":"2024-01-24T17:48:26.929585+0000","last_became_active":"2024-01-24T17:48:26.917312+0000","last_became_peered":"2024-01-24T17:48:26.917312+0000","last_unstale":"2024-01-24T17:48:26.929585+0000","last_undegraded":"2024-01-24T17:48:26.929585+0000","last_fullsized":"2024-01-24T17:48:26.929585+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:16:42.532463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923167+0000","last_change":"2024-01-24T17:48:26.919152+0000","last_active":"2024-01-24T17:48:26.923167+0000","last_peered":"2024-01-24T17:48:26.923167+0000","last_clean":"2024-01-24T17:48:26.923167+0000","last_became_active":"2024-01-24T17:48:26.918789+0000","last_became_peered":"2024-01-24T17:48:26.918789+0000","last_unstale":"2024-01-24T17:48:26.923167+0000","last_undegraded":"2024-01-24T17:48:26.923167+0000","last_fullsized":"2024-01-24T17:48:26.923167+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:57:05.790265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.920877+0000","last_change":"2024-01-24T17:48:26.912980+0000","last_active":"2024-01-24T17:48:26.920877+0000","last_peered":"2024-01-24T17:48:26.920877+0000","last_clean":"2024-01-24T17:48:26.920877+0000","last_became_active":"2024-01-24T17:48:26.912720+0000","last_became_peered":"2024-01-24T17:48:26.912720+0000","last_unstale":"2024-01-24T17:48:26.920877+0000","last_undegraded":"2024-01-24T17:48:26.920877+0000","last_fullsized":"2024-01-24T17:48:26.920877+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:47:44.117143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933468+0000","last_change":"2024-01-24T17:48:26.928405+0000","last_active":"2024-01-24T17:48:26.933468+0000","last_peered":"2024-01-24T17:48:26.933468+0000","last_clean":"2024-01-24T17:48:26.933468+0000","last_became_active":"2024-01-24T17:48:26.928063+0000","last_became_peered":"2024-01-24T17:48:26.928063+0000","last_unstale":"2024-01-24T17:48:26.933468+0000","last_undegraded":"2024-01-24T17:48:26.933468+0000","last_fullsized":"2024-01-24T17:48:26.933468+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:40.339905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923128+0000","last_change":"2024-01-24T17:48:26.915142+0000","last_active":"2024-01-24T17:48:26.923128+0000","last_peered":"2024-01-24T17:48:26.923128+0000","last_clean":"2024-01-24T17:48:26.923128+0000","last_became_active":"2024-01-24T17:48:26.914859+0000","last_became_peered":"2024-01-24T17:48:26.914859+0000","last_unstale":"2024-01-24T17:48:26.923128+0000","last_undegraded":"2024-01-24T17:48:26.923128+0000","last_fullsized":"2024-01-24T17:48:26.923128+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:19:24.310411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.927560+0000","last_change":"2024-01-24T17:48:26.926259+0000","last_active":"2024-01-24T17:48:26.927560+0000","last_peered":"2024-01-24T17:48:26.927560+0000","last_clean":"2024-01-24T17:48:26.927560+0000","last_became_active":"2024-01-24T17:48:26.926005+0000","last_became_peered":"2024-01-24T17:48:26.926005+0000","last_unstale":"2024-01-24T17:48:26.927560+0000","last_undegraded":"2024-01-24T17:48:26.927560+0000","last_fullsized":"2024-01-24T17:48:26.927560+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:19:29.557377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933038+0000","last_change":"2024-01-24T17:48:26.926546+0000","last_active":"2024-01-24T17:48:26.933038+0000","last_peered":"2024-01-24T17:48:26.933038+0000","last_clean":"2024-01-24T17:48:26.933038+0000","last_became_active":"2024-01-24T17:48:26.926231+0000","last_became_peered":"2024-01-24T17:48:26.926231+0000","last_unstale":"2024-01-24T17:48:26.933038+0000","last_undegraded":"2024-01-24T17:48:26.933038+0000","last_fullsized":"2024-01-24T17:48:26.933038+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:40:28.505137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.940162+0000","last_change":"2024-01-24T17:48:26.934236+0000","last_active":"2024-01-24T17:48:26.940162+0000","last_peered":"2024-01-24T17:48:26.940162+0000","last_clean":"2024-01-24T17:48:26.940162+0000","last_became_active":"2024-01-24T17:48:26.933895+0000","last_became_peered":"2024-01-24T17:48:26.933895+0000","last_unstale":"2024-01-24T17:48:26.940162+0000","last_undegraded":"2024-01-24T17:48:26.940162+0000","last_fullsized":"2024-01-24T17:48:26.940162+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:24:02.371020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930187+0000","last_change":"2024-01-24T17:48:26.923487+0000","last_active":"2024-01-24T17:48:26.930187+0000","last_peered":"2024-01-24T17:48:26.930187+0000","last_clean":"2024-01-24T17:48:26.930187+0000","last_became_active":"2024-01-24T17:48:26.923151+0000","last_became_peered":"2024-01-24T17:48:26.923151+0000","last_unstale":"2024-01-24T17:48:26.930187+0000","last_undegraded":"2024-01-24T17:48:26.930187+0000","last_fullsized":"2024-01-24T17:48:26.930187+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:42:42.058056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923633+0000","last_change":"2024-01-24T17:48:26.915693+0000","last_active":"2024-01-24T17:48:26.923633+0000","last_peered":"2024-01-24T17:48:26.923633+0000","last_clean":"2024-01-24T17:48:26.923633+0000","last_became_active":"2024-01-24T17:48:26.915339+0000","last_became_peered":"2024-01-24T17:48:26.915339+0000","last_unstale":"2024-01-24T17:48:26.923633+0000","last_undegraded":"2024-01-24T17:48:26.923633+0000","last_fullsized":"2024-01-24T17:48:26.923633+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:23:15.094402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.914872+0000","last_change":"2024-01-24T17:48:26.911079+0000","last_active":"2024-01-24T17:48:26.914872+0000","last_peered":"2024-01-24T17:48:26.914872+0000","last_clean":"2024-01-24T17:48:26.914872+0000","last_became_active":"2024-01-24T17:48:26.910777+0000","last_became_peered":"2024-01-24T17:48:26.910777+0000","last_unstale":"2024-01-24T17:48:26.914872+0000","last_undegraded":"2024-01-24T17:48:26.914872+0000","last_fullsized":"2024-01-24T17:48:26.914872+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:36:51.824015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1f","version":"99'4","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.934074+0000","last_change":"2024-01-24T17:48:26.436409+0000","last_active":"2024-01-24T17:48:26.934074+0000","last_peered":"2024-01-24T17:48:26.934074+0000","last_clean":"2024-01-24T17:48:26.934074+0000","last_became_active":"2024-01-24T17:48:26.435501+0000","last_became_peered":"2024-01-24T17:48:26.435501+0000","last_unstale":"2024-01-24T17:48:26.934074+0000","last_undegraded":"2024-01-24T17:48:26.934074+0000","last_fullsized":"2024-01-24T17:48:26.934074+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:49.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.940187+0000","last_change":"2024-01-24T17:48:26.935171+0000","last_active":"2024-01-24T17:48:26.940187+0000","last_peered":"2024-01-24T17:48:26.940187+0000","last_clean":"2024-01-24T17:48:26.940187+0000","last_became_active":"2024-01-24T17:48:26.934878+0000","last_became_peered":"2024-01-24T17:48:26.934878+0000","last_unstale":"2024-01-24T17:48:26.940187+0000","last_undegraded":"2024-01-24T17:48:26.940187+0000","last_fullsized":"2024-01-24T17:48:26.940187+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:27:29.531410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1e","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.917903+0000","last_change":"2024-01-24T17:48:26.422468+0000","last_active":"2024-01-24T17:48:26.917903+0000","last_peered":"2024-01-24T17:48:26.917903+0000","last_clean":"2024-01-24T17:48:26.917903+0000","last_became_active":"2024-01-24T17:48:26.422078+0000","last_became_peered":"2024-01-24T17:48:26.422078+0000","last_unstale":"2024-01-24T17:48:26.917903+0000","last_undegraded":"2024-01-24T17:48:26.917903+0000","last_fullsized":"2024-01-24T17:48:26.917903+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:24.481188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930207+0000","last_change":"2024-01-24T17:48:26.917480+0000","last_active":"2024-01-24T17:48:26.930207+0000","last_peered":"2024-01-24T17:48:26.930207+0000","last_clean":"2024-01-24T17:48:26.930207+0000","last_became_active":"2024-01-24T17:48:26.914739+0000","last_became_peered":"2024-01-24T17:48:26.914739+0000","last_unstale":"2024-01-24T17:48:26.930207+0000","last_undegraded":"2024-01-24T17:48:26.930207+0000","last_fullsized":"2024-01-24T17:48:26.930207+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:55:24.010659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930292+0000","last_change":"2024-01-24T17:48:26.440810+0000","last_active":"2024-01-24T17:48:26.930292+0000","last_peered":"2024-01-24T17:48:26.930292+0000","last_clean":"2024-01-24T17:48:26.930292+0000","last_became_active":"2024-01-24T17:48:26.440235+0000","last_became_peered":"2024-01-24T17:48:26.440235+0000","last_unstale":"2024-01-24T17:48:26.930292+0000","last_undegraded":"2024-01-24T17:48:26.930292+0000","last_fullsized":"2024-01-24T17:48:26.930292+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:17:03.731286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.921106+0000","last_change":"2024-01-24T17:48:26.915047+0000","last_active":"2024-01-24T17:48:26.921106+0000","last_peered":"2024-01-24T17:48:26.921106+0000","last_clean":"2024-01-24T17:48:26.921106+0000","last_became_active":"2024-01-24T17:48:26.914645+0000","last_became_peered":"2024-01-24T17:48:26.914645+0000","last_unstale":"2024-01-24T17:48:26.921106+0000","last_undegraded":"2024-01-24T17:48:26.921106+0000","last_fullsized":"2024-01-24T17:48:26.921106+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:16:26.055063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933526+0000","last_change":"2024-01-24T17:48:26.434616+0000","last_active":"2024-01-24T17:48:26.933526+0000","last_peered":"2024-01-24T17:48:26.933526+0000","last_clean":"2024-01-24T17:48:26.933526+0000","last_became_active":"2024-01-24T17:48:26.434308+0000","last_became_peered":"2024-01-24T17:48:26.434308+0000","last_unstale":"2024-01-24T17:48:26.933526+0000","last_undegraded":"2024-01-24T17:48:26.933526+0000","last_fullsized":"2024-01-24T17:48:26.933526+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:16.864381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.920315+0000","last_change":"2024-01-24T17:48:26.915182+0000","last_active":"2024-01-24T17:48:26.920315+0000","last_peered":"2024-01-24T17:48:26.920315+0000","last_clean":"2024-01-24T17:48:26.920315+0000","last_became_active":"2024-01-24T17:48:26.914921+0000","last_became_peered":"2024-01-24T17:48:26.914921+0000","last_unstale":"2024-01-24T17:48:26.920315+0000","last_undegraded":"2024-01-24T17:48:26.920315+0000","last_fullsized":"2024-01-24T17:48:26.920315+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:35:22.864785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1b","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.917886+0000","last_change":"2024-01-24T17:48:26.425387+0000","last_active":"2024-01-24T17:48:26.917886+0000","last_peered":"2024-01-24T17:48:26.917886+0000","last_clean":"2024-01-24T17:48:26.917886+0000","last_became_active":"2024-01-24T17:48:26.425109+0000","last_became_peered":"2024-01-24T17:48:26.425109+0000","last_unstale":"2024-01-24T17:48:26.917886+0000","last_undegraded":"2024-01-24T17:48:26.917886+0000","last_fullsized":"2024-01-24T17:48:26.917886+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:49.530193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.918962+0000","last_change":"2024-01-24T17:48:26.913734+0000","last_active":"2024-01-24T17:48:26.918962+0000","last_peered":"2024-01-24T17:48:26.918962+0000","last_clean":"2024-01-24T17:48:26.918962+0000","last_became_active":"2024-01-24T17:48:26.913393+0000","last_became_peered":"2024-01-24T17:48:26.913393+0000","last_unstale":"2024-01-24T17:48:26.918962+0000","last_undegraded":"2024-01-24T17:48:26.918962+0000","last_fullsized":"2024-01-24T17:48:26.918962+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:25:35.588913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.932306+0000","last_change":"2024-01-24T17:48:26.430251+0000","last_active":"2024-01-24T17:48:26.932306+0000","last_peered":"2024-01-24T17:48:26.932306+0000","last_clean":"2024-01-24T17:48:26.932306+0000","last_became_active":"2024-01-24T17:48:26.427346+0000","last_became_peered":"2024-01-24T17:48:26.427346+0000","last_unstale":"2024-01-24T17:48:26.932306+0000","last_undegraded":"2024-01-24T17:48:26.932306+0000","last_fullsized":"2024-01-24T17:48:26.932306+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:38:16.109886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.940157+0000","last_change":"2024-01-24T17:48:26.933163+0000","last_active":"2024-01-24T17:48:26.940157+0000","last_peered":"2024-01-24T17:48:26.940157+0000","last_clean":"2024-01-24T17:48:26.940157+0000","last_became_active":"2024-01-24T17:48:26.931543+0000","last_became_peered":"2024-01-24T17:48:26.931543+0000","last_unstale":"2024-01-24T17:48:26.940157+0000","last_undegraded":"2024-01-24T17:48:26.940157+0000","last_fullsized":"2024-01-24T17:48:26.940157+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:55:24.010659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.900520+0000","last_change":"2024-01-24T17:48:26.429814+0000","last_active":"2024-01-24T17:48:26.900520+0000","last_peered":"2024-01-24T17:48:26.900520+0000","last_clean":"2024-01-24T17:48:26.900520+0000","last_became_active":"2024-01-24T17:48:26.429512+0000","last_became_peered":"2024-01-24T17:48:26.429512+0000","last_unstale":"2024-01-24T17:48:26.900520+0000","last_undegraded":"2024-01-24T17:48:26.900520+0000","last_fullsized":"2024-01-24T17:48:26.900520+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:48.343913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.911801+0000","last_change":"2024-01-24T17:48:26.905009+0000","last_active":"2024-01-24T17:48:26.911801+0000","last_peered":"2024-01-24T17:48:26.911801+0000","last_clean":"2024-01-24T17:48:26.911801+0000","last_became_active":"2024-01-24T17:48:26.904642+0000","last_became_peered":"2024-01-24T17:48:26.904642+0000","last_unstale":"2024-01-24T17:48:26.911801+0000","last_undegraded":"2024-01-24T17:48:26.911801+0000","last_fullsized":"2024-01-24T17:48:26.911801+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:27:05.102120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.907227+0000","last_change":"2024-01-24T17:48:26.429704+0000","last_active":"2024-01-24T17:48:26.907227+0000","last_peered":"2024-01-24T17:48:26.907227+0000","last_clean":"2024-01-24T17:48:26.907227+0000","last_became_active":"2024-01-24T17:48:26.428050+0000","last_became_peered":"2024-01-24T17:48:26.428050+0000","last_unstale":"2024-01-24T17:48:26.907227+0000","last_undegraded":"2024-01-24T17:48:26.907227+0000","last_fullsized":"2024-01-24T17:48:26.907227+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:55:22.994196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929695+0000","last_change":"2024-01-24T17:48:26.918741+0000","last_active":"2024-01-24T17:48:26.929695+0000","last_peered":"2024-01-24T17:48:26.929695+0000","last_clean":"2024-01-24T17:48:26.929695+0000","last_became_active":"2024-01-24T17:48:26.918464+0000","last_became_peered":"2024-01-24T17:48:26.918464+0000","last_unstale":"2024-01-24T17:48:26.929695+0000","last_undegraded":"2024-01-24T17:48:26.929695+0000","last_fullsized":"2024-01-24T17:48:26.929695+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:50.891831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.917673+0000","last_change":"2024-01-24T17:48:26.437385+0000","last_active":"2024-01-24T17:48:26.917673+0000","last_peered":"2024-01-24T17:48:26.917673+0000","last_clean":"2024-01-24T17:48:26.917673+0000","last_became_active":"2024-01-24T17:48:26.437107+0000","last_became_peered":"2024-01-24T17:48:26.437107+0000","last_unstale":"2024-01-24T17:48:26.917673+0000","last_undegraded":"2024-01-24T17:48:26.917673+0000","last_fullsized":"2024-01-24T17:48:26.917673+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:37:38.371998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.920253+0000","last_change":"2024-01-24T17:48:26.912406+0000","last_active":"2024-01-24T17:48:26.920253+0000","last_peered":"2024-01-24T17:48:26.920253+0000","last_clean":"2024-01-24T17:48:26.920253+0000","last_became_active":"2024-01-24T17:48:26.912125+0000","last_became_peered":"2024-01-24T17:48:26.912125+0000","last_unstale":"2024-01-24T17:48:26.920253+0000","last_undegraded":"2024-01-24T17:48:26.920253+0000","last_fullsized":"2024-01-24T17:48:26.920253+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:50.891831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.905732+0000","last_change":"2024-01-24T17:48:26.435724+0000","last_active":"2024-01-24T17:48:26.905732+0000","last_peered":"2024-01-24T17:48:26.905732+0000","last_clean":"2024-01-24T17:48:26.905732+0000","last_became_active":"2024-01-24T17:48:26.434914+0000","last_became_peered":"2024-01-24T17:48:26.434914+0000","last_unstale":"2024-01-24T17:48:26.905732+0000","last_undegraded":"2024-01-24T17:48:26.905732+0000","last_fullsized":"2024-01-24T17:48:26.905732+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:19:57.644580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.926144+0000","last_change":"2024-01-24T17:48:26.921371+0000","last_active":"2024-01-24T17:48:26.926144+0000","last_peered":"2024-01-24T17:48:26.926144+0000","last_clean":"2024-01-24T17:48:26.926144+0000","last_became_active":"2024-01-24T17:48:26.921083+0000","last_became_peered":"2024-01-24T17:48:26.921083+0000","last_unstale":"2024-01-24T17:48:26.926144+0000","last_undegraded":"2024-01-24T17:48:26.926144+0000","last_fullsized":"2024-01-24T17:48:26.926144+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:59:00.321722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"99'3","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.921371+0000","last_change":"2024-01-24T17:48:26.432421+0000","last_active":"2024-01-24T17:48:26.921371+0000","last_peered":"2024-01-24T17:48:26.921371+0000","last_clean":"2024-01-24T17:48:26.921371+0000","last_became_active":"2024-01-24T17:48:26.432170+0000","last_became_peered":"2024-01-24T17:48:26.432170+0000","last_unstale":"2024-01-24T17:48:26.921371+0000","last_undegraded":"2024-01-24T17:48:26.921371+0000","last_fullsized":"2024-01-24T17:48:26.921371+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:45:38.433706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923147+0000","last_change":"2024-01-24T17:48:26.918604+0000","last_active":"2024-01-24T17:48:26.923147+0000","last_peered":"2024-01-24T17:48:26.923147+0000","last_clean":"2024-01-24T17:48:26.923147+0000","last_became_active":"2024-01-24T17:48:26.918356+0000","last_became_peered":"2024-01-24T17:48:26.918356+0000","last_unstale":"2024-01-24T17:48:26.923147+0000","last_undegraded":"2024-01-24T17:48:26.923147+0000","last_fullsized":"2024-01-24T17:48:26.923147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:28:38.063901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.14","version":"99'6","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933949+0000","last_change":"2024-01-24T17:48:26.429468+0000","last_active":"2024-01-24T17:48:26.933949+0000","last_peered":"2024-01-24T17:48:26.933949+0000","last_clean":"2024-01-24T17:48:26.933949+0000","last_became_active":"2024-01-24T17:48:26.429095+0000","last_became_peered":"2024-01-24T17:48:26.429095+0000","last_unstale":"2024-01-24T17:48:26.933949+0000","last_undegraded":"2024-01-24T17:48:26.933949+0000","last_fullsized":"2024-01-24T17:48:26.933949+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:38:16.109886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929603+0000","last_change":"2024-01-24T17:48:26.920781+0000","last_active":"2024-01-24T17:48:26.929603+0000","last_peered":"2024-01-24T17:48:26.929603+0000","last_clean":"2024-01-24T17:48:26.929603+0000","last_became_active":"2024-01-24T17:48:26.920278+0000","last_became_peered":"2024-01-24T17:48:26.920278+0000","last_unstale":"2024-01-24T17:48:26.929603+0000","last_undegraded":"2024-01-24T17:48:26.929603+0000","last_fullsized":"2024-01-24T17:48:26.929603+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:41:28.702211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.924627+0000","last_change":"2024-01-24T17:48:26.424391+0000","last_active":"2024-01-24T17:48:26.924627+0000","last_peered":"2024-01-24T17:48:26.924627+0000","last_clean":"2024-01-24T17:48:26.924627+0000","last_became_active":"2024-01-24T17:48:26.421024+0000","last_became_peered":"2024-01-24T17:48:26.421024+0000","last_unstale":"2024-01-24T17:48:26.924627+0000","last_undegraded":"2024-01-24T17:48:26.924627+0000","last_fullsized":"2024-01-24T17:48:26.924627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:27:28.514947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.937752+0000","last_change":"2024-01-24T17:48:26.934723+0000","last_active":"2024-01-24T17:48:26.937752+0000","last_peered":"2024-01-24T17:48:26.937752+0000","last_clean":"2024-01-24T17:48:26.937752+0000","last_became_active":"2024-01-24T17:48:26.933851+0000","last_became_peered":"2024-01-24T17:48:26.933851+0000","last_unstale":"2024-01-24T17:48:26.937752+0000","last_undegraded":"2024-01-24T17:48:26.937752+0000","last_fullsized":"2024-01-24T17:48:26.937752+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:41:28.702211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.905072+0000","last_change":"2024-01-24T17:48:26.427403+0000","last_active":"2024-01-24T17:48:26.905072+0000","last_peered":"2024-01-24T17:48:26.905072+0000","last_clean":"2024-01-24T17:48:26.905072+0000","last_became_active":"2024-01-24T17:48:26.427120+0000","last_became_peered":"2024-01-24T17:48:26.427120+0000","last_unstale":"2024-01-24T17:48:26.905072+0000","last_undegraded":"2024-01-24T17:48:26.905072+0000","last_fullsized":"2024-01-24T17:48:26.905072+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:27:04.085657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.918754+0000","last_change":"2024-01-24T17:48:26.912679+0000","last_active":"2024-01-24T17:48:26.918754+0000","last_peered":"2024-01-24T17:48:26.918754+0000","last_clean":"2024-01-24T17:48:26.918754+0000","last_became_active":"2024-01-24T17:48:26.912276+0000","last_became_peered":"2024-01-24T17:48:26.912276+0000","last_unstale":"2024-01-24T17:48:26.918754+0000","last_undegraded":"2024-01-24T17:48:26.918754+0000","last_fullsized":"2024-01-24T17:48:26.918754+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:40:24.661312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.893145+0000","last_change":"2024-01-24T17:48:26.432311+0000","last_active":"2024-01-24T17:48:26.893145+0000","last_peered":"2024-01-24T17:48:26.893145+0000","last_clean":"2024-01-24T17:48:26.893145+0000","last_became_active":"2024-01-24T17:48:26.432000+0000","last_became_peered":"2024-01-24T17:48:26.432000+0000","last_unstale":"2024-01-24T17:48:26.893145+0000","last_undegraded":"2024-01-24T17:48:26.893145+0000","last_fullsized":"2024-01-24T17:48:26.893145+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:37:38.371998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930164+0000","last_change":"2024-01-24T17:48:26.921033+0000","last_active":"2024-01-24T17:48:26.930164+0000","last_peered":"2024-01-24T17:48:26.930164+0000","last_clean":"2024-01-24T17:48:26.930164+0000","last_became_active":"2024-01-24T17:48:26.920684+0000","last_became_peered":"2024-01-24T17:48:26.920684+0000","last_unstale":"2024-01-24T17:48:26.930164+0000","last_undegraded":"2024-01-24T17:48:26.930164+0000","last_fullsized":"2024-01-24T17:48:26.930164+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:18:48.467429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.10","version":"99'8","reported_seq":18,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.935700+0000","last_change":"2024-01-24T17:48:26.428667+0000","last_active":"2024-01-24T17:48:26.935700+0000","last_peered":"2024-01-24T17:48:26.935700+0000","last_clean":"2024-01-24T17:48:26.935700+0000","last_became_active":"2024-01-24T17:48:26.428319+0000","last_became_peered":"2024-01-24T17:48:26.428319+0000","last_unstale":"2024-01-24T17:48:26.935700+0000","last_undegraded":"2024-01-24T17:48:26.935700+0000","last_fullsized":"2024-01-24T17:48:26.935700+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:16:26.347716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.911872+0000","last_change":"2024-01-24T17:48:26.904743+0000","last_active":"2024-01-24T17:48:26.911872+0000","last_peered":"2024-01-24T17:48:26.911872+0000","last_clean":"2024-01-24T17:48:26.911872+0000","last_became_active":"2024-01-24T17:48:26.904459+0000","last_became_peered":"2024-01-24T17:48:26.904459+0000","last_unstale":"2024-01-24T17:48:26.911872+0000","last_undegraded":"2024-01-24T17:48:26.911872+0000","last_fullsized":"2024-01-24T17:48:26.911872+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:51:23.333640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.e","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.913873+0000","last_change":"2024-01-24T17:48:26.437498+0000","last_active":"2024-01-24T17:48:26.913873+0000","last_peered":"2024-01-24T17:48:26.913873+0000","last_clean":"2024-01-24T17:48:26.913873+0000","last_became_active":"2024-01-24T17:48:26.437175+0000","last_became_peered":"2024-01-24T17:48:26.437175+0000","last_unstale":"2024-01-24T17:48:26.913873+0000","last_undegraded":"2024-01-24T17:48:26.913873+0000","last_fullsized":"2024-01-24T17:48:26.913873+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:38.094327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.914708+0000","last_change":"2024-01-24T17:48:26.907099+0000","last_active":"2024-01-24T17:48:26.914708+0000","last_peered":"2024-01-24T17:48:26.914708+0000","last_clean":"2024-01-24T17:48:26.914708+0000","last_became_active":"2024-01-24T17:48:26.906615+0000","last_became_peered":"2024-01-24T17:48:26.906615+0000","last_unstale":"2024-01-24T17:48:26.914708+0000","last_undegraded":"2024-01-24T17:48:26.914708+0000","last_fullsized":"2024-01-24T17:48:26.914708+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:16:11.808000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.d","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.915494+0000","last_change":"2024-01-24T17:48:26.426985+0000","last_active":"2024-01-24T17:48:26.915494+0000","last_peered":"2024-01-24T17:48:26.915494+0000","last_clean":"2024-01-24T17:48:26.915494+0000","last_became_active":"2024-01-24T17:48:26.426675+0000","last_became_peered":"2024-01-24T17:48:26.426675+0000","last_unstale":"2024-01-24T17:48:26.915494+0000","last_undegraded":"2024-01-24T17:48:26.915494+0000","last_fullsized":"2024-01-24T17:48:26.915494+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:39:09.540390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.919717+0000","last_change":"2024-01-24T17:48:26.914171+0000","last_active":"2024-01-24T17:48:26.919717+0000","last_peered":"2024-01-24T17:48:26.919717+0000","last_clean":"2024-01-24T17:48:26.919717+0000","last_became_active":"2024-01-24T17:48:26.913928+0000","last_became_peered":"2024-01-24T17:48:26.913928+0000","last_unstale":"2024-01-24T17:48:26.919717+0000","last_undegraded":"2024-01-24T17:48:26.919717+0000","last_fullsized":"2024-01-24T17:48:26.919717+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:19:54.838909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.908264+0000","last_change":"2024-01-24T17:48:26.421304+0000","last_active":"2024-01-24T17:48:26.908264+0000","last_peered":"2024-01-24T17:48:26.908264+0000","last_clean":"2024-01-24T17:48:26.908264+0000","last_became_active":"2024-01-24T17:48:26.420499+0000","last_became_peered":"2024-01-24T17:48:26.420499+0000","last_unstale":"2024-01-24T17:48:26.908264+0000","last_undegraded":"2024-01-24T17:48:26.908264+0000","last_fullsized":"2024-01-24T17:48:26.908264+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:40:27.488675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933614+0000","last_change":"2024-01-24T17:48:26.929098+0000","last_active":"2024-01-24T17:48:26.933614+0000","last_peered":"2024-01-24T17:48:26.933614+0000","last_clean":"2024-01-24T17:48:26.933614+0000","last_became_active":"2024-01-24T17:48:26.928842+0000","last_became_peered":"2024-01-24T17:48:26.928842+0000","last_unstale":"2024-01-24T17:48:26.933614+0000","last_undegraded":"2024-01-24T17:48:26.933614+0000","last_fullsized":"2024-01-24T17:48:26.933614+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:21:35.521611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.925521+0000","last_change":"2024-01-24T17:48:26.434737+0000","last_active":"2024-01-24T17:48:26.925521+0000","last_peered":"2024-01-24T17:48:26.925521+0000","last_clean":"2024-01-24T17:48:26.925521+0000","last_became_active":"2024-01-24T17:48:26.434343+0000","last_became_peered":"2024-01-24T17:48:26.434343+0000","last_unstale":"2024-01-24T17:48:26.925521+0000","last_undegraded":"2024-01-24T17:48:26.925521+0000","last_fullsized":"2024-01-24T17:48:26.925521+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:35:21.848322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933399+0000","last_change":"2024-01-24T17:48:26.927592+0000","last_active":"2024-01-24T17:48:26.933399+0000","last_peered":"2024-01-24T17:48:26.933399+0000","last_clean":"2024-01-24T17:48:26.933399+0000","last_became_active":"2024-01-24T17:48:26.927316+0000","last_became_peered":"2024-01-24T17:48:26.927316+0000","last_unstale":"2024-01-24T17:48:26.933399+0000","last_undegraded":"2024-01-24T17:48:26.933399+0000","last_fullsized":"2024-01-24T17:48:26.933399+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:04:32.591869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.897316+0000","last_change":"2024-01-24T17:48:26.427383+0000","last_active":"2024-01-24T17:48:26.897316+0000","last_peered":"2024-01-24T17:48:26.897316+0000","last_clean":"2024-01-24T17:48:26.897316+0000","last_became_active":"2024-01-24T17:48:26.427089+0000","last_became_peered":"2024-01-24T17:48:26.427089+0000","last_unstale":"2024-01-24T17:48:26.897316+0000","last_undegraded":"2024-01-24T17:48:26.897316+0000","last_fullsized":"2024-01-24T17:48:26.897316+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:07:20.592462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923607+0000","last_change":"2024-01-24T17:48:26.914951+0000","last_active":"2024-01-24T17:48:26.923607+0000","last_peered":"2024-01-24T17:48:26.923607+0000","last_clean":"2024-01-24T17:48:26.923607+0000","last_became_active":"2024-01-24T17:48:26.914677+0000","last_became_peered":"2024-01-24T17:48:26.914677+0000","last_unstale":"2024-01-24T17:48:26.923607+0000","last_undegraded":"2024-01-24T17:48:26.923607+0000","last_fullsized":"2024-01-24T17:48:26.923607+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:24:02.371020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.891159+0000","last_change":"2024-01-24T17:48:26.429921+0000","last_active":"2024-01-24T17:48:26.891159+0000","last_peered":"2024-01-24T17:48:26.891159+0000","last_clean":"2024-01-24T17:48:26.891159+0000","last_became_active":"2024-01-24T17:48:26.429551+0000","last_became_peered":"2024-01-24T17:48:26.429551+0000","last_unstale":"2024-01-24T17:48:26.891159+0000","last_undegraded":"2024-01-24T17:48:26.891159+0000","last_fullsized":"2024-01-24T17:48:26.891159+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:40:15.685823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.924109+0000","last_change":"2024-01-24T17:48:26.922640+0000","last_active":"2024-01-24T17:48:26.924109+0000","last_peered":"2024-01-24T17:48:26.924109+0000","last_clean":"2024-01-24T17:48:26.924109+0000","last_became_active":"2024-01-24T17:48:26.922418+0000","last_became_peered":"2024-01-24T17:48:26.922418+0000","last_unstale":"2024-01-24T17:48:26.924109+0000","last_undegraded":"2024-01-24T17:48:26.924109+0000","last_fullsized":"2024-01-24T17:48:26.924109+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:03:08.285397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.915815+0000","last_change":"2024-01-24T17:48:26.438815+0000","last_active":"2024-01-24T17:48:26.915815+0000","last_peered":"2024-01-24T17:48:26.915815+0000","last_clean":"2024-01-24T17:48:26.915815+0000","last_became_active":"2024-01-24T17:48:26.438290+0000","last_became_peered":"2024-01-24T17:48:26.438290+0000","last_unstale":"2024-01-24T17:48:26.915815+0000","last_undegraded":"2024-01-24T17:48:26.915815+0000","last_fullsized":"2024-01-24T17:48:26.915815+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:38.094327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929158+0000","last_change":"2024-01-24T17:48:26.914775+0000","last_active":"2024-01-24T17:48:26.929158+0000","last_peered":"2024-01-24T17:48:26.929158+0000","last_clean":"2024-01-24T17:48:26.929158+0000","last_became_active":"2024-01-24T17:48:26.914434+0000","last_became_peered":"2024-01-24T17:48:26.914434+0000","last_unstale":"2024-01-24T17:48:26.929158+0000","last_undegraded":"2024-01-24T17:48:26.929158+0000","last_fullsized":"2024-01-24T17:48:26.929158+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:41:47.000130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.897206+0000","last_change":"2024-01-24T17:48:26.432467+0000","last_active":"2024-01-24T17:48:26.897206+0000","last_peered":"2024-01-24T17:48:26.897206+0000","last_clean":"2024-01-24T17:48:26.897206+0000","last_became_active":"2024-01-24T17:48:26.431971+0000","last_became_peered":"2024-01-24T17:48:26.431971+0000","last_unstale":"2024-01-24T17:48:26.897206+0000","last_undegraded":"2024-01-24T17:48:26.897206+0000","last_fullsized":"2024-01-24T17:48:26.897206+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:47.757639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.931825+0000","last_change":"2024-01-24T17:48:26.921114+0000","last_active":"2024-01-24T17:48:26.931825+0000","last_peered":"2024-01-24T17:48:26.931825+0000","last_clean":"2024-01-24T17:48:26.931825+0000","last_became_active":"2024-01-24T17:48:26.920853+0000","last_became_peered":"2024-01-24T17:48:26.920853+0000","last_unstale":"2024-01-24T17:48:26.931825+0000","last_undegraded":"2024-01-24T17:48:26.931825+0000","last_fullsized":"2024-01-24T17:48:26.931825+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:33:04.140368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.907870+0000","last_change":"2024-01-24T17:48:26.434756+0000","last_active":"2024-01-24T17:48:26.907870+0000","last_peered":"2024-01-24T17:48:26.907870+0000","last_clean":"2024-01-24T17:48:26.907870+0000","last_became_active":"2024-01-24T17:48:26.434466+0000","last_became_peered":"2024-01-24T17:48:26.434466+0000","last_unstale":"2024-01-24T17:48:26.907870+0000","last_undegraded":"2024-01-24T17:48:26.907870+0000","last_fullsized":"2024-01-24T17:48:26.907870+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:38:43.008776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933466+0000","last_change":"2024-01-24T17:48:26.930084+0000","last_active":"2024-01-24T17:48:26.933466+0000","last_peered":"2024-01-24T17:48:26.933466+0000","last_clean":"2024-01-24T17:48:26.933466+0000","last_became_active":"2024-01-24T17:48:26.925847+0000","last_became_peered":"2024-01-24T17:48:26.925847+0000","last_unstale":"2024-01-24T17:48:26.933466+0000","last_undegraded":"2024-01-24T17:48:26.933466+0000","last_fullsized":"2024-01-24T17:48:26.933466+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:50.103242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.908767+0000","last_change":"2024-01-24T17:48:26.429485+0000","last_active":"2024-01-24T17:48:26.908767+0000","last_peered":"2024-01-24T17:48:26.908767+0000","last_clean":"2024-01-24T17:48:26.908767+0000","last_became_active":"2024-01-24T17:48:26.427816+0000","last_became_peered":"2024-01-24T17:48:26.427816+0000","last_unstale":"2024-01-24T17:48:26.908767+0000","last_undegraded":"2024-01-24T17:48:26.908767+0000","last_fullsized":"2024-01-24T17:48:26.908767+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:39.323443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.918783+0000","last_change":"2024-01-24T17:48:26.912842+0000","last_active":"2024-01-24T17:48:26.918783+0000","last_peered":"2024-01-24T17:48:26.918783+0000","last_clean":"2024-01-24T17:48:26.918783+0000","last_became_active":"2024-01-24T17:48:26.912574+0000","last_became_peered":"2024-01-24T17:48:26.912574+0000","last_unstale":"2024-01-24T17:48:26.918783+0000","last_undegraded":"2024-01-24T17:48:26.918783+0000","last_fullsized":"2024-01-24T17:48:26.918783+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:47:04.048590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.898990+0000","last_change":"2024-01-24T17:48:26.427943+0000","last_active":"2024-01-24T17:48:26.898990+0000","last_peered":"2024-01-24T17:48:26.898990+0000","last_clean":"2024-01-24T17:48:26.898990+0000","last_became_active":"2024-01-24T17:48:26.427470+0000","last_became_peered":"2024-01-24T17:48:26.427470+0000","last_unstale":"2024-01-24T17:48:26.898990+0000","last_undegraded":"2024-01-24T17:48:26.898990+0000","last_fullsized":"2024-01-24T17:48:26.898990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:47:43.100680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.932497+0000","last_change":"2024-01-24T17:48:26.918522+0000","last_active":"2024-01-24T17:48:26.932497+0000","last_peered":"2024-01-24T17:48:26.932497+0000","last_clean":"2024-01-24T17:48:26.932497+0000","last_became_active":"2024-01-24T17:48:26.918180+0000","last_became_peered":"2024-01-24T17:48:26.918180+0000","last_unstale":"2024-01-24T17:48:26.932497+0000","last_undegraded":"2024-01-24T17:48:26.932497+0000","last_fullsized":"2024-01-24T17:48:26.932497+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:38:48.469680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.916549+0000","last_change":"2024-01-24T17:48:26.440870+0000","last_active":"2024-01-24T17:48:26.916549+0000","last_peered":"2024-01-24T17:48:26.916549+0000","last_clean":"2024-01-24T17:48:26.916549+0000","last_became_active":"2024-01-24T17:48:26.440549+0000","last_became_peered":"2024-01-24T17:48:26.440549+0000","last_unstale":"2024-01-24T17:48:26.916549+0000","last_undegraded":"2024-01-24T17:48:26.916549+0000","last_fullsized":"2024-01-24T17:48:26.916549+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:13.536137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.934294+0000","last_change":"2024-01-24T17:48:26.927450+0000","last_active":"2024-01-24T17:48:26.934294+0000","last_peered":"2024-01-24T17:48:26.934294+0000","last_clean":"2024-01-24T17:48:26.934294+0000","last_became_active":"2024-01-24T17:48:26.927182+0000","last_became_peered":"2024-01-24T17:48:26.927182+0000","last_unstale":"2024-01-24T17:48:26.934294+0000","last_undegraded":"2024-01-24T17:48:26.934294+0000","last_fullsized":"2024-01-24T17:48:26.934294+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:31:12.583303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.898671+0000","last_change":"2024-01-24T17:48:26.423543+0000","last_active":"2024-01-24T17:48:26.898671+0000","last_peered":"2024-01-24T17:48:26.898671+0000","last_clean":"2024-01-24T17:48:26.898671+0000","last_became_active":"2024-01-24T17:48:26.423221+0000","last_became_peered":"2024-01-24T17:48:26.423221+0000","last_unstale":"2024-01-24T17:48:26.898671+0000","last_undegraded":"2024-01-24T17:48:26.898671+0000","last_fullsized":"2024-01-24T17:48:26.898671+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:07:56.196666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930799+0000","last_change":"2024-01-24T17:48:26.917832+0000","last_active":"2024-01-24T17:48:26.930799+0000","last_peered":"2024-01-24T17:48:26.930799+0000","last_clean":"2024-01-24T17:48:26.930799+0000","last_became_active":"2024-01-24T17:48:26.917584+0000","last_became_peered":"2024-01-24T17:48:26.917584+0000","last_unstale":"2024-01-24T17:48:26.930799+0000","last_undegraded":"2024-01-24T17:48:26.930799+0000","last_fullsized":"2024-01-24T17:48:26.930799+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:19:29.557377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.912432+0000","last_change":"2024-01-24T17:48:26.438811+0000","last_active":"2024-01-24T17:48:26.912432+0000","last_peered":"2024-01-24T17:48:26.912432+0000","last_clean":"2024-01-24T17:48:26.912432+0000","last_became_active":"2024-01-24T17:48:26.438127+0000","last_became_peered":"2024-01-24T17:48:26.438127+0000","last_unstale":"2024-01-24T17:48:26.912432+0000","last_undegraded":"2024-01-24T17:48:26.912432+0000","last_fullsized":"2024-01-24T17:48:26.912432+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:19:53.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.924911+0000","last_change":"2024-01-24T17:48:26.918526+0000","last_active":"2024-01-24T17:48:26.924911+0000","last_peered":"2024-01-24T17:48:26.924911+0000","last_clean":"2024-01-24T17:48:26.924911+0000","last_became_active":"2024-01-24T17:48:26.918296+0000","last_became_peered":"2024-01-24T17:48:26.918296+0000","last_unstale":"2024-01-24T17:48:26.924911+0000","last_undegraded":"2024-01-24T17:48:26.924911+0000","last_fullsized":"2024-01-24T17:48:26.924911+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:40:16.702286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.915916+0000","last_change":"2024-01-24T17:48:26.438598+0000","last_active":"2024-01-24T17:48:26.915916+0000","last_peered":"2024-01-24T17:48:26.915916+0000","last_clean":"2024-01-24T17:48:26.915916+0000","last_became_active":"2024-01-24T17:48:26.437938+0000","last_became_peered":"2024-01-24T17:48:26.437938+0000","last_unstale":"2024-01-24T17:48:26.915916+0000","last_undegraded":"2024-01-24T17:48:26.915916+0000","last_fullsized":"2024-01-24T17:48:26.915916+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:47.757639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933545+0000","last_change":"2024-01-24T17:48:26.930709+0000","last_active":"2024-01-24T17:48:26.933545+0000","last_peered":"2024-01-24T17:48:26.933545+0000","last_clean":"2024-01-24T17:48:26.933545+0000","last_became_active":"2024-01-24T17:48:26.930432+0000","last_became_peered":"2024-01-24T17:48:26.930432+0000","last_unstale":"2024-01-24T17:48:26.933545+0000","last_undegraded":"2024-01-24T17:48:26.933545+0000","last_fullsized":"2024-01-24T17:48:26.933545+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:47:44.117143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.f","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.913974+0000","last_change":"2024-01-24T17:48:26.436022+0000","last_active":"2024-01-24T17:48:26.913974+0000","last_peered":"2024-01-24T17:48:26.913974+0000","last_clean":"2024-01-24T17:48:26.913974+0000","last_became_active":"2024-01-24T17:48:26.435638+0000","last_became_peered":"2024-01-24T17:48:26.435638+0000","last_unstale":"2024-01-24T17:48:26.913974+0000","last_undegraded":"2024-01-24T17:48:26.913974+0000","last_fullsized":"2024-01-24T17:48:26.913974+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:55:40.269339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":235,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.424336+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:48:26.424336+0000","last_peered":"2024-01-24T17:48:26.424336+0000","last_clean":"2024-01-24T17:48:26.424336+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:48:26.424336+0000","last_undegraded":"2024-01-24T17:48:26.424336+0000","last_fullsized":"2024-01-24T17:48:26.424336+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574936,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1228800,"data_stored":526817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.081,"15min":1.0349999999999999},"min":{"1min":0.70499999999999996,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.1399999999999999,"15min":1.0640000000000001},"min":{"1min":0.70099999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.91300000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.093,"15min":1.1020000000000001},"min":{"1min":0.76300000000000001,"5min":0.621,"15min":0.621},"max":{"1min":1.6479999999999999,"5min":2.04,"15min":2.04},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.202,"15min":1.198},"min":{"1min":0.81399999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.944,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.218,"15min":1.1739999999999999},"min":{"1min":0.75800000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.992,"5min":1.992,"15min":2.1400000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.1819999999999999,"15min":1.2190000000000001},"min":{"1min":0.76000000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7769999999999999,"5min":2.484,"15min":2.484},"last":1.343}]},{"osd":3,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2390000000000001,"15min":1.25},"min":{"1min":0.75900000000000001,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7450000000000001,"5min":2.0179999999999998,"15min":2.2930000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3140000000000001,"15min":1.3200000000000001},"min":{"1min":0.872,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.833,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.0820000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.28,"15min":1.3149999999999999},"min":{"1min":0.85799999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.6179999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.423,"15min":1.4339999999999999},"min":{"1min":1.212,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.9330000000000001,"5min":2.5449999999999999,"15min":2.6459999999999999},"last":1.202}]},{"osd":5,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3029999999999999,"15min":1.347},"min":{"1min":1.131,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7070000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.163},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.361,"15min":1.423},"min":{"1min":0.97999999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2480000000000002,"5min":2.2480000000000002,"15min":2.3719999999999999},"last":1.032}]},{"osd":6,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.468,"15min":1.5249999999999999},"min":{"1min":1.1990000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.4129999999999998,"5min":2.5720000000000001,"15min":2.6960000000000002},"last":1.125},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.4370000000000001,"15min":1.476},"min":{"1min":1.0920000000000001,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.923,"5min":2.2450000000000001,"15min":2.4830000000000001},"last":1.425}]}]},{"osd":6,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1212416,"data_stored":519460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.081,"15min":1.04},"min":{"1min":0.748,"5min":0.623,"15min":0.61199999999999999},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.0920000000000001,"15min":1.0349999999999999},"min":{"1min":0.79100000000000004,"5min":0.69499999999999995,"15min":0.66300000000000003},"max":{"1min":1.4390000000000001,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.113,"15min":1.022},"min":{"1min":0.78400000000000003,"5min":0.66700000000000004,"15min":0.64200000000000002},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.234},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.173,"15min":1.131},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.9129999999999998,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1759999999999999,"15min":1.1479999999999999},"min":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.63600000000000001},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1919999999999999,"15min":1.1739999999999999},"min":{"1min":0.755,"5min":0.69099999999999995,"15min":0.56599999999999995},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.0910000000000002},"last":1.5069999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.236,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.75,"5min":2.0009999999999999,"15min":2.1669999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2090000000000001,"15min":1.2450000000000001},"min":{"1min":0.95299999999999996,"5min":0.81899999999999995,"15min":0.69799999999999995},"max":{"1min":1.625,"5min":1.758,"15min":2.46},"last":1.347}]},{"osd":4,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3089999999999999,"15min":1.2549999999999999},"min":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.877,"5min":1.9179999999999999,"15min":1.988},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.753,"5min":0.73899999999999999,"15min":0.44600000000000001},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.1619999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.339,"15min":1.3069999999999999},"min":{"1min":0.77600000000000002,"5min":0.65300000000000002,"15min":0.46800000000000003},"max":{"1min":1.694,"5min":1.8879999999999999,"15min":2.2170000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.4419999999999999,"15min":1.4399999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.9750000000000001,"5min":2.0030000000000001,"15min":2.4849999999999999},"last":1.4790000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3480000000000001,"15min":1.365},"min":{"1min":0.91200000000000003,"5min":0.85399999999999998,"15min":0.51400000000000001},"max":{"1min":1.911,"5min":1.978,"15min":2.2050000000000001},"last":1.298},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4470000000000001,"15min":1.4610000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.52300000000000002},"max":{"1min":1.7649999999999999,"5min":1.9019999999999999,"15min":2.6920000000000002},"last":1.7170000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574937,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1806336,"data_stored":1109828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0309999999999999,"15min":1.087},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5389999999999999,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":3.609},{"interface":"front","average":{"1min":1.012,"5min":1.0329999999999999,"15min":1.032},"min":{"1min":0.73099999999999998,"5min":0.67300000000000004,"15min":0.65300000000000002},"max":{"1min":1.28,"5min":1.889,"15min":1.889},"last":3.7240000000000002}]},{"osd":1,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.0920000000000001,"15min":1.079},"min":{"1min":0.71199999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.7030000000000001,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":3.5470000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1499999999999999,"15min":1.24},"min":{"1min":0.76200000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.657,"5min":1.657,"15min":2.1579999999999999},"last":1.4219999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.212,"15min":1.226},"min":{"1min":0.85299999999999998,"5min":0.79400000000000004,"15min":0.59199999999999997},"max":{"1min":1.6339999999999999,"5min":1.655,"15min":1.8400000000000001},"last":2.8860000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.242,"15min":1.3},"min":{"1min":0.79700000000000004,"5min":0.67600000000000005,"15min":0.66300000000000003},"max":{"1min":1.5669999999999999,"5min":1.9870000000000001,"15min":3.1349999999999998},"last":8.1509999999999998}]},{"osd":3,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.208,"15min":1.3129999999999999},"min":{"1min":0.82399999999999995,"5min":0.76400000000000001,"15min":0.70799999999999996},"max":{"1min":1.611,"5min":2.0150000000000001,"15min":3.036},"last":2.7559999999999998},{"interface":"front","average":{"1min":1.26,"5min":1.262,"15min":1.3169999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.5980000000000001,"5min":1.7290000000000001,"15min":1.7769999999999999},"last":4.1230000000000002}]},{"osd":4,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.401,"15min":1.4419999999999999},"min":{"1min":0.58299999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.7130000000000001,"5min":2.069,"15min":2.069},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.274,"15min":1.3420000000000001},"min":{"1min":0.87,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.837,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":2.1139999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3080000000000001,"15min":1.3460000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.028,"5min":2.504,"15min":2.504},"last":1.8640000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.4530000000000001,"15min":1.506},"min":{"1min":0.71399999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.806,"5min":2.1579999999999999,"15min":2.2290000000000001},"last":1.52}]},{"osd":7,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3680000000000001,"15min":1.4630000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.8420000000000001,"5min":3.161,"15min":3.161},"last":2.2010000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.508,"15min":1.607},"min":{"1min":0.78400000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.113,"5min":2.1389999999999998,"15min":3.2410000000000001},"last":1.978}]}]},{"osd":4,"up_from":13,"seq":55834574938,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1212416,"data_stored":519460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.1000000000000001,"15min":1.056},"min":{"1min":0.72099999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6120000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.095,"5min":1.087,"15min":1.0940000000000001},"min":{"1min":0.75600000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.464,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.1659999999999999,"15min":1.0800000000000001},"min":{"1min":0.77500000000000002,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.113,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.159,"5min":1.1319999999999999,"15min":1.198},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5029999999999999,"5min":1.849,"15min":1.849},"last":1.1419999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.2110000000000001,"15min":1.214},"min":{"1min":0.89100000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.9850000000000001,"5min":2.105,"15min":2.105},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.2669999999999999,"15min":1.3220000000000001},"min":{"1min":0.95399999999999996,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.6619999999999999,"5min":1.986,"15min":1.986},"last":0.96399999999999997}]},{"osd":3,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1719999999999999,"15min":1.1870000000000001},"min":{"1min":0.69599999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4330000000000001,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.268,"15min":1.3260000000000001},"min":{"1min":0.80100000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.7929999999999999,"5min":1.8029999999999999,"15min":2.9409999999999998},"last":1.6679999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2729999999999999,"15min":1.3480000000000001},"min":{"1min":0.76900000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.625,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.25},{"interface":"front","average":{"1min":1.252,"5min":1.256,"15min":1.292},"min":{"1min":0.93200000000000005,"5min":0.65600000000000003,"15min":0.55200000000000005},"max":{"1min":1.738,"5min":1.738,"15min":1.9039999999999999},"last":1.2969999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.88900000000000001,"5min":0.67800000000000005,"15min":0.64400000000000002},"max":{"1min":2.1930000000000001,"5min":2.3159999999999998,"15min":3.0840000000000001},"last":1.722},{"interface":"front","average":{"1min":1.464,"5min":1.4139999999999999,"15min":1.4630000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.66500000000000004},"max":{"1min":1.931,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.532}]},{"osd":7,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.389,"15min":1.411},"min":{"1min":0.78600000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.627,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.411},{"interface":"front","average":{"1min":1.532,"5min":1.4790000000000001,"15min":1.581},"min":{"1min":1.008,"5min":0.79300000000000004,"15min":0.60399999999999998},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":4.3250000000000002},"last":1.7729999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574938,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1236992,"data_stored":526429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90100000000000002,"15min":0.86699999999999999},"min":{"1min":0.53800000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.2909999999999999,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.97399999999999998,"15min":0.89800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.3360000000000001,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.80800000000000005}]},{"osd":1,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.92800000000000005},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.2789999999999999,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.054,"15min":1.0600000000000001},"min":{"1min":0.57399999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.45,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.69899999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0860000000000001,"15min":1.1040000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.502,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1659999999999999,"15min":1.159},"min":{"1min":0.60299999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.7250000000000001,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.76100000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.282,"15min":1.28},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.0049999999999999,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.353,"5min":1.3080000000000001,"15min":1.3400000000000001},"min":{"1min":0.85699999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.6399999999999999,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.177}]},{"osd":5,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.367,"15min":1.415},"min":{"1min":1.006,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.6659999999999999,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.095},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.3859999999999999,"15min":1.4139999999999999},"min":{"1min":0.80800000000000005,"5min":0.746,"15min":0.746},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.98899999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.5029999999999999,"15min":1.4930000000000001},"min":{"1min":0.90600000000000003,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.3210000000000002,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.397,"5min":1.415,"15min":1.444},"min":{"1min":0.76500000000000001,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.093,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.0009999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.4750000000000001,"15min":1.4950000000000001},"min":{"1min":0.97399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1829999999999998,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.206},{"interface":"front","average":{"1min":1.45,"5min":1.502,"15min":1.512},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.77,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.0489999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1212416,"data_stored":519460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78900000000000003,"15min":0.79300000000000004},"min":{"1min":0.443,"5min":0.318,"15min":0.318},"max":{"1min":1.3420000000000001,"5min":1.667,"15min":1.667},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.66200000000000003,"5min":0.375,"15min":0.375},"max":{"1min":1.4370000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0109999999999999,"15min":1.0469999999999999},"min":{"1min":0.505,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.569,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.296},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.111,"15min":1.1950000000000001},"min":{"1min":0.76300000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.677,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.6499999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94299999999999995,"15min":0.98299999999999998},"min":{"1min":0.51900000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.5089999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.26},{"interface":"front","average":{"1min":1.129,"5min":1.135,"15min":1.1950000000000001},"min":{"1min":0.68000000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5680000000000001,"5min":2.996,"15min":2.996},"last":1.5249999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.248,"15min":1.3100000000000001},"min":{"1min":0.78500000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.5,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.397},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.2589999999999999,"15min":1.3779999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.831,"5min":1.889,"15min":2.3839999999999999},"last":1.349}]},{"osd":5,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3009999999999999,"15min":1.389},"min":{"1min":0.873,"5min":0.73999999999999999,"15min":0.73899999999999999},"max":{"1min":1.851,"5min":1.851,"15min":2.3140000000000001},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.202,"15min":1.3},"min":{"1min":0.754,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.4750000000000001,"5min":1.889,"15min":2.0720000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.304,"15min":1.3799999999999999},"min":{"1min":0.63200000000000001,"5min":0.623,"15min":0.623},"max":{"1min":1.615,"5min":1.9350000000000001,"15min":2.2679999999999998},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.3380000000000001,"15min":1.4490000000000001},"min":{"1min":0.95499999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.782,"5min":2.2530000000000001,"15min":3.7410000000000001},"last":1.589}]},{"osd":7,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.401,"15min":1.5449999999999999},"min":{"1min":0.81999999999999995,"5min":0.80800000000000005,"15min":0.80300000000000005},"max":{"1min":1.9670000000000001,"5min":2.1499999999999999,"15min":3.343},"last":1.7490000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.367,"15min":1.48},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.7190000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.716}]}]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1806336,"data_stored":1109828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88200000000000001,"15min":0.86299999999999999},"min":{"1min":0.43099999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2769999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.92100000000000004,"15min":0.94999999999999996},"min":{"1min":0.62,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.395,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.80000000000000004}]},{"osd":2,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.875,"15min":0.95799999999999996},"min":{"1min":0.54700000000000004,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3140000000000001,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.0509999999999999,"15min":1.1000000000000001},"min":{"1min":0.67100000000000004,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4239999999999999,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.262}]},{"osd":3,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.141,"15min":1.163},"min":{"1min":0.54500000000000004,"5min":0.53000000000000003,"15min":0.47699999999999998},"max":{"1min":1.8440000000000001,"5min":1.863,"15min":1.9610000000000001},"last":1.5},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.073,"15min":1.204},"min":{"1min":0.68200000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":3.149},"last":1.46}]},{"osd":4,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.333,"15min":1.359},"min":{"1min":0.96799999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.772,"5min":1.962,"15min":2.2410000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.244,"5min":1.2609999999999999,"15min":1.3120000000000001},"min":{"1min":0.871,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.734,"5min":1.734,"15min":2.3849999999999998},"last":1.5369999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.2729999999999999,"15min":1.341},"min":{"1min":0.76700000000000002,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.964,"5min":1.964,"15min":2.577},"last":1.629},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.363,"15min":1.377},"min":{"1min":0.92400000000000004,"5min":0.83199999999999996,"15min":0.81299999999999994},"max":{"1min":2.2770000000000001,"5min":2.2770000000000001,"15min":2.9809999999999999},"last":2.169}]},{"osd":6,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.3400000000000001,"15min":1.4239999999999999},"min":{"1min":0.94099999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7929999999999999,"5min":2.008,"15min":3.2850000000000001},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.345,"15min":1.4930000000000001},"min":{"1min":0.96199999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":2.077,"15min":3.5859999999999999},"last":1.7190000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.415,"15min":1.4610000000000001},"min":{"1min":1.0249999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":2.8439999999999999},"last":2.0819999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.4339999999999999,"15min":1.532},"min":{"1min":0.89700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":3.4350000000000001},"last":1.6779999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1212416,"data_stored":519460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.79800000000000004,"15min":0.81699999999999995},"min":{"1min":0.60899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.7290000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.92600000000000005,"15min":0.96799999999999997},"min":{"1min":0.433,"5min":0.376,"15min":0.376},"max":{"1min":1.5309999999999999,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.621}]},{"osd":2,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.86799999999999999,"15min":0.84199999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.34699999999999998},"max":{"1min":1.3260000000000001,"5min":1.331,"15min":1.331},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.117,"5min":0.98899999999999999,"15min":1.081},"min":{"1min":0.50600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.5049999999999999,"5min":1.55,"15min":2.0499999999999998},"last":0.91500000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1040000000000001,"15min":1.1779999999999999},"min":{"1min":0.59399999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.9670000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.173,"15min":1.1910000000000001},"min":{"1min":0.49299999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.806,"5min":1.806,"15min":2.407},"last":0.84499999999999997}]},{"osd":4,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.2529999999999999,"15min":1.385},"min":{"1min":1.0129999999999999,"5min":0.73299999999999998,"15min":0.70799999999999996},"max":{"1min":1.6279999999999999,"5min":2.056,"15min":2.2589999999999999},"last":1.431},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.2889999999999999,"15min":1.446},"min":{"1min":1.01,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":3.3740000000000001},"last":1.325}]},{"osd":5,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.2689999999999999,"15min":1.375},"min":{"1min":0.90700000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.746,"5min":1.7490000000000001,"15min":2.9300000000000002},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.246,"15min":1.2609999999999999},"min":{"1min":0.88600000000000001,"5min":0.77000000000000002,"15min":0.65600000000000003},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":2.3799999999999999},"last":1.55}]},{"osd":6,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.327,"15min":1.357},"min":{"1min":0.83399999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.954,"5min":1.954,"15min":2.335},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.3799999999999999,"15min":1.4490000000000001},"min":{"1min":0.92100000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.931,"15min":2.2389999999999999},"last":1.246}]},{"osd":7,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.381,"15min":1.5069999999999999},"min":{"1min":0.97699999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":3.0230000000000001},"last":1.347},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.4450000000000001,"15min":1.5740000000000001},"min":{"1min":0.99299999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":3.1600000000000001},"last":1.2989999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:48:31.832 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:48:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.087+0000 7f8476a00640 1 Processor -- start 2024-01-24T17:48:32.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.087+0000 7f8476a00640 1 -- start start 2024-01-24T17:48:32.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.089+0000 7f8476a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f8470070920 0x7f84700b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.089+0000 7f8476a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8470070470 con 0x7f84700b0010 2024-01-24T17:48:32.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.089+0000 7f8476a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847015ef00 con 0x7f847015f390 2024-01-24T17:48:32.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.089+0000 7f8476a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84700710a0 con 0x7f8470070920 2024-01-24T17:48:32.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.089+0000 7f846fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f84700b0010 0x7f847015bb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39438/0 (socket says 172.21.15.39:39438) 2024-01-24T17:48:32.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.089+0000 7f846fe00640 1 -- 172.21.15.39:0/480097579 learned_addr learned my addr 172.21.15.39:0/480097579 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f8467e00640 1 --2- 172.21.15.39:0/480097579 >> v2:172.21.15.100:3300/0 conn(0x7f8470070920 0x7f84700b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845402543 0 0) 0x7f8470070470 con 0x7f84700b0010 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8450003520 con 0x7f84700b0010 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3693723262 0 0) 0x7f8450003520 con 0x7f84700b0010 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 >> v2:172.21.15.100:3300/0 conn(0x7f8470070920 msgr2=0x7f84700b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 --2- 172.21.15.39:0/480097579 >> v2:172.21.15.100:3300/0 conn(0x7f8470070920 0x7f84700b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 >> v1:172.21.15.100:6789/0 conn(0x7f847015f390 legacy=0x7f84700c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84700b0930 con 0x7f84700b0010 2024-01-24T17:48:32.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8460001670 con 0x7f84700b0010 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 >> v1:172.21.15.39:6789/0 conn(0x7f84700b0010 legacy=0x7f847015bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84500039c0 0x7f8450003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84500047c0 0x7f8450004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8450005180 0x7f845000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84700710a0 con 0x7f84500047c0 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845000c060 con 0x7f84500039c0 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.090+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845000c220 con 0x7f8450005180 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8467e00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84500047c0 0x7f8450004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f846fe00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84500039c0 0x7f8450003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8474a00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8450005180 0x7f845000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8467e00640 1 -- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8450005180 msgr2=0x7f845000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:32.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8467e00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8450005180 0x7f845000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8467e00640 1 -- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84500039c0 msgr2=0x7f8450003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8467e00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84500039c0 0x7f8450003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8467e00640 1 -- 172.21.15.39:0/480097579 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450003520 con 0x7f84500047c0 2024-01-24T17:48:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8474a00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f8450005180 0x7f845000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:32.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.091+0000 7f8474a00640 1 -- 172.21.15.39:0/480097579 reap_dead start 2024-01-24T17:48:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.092+0000 7f8467e00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84500047c0 0x7f8450004c40 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f8458009ac0 tx=0x7f84500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f974e376ec35430 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.092+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845802c6b0 con 0x7f84500047c0 2024-01-24T17:48:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845802c810 con 0x7f84500047c0 2024-01-24T17:48:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f846f400640 1 -- 172.21.15.39:0/480097579 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845802cba0 con 0x7f84500047c0 2024-01-24T17:48:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f8476a00640 1 -- 172.21.15.39:0/480097579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84500047c0 msgr2=0x7f8450004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f8476a00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84500047c0 0x7f8450004c40 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f8458009ac0 tx=0x7f84500057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f8476a00640 1 -- 172.21.15.39:0/480097579 shutdown_connections 2024-01-24T17:48:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f8476a00640 1 --2- 172.21.15.39:0/480097579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84500047c0 0x7f8450004c40 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f8476a00640 1 -- 172.21.15.39:0/480097579 >> 172.21.15.39:0/480097579 conn(0x7f847006bf20 msgr2=0x7f847006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f8476a00640 1 -- 172.21.15.39:0/480097579 shutdown_connections 2024-01-24T17:48:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.093+0000 7f8476a00640 1 -- 172.21.15.39:0/480097579 wait complete. 2024-01-24T17:48:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.095+0000 7f8476a00640 1 Processor -- start 2024-01-24T17:48:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.095+0000 7f8476a00640 1 -- start start 2024-01-24T17:48:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.096+0000 7f8476a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84500047c0 0x7f84700ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.096+0000 7f8476a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84700cd240 0x7f84700cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.096+0000 7f8476a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84700cafa0 0x7f84700cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.096+0000 7f8476a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84701f35a0 con 0x7f84700cafa0 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.096+0000 7f8476a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84701f2a10 con 0x7f84700cd240 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.096+0000 7f8476a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84700cdbe0 con 0x7f84500047c0 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.096+0000 7f8474a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84700cafa0 0x7f84700cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8474a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84700cafa0 0x7f84700cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49776/0 (socket says 172.21.15.39:49776) 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8474a00640 1 -- 172.21.15.39:0/760509378 learned_addr learned my addr 172.21.15.39:0/760509378 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f846fe00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84500047c0 0x7f84700ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8467e00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84700cd240 0x7f84700cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8474a00640 1 -- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84500047c0 msgr2=0x7f84700ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8474a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84500047c0 0x7f84700ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8474a00640 1 -- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84700cd240 msgr2=0x7f84700cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8474a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84700cd240 0x7f84700cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8474a00640 1 -- 172.21.15.39:0/760509378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847015a450 con 0x7f84700cafa0 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.097+0000 7f8474a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84700cafa0 0x7f84700cb420 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f84700c5610 tx=0x7f846800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.098+0000 7f846cc00640 1 -- 172.21.15.39:0/760509378 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8468003b70 con 0x7f84700cafa0 2024-01-24T17:48:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.098+0000 7f846cc00640 1 -- 172.21.15.39:0/760509378 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8468003cd0 con 0x7f84700cafa0 2024-01-24T17:48:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.098+0000 7f846cc00640 1 -- 172.21.15.39:0/760509378 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84680052b0 con 0x7f84700cafa0 2024-01-24T17:48:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.098+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84700cb960 con 0x7f84700cafa0 2024-01-24T17:48:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.098+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84700cde20 con 0x7f84700cafa0 2024-01-24T17:48:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.099+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84701f36e0 con 0x7f84700cafa0 2024-01-24T17:48:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.100+0000 7f846cc00640 1 -- 172.21.15.39:0/760509378 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f8468005410 con 0x7f84700cafa0 2024-01-24T17:48:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.101+0000 7f846cc00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f844806a460 0x7f844806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.101+0000 7f846cc00640 1 -- 172.21.15.39:0/760509378 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8468088220 con 0x7f84700cafa0 2024-01-24T17:48:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.102+0000 7f846fe00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f844806a460 0x7f844806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.103+0000 7f846fe00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f844806a460 0x7f844806c920 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8460001050 tx=0x7f8460009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:32.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.105+0000 7f846cc00640 1 -- 172.21.15.39:0/760509378 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f84701f36e0 con 0x7f84700cafa0 2024-01-24T17:48:32.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.295+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f847015e8b0 con 0x7f844806a460 2024-01-24T17:48:32.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.318+0000 7f846cc00640 1 -- 172.21.15.39:0/760509378 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287090 (secure 0 0 0) 0x7f8460010860 con 0x7f844806a460 2024-01-24T17:48:32.323 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:32.326 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:48:32.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f844806a460 msgr2=0x7f844806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f844806a460 0x7f844806c920 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8460001050 tx=0x7f8460009040 comp rx=0 tx=0).stop 2024-01-24T17:48:32.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84700cafa0 msgr2=0x7f84700cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84700cafa0 0x7f84700cb420 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f84700c5610 tx=0x7f846800dc00 comp rx=0 tx=0).stop 2024-01-24T17:48:32.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 shutdown_connections 2024-01-24T17:48:32.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f844806a460 0x7f844806c920 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84700cafa0 0x7f84700cb420 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84700cd240 0x7f84700cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 --2- 172.21.15.39:0/760509378 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f84500047c0 0x7f84700ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 >> 172.21.15.39:0/760509378 conn(0x7f847006bf20 msgr2=0x7f84700c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:32.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.324+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 shutdown_connections 2024-01-24T17:48:32.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.325+0000 7f8476a00640 1 -- 172.21.15.39:0/760509378 wait complete. 2024-01-24T17:48:32.345 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-01-24T17:48:30.359933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259612,"kb_used_data":10672,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754715108,"statfs":{"total":773094113280,"available":772828270592,"internally_reserved":0,"allocated":10928128,"data_stored":5350742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.979674"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929630+0000","last_change":"2024-01-24T17:48:26.919046+0000","last_active":"2024-01-24T17:48:26.929630+0000","last_peered":"2024-01-24T17:48:26.929630+0000","last_clean":"2024-01-24T17:48:26.929630+0000","last_became_active":"2024-01-24T17:48:26.917400+0000","last_became_peered":"2024-01-24T17:48:26.917400+0000","last_unstale":"2024-01-24T17:48:26.929630+0000","last_undegraded":"2024-01-24T17:48:26.929630+0000","last_fullsized":"2024-01-24T17:48:26.929630+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:27:29.531410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.919147+0000","last_change":"2024-01-24T17:48:26.913733+0000","last_active":"2024-01-24T17:48:26.919147+0000","last_peered":"2024-01-24T17:48:26.919147+0000","last_clean":"2024-01-24T17:48:26.919147+0000","last_became_active":"2024-01-24T17:48:26.913390+0000","last_became_peered":"2024-01-24T17:48:26.913390+0000","last_unstale":"2024-01-24T17:48:26.919147+0000","last_undegraded":"2024-01-24T17:48:26.919147+0000","last_fullsized":"2024-01-24T17:48:26.919147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:40:28.505137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.931746+0000","last_change":"2024-01-24T17:48:26.925051+0000","last_active":"2024-01-24T17:48:26.931746+0000","last_peered":"2024-01-24T17:48:26.931746+0000","last_clean":"2024-01-24T17:48:26.931746+0000","last_became_active":"2024-01-24T17:48:26.924187+0000","last_became_peered":"2024-01-24T17:48:26.924187+0000","last_unstale":"2024-01-24T17:48:26.931746+0000","last_undegraded":"2024-01-24T17:48:26.931746+0000","last_fullsized":"2024-01-24T17:48:26.931746+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:16:26.055063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930228+0000","last_change":"2024-01-24T17:48:26.918982+0000","last_active":"2024-01-24T17:48:26.930228+0000","last_peered":"2024-01-24T17:48:26.930228+0000","last_clean":"2024-01-24T17:48:26.930228+0000","last_became_active":"2024-01-24T17:48:26.918660+0000","last_became_peered":"2024-01-24T17:48:26.918660+0000","last_unstale":"2024-01-24T17:48:26.930228+0000","last_undegraded":"2024-01-24T17:48:26.930228+0000","last_fullsized":"2024-01-24T17:48:26.930228+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:06:33.271424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.926293+0000","last_change":"2024-01-24T17:48:26.923046+0000","last_active":"2024-01-24T17:48:26.926293+0000","last_peered":"2024-01-24T17:48:26.926293+0000","last_clean":"2024-01-24T17:48:26.926293+0000","last_became_active":"2024-01-24T17:48:26.922781+0000","last_became_peered":"2024-01-24T17:48:26.922781+0000","last_unstale":"2024-01-24T17:48:26.926293+0000","last_undegraded":"2024-01-24T17:48:26.926293+0000","last_fullsized":"2024-01-24T17:48:26.926293+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:33:54.874293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.926283+0000","last_change":"2024-01-24T17:48:26.921781+0000","last_active":"2024-01-24T17:48:26.926283+0000","last_peered":"2024-01-24T17:48:26.926283+0000","last_clean":"2024-01-24T17:48:26.926283+0000","last_became_active":"2024-01-24T17:48:26.921087+0000","last_became_peered":"2024-01-24T17:48:26.921087+0000","last_unstale":"2024-01-24T17:48:26.926283+0000","last_undegraded":"2024-01-24T17:48:26.926283+0000","last_fullsized":"2024-01-24T17:48:26.926283+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:00:45.740090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.919364+0000","last_change":"2024-01-24T17:48:26.915775+0000","last_active":"2024-01-24T17:48:26.919364+0000","last_peered":"2024-01-24T17:48:26.919364+0000","last_clean":"2024-01-24T17:48:26.919364+0000","last_became_active":"2024-01-24T17:48:26.915490+0000","last_became_peered":"2024-01-24T17:48:26.915490+0000","last_unstale":"2024-01-24T17:48:26.919364+0000","last_undegraded":"2024-01-24T17:48:26.919364+0000","last_fullsized":"2024-01-24T17:48:26.919364+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:32:22.150580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930305+0000","last_change":"2024-01-24T17:48:26.920702+0000","last_active":"2024-01-24T17:48:26.930305+0000","last_peered":"2024-01-24T17:48:26.930305+0000","last_clean":"2024-01-24T17:48:26.930305+0000","last_became_active":"2024-01-24T17:48:26.920434+0000","last_became_peered":"2024-01-24T17:48:26.920434+0000","last_unstale":"2024-01-24T17:48:26.930305+0000","last_undegraded":"2024-01-24T17:48:26.930305+0000","last_fullsized":"2024-01-24T17:48:26.930305+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:51:40.896415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929286+0000","last_change":"2024-01-24T17:48:26.925409+0000","last_active":"2024-01-24T17:48:26.929286+0000","last_peered":"2024-01-24T17:48:26.929286+0000","last_clean":"2024-01-24T17:48:26.929286+0000","last_became_active":"2024-01-24T17:48:26.925233+0000","last_became_peered":"2024-01-24T17:48:26.925233+0000","last_unstale":"2024-01-24T17:48:26.929286+0000","last_undegraded":"2024-01-24T17:48:26.929286+0000","last_fullsized":"2024-01-24T17:48:26.929286+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:30:28.625862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.921359+0000","last_change":"2024-01-24T17:48:26.914375+0000","last_active":"2024-01-24T17:48:26.921359+0000","last_peered":"2024-01-24T17:48:26.921359+0000","last_clean":"2024-01-24T17:48:26.921359+0000","last_became_active":"2024-01-24T17:48:26.914123+0000","last_became_peered":"2024-01-24T17:48:26.914123+0000","last_unstale":"2024-01-24T17:48:26.921359+0000","last_undegraded":"2024-01-24T17:48:26.921359+0000","last_fullsized":"2024-01-24T17:48:26.921359+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:18:48.467429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.918930+0000","last_change":"2024-01-24T17:48:26.913663+0000","last_active":"2024-01-24T17:48:26.918930+0000","last_peered":"2024-01-24T17:48:26.918930+0000","last_clean":"2024-01-24T17:48:26.918930+0000","last_became_active":"2024-01-24T17:48:26.913375+0000","last_became_peered":"2024-01-24T17:48:26.913375+0000","last_unstale":"2024-01-24T17:48:26.918930+0000","last_undegraded":"2024-01-24T17:48:26.918930+0000","last_fullsized":"2024-01-24T17:48:26.918930+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:14:01.025561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929731+0000","last_change":"2024-01-24T17:48:26.921337+0000","last_active":"2024-01-24T17:48:26.929731+0000","last_peered":"2024-01-24T17:48:26.929731+0000","last_clean":"2024-01-24T17:48:26.929731+0000","last_became_active":"2024-01-24T17:48:26.921068+0000","last_became_peered":"2024-01-24T17:48:26.921068+0000","last_unstale":"2024-01-24T17:48:26.929731+0000","last_undegraded":"2024-01-24T17:48:26.929731+0000","last_fullsized":"2024-01-24T17:48:26.929731+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:50.103242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.940117+0000","last_change":"2024-01-24T17:48:26.932900+0000","last_active":"2024-01-24T17:48:26.940117+0000","last_peered":"2024-01-24T17:48:26.940117+0000","last_clean":"2024-01-24T17:48:26.940117+0000","last_became_active":"2024-01-24T17:48:26.930460+0000","last_became_peered":"2024-01-24T17:48:26.930460+0000","last_unstale":"2024-01-24T17:48:26.940117+0000","last_undegraded":"2024-01-24T17:48:26.940117+0000","last_fullsized":"2024-01-24T17:48:26.940117+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:38:44.025239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.935362+0000","last_change":"2024-01-24T17:48:26.926123+0000","last_active":"2024-01-24T17:48:26.935362+0000","last_peered":"2024-01-24T17:48:26.935362+0000","last_clean":"2024-01-24T17:48:26.935362+0000","last_became_active":"2024-01-24T17:48:26.925865+0000","last_became_peered":"2024-01-24T17:48:26.925865+0000","last_unstale":"2024-01-24T17:48:26.935362+0000","last_undegraded":"2024-01-24T17:48:26.935362+0000","last_fullsized":"2024-01-24T17:48:26.935362+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:57:05.790265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.935504+0000","last_change":"2024-01-24T17:48:26.928443+0000","last_active":"2024-01-24T17:48:26.935504+0000","last_peered":"2024-01-24T17:48:26.935504+0000","last_clean":"2024-01-24T17:48:26.935504+0000","last_became_active":"2024-01-24T17:48:26.927549+0000","last_became_peered":"2024-01-24T17:48:26.927549+0000","last_unstale":"2024-01-24T17:48:26.935504+0000","last_undegraded":"2024-01-24T17:48:26.935504+0000","last_fullsized":"2024-01-24T17:48:26.935504+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:51:23.333640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.924180+0000","last_change":"2024-01-24T17:48:26.918229+0000","last_active":"2024-01-24T17:48:26.924180+0000","last_peered":"2024-01-24T17:48:26.924180+0000","last_clean":"2024-01-24T17:48:26.924180+0000","last_became_active":"2024-01-24T17:48:26.917618+0000","last_became_peered":"2024-01-24T17:48:26.917618+0000","last_unstale":"2024-01-24T17:48:26.924180+0000","last_undegraded":"2024-01-24T17:48:26.924180+0000","last_fullsized":"2024-01-24T17:48:26.924180+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:15:06.378822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933724+0000","last_change":"2024-01-24T17:48:26.930098+0000","last_active":"2024-01-24T17:48:26.933724+0000","last_peered":"2024-01-24T17:48:26.933724+0000","last_clean":"2024-01-24T17:48:26.933724+0000","last_became_active":"2024-01-24T17:48:26.929843+0000","last_became_peered":"2024-01-24T17:48:26.929843+0000","last_unstale":"2024-01-24T17:48:26.933724+0000","last_undegraded":"2024-01-24T17:48:26.933724+0000","last_fullsized":"2024-01-24T17:48:26.933724+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T01:33:37.315009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.935532+0000","last_change":"2024-01-24T17:48:26.928528+0000","last_active":"2024-01-24T17:48:26.935532+0000","last_peered":"2024-01-24T17:48:26.935532+0000","last_clean":"2024-01-24T17:48:26.935532+0000","last_became_active":"2024-01-24T17:48:26.927985+0000","last_became_peered":"2024-01-24T17:48:26.927985+0000","last_unstale":"2024-01-24T17:48:26.935532+0000","last_undegraded":"2024-01-24T17:48:26.935532+0000","last_fullsized":"2024-01-24T17:48:26.935532+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:16:11.808000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933318+0000","last_change":"2024-01-24T17:48:26.926495+0000","last_active":"2024-01-24T17:48:26.933318+0000","last_peered":"2024-01-24T17:48:26.933318+0000","last_clean":"2024-01-24T17:48:26.933318+0000","last_became_active":"2024-01-24T17:48:26.921079+0000","last_became_peered":"2024-01-24T17:48:26.921079+0000","last_unstale":"2024-01-24T17:48:26.933318+0000","last_undegraded":"2024-01-24T17:48:26.933318+0000","last_fullsized":"2024-01-24T17:48:26.933318+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:50:11.805736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.925056+0000","last_change":"2024-01-24T17:48:26.917987+0000","last_active":"2024-01-24T17:48:26.925056+0000","last_peered":"2024-01-24T17:48:26.925056+0000","last_clean":"2024-01-24T17:48:26.925056+0000","last_became_active":"2024-01-24T17:48:26.917105+0000","last_became_peered":"2024-01-24T17:48:26.917105+0000","last_unstale":"2024-01-24T17:48:26.925056+0000","last_undegraded":"2024-01-24T17:48:26.925056+0000","last_fullsized":"2024-01-24T17:48:26.925056+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:19:08.635766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930071+0000","last_change":"2024-01-24T17:48:26.924914+0000","last_active":"2024-01-24T17:48:26.930071+0000","last_peered":"2024-01-24T17:48:26.930071+0000","last_clean":"2024-01-24T17:48:26.930071+0000","last_became_active":"2024-01-24T17:48:26.924548+0000","last_became_peered":"2024-01-24T17:48:26.924548+0000","last_unstale":"2024-01-24T17:48:26.930071+0000","last_undegraded":"2024-01-24T17:48:26.930071+0000","last_fullsized":"2024-01-24T17:48:26.930071+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:19:54.838909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.931875+0000","last_change":"2024-01-24T17:48:26.927897+0000","last_active":"2024-01-24T17:48:26.931875+0000","last_peered":"2024-01-24T17:48:26.931875+0000","last_clean":"2024-01-24T17:48:26.931875+0000","last_became_active":"2024-01-24T17:48:26.927529+0000","last_became_peered":"2024-01-24T17:48:26.927529+0000","last_unstale":"2024-01-24T17:48:26.931875+0000","last_undegraded":"2024-01-24T17:48:26.931875+0000","last_fullsized":"2024-01-24T17:48:26.931875+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:19:08.635766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929585+0000","last_change":"2024-01-24T17:48:26.917605+0000","last_active":"2024-01-24T17:48:26.929585+0000","last_peered":"2024-01-24T17:48:26.929585+0000","last_clean":"2024-01-24T17:48:26.929585+0000","last_became_active":"2024-01-24T17:48:26.917312+0000","last_became_peered":"2024-01-24T17:48:26.917312+0000","last_unstale":"2024-01-24T17:48:26.929585+0000","last_undegraded":"2024-01-24T17:48:26.929585+0000","last_fullsized":"2024-01-24T17:48:26.929585+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:16:42.532463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923167+0000","last_change":"2024-01-24T17:48:26.919152+0000","last_active":"2024-01-24T17:48:26.923167+0000","last_peered":"2024-01-24T17:48:26.923167+0000","last_clean":"2024-01-24T17:48:26.923167+0000","last_became_active":"2024-01-24T17:48:26.918789+0000","last_became_peered":"2024-01-24T17:48:26.918789+0000","last_unstale":"2024-01-24T17:48:26.923167+0000","last_undegraded":"2024-01-24T17:48:26.923167+0000","last_fullsized":"2024-01-24T17:48:26.923167+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:57:05.790265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.920877+0000","last_change":"2024-01-24T17:48:26.912980+0000","last_active":"2024-01-24T17:48:26.920877+0000","last_peered":"2024-01-24T17:48:26.920877+0000","last_clean":"2024-01-24T17:48:26.920877+0000","last_became_active":"2024-01-24T17:48:26.912720+0000","last_became_peered":"2024-01-24T17:48:26.912720+0000","last_unstale":"2024-01-24T17:48:26.920877+0000","last_undegraded":"2024-01-24T17:48:26.920877+0000","last_fullsized":"2024-01-24T17:48:26.920877+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:47:44.117143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933468+0000","last_change":"2024-01-24T17:48:26.928405+0000","last_active":"2024-01-24T17:48:26.933468+0000","last_peered":"2024-01-24T17:48:26.933468+0000","last_clean":"2024-01-24T17:48:26.933468+0000","last_became_active":"2024-01-24T17:48:26.928063+0000","last_became_peered":"2024-01-24T17:48:26.928063+0000","last_unstale":"2024-01-24T17:48:26.933468+0000","last_undegraded":"2024-01-24T17:48:26.933468+0000","last_fullsized":"2024-01-24T17:48:26.933468+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:40.339905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923128+0000","last_change":"2024-01-24T17:48:26.915142+0000","last_active":"2024-01-24T17:48:26.923128+0000","last_peered":"2024-01-24T17:48:26.923128+0000","last_clean":"2024-01-24T17:48:26.923128+0000","last_became_active":"2024-01-24T17:48:26.914859+0000","last_became_peered":"2024-01-24T17:48:26.914859+0000","last_unstale":"2024-01-24T17:48:26.923128+0000","last_undegraded":"2024-01-24T17:48:26.923128+0000","last_fullsized":"2024-01-24T17:48:26.923128+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:19:24.310411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.927560+0000","last_change":"2024-01-24T17:48:26.926259+0000","last_active":"2024-01-24T17:48:26.927560+0000","last_peered":"2024-01-24T17:48:26.927560+0000","last_clean":"2024-01-24T17:48:26.927560+0000","last_became_active":"2024-01-24T17:48:26.926005+0000","last_became_peered":"2024-01-24T17:48:26.926005+0000","last_unstale":"2024-01-24T17:48:26.927560+0000","last_undegraded":"2024-01-24T17:48:26.927560+0000","last_fullsized":"2024-01-24T17:48:26.927560+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:19:29.557377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933038+0000","last_change":"2024-01-24T17:48:26.926546+0000","last_active":"2024-01-24T17:48:26.933038+0000","last_peered":"2024-01-24T17:48:26.933038+0000","last_clean":"2024-01-24T17:48:26.933038+0000","last_became_active":"2024-01-24T17:48:26.926231+0000","last_became_peered":"2024-01-24T17:48:26.926231+0000","last_unstale":"2024-01-24T17:48:26.933038+0000","last_undegraded":"2024-01-24T17:48:26.933038+0000","last_fullsized":"2024-01-24T17:48:26.933038+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:40:28.505137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.940162+0000","last_change":"2024-01-24T17:48:26.934236+0000","last_active":"2024-01-24T17:48:26.940162+0000","last_peered":"2024-01-24T17:48:26.940162+0000","last_clean":"2024-01-24T17:48:26.940162+0000","last_became_active":"2024-01-24T17:48:26.933895+0000","last_became_peered":"2024-01-24T17:48:26.933895+0000","last_unstale":"2024-01-24T17:48:26.940162+0000","last_undegraded":"2024-01-24T17:48:26.940162+0000","last_fullsized":"2024-01-24T17:48:26.940162+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:24:02.371020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930187+0000","last_change":"2024-01-24T17:48:26.923487+0000","last_active":"2024-01-24T17:48:26.930187+0000","last_peered":"2024-01-24T17:48:26.930187+0000","last_clean":"2024-01-24T17:48:26.930187+0000","last_became_active":"2024-01-24T17:48:26.923151+0000","last_became_peered":"2024-01-24T17:48:26.923151+0000","last_unstale":"2024-01-24T17:48:26.930187+0000","last_undegraded":"2024-01-24T17:48:26.930187+0000","last_fullsized":"2024-01-24T17:48:26.930187+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:42:42.058056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923633+0000","last_change":"2024-01-24T17:48:26.915693+0000","last_active":"2024-01-24T17:48:26.923633+0000","last_peered":"2024-01-24T17:48:26.923633+0000","last_clean":"2024-01-24T17:48:26.923633+0000","last_became_active":"2024-01-24T17:48:26.915339+0000","last_became_peered":"2024-01-24T17:48:26.915339+0000","last_unstale":"2024-01-24T17:48:26.923633+0000","last_undegraded":"2024-01-24T17:48:26.923633+0000","last_fullsized":"2024-01-24T17:48:26.923633+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:23:15.094402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.914872+0000","last_change":"2024-01-24T17:48:26.911079+0000","last_active":"2024-01-24T17:48:26.914872+0000","last_peered":"2024-01-24T17:48:26.914872+0000","last_clean":"2024-01-24T17:48:26.914872+0000","last_became_active":"2024-01-24T17:48:26.910777+0000","last_became_peered":"2024-01-24T17:48:26.910777+0000","last_unstale":"2024-01-24T17:48:26.914872+0000","last_undegraded":"2024-01-24T17:48:26.914872+0000","last_fullsized":"2024-01-24T17:48:26.914872+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:36:51.824015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1f","version":"99'4","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.934074+0000","last_change":"2024-01-24T17:48:26.436409+0000","last_active":"2024-01-24T17:48:26.934074+0000","last_peered":"2024-01-24T17:48:26.934074+0000","last_clean":"2024-01-24T17:48:26.934074+0000","last_became_active":"2024-01-24T17:48:26.435501+0000","last_became_peered":"2024-01-24T17:48:26.435501+0000","last_unstale":"2024-01-24T17:48:26.934074+0000","last_undegraded":"2024-01-24T17:48:26.934074+0000","last_fullsized":"2024-01-24T17:48:26.934074+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:49.086780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.940187+0000","last_change":"2024-01-24T17:48:26.935171+0000","last_active":"2024-01-24T17:48:26.940187+0000","last_peered":"2024-01-24T17:48:26.940187+0000","last_clean":"2024-01-24T17:48:26.940187+0000","last_became_active":"2024-01-24T17:48:26.934878+0000","last_became_peered":"2024-01-24T17:48:26.934878+0000","last_unstale":"2024-01-24T17:48:26.940187+0000","last_undegraded":"2024-01-24T17:48:26.940187+0000","last_fullsized":"2024-01-24T17:48:26.940187+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:27:29.531410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1e","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.917903+0000","last_change":"2024-01-24T17:48:26.422468+0000","last_active":"2024-01-24T17:48:26.917903+0000","last_peered":"2024-01-24T17:48:26.917903+0000","last_clean":"2024-01-24T17:48:26.917903+0000","last_became_active":"2024-01-24T17:48:26.422078+0000","last_became_peered":"2024-01-24T17:48:26.422078+0000","last_unstale":"2024-01-24T17:48:26.917903+0000","last_undegraded":"2024-01-24T17:48:26.917903+0000","last_fullsized":"2024-01-24T17:48:26.917903+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T17:54:24.481188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930207+0000","last_change":"2024-01-24T17:48:26.917480+0000","last_active":"2024-01-24T17:48:26.930207+0000","last_peered":"2024-01-24T17:48:26.930207+0000","last_clean":"2024-01-24T17:48:26.930207+0000","last_became_active":"2024-01-24T17:48:26.914739+0000","last_became_peered":"2024-01-24T17:48:26.914739+0000","last_unstale":"2024-01-24T17:48:26.930207+0000","last_undegraded":"2024-01-24T17:48:26.930207+0000","last_fullsized":"2024-01-24T17:48:26.930207+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:55:24.010659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930292+0000","last_change":"2024-01-24T17:48:26.440810+0000","last_active":"2024-01-24T17:48:26.930292+0000","last_peered":"2024-01-24T17:48:26.930292+0000","last_clean":"2024-01-24T17:48:26.930292+0000","last_became_active":"2024-01-24T17:48:26.440235+0000","last_became_peered":"2024-01-24T17:48:26.440235+0000","last_unstale":"2024-01-24T17:48:26.930292+0000","last_undegraded":"2024-01-24T17:48:26.930292+0000","last_fullsized":"2024-01-24T17:48:26.930292+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:17:03.731286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.921106+0000","last_change":"2024-01-24T17:48:26.915047+0000","last_active":"2024-01-24T17:48:26.921106+0000","last_peered":"2024-01-24T17:48:26.921106+0000","last_clean":"2024-01-24T17:48:26.921106+0000","last_became_active":"2024-01-24T17:48:26.914645+0000","last_became_peered":"2024-01-24T17:48:26.914645+0000","last_unstale":"2024-01-24T17:48:26.921106+0000","last_undegraded":"2024-01-24T17:48:26.921106+0000","last_fullsized":"2024-01-24T17:48:26.921106+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:16:26.055063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933526+0000","last_change":"2024-01-24T17:48:26.434616+0000","last_active":"2024-01-24T17:48:26.933526+0000","last_peered":"2024-01-24T17:48:26.933526+0000","last_clean":"2024-01-24T17:48:26.933526+0000","last_became_active":"2024-01-24T17:48:26.434308+0000","last_became_peered":"2024-01-24T17:48:26.434308+0000","last_unstale":"2024-01-24T17:48:26.933526+0000","last_undegraded":"2024-01-24T17:48:26.933526+0000","last_fullsized":"2024-01-24T17:48:26.933526+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:16.864381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.920315+0000","last_change":"2024-01-24T17:48:26.915182+0000","last_active":"2024-01-24T17:48:26.920315+0000","last_peered":"2024-01-24T17:48:26.920315+0000","last_clean":"2024-01-24T17:48:26.920315+0000","last_became_active":"2024-01-24T17:48:26.914921+0000","last_became_peered":"2024-01-24T17:48:26.914921+0000","last_unstale":"2024-01-24T17:48:26.920315+0000","last_undegraded":"2024-01-24T17:48:26.920315+0000","last_fullsized":"2024-01-24T17:48:26.920315+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:35:22.864785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1b","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.917886+0000","last_change":"2024-01-24T17:48:26.425387+0000","last_active":"2024-01-24T17:48:26.917886+0000","last_peered":"2024-01-24T17:48:26.917886+0000","last_clean":"2024-01-24T17:48:26.917886+0000","last_became_active":"2024-01-24T17:48:26.425109+0000","last_became_peered":"2024-01-24T17:48:26.425109+0000","last_unstale":"2024-01-24T17:48:26.917886+0000","last_undegraded":"2024-01-24T17:48:26.917886+0000","last_fullsized":"2024-01-24T17:48:26.917886+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:07:49.530193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.918962+0000","last_change":"2024-01-24T17:48:26.913734+0000","last_active":"2024-01-24T17:48:26.918962+0000","last_peered":"2024-01-24T17:48:26.918962+0000","last_clean":"2024-01-24T17:48:26.918962+0000","last_became_active":"2024-01-24T17:48:26.913393+0000","last_became_peered":"2024-01-24T17:48:26.913393+0000","last_unstale":"2024-01-24T17:48:26.918962+0000","last_undegraded":"2024-01-24T17:48:26.918962+0000","last_fullsized":"2024-01-24T17:48:26.918962+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:25:35.588913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.932306+0000","last_change":"2024-01-24T17:48:26.430251+0000","last_active":"2024-01-24T17:48:26.932306+0000","last_peered":"2024-01-24T17:48:26.932306+0000","last_clean":"2024-01-24T17:48:26.932306+0000","last_became_active":"2024-01-24T17:48:26.427346+0000","last_became_peered":"2024-01-24T17:48:26.427346+0000","last_unstale":"2024-01-24T17:48:26.932306+0000","last_undegraded":"2024-01-24T17:48:26.932306+0000","last_fullsized":"2024-01-24T17:48:26.932306+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:38:16.109886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.940157+0000","last_change":"2024-01-24T17:48:26.933163+0000","last_active":"2024-01-24T17:48:26.940157+0000","last_peered":"2024-01-24T17:48:26.940157+0000","last_clean":"2024-01-24T17:48:26.940157+0000","last_became_active":"2024-01-24T17:48:26.931543+0000","last_became_peered":"2024-01-24T17:48:26.931543+0000","last_unstale":"2024-01-24T17:48:26.940157+0000","last_undegraded":"2024-01-24T17:48:26.940157+0000","last_fullsized":"2024-01-24T17:48:26.940157+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:55:24.010659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.900520+0000","last_change":"2024-01-24T17:48:26.429814+0000","last_active":"2024-01-24T17:48:26.900520+0000","last_peered":"2024-01-24T17:48:26.900520+0000","last_clean":"2024-01-24T17:48:26.900520+0000","last_became_active":"2024-01-24T17:48:26.429512+0000","last_became_peered":"2024-01-24T17:48:26.429512+0000","last_unstale":"2024-01-24T17:48:26.900520+0000","last_undegraded":"2024-01-24T17:48:26.900520+0000","last_fullsized":"2024-01-24T17:48:26.900520+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:48:48.343913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.911801+0000","last_change":"2024-01-24T17:48:26.905009+0000","last_active":"2024-01-24T17:48:26.911801+0000","last_peered":"2024-01-24T17:48:26.911801+0000","last_clean":"2024-01-24T17:48:26.911801+0000","last_became_active":"2024-01-24T17:48:26.904642+0000","last_became_peered":"2024-01-24T17:48:26.904642+0000","last_unstale":"2024-01-24T17:48:26.911801+0000","last_undegraded":"2024-01-24T17:48:26.911801+0000","last_fullsized":"2024-01-24T17:48:26.911801+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:27:05.102120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.907227+0000","last_change":"2024-01-24T17:48:26.429704+0000","last_active":"2024-01-24T17:48:26.907227+0000","last_peered":"2024-01-24T17:48:26.907227+0000","last_clean":"2024-01-24T17:48:26.907227+0000","last_became_active":"2024-01-24T17:48:26.428050+0000","last_became_peered":"2024-01-24T17:48:26.428050+0000","last_unstale":"2024-01-24T17:48:26.907227+0000","last_undegraded":"2024-01-24T17:48:26.907227+0000","last_fullsized":"2024-01-24T17:48:26.907227+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:55:22.994196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929695+0000","last_change":"2024-01-24T17:48:26.918741+0000","last_active":"2024-01-24T17:48:26.929695+0000","last_peered":"2024-01-24T17:48:26.929695+0000","last_clean":"2024-01-24T17:48:26.929695+0000","last_became_active":"2024-01-24T17:48:26.918464+0000","last_became_peered":"2024-01-24T17:48:26.918464+0000","last_unstale":"2024-01-24T17:48:26.929695+0000","last_undegraded":"2024-01-24T17:48:26.929695+0000","last_fullsized":"2024-01-24T17:48:26.929695+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:50.891831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.917673+0000","last_change":"2024-01-24T17:48:26.437385+0000","last_active":"2024-01-24T17:48:26.917673+0000","last_peered":"2024-01-24T17:48:26.917673+0000","last_clean":"2024-01-24T17:48:26.917673+0000","last_became_active":"2024-01-24T17:48:26.437107+0000","last_became_peered":"2024-01-24T17:48:26.437107+0000","last_unstale":"2024-01-24T17:48:26.917673+0000","last_undegraded":"2024-01-24T17:48:26.917673+0000","last_fullsized":"2024-01-24T17:48:26.917673+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:37:38.371998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.920253+0000","last_change":"2024-01-24T17:48:26.912406+0000","last_active":"2024-01-24T17:48:26.920253+0000","last_peered":"2024-01-24T17:48:26.920253+0000","last_clean":"2024-01-24T17:48:26.920253+0000","last_became_active":"2024-01-24T17:48:26.912125+0000","last_became_peered":"2024-01-24T17:48:26.912125+0000","last_unstale":"2024-01-24T17:48:26.920253+0000","last_undegraded":"2024-01-24T17:48:26.920253+0000","last_fullsized":"2024-01-24T17:48:26.920253+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:50.891831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.905732+0000","last_change":"2024-01-24T17:48:26.435724+0000","last_active":"2024-01-24T17:48:26.905732+0000","last_peered":"2024-01-24T17:48:26.905732+0000","last_clean":"2024-01-24T17:48:26.905732+0000","last_became_active":"2024-01-24T17:48:26.434914+0000","last_became_peered":"2024-01-24T17:48:26.434914+0000","last_unstale":"2024-01-24T17:48:26.905732+0000","last_undegraded":"2024-01-24T17:48:26.905732+0000","last_fullsized":"2024-01-24T17:48:26.905732+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:19:57.644580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.926144+0000","last_change":"2024-01-24T17:48:26.921371+0000","last_active":"2024-01-24T17:48:26.926144+0000","last_peered":"2024-01-24T17:48:26.926144+0000","last_clean":"2024-01-24T17:48:26.926144+0000","last_became_active":"2024-01-24T17:48:26.921083+0000","last_became_peered":"2024-01-24T17:48:26.921083+0000","last_unstale":"2024-01-24T17:48:26.926144+0000","last_undegraded":"2024-01-24T17:48:26.926144+0000","last_fullsized":"2024-01-24T17:48:26.926144+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:59:00.321722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.15","version":"99'3","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.921371+0000","last_change":"2024-01-24T17:48:26.432421+0000","last_active":"2024-01-24T17:48:26.921371+0000","last_peered":"2024-01-24T17:48:26.921371+0000","last_clean":"2024-01-24T17:48:26.921371+0000","last_became_active":"2024-01-24T17:48:26.432170+0000","last_became_peered":"2024-01-24T17:48:26.432170+0000","last_unstale":"2024-01-24T17:48:26.921371+0000","last_undegraded":"2024-01-24T17:48:26.921371+0000","last_fullsized":"2024-01-24T17:48:26.921371+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:45:38.433706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923147+0000","last_change":"2024-01-24T17:48:26.918604+0000","last_active":"2024-01-24T17:48:26.923147+0000","last_peered":"2024-01-24T17:48:26.923147+0000","last_clean":"2024-01-24T17:48:26.923147+0000","last_became_active":"2024-01-24T17:48:26.918356+0000","last_became_peered":"2024-01-24T17:48:26.918356+0000","last_unstale":"2024-01-24T17:48:26.923147+0000","last_undegraded":"2024-01-24T17:48:26.923147+0000","last_fullsized":"2024-01-24T17:48:26.923147+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:28:38.063901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.14","version":"99'6","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933949+0000","last_change":"2024-01-24T17:48:26.429468+0000","last_active":"2024-01-24T17:48:26.933949+0000","last_peered":"2024-01-24T17:48:26.933949+0000","last_clean":"2024-01-24T17:48:26.933949+0000","last_became_active":"2024-01-24T17:48:26.429095+0000","last_became_peered":"2024-01-24T17:48:26.429095+0000","last_unstale":"2024-01-24T17:48:26.933949+0000","last_undegraded":"2024-01-24T17:48:26.933949+0000","last_fullsized":"2024-01-24T17:48:26.933949+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:38:16.109886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929603+0000","last_change":"2024-01-24T17:48:26.920781+0000","last_active":"2024-01-24T17:48:26.929603+0000","last_peered":"2024-01-24T17:48:26.929603+0000","last_clean":"2024-01-24T17:48:26.929603+0000","last_became_active":"2024-01-24T17:48:26.920278+0000","last_became_peered":"2024-01-24T17:48:26.920278+0000","last_unstale":"2024-01-24T17:48:26.929603+0000","last_undegraded":"2024-01-24T17:48:26.929603+0000","last_fullsized":"2024-01-24T17:48:26.929603+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:41:28.702211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.924627+0000","last_change":"2024-01-24T17:48:26.424391+0000","last_active":"2024-01-24T17:48:26.924627+0000","last_peered":"2024-01-24T17:48:26.924627+0000","last_clean":"2024-01-24T17:48:26.924627+0000","last_became_active":"2024-01-24T17:48:26.421024+0000","last_became_peered":"2024-01-24T17:48:26.421024+0000","last_unstale":"2024-01-24T17:48:26.924627+0000","last_undegraded":"2024-01-24T17:48:26.924627+0000","last_fullsized":"2024-01-24T17:48:26.924627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:27:28.514947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.937752+0000","last_change":"2024-01-24T17:48:26.934723+0000","last_active":"2024-01-24T17:48:26.937752+0000","last_peered":"2024-01-24T17:48:26.937752+0000","last_clean":"2024-01-24T17:48:26.937752+0000","last_became_active":"2024-01-24T17:48:26.933851+0000","last_became_peered":"2024-01-24T17:48:26.933851+0000","last_unstale":"2024-01-24T17:48:26.937752+0000","last_undegraded":"2024-01-24T17:48:26.937752+0000","last_fullsized":"2024-01-24T17:48:26.937752+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:41:28.702211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.905072+0000","last_change":"2024-01-24T17:48:26.427403+0000","last_active":"2024-01-24T17:48:26.905072+0000","last_peered":"2024-01-24T17:48:26.905072+0000","last_clean":"2024-01-24T17:48:26.905072+0000","last_became_active":"2024-01-24T17:48:26.427120+0000","last_became_peered":"2024-01-24T17:48:26.427120+0000","last_unstale":"2024-01-24T17:48:26.905072+0000","last_undegraded":"2024-01-24T17:48:26.905072+0000","last_fullsized":"2024-01-24T17:48:26.905072+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:27:04.085657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.918754+0000","last_change":"2024-01-24T17:48:26.912679+0000","last_active":"2024-01-24T17:48:26.918754+0000","last_peered":"2024-01-24T17:48:26.918754+0000","last_clean":"2024-01-24T17:48:26.918754+0000","last_became_active":"2024-01-24T17:48:26.912276+0000","last_became_peered":"2024-01-24T17:48:26.912276+0000","last_unstale":"2024-01-24T17:48:26.918754+0000","last_undegraded":"2024-01-24T17:48:26.918754+0000","last_fullsized":"2024-01-24T17:48:26.918754+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:40:24.661312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.893145+0000","last_change":"2024-01-24T17:48:26.432311+0000","last_active":"2024-01-24T17:48:26.893145+0000","last_peered":"2024-01-24T17:48:26.893145+0000","last_clean":"2024-01-24T17:48:26.893145+0000","last_became_active":"2024-01-24T17:48:26.432000+0000","last_became_peered":"2024-01-24T17:48:26.432000+0000","last_unstale":"2024-01-24T17:48:26.893145+0000","last_undegraded":"2024-01-24T17:48:26.893145+0000","last_fullsized":"2024-01-24T17:48:26.893145+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:37:38.371998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930164+0000","last_change":"2024-01-24T17:48:26.921033+0000","last_active":"2024-01-24T17:48:26.930164+0000","last_peered":"2024-01-24T17:48:26.930164+0000","last_clean":"2024-01-24T17:48:26.930164+0000","last_became_active":"2024-01-24T17:48:26.920684+0000","last_became_peered":"2024-01-24T17:48:26.920684+0000","last_unstale":"2024-01-24T17:48:26.930164+0000","last_undegraded":"2024-01-24T17:48:26.930164+0000","last_fullsized":"2024-01-24T17:48:26.930164+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:18:48.467429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.10","version":"99'8","reported_seq":18,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.935700+0000","last_change":"2024-01-24T17:48:26.428667+0000","last_active":"2024-01-24T17:48:26.935700+0000","last_peered":"2024-01-24T17:48:26.935700+0000","last_clean":"2024-01-24T17:48:26.935700+0000","last_became_active":"2024-01-24T17:48:26.428319+0000","last_became_peered":"2024-01-24T17:48:26.428319+0000","last_unstale":"2024-01-24T17:48:26.935700+0000","last_undegraded":"2024-01-24T17:48:26.935700+0000","last_fullsized":"2024-01-24T17:48:26.935700+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:16:26.347716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.911872+0000","last_change":"2024-01-24T17:48:26.904743+0000","last_active":"2024-01-24T17:48:26.911872+0000","last_peered":"2024-01-24T17:48:26.911872+0000","last_clean":"2024-01-24T17:48:26.911872+0000","last_became_active":"2024-01-24T17:48:26.904459+0000","last_became_peered":"2024-01-24T17:48:26.904459+0000","last_unstale":"2024-01-24T17:48:26.911872+0000","last_undegraded":"2024-01-24T17:48:26.911872+0000","last_fullsized":"2024-01-24T17:48:26.911872+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:51:23.333640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.e","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.913873+0000","last_change":"2024-01-24T17:48:26.437498+0000","last_active":"2024-01-24T17:48:26.913873+0000","last_peered":"2024-01-24T17:48:26.913873+0000","last_clean":"2024-01-24T17:48:26.913873+0000","last_became_active":"2024-01-24T17:48:26.437175+0000","last_became_peered":"2024-01-24T17:48:26.437175+0000","last_unstale":"2024-01-24T17:48:26.913873+0000","last_undegraded":"2024-01-24T17:48:26.913873+0000","last_fullsized":"2024-01-24T17:48:26.913873+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:38.094327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.914708+0000","last_change":"2024-01-24T17:48:26.907099+0000","last_active":"2024-01-24T17:48:26.914708+0000","last_peered":"2024-01-24T17:48:26.914708+0000","last_clean":"2024-01-24T17:48:26.914708+0000","last_became_active":"2024-01-24T17:48:26.906615+0000","last_became_peered":"2024-01-24T17:48:26.906615+0000","last_unstale":"2024-01-24T17:48:26.914708+0000","last_undegraded":"2024-01-24T17:48:26.914708+0000","last_fullsized":"2024-01-24T17:48:26.914708+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:16:11.808000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.d","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.915494+0000","last_change":"2024-01-24T17:48:26.426985+0000","last_active":"2024-01-24T17:48:26.915494+0000","last_peered":"2024-01-24T17:48:26.915494+0000","last_clean":"2024-01-24T17:48:26.915494+0000","last_became_active":"2024-01-24T17:48:26.426675+0000","last_became_peered":"2024-01-24T17:48:26.426675+0000","last_unstale":"2024-01-24T17:48:26.915494+0000","last_undegraded":"2024-01-24T17:48:26.915494+0000","last_fullsized":"2024-01-24T17:48:26.915494+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:39:09.540390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.919717+0000","last_change":"2024-01-24T17:48:26.914171+0000","last_active":"2024-01-24T17:48:26.919717+0000","last_peered":"2024-01-24T17:48:26.919717+0000","last_clean":"2024-01-24T17:48:26.919717+0000","last_became_active":"2024-01-24T17:48:26.913928+0000","last_became_peered":"2024-01-24T17:48:26.913928+0000","last_unstale":"2024-01-24T17:48:26.919717+0000","last_undegraded":"2024-01-24T17:48:26.919717+0000","last_fullsized":"2024-01-24T17:48:26.919717+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:19:54.838909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.908264+0000","last_change":"2024-01-24T17:48:26.421304+0000","last_active":"2024-01-24T17:48:26.908264+0000","last_peered":"2024-01-24T17:48:26.908264+0000","last_clean":"2024-01-24T17:48:26.908264+0000","last_became_active":"2024-01-24T17:48:26.420499+0000","last_became_peered":"2024-01-24T17:48:26.420499+0000","last_unstale":"2024-01-24T17:48:26.908264+0000","last_undegraded":"2024-01-24T17:48:26.908264+0000","last_fullsized":"2024-01-24T17:48:26.908264+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:40:27.488675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933614+0000","last_change":"2024-01-24T17:48:26.929098+0000","last_active":"2024-01-24T17:48:26.933614+0000","last_peered":"2024-01-24T17:48:26.933614+0000","last_clean":"2024-01-24T17:48:26.933614+0000","last_became_active":"2024-01-24T17:48:26.928842+0000","last_became_peered":"2024-01-24T17:48:26.928842+0000","last_unstale":"2024-01-24T17:48:26.933614+0000","last_undegraded":"2024-01-24T17:48:26.933614+0000","last_fullsized":"2024-01-24T17:48:26.933614+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:21:35.521611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.925521+0000","last_change":"2024-01-24T17:48:26.434737+0000","last_active":"2024-01-24T17:48:26.925521+0000","last_peered":"2024-01-24T17:48:26.925521+0000","last_clean":"2024-01-24T17:48:26.925521+0000","last_became_active":"2024-01-24T17:48:26.434343+0000","last_became_peered":"2024-01-24T17:48:26.434343+0000","last_unstale":"2024-01-24T17:48:26.925521+0000","last_undegraded":"2024-01-24T17:48:26.925521+0000","last_fullsized":"2024-01-24T17:48:26.925521+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:35:21.848322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933399+0000","last_change":"2024-01-24T17:48:26.927592+0000","last_active":"2024-01-24T17:48:26.933399+0000","last_peered":"2024-01-24T17:48:26.933399+0000","last_clean":"2024-01-24T17:48:26.933399+0000","last_became_active":"2024-01-24T17:48:26.927316+0000","last_became_peered":"2024-01-24T17:48:26.927316+0000","last_unstale":"2024-01-24T17:48:26.933399+0000","last_undegraded":"2024-01-24T17:48:26.933399+0000","last_fullsized":"2024-01-24T17:48:26.933399+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:04:32.591869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.897316+0000","last_change":"2024-01-24T17:48:26.427383+0000","last_active":"2024-01-24T17:48:26.897316+0000","last_peered":"2024-01-24T17:48:26.897316+0000","last_clean":"2024-01-24T17:48:26.897316+0000","last_became_active":"2024-01-24T17:48:26.427089+0000","last_became_peered":"2024-01-24T17:48:26.427089+0000","last_unstale":"2024-01-24T17:48:26.897316+0000","last_undegraded":"2024-01-24T17:48:26.897316+0000","last_fullsized":"2024-01-24T17:48:26.897316+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:07:20.592462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.923607+0000","last_change":"2024-01-24T17:48:26.914951+0000","last_active":"2024-01-24T17:48:26.923607+0000","last_peered":"2024-01-24T17:48:26.923607+0000","last_clean":"2024-01-24T17:48:26.923607+0000","last_became_active":"2024-01-24T17:48:26.914677+0000","last_became_peered":"2024-01-24T17:48:26.914677+0000","last_unstale":"2024-01-24T17:48:26.923607+0000","last_undegraded":"2024-01-24T17:48:26.923607+0000","last_fullsized":"2024-01-24T17:48:26.923607+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:24:02.371020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.891159+0000","last_change":"2024-01-24T17:48:26.429921+0000","last_active":"2024-01-24T17:48:26.891159+0000","last_peered":"2024-01-24T17:48:26.891159+0000","last_clean":"2024-01-24T17:48:26.891159+0000","last_became_active":"2024-01-24T17:48:26.429551+0000","last_became_peered":"2024-01-24T17:48:26.429551+0000","last_unstale":"2024-01-24T17:48:26.891159+0000","last_undegraded":"2024-01-24T17:48:26.891159+0000","last_fullsized":"2024-01-24T17:48:26.891159+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:40:15.685823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.924109+0000","last_change":"2024-01-24T17:48:26.922640+0000","last_active":"2024-01-24T17:48:26.924109+0000","last_peered":"2024-01-24T17:48:26.924109+0000","last_clean":"2024-01-24T17:48:26.924109+0000","last_became_active":"2024-01-24T17:48:26.922418+0000","last_became_peered":"2024-01-24T17:48:26.922418+0000","last_unstale":"2024-01-24T17:48:26.924109+0000","last_undegraded":"2024-01-24T17:48:26.924109+0000","last_fullsized":"2024-01-24T17:48:26.924109+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:03:08.285397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.915815+0000","last_change":"2024-01-24T17:48:26.438815+0000","last_active":"2024-01-24T17:48:26.915815+0000","last_peered":"2024-01-24T17:48:26.915815+0000","last_clean":"2024-01-24T17:48:26.915815+0000","last_became_active":"2024-01-24T17:48:26.438290+0000","last_became_peered":"2024-01-24T17:48:26.438290+0000","last_unstale":"2024-01-24T17:48:26.915815+0000","last_undegraded":"2024-01-24T17:48:26.915815+0000","last_fullsized":"2024-01-24T17:48:26.915815+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:47:38.094327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.929158+0000","last_change":"2024-01-24T17:48:26.914775+0000","last_active":"2024-01-24T17:48:26.929158+0000","last_peered":"2024-01-24T17:48:26.929158+0000","last_clean":"2024-01-24T17:48:26.929158+0000","last_became_active":"2024-01-24T17:48:26.914434+0000","last_became_peered":"2024-01-24T17:48:26.914434+0000","last_unstale":"2024-01-24T17:48:26.929158+0000","last_undegraded":"2024-01-24T17:48:26.929158+0000","last_fullsized":"2024-01-24T17:48:26.929158+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:41:47.000130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"99'2","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.897206+0000","last_change":"2024-01-24T17:48:26.432467+0000","last_active":"2024-01-24T17:48:26.897206+0000","last_peered":"2024-01-24T17:48:26.897206+0000","last_clean":"2024-01-24T17:48:26.897206+0000","last_became_active":"2024-01-24T17:48:26.431971+0000","last_became_peered":"2024-01-24T17:48:26.431971+0000","last_unstale":"2024-01-24T17:48:26.897206+0000","last_undegraded":"2024-01-24T17:48:26.897206+0000","last_fullsized":"2024-01-24T17:48:26.897206+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:47.757639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.931825+0000","last_change":"2024-01-24T17:48:26.921114+0000","last_active":"2024-01-24T17:48:26.931825+0000","last_peered":"2024-01-24T17:48:26.931825+0000","last_clean":"2024-01-24T17:48:26.931825+0000","last_became_active":"2024-01-24T17:48:26.920853+0000","last_became_peered":"2024-01-24T17:48:26.920853+0000","last_unstale":"2024-01-24T17:48:26.931825+0000","last_undegraded":"2024-01-24T17:48:26.931825+0000","last_fullsized":"2024-01-24T17:48:26.931825+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T05:33:04.140368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.907870+0000","last_change":"2024-01-24T17:48:26.434756+0000","last_active":"2024-01-24T17:48:26.907870+0000","last_peered":"2024-01-24T17:48:26.907870+0000","last_clean":"2024-01-24T17:48:26.907870+0000","last_became_active":"2024-01-24T17:48:26.434466+0000","last_became_peered":"2024-01-24T17:48:26.434466+0000","last_unstale":"2024-01-24T17:48:26.907870+0000","last_undegraded":"2024-01-24T17:48:26.907870+0000","last_fullsized":"2024-01-24T17:48:26.907870+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:38:43.008776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933466+0000","last_change":"2024-01-24T17:48:26.930084+0000","last_active":"2024-01-24T17:48:26.933466+0000","last_peered":"2024-01-24T17:48:26.933466+0000","last_clean":"2024-01-24T17:48:26.933466+0000","last_became_active":"2024-01-24T17:48:26.925847+0000","last_became_peered":"2024-01-24T17:48:26.925847+0000","last_unstale":"2024-01-24T17:48:26.933466+0000","last_undegraded":"2024-01-24T17:48:26.933466+0000","last_fullsized":"2024-01-24T17:48:26.933466+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:19:50.103242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.908767+0000","last_change":"2024-01-24T17:48:26.429485+0000","last_active":"2024-01-24T17:48:26.908767+0000","last_peered":"2024-01-24T17:48:26.908767+0000","last_clean":"2024-01-24T17:48:26.908767+0000","last_became_active":"2024-01-24T17:48:26.427816+0000","last_became_peered":"2024-01-24T17:48:26.427816+0000","last_unstale":"2024-01-24T17:48:26.908767+0000","last_undegraded":"2024-01-24T17:48:26.908767+0000","last_fullsized":"2024-01-24T17:48:26.908767+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T19:39:39.323443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.918783+0000","last_change":"2024-01-24T17:48:26.912842+0000","last_active":"2024-01-24T17:48:26.918783+0000","last_peered":"2024-01-24T17:48:26.918783+0000","last_clean":"2024-01-24T17:48:26.918783+0000","last_became_active":"2024-01-24T17:48:26.912574+0000","last_became_peered":"2024-01-24T17:48:26.912574+0000","last_unstale":"2024-01-24T17:48:26.918783+0000","last_undegraded":"2024-01-24T17:48:26.918783+0000","last_fullsized":"2024-01-24T17:48:26.918783+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:47:04.048590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.898990+0000","last_change":"2024-01-24T17:48:26.427943+0000","last_active":"2024-01-24T17:48:26.898990+0000","last_peered":"2024-01-24T17:48:26.898990+0000","last_clean":"2024-01-24T17:48:26.898990+0000","last_became_active":"2024-01-24T17:48:26.427470+0000","last_became_peered":"2024-01-24T17:48:26.427470+0000","last_unstale":"2024-01-24T17:48:26.898990+0000","last_undegraded":"2024-01-24T17:48:26.898990+0000","last_fullsized":"2024-01-24T17:48:26.898990+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:47:43.100680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.932497+0000","last_change":"2024-01-24T17:48:26.918522+0000","last_active":"2024-01-24T17:48:26.932497+0000","last_peered":"2024-01-24T17:48:26.932497+0000","last_clean":"2024-01-24T17:48:26.932497+0000","last_became_active":"2024-01-24T17:48:26.918180+0000","last_became_peered":"2024-01-24T17:48:26.918180+0000","last_unstale":"2024-01-24T17:48:26.932497+0000","last_undegraded":"2024-01-24T17:48:26.932497+0000","last_fullsized":"2024-01-24T17:48:26.932497+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:38:48.469680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.916549+0000","last_change":"2024-01-24T17:48:26.440870+0000","last_active":"2024-01-24T17:48:26.916549+0000","last_peered":"2024-01-24T17:48:26.916549+0000","last_clean":"2024-01-24T17:48:26.916549+0000","last_became_active":"2024-01-24T17:48:26.440549+0000","last_became_peered":"2024-01-24T17:48:26.440549+0000","last_unstale":"2024-01-24T17:48:26.916549+0000","last_undegraded":"2024-01-24T17:48:26.916549+0000","last_fullsized":"2024-01-24T17:48:26.916549+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:49:13.536137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.934294+0000","last_change":"2024-01-24T17:48:26.927450+0000","last_active":"2024-01-24T17:48:26.934294+0000","last_peered":"2024-01-24T17:48:26.934294+0000","last_clean":"2024-01-24T17:48:26.934294+0000","last_became_active":"2024-01-24T17:48:26.927182+0000","last_became_peered":"2024-01-24T17:48:26.927182+0000","last_unstale":"2024-01-24T17:48:26.934294+0000","last_undegraded":"2024-01-24T17:48:26.934294+0000","last_fullsized":"2024-01-24T17:48:26.934294+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T20:31:12.583303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.898671+0000","last_change":"2024-01-24T17:48:26.423543+0000","last_active":"2024-01-24T17:48:26.898671+0000","last_peered":"2024-01-24T17:48:26.898671+0000","last_clean":"2024-01-24T17:48:26.898671+0000","last_became_active":"2024-01-24T17:48:26.423221+0000","last_became_peered":"2024-01-24T17:48:26.423221+0000","last_unstale":"2024-01-24T17:48:26.898671+0000","last_undegraded":"2024-01-24T17:48:26.898671+0000","last_fullsized":"2024-01-24T17:48:26.898671+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:07:56.196666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.930799+0000","last_change":"2024-01-24T17:48:26.917832+0000","last_active":"2024-01-24T17:48:26.930799+0000","last_peered":"2024-01-24T17:48:26.930799+0000","last_clean":"2024-01-24T17:48:26.930799+0000","last_became_active":"2024-01-24T17:48:26.917584+0000","last_became_peered":"2024-01-24T17:48:26.917584+0000","last_unstale":"2024-01-24T17:48:26.930799+0000","last_undegraded":"2024-01-24T17:48:26.930799+0000","last_fullsized":"2024-01-24T17:48:26.930799+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T00:19:29.557377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.912432+0000","last_change":"2024-01-24T17:48:26.438811+0000","last_active":"2024-01-24T17:48:26.912432+0000","last_peered":"2024-01-24T17:48:26.912432+0000","last_clean":"2024-01-24T17:48:26.912432+0000","last_became_active":"2024-01-24T17:48:26.438127+0000","last_became_peered":"2024-01-24T17:48:26.438127+0000","last_unstale":"2024-01-24T17:48:26.912432+0000","last_undegraded":"2024-01-24T17:48:26.912432+0000","last_fullsized":"2024-01-24T17:48:26.912432+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T21:19:53.822446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.924911+0000","last_change":"2024-01-24T17:48:26.918526+0000","last_active":"2024-01-24T17:48:26.924911+0000","last_peered":"2024-01-24T17:48:26.924911+0000","last_clean":"2024-01-24T17:48:26.924911+0000","last_became_active":"2024-01-24T17:48:26.918296+0000","last_became_peered":"2024-01-24T17:48:26.918296+0000","last_unstale":"2024-01-24T17:48:26.924911+0000","last_undegraded":"2024-01-24T17:48:26.924911+0000","last_fullsized":"2024-01-24T17:48:26.924911+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T04:40:16.702286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.915916+0000","last_change":"2024-01-24T17:48:26.438598+0000","last_active":"2024-01-24T17:48:26.915916+0000","last_peered":"2024-01-24T17:48:26.915916+0000","last_clean":"2024-01-24T17:48:26.915916+0000","last_became_active":"2024-01-24T17:48:26.437938+0000","last_became_peered":"2024-01-24T17:48:26.437938+0000","last_unstale":"2024-01-24T17:48:26.915916+0000","last_undegraded":"2024-01-24T17:48:26.915916+0000","last_fullsized":"2024-01-24T17:48:26.915916+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T03:57:47.757639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.933545+0000","last_change":"2024-01-24T17:48:26.930709+0000","last_active":"2024-01-24T17:48:26.933545+0000","last_peered":"2024-01-24T17:48:26.933545+0000","last_clean":"2024-01-24T17:48:26.933545+0000","last_became_active":"2024-01-24T17:48:26.930432+0000","last_became_peered":"2024-01-24T17:48:26.930432+0000","last_unstale":"2024-01-24T17:48:26.933545+0000","last_undegraded":"2024-01-24T17:48:26.933545+0000","last_fullsized":"2024-01-24T17:48:26.933545+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:26.387388+0000","last_clean_scrub_stamp":"2024-01-24T17:48:26.387388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-26T02:47:44.117143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.f","version":"99'1","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.913974+0000","last_change":"2024-01-24T17:48:26.436022+0000","last_active":"2024-01-24T17:48:26.913974+0000","last_peered":"2024-01-24T17:48:26.913974+0000","last_clean":"2024-01-24T17:48:26.913974+0000","last_became_active":"2024-01-24T17:48:26.435638+0000","last_became_peered":"2024-01-24T17:48:26.435638+0000","last_unstale":"2024-01-24T17:48:26.913974+0000","last_undegraded":"2024-01-24T17:48:26.913974+0000","last_fullsized":"2024-01-24T17:48:26.913974+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:48:25.370925+0000","last_clean_scrub_stamp":"2024-01-24T17:48:25.370925+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T18:55:40.269339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":235,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-24T17:48:26.424336+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:48:26.424336+0000","last_peered":"2024-01-24T17:48:26.424336+0000","last_clean":"2024-01-24T17:48:26.424336+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:48:26.424336+0000","last_undegraded":"2024-01-24T17:48:26.424336+0000","last_fullsized":"2024-01-24T17:48:26.424336+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574936,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32320,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339520,"statfs":{"total":96636764160,"available":96603668480,"internally_reserved":0,"allocated":1228800,"data_stored":526817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.081,"15min":1.0349999999999999},"min":{"1min":0.70499999999999996,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.1399999999999999,"15min":1.0640000000000001},"min":{"1min":0.70099999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.91300000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.093,"15min":1.1020000000000001},"min":{"1min":0.76300000000000001,"5min":0.621,"15min":0.621},"max":{"1min":1.6479999999999999,"5min":2.04,"15min":2.04},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.202,"15min":1.198},"min":{"1min":0.81399999999999995,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.944,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.218,"15min":1.1739999999999999},"min":{"1min":0.75800000000000001,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.992,"5min":1.992,"15min":2.1400000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.306,"5min":1.1819999999999999,"15min":1.2190000000000001},"min":{"1min":0.76000000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7769999999999999,"5min":2.484,"15min":2.484},"last":1.343}]},{"osd":3,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.2390000000000001,"15min":1.25},"min":{"1min":0.75900000000000001,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.7450000000000001,"5min":2.0179999999999998,"15min":2.2930000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3140000000000001,"15min":1.3200000000000001},"min":{"1min":0.872,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.833,"5min":2.6269999999999998,"15min":2.6269999999999998},"last":1.0820000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.28,"15min":1.3149999999999999},"min":{"1min":0.85799999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.6179999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.423,"15min":1.4339999999999999},"min":{"1min":1.212,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.9330000000000001,"5min":2.5449999999999999,"15min":2.6459999999999999},"last":1.202}]},{"osd":5,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3029999999999999,"15min":1.347},"min":{"1min":1.131,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7070000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":1.163},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.361,"15min":1.423},"min":{"1min":0.97999999999999998,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.2480000000000002,"5min":2.2480000000000002,"15min":2.3719999999999999},"last":1.032}]},{"osd":6,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.468,"15min":1.5249999999999999},"min":{"1min":1.1990000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.4129999999999998,"5min":2.5720000000000001,"15min":2.6960000000000002},"last":1.125},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.4370000000000001,"15min":1.476},"min":{"1min":1.0920000000000001,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.923,"5min":2.2450000000000001,"15min":2.4830000000000001},"last":1.425}]}]},{"osd":6,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1212416,"data_stored":519460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.081,"15min":1.04},"min":{"1min":0.748,"5min":0.623,"15min":0.61199999999999999},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.0920000000000001,"15min":1.0349999999999999},"min":{"1min":0.79100000000000004,"5min":0.69499999999999995,"15min":0.66300000000000003},"max":{"1min":1.4390000000000001,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.113,"15min":1.022},"min":{"1min":0.78400000000000003,"5min":0.66700000000000004,"15min":0.64200000000000002},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.234},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.173,"15min":1.131},"min":{"1min":0.69999999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.9129999999999998,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.1759999999999999,"15min":1.1479999999999999},"min":{"1min":0.71899999999999997,"5min":0.70399999999999996,"15min":0.63600000000000001},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.1919999999999999,"15min":1.1739999999999999},"min":{"1min":0.755,"5min":0.69099999999999995,"15min":0.56599999999999995},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.0910000000000002},"last":1.5069999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.236,"15min":1.252},"min":{"1min":0.81100000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.75,"5min":2.0009999999999999,"15min":2.1669999999999998},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2090000000000001,"15min":1.2450000000000001},"min":{"1min":0.95299999999999996,"5min":0.81899999999999995,"15min":0.69799999999999995},"max":{"1min":1.625,"5min":1.758,"15min":2.46},"last":1.347}]},{"osd":4,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.3089999999999999,"15min":1.2549999999999999},"min":{"1min":0.52800000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.877,"5min":1.9179999999999999,"15min":1.988},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.753,"5min":0.73899999999999999,"15min":0.44600000000000001},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.1619999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.339,"15min":1.3069999999999999},"min":{"1min":0.77600000000000002,"5min":0.65300000000000002,"15min":0.46800000000000003},"max":{"1min":1.694,"5min":1.8879999999999999,"15min":2.2170000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.4419999999999999,"15min":1.4399999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.9750000000000001,"5min":2.0030000000000001,"15min":2.4849999999999999},"last":1.4790000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:47:30 2024","interfaces":[{"interface":"back","average":{"1min":1.345,"5min":1.3480000000000001,"15min":1.365},"min":{"1min":0.91200000000000003,"5min":0.85399999999999998,"15min":0.51400000000000001},"max":{"1min":1.911,"5min":1.978,"15min":2.2050000000000001},"last":1.298},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4470000000000001,"15min":1.4610000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.52300000000000002},"max":{"1min":1.7649999999999999,"5min":1.9019999999999999,"15min":2.6920000000000002},"last":1.7170000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574937,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1806336,"data_stored":1109828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0309999999999999,"15min":1.087},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.5389999999999999,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":3.609},{"interface":"front","average":{"1min":1.012,"5min":1.0329999999999999,"15min":1.032},"min":{"1min":0.73099999999999998,"5min":0.67300000000000004,"15min":0.65300000000000002},"max":{"1min":1.28,"5min":1.889,"15min":1.889},"last":3.7240000000000002}]},{"osd":1,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.0920000000000001,"15min":1.079},"min":{"1min":0.71199999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.7030000000000001,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":3.5470000000000002},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1499999999999999,"15min":1.24},"min":{"1min":0.76200000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.657,"5min":1.657,"15min":2.1579999999999999},"last":1.4219999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.212,"15min":1.226},"min":{"1min":0.85299999999999998,"5min":0.79400000000000004,"15min":0.59199999999999997},"max":{"1min":1.6339999999999999,"5min":1.655,"15min":1.8400000000000001},"last":2.8860000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.242,"15min":1.3},"min":{"1min":0.79700000000000004,"5min":0.67600000000000005,"15min":0.66300000000000003},"max":{"1min":1.5669999999999999,"5min":1.9870000000000001,"15min":3.1349999999999998},"last":8.1509999999999998}]},{"osd":3,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.208,"15min":1.3129999999999999},"min":{"1min":0.82399999999999995,"5min":0.76400000000000001,"15min":0.70799999999999996},"max":{"1min":1.611,"5min":2.0150000000000001,"15min":3.036},"last":2.7559999999999998},{"interface":"front","average":{"1min":1.26,"5min":1.262,"15min":1.3169999999999999},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.5980000000000001,"5min":1.7290000000000001,"15min":1.7769999999999999},"last":4.1230000000000002}]},{"osd":4,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.401,"15min":1.4419999999999999},"min":{"1min":0.58299999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.7130000000000001,"5min":2.069,"15min":2.069},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.274,"15min":1.3420000000000001},"min":{"1min":0.87,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.837,"5min":3.2810000000000001,"15min":3.2810000000000001},"last":2.1139999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3080000000000001,"15min":1.3460000000000001},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.028,"5min":2.504,"15min":2.504},"last":1.8640000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.4530000000000001,"15min":1.506},"min":{"1min":0.71399999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.806,"5min":2.1579999999999999,"15min":2.2290000000000001},"last":1.52}]},{"osd":7,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.3680000000000001,"15min":1.4630000000000001},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.8420000000000001,"5min":3.161,"15min":3.161},"last":2.2010000000000001},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.508,"15min":1.607},"min":{"1min":0.78400000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.113,"5min":2.1389999999999998,"15min":3.2410000000000001},"last":1.978}]}]},{"osd":4,"up_from":13,"seq":55834574938,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1212416,"data_stored":519460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.1000000000000001,"15min":1.056},"min":{"1min":0.72099999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.6120000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.095,"5min":1.087,"15min":1.0940000000000001},"min":{"1min":0.75600000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.464,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.1659999999999999,"15min":1.0800000000000001},"min":{"1min":0.77500000000000002,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.113,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.159,"5min":1.1319999999999999,"15min":1.198},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.5029999999999999,"5min":1.849,"15min":1.849},"last":1.1419999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.2110000000000001,"15min":1.214},"min":{"1min":0.89100000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.9850000000000001,"5min":2.105,"15min":2.105},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.2669999999999999,"15min":1.3220000000000001},"min":{"1min":0.95399999999999996,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.6619999999999999,"5min":1.986,"15min":1.986},"last":0.96399999999999997}]},{"osd":3,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.1719999999999999,"15min":1.1870000000000001},"min":{"1min":0.69599999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4330000000000001,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.268,"15min":1.3260000000000001},"min":{"1min":0.80100000000000005,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.7929999999999999,"5min":1.8029999999999999,"15min":2.9409999999999998},"last":1.6679999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.2729999999999999,"15min":1.3480000000000001},"min":{"1min":0.76900000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.625,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.25},{"interface":"front","average":{"1min":1.252,"5min":1.256,"15min":1.292},"min":{"1min":0.93200000000000005,"5min":0.65600000000000003,"15min":0.55200000000000005},"max":{"1min":1.738,"5min":1.738,"15min":1.9039999999999999},"last":1.2969999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.88900000000000001,"5min":0.67800000000000005,"15min":0.64400000000000002},"max":{"1min":2.1930000000000001,"5min":2.3159999999999998,"15min":3.0840000000000001},"last":1.722},{"interface":"front","average":{"1min":1.464,"5min":1.4139999999999999,"15min":1.4630000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.66500000000000004},"max":{"1min":1.931,"5min":2.2610000000000001,"15min":2.2610000000000001},"last":1.532}]},{"osd":7,"last update":"Wed Jan 24 17:47:40 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.389,"15min":1.411},"min":{"1min":0.78600000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.627,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.411},{"interface":"front","average":{"1min":1.532,"5min":1.4790000000000001,"15min":1.581},"min":{"1min":1.008,"5min":0.79300000000000004,"15min":0.60399999999999998},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":4.3250000000000002},"last":1.7729999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574938,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1236992,"data_stored":526429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90100000000000002,"15min":0.86699999999999999},"min":{"1min":0.53800000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.2909999999999999,"5min":5.1929999999999996,"15min":5.1929999999999996},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.97399999999999998,"15min":0.89800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.3360000000000001,"5min":8.6989999999999998,"15min":8.6989999999999998},"last":0.80800000000000005}]},{"osd":1,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.87,"15min":0.92800000000000005},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.2789999999999999,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.054,"15min":1.0600000000000001},"min":{"1min":0.57399999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.45,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.69899999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0860000000000001,"15min":1.1040000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.502,"5min":3.4630000000000001,"15min":3.4630000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1659999999999999,"15min":1.159},"min":{"1min":0.60299999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.7250000000000001,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.76100000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.282,"15min":1.28},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.0049999999999999,"5min":6.7469999999999999,"15min":6.7469999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.353,"5min":1.3080000000000001,"15min":1.3400000000000001},"min":{"1min":0.85699999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.6399999999999999,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.177}]},{"osd":5,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.367,"15min":1.415},"min":{"1min":1.006,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.6659999999999999,"5min":4.3789999999999996,"15min":4.3789999999999996},"last":1.095},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.3859999999999999,"15min":1.4139999999999999},"min":{"1min":0.80800000000000005,"5min":0.746,"15min":0.746},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.98899999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.5029999999999999,"15min":1.4930000000000001},"min":{"1min":0.90600000000000003,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.3210000000000002,"5min":9.5440000000000005,"15min":9.5440000000000005},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.397,"5min":1.415,"15min":1.444},"min":{"1min":0.76500000000000001,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.093,"5min":4.8170000000000002,"15min":4.8170000000000002},"last":1.0009999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:47:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.4750000000000001,"15min":1.4950000000000001},"min":{"1min":0.97399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.1829999999999998,"5min":6.6970000000000001,"15min":6.6970000000000001},"last":1.206},{"interface":"front","average":{"1min":1.45,"5min":1.502,"15min":1.512},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.77,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":1.0489999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1212416,"data_stored":519460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78900000000000003,"15min":0.79300000000000004},"min":{"1min":0.443,"5min":0.318,"15min":0.318},"max":{"1min":1.3420000000000001,"5min":1.667,"15min":1.667},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.66200000000000003,"5min":0.375,"15min":0.375},"max":{"1min":1.4370000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"last":0.85599999999999998}]},{"osd":1,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0109999999999999,"15min":1.0469999999999999},"min":{"1min":0.505,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.569,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.296},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.111,"15min":1.1950000000000001},"min":{"1min":0.76300000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.677,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.6499999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.94299999999999995,"15min":0.98299999999999998},"min":{"1min":0.51900000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.5089999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":1.26},{"interface":"front","average":{"1min":1.129,"5min":1.135,"15min":1.1950000000000001},"min":{"1min":0.68000000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5680000000000001,"5min":2.996,"15min":2.996},"last":1.5249999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.248,"15min":1.3100000000000001},"min":{"1min":0.78500000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.5,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.397},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.2589999999999999,"15min":1.3779999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.831,"5min":1.889,"15min":2.3839999999999999},"last":1.349}]},{"osd":5,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3009999999999999,"15min":1.389},"min":{"1min":0.873,"5min":0.73999999999999999,"15min":0.73899999999999999},"max":{"1min":1.851,"5min":1.851,"15min":2.3140000000000001},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.202,"15min":1.3},"min":{"1min":0.754,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.4750000000000001,"5min":1.889,"15min":2.0720000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.304,"15min":1.3799999999999999},"min":{"1min":0.63200000000000001,"5min":0.623,"15min":0.623},"max":{"1min":1.615,"5min":1.9350000000000001,"15min":2.2679999999999998},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.3380000000000001,"15min":1.4490000000000001},"min":{"1min":0.95499999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.782,"5min":2.2530000000000001,"15min":3.7410000000000001},"last":1.589}]},{"osd":7,"last update":"Wed Jan 24 17:47:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.401,"15min":1.5449999999999999},"min":{"1min":0.81999999999999995,"5min":0.80800000000000005,"15min":0.80300000000000005},"max":{"1min":1.9670000000000001,"5min":2.1499999999999999,"15min":3.343},"last":1.7490000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.367,"15min":1.48},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.7190000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.716}]}]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1806336,"data_stored":1109828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88200000000000001,"15min":0.86299999999999999},"min":{"1min":0.43099999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.2769999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.92100000000000004,"15min":0.94999999999999996},"min":{"1min":0.62,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.395,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.80000000000000004}]},{"osd":2,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.875,"15min":0.95799999999999996},"min":{"1min":0.54700000000000004,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3140000000000001,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.0509999999999999,"15min":1.1000000000000001},"min":{"1min":0.67100000000000004,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4239999999999999,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.262}]},{"osd":3,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.141,"15min":1.163},"min":{"1min":0.54500000000000004,"5min":0.53000000000000003,"15min":0.47699999999999998},"max":{"1min":1.8440000000000001,"5min":1.863,"15min":1.9610000000000001},"last":1.5},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.073,"15min":1.204},"min":{"1min":0.68200000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":3.149},"last":1.46}]},{"osd":4,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.333,"15min":1.359},"min":{"1min":0.96799999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.772,"5min":1.962,"15min":2.2410000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.244,"5min":1.2609999999999999,"15min":1.3120000000000001},"min":{"1min":0.871,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.734,"5min":1.734,"15min":2.3849999999999998},"last":1.5369999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.2729999999999999,"15min":1.341},"min":{"1min":0.76700000000000002,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.964,"5min":1.964,"15min":2.577},"last":1.629},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.363,"15min":1.377},"min":{"1min":0.92400000000000004,"5min":0.83199999999999996,"15min":0.81299999999999994},"max":{"1min":2.2770000000000001,"5min":2.2770000000000001,"15min":2.9809999999999999},"last":2.169}]},{"osd":6,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.3400000000000001,"15min":1.4239999999999999},"min":{"1min":0.94099999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7929999999999999,"5min":2.008,"15min":3.2850000000000001},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.345,"15min":1.4930000000000001},"min":{"1min":0.96199999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.887,"5min":2.077,"15min":3.5859999999999999},"last":1.7190000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.415,"15min":1.4610000000000001},"min":{"1min":1.0249999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.5880000000000001,"5min":2.5880000000000001,"15min":2.8439999999999999},"last":2.0819999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.4339999999999999,"15min":1.532},"min":{"1min":0.89700000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":3.4350000000000001},"last":1.6779999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1212416,"data_stored":519460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.79800000000000004,"15min":0.81699999999999995},"min":{"1min":0.60899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.7290000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.92600000000000005,"15min":0.96799999999999997},"min":{"1min":0.433,"5min":0.376,"15min":0.376},"max":{"1min":1.5309999999999999,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.621}]},{"osd":2,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.86799999999999999,"15min":0.84199999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.34699999999999998},"max":{"1min":1.3260000000000001,"5min":1.331,"15min":1.331},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.117,"5min":0.98899999999999999,"15min":1.081},"min":{"1min":0.50600000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.5049999999999999,"5min":1.55,"15min":2.0499999999999998},"last":0.91500000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1040000000000001,"15min":1.1779999999999999},"min":{"1min":0.59399999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.9670000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.173,"15min":1.1910000000000001},"min":{"1min":0.49299999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.806,"5min":1.806,"15min":2.407},"last":0.84499999999999997}]},{"osd":4,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.2529999999999999,"15min":1.385},"min":{"1min":1.0129999999999999,"5min":0.73299999999999998,"15min":0.70799999999999996},"max":{"1min":1.6279999999999999,"5min":2.056,"15min":2.2589999999999999},"last":1.431},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.2889999999999999,"15min":1.446},"min":{"1min":1.01,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":3.3740000000000001},"last":1.325}]},{"osd":5,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.2689999999999999,"15min":1.375},"min":{"1min":0.90700000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.746,"5min":1.7490000000000001,"15min":2.9300000000000002},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.246,"15min":1.2609999999999999},"min":{"1min":0.88600000000000001,"5min":0.77000000000000002,"15min":0.65600000000000003},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":2.3799999999999999},"last":1.55}]},{"osd":6,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.327,"15min":1.357},"min":{"1min":0.83399999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.954,"5min":1.954,"15min":2.335},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.478,"5min":1.3799999999999999,"15min":1.4490000000000001},"min":{"1min":0.92100000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.871,"5min":1.931,"15min":2.2389999999999999},"last":1.246}]},{"osd":7,"last update":"Wed Jan 24 17:47:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.381,"15min":1.5069999999999999},"min":{"1min":0.97699999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":3.0230000000000001},"last":1.347},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.4450000000000001,"15min":1.5740000000000001},"min":{"1min":0.99299999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":3.1600000000000001},"last":1.2989999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:48:32.351 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-24T17:48:32.351 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-24T17:48:32.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.621+0000 7fae1e800640 1 Processor -- start 2024-01-24T17:48:32.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.621+0000 7fae1e800640 1 -- start start 2024-01-24T17:48:32.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.623+0000 7fae1e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fae181516c0 0x7fae180b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.623+0000 7fae1e800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae180b12b0 con 0x7fae18070430 2024-01-24T17:48:32.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.623+0000 7fae1e800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1815b170 con 0x7fae1815ac70 2024-01-24T17:48:32.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.623+0000 7fae1e800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1815b450 con 0x7fae181516c0 2024-01-24T17:48:32.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.623+0000 7fae1d400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fae181516c0 0x7fae180b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.623+0000 7fae1de00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fae18070430 0x7fae180ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:39446/0 (socket says 172.21.15.39:39446) 2024-01-24T17:48:32.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.623+0000 7fae1de00640 1 -- 172.21.15.39:0/3115230078 learned_addr learned my addr 172.21.15.39:0/3115230078 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.624+0000 7fae1d400640 1 -- 172.21.15.39:0/3115230078 >> v1:172.21.15.100:6789/0 conn(0x7fae1815ac70 legacy=0x7fae1806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.624+0000 7fae1d400640 1 -- 172.21.15.39:0/3115230078 >> v1:172.21.15.39:6789/0 conn(0x7fae18070430 legacy=0x7fae180ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.624+0000 7fae1d400640 1 -- 172.21.15.39:0/3115230078 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1815b730 con 0x7fae181516c0 2024-01-24T17:48:32.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.624+0000 7fae17e00640 1 -- 172.21.15.39:0/3115230078 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062379337 0 0) 0x7fae180b12b0 con 0x7fae18070430 2024-01-24T17:48:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.625+0000 7fae1d400640 1 --2- 172.21.15.39:0/3115230078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae180b3d80 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fae080097a0 tx=0x7fae0802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c42d2bd5d2841eb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.625+0000 7fae17e00640 1 -- 172.21.15.39:0/3115230078 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08030610 con 0x7fae181516c0 2024-01-24T17:48:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.625+0000 7fae17e00640 1 -- 172.21.15.39:0/3115230078 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae08030770 con 0x7fae181516c0 2024-01-24T17:48:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.625+0000 7fae17e00640 1 -- 172.21.15.39:0/3115230078 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08030b00 con 0x7fae181516c0 2024-01-24T17:48:32.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.626+0000 7fae1e800640 1 -- 172.21.15.39:0/3115230078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 msgr2=0x7fae180b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.626+0000 7fae1e800640 1 --2- 172.21.15.39:0/3115230078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae180b3d80 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fae080097a0 tx=0x7fae0802b960 comp rx=0 tx=0).stop 2024-01-24T17:48:32.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.626+0000 7fae1e800640 1 -- 172.21.15.39:0/3115230078 shutdown_connections 2024-01-24T17:48:32.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.626+0000 7fae1e800640 1 --2- 172.21.15.39:0/3115230078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae180b3d80 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.626+0000 7fae1e800640 1 -- 172.21.15.39:0/3115230078 >> 172.21.15.39:0/3115230078 conn(0x7fae1806a200 msgr2=0x7fae180affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:32.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.626+0000 7fae1e800640 1 -- 172.21.15.39:0/3115230078 shutdown_connections 2024-01-24T17:48:32.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.626+0000 7fae1e800640 1 -- 172.21.15.39:0/3115230078 wait complete. 2024-01-24T17:48:32.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.628+0000 7fae1e800640 1 Processor -- start 2024-01-24T17:48:32.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.629+0000 7fae1e800640 1 -- start start 2024-01-24T17:48:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1e800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae18070430 0x7fae180c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae181efdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1e800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1815ac70 0x7fae1814de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c35d0 con 0x7fae181516c0 2024-01-24T17:48:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1e800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c2530 con 0x7fae18070430 2024-01-24T17:48:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1e800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c2810 con 0x7fae1815ac70 2024-01-24T17:48:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae181efdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae181efdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49796/0 (socket says 172.21.15.39:49796) 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.630+0000 7fae1ca00640 1 -- 172.21.15.39:0/1522916671 learned_addr learned my addr 172.21.15.39:0/1522916671 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1d400640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae18070430 0x7fae180c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1de00640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1815ac70 0x7fae1814de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1ca00640 1 -- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1815ac70 msgr2=0x7fae1814de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1ca00640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1815ac70 0x7fae1814de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1ca00640 1 -- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae18070430 msgr2=0x7fae180c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1ca00640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae18070430 0x7fae180c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1ca00640 1 -- 172.21.15.39:0/1522916671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae08009400 con 0x7fae181516c0 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1d400640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae18070430 0x7fae180c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1ca00640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae181efdd0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fae0000c450 tx=0x7fae0000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae15600640 1 -- 172.21.15.39:0/1522916671 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae000052d0 con 0x7fae181516c0 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae15600640 1 -- 172.21.15.39:0/1522916671 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae00005430 con 0x7fae181516c0 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae15600640 1 -- 172.21.15.39:0/1522916671 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00012870 con 0x7fae181516c0 2024-01-24T17:48:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.631+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae180c2af0 con 0x7fae181516c0 2024-01-24T17:48:32.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.632+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae180c30b0 con 0x7fae181516c0 2024-01-24T17:48:32.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.633+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1804e840 con 0x7fae181516c0 2024-01-24T17:48:32.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.634+0000 7fae15600640 1 -- 172.21.15.39:0/1522916671 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fae000129d0 con 0x7fae181516c0 2024-01-24T17:48:32.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.635+0000 7fae15600640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadf006a430 0x7fadf006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:32.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.635+0000 7fae15600640 1 -- 172.21.15.39:0/1522916671 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae000883e0 con 0x7fae181516c0 2024-01-24T17:48:32.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.636+0000 7fae1d400640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadf006a430 0x7fadf006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:32.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.637+0000 7fae1d400640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadf006a430 0x7fadf006c8f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fae08036000 tx=0x7fae0803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:32.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.639+0000 7fae15600640 1 -- 172.21.15.39:0/1522916671 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fae000539d0 con 0x7fae181516c0 2024-01-24T17:48:32.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.904+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fae180b0d20 con 0x7fae181516c0 2024-01-24T17:48:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.912+0000 7fae15600640 1 -- 172.21.15.39:0/1522916671 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7fae00056ff0 con 0x7fae181516c0 2024-01-24T17:48:32.912 INFO:teuthology.orchestra.run.smithi039.stderr:updated caps for client.0 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.914+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadf006a430 msgr2=0x7fadf006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.914+0000 7fae1e800640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadf006a430 0x7fadf006c8f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fae08036000 tx=0x7fae0803b040 comp rx=0 tx=0).stop 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.914+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 msgr2=0x7fae181efdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.914+0000 7fae1e800640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae181efdd0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fae0000c450 tx=0x7fae0000c910 comp rx=0 tx=0).stop 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.914+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 shutdown_connections 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.915+0000 7fae1e800640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae1815ac70 0x7fae1814de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.915+0000 7fae1e800640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fadf006a430 0x7fadf006c8f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.915+0000 7fae1e800640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae181516c0 0x7fae181efdd0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.915+0000 7fae1e800640 1 --2- 172.21.15.39:0/1522916671 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae18070430 0x7fae180c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.915+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 >> 172.21.15.39:0/1522916671 conn(0x7fae1806a200 msgr2=0x7fae18158e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.915+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 shutdown_connections 2024-01-24T17:48:32.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:32.915+0000 7fae1e800640 1 -- 172.21.15.39:0/1522916671 wait complete. 2024-01-24T17:48:32.931 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:33.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.142+0000 7f18b7e00640 1 Processor -- start 2024-01-24T17:48:33.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.142+0000 7f18b7e00640 1 -- start start 2024-01-24T17:48:33.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.143+0000 7f18b7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18b80ca320 0x7f18b815d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:33.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f18b80ca700 0x7f18b815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:33.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b7e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b81540f0 con 0x7f18b815a910 2024-01-24T17:48:33.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b7e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b8159fd0 con 0x7f18b80ca320 2024-01-24T17:48:33.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b7e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b815da00 con 0x7f18b80ca700 2024-01-24T17:48:33.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b6a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18b80ca320 0x7f18b815d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:33.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b6a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18b80ca320 0x7f18b815d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48698/0 (socket says 172.21.15.39:48698) 2024-01-24T17:48:33.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b6a00640 1 -- 172.21.15.39:0/1667767208 learned_addr learned my addr 172.21.15.39:0/1667767208 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:33.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b6a00640 1 -- 172.21.15.39:0/1667767208 >> v2:172.21.15.100:3300/0 conn(0x7f18b80ca700 msgr2=0x7f18b815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:33.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b6a00640 1 --2- 172.21.15.39:0/1667767208 >> v2:172.21.15.100:3300/0 conn(0x7f18b80ca700 0x7f18b815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b6a00640 1 -- 172.21.15.39:0/1667767208 >> v1:172.21.15.100:6789/0 conn(0x7f18b815a910 legacy=0x7f18b806bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:33.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.144+0000 7f18b6a00640 1 -- 172.21.15.39:0/1667767208 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b815dce0 con 0x7f18b80ca320 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.145+0000 7f18b6a00640 1 --2- 172.21.15.39:0/1667767208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca320 0x7f18b815d1b0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f18b8070600 tx=0x7f18a802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d377aa9b60312b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b6000640 1 -- 172.21.15.39:0/1667767208 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8004230 con 0x7f18b80ca320 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b6000640 1 -- 172.21.15.39:0/1667767208 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a8004390 con 0x7f18b80ca320 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b6000640 1 -- 172.21.15.39:0/1667767208 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8003810 con 0x7f18b80ca320 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b7e00640 1 -- 172.21.15.39:0/1667767208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca320 msgr2=0x7f18b815d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b7e00640 1 --2- 172.21.15.39:0/1667767208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca320 0x7f18b815d1b0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f18b8070600 tx=0x7f18a802b7e0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b7e00640 1 -- 172.21.15.39:0/1667767208 shutdown_connections 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b7e00640 1 --2- 172.21.15.39:0/1667767208 >> v2:172.21.15.100:3300/0 conn(0x7f18b80ca700 0x7f18b815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b7e00640 1 --2- 172.21.15.39:0/1667767208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca320 0x7f18b815d1b0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b7e00640 1 -- 172.21.15.39:0/1667767208 >> 172.21.15.39:0/1667767208 conn(0x7f18b806a340 msgr2=0x7f18b80cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:33.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b7e00640 1 -- 172.21.15.39:0/1667767208 shutdown_connections 2024-01-24T17:48:33.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.146+0000 7f18b7e00640 1 -- 172.21.15.39:0/1667767208 wait complete. 2024-01-24T17:48:33.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.148+0000 7f18b7e00640 1 Processor -- start 2024-01-24T17:48:33.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.148+0000 7f18b7e00640 1 -- start start 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.149+0000 7f18b7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18b80ca320 0x7f18b8152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.149+0000 7f18b7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca700 0x7f18b81536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.149+0000 7f18b7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18b815a910 0x7f18b815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.149+0000 7f18b7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8153280 con 0x7f18b80ca700 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.149+0000 7f18b7e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8152c90 con 0x7f18b80ca320 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.149+0000 7f18b7e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b815c870 con 0x7f18b815a910 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca700 0x7f18b81536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca700 0x7f18b81536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48704/0 (socket says 172.21.15.39:48704) 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 -- 172.21.15.39:0/3751016343 learned_addr learned my addr 172.21.15.39:0/3751016343 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18b6a00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18b80ca320 0x7f18b8152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18b7400640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18b815a910 0x7f18b815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:33.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 -- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18b815a910 msgr2=0x7f18b815bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:33.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18b815a910 0x7f18b815bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 -- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18b80ca320 msgr2=0x7f18b8152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:33.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18b80ca320 0x7f18b8152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 -- 172.21.15.39:0/3751016343 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a8009460 con 0x7f18b80ca700 2024-01-24T17:48:33.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.150+0000 7f18afe00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca700 0x7f18b81536a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f18a000b4a0 tx=0x7f18a000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:33.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.151+0000 7f18aea00640 1 -- 172.21.15.39:0/3751016343 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0010610 con 0x7f18b80ca700 2024-01-24T17:48:33.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.151+0000 7f18aea00640 1 -- 172.21.15.39:0/3751016343 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a0010770 con 0x7f18b80ca700 2024-01-24T17:48:33.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.151+0000 7f18aea00640 1 -- 172.21.15.39:0/3751016343 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0010b20 con 0x7f18b80ca700 2024-01-24T17:48:33.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.151+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b815cb50 con 0x7f18b80ca700 2024-01-24T17:48:33.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.151+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b80ad2b0 con 0x7f18b80ca700 2024-01-24T17:48:33.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.152+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b815cd70 con 0x7f18b80ca700 2024-01-24T17:48:33.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.153+0000 7f18aea00640 1 -- 172.21.15.39:0/3751016343 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f18a0010c80 con 0x7f18b80ca700 2024-01-24T17:48:33.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.154+0000 7f18aea00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f189006a430 0x7f189006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:33.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.154+0000 7f18aea00640 1 -- 172.21.15.39:0/3751016343 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18a0088480 con 0x7f18b80ca700 2024-01-24T17:48:33.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.154+0000 7f18b6a00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f189006a430 0x7f189006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:33.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.156+0000 7f18b6a00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f189006a430 0x7f189006c8f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f18a8035000 tx=0x7f18a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:33.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.158+0000 7f18aea00640 1 -- 172.21.15.39:0/3751016343 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f18a0053b00 con 0x7f18b80ca700 2024-01-24T17:48:33.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.394+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18b81533c0 con 0x7f18b80ca700 2024-01-24T17:48:33.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.395+0000 7f18aea00640 1 -- 172.21.15.39:0/3751016343 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f18b81533c0 con 0x7f18b80ca700 2024-01-24T17:48:33.396 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:33.396 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5717,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1968583805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6835","nonce":1968583805},{"type":"v1","addr":"172.21.15.100:6837","nonce":1968583805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":6436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/872967797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":872967797},{"type":"v1","addr":"172.21.15.39:6833","nonce":872967797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54},{"gid":6439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6836/3116066709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":3116066709},{"type":"v1","addr":"172.21.15.100:6836","nonce":3116066709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:48:30.324664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7761},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7761":{"gid":7761,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6835/3495904204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":3495904204},{"type":"v1","addr":"172.21.15.39:6835","nonce":3495904204}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:48:33.396 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 59 2024-01-24T17:48:33.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f189006a430 msgr2=0x7f189006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:33.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f189006a430 0x7f189006c8f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f18a8035000 tx=0x7f18a8008030 comp rx=0 tx=0).stop 2024-01-24T17:48:33.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca700 msgr2=0x7f18b81536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:33.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca700 0x7f18b81536a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f18a000b4a0 tx=0x7f18a000b960 comp rx=0 tx=0).stop 2024-01-24T17:48:33.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 shutdown_connections 2024-01-24T17:48:33.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18b815a910 0x7f18b815bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f189006a430 0x7f189006c8f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18b80ca700 0x7f18b81536a0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 --2- 172.21.15.39:0/3751016343 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18b80ca320 0x7f18b8152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:33.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 >> 172.21.15.39:0/3751016343 conn(0x7f18b806a340 msgr2=0x7f18b80cd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:33.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.398+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 shutdown_connections 2024-01-24T17:48:33.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:33.399+0000 7f18b7e00640 1 -- 172.21.15.39:0/3751016343 wait complete. 2024-01-24T17:48:33.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:48:26.880600+0000', 'modified': '2024-01-24T17:48:30.324664+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7761}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7761': {'gid': 7761, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6835/3495904204', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6834', 'nonce': 3495904204}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 3495904204}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:48:33.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T17:48:33.820 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-24T17:48:34.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.054+0000 7f2493e00640 1 Processor -- start 2024-01-24T17:48:34.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.054+0000 7f2493e00640 1 -- start start 2024-01-24T17:48:34.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.056+0000 7f2493e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f249415cbf0 0x7f24940cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.056+0000 7f2493e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249406acb0 0x7f24940cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.056+0000 7f2493e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24940ce010 con 0x7f24940b1570 2024-01-24T17:48:34.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.056+0000 7f2493e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24940ce870 con 0x7f249415cbf0 2024-01-24T17:48:34.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.056+0000 7f2493e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24940ce2b0 con 0x7f249406acb0 2024-01-24T17:48:34.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.056+0000 7f2492000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249406acb0 0x7f24940cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.057+0000 7f2492a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f249415cbf0 0x7f24940cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.057+0000 7f2492a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f249415cbf0 0x7f24940cf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:36090/0 (socket says 172.21.15.39:36090) 2024-01-24T17:48:34.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.057+0000 7f2492a00640 1 -- 172.21.15.39:0/2315584189 learned_addr learned my addr 172.21.15.39:0/2315584189 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:34.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.057+0000 7f2492000640 1 -- 172.21.15.39:0/2315584189 >> v2:172.21.15.100:3300/0 conn(0x7f249415cbf0 msgr2=0x7f24940cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.057+0000 7f2492000640 1 --2- 172.21.15.39:0/2315584189 >> v2:172.21.15.100:3300/0 conn(0x7f249415cbf0 0x7f24940cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.057+0000 7f2492000640 1 -- 172.21.15.39:0/2315584189 >> v1:172.21.15.100:6789/0 conn(0x7f24940b1570 legacy=0x7f24941605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.057+0000 7f2492000640 1 -- 172.21.15.39:0/2315584189 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24940ce590 con 0x7f249406acb0 2024-01-24T17:48:34.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.059+0000 7f2492000640 1 --2- 172.21.15.39:0/2315584189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249406acb0 0x7f24940cf6d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f247c0097a0 tx=0x7f247c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1f6dde424ea833b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.060+0000 7f2491600640 1 -- 172.21.15.39:0/2315584189 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c02e070 con 0x7f249406acb0 2024-01-24T17:48:34.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.060+0000 7f2491600640 1 -- 172.21.15.39:0/2315584189 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c0315a0 con 0x7f249406acb0 2024-01-24T17:48:34.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.060+0000 7f2491600640 1 -- 172.21.15.39:0/2315584189 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c031930 con 0x7f249406acb0 2024-01-24T17:48:34.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.061+0000 7f2493e00640 1 -- 172.21.15.39:0/2315584189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249406acb0 msgr2=0x7f24940cf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.061+0000 7f2493e00640 1 --2- 172.21.15.39:0/2315584189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249406acb0 0x7f24940cf6d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f247c0097a0 tx=0x7f247c02b840 comp rx=0 tx=0).stop 2024-01-24T17:48:34.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.061+0000 7f2493e00640 1 -- 172.21.15.39:0/2315584189 shutdown_connections 2024-01-24T17:48:34.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.061+0000 7f2493e00640 1 --2- 172.21.15.39:0/2315584189 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249406acb0 0x7f24940cf6d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.061+0000 7f2493e00640 1 --2- 172.21.15.39:0/2315584189 >> v2:172.21.15.100:3300/0 conn(0x7f249415cbf0 0x7f24940cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.061+0000 7f2493e00640 1 -- 172.21.15.39:0/2315584189 >> 172.21.15.39:0/2315584189 conn(0x7f249409bad0 msgr2=0x7f24940d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:34.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.061+0000 7f2493e00640 1 -- 172.21.15.39:0/2315584189 shutdown_connections 2024-01-24T17:48:34.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.061+0000 7f2493e00640 1 -- 172.21.15.39:0/2315584189 wait complete. 2024-01-24T17:48:34.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.063+0000 7f2493e00640 1 Processor -- start 2024-01-24T17:48:34.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.063+0000 7f2493e00640 1 -- start start 2024-01-24T17:48:34.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2493e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249406acb0 0x7f24940b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2493e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24940b1570 0x7f249409ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2493e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249415cbf0 0x7f24940cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2493e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2494159620 con 0x7f249406acb0 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2493e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24940a04f0 con 0x7f24940b1570 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2493e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24940ccba0 con 0x7f249415cbf0 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2492a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249406acb0 0x7f24940b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2492a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249406acb0 0x7f24940b6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48712/0 (socket says 172.21.15.39:48712) 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.064+0000 7f2492a00640 1 -- 172.21.15.39:0/4005035278 learned_addr learned my addr 172.21.15.39:0/4005035278 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2492a00640 1 -- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249415cbf0 msgr2=0x7f24940cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2492a00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249415cbf0 0x7f24940cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2492000640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24940b1570 0x7f249409ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2492a00640 1 -- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24940b1570 msgr2=0x7f249409ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2492a00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24940b1570 0x7f249409ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2492a00640 1 -- 172.21.15.39:0/4005035278 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c009400 con 0x7f249406acb0 2024-01-24T17:48:34.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2492a00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249406acb0 0x7f24940b6ef0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f2484009d40 tx=0x7f248400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f248aa00640 1 -- 172.21.15.39:0/4005035278 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248400cda0 con 0x7f249406acb0 2024-01-24T17:48:34.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f248aa00640 1 -- 172.21.15.39:0/4005035278 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248400f040 con 0x7f249406acb0 2024-01-24T17:48:34.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2493e00640 1 -- 172.21.15.39:0/4005035278 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24940c99e0 con 0x7f249406acb0 2024-01-24T17:48:34.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f248aa00640 1 -- 172.21.15.39:0/4005035278 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484014680 con 0x7f249406acb0 2024-01-24T17:48:34.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.065+0000 7f2493e00640 1 -- 172.21.15.39:0/4005035278 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24940c9f60 con 0x7f249406acb0 2024-01-24T17:48:34.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.067+0000 7f248aa00640 1 -- 172.21.15.39:0/4005035278 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f24840147e0 con 0x7f249406acb0 2024-01-24T17:48:34.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.068+0000 7f248aa00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f246c06a430 0x7f246c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.068+0000 7f248aa00640 1 -- 172.21.15.39:0/4005035278 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24840892d0 con 0x7f249406acb0 2024-01-24T17:48:34.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.069+0000 7f2492000640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f246c06a430 0x7f246c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.069+0000 7f249c77d740 1 Processor -- start 2024-01-24T17:48:34.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.069+0000 7f249c77d740 1 -- start start 2024-01-24T17:48:34.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.070+0000 7f249c77d740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 0x56548f0015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.070+0000 7f249c77d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd1150 0x56548effe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.070+0000 7f249c77d740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548efd4d50 0x56548effe980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.070+0000 7f249c77d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548efac6c0 con 0x56548efd1150 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.070+0000 7f249c77d740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548ef41070 con 0x56548f0011d0 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.070+0000 7f249c77d740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56548ef3e800 con 0x56548efd4d50 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.070+0000 7f2492000640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f246c06a430 0x7f246c06c8f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f247c036000 tx=0x7f247c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.070+0000 7f2492000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd1150 0x56548effe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd1150 0x56548effe440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48716/0 (socket says 172.21.15.39:48716) 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492000640 1 -- 172.21.15.39:0/1908816997 learned_addr learned my addr 172.21.15.39:0/1908816997 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492a00640 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 0x56548f0015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2493400640 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548efd4d50 0x56548effe980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492000640 1 -- 172.21.15.39:0/1908816997 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548efd4d50 msgr2=0x56548effe980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492000640 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548efd4d50 0x56548effe980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492000640 1 -- 172.21.15.39:0/1908816997 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 msgr2=0x56548f0015b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492000640 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 0x56548f0015b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492000640 1 -- 172.21.15.39:0/1908816997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56548ef01780 con 0x56548efd1150 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.071+0000 7f2492000640 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd1150 0x56548effe440 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f247c000c00 tx=0x7f247c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf390190f24a7840 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f2483400640 1 -- 172.21.15.39:0/1908816997 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c032980 con 0x56548efd1150 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f2483400640 1 -- 172.21.15.39:0/1908816997 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c032ae0 con 0x56548efd1150 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f2483400640 1 -- 172.21.15.39:0/1908816997 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c032e70 con 0x56548efd1150 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 -- 172.21.15.39:0/1908816997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd1150 msgr2=0x56548effe440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd1150 0x56548effe440 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f247c000c00 tx=0x7f247c030000 comp rx=0 tx=0).stop 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 -- 172.21.15.39:0/1908816997 shutdown_connections 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548efd4d50 0x56548effe980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd1150 0x56548effe440 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 --2- 172.21.15.39:0/1908816997 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 0x56548f0015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 -- 172.21.15.39:0/1908816997 >> 172.21.15.39:0/1908816997 conn(0x56548efd3460 msgr2=0x56548f01bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 -- 172.21.15.39:0/1908816997 shutdown_connections 2024-01-24T17:48:34.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.072+0000 7f249c77d740 1 -- 172.21.15.39:0/1908816997 wait complete. 2024-01-24T17:48:34.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.073+0000 7f249c77d740 1 -- start start 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.073+0000 7f249c77d740 1 Processor -- start 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.074+0000 7f249c77d740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548f013240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.074+0000 7f249c77d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548effb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.074+0000 7f249c77d740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548f015bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.074+0000 7f249c77d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548ef3a0f0 con 0x56548f0011d0 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.074+0000 7f249c77d740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548eedb8c0 con 0x56548efd4d50 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.074+0000 7f249c77d740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56548eedb560 con 0x56548effc4f0 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.074+0000 7f249c77d740 10 client.0 _ll_register_callbacks cb 0x56548ef33a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548effb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548effb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48730/0 (socket says 172.21.15.39:48730) 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 -- 172.21.15.39:0/43979350 learned_addr learned my addr 172.21.15.39:0/43979350 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:34.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492a00640 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548f013240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2493400640 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548f015bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 -- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 msgr2=0x56548f015bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548f015bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 -- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 msgr2=0x56548f013240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548f013240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 -- 172.21.15.39:0/43979350 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24940ce590 con 0x56548f0011d0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492a00640 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548f013240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f2492000640 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548effb6d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f247c032040 tx=0x7f247c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3ed43f4ec899ce server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.075+0000 7f247be00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f247be00640 1 -- 172.21.15.39:0/43979350 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c003a20 con 0x56548f0011d0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 1 -- 172.21.15.39:0/43979350 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56548ef64640 con 0x56548f0011d0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f247be00640 1 -- 172.21.15.39:0/43979350 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f247c003b80 con 0x56548f0011d0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f247be00640 1 -- 172.21.15.39:0/43979350 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f247c003d80 con 0x56548f0011d0 2024-01-24T17:48:34.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 10 client.7899 fetch_fsmap learned FSMap version 59 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 1 -- 172.21.15.39:0/43979350 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x56548ef38770 con 0x56548f0011d0 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f247be00640 1 -- 172.21.15.39:0/43979350 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56548ef64640 con 0x56548f0011d0 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f247be00640 1 -- 172.21.15.39:0/43979350 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 59) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f247c050480 con 0x56548f0011d0 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 10 client.7899 fetch_fsmap finished waiting for FSMap version 59 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 10 client.7899 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 20 client.7899 populate_metadata read hostname 'smithi039' 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] conn(0x56548f037170 0x56548f039560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 4 client.7899 mds_command: new command op to 7761 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.076+0000 7f249c77d740 1 -- 172.21.15.39:0/43979350 --> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56548ef2c940 con 0x56548f037170 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.077+0000 7f2492a00640 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] conn(0x56548f037170 0x56548f039560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.077+0000 7f2492a00640 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] conn(0x56548f037170 0x56548f039560 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8034e6e42173b8f5 server_cookie=35ad08560f492537 in_seq=0 out_seq=0 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.077+0000 7f247be00640 10 client.7899 ms_handle_connect on v2:172.21.15.39:6834/3495904204 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f247be00640 1 -- 172.21.15.39:0/43979350 <== mds.0 v2:172.21.15.39:6834/3495904204 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56548ef2c940 con 0x56548f037170 2024-01-24T17:48:34.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f247be00640 10 client.7899 handle_command_reply: tid=0 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 client.7899 shutdown 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 -- 172.21.15.39:0/43979350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 msgr2=0x56548effb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548effb6d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f247c032040 tx=0x7f247c004730 comp rx=0 tx=0).stop 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 -- 172.21.15.39:0/43979350 shutdown_connections 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] conn(0x56548f037170 0x56548f039560 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548f015bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548effb6d0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 --2- 172.21.15.39:0/43979350 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548f013240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 -- 172.21.15.39:0/43979350 >> 172.21.15.39:0/43979350 conn(0x56548efd1150 msgr2=0x56548f004d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f247be00640 0 client.7899 ms_handle_reset on v2:172.21.15.39:6834/3495904204 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 -- 172.21.15.39:0/43979350 shutdown_connections 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.078+0000 7f249c77d740 1 -- 172.21.15.39:0/43979350 wait complete. 2024-01-24T17:48:34.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.079+0000 7f249c77d740 20 client.7899 trim_cache size 0 max 16384 2024-01-24T17:48:34.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.095+0000 7f249c77d740 1 Processor -- start 2024-01-24T17:48:34.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.095+0000 7f249c77d740 1 -- start start 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f249c77d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd4d50 0x56548effcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f249c77d740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548f01ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f249c77d740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 0x56548f016ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f249c77d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548eca1f20 con 0x56548efd4d50 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f249c77d740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548efd2d00 con 0x56548f0011d0 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f249c77d740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56548f01f1f0 con 0x56548effc4f0 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f2492a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd4d50 0x56548effcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f2492a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd4d50 0x56548effcf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48734/0 (socket says 172.21.15.39:48734) 2024-01-24T17:48:34.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.096+0000 7f2492a00640 1 -- 172.21.15.39:0/319226717 learned_addr learned my addr 172.21.15.39:0/319226717 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2492a00640 1 -- 172.21.15.39:0/319226717 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 msgr2=0x56548f01ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2492a00640 1 --2- 172.21.15.39:0/319226717 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548f01ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2492a00640 1 -- 172.21.15.39:0/319226717 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 msgr2=0x56548f016ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2492a00640 1 --2- 172.21.15.39:0/319226717 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 0x56548f016ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2492a00640 1 -- 172.21.15.39:0/319226717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56548ef6e2f0 con 0x56548efd4d50 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2492a00640 1 --2- 172.21.15.39:0/319226717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd4d50 0x56548effcf00 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f2484009d10 tx=0x7f2484087630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642888613182be05 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2483e00640 1 -- 172.21.15.39:0/319226717 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248405c310 con 0x56548efd4d50 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2483e00640 1 -- 172.21.15.39:0/319226717 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24840604c0 con 0x56548efd4d50 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.097+0000 7f2483e00640 1 -- 172.21.15.39:0/319226717 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2484087870 con 0x56548efd4d50 2024-01-24T17:48:34.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 -- 172.21.15.39:0/319226717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd4d50 msgr2=0x56548effcf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 --2- 172.21.15.39:0/319226717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd4d50 0x56548effcf00 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f2484009d10 tx=0x7f2484087630 comp rx=0 tx=0).stop 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 -- 172.21.15.39:0/319226717 shutdown_connections 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 --2- 172.21.15.39:0/319226717 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548f0011d0 0x56548f016ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 --2- 172.21.15.39:0/319226717 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548f01ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 --2- 172.21.15.39:0/319226717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548efd4d50 0x56548effcf00 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 -- 172.21.15.39:0/319226717 >> 172.21.15.39:0/319226717 conn(0x56548efd1150 msgr2=0x56548efcc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 -- 172.21.15.39:0/319226717 shutdown_connections 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.098+0000 7f249c77d740 1 -- 172.21.15.39:0/319226717 wait complete. 2024-01-24T17:48:34.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.099+0000 7f249c77d740 1 -- start start 2024-01-24T17:48:34.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.099+0000 7f249c77d740 1 Processor -- start 2024-01-24T17:48:34.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f249c77d740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548efd82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f249c77d740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548effcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f249c77d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548f067c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f249c77d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548f015450 con 0x56548f0011d0 2024-01-24T17:48:34.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f249c77d740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56548efd05f0 con 0x56548efd4d50 2024-01-24T17:48:34.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f249c77d740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56548effd260 con 0x56548effc4f0 2024-01-24T17:48:34.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f249c77d740 10 client.0 _ll_register_callbacks cb 0x7f247c009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f2493400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548f067c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f2493400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548f067c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48746/0 (socket says 172.21.15.39:48746) 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f2493400640 1 -- 172.21.15.39:0/1280850239 learned_addr learned my addr 172.21.15.39:0/1280850239 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f2492000640 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548effcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.100+0000 7f2492a00640 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548efd82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f2493400640 1 -- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 msgr2=0x56548effcd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f2493400640 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548effcd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f2493400640 1 -- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 msgr2=0x56548efd82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f2493400640 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548efd82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f2493400640 1 -- 172.21.15.39:0/1280850239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24940c9f60 con 0x56548f0011d0 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f2492a00640 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548efd82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f2493400640 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548f067c90 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f248c000f60 tx=0x7f248c00f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cea5c1948e3a25 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f247be00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:48:34.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f247be00640 1 -- 172.21.15.39:0/1280850239 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c00fe10 con 0x56548f0011d0 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f247be00640 1 -- 172.21.15.39:0/1280850239 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f248c0050d0 con 0x56548f0011d0 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f249c77d740 1 -- 172.21.15.39:0/1280850239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f247c032ae0 con 0x56548f0011d0 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f247be00640 1 -- 172.21.15.39:0/1280850239 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f248c005290 con 0x56548f0011d0 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f249c77d740 10 client.7905 fetch_fsmap learned FSMap version 59 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.101+0000 7f249c77d740 1 -- 172.21.15.39:0/1280850239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x56548eedaad0 con 0x56548f0011d0 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f247be00640 1 -- 172.21.15.39:0/1280850239 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c032ae0 con 0x56548f0011d0 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f247be00640 1 -- 172.21.15.39:0/1280850239 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 59) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f248c012070 con 0x56548f0011d0 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f249c77d740 10 client.7905 fetch_fsmap finished waiting for FSMap version 59 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f249c77d740 10 client.7905 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f249c77d740 20 client.7905 populate_metadata read hostname 'smithi039' 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f249c77d740 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] conn(0x56548f02e160 0x56548f085980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f249c77d740 4 client.7905 mds_command: new command op to 7761 tid=0[{"prefix": "status"}] 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f249c77d740 1 -- 172.21.15.39:0/1280850239 --> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56548ef33170 con 0x56548f02e160 2024-01-24T17:48:34.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.102+0000 7f2492a00640 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] conn(0x56548f02e160 0x56548f085980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f2492a00640 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] conn(0x56548f02e160 0x56548f085980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ccd15d48574ce97 server_cookie=c00651d7bdb84440 in_seq=0 out_seq=0 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f247be00640 10 client.7905 ms_handle_connect on v2:172.21.15.39:6834/3495904204 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f247be00640 1 -- 172.21.15.39:0/1280850239 <== mds.0 v2:172.21.15.39:6834/3495904204 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x56548ef33170 con 0x56548f02e160 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f247be00640 10 client.7905 handle_command_reply: tid=0 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 client.7905 shutdown 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 -- 172.21.15.39:0/1280850239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 msgr2=0x56548f067c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548f067c90 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f248c000f60 tx=0x7f248c00f5a0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 -- 172.21.15.39:0/1280850239 shutdown_connections 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.39:6834/3495904204,v1:172.21.15.39:6835/3495904204] conn(0x56548f02e160 0x56548f085980 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56548f0011d0 0x56548f067c90 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56548effc4f0 0x56548effcd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 --2- 172.21.15.39:0/1280850239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56548efd4d50 0x56548efd82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f247be00640 0 client.7905 ms_handle_reset on v2:172.21.15.39:6834/3495904204 2024-01-24T17:48:34.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 -- 172.21.15.39:0/1280850239 >> 172.21.15.39:0/1280850239 conn(0x56548efd1150 msgr2=0x56548eff4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 -- 172.21.15.39:0/1280850239 shutdown_connections 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.103+0000 7f249c77d740 1 -- 172.21.15.39:0/1280850239 wait complete. 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f249c77d740 20 client.7905 trim_cache size 0 max 16384 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 7761, 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:48:34.108 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:48:34.109 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:48:34.109 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 7.21841811, 2024-01-24T17:48:34.109 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 59, 2024-01-24T17:48:34.109 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 100, 2024-01-24T17:48:34.109 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 100, 2024-01-24T17:48:34.109 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 13.037332519 2024-01-24T17:48:34.109 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 -- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f246c06a430 msgr2=0x7f246c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f246c06a430 0x7f246c06c8f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f247c036000 tx=0x7f247c03d040 comp rx=0 tx=0).stop 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 -- 172.21.15.39:0/4005035278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249406acb0 msgr2=0x7f24940b6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249406acb0 0x7f24940b6ef0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f2484009d40 tx=0x7f248400c490 comp rx=0 tx=0).stop 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 -- 172.21.15.39:0/4005035278 shutdown_connections 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f249415cbf0 0x7f24940cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f24940b1570 0x7f249409ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f246c06a430 0x7f246c06c8f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 --2- 172.21.15.39:0/4005035278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f249406acb0 0x7f24940b6ef0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 -- 172.21.15.39:0/4005035278 >> 172.21.15.39:0/4005035278 conn(0x7f249409bad0 msgr2=0x7f249406bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:34.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 -- 172.21.15.39:0/4005035278 shutdown_connections 2024-01-24T17:48:34.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.104+0000 7f2493e00640 1 -- 172.21.15.39:0/4005035278 wait complete. 2024-01-24T17:48:34.121 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:48:34.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.331+0000 7fd95e800640 1 Processor -- start 2024-01-24T17:48:34.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.331+0000 7fd95e800640 1 -- start start 2024-01-24T17:48:34.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.332+0000 7fd95e800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd95815cbe0 0x7fd9580cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.332+0000 7fd95e800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd9580ce510 0x7fd9580d0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.332+0000 7fd95e800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9580d1800 con 0x7fd9580d13d0 2024-01-24T17:48:34.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.332+0000 7fd95e800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9580b8820 con 0x7fd95815cbe0 2024-01-24T17:48:34.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.332+0000 7fd95e800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9580b5790 con 0x7fd9580ce510 2024-01-24T17:48:34.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.333+0000 7fd957400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd95815cbe0 0x7fd9580cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.333+0000 7fd957400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd95815cbe0 0x7fd9580cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48752/0 (socket says 172.21.15.39:48752) 2024-01-24T17:48:34.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.333+0000 7fd957400640 1 -- 172.21.15.39:0/2928889021 learned_addr learned my addr 172.21.15.39:0/2928889021 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:34.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.333+0000 7fd956a00640 1 --2- 172.21.15.39:0/2928889021 >> v2:172.21.15.100:3300/0 conn(0x7fd9580ce510 0x7fd9580d0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.333+0000 7fd957400640 1 -- 172.21.15.39:0/2928889021 >> v2:172.21.15.100:3300/0 conn(0x7fd9580ce510 msgr2=0x7fd9580d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:34.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.333+0000 7fd957400640 1 --2- 172.21.15.39:0/2928889021 >> v2:172.21.15.100:3300/0 conn(0x7fd9580ce510 0x7fd9580d0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.333+0000 7fd957400640 1 -- 172.21.15.39:0/2928889021 >> v1:172.21.15.39:6789/0 conn(0x7fd9580d13d0 legacy=0x7fd958160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.333+0000 7fd957400640 1 -- 172.21.15.39:0/2928889021 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9580b5a50 con 0x7fd95815cbe0 2024-01-24T17:48:34.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.335+0000 7fd957400640 1 --2- 172.21.15.39:0/2928889021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd95815cbe0 0x7fd9580cdfd0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd95808bdb0 tx=0x7fd94802bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b128455c51252526 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.335+0000 7fd956000640 1 -- 172.21.15.39:0/2928889021 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94802cb60 con 0x7fd95815cbe0 2024-01-24T17:48:34.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.335+0000 7fd956000640 1 -- 172.21.15.39:0/2928889021 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94802ccc0 con 0x7fd95815cbe0 2024-01-24T17:48:34.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.335+0000 7fd956000640 1 -- 172.21.15.39:0/2928889021 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948034610 con 0x7fd95815cbe0 2024-01-24T17:48:34.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.336+0000 7fd95e800640 1 -- 172.21.15.39:0/2928889021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd95815cbe0 msgr2=0x7fd9580cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.336+0000 7fd95e800640 1 --2- 172.21.15.39:0/2928889021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd95815cbe0 0x7fd9580cdfd0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fd95808bdb0 tx=0x7fd94802bfc0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.336+0000 7fd95e800640 1 -- 172.21.15.39:0/2928889021 shutdown_connections 2024-01-24T17:48:34.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.336+0000 7fd95e800640 1 --2- 172.21.15.39:0/2928889021 >> v2:172.21.15.100:3300/0 conn(0x7fd9580ce510 0x7fd9580d0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.336+0000 7fd95e800640 1 --2- 172.21.15.39:0/2928889021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd95815cbe0 0x7fd9580cdfd0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.336+0000 7fd95e800640 1 -- 172.21.15.39:0/2928889021 >> 172.21.15.39:0/2928889021 conn(0x7fd95809bc80 msgr2=0x7fd9580d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:34.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.336+0000 7fd95e800640 1 -- 172.21.15.39:0/2928889021 shutdown_connections 2024-01-24T17:48:34.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.336+0000 7fd95e800640 1 -- 172.21.15.39:0/2928889021 wait complete. 2024-01-24T17:48:34.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.338+0000 7fd95e800640 1 Processor -- start 2024-01-24T17:48:34.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.338+0000 7fd95e800640 1 -- start start 2024-01-24T17:48:34.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd95e800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9580ce510 0x7fd95815ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd95e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9580d13d0 0x7fd95815a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd95e800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd95815cbe0 0x7fd9580cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd95e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95815fd80 con 0x7fd9580d13d0 2024-01-24T17:48:34.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd95e800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95815f240 con 0x7fd9580ce510 2024-01-24T17:48:34.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd95e800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95815f4e0 con 0x7fd95815cbe0 2024-01-24T17:48:34.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd956a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9580d13d0 0x7fd95815a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd956a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9580d13d0 0x7fd95815a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48754/0 (socket says 172.21.15.39:48754) 2024-01-24T17:48:34.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd956a00640 1 -- 172.21.15.39:0/2876486507 learned_addr learned my addr 172.21.15.39:0/2876486507 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:34.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd956a00640 1 -- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd95815cbe0 msgr2=0x7fd9580cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd956a00640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd95815cbe0 0x7fd9580cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd956a00640 1 -- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9580ce510 msgr2=0x7fd95815ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd956a00640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9580ce510 0x7fd95815ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.339+0000 7fd956a00640 1 -- 172.21.15.39:0/2876486507 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948009b30 con 0x7fd9580d13d0 2024-01-24T17:48:34.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.340+0000 7fd956a00640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9580d13d0 0x7fd95815a750 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fd94000d5f0 tx=0x7fd94000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.340+0000 7fd94f400640 1 -- 172.21.15.39:0/2876486507 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400125f0 con 0x7fd9580d13d0 2024-01-24T17:48:34.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.340+0000 7fd94f400640 1 -- 172.21.15.39:0/2876486507 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940012750 con 0x7fd9580d13d0 2024-01-24T17:48:34.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.340+0000 7fd94f400640 1 -- 172.21.15.39:0/2876486507 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940012b00 con 0x7fd9580d13d0 2024-01-24T17:48:34.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.340+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95815f7c0 con 0x7fd9580d13d0 2024-01-24T17:48:34.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.340+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd958158af0 con 0x7fd9580d13d0 2024-01-24T17:48:34.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.342+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd958158950 con 0x7fd9580d13d0 2024-01-24T17:48:34.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.342+0000 7fd94f400640 1 -- 172.21.15.39:0/2876486507 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd940012c60 con 0x7fd9580d13d0 2024-01-24T17:48:34.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.343+0000 7fd94f400640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd93006a430 0x7fd93006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:34.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.343+0000 7fd94f400640 1 -- 172.21.15.39:0/2876486507 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd940088de0 con 0x7fd9580d13d0 2024-01-24T17:48:34.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.344+0000 7fd957400640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd93006a430 0x7fd93006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:34.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.345+0000 7fd957400640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd93006a430 0x7fd93006c8f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd948035000 tx=0x7fd94802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:34.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.348+0000 7fd94f400640 1 -- 172.21.15.39:0/2876486507 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd958158950 con 0x7fd9580d13d0 2024-01-24T17:48:34.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.585+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd958158950 con 0x7fd9580d13d0 2024-01-24T17:48:34.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.585+0000 7fd94f400640 1 -- 172.21.15.39:0/2876486507 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v59) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd958158950 con 0x7fd9580d13d0 2024-01-24T17:48:34.586 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:48:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd93006a430 msgr2=0x7fd93006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd93006a430 0x7fd93006c8f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd948035000 tx=0x7fd94802e000 comp rx=0 tx=0).stop 2024-01-24T17:48:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9580d13d0 msgr2=0x7fd95815a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9580d13d0 0x7fd95815a750 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fd94000d5f0 tx=0x7fd94000dab0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 shutdown_connections 2024-01-24T17:48:34.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd95815cbe0 0x7fd9580cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd93006a430 0x7fd93006c8f0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9580d13d0 0x7fd95815a750 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 --2- 172.21.15.39:0/2876486507 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9580ce510 0x7fd95815ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 >> 172.21.15.39:0/2876486507 conn(0x7fd95809bc80 msgr2=0x7fd9580d2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.588+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 shutdown_connections 2024-01-24T17:48:34.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:34.589+0000 7fd95e800640 1 -- 172.21.15.39:0/2876486507 wait complete. 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:48:34.608 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:48:34.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:34.609 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:48:34.663 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:48:34.663 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:48:34.664 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42217sec preferred_lft 42217sec 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:34.665 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:48:34.666 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:34.666 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:48:34.666 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:48:34.666 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:48:34.666 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:34.666 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:48:34.666 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:48:34.667 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:48:34.667 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:34.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:34.668 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:48:34.765 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:48:34.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:34.770 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:48:34.825 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:48:34.825 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:48:34.825 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:48:34.825 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:48:34.825 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:48:34.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:34.826 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:48:34.826 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:48:34.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:48:34.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:48:34.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:48:34.826 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:48:35.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:35.019 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:48:35.074 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:48:35.075 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:48:35.075 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:48:35.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:35.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:48:35.176 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:48:35.176 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:48:35.176 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:48:35.232 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:48:35.232 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:48:35.232 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:48:35.287 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:48:35.391 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:48:35.392 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:48:35.489 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:48:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:48:35.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:35.494 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:48:35.553 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:48:35.553 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:48:35.558 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:48:35.637 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:48:35.637 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:48:35.718 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:48:35.722 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:48:35.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:35.723 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:48:35.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:48:35.761+0000 7fd68f638480 -1 init, newargv = 0x55b4553d3b90 newargc=15 2024-01-24T17:48:35.762 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[70338]: starting ceph client 2024-01-24T17:48:35.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[70338]: starting fuse 2024-01-24T17:48:36.873 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:48:36.981 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:48:36.981 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:48:37.078 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:48:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:48:37.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:37.083 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:48:37.140 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:48:37.140 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:48:37.140 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:48:37.140 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:48:37.140 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:48:37.141 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:48:37.142 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:48:37.332 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.70338.asok 2024-01-24T17:48:37.511 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.70338.asok 2024-01-24T17:48:37.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:37.511 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.70338.asok status 2024-01-24T17:48:37.761 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:48:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:48:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:48:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:48:37.761 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "70338", 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 7920, 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 7920 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:48:37.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3453113683 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3453113683 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.7920 192.168.0.1:0/3453113683", 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/3453113683", 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 59, 2024-01-24T17:48:37.763 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 100, 2024-01-24T17:48:37.764 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:48:37.764 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:48:37.764 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:48:37.764 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:48:37.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:37.777 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:48:37.777 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:48:37.834 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:48:37.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:48:37.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:37.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:48:37.937 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:48:38.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.160+0000 7fa70b000640 1 Processor -- start 2024-01-24T17:48:38.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.160+0000 7fa70b000640 1 -- start start 2024-01-24T17:48:38.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa70b000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa704159d00 0x7fa7040cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:38.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa70b000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7040b3f20 0x7fa7040714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:38.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa70b000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7040ccd10 con 0x7fa704071a30 2024-01-24T17:48:38.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa70b000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7040ce230 con 0x7fa704159d00 2024-01-24T17:48:38.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa70b000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040ccff0 con 0x7fa7040b3f20 2024-01-24T17:48:38.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa703e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa704159d00 0x7fa7040cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:38.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa703e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa704159d00 0x7fa7040cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48770/0 (socket says 172.21.15.39:48770) 2024-01-24T17:48:38.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa703e00640 1 -- 172.21.15.39:0/2061042074 learned_addr learned my addr 172.21.15.39:0/2061042074 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:38.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa709000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa704071a30 0x7fa7040c8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:42650/0 (socket says 172.21.15.39:42650) 2024-01-24T17:48:38.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.162+0000 7fa703400640 1 --2- 172.21.15.39:0/2061042074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7040b3f20 0x7fa7040714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:38.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.163+0000 7fa703e00640 1 -- 172.21.15.39:0/2061042074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7040b3f20 msgr2=0x7fa7040714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:38.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.163+0000 7fa703e00640 1 --2- 172.21.15.39:0/2061042074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7040b3f20 0x7fa7040714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.163+0000 7fa703e00640 1 -- 172.21.15.39:0/2061042074 >> v1:172.21.15.39:6789/0 conn(0x7fa704071a30 legacy=0x7fa7040c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:38.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.163+0000 7fa703e00640 1 -- 172.21.15.39:0/2061042074 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040ad720 con 0x7fa704159d00 2024-01-24T17:48:38.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.164+0000 7fa703e00640 1 --2- 172.21.15.39:0/2061042074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa704159d00 0x7fa7040cdcf0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400bb10 tx=0x7fa6f402e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816df703af3281d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:38.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.164+0000 7fa702a00640 1 -- 172.21.15.39:0/2061042074 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4036440 con 0x7fa704159d00 2024-01-24T17:48:38.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.164+0000 7fa702a00640 1 -- 172.21.15.39:0/2061042074 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f40365a0 con 0x7fa704159d00 2024-01-24T17:48:38.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.164+0000 7fa702a00640 1 -- 172.21.15.39:0/2061042074 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4036930 con 0x7fa704159d00 2024-01-24T17:48:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.165+0000 7fa70b000640 1 -- 172.21.15.39:0/2061042074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa704159d00 msgr2=0x7fa7040cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.165+0000 7fa70b000640 1 --2- 172.21.15.39:0/2061042074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa704159d00 0x7fa7040cdcf0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400bb10 tx=0x7fa6f402e150 comp rx=0 tx=0).stop 2024-01-24T17:48:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.165+0000 7fa70b000640 1 -- 172.21.15.39:0/2061042074 shutdown_connections 2024-01-24T17:48:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.165+0000 7fa70b000640 1 --2- 172.21.15.39:0/2061042074 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7040b3f20 0x7fa7040714f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.165+0000 7fa70b000640 1 --2- 172.21.15.39:0/2061042074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa704159d00 0x7fa7040cdcf0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.165+0000 7fa70b000640 1 -- 172.21.15.39:0/2061042074 >> 172.21.15.39:0/2061042074 conn(0x7fa70406bf20 msgr2=0x7fa70406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.165+0000 7fa70b000640 1 -- 172.21.15.39:0/2061042074 shutdown_connections 2024-01-24T17:48:38.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.165+0000 7fa70b000640 1 -- 172.21.15.39:0/2061042074 wait complete. 2024-01-24T17:48:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.166+0000 7fa70b000640 1 Processor -- start 2024-01-24T17:48:38.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.167+0000 7fa70b000640 1 -- start start 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.167+0000 7fa70b000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa704071a30 0x7fa7041f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa70b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7040b3f20 0x7fa70415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa70b000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa704159d00 0x7fa7041efa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa70b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70415b1a0 con 0x7fa7040b3f20 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa70b000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70415a7d0 con 0x7fa704159d00 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa70b000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7041effd0 con 0x7fa704071a30 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7040b3f20 0x7fa70415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7040b3f20 0x7fa70415ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48782/0 (socket says 172.21.15.39:48782) 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703400640 1 -- 172.21.15.39:0/766492603 learned_addr learned my addr 172.21.15.39:0/766492603 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703e00640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa704071a30 0x7fa7041f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703400640 1 -- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa704071a30 msgr2=0x7fa7041f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:38.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703400640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa704071a30 0x7fa7041f31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703400640 1 -- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa704159d00 msgr2=0x7fa7041efa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703400640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa704159d00 0x7fa7041efa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.168+0000 7fa703400640 1 -- 172.21.15.39:0/766492603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f400b770 con 0x7fa7040b3f20 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.169+0000 7fa703400640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7040b3f20 0x7fa70415ac60 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c820 tx=0x7fa6ec00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.169+0000 7fa6fbe00640 1 -- 172.21.15.39:0/766492603 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec004220 con 0x7fa7040b3f20 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.169+0000 7fa6fbe00640 1 -- 172.21.15.39:0/766492603 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec0043a0 con 0x7fa7040b3f20 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.169+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7040c6d00 con 0x7fa7040b3f20 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.169+0000 7fa6fbe00640 1 -- 172.21.15.39:0/766492603 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec007a70 con 0x7fa7040b3f20 2024-01-24T17:48:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.169+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70415a460 con 0x7fa7040b3f20 2024-01-24T17:48:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.170+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7040b3370 con 0x7fa7040b3f20 2024-01-24T17:48:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.171+0000 7fa6fbe00640 1 -- 172.21.15.39:0/766492603 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa6ec007bd0 con 0x7fa7040b3f20 2024-01-24T17:48:38.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.172+0000 7fa6fbe00640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6dc06a430 0x7fa6dc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:38.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.172+0000 7fa6fbe00640 1 -- 172.21.15.39:0/766492603 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6ec0886a0 con 0x7fa7040b3f20 2024-01-24T17:48:38.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.173+0000 7fa703e00640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6dc06a430 0x7fa6dc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:38.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.174+0000 7fa703e00640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6dc06a430 0x7fa6dc06c8f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4038000 tx=0x7fa6f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.177+0000 7fa6fbe00640 1 -- 172.21.15.39:0/766492603 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa6ec053c90 con 0x7fa7040b3f20 2024-01-24T17:48:38.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.414+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa70404dff0 con 0x7fa7040b3f20 2024-01-24T17:48:38.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.414+0000 7fa6fbe00640 1 -- 172.21.15.39:0/766492603 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v59) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa6ec01e070 con 0x7fa7040b3f20 2024-01-24T17:48:38.415 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6dc06a430 msgr2=0x7fa6dc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6dc06a430 0x7fa6dc06c8f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4038000 tx=0x7fa6f4008040 comp rx=0 tx=0).stop 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7040b3f20 msgr2=0x7fa70415ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7040b3f20 0x7fa70415ac60 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c820 tx=0x7fa6ec00cce0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 shutdown_connections 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa704159d00 0x7fa7041efa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa6dc06a430 0x7fa6dc06c8f0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7040b3f20 0x7fa70415ac60 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 --2- 172.21.15.39:0/766492603 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa704071a30 0x7fa7041f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 >> 172.21.15.39:0/766492603 conn(0x7fa70406bf20 msgr2=0x7fa7040c20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 shutdown_connections 2024-01-24T17:48:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:38.417+0000 7fa70b000640 1 -- 172.21.15.39:0/766492603 wait complete. 2024-01-24T17:48:38.434 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:48:38.434 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:48:38.434 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:48:38.434 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-24T17:48:38.434 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:48:38.434 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:48:38.434 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:48:38.435 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:48:38.435 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:48:38.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:38.435 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:48:38.493 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42211sec preferred_lft 42211sec 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:48:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:48:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:48:38.496 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:48:38.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:38.497 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:48:38.583 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:48:38.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:38.587 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:48:38.640 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:48:38.640 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:48:38.640 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:48:38.640 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:48:38.640 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:48:38.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:38.641 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:48:38.641 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:48:38.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:48:38.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:48:38.641 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:48:38.641 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:48:38.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:38.819 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:48:38.867 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:48:38.868 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:48:38.870 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:48:38.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:38.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:48:38.968 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-24T17:48:38.969 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:48:38.969 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:48:39.019 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:48:39.020 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:48:39.020 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:48:39.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:48:39.159 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:48:39.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:48:39.254 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:48:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:48:39.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:39.257 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:48:39.308 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:48:39.308 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-24T17:48:39.312 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:48:39.388 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:48:39.388 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:48:39.456 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:48:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:48:39.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:39.460 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:48:39.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:48:39.496+0000 7f349406f480 -1 init, newargv = 0x55bd42941280 newargc=15 2024-01-24T17:48:39.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[48667]: starting ceph client 2024-01-24T17:48:39.508 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[48667]: starting fuse 2024-01-24T17:48:40.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:48:40.712 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:48:40.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:48:40.806 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:48:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:48:40.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:40.811 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:48:40.863 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:48:40.863 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:48:40.864 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2024-01-24T17:48:40.865 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:48:41.037 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.48667.asok 2024-01-24T17:48:41.219 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.48667.asok 2024-01-24T17:48:41.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:41.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.48667.asok status 2024-01-24T17:48:41.454 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:48:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:48:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:48:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:48:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "48667", 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 6538, 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 6538 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:48:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 469665271 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 469665271 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.6538 192.168.0.1:0/469665271", 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/469665271", 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 59, 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 100, 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:48:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:48:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:48:41.457 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:48:41.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:41.470 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:48:41.470 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:48:41.522 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:48:41.523 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:48:41.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:41.523 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:48:41.618 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:48:41.618 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:41.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:41.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.905+0000 7f0d1c200640 1 Processor -- start 2024-01-24T17:48:41.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.906+0000 7f0d1c200640 1 -- start start 2024-01-24T17:48:41.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.907+0000 7f0d1c200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0d14071c00 0x7f0d140cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:41.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.907+0000 7f0d1c200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0d140b3f20 0x7f0d140ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:41.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.907+0000 7f0d1c200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140ccd10 con 0x7f0d14159d00 2024-01-24T17:48:41.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.907+0000 7f0d1c200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d140ce230 con 0x7f0d14071c00 2024-01-24T17:48:41.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.907+0000 7f0d1c200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d140ccff0 con 0x7f0d140b3f20 2024-01-24T17:48:41.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.907+0000 7f0d19800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0d14159d00 0x7f0d140714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:42666/0 (socket says 172.21.15.39:42666) 2024-01-24T17:48:41.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.907+0000 7f0d19800640 1 -- 172.21.15.39:0/3747512728 learned_addr learned my addr 172.21.15.39:0/3747512728 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:41.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.908+0000 7f0d18e00640 1 --2- 172.21.15.39:0/3747512728 >> v2:172.21.15.39:3300/0 conn(0x7f0d14071c00 0x7f0d140cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:41.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.908+0000 7f0d1a200640 1 --2- 172.21.15.39:0/3747512728 >> v2:172.21.15.100:3300/0 conn(0x7f0d140b3f20 0x7f0d140ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:41.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.908+0000 7f0d13e00640 1 -- 172.21.15.39:0/3747512728 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344540644 0 0) 0x7f0d140ccd10 con 0x7f0d14159d00 2024-01-24T17:48:41.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.908+0000 7f0d13e00640 1 -- 172.21.15.39:0/3747512728 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf4005730 con 0x7f0d14159d00 2024-01-24T17:48:41.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.908+0000 7f0d18e00640 1 -- 172.21.15.39:0/3747512728 >> v2:172.21.15.100:3300/0 conn(0x7f0d140b3f20 msgr2=0x7f0d140ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:41.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.908+0000 7f0d18e00640 1 --2- 172.21.15.39:0/3747512728 >> v2:172.21.15.100:3300/0 conn(0x7f0d140b3f20 0x7f0d140ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:41.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.908+0000 7f0d18e00640 1 -- 172.21.15.39:0/3747512728 >> v1:172.21.15.39:6789/0 conn(0x7f0d14159d00 legacy=0x7f0d140714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:41.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.908+0000 7f0d18e00640 1 -- 172.21.15.39:0/3747512728 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140ad720 con 0x7f0d14071c00 2024-01-24T17:48:41.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.909+0000 7f0d18e00640 1 --2- 172.21.15.39:0/3747512728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14071c00 0x7f0d140cdcf0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f0d1406fea0 tx=0x7f0cfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121e3f5ab729f3e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:41.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d13e00640 1 -- 172.21.15.39:0/3747512728 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc031830 con 0x7f0d14071c00 2024-01-24T17:48:41.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d13e00640 1 -- 172.21.15.39:0/3747512728 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc031990 con 0x7f0d14071c00 2024-01-24T17:48:41.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d13e00640 1 -- 172.21.15.39:0/3747512728 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc031d20 con 0x7f0d14071c00 2024-01-24T17:48:41.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d1c200640 1 -- 172.21.15.39:0/3747512728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14071c00 msgr2=0x7f0d140cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:41.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3747512728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14071c00 0x7f0d140cdcf0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f0d1406fea0 tx=0x7f0cfc02b8d0 comp rx=0 tx=0).stop 2024-01-24T17:48:41.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d1c200640 1 -- 172.21.15.39:0/3747512728 shutdown_connections 2024-01-24T17:48:41.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3747512728 >> v2:172.21.15.100:3300/0 conn(0x7f0d140b3f20 0x7f0d140ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:41.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3747512728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14071c00 0x7f0d140cdcf0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:41.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.910+0000 7f0d1c200640 1 -- 172.21.15.39:0/3747512728 >> 172.21.15.39:0/3747512728 conn(0x7f0d1406bf20 msgr2=0x7f0d1406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:41.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.911+0000 7f0d1c200640 1 -- 172.21.15.39:0/3747512728 shutdown_connections 2024-01-24T17:48:41.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.911+0000 7f0d1c200640 1 -- 172.21.15.39:0/3747512728 wait complete. 2024-01-24T17:48:41.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.912+0000 7f0d1c200640 1 Processor -- start 2024-01-24T17:48:41.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.912+0000 7f0d1c200640 1 -- start start 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.913+0000 7f0d1c200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d14071c00 0x7f0d1414c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.913+0000 7f0d1c200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b3f20 0x7f0d1415b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.913+0000 7f0d1c200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14159d00 0x7f0d140c3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.913+0000 7f0d1c200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140c7dd0 con 0x7f0d14159d00 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.913+0000 7f0d1c200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140c6d30 con 0x7f0d14071c00 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.913+0000 7f0d1c200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140c7010 con 0x7f0d140b3f20 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14159d00 0x7f0d140c3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14159d00 0x7f0d140c3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48800/0 (socket says 172.21.15.39:48800) 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 -- 172.21.15.39:0/3729740985 learned_addr learned my addr 172.21.15.39:0/3729740985 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 -- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b3f20 msgr2=0x7f0d1415b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b3f20 0x7f0d1415b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 -- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d14071c00 msgr2=0x7f0d1414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:41.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d14071c00 0x7f0d1414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:41.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 -- 172.21.15.39:0/3729740985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc009400 con 0x7f0d14159d00 2024-01-24T17:48:41.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.914+0000 7f0d1a200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14159d00 0x7f0d140c3880 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00efc0 tx=0x7f0d0c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:41.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.915+0000 7f0d11600640 1 -- 172.21.15.39:0/3729740985 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c014480 con 0x7f0d14159d00 2024-01-24T17:48:41.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.915+0000 7f0d11600640 1 -- 172.21.15.39:0/3729740985 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c00f040 con 0x7f0d14159d00 2024-01-24T17:48:41.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.915+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d140c72f0 con 0x7f0d14159d00 2024-01-24T17:48:41.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.915+0000 7f0d11600640 1 -- 172.21.15.39:0/3729740985 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c014820 con 0x7f0d14159d00 2024-01-24T17:48:41.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.915+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d140c78b0 con 0x7f0d14159d00 2024-01-24T17:48:41.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.916+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d140c7aa0 con 0x7f0d14159d00 2024-01-24T17:48:41.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.917+0000 7f0d11600640 1 -- 172.21.15.39:0/3729740985 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0d0c014980 con 0x7f0d14159d00 2024-01-24T17:48:41.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.918+0000 7f0d11600640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cec06a430 0x7f0cec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:41.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.918+0000 7f0d11600640 1 -- 172.21.15.39:0/3729740985 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d0c0895c0 con 0x7f0d14159d00 2024-01-24T17:48:41.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.918+0000 7f0d19800640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cec06a430 0x7f0cec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:41.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.920+0000 7f0d19800640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cec06a430 0x7f0cec06c8f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0d04001570 tx=0x7f0d04007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:41.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:41.923+0000 7f0d11600640 1 -- 172.21.15.39:0/3729740985 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0d0c054b10 con 0x7f0d14159d00 2024-01-24T17:48:42.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.119+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d1404dff0 con 0x7f0d14159d00 2024-01-24T17:48:42.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.121+0000 7f0d11600640 1 -- 172.21.15.39:0/3729740985 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16042 (secure 0 0 0) 0x7f0d0c058130 con 0x7f0d14159d00 2024-01-24T17:48:42.122 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:42.122 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":100,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:48:27.885313+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cec06a430 msgr2=0x7f0cec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cec06a430 0x7f0cec06c8f0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0d04001570 tx=0x7f0d04007930 comp rx=0 tx=0).stop 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14159d00 msgr2=0x7f0d140c3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14159d00 0x7f0d140c3880 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00efc0 tx=0x7f0d0c00c490 comp rx=0 tx=0).stop 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 shutdown_connections 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0cec06a430 0x7f0cec06c8f0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d14159d00 0x7f0d140c3880 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0d140b3f20 0x7f0d1415b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 --2- 172.21.15.39:0/3729740985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0d14071c00 0x7f0d1414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 >> 172.21.15.39:0/3729740985 conn(0x7f0d1406bf20 msgr2=0x7f0d14157890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:42.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 shutdown_connections 2024-01-24T17:48:42.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.124+0000 7f0d1c200640 1 -- 172.21.15.39:0/3729740985 wait complete. 2024-01-24T17:48:42.144 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:48:42.145 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:48:42.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.370+0000 7f0741000640 1 Processor -- start 2024-01-24T17:48:42.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.370+0000 7f0741000640 1 -- start start 2024-01-24T17:48:42.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.371+0000 7f0741000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f073c0d1520 0x7f073c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.371+0000 7f0741000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c0d0610 0x7f073c0b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.371+0000 7f0741000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0d0b40 con 0x7f073c15c920 2024-01-24T17:48:42.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.371+0000 7f0741000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073c0b1d00 con 0x7f073c0d1520 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.371+0000 7f0741000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c0b1fe0 con 0x7f073c0d0610 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.372+0000 7f0733e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f073c0d1520 0x7f073c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.372+0000 7f0733e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f073c0d1520 0x7f073c0cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48804/0 (socket says 172.21.15.39:48804) 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.372+0000 7f0733e00640 1 -- 172.21.15.39:0/4172649341 learned_addr learned my addr 172.21.15.39:0/4172649341 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.372+0000 7f073be00640 1 --2- 172.21.15.39:0/4172649341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c0d0610 0x7f073c0b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.372+0000 7f0733e00640 1 -- 172.21.15.39:0/4172649341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c0d0610 msgr2=0x7f073c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.372+0000 7f0733e00640 1 --2- 172.21.15.39:0/4172649341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c0d0610 0x7f073c0b2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.372+0000 7f0733e00640 1 -- 172.21.15.39:0/4172649341 >> v1:172.21.15.100:6789/0 conn(0x7f073c15c920 legacy=0x7f073c0d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:42.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.372+0000 7f0733e00640 1 -- 172.21.15.39:0/4172649341 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c0b22c0 con 0x7f073c0d1520 2024-01-24T17:48:42.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.373+0000 7f0733e00640 1 --2- 172.21.15.39:0/4172649341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d1520 0x7f073c0cf870 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f073c08cb80 tx=0x7f072402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286d61aa2898e5e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:42.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.374+0000 7f073aa00640 1 -- 172.21.15.39:0/4172649341 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724030610 con 0x7f073c0d1520 2024-01-24T17:48:42.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.374+0000 7f073aa00640 1 -- 172.21.15.39:0/4172649341 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0724030770 con 0x7f073c0d1520 2024-01-24T17:48:42.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.374+0000 7f073aa00640 1 -- 172.21.15.39:0/4172649341 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724030b00 con 0x7f073c0d1520 2024-01-24T17:48:42.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.374+0000 7f0741000640 1 -- 172.21.15.39:0/4172649341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d1520 msgr2=0x7f073c0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.374+0000 7f0741000640 1 --2- 172.21.15.39:0/4172649341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d1520 0x7f073c0cf870 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f073c08cb80 tx=0x7f072402b960 comp rx=0 tx=0).stop 2024-01-24T17:48:42.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.375+0000 7f0741000640 1 -- 172.21.15.39:0/4172649341 shutdown_connections 2024-01-24T17:48:42.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.375+0000 7f0741000640 1 --2- 172.21.15.39:0/4172649341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d1520 0x7f073c0cf870 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.375+0000 7f0741000640 1 --2- 172.21.15.39:0/4172649341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c0d0610 0x7f073c0b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.375+0000 7f0741000640 1 -- 172.21.15.39:0/4172649341 >> 172.21.15.39:0/4172649341 conn(0x7f073c0d2830 msgr2=0x7f073c0d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:42.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.375+0000 7f0741000640 1 -- 172.21.15.39:0/4172649341 shutdown_connections 2024-01-24T17:48:42.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.375+0000 7f0741000640 1 -- 172.21.15.39:0/4172649341 wait complete. 2024-01-24T17:48:42.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.377+0000 7f0741000640 1 Processor -- start 2024-01-24T17:48:42.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.377+0000 7f0741000640 1 -- start start 2024-01-24T17:48:42.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f0741000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d0610 0x7f073c15b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f0741000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f073c0d1520 0x7f073c15e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f0741000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c15c920 0x7f073c1f2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f0741000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c15f500 con 0x7f073c0d0610 2024-01-24T17:48:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f0741000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c15e650 con 0x7f073c0d1520 2024-01-24T17:48:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f0741000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c15e8f0 con 0x7f073c15c920 2024-01-24T17:48:42.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f073b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d0610 0x7f073c15b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f073b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d0610 0x7f073c15b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48814/0 (socket says 172.21.15.39:48814) 2024-01-24T17:48:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f073b400640 1 -- 172.21.15.39:0/3486467445 learned_addr learned my addr 172.21.15.39:0/3486467445 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f0733e00640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f073c0d1520 0x7f073c15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.378+0000 7f073be00640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c15c920 0x7f073c1f2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f073b400640 1 -- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c15c920 msgr2=0x7f073c1f2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f073b400640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c15c920 0x7f073c1f2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f073b400640 1 -- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f073c0d1520 msgr2=0x7f073c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f073b400640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f073c0d1520 0x7f073c15e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f073b400640 1 -- 172.21.15.39:0/3486467445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07240094d0 con 0x7f073c0d0610 2024-01-24T17:48:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f073b400640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d0610 0x7f073c15b170 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f072c00b3f0 tx=0x7f072c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f0733400640 1 -- 172.21.15.39:0/3486467445 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c011830 con 0x7f073c0d0610 2024-01-24T17:48:42.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f0733400640 1 -- 172.21.15.39:0/3486467445 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072c011990 con 0x7f073c0d0610 2024-01-24T17:48:42.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f0733400640 1 -- 172.21.15.39:0/3486467445 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c011d40 con 0x7f073c0d0610 2024-01-24T17:48:42.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.379+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c15ebd0 con 0x7f073c0d0610 2024-01-24T17:48:42.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.380+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c15f190 con 0x7f073c0d0610 2024-01-24T17:48:42.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.381+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c04e840 con 0x7f073c0d0610 2024-01-24T17:48:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.382+0000 7f0733400640 1 -- 172.21.15.39:0/3486467445 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f072c01a400 con 0x7f073c0d0610 2024-01-24T17:48:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.382+0000 7f0733400640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f071406a430 0x7f071406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.383+0000 7f0733400640 1 -- 172.21.15.39:0/3486467445 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f072c088780 con 0x7f073c0d0610 2024-01-24T17:48:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.383+0000 7f0733e00640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f071406a430 0x7f071406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.384+0000 7f0733e00640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f071406a430 0x7f071406c8f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0724036000 tx=0x7f0724008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:42.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.388+0000 7f0733400640 1 -- 172.21.15.39:0/3486467445 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f072c053d50 con 0x7f073c0d0610 2024-01-24T17:48:42.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.589+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f073c0d2040 con 0x7f073c0d0610 2024-01-24T17:48:42.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.590+0000 7f0733400640 1 -- 172.21.15.39:0/3486467445 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f072c057370 con 0x7f073c0d0610 2024-01-24T17:48:42.590 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f071406a430 msgr2=0x7f071406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f071406a430 0x7f071406c8f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0724036000 tx=0x7f0724008030 comp rx=0 tx=0).stop 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d0610 msgr2=0x7f073c15b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d0610 0x7f073c15b170 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f072c00b3f0 tx=0x7f072c00b8b0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 shutdown_connections 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f073c15c920 0x7f073c1f2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f073c0d1520 0x7f073c15e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f071406a430 0x7f071406c8f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 --2- 172.21.15.39:0/3486467445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f073c0d0610 0x7f073c15b170 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 >> 172.21.15.39:0/3486467445 conn(0x7f073c0d2830 msgr2=0x7f073c0d0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 shutdown_connections 2024-01-24T17:48:42.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.593+0000 7f0741000640 1 -- 172.21.15.39:0/3486467445 wait complete. 2024-01-24T17:48:42.614 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:48:42.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.843+0000 7fe047e00640 1 Processor -- start 2024-01-24T17:48:42.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.843+0000 7fe047e00640 1 -- start start 2024-01-24T17:48:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.844+0000 7fe047e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe04806acd0 0x7fe04806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.844+0000 7fe047e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04806bde0 con 0x7fe0480ce660 2024-01-24T17:48:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.844+0000 7fe047e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04806b1d0 con 0x7fe0480b4310 2024-01-24T17:48:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.844+0000 7fe047e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0480afaf0 con 0x7fe04806acd0 2024-01-24T17:48:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.845+0000 7fe046000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe0480ce660 0x7fe0480af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:42680/0 (socket says 172.21.15.39:42680) 2024-01-24T17:48:42.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.845+0000 7fe046000640 1 -- 172.21.15.39:0/3820336122 learned_addr learned my addr 172.21.15.39:0/3820336122 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.845+0000 7fe047400640 1 --2- 172.21.15.39:0/3820336122 >> v2:172.21.15.100:3300/0 conn(0x7fe04806acd0 0x7fe04806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.845+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982194159 0 0) 0x7fe04806bde0 con 0x7fe0480ce660 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.845+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe028003520 con 0x7fe0480ce660 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2582551341 0 0) 0x7fe028003520 con 0x7fe0480ce660 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 >> v2:172.21.15.100:3300/0 conn(0x7fe04806acd0 msgr2=0x7fe04806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 --2- 172.21.15.39:0/3820336122 >> v2:172.21.15.100:3300/0 conn(0x7fe04806acd0 0x7fe04806c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 >> v1:172.21.15.100:6789/0 conn(0x7fe0480b4310 legacy=0x7fe0480b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04806b4b0 con 0x7fe0480ce660 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fe030001670 con 0x7fe0480ce660 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 >> v1:172.21.15.39:6789/0 conn(0x7fe0480ce660 legacy=0x7fe0480af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0280039c0 0x7fe028003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0280047c0 0x7fe028004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe028005180 0x7fe02800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480afaf0 con 0x7fe0280039c0 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02800c080 con 0x7fe0280047c0 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02800c240 con 0x7fe028005180 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe046a00640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0280039c0 0x7fe028003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.846+0000 7fe046000640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0280047c0 0x7fe028004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe047400640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe028005180 0x7fe02800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe046a00640 1 -- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe028005180 msgr2=0x7fe02800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe046a00640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe028005180 0x7fe02800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe046a00640 1 -- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0280047c0 msgr2=0x7fe028004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe046a00640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0280047c0 0x7fe028004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe046a00640 1 -- 172.21.15.39:0/3820336122 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028003520 con 0x7fe0280039c0 2024-01-24T17:48:42.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe047400640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe028005180 0x7fe02800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:42.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe046000640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0280047c0 0x7fe028004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:48:42.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.847+0000 7fe047400640 1 -- 172.21.15.39:0/3820336122 reap_dead start 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.848+0000 7fe046a00640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0280039c0 0x7fe028003de0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe038001e70 tx=0x7fe0280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bceaa73caf6ffb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.848+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe038003bc0 con 0x7fe0280039c0 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.848+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe038003d20 con 0x7fe0280039c0 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.848+0000 7fe045600640 1 -- 172.21.15.39:0/3820336122 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe038005310 con 0x7fe0280039c0 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.849+0000 7fe047e00640 1 -- 172.21.15.39:0/3820336122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0280039c0 msgr2=0x7fe028003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.849+0000 7fe047e00640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0280039c0 0x7fe028003de0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fe038001e70 tx=0x7fe0280057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.849+0000 7fe047e00640 1 -- 172.21.15.39:0/3820336122 shutdown_connections 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.849+0000 7fe047e00640 1 --2- 172.21.15.39:0/3820336122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0280039c0 0x7fe028003de0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.849+0000 7fe047e00640 1 -- 172.21.15.39:0/3820336122 >> 172.21.15.39:0/3820336122 conn(0x7fe0480d2890 msgr2=0x7fe0480d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.849+0000 7fe047e00640 1 -- 172.21.15.39:0/3820336122 shutdown_connections 2024-01-24T17:48:42.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.849+0000 7fe047e00640 1 -- 172.21.15.39:0/3820336122 wait complete. 2024-01-24T17:48:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.851+0000 7fe047e00640 1 Processor -- start 2024-01-24T17:48:42.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.851+0000 7fe047e00640 1 -- start start 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe047e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0280039c0 0x7fe04815a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe047e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0480c6310 0x7fe0480c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe047e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480c9e10 0x7fe04815f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe047e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04815a690 con 0x7fe0480c6310 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe047e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480ca320 con 0x7fe0480c9e10 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe047e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480ca600 con 0x7fe0280039c0 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe046000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0480c6310 0x7fe0480c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe046000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0480c6310 0x7fe0480c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48838/0 (socket says 172.21.15.39:48838) 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.852+0000 7fe046000640 1 -- 172.21.15.39:0/49507955 learned_addr learned my addr 172.21.15.39:0/49507955 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe047400640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480c9e10 0x7fe04815f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe046000640 1 -- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0280039c0 msgr2=0x7fe04815a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe046a00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0280039c0 0x7fe04815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe046000640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0280039c0 0x7fe04815a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe046000640 1 -- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480c9e10 msgr2=0x7fe04815f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe046000640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480c9e10 0x7fe04815f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe046000640 1 -- 172.21.15.39:0/49507955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0480ca8e0 con 0x7fe0480c6310 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe046000640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0480c6310 0x7fe0480c98d0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe03000e800 tx=0x7fe03000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe03f400640 1 -- 172.21.15.39:0/49507955 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03000c6c0 con 0x7fe0480c6310 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe03f400640 1 -- 172.21.15.39:0/49507955 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03000c820 con 0x7fe0480c6310 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe03f400640 1 -- 172.21.15.39:0/49507955 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03000cb90 con 0x7fe0480c6310 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0480cabc0 con 0x7fe0480c6310 2024-01-24T17:48:42.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.853+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0480c6770 con 0x7fe0480c6310 2024-01-24T17:48:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.855+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0480c6900 con 0x7fe0480c6310 2024-01-24T17:48:42.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.855+0000 7fe03f400640 1 -- 172.21.15.39:0/49507955 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe03000ccf0 con 0x7fe0480c6310 2024-01-24T17:48:42.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.856+0000 7fe03f400640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe02006a460 0x7fe02006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:42.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.856+0000 7fe03f400640 1 -- 172.21.15.39:0/49507955 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe030088860 con 0x7fe0480c6310 2024-01-24T17:48:42.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.857+0000 7fe046a00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe02006a460 0x7fe02006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:42.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.858+0000 7fe046a00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe02006a460 0x7fe02006c920 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe038034000 tx=0x7fe038009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:42.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:42.861+0000 7fe03f400640 1 -- 172.21.15.39:0/49507955 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe030057850 con 0x7fe0480c6310 2024-01-24T17:48:43.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.060+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe04804e840 con 0x7fe0480c6310 2024-01-24T17:48:43.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.061+0000 7fe03f400640 1 -- 172.21.15.39:0/49507955 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe030001b30 con 0x7fe0480c6310 2024-01-24T17:48:43.062 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:48:43.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.063+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe02006a460 msgr2=0x7fe02006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.063+0000 7fe047e00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe02006a460 0x7fe02006c920 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe038034000 tx=0x7fe038009030 comp rx=0 tx=0).stop 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0480c6310 msgr2=0x7fe0480c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0480c6310 0x7fe0480c98d0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe03000e800 tx=0x7fe03000ecc0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 shutdown_connections 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480c9e10 0x7fe04815f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe02006a460 0x7fe02006c920 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0480c6310 0x7fe0480c98d0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 --2- 172.21.15.39:0/49507955 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe0280039c0 0x7fe04815a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 >> 172.21.15.39:0/49507955 conn(0x7fe0480d2890 msgr2=0x7fe0480b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 shutdown_connections 2024-01-24T17:48:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.064+0000 7fe047e00640 1 -- 172.21.15.39:0/49507955 wait complete. 2024-01-24T17:48:43.084 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:48:43.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.310+0000 7f429cc00640 1 Processor -- start 2024-01-24T17:48:43.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.310+0000 7f429cc00640 1 -- start start 2024-01-24T17:48:43.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f429cc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f429815d450 0x7f42980c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f429cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980cbf90 0x7f429815d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f429cc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f42980ce350 0x7f42980ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f429cc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42980ced20 con 0x7f429815d450 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f429cc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42980c9290 con 0x7f42980ce350 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f429cc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42980c9570 con 0x7f42980cbf90 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f4297400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f429815d450 0x7f42980c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f4297400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f429815d450 0x7f42980c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32982/0 (socket says 172.21.15.39:32982) 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.312+0000 7f4297400640 1 -- 172.21.15.39:0/265013524 learned_addr learned my addr 172.21.15.39:0/265013524 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.313+0000 7f428e800640 1 --2- 172.21.15.39:0/265013524 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980cbf90 0x7f429815d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.313+0000 7f4297e00640 1 --2- 172.21.15.39:0/265013524 >> v2:172.21.15.100:3300/0 conn(0x7f42980ce350 0x7f42980ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.313+0000 7f4297400640 1 -- 172.21.15.39:0/265013524 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980cbf90 msgr2=0x7f429815d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.313+0000 7f4297400640 1 --2- 172.21.15.39:0/265013524 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980cbf90 0x7f429815d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.313+0000 7f4297400640 1 -- 172.21.15.39:0/265013524 >> v2:172.21.15.100:3300/0 conn(0x7f42980ce350 msgr2=0x7f42980ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.313+0000 7f4297400640 1 --2- 172.21.15.39:0/265013524 >> v2:172.21.15.100:3300/0 conn(0x7f42980ce350 0x7f42980ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.313+0000 7f4297400640 1 -- 172.21.15.39:0/265013524 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429806fed0 con 0x7f429815d450 2024-01-24T17:48:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.314+0000 7f4297400640 1 --2- 172.21.15.39:0/265013524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429815d450 0x7f42980c8bd0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f429806dfc0 tx=0x7f428802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0830f7d76f89a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.315+0000 7f4296a00640 1 -- 172.21.15.39:0/265013524 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288031870 con 0x7f429815d450 2024-01-24T17:48:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.315+0000 7f4296a00640 1 -- 172.21.15.39:0/265013524 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42880319d0 con 0x7f429815d450 2024-01-24T17:48:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.315+0000 7f4296a00640 1 -- 172.21.15.39:0/265013524 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288031d60 con 0x7f429815d450 2024-01-24T17:48:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.315+0000 7f429cc00640 1 -- 172.21.15.39:0/265013524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429815d450 msgr2=0x7f42980c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.315+0000 7f429cc00640 1 --2- 172.21.15.39:0/265013524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429815d450 0x7f42980c8bd0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f429806dfc0 tx=0x7f428802b910 comp rx=0 tx=0).stop 2024-01-24T17:48:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.316+0000 7f429cc00640 1 -- 172.21.15.39:0/265013524 shutdown_connections 2024-01-24T17:48:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.316+0000 7f429cc00640 1 --2- 172.21.15.39:0/265013524 >> v2:172.21.15.100:3300/0 conn(0x7f42980ce350 0x7f42980ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.316+0000 7f429cc00640 1 --2- 172.21.15.39:0/265013524 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980cbf90 0x7f429815d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.316+0000 7f429cc00640 1 --2- 172.21.15.39:0/265013524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f429815d450 0x7f42980c8bd0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.316+0000 7f429cc00640 1 -- 172.21.15.39:0/265013524 >> 172.21.15.39:0/265013524 conn(0x7f429806a200 msgr2=0x7f42980affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.316+0000 7f429cc00640 1 -- 172.21.15.39:0/265013524 shutdown_connections 2024-01-24T17:48:43.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.316+0000 7f429cc00640 1 -- 172.21.15.39:0/265013524 wait complete. 2024-01-24T17:48:43.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.318+0000 7f429cc00640 1 Processor -- start 2024-01-24T17:48:43.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.318+0000 7f429cc00640 1 -- start start 2024-01-24T17:48:43.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f429cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42980cbf90 0x7f42980c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f429cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980ce350 0x7f429815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f429cc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f429815d450 0x7f4298153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f429cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42981f2c00 con 0x7f42980cbf90 2024-01-24T17:48:43.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f429cc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429815f730 con 0x7f429815d450 2024-01-24T17:48:43.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f429cc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42981f2310 con 0x7f42980ce350 2024-01-24T17:48:43.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f4297400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42980cbf90 0x7f42980c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f4297400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42980cbf90 0x7f42980c8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32994/0 (socket says 172.21.15.39:32994) 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f4297400640 1 -- 172.21.15.39:0/4029543964 learned_addr learned my addr 172.21.15.39:0/4029543964 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f428e800640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980ce350 0x7f429815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.319+0000 7f4297e00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f429815d450 0x7f4298153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f4297400640 1 -- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980ce350 msgr2=0x7f429815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f4297400640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980ce350 0x7f429815ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f4297400640 1 -- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f429815d450 msgr2=0x7f4298153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f4297400640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f429815d450 0x7f4298153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f4297400640 1 -- 172.21.15.39:0/4029543964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42880094d0 con 0x7f42980cbf90 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f4297e00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f429815d450 0x7f4298153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f4297400640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42980cbf90 0x7f42980c8100 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f428802b950 tx=0x7f42880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f428fe00640 1 -- 172.21.15.39:0/4029543964 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428803ac60 con 0x7f42980cbf90 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f428fe00640 1 -- 172.21.15.39:0/4029543964 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428803adc0 con 0x7f42980cbf90 2024-01-24T17:48:43.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f428fe00640 1 -- 172.21.15.39:0/4029543964 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42880445f0 con 0x7f42980cbf90 2024-01-24T17:48:43.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429815f490 con 0x7f42980cbf90 2024-01-24T17:48:43.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.320+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42981f25b0 con 0x7f42980cbf90 2024-01-24T17:48:43.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.322+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42980738c0 con 0x7f42980cbf90 2024-01-24T17:48:43.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.322+0000 7f428fe00640 1 -- 172.21.15.39:0/4029543964 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f428804d450 con 0x7f42980cbf90 2024-01-24T17:48:43.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.323+0000 7f428fe00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f427006a430 0x7f427006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.323+0000 7f428fe00640 1 -- 172.21.15.39:0/4029543964 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f428802d040 con 0x7f42980cbf90 2024-01-24T17:48:43.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.324+0000 7f428e800640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f427006a430 0x7f427006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.325+0000 7f428e800640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f427006a430 0x7f427006c8f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4280006fd0 tx=0x7f4280008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:43.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.327+0000 7f428fe00640 1 -- 172.21.15.39:0/4029543964 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4288076580 con 0x7f42980cbf90 2024-01-24T17:48:43.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.519+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f429804e840 con 0x7f42980cbf90 2024-01-24T17:48:43.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.519+0000 7f428fe00640 1 -- 172.21.15.39:0/4029543964 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7f42880ac0a0 con 0x7f42980cbf90 2024-01-24T17:48:43.520 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:48:43.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f427006a430 msgr2=0x7f427006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f427006a430 0x7f427006c8f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4280006fd0 tx=0x7f4280008040 comp rx=0 tx=0).stop 2024-01-24T17:48:43.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42980cbf90 msgr2=0x7f42980c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42980cbf90 0x7f42980c8100 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f428802b950 tx=0x7f42880046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 shutdown_connections 2024-01-24T17:48:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f429815d450 0x7f4298153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f42980ce350 0x7f429815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f427006a430 0x7f427006c8f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 --2- 172.21.15.39:0/4029543964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42980cbf90 0x7f42980c8100 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.522+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 >> 172.21.15.39:0/4029543964 conn(0x7f429806a200 msgr2=0x7f42980cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.523+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 shutdown_connections 2024-01-24T17:48:43.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.523+0000 7f429cc00640 1 -- 172.21.15.39:0/4029543964 wait complete. 2024-01-24T17:48:43.541 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:48:43.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.753+0000 7f1963200640 1 Processor -- start 2024-01-24T17:48:43.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.753+0000 7f1963200640 1 -- start start 2024-01-24T17:48:43.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.755+0000 7f1963200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19640cadc0 0x7f196415a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.755+0000 7f1963200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f196406bd40 0x7f19640ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.755+0000 7f1963200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640cac10 con 0x7f19640cb1a0 2024-01-24T17:48:43.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.755+0000 7f1963200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196415b040 con 0x7f196406bd40 2024-01-24T17:48:43.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.755+0000 7f1963200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196406c1a0 con 0x7f19640cadc0 2024-01-24T17:48:43.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.756+0000 7f1961400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f196406bd40 0x7f19640ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.756+0000 7f1961e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19640cadc0 0x7f196415a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.756+0000 7f1961400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f196406bd40 0x7f19640ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.39:58566/0 (socket says 172.21.15.39:58566) 2024-01-24T17:48:43.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.756+0000 7f1961400640 1 -- 172.21.15.39:0/1679914219 learned_addr learned my addr 172.21.15.39:0/1679914219 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:43.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.756+0000 7f1961e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19640cadc0 0x7f196415a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:40718/0 (socket says 172.21.15.39:40718) 2024-01-24T17:48:43.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.757+0000 7f1961400640 1 -- 172.21.15.39:0/1679914219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19640cadc0 msgr2=0x7f196415a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:43.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.757+0000 7f1961400640 1 --2- 172.21.15.39:0/1679914219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19640cadc0 0x7f196415a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.757+0000 7f1961400640 1 -- 172.21.15.39:0/1679914219 >> v1:172.21.15.100:6789/0 conn(0x7f19640cb1a0 legacy=0x7f196415b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.757+0000 7f1961400640 1 -- 172.21.15.39:0/1679914219 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196415bbe0 con 0x7f196406bd40 2024-01-24T17:48:43.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.757+0000 7f1961e00640 1 --2- 172.21.15.39:0/1679914219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19640cadc0 0x7f196415a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:48:43.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.759+0000 7f1961400640 1 --2- 172.21.15.39:0/1679914219 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f196406bd40 0x7f19640ca6d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f194c009e50 tx=0x7f194c02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4e00de72fe4e644 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:43.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.759+0000 7f1960a00640 1 -- 172.21.15.39:0/1679914219 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c02ca80 con 0x7f196406bd40 2024-01-24T17:48:43.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.759+0000 7f1960a00640 1 -- 172.21.15.39:0/1679914219 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c02cbe0 con 0x7f196406bd40 2024-01-24T17:48:43.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.759+0000 7f1960a00640 1 -- 172.21.15.39:0/1679914219 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c0343f0 con 0x7f196406bd40 2024-01-24T17:48:43.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.760+0000 7f1963200640 1 -- 172.21.15.39:0/1679914219 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f196406bd40 msgr2=0x7f19640ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.760+0000 7f1963200640 1 --2- 172.21.15.39:0/1679914219 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f196406bd40 0x7f19640ca6d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f194c009e50 tx=0x7f194c02bee0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.760+0000 7f1963200640 1 -- 172.21.15.39:0/1679914219 shutdown_connections 2024-01-24T17:48:43.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.760+0000 7f1963200640 1 --2- 172.21.15.39:0/1679914219 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f196406bd40 0x7f19640ca6d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.760+0000 7f1963200640 1 --2- 172.21.15.39:0/1679914219 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f19640cadc0 0x7f196415a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.760+0000 7f1963200640 1 -- 172.21.15.39:0/1679914219 >> 172.21.15.39:0/1679914219 conn(0x7f196406a200 msgr2=0x7f19640aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:43.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.760+0000 7f1963200640 1 -- 172.21.15.39:0/1679914219 shutdown_connections 2024-01-24T17:48:43.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.760+0000 7f1963200640 1 -- 172.21.15.39:0/1679914219 wait complete. 2024-01-24T17:48:43.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.762+0000 7f1963200640 1 Processor -- start 2024-01-24T17:48:43.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.762+0000 7f1963200640 1 -- start start 2024-01-24T17:48:43.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1963200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f196406bd40 0x7f19641f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1963200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19640cadc0 0x7f19640ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1963200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19640cb1a0 0x7f19640c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1963200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19640c77e0 con 0x7f19640cb1a0 2024-01-24T17:48:43.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1963200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19640ac1c0 con 0x7f19640cadc0 2024-01-24T17:48:43.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1963200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19640ac4a0 con 0x7f196406bd40 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1962800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19640cb1a0 0x7f19640c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1962800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19640cb1a0 0x7f19640c7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32996/0 (socket says 172.21.15.39:32996) 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1962800640 1 -- 172.21.15.39:0/325256567 learned_addr learned my addr 172.21.15.39:0/325256567 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1961e00640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f196406bd40 0x7f19641f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.763+0000 7f1961400640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19640cadc0 0x7f19640ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f1962800640 1 -- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f196406bd40 msgr2=0x7f19641f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f1962800640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f196406bd40 0x7f19641f18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f1962800640 1 -- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19640cadc0 msgr2=0x7f19640ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f1962800640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19640cadc0 0x7f19640ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f1962800640 1 -- 172.21.15.39:0/325256567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19540093a0 con 0x7f19640cb1a0 2024-01-24T17:48:43.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f1962800640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19640cb1a0 0x7f19640c7c00 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f195c001eb0 tx=0x7f195c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:43.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f195a000640 1 -- 172.21.15.39:0/325256567 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c0052d0 con 0x7f19640cb1a0 2024-01-24T17:48:43.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f195a000640 1 -- 172.21.15.39:0/325256567 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c005430 con 0x7f19640cb1a0 2024-01-24T17:48:43.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c009b30 con 0x7f19640cb1a0 2024-01-24T17:48:43.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f195a000640 1 -- 172.21.15.39:0/325256567 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c003810 con 0x7f19640cb1a0 2024-01-24T17:48:43.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.764+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19640ac780 con 0x7f19640cb1a0 2024-01-24T17:48:43.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.766+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19640c7920 con 0x7f19640cb1a0 2024-01-24T17:48:43.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.766+0000 7f195a000640 1 -- 172.21.15.39:0/325256567 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f195c003970 con 0x7f19640cb1a0 2024-01-24T17:48:43.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.767+0000 7f195a000640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f193c06a430 0x7f193c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:43.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.767+0000 7f195a000640 1 -- 172.21.15.39:0/325256567 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f195c087f60 con 0x7f19640cb1a0 2024-01-24T17:48:43.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.768+0000 7f1961e00640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f193c06a430 0x7f193c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:43.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.769+0000 7f1961e00640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f193c06a430 0x7f193c06c8f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1954009c50 tx=0x7f1954008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:43.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.771+0000 7f195a000640 1 -- 172.21.15.39:0/325256567 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f19640c7920 con 0x7f19640cb1a0 2024-01-24T17:48:43.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.983+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f19640c7920 con 0x7f19640cb1a0 2024-01-24T17:48:43.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.984+0000 7f195a000640 1 -- 172.21.15.39:0/325256567 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v100) v1 ==== 80+0+0 (secure 0 0 0) 0x7f19640c7920 con 0x7f19640cb1a0 2024-01-24T17:48:43.985 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:48:43.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.986+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f193c06a430 msgr2=0x7f193c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.986+0000 7f1963200640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f193c06a430 0x7f193c06c8f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1954009c50 tx=0x7f1954008040 comp rx=0 tx=0).stop 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.986+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19640cb1a0 msgr2=0x7f19640c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.986+0000 7f1963200640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19640cb1a0 0x7f19640c7c00 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f195c001eb0 tx=0x7f195c00c890 comp rx=0 tx=0).stop 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.987+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 shutdown_connections 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.987+0000 7f1963200640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f193c06a430 0x7f193c06c8f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.987+0000 7f1963200640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19640cb1a0 0x7f19640c7c00 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.987+0000 7f1963200640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f19640cadc0 0x7f19640ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.987+0000 7f1963200640 1 --2- 172.21.15.39:0/325256567 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f196406bd40 0x7f19641f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.987+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 >> 172.21.15.39:0/325256567 conn(0x7f196406a200 msgr2=0x7f196415be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.987+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 shutdown_connections 2024-01-24T17:48:43.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:43.987+0000 7f1963200640 1 -- 172.21.15.39:0/325256567 wait complete. 2024-01-24T17:48:44.008 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:48:44.008 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:44.060 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:44.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.278+0000 7f2197800640 1 Processor -- start 2024-01-24T17:48:44.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.278+0000 7f2197800640 1 -- start start 2024-01-24T17:48:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.279+0000 7f2197800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2190159d00 0x7f21900cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.279+0000 7f2197800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2190071c00 0x7f21900ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.279+0000 7f2197800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21900ccd10 con 0x7f21900b3f20 2024-01-24T17:48:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.279+0000 7f2197800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21900ce230 con 0x7f2190159d00 2024-01-24T17:48:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.279+0000 7f2197800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21900ccff0 con 0x7f2190071c00 2024-01-24T17:48:44.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f218fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f21900b3f20 0x7f21900714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40842/0 (socket says 172.21.15.39:40842) 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f218fe00640 1 -- 172.21.15.39:0/4291308038 learned_addr learned my addr 172.21.15.39:0/4291308038 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f2194e00640 1 --2- 172.21.15.39:0/4291308038 >> v2:172.21.15.39:3300/0 conn(0x7f2190159d00 0x7f21900cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f218f400640 1 -- 172.21.15.39:0/4291308038 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932618634 0 0) 0x7f21900ccd10 con 0x7f21900b3f20 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f218f400640 1 -- 172.21.15.39:0/4291308038 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2170005730 con 0x7f21900b3f20 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f2195800640 1 --2- 172.21.15.39:0/4291308038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2190071c00 0x7f21900ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f2194e00640 1 -- 172.21.15.39:0/4291308038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2190071c00 msgr2=0x7f21900ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f2194e00640 1 --2- 172.21.15.39:0/4291308038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2190071c00 0x7f21900ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.280+0000 7f2194e00640 1 -- 172.21.15.39:0/4291308038 >> v1:172.21.15.39:6789/0 conn(0x7f21900b3f20 legacy=0x7f21900714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.281+0000 7f2194e00640 1 -- 172.21.15.39:0/4291308038 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21900ad720 con 0x7f2190159d00 2024-01-24T17:48:44.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.281+0000 7f2195800640 1 --2- 172.21.15.39:0/4291308038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2190071c00 0x7f21900ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.282+0000 7f2194e00640 1 --2- 172.21.15.39:0/4291308038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f21900cdcf0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f21800097a0 tx=0x7f218002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fd24672ecfe2d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.282+0000 7f218f400640 1 -- 172.21.15.39:0/4291308038 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180031780 con 0x7f2190159d00 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.282+0000 7f218f400640 1 -- 172.21.15.39:0/4291308038 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21800318e0 con 0x7f2190159d00 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.282+0000 7f218f400640 1 -- 172.21.15.39:0/4291308038 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180031c70 con 0x7f2190159d00 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.283+0000 7f2197800640 1 -- 172.21.15.39:0/4291308038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 msgr2=0x7f21900cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.283+0000 7f2197800640 1 --2- 172.21.15.39:0/4291308038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f21900cdcf0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f21800097a0 tx=0x7f218002b820 comp rx=0 tx=0).stop 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.283+0000 7f2197800640 1 -- 172.21.15.39:0/4291308038 shutdown_connections 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.283+0000 7f2197800640 1 --2- 172.21.15.39:0/4291308038 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2190071c00 0x7f21900ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.283+0000 7f2197800640 1 --2- 172.21.15.39:0/4291308038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f21900cdcf0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.283+0000 7f2197800640 1 -- 172.21.15.39:0/4291308038 >> 172.21.15.39:0/4291308038 conn(0x7f219006bf20 msgr2=0x7f219006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.283+0000 7f2197800640 1 -- 172.21.15.39:0/4291308038 shutdown_connections 2024-01-24T17:48:44.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.283+0000 7f2197800640 1 -- 172.21.15.39:0/4291308038 wait complete. 2024-01-24T17:48:44.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.285+0000 7f2197800640 1 Processor -- start 2024-01-24T17:48:44.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.285+0000 7f2197800640 1 -- start start 2024-01-24T17:48:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2197800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2190071c00 0x7f219015c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2197800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21900b3f20 0x7f21901eea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2197800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f219014d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2197800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21900c7350 con 0x7f2190159d00 2024-01-24T17:48:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2197800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21901eef70 con 0x7f2190071c00 2024-01-24T17:48:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2197800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21900c69b0 con 0x7f21900b3f20 2024-01-24T17:48:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2195800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f219014d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2195800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f219014d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33022/0 (socket says 172.21.15.39:33022) 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2195800640 1 -- 172.21.15.39:0/2488562351 learned_addr learned my addr 172.21.15.39:0/2488562351 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f2194e00640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2190071c00 0x7f219015c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.286+0000 7f218fe00640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21900b3f20 0x7f21901eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2195800640 1 -- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21900b3f20 msgr2=0x7f21901eea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2195800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21900b3f20 0x7f21901eea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2195800640 1 -- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2190071c00 msgr2=0x7f219015c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2195800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2190071c00 0x7f219015c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2195800640 1 -- 172.21.15.39:0/2488562351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2180009400 con 0x7f2190159d00 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2194e00640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2190071c00 0x7f219015c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:44.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2195800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f219014d6f0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f218800cef0 tx=0x7f218800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f218cc00640 1 -- 172.21.15.39:0/2488562351 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218800d840 con 0x7f2190159d00 2024-01-24T17:48:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f218cc00640 1 -- 172.21.15.39:0/2488562351 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218800d9c0 con 0x7f2190159d00 2024-01-24T17:48:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f218cc00640 1 -- 172.21.15.39:0/2488562351 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218800dd70 con 0x7f2190159d00 2024-01-24T17:48:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21900c6c90 con 0x7f2190159d00 2024-01-24T17:48:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.287+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219015cc00 con 0x7f2190159d00 2024-01-24T17:48:44.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.288+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21900c7490 con 0x7f2190159d00 2024-01-24T17:48:44.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.289+0000 7f218cc00640 1 -- 172.21.15.39:0/2488562351 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f218801e3b0 con 0x7f2190159d00 2024-01-24T17:48:44.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.290+0000 7f218cc00640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f216806a430 0x7f216806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.290+0000 7f218cc00640 1 -- 172.21.15.39:0/2488562351 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2188089440 con 0x7f2190159d00 2024-01-24T17:48:44.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.291+0000 7f2194e00640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f216806a430 0x7f216806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.292+0000 7f2194e00640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f216806a430 0x7f216806c8f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2180036000 tx=0x7f218003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:44.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.294+0000 7f218cc00640 1 -- 172.21.15.39:0/2488562351 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f21900c7490 con 0x7f2190159d00 2024-01-24T17:48:44.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.490+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21900c7490 con 0x7f2190159d00 2024-01-24T17:48:44.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.493+0000 7f218cc00640 1 -- 172.21.15.39:0/2488562351 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16042 (secure 0 0 0) 0x7f21900c7490 con 0x7f2190159d00 2024-01-24T17:48:44.495 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:44.495 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":100,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:48:27.885313+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:44.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f216806a430 msgr2=0x7f216806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f216806a430 0x7f216806c8f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2180036000 tx=0x7f218003d040 comp rx=0 tx=0).stop 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 msgr2=0x7f219014d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f219014d6f0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f218800cef0 tx=0x7f218800a710 comp rx=0 tx=0).stop 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 shutdown_connections 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f216806a430 0x7f216806c8f0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2190159d00 0x7f219014d6f0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f21900b3f20 0x7f21901eea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 --2- 172.21.15.39:0/2488562351 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2190071c00 0x7f219015c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 >> 172.21.15.39:0/2488562351 conn(0x7f219006bf20 msgr2=0x7f21900c25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.495+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 shutdown_connections 2024-01-24T17:48:44.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.496+0000 7f2197800640 1 -- 172.21.15.39:0/2488562351 wait complete. 2024-01-24T17:48:44.517 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:48:44.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:48:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.735+0000 7f2bbe600640 1 Processor -- start 2024-01-24T17:48:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.736+0000 7f2bbe600640 1 -- start start 2024-01-24T17:48:44.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.737+0000 7f2bbe600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2bb8160dc0 0x7f2bb8167a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.737+0000 7f2bbe600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80d13c0 con 0x7f2bb815ca30 2024-01-24T17:48:44.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.737+0000 7f2bbe600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80d16a0 con 0x7f2bb80d0e90 2024-01-24T17:48:44.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.737+0000 7f2bbe600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80d1980 con 0x7f2bb8160dc0 2024-01-24T17:48:44.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.738+0000 7f2bb7400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2bb815ca30 0x7f2bb80d3260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40854/0 (socket says 172.21.15.39:40854) 2024-01-24T17:48:44.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.738+0000 7f2bb7400640 1 -- 172.21.15.39:0/833532713 learned_addr learned my addr 172.21.15.39:0/833532713 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.738+0000 7f2bb7e00640 1 --2- 172.21.15.39:0/833532713 >> v2:172.21.15.100:3300/0 conn(0x7f2bb8160dc0 0x7f2bb8167a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.738+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164220946 0 0) 0x7f2bb80d13c0 con 0x7f2bb815ca30 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.738+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b98003520 con 0x7f2bb815ca30 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1696549534 0 0) 0x7f2b98003520 con 0x7f2bb815ca30 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 >> v2:172.21.15.100:3300/0 conn(0x7f2bb8160dc0 msgr2=0x7f2bb8167a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 --2- 172.21.15.39:0/833532713 >> v2:172.21.15.100:3300/0 conn(0x7f2bb8160dc0 0x7f2bb8167a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 >> v1:172.21.15.100:6789/0 conn(0x7f2bb80d0e90 legacy=0x7f2bb81606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80af510 con 0x7f2bb815ca30 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2ba8001670 con 0x7f2bb815ca30 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 >> v1:172.21.15.39:6789/0 conn(0x7f2bb815ca30 legacy=0x7f2bb80d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b980039c0 0x7f2b98003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b980047c0 0x7f2b98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2b9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80d1980 con 0x7f2b98005180 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9800c080 con 0x7f2b980047c0 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9800c240 con 0x7f2b980039c0 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb7e00640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2b9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb6a00640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b980047c0 0x7f2b98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.739+0000 7f2bb7e00640 1 -- 172.21.15.39:0/833532713 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b980039c0 msgr2=0x7f2b98003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.740+0000 7f2bb7e00640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2b980039c0 0x7f2b98003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.740+0000 7f2bb7e00640 1 -- 172.21.15.39:0/833532713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b980047c0 msgr2=0x7f2b98004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.740+0000 7f2bb7e00640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b980047c0 0x7f2b98004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.740+0000 7f2bb7e00640 1 -- 172.21.15.39:0/833532713 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b98003520 con 0x7f2b98005180 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.740+0000 7f2bb6a00640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2b980047c0 0x7f2b98004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.740+0000 7f2bb7e00640 1 -- 172.21.15.39:0/833532713 reap_dead start 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.741+0000 7f2bb7e00640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2b9800b980 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80c9930 tx=0x7f2b980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20bb6ededfea8cb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.741+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb002c680 con 0x7f2b98005180 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.741+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb002c7e0 con 0x7f2b98005180 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.741+0000 7f2bb6000640 1 -- 172.21.15.39:0/833532713 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb002cb90 con 0x7f2b98005180 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.741+0000 7f2bbe600640 1 -- 172.21.15.39:0/833532713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 msgr2=0x7f2b9800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.741+0000 7f2bbe600640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2b9800b980 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80c9930 tx=0x7f2b980057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.742+0000 7f2bbe600640 1 -- 172.21.15.39:0/833532713 shutdown_connections 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.742+0000 7f2bbe600640 1 --2- 172.21.15.39:0/833532713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2b9800b980 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.742+0000 7f2bbe600640 1 -- 172.21.15.39:0/833532713 >> 172.21.15.39:0/833532713 conn(0x7f2bb809bc80 msgr2=0x7f2bb80d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.742+0000 7f2bbe600640 1 -- 172.21.15.39:0/833532713 shutdown_connections 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.742+0000 7f2bbe600640 1 -- 172.21.15.39:0/833532713 wait complete. 2024-01-24T17:48:44.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.743+0000 7f2bbe600640 1 Processor -- start 2024-01-24T17:48:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.744+0000 7f2bbe600640 1 -- start start 2024-01-24T17:48:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bbe600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2bb80c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bbe600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb80b43b0 0x7f2bb8156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bbe600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8158e50 0x7f2bb81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bbe600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8159810 con 0x7f2b98005180 2024-01-24T17:48:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bbe600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb81f6570 con 0x7f2bb8158e50 2024-01-24T17:48:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bbe600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80c6bd0 con 0x7f2bb80b43b0 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2bb80c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2bb80c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33040/0 (socket says 172.21.15.39:33040) 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7400640 1 -- 172.21.15.39:0/1407148361 learned_addr learned my addr 172.21.15.39:0/1407148361 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7e00640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8158e50 0x7f2bb81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7400640 1 -- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb80b43b0 msgr2=0x7f2bb8156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb6a00640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb80b43b0 0x7f2bb8156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7400640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb80b43b0 0x7f2bb8156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7400640 1 -- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8158e50 msgr2=0x7f2bb81592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7400640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8158e50 0x7f2bb81592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.745+0000 7f2bb7400640 1 -- 172.21.15.39:0/1407148361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80c6900 con 0x7f2b98005180 2024-01-24T17:48:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.746+0000 7f2bb7400640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2bb80c63c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800c760 tx=0x7f2ba800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:44.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.746+0000 7f2baf400640 1 -- 172.21.15.39:0/1407148361 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80072d0 con 0x7f2b98005180 2024-01-24T17:48:44.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.746+0000 7f2baf400640 1 -- 172.21.15.39:0/1407148361 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba8007430 con 0x7f2b98005180 2024-01-24T17:48:44.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.746+0000 7f2baf400640 1 -- 172.21.15.39:0/1407148361 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80047e0 con 0x7f2b98005180 2024-01-24T17:48:44.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.746+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb81f57b0 con 0x7f2b98005180 2024-01-24T17:48:44.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.746+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb81f5d70 con 0x7f2b98005180 2024-01-24T17:48:44.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.747+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb804dff0 con 0x7f2b98005180 2024-01-24T17:48:44.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.748+0000 7f2baf400640 1 -- 172.21.15.39:0/1407148361 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2ba8004940 con 0x7f2b98005180 2024-01-24T17:48:44.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.749+0000 7f2baf400640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:44.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.749+0000 7f2baf400640 1 -- 172.21.15.39:0/1407148361 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ba8087c30 con 0x7f2b98005180 2024-01-24T17:48:44.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.750+0000 7f2bb6a00640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:44.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.751+0000 7f2bb6a00640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80cab60 tx=0x7f2ba0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:44.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.754+0000 7f2baf400640 1 -- 172.21.15.39:0/1407148361 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2ba80532b0 con 0x7f2b98005180 2024-01-24T17:48:44.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.948+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2bb80b5110 con 0x7f2b98005180 2024-01-24T17:48:44.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.949+0000 7f2baf400640 1 -- 172.21.15.39:0/1407148361 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ba8018070 con 0x7f2b98005180 2024-01-24T17:48:44.951 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:48:44.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.951+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 msgr2=0x7f2b9006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.951+0000 7f2bbe600640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80cab60 tx=0x7f2ba0009040 comp rx=0 tx=0).stop 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.951+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 msgr2=0x7f2bb80c63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.951+0000 7f2bbe600640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2bb80c63c0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800c760 tx=0x7f2ba800cc20 comp rx=0 tx=0).stop 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.952+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 shutdown_connections 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.952+0000 7f2bbe600640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2bb8158e50 0x7f2bb81592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.952+0000 7f2bbe600640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2bb80b43b0 0x7f2bb8156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.952+0000 7f2bbe600640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2b9006a430 0x7f2b9006c8f0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.952+0000 7f2bbe600640 1 --2- 172.21.15.39:0/1407148361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b98005180 0x7f2bb80c63c0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.952+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 >> 172.21.15.39:0/1407148361 conn(0x7f2bb809bc80 msgr2=0x7f2bb815d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.952+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 shutdown_connections 2024-01-24T17:48:44.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:44.952+0000 7f2bbe600640 1 -- 172.21.15.39:0/1407148361 wait complete. 2024-01-24T17:48:44.970 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:48:45.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.184+0000 7f472d400640 1 Processor -- start 2024-01-24T17:48:45.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.184+0000 7f472d400640 1 -- start start 2024-01-24T17:48:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.186+0000 7f472d400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 0x7f47280ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.186+0000 7f472d400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47280ca4c0 con 0x7f47280cb990 2024-01-24T17:48:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.186+0000 7f472d400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472806da00 con 0x7f472815aa60 2024-01-24T17:48:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.186+0000 7f472d400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47280cae90 con 0x7f472806d5d0 2024-01-24T17:48:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.186+0000 7f471fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f47280cb990 0x7f472806cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40856/0 (socket says 172.21.15.39:40856) 2024-01-24T17:48:45.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.186+0000 7f471fe00640 1 -- 172.21.15.39:0/2550750188 learned_addr learned my addr 172.21.15.39:0/2550750188 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4727200640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 0x7f47280ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220450071 0 0) 0x7f47280ca4c0 con 0x7f47280cb990 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4708003520 con 0x7f47280cb990 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2212328304 0 0) 0x7f4708003520 con 0x7f47280cb990 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 msgr2=0x7f47280ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 0x7f47280ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 >> v1:172.21.15.100:6789/0 conn(0x7f472815aa60 legacy=0x7f47280cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472806dc90 con 0x7f47280cb990 2024-01-24T17:48:45.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f4710001670 con 0x7f47280cb990 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 >> v1:172.21.15.39:6789/0 conn(0x7f47280cb990 legacy=0x7f472806cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4708003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 0x7f47080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4708004d00 0x7f4708005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47280cae90 con 0x7f47080039a0 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470800ba60 con 0x7f4708004d00 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.187+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470800bc20 con 0x7f472806d5d0 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f4726800640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4708003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f471fe00640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 0x7f47080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f4727200640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4708004d00 0x7f4708005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f4726800640 1 -- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 msgr2=0x7f47080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f4726800640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 0x7f47080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f4726800640 1 -- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4708004d00 msgr2=0x7f4708005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:45.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f4726800640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4708004d00 0x7f4708005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f4726800640 1 -- 172.21.15.39:0/2550750188 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4708003520 con 0x7f47080039a0 2024-01-24T17:48:45.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.188+0000 7f471fe00640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f472806d5d0 0x7f47080047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.189+0000 7f4726800640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4708003dc0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f4718001720 tx=0x7f47080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976fb379ef24fe7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.190+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47180052d0 con 0x7f47080039a0 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.190+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471802e050 con 0x7f47080039a0 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.190+0000 7f4725e00640 1 -- 172.21.15.39:0/2550750188 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47180035f0 con 0x7f47080039a0 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.190+0000 7f472d400640 1 -- 172.21.15.39:0/2550750188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 msgr2=0x7f4708003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.190+0000 7f472d400640 1 --2- 172.21.15.39:0/2550750188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4708003dc0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f4718001720 tx=0x7f47080057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.190+0000 7f4727200640 1 -- 172.21.15.39:0/2550750188 reap_dead start 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.191+0000 7f472d400640 1 -- 172.21.15.39:0/2550750188 shutdown_connections 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.191+0000 7f472d400640 1 -- 172.21.15.39:0/2550750188 >> 172.21.15.39:0/2550750188 conn(0x7f47280ce9e0 msgr2=0x7f47280cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.191+0000 7f472d400640 1 -- 172.21.15.39:0/2550750188 shutdown_connections 2024-01-24T17:48:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.191+0000 7f472d400640 1 -- 172.21.15.39:0/2550750188 wait complete. 2024-01-24T17:48:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.192+0000 7f472d400640 1 Processor -- start 2024-01-24T17:48:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.193+0000 7f472d400640 1 -- start start 2024-01-24T17:48:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.193+0000 7f472d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4728155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.193+0000 7f472d400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47280c7990 0x7f47281f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.193+0000 7f472d400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47280b2190 0x7f47280b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f472d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47280c8550 con 0x7f47080039a0 2024-01-24T17:48:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f472d400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47280b2b50 con 0x7f47280b2190 2024-01-24T17:48:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f472d400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4728155e50 con 0x7f47280c7990 2024-01-24T17:48:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4726800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4728155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4726800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4728155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33054/0 (socket says 172.21.15.39:33054) 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4726800640 1 -- 172.21.15.39:0/234397089 learned_addr learned my addr 172.21.15.39:0/234397089 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4727200640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47280b2190 0x7f47280b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4726800640 1 -- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47280c7990 msgr2=0x7f47281f1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f471fe00640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47280c7990 0x7f47281f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4726800640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47280c7990 0x7f47281f1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4726800640 1 -- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47280b2190 msgr2=0x7f47280b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4726800640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47280b2190 0x7f47280b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.194+0000 7f4726800640 1 -- 172.21.15.39:0/234397089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4728156080 con 0x7f47080039a0 2024-01-24T17:48:45.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.195+0000 7f4726800640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4728155910 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4718008fd0 tx=0x7f4718001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.195+0000 7f471ea00640 1 -- 172.21.15.39:0/234397089 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47180076c0 con 0x7f47080039a0 2024-01-24T17:48:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.195+0000 7f471ea00640 1 -- 172.21.15.39:0/234397089 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4718007a30 con 0x7f47080039a0 2024-01-24T17:48:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.195+0000 7f471ea00640 1 -- 172.21.15.39:0/234397089 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718006be0 con 0x7f47080039a0 2024-01-24T17:48:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.195+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47280c7710 con 0x7f47080039a0 2024-01-24T17:48:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.195+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47281f2070 con 0x7f47080039a0 2024-01-24T17:48:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.196+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472804dff0 con 0x7f47080039a0 2024-01-24T17:48:45.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.197+0000 7f471ea00640 1 -- 172.21.15.39:0/234397089 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4718006d40 con 0x7f47080039a0 2024-01-24T17:48:45.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.198+0000 7f471ea00640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f470006a430 0x7f470006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.199+0000 7f471ea00640 1 -- 172.21.15.39:0/234397089 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47180aaa80 con 0x7f47080039a0 2024-01-24T17:48:45.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.199+0000 7f471fe00640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f470006a430 0x7f470006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.200+0000 7f471fe00640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f470006a430 0x7f470006c8f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4710001200 tx=0x7f4710009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:45.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.203+0000 7f471ea00640 1 -- 172.21.15.39:0/234397089 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4718076070 con 0x7f47080039a0 2024-01-24T17:48:45.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.396+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f472806c8a0 con 0x7f47080039a0 2024-01-24T17:48:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.397+0000 7f471ea00640 1 -- 172.21.15.39:0/234397089 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f471804b020 con 0x7f47080039a0 2024-01-24T17:48:45.399 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:48:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.399+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f470006a430 msgr2=0x7f470006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.399+0000 7f472d400640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f470006a430 0x7f470006c8f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4710001200 tx=0x7f4710009040 comp rx=0 tx=0).stop 2024-01-24T17:48:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.399+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 msgr2=0x7f4728155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.399+0000 7f472d400640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4728155910 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4718008fd0 tx=0x7f4718001120 comp rx=0 tx=0).stop 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.400+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 shutdown_connections 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.400+0000 7f472d400640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47280b2190 0x7f47280b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.400+0000 7f472d400640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f47280c7990 0x7f47281f1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.400+0000 7f472d400640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f470006a430 0x7f470006c8f0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.400+0000 7f472d400640 1 --2- 172.21.15.39:0/234397089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47080039a0 0x7f4728155910 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.400+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 >> 172.21.15.39:0/234397089 conn(0x7f47280ce9e0 msgr2=0x7f47280ae350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.400+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 shutdown_connections 2024-01-24T17:48:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.400+0000 7f472d400640 1 -- 172.21.15.39:0/234397089 wait complete. 2024-01-24T17:48:45.420 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:48:45.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.638+0000 7fc72a400640 1 Processor -- start 2024-01-24T17:48:45.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.638+0000 7fc72a400640 1 -- start start 2024-01-24T17:48:45.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc72a400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc72415ca30 0x7fc7240d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc72a400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc7241606b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc72a400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240d37a0 con 0x7fc724160bf0 2024-01-24T17:48:45.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc72a400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7241610a0 con 0x7fc72415ca30 2024-01-24T17:48:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc72a400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240d13c0 con 0x7fc7240d0e90 2024-01-24T17:48:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc723e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc724160bf0 0x7fc7240cd570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40858/0 (socket says 172.21.15.39:40858) 2024-01-24T17:48:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc723e00640 1 -- 172.21.15.39:0/2277154663 learned_addr learned my addr 172.21.15.39:0/2277154663 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc71be00640 1 --2- 172.21.15.39:0/2277154663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc7241606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.640+0000 7fc723400640 1 --2- 172.21.15.39:0/2277154663 >> v2:172.21.15.100:3300/0 conn(0x7fc72415ca30 0x7fc7240d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.641+0000 7fc722a00640 1 -- 172.21.15.39:0/2277154663 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315365163 0 0) 0x7fc7240d37a0 con 0x7fc724160bf0 2024-01-24T17:48:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.641+0000 7fc722a00640 1 -- 172.21.15.39:0/2277154663 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc704003520 con 0x7fc724160bf0 2024-01-24T17:48:45.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.641+0000 7fc71be00640 1 -- 172.21.15.39:0/2277154663 >> v2:172.21.15.100:3300/0 conn(0x7fc72415ca30 msgr2=0x7fc7240d3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.641+0000 7fc71be00640 1 --2- 172.21.15.39:0/2277154663 >> v2:172.21.15.100:3300/0 conn(0x7fc72415ca30 0x7fc7240d3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.641+0000 7fc71be00640 1 -- 172.21.15.39:0/2277154663 >> v1:172.21.15.39:6789/0 conn(0x7fc724160bf0 legacy=0x7fc7240cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.641+0000 7fc722a00640 1 -- 172.21.15.39:0/2277154663 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3700277715 0 0) 0x7fc704003520 con 0x7fc724160bf0 2024-01-24T17:48:45.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.641+0000 7fc71be00640 1 -- 172.21.15.39:0/2277154663 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7240d16a0 con 0x7fc7240d0e90 2024-01-24T17:48:45.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.642+0000 7fc71be00640 1 --2- 172.21.15.39:0/2277154663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc7241606b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0097a0 tx=0x7fc70c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44cfbae230aef882 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.643+0000 7fc722a00640 1 -- 172.21.15.39:0/2277154663 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c0019f0 con 0x7fc7240d0e90 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.643+0000 7fc722a00640 1 -- 172.21.15.39:0/2277154663 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c02e050 con 0x7fc7240d0e90 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.643+0000 7fc722a00640 1 -- 172.21.15.39:0/2277154663 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c030710 con 0x7fc7240d0e90 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.644+0000 7fc72a400640 1 -- 172.21.15.39:0/2277154663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 msgr2=0x7fc7241606b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.644+0000 7fc72a400640 1 --2- 172.21.15.39:0/2277154663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc7241606b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0097a0 tx=0x7fc70c02b790 comp rx=0 tx=0).stop 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.644+0000 7fc72a400640 1 -- 172.21.15.39:0/2277154663 shutdown_connections 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.644+0000 7fc72a400640 1 --2- 172.21.15.39:0/2277154663 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc7241606b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.644+0000 7fc72a400640 1 --2- 172.21.15.39:0/2277154663 >> v2:172.21.15.100:3300/0 conn(0x7fc72415ca30 0x7fc7240d3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.644+0000 7fc72a400640 1 -- 172.21.15.39:0/2277154663 >> 172.21.15.39:0/2277154663 conn(0x7fc72409bc80 msgr2=0x7fc7240d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.644+0000 7fc72a400640 1 -- 172.21.15.39:0/2277154663 shutdown_connections 2024-01-24T17:48:45.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.644+0000 7fc72a400640 1 -- 172.21.15.39:0/2277154663 wait complete. 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.646+0000 7fc72a400640 1 Processor -- start 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.646+0000 7fc72a400640 1 -- start start 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc72a400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc72415d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc72a400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc72415ca30 0x7fc724157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc72a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc724160bf0 0x7fc7241f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc72a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7241f2ff0 con 0x7fc724160bf0 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc72a400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415d800 con 0x7fc72415ca30 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc72a400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415dae0 con 0x7fc7240d0e90 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc723e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc724160bf0 0x7fc7241f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc723e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc724160bf0 0x7fc7241f2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33070/0 (socket says 172.21.15.39:33070) 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc723e00640 1 -- 172.21.15.39:0/2722975094 learned_addr learned my addr 172.21.15.39:0/2722975094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:45.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc71be00640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc72415ca30 0x7fc724157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.647+0000 7fc723400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc72415d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc723e00640 1 -- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 msgr2=0x7fc72415d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc723e00640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc72415d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc723e00640 1 -- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc72415ca30 msgr2=0x7fc724157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc723e00640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc72415ca30 0x7fc724157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc723e00640 1 -- 172.21.15.39:0/2722975094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c009400 con 0x7fc724160bf0 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc723e00640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc724160bf0 0x7fc7241f2ab0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00cc00 tx=0x7fc71c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc71b400640 1 -- 172.21.15.39:0/2722975094 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c00ce10 con 0x7fc724160bf0 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc71b400640 1 -- 172.21.15.39:0/2722975094 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c012400 con 0x7fc724160bf0 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc71b400640 1 -- 172.21.15.39:0/2722975094 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c0127b0 con 0x7fc724160bf0 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72415ddc0 con 0x7fc724160bf0 2024-01-24T17:48:45.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.648+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7240cf700 con 0x7fc724160bf0 2024-01-24T17:48:45.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.649+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72406cd10 con 0x7fc724160bf0 2024-01-24T17:48:45.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.650+0000 7fc71b400640 1 -- 172.21.15.39:0/2722975094 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc71c012910 con 0x7fc724160bf0 2024-01-24T17:48:45.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.651+0000 7fc71b400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6fc06a430 0x7fc6fc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:45.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.652+0000 7fc71b400640 1 -- 172.21.15.39:0/2722975094 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc71c0890b0 con 0x7fc724160bf0 2024-01-24T17:48:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.652+0000 7fc723400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6fc06a430 0x7fc6fc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:45.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.653+0000 7fc723400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6fc06a430 0x7fc6fc06c8f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc714009a10 tx=0x7fc714008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:45.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.656+0000 7fc71b400640 1 -- 172.21.15.39:0/2722975094 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc71c054680 con 0x7fc724160bf0 2024-01-24T17:48:45.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.843+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc72404dff0 con 0x7fc724160bf0 2024-01-24T17:48:45.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.844+0000 7fc71b400640 1 -- 172.21.15.39:0/2722975094 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc71c057ca0 con 0x7fc724160bf0 2024-01-24T17:48:45.846 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:48:45.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.846+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6fc06a430 msgr2=0x7fc6fc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.846+0000 7fc72a400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6fc06a430 0x7fc6fc06c8f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fc714009a10 tx=0x7fc714008040 comp rx=0 tx=0).stop 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.846+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc724160bf0 msgr2=0x7fc7241f2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.846+0000 7fc72a400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc724160bf0 0x7fc7241f2ab0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00cc00 tx=0x7fc71c00a430 comp rx=0 tx=0).stop 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.847+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 shutdown_connections 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.847+0000 7fc72a400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc6fc06a430 0x7fc6fc06c8f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.847+0000 7fc72a400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc724160bf0 0x7fc7241f2ab0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.847+0000 7fc72a400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc72415ca30 0x7fc724157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.847+0000 7fc72a400640 1 --2- 172.21.15.39:0/2722975094 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc7240d0e90 0x7fc72415d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.847+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 >> 172.21.15.39:0/2722975094 conn(0x7fc72409bc80 msgr2=0x7fc72415a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.847+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 shutdown_connections 2024-01-24T17:48:45.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:45.847+0000 7fc72a400640 1 -- 172.21.15.39:0/2722975094 wait complete. 2024-01-24T17:48:45.866 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:48:46.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.077+0000 7f6011200640 1 Processor -- start 2024-01-24T17:48:46.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.077+0000 7f6011200640 1 -- start start 2024-01-24T17:48:46.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f6011200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f600c0cf8a0 0x7f600c0d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:46.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f6011200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f600c06acb0 0x7f600c0d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:46.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f6011200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0ce010 con 0x7f600c15cbf0 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f6011200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c0ce870 con 0x7f600c06acb0 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f6011200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c0ce2b0 con 0x7f600c0cf8a0 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f600b400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f600c15cbf0 0x7f600c0cf190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40862/0 (socket says 172.21.15.39:40862) 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f600b400640 1 -- 172.21.15.39:0/4243629300 learned_addr learned my addr 172.21.15.39:0/4243629300 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f600be00640 1 --2- 172.21.15.39:0/4243629300 >> v2:172.21.15.39:3300/0 conn(0x7f600c06acb0 0x7f600c0d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f600a000640 1 -- 172.21.15.39:0/4243629300 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995085666 0 0) 0x7f600c0ce010 con 0x7f600c15cbf0 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f600a000640 1 -- 172.21.15.39:0/4243629300 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec003520 con 0x7f600c15cbf0 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.079+0000 7f600aa00640 1 --2- 172.21.15.39:0/4243629300 >> v2:172.21.15.100:3300/0 conn(0x7f600c0cf8a0 0x7f600c0d0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.080+0000 7f600be00640 1 -- 172.21.15.39:0/4243629300 >> v2:172.21.15.100:3300/0 conn(0x7f600c0cf8a0 msgr2=0x7f600c0d0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:46.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.080+0000 7f600be00640 1 --2- 172.21.15.39:0/4243629300 >> v2:172.21.15.100:3300/0 conn(0x7f600c0cf8a0 0x7f600c0d0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.080+0000 7f600be00640 1 -- 172.21.15.39:0/4243629300 >> v1:172.21.15.39:6789/0 conn(0x7f600c15cbf0 legacy=0x7f600c0cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.080+0000 7f600be00640 1 -- 172.21.15.39:0/4243629300 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c0ce590 con 0x7f600c06acb0 2024-01-24T17:48:46.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.080+0000 7f600aa00640 1 --2- 172.21.15.39:0/4243629300 >> v2:172.21.15.100:3300/0 conn(0x7f600c0cf8a0 0x7f600c0d0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.081+0000 7f600be00640 1 --2- 172.21.15.39:0/4243629300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c0d0fd0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f600c0b53b0 tx=0x7f600402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c138612bb40e95b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.081+0000 7f600a000640 1 -- 172.21.15.39:0/4243629300 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600402e070 con 0x7f600c06acb0 2024-01-24T17:48:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.081+0000 7f600a000640 1 -- 172.21.15.39:0/4243629300 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60040315a0 con 0x7f600c06acb0 2024-01-24T17:48:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.081+0000 7f600a000640 1 -- 172.21.15.39:0/4243629300 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004031930 con 0x7f600c06acb0 2024-01-24T17:48:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.082+0000 7f6011200640 1 -- 172.21.15.39:0/4243629300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 msgr2=0x7f600c0d0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:46.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.082+0000 7f6011200640 1 --2- 172.21.15.39:0/4243629300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c0d0fd0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f600c0b53b0 tx=0x7f600402b840 comp rx=0 tx=0).stop 2024-01-24T17:48:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.082+0000 7f6011200640 1 -- 172.21.15.39:0/4243629300 shutdown_connections 2024-01-24T17:48:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.082+0000 7f6011200640 1 --2- 172.21.15.39:0/4243629300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c0d0fd0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.082+0000 7f6011200640 1 --2- 172.21.15.39:0/4243629300 >> v2:172.21.15.100:3300/0 conn(0x7f600c0cf8a0 0x7f600c0d0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.082+0000 7f6011200640 1 -- 172.21.15.39:0/4243629300 >> 172.21.15.39:0/4243629300 conn(0x7f600c09bad0 msgr2=0x7f600c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.082+0000 7f6011200640 1 -- 172.21.15.39:0/4243629300 shutdown_connections 2024-01-24T17:48:46.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.082+0000 7f6011200640 1 -- 172.21.15.39:0/4243629300 wait complete. 2024-01-24T17:48:46.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.083+0000 7f6011200640 1 Processor -- start 2024-01-24T17:48:46.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.084+0000 7f6011200640 1 -- start start 2024-01-24T17:48:46.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.084+0000 7f6011200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:46.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.084+0000 7f6011200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f600c0cf8a0 0x7f600c1f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:46.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.084+0000 7f6011200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f600c15cbf0 0x7f600c15a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f6011200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c09e2d0 con 0x7f600c06acb0 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f6011200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1f2b10 con 0x7f600c0cf8a0 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f6011200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c09e870 con 0x7f600c15cbf0 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33086/0 (socket says 172.21.15.39:33086) 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600b400640 1 -- 172.21.15.39:0/1936810933 learned_addr learned my addr 172.21.15.39:0/1936810933 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600aa00640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f600c0cf8a0 0x7f600c1f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600be00640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f600c15cbf0 0x7f600c15a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600b400640 1 -- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f600c15cbf0 msgr2=0x7f600c15a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600b400640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f600c15cbf0 0x7f600c15a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600b400640 1 -- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f600c0cf8a0 msgr2=0x7f600c1f1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600b400640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f600c0cf8a0 0x7f600c1f1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600b400640 1 -- 172.21.15.39:0/1936810933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6004009400 con 0x7f600c06acb0 2024-01-24T17:48:46.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.085+0000 7f600aa00640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f600c0cf8a0 0x7f600c1f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.086+0000 7f600b400640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c159e80 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc001260 tx=0x7f5ffc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.086+0000 7f6003400640 1 -- 172.21.15.39:0/1936810933 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0074e0 con 0x7f600c06acb0 2024-01-24T17:48:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.086+0000 7f6003400640 1 -- 172.21.15.39:0/1936810933 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc007640 con 0x7f600c06acb0 2024-01-24T17:48:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.086+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c1f2550 con 0x7f600c06acb0 2024-01-24T17:48:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.086+0000 7f6003400640 1 -- 172.21.15.39:0/1936810933 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0059a0 con 0x7f600c06acb0 2024-01-24T17:48:46.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.086+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c09e500 con 0x7f600c06acb0 2024-01-24T17:48:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.087+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600c04dff0 con 0x7f600c06acb0 2024-01-24T17:48:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.088+0000 7f6003400640 1 -- 172.21.15.39:0/1936810933 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5ffc005b00 con 0x7f600c06acb0 2024-01-24T17:48:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.089+0000 7f6003400640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5fe406a430 0x7f5fe406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.089+0000 7f6003400640 1 -- 172.21.15.39:0/1936810933 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ffc087e20 con 0x7f600c06acb0 2024-01-24T17:48:46.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.089+0000 7f600aa00640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5fe406a430 0x7f5fe406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:46.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.091+0000 7f600aa00640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5fe406a430 0x7f5fe406c8f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f600c0b3fd0 tx=0x7f5ff4007530 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:46.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.093+0000 7f6003400640 1 -- 172.21.15.39:0/1936810933 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5ffc0534b0 con 0x7f600c06acb0 2024-01-24T17:48:46.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.300+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f600c0d0000 con 0x7f600c06acb0 2024-01-24T17:48:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.301+0000 7f6003400640 1 -- 172.21.15.39:0/1936810933 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v100) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5ffc017020 con 0x7f600c06acb0 2024-01-24T17:48:46.303 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:48:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5fe406a430 msgr2=0x7f5fe406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:46.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5fe406a430 0x7f5fe406c8f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f600c0b3fd0 tx=0x7f5ff4007530 comp rx=0 tx=0).stop 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 msgr2=0x7f600c159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c159e80 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc001260 tx=0x7f5ffc00c960 comp rx=0 tx=0).stop 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 shutdown_connections 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f600c15cbf0 0x7f600c15a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f600c0cf8a0 0x7f600c1f1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f5fe406a430 0x7f5fe406c8f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 --2- 172.21.15.39:0/1936810933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c06acb0 0x7f600c159e80 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 >> 172.21.15.39:0/1936810933 conn(0x7f600c09bad0 msgr2=0x7f600c0cd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 shutdown_connections 2024-01-24T17:48:46.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:46.304+0000 7f6011200640 1 -- 172.21.15.39:0/1936810933 wait complete. 2024-01-24T17:48:46.324 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:48:46.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:46.325 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:48:46.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[70338]: fuse finished with error 0 and tester_r 0 2024-01-24T17:48:46.423 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:51.671 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:48:51.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:48:52.525 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:48:52.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:52.525 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:48:52.587 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:48:52.588 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:52.644 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:52.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.874+0000 7f16d7e00640 1 Processor -- start 2024-01-24T17:48:52.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.874+0000 7f16d7e00640 1 -- start start 2024-01-24T17:48:52.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.875+0000 7f16d7e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f16d80ca700 0x7f16d815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.875+0000 7f16d7e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d81540f0 con 0x7f16d815a910 2024-01-24T17:48:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.876+0000 7f16d7e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d815da00 con 0x7f16d80ca320 2024-01-24T17:48:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.876+0000 7f16d7e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16d8159fd0 con 0x7f16d80ca700 2024-01-24T17:48:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.876+0000 7f16d7400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f16d815a910 0x7f16d806bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:40866/0 (socket says 172.21.15.39:40866) 2024-01-24T17:48:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.876+0000 7f16d7400640 1 -- 172.21.15.39:0/1798392371 learned_addr learned my addr 172.21.15.39:0/1798392371 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.876+0000 7f16cfe00640 1 --2- 172.21.15.39:0/1798392371 >> v2:172.21.15.100:3300/0 conn(0x7f16d80ca700 0x7f16d815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.876+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300548855 0 0) 0x7f16d81540f0 con 0x7f16d815a910 2024-01-24T17:48:52.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.876+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16b8003520 con 0x7f16d815a910 2024-01-24T17:48:52.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3034104951 0 0) 0x7f16b8003520 con 0x7f16d815a910 2024-01-24T17:48:52.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 >> v2:172.21.15.100:3300/0 conn(0x7f16d80ca700 msgr2=0x7f16d815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:52.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 --2- 172.21.15.39:0/1798392371 >> v2:172.21.15.100:3300/0 conn(0x7f16d80ca700 0x7f16d815a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:52.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 >> v1:172.21.15.100:6789/0 conn(0x7f16d80ca320 legacy=0x7f16d815d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d815dce0 con 0x7f16d815a910 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f16d00019f0 con 0x7f16d815a910 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 >> v1:172.21.15.39:6789/0 conn(0x7f16d815a910 legacy=0x7f16d806bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16b8003970 0x7f16b8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b80052a0 0x7f16b800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8159fd0 con 0x7f16b80052a0 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b800e400 con 0x7f16b80047c0 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b800e5c0 con 0x7f16b8003970 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d7400640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b80052a0 0x7f16b800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d6a00640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16b8003970 0x7f16b8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:52.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16cfe00640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d7400640 1 -- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16b8003970 msgr2=0x7f16b8003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d7400640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16b8003970 0x7f16b8003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d7400640 1 -- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80047c0 msgr2=0x7f16b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d7400640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.877+0000 7f16d7400640 1 -- 172.21.15.39:0/1798392371 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8003520 con 0x7f16b80052a0 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.878+0000 7f16cfe00640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80047c0 0x7f16b8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.878+0000 7f16d7400640 1 -- 172.21.15.39:0/1798392371 reap_dead start 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d7400640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b80052a0 0x7f16b800db90 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f16d0002230 tx=0x7f16b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bd8ceb13ea5633 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d0034630 con 0x7f16b80052a0 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16d002f070 con 0x7f16b80052a0 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d6000640 1 -- 172.21.15.39:0/1798392371 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d00349f0 con 0x7f16b80052a0 2024-01-24T17:48:52.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d7e00640 1 -- 172.21.15.39:0/1798392371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b80052a0 msgr2=0x7f16b800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:52.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d7e00640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b80052a0 0x7f16b800db90 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f16d0002230 tx=0x7f16b80057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:52.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d7e00640 1 -- 172.21.15.39:0/1798392371 shutdown_connections 2024-01-24T17:48:52.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d7e00640 1 --2- 172.21.15.39:0/1798392371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b80052a0 0x7f16b800db90 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:52.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.879+0000 7f16d7e00640 1 -- 172.21.15.39:0/1798392371 >> 172.21.15.39:0/1798392371 conn(0x7f16d806a340 msgr2=0x7f16d80cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:52.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.880+0000 7f16d7e00640 1 -- 172.21.15.39:0/1798392371 shutdown_connections 2024-01-24T17:48:52.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.880+0000 7f16d7e00640 1 -- 172.21.15.39:0/1798392371 wait complete. 2024-01-24T17:48:52.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.881+0000 7f16d7e00640 1 Processor -- start 2024-01-24T17:48:52.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.881+0000 7f16d7e00640 1 -- start start 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.882+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80052a0 0x7f16d8141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.882+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16d813f260 0x7f16d81f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.882+0000 7f16d7e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16d81408a0 0x7f16d8142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.882+0000 7f16d7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d8142240 con 0x7f16d813f260 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.882+0000 7f16d7e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d813f800 con 0x7f16b80052a0 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.882+0000 7f16d7e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d813fae0 con 0x7f16d81408a0 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16cfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16d813f260 0x7f16d81f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16cfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16d813f260 0x7f16d81f1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33110/0 (socket says 172.21.15.39:33110) 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16cfe00640 1 -- 172.21.15.39:0/2457833959 learned_addr learned my addr 172.21.15.39:0/2457833959 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16d7400640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16d81408a0 0x7f16d8142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:52.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16cfe00640 1 -- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16d81408a0 msgr2=0x7f16d8142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16cfe00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16d81408a0 0x7f16d8142750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16cfe00640 1 -- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80052a0 msgr2=0x7f16d8141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16cfe00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80052a0 0x7f16d8141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.883+0000 7f16cfe00640 1 -- 172.21.15.39:0/2457833959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d813fdc0 con 0x7f16d813f260 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.884+0000 7f16cfe00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16d813f260 0x7f16d81f1ad0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f16c000e7e0 tx=0x7f16c000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.884+0000 7f16cf400640 1 -- 172.21.15.39:0/2457833959 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c000c6d0 con 0x7f16d813f260 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.884+0000 7f16cf400640 1 -- 172.21.15.39:0/2457833959 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c000c830 con 0x7f16d813f260 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.884+0000 7f16cf400640 1 -- 172.21.15.39:0/2457833959 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c000cba0 con 0x7f16d813f260 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.884+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d81400a0 con 0x7f16d813f260 2024-01-24T17:48:52.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.884+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d81f2010 con 0x7f16d813f260 2024-01-24T17:48:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.886+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d804dff0 con 0x7f16d813f260 2024-01-24T17:48:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.886+0000 7f16cf400640 1 -- 172.21.15.39:0/2457833959 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f16c000cd00 con 0x7f16d813f260 2024-01-24T17:48:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.887+0000 7f16cf400640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16b006a430 0x7f16b006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:52.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.887+0000 7f16cf400640 1 -- 172.21.15.39:0/2457833959 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16c00886f0 con 0x7f16d813f260 2024-01-24T17:48:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.888+0000 7f16d6a00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16b006a430 0x7f16b006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:52.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.889+0000 7f16d6a00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16b006a430 0x7f16b006c8f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f16c8003d40 tx=0x7f16c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:52.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:52.892+0000 7f16cf400640 1 -- 172.21.15.39:0/2457833959 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f16c0053d70 con 0x7f16d813f260 2024-01-24T17:48:53.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.085+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16d8158a40 con 0x7f16d813f260 2024-01-24T17:48:53.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.088+0000 7f16cf400640 1 -- 172.21.15.39:0/2457833959 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16042 (secure 0 0 0) 0x7f16c0057390 con 0x7f16d813f260 2024-01-24T17:48:53.090 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:53.091 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":100,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:48:27.885313+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:53.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.091+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16b006a430 msgr2=0x7f16b006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.091+0000 7f16d7e00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16b006a430 0x7f16b006c8f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f16c8003d40 tx=0x7f16c8009040 comp rx=0 tx=0).stop 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.091+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16d813f260 msgr2=0x7f16d81f1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.091+0000 7f16d7e00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16d813f260 0x7f16d81f1ad0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f16c000e7e0 tx=0x7f16c000eca0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.091+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 shutdown_connections 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.091+0000 7f16d7e00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f16d81408a0 0x7f16d8142750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.091+0000 7f16d7e00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f16b006a430 0x7f16b006c8f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.091+0000 7f16d7e00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16d813f260 0x7f16d81f1ad0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.092+0000 7f16d7e00640 1 --2- 172.21.15.39:0/2457833959 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f16b80052a0 0x7f16d8141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.092+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 >> 172.21.15.39:0/2457833959 conn(0x7f16d806a340 msgr2=0x7f16d815bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.092+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 shutdown_connections 2024-01-24T17:48:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.092+0000 7f16d7e00640 1 -- 172.21.15.39:0/2457833959 wait complete. 2024-01-24T17:48:53.116 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:48:53.116 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:48:53.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.342+0000 7f1050e00640 1 Processor -- start 2024-01-24T17:48:53.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.343+0000 7f1050e00640 1 -- start start 2024-01-24T17:48:53.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.344+0000 7f1050e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f104c159530 0x7f104c070de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.344+0000 7f1050e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c0cbf90 con 0x7f104c0cb250 2024-01-24T17:48:53.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.344+0000 7f1050e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104c0cc270 con 0x7f104c071e40 2024-01-24T17:48:53.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.344+0000 7f1050e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104c0cc550 con 0x7f104c159530 2024-01-24T17:48:53.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.345+0000 7f104be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f104c0cb250 0x7f104c0c8a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58296/0 (socket says 172.21.15.39:58296) 2024-01-24T17:48:53.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.345+0000 7f104be00640 1 -- 172.21.15.39:0/2412725143 learned_addr learned my addr 172.21.15.39:0/2412725143 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:53.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.345+0000 7f104b400640 1 --2- 172.21.15.39:0/2412725143 >> v2:172.21.15.100:3300/0 conn(0x7f104c159530 0x7f104c070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.345+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435149861 0 0) 0x7f104c0cbf90 con 0x7f104c0cb250 2024-01-24T17:48:53.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.345+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102c005730 con 0x7f104c0cb250 2024-01-24T17:48:53.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 352208399 0 0) 0x7f102c005730 con 0x7f104c0cb250 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 >> v2:172.21.15.100:3300/0 conn(0x7f104c159530 msgr2=0x7f104c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 --2- 172.21.15.39:0/2412725143 >> v2:172.21.15.100:3300/0 conn(0x7f104c159530 0x7f104c070de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 >> v1:172.21.15.100:6789/0 conn(0x7f104c071e40 legacy=0x7f104c0cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c0cc830 con 0x7f104c0cb250 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f10440019f0 con 0x7f104c0cb250 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 >> v1:172.21.15.39:6789/0 conn(0x7f104c0cb250 legacy=0x7f104c0c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102c005bd0 0x7f102c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c0069d0 0x7f102c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f102c007390 0x7f102c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c0cc550 con 0x7f102c005bd0 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c00e290 con 0x7f102c0069d0 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c00e450 con 0x7f102c007390 2024-01-24T17:48:53.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104b400640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102c005bd0 0x7f102c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104b400640 1 -- 172.21.15.39:0/2412725143 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f102c007390 msgr2=0x7f102c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:53.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104b400640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f102c007390 0x7f102c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104b400640 1 -- 172.21.15.39:0/2412725143 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c0069d0 msgr2=0x7f102c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:53.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104b400640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c0069d0 0x7f102c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.346+0000 7f104b400640 1 -- 172.21.15.39:0/2412725143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c005730 con 0x7f102c005bd0 2024-01-24T17:48:53.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.347+0000 7f104be00640 1 -- 172.21.15.39:0/2412725143 reap_dead start 2024-01-24T17:48:53.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.348+0000 7f104b400640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102c005bd0 0x7f102c005ff0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f103c009ac0 tx=0x7f102c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c9ee836b6d9c22 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:53.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.348+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c02c6b0 con 0x7f102c005bd0 2024-01-24T17:48:53.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.348+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c02c810 con 0x7f102c005bd0 2024-01-24T17:48:53.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.348+0000 7f104a000640 1 -- 172.21.15.39:0/2412725143 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c02cba0 con 0x7f102c005bd0 2024-01-24T17:48:53.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.348+0000 7f1050e00640 1 -- 172.21.15.39:0/2412725143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102c005bd0 msgr2=0x7f102c005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.348+0000 7f1050e00640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102c005bd0 0x7f102c005ff0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f103c009ac0 tx=0x7f102c007a00 comp rx=0 tx=0).stop 2024-01-24T17:48:53.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.349+0000 7f1050e00640 1 -- 172.21.15.39:0/2412725143 shutdown_connections 2024-01-24T17:48:53.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.349+0000 7f1050e00640 1 --2- 172.21.15.39:0/2412725143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f102c005bd0 0x7f102c005ff0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.349+0000 7f1050e00640 1 -- 172.21.15.39:0/2412725143 >> 172.21.15.39:0/2412725143 conn(0x7f104c06a200 msgr2=0x7f104c0affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:53.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.349+0000 7f1050e00640 1 -- 172.21.15.39:0/2412725143 shutdown_connections 2024-01-24T17:48:53.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.349+0000 7f1050e00640 1 -- 172.21.15.39:0/2412725143 wait complete. 2024-01-24T17:48:53.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.350+0000 7f1050e00640 1 Processor -- start 2024-01-24T17:48:53.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.351+0000 7f1050e00640 1 -- start start 2024-01-24T17:48:53.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.351+0000 7f1050e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c005bd0 0x7f104c156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.351+0000 7f1050e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f104c0c5d80 0x7f104c0c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.351+0000 7f1050e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f104c151640 0x7f104c151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.351+0000 7f1050e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c152000 con 0x7f104c151640 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f1050e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c1f64b0 con 0x7f102c005bd0 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f1050e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104c1f58a0 con 0x7f104c0c5d80 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f104c151640 0x7f104c151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f104c151640 0x7f104c151ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57190/0 (socket says 172.21.15.39:57190) 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104be00640 1 -- 172.21.15.39:0/3673255293 learned_addr learned my addr 172.21.15.39:0/3673255293 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104b400640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c005bd0 0x7f104c156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104aa00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f104c0c5d80 0x7f104c0c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104be00640 1 -- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f104c0c5d80 msgr2=0x7f104c0c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104be00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f104c0c5d80 0x7f104c0c7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104be00640 1 -- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c005bd0 msgr2=0x7f104c156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104be00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c005bd0 0x7f104c156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104be00640 1 -- 172.21.15.39:0/3673255293 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c1f5b80 con 0x7f104c151640 2024-01-24T17:48:53.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.352+0000 7f104b400640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c005bd0 0x7f104c156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:53.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.353+0000 7f104be00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f104c151640 0x7f104c151ac0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f104c0b0b50 tx=0x7f104400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.353+0000 7f1043400640 1 -- 172.21.15.39:0/3673255293 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104400ce50 con 0x7f104c151640 2024-01-24T17:48:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.353+0000 7f1043400640 1 -- 172.21.15.39:0/3673255293 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104400a610 con 0x7f104c151640 2024-01-24T17:48:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.353+0000 7f1043400640 1 -- 172.21.15.39:0/3673255293 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104400a9a0 con 0x7f104c151640 2024-01-24T17:48:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.353+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c1f5e60 con 0x7f104c151640 2024-01-24T17:48:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.353+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c0c8210 con 0x7f104c151640 2024-01-24T17:48:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.354+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104c0738c0 con 0x7f104c151640 2024-01-24T17:48:53.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.355+0000 7f1043400640 1 -- 172.21.15.39:0/3673255293 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f104400ab00 con 0x7f104c151640 2024-01-24T17:48:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.356+0000 7f1043400640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f102406a460 0x7f102406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.356+0000 7f1043400640 1 -- 172.21.15.39:0/3673255293 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1044087ff0 con 0x7f104c151640 2024-01-24T17:48:53.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.357+0000 7f104b400640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f102406a460 0x7f102406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.358+0000 7f104b400640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f102406a460 0x7f102406c920 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f103c036000 tx=0x7f103c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:53.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.360+0000 7f1043400640 1 -- 172.21.15.39:0/3673255293 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1044053580 con 0x7f104c151640 2024-01-24T17:48:53.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.557+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f104c0cdb10 con 0x7f104c151640 2024-01-24T17:48:53.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.557+0000 7f1043400640 1 -- 172.21.15.39:0/3673255293 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1044056ba0 con 0x7f104c151640 2024-01-24T17:48:53.559 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f102406a460 msgr2=0x7f102406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f102406a460 0x7f102406c920 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f103c036000 tx=0x7f103c030000 comp rx=0 tx=0).stop 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f104c151640 msgr2=0x7f104c151ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f104c151640 0x7f104c151ac0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f104c0b0b50 tx=0x7f104400cc00 comp rx=0 tx=0).stop 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 shutdown_connections 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f102406a460 0x7f102406c920 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f104c151640 0x7f104c151ac0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f104c0c5d80 0x7f104c0c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 --2- 172.21.15.39:0/3673255293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f102c005bd0 0x7f104c156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 >> 172.21.15.39:0/3673255293 conn(0x7f104c06a200 msgr2=0x7f104c0aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:53.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 shutdown_connections 2024-01-24T17:48:53.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.560+0000 7f1050e00640 1 -- 172.21.15.39:0/3673255293 wait complete. 2024-01-24T17:48:53.581 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:48:53.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.810+0000 7efd4cc00640 1 Processor -- start 2024-01-24T17:48:53.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.811+0000 7efd4cc00640 1 -- start start 2024-01-24T17:48:53.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.812+0000 7efd4cc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7efd480cd960 0x7efd480cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.812+0000 7efd4cc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd480cf360 con 0x7efd480d1360 2024-01-24T17:48:53.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.813+0000 7efd4cc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4815aec0 con 0x7efd4815aae0 2024-01-24T17:48:53.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.813+0000 7efd4cc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd480b00a0 con 0x7efd480cd960 2024-01-24T17:48:53.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.813+0000 7efd46800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7efd480d1360 0x7efd480b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58300/0 (socket says 172.21.15.39:58300) 2024-01-24T17:48:53.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.813+0000 7efd46800640 1 -- 172.21.15.39:0/3756570803 learned_addr learned my addr 172.21.15.39:0/3756570803 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:53.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.813+0000 7efd3fe00640 1 --2- 172.21.15.39:0/3756570803 >> v2:172.21.15.100:3300/0 conn(0x7efd480cd960 0x7efd480cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247196571 0 0) 0x7efd480cf360 con 0x7efd480d1360 2024-01-24T17:48:53.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd28003520 con 0x7efd480d1360 2024-01-24T17:48:53.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3088747117 0 0) 0x7efd28003520 con 0x7efd480d1360 2024-01-24T17:48:53.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 >> v2:172.21.15.100:3300/0 conn(0x7efd480cd960 msgr2=0x7efd480cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:53.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 --2- 172.21.15.39:0/3756570803 >> v2:172.21.15.100:3300/0 conn(0x7efd480cd960 0x7efd480cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 >> v1:172.21.15.100:6789/0 conn(0x7efd4815aae0 legacy=0x7efd480cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd480cf590 con 0x7efd480d1360 2024-01-24T17:48:53.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7efd400019f0 con 0x7efd480d1360 2024-01-24T17:48:53.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 >> v1:172.21.15.39:6789/0 conn(0x7efd480d1360 legacy=0x7efd480b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd280039c0 0x7efd28003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd280047c0 0x7efd28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd28005180 0x7efd2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd480b00a0 con 0x7efd280047c0 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2800c080 con 0x7efd280039c0 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2800c240 con 0x7efd28005180 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.814+0000 7efd3fe00640 1 --2- 172.21.15.39:0/3756570803 >> v2:172.21.15.100:3300/0 conn(0x7efd480cd960 0x7efd480cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd3fe00640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd280047c0 0x7efd28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd46800640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd28005180 0x7efd2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd45e00640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd280039c0 0x7efd28003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd3fe00640 1 -- 172.21.15.39:0/3756570803 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd28005180 msgr2=0x7efd2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd3fe00640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd28005180 0x7efd2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd3fe00640 1 -- 172.21.15.39:0/3756570803 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd280039c0 msgr2=0x7efd28003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:53.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd3fe00640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd280039c0 0x7efd28003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd3fe00640 1 -- 172.21.15.39:0/3756570803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd28003520 con 0x7efd280047c0 2024-01-24T17:48:53.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.815+0000 7efd46800640 1 -- 172.21.15.39:0/3756570803 reap_dead start 2024-01-24T17:48:53.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.816+0000 7efd3fe00640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd280047c0 0x7efd28004c40 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7efd30009c70 tx=0x7efd280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3759b71ee8cb82a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:53.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.816+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd30004020 con 0x7efd280047c0 2024-01-24T17:48:53.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd30004180 con 0x7efd280047c0 2024-01-24T17:48:53.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd45400640 1 -- 172.21.15.39:0/3756570803 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3002c8f0 con 0x7efd280047c0 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd4cc00640 1 -- 172.21.15.39:0/3756570803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd280047c0 msgr2=0x7efd28004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd4cc00640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd280047c0 0x7efd28004c40 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7efd30009c70 tx=0x7efd280057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd4cc00640 1 -- 172.21.15.39:0/3756570803 shutdown_connections 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd4cc00640 1 --2- 172.21.15.39:0/3756570803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd280047c0 0x7efd28004c40 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd4cc00640 1 -- 172.21.15.39:0/3756570803 >> 172.21.15.39:0/3756570803 conn(0x7efd480d0840 msgr2=0x7efd480d0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd4cc00640 1 -- 172.21.15.39:0/3756570803 shutdown_connections 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.817+0000 7efd4cc00640 1 -- 172.21.15.39:0/3756570803 wait complete. 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.819+0000 7efd4cc00640 1 Processor -- start 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.819+0000 7efd4cc00640 1 -- start start 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd4cc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd280047c0 0x7efd481566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd4cc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd4815b0b0 0x7efd480ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd4cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4806ee70 0x7efd4806f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd4cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd48156c00 con 0x7efd4806ee70 2024-01-24T17:48:53.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd4cc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4806f840 con 0x7efd4815b0b0 2024-01-24T17:48:53.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd4cc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4815d2f0 con 0x7efd280047c0 2024-01-24T17:48:53.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd46800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4806ee70 0x7efd4806f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd46800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4806ee70 0x7efd4806f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57216/0 (socket says 172.21.15.39:57216) 2024-01-24T17:48:53.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.820+0000 7efd46800640 1 -- 172.21.15.39:0/1942242455 learned_addr learned my addr 172.21.15.39:0/1942242455 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:53.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd3fe00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd4815b0b0 0x7efd480ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd45e00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd280047c0 0x7efd481566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd46800640 1 -- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd280047c0 msgr2=0x7efd481566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd46800640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd280047c0 0x7efd481566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd46800640 1 -- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd4815b0b0 msgr2=0x7efd480ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:53.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd46800640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd4815b0b0 0x7efd480ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd46800640 1 -- 172.21.15.39:0/1942242455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4815b590 con 0x7efd4806ee70 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd46800640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4806ee70 0x7efd4806f300 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7efd48072150 tx=0x7efd4000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd3e000640 1 -- 172.21.15.39:0/1942242455 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4000ec40 con 0x7efd4806ee70 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd3e000640 1 -- 172.21.15.39:0/1942242455 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4000eda0 con 0x7efd4806ee70 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd3e000640 1 -- 172.21.15.39:0/1942242455 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40014670 con 0x7efd4806ee70 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.821+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4815b870 con 0x7efd4806ee70 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.822+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4815be30 con 0x7efd4806ee70 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.823+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd48067050 con 0x7efd4806ee70 2024-01-24T17:48:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.824+0000 7efd3e000640 1 -- 172.21.15.39:0/1942242455 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7efd400147d0 con 0x7efd4806ee70 2024-01-24T17:48:53.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.824+0000 7efd3e000640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2006a450 0x7efd2006c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:53.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.825+0000 7efd3e000640 1 -- 172.21.15.39:0/1942242455 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd40088750 con 0x7efd4806ee70 2024-01-24T17:48:53.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.825+0000 7efd45e00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2006a450 0x7efd2006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:53.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.826+0000 7efd45e00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2006a450 0x7efd2006c910 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efd38000ef0 tx=0x7efd38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:53.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:53.829+0000 7efd3e000640 1 -- 172.21.15.39:0/1942242455 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7efd40053ca0 con 0x7efd4806ee70 2024-01-24T17:48:54.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.021+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efd4806d1a0 con 0x7efd4806ee70 2024-01-24T17:48:54.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.023+0000 7efd3e000640 1 -- 172.21.15.39:0/1942242455 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7efd400572c0 con 0x7efd4806ee70 2024-01-24T17:48:54.024 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2006a450 msgr2=0x7efd2006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2006a450 0x7efd2006c910 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7efd38000ef0 tx=0x7efd38009040 comp rx=0 tx=0).stop 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4806ee70 msgr2=0x7efd4806f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4806ee70 0x7efd4806f300 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7efd48072150 tx=0x7efd4000e530 comp rx=0 tx=0).stop 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 shutdown_connections 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7efd2006a450 0x7efd2006c910 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd4806ee70 0x7efd4806f300 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efd4815b0b0 0x7efd480ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 --2- 172.21.15.39:0/1942242455 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7efd280047c0 0x7efd481566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 >> 172.21.15.39:0/1942242455 conn(0x7efd480d0840 msgr2=0x7efd48158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:54.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 shutdown_connections 2024-01-24T17:48:54.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.026+0000 7efd4cc00640 1 -- 172.21.15.39:0/1942242455 wait complete. 2024-01-24T17:48:54.047 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:48:54.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.270+0000 7fa1f1a00640 1 Processor -- start 2024-01-24T17:48:54.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.271+0000 7fa1f1a00640 1 -- start start 2024-01-24T17:48:54.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.272+0000 7fa1f1a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1ec0cdc20 0x7fa1ec0ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.272+0000 7fa1f1a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1ec0cd400 0x7fa1ec0cba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.272+0000 7fa1f1a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0cb5c0 con 0x7fa1ec15cb30 2024-01-24T17:48:54.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.272+0000 7fa1f1a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cd890 con 0x7fa1ec0cd400 2024-01-24T17:48:54.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.272+0000 7fa1f1a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cfb70 con 0x7fa1ec0cdc20 2024-01-24T17:48:54.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.273+0000 7fa1f1000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1ec0cd400 0x7fa1ec0cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.273+0000 7fa1f1000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fa1ec0cd400 0x7fa1ec0cba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57224/0 (socket says 172.21.15.39:57224) 2024-01-24T17:48:54.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.273+0000 7fa1f1000640 1 -- 172.21.15.39:0/2273012165 learned_addr learned my addr 172.21.15.39:0/2273012165 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:54.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.273+0000 7fa1f1000640 1 -- 172.21.15.39:0/2273012165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1ec0cdc20 msgr2=0x7fa1ec0ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:54.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.273+0000 7fa1f1000640 1 --2- 172.21.15.39:0/2273012165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1ec0cdc20 0x7fa1ec0ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.273+0000 7fa1f1000640 1 -- 172.21.15.39:0/2273012165 >> v1:172.21.15.100:6789/0 conn(0x7fa1ec15cb30 legacy=0x7fa1ec0cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:54.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.274+0000 7fa1f1000640 1 -- 172.21.15.39:0/2273012165 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0cac90 con 0x7fa1ec0cd400 2024-01-24T17:48:54.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.275+0000 7fa1f1000640 1 --2- 172.21.15.39:0/2273012165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec0cba50 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009870 tx=0x7fa1e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef554c8d54d85109 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:54.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.275+0000 7fa1eaa00640 1 -- 172.21.15.39:0/2273012165 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4030840 con 0x7fa1ec0cd400 2024-01-24T17:48:54.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.275+0000 7fa1eaa00640 1 -- 172.21.15.39:0/2273012165 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40309a0 con 0x7fa1ec0cd400 2024-01-24T17:48:54.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.275+0000 7fa1eaa00640 1 -- 172.21.15.39:0/2273012165 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4030d30 con 0x7fa1ec0cd400 2024-01-24T17:48:54.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.276+0000 7fa1f1a00640 1 -- 172.21.15.39:0/2273012165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 msgr2=0x7fa1ec0cba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.276+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/2273012165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec0cba50 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009870 tx=0x7fa1e402b980 comp rx=0 tx=0).stop 2024-01-24T17:48:54.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.276+0000 7fa1f1a00640 1 -- 172.21.15.39:0/2273012165 shutdown_connections 2024-01-24T17:48:54.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.276+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/2273012165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec0cba50 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.276+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/2273012165 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1ec0cdc20 0x7fa1ec0ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.276+0000 7fa1f1a00640 1 -- 172.21.15.39:0/2273012165 >> 172.21.15.39:0/2273012165 conn(0x7fa1ec0d0640 msgr2=0x7fa1ec0d0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:54.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.276+0000 7fa1f1a00640 1 -- 172.21.15.39:0/2273012165 shutdown_connections 2024-01-24T17:48:54.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.276+0000 7fa1f1a00640 1 -- 172.21.15.39:0/2273012165 wait complete. 2024-01-24T17:48:54.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.278+0000 7fa1f1a00640 1 Processor -- start 2024-01-24T17:48:54.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.278+0000 7fa1f1a00640 1 -- start start 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1f1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1f1a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1ec0cdc20 0x7fa1ec1555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1f1a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1ec15cb30 0x7fa1ec0c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1f1a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0b2060 con 0x7fa1ec0cd400 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1f1a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1562e0 con 0x7fa1ec0cdc20 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1f1a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec155b00 con 0x7fa1ec15cb30 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1ebe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1ebe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57230/0 (socket says 172.21.15.39:57230) 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1ebe00640 1 -- 172.21.15.39:0/3815744273 learned_addr learned my addr 172.21.15.39:0/3815744273 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1ebe00640 1 -- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1ec15cb30 msgr2=0x7fa1ec0c68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1ebe00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1ec15cb30 0x7fa1ec0c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1ebe00640 1 -- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1ec0cdc20 msgr2=0x7fa1ec1555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1ebe00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1ec0cdc20 0x7fa1ec1555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.279+0000 7fa1ebe00640 1 -- 172.21.15.39:0/3815744273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e40094d0 con 0x7fa1ec0cd400 2024-01-24T17:48:54.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.280+0000 7fa1ebe00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec155da0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c450 tx=0x7fa1dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:54.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.280+0000 7fa1e3e00640 1 -- 172.21.15.39:0/3815744273 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0052d0 con 0x7fa1ec0cd400 2024-01-24T17:48:54.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.280+0000 7fa1e3e00640 1 -- 172.21.15.39:0/3815744273 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc005450 con 0x7fa1ec0cd400 2024-01-24T17:48:54.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.280+0000 7fa1e3e00640 1 -- 172.21.15.39:0/3815744273 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc003880 con 0x7fa1ec0cd400 2024-01-24T17:48:54.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.280+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec0c5840 con 0x7fa1ec0cd400 2024-01-24T17:48:54.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.280+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec0c5e00 con 0x7fa1ec0cd400 2024-01-24T17:48:54.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.281+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec04e840 con 0x7fa1ec0cd400 2024-01-24T17:48:54.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.282+0000 7fa1e3e00640 1 -- 172.21.15.39:0/3815744273 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa1dc0039e0 con 0x7fa1ec0cd400 2024-01-24T17:48:54.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.283+0000 7fa1e3e00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1c406a460 0x7fa1c406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.283+0000 7fa1e3e00640 1 -- 172.21.15.39:0/3815744273 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1dc088460 con 0x7fa1ec0cd400 2024-01-24T17:48:54.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.284+0000 7fa1eb400640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1c406a460 0x7fa1c406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.285+0000 7fa1eb400640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1c406a460 0x7fa1c406c920 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec15b8c0 tx=0x7fa1d4009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:54.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.288+0000 7fa1e3e00640 1 -- 172.21.15.39:0/3815744273 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa1dc053a40 con 0x7fa1ec0cd400 2024-01-24T17:48:54.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.485+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1ec0ca970 con 0x7fa1ec0cd400 2024-01-24T17:48:54.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.486+0000 7fa1e3e00640 1 -- 172.21.15.39:0/3815744273 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1dc057060 con 0x7fa1ec0cd400 2024-01-24T17:48:54.488 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:48:54.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.488+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1c406a460 msgr2=0x7fa1c406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.488+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1c406a460 0x7fa1c406c920 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec15b8c0 tx=0x7fa1d4009250 comp rx=0 tx=0).stop 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.488+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 msgr2=0x7fa1ec155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.488+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec155da0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c450 tx=0x7fa1dc00c910 comp rx=0 tx=0).stop 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.488+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 shutdown_connections 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.488+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa1ec15cb30 0x7fa1ec0c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.489+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1ec0cdc20 0x7fa1ec1555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.489+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa1c406a460 0x7fa1c406c920 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.489+0000 7fa1f1a00640 1 --2- 172.21.15.39:0/3815744273 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1ec0cd400 0x7fa1ec155da0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.489+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 >> 172.21.15.39:0/3815744273 conn(0x7fa1ec0d0640 msgr2=0x7fa1ec0af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.489+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 shutdown_connections 2024-01-24T17:48:54.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.489+0000 7fa1f1a00640 1 -- 172.21.15.39:0/3815744273 wait complete. 2024-01-24T17:48:54.510 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:48:54.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.733+0000 7f2942a00640 1 Processor -- start 2024-01-24T17:48:54.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.733+0000 7f2942a00640 1 -- start start 2024-01-24T17:48:54.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.734+0000 7f2942a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c0cadc0 0x7f293c15a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.734+0000 7f2942a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f293c0ca6d0 0x7f293c15b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.734+0000 7f2942a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293c15b040 con 0x7f293c06bd40 2024-01-24T17:48:54.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.734+0000 7f2942a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293c06c1a0 con 0x7f293c0ca6d0 2024-01-24T17:48:54.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.734+0000 7f2942a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c06c420 con 0x7f293c0cadc0 2024-01-24T17:48:54.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f2940c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f293c06bd40 0x7f293c0cb1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58302/0 (socket says 172.21.15.39:58302) 2024-01-24T17:48:54.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f2940c00640 1 -- 172.21.15.39:0/1362568945 learned_addr learned my addr 172.21.15.39:0/1362568945 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:54.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f2942000640 1 --2- 172.21.15.39:0/1362568945 >> v2:172.21.15.39:3300/0 conn(0x7f293c0ca6d0 0x7f293c15b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f2941600640 1 --2- 172.21.15.39:0/1362568945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c0cadc0 0x7f293c15a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f293be00640 1 -- 172.21.15.39:0/1362568945 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660543857 0 0) 0x7f293c15b040 con 0x7f293c06bd40 2024-01-24T17:48:54.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f293be00640 1 -- 172.21.15.39:0/1362568945 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f291c003520 con 0x7f293c06bd40 2024-01-24T17:48:54.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f2942000640 1 -- 172.21.15.39:0/1362568945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c0cadc0 msgr2=0x7f293c15a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:54.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f2942000640 1 --2- 172.21.15.39:0/1362568945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c0cadc0 0x7f293c15a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.735+0000 7f2942000640 1 -- 172.21.15.39:0/1362568945 >> v1:172.21.15.39:6789/0 conn(0x7f293c06bd40 legacy=0x7f293c0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.736+0000 7f2942000640 1 -- 172.21.15.39:0/1362568945 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c06c700 con 0x7f293c0ca6d0 2024-01-24T17:48:54.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.737+0000 7f2942000640 1 --2- 172.21.15.39:0/1362568945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c15b4d0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2934009d90 tx=0x7f293402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa11c722c20cf31c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:54.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.737+0000 7f293be00640 1 -- 172.21.15.39:0/1362568945 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293402cbc0 con 0x7f293c0ca6d0 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.737+0000 7f293be00640 1 -- 172.21.15.39:0/1362568945 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293402cd20 con 0x7f293c0ca6d0 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.737+0000 7f293be00640 1 -- 172.21.15.39:0/1362568945 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934034660 con 0x7f293c0ca6d0 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.737+0000 7f2942a00640 1 -- 172.21.15.39:0/1362568945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 msgr2=0x7f293c15b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.737+0000 7f2942a00640 1 --2- 172.21.15.39:0/1362568945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c15b4d0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2934009d90 tx=0x7f293402be10 comp rx=0 tx=0).stop 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.738+0000 7f2942a00640 1 -- 172.21.15.39:0/1362568945 shutdown_connections 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.738+0000 7f2942a00640 1 --2- 172.21.15.39:0/1362568945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c15b4d0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.738+0000 7f2942a00640 1 --2- 172.21.15.39:0/1362568945 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c0cadc0 0x7f293c15a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.738+0000 7f2942a00640 1 -- 172.21.15.39:0/1362568945 >> 172.21.15.39:0/1362568945 conn(0x7f293c06a200 msgr2=0x7f293c0aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.738+0000 7f2942a00640 1 -- 172.21.15.39:0/1362568945 shutdown_connections 2024-01-24T17:48:54.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.738+0000 7f2942a00640 1 -- 172.21.15.39:0/1362568945 wait complete. 2024-01-24T17:48:54.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.739+0000 7f2942a00640 1 Processor -- start 2024-01-24T17:48:54.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.740+0000 7f2942a00640 1 -- start start 2024-01-24T17:48:54.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2942a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c06bd40 0x7f293c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2942a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c0ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2942a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f293c0cadc0 0x7f293c0c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2942a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0ad200 con 0x7f293c0ca6d0 2024-01-24T17:48:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2942a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0acc40 con 0x7f293c0cadc0 2024-01-24T17:48:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2942a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f293c0acf20 con 0x7f293c06bd40 2024-01-24T17:48:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2940c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c0ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2940c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c0ac130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57252/0 (socket says 172.21.15.39:57252) 2024-01-24T17:48:54.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2940c00640 1 -- 172.21.15.39:0/911988265 learned_addr learned my addr 172.21.15.39:0/911988265 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2941600640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c06bd40 0x7f293c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2942000640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f293c0cadc0 0x7f293c0c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2940c00640 1 -- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c06bd40 msgr2=0x7f293c0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2940c00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c06bd40 0x7f293c0b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2940c00640 1 -- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f293c0cadc0 msgr2=0x7f293c0c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2940c00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f293c0cadc0 0x7f293c0c77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.741+0000 7f2940c00640 1 -- 172.21.15.39:0/911988265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2934009a70 con 0x7f293c0ca6d0 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.742+0000 7f2940c00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c0ac130 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f292400c500 tx=0x7f292400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.742+0000 7f2939600640 1 -- 172.21.15.39:0/911988265 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2924005ce0 con 0x7f293c0ca6d0 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.742+0000 7f2939600640 1 -- 172.21.15.39:0/911988265 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2924005e40 con 0x7f293c0ca6d0 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.742+0000 7f2939600640 1 -- 172.21.15.39:0/911988265 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2924007330 con 0x7f293c0ca6d0 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.742+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c0cc0b0 con 0x7f293c0ca6d0 2024-01-24T17:48:54.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.742+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c14e880 con 0x7f293c0ca6d0 2024-01-24T17:48:54.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.743+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c04dff0 con 0x7f293c0ca6d0 2024-01-24T17:48:54.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.744+0000 7f2939600640 1 -- 172.21.15.39:0/911988265 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2924007550 con 0x7f293c0ca6d0 2024-01-24T17:48:54.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.745+0000 7f2939600640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f291406a430 0x7f291406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:54.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.745+0000 7f2939600640 1 -- 172.21.15.39:0/911988265 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2924087fb0 con 0x7f293c0ca6d0 2024-01-24T17:48:54.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.746+0000 7f2941600640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f291406a430 0x7f291406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:54.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.747+0000 7f2941600640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f291406a430 0x7f291406c8f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f292c0096d0 tx=0x7f292c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:54.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.750+0000 7f2939600640 1 -- 172.21.15.39:0/911988265 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2924053650 con 0x7f293c0ca6d0 2024-01-24T17:48:54.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.955+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f293c14eb00 con 0x7f293c0ca6d0 2024-01-24T17:48:54.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.956+0000 7f2939600640 1 -- 172.21.15.39:0/911988265 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v100) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2924017080 con 0x7f293c0ca6d0 2024-01-24T17:48:54.958 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.958+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f291406a430 msgr2=0x7f291406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.958+0000 7f2942a00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f291406a430 0x7f291406c8f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f292c0096d0 tx=0x7f292c008040 comp rx=0 tx=0).stop 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.958+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 msgr2=0x7f293c0ac130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.958+0000 7f2942a00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c0ac130 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f292400c500 tx=0x7f292400c9c0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.959+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 shutdown_connections 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.959+0000 7f2942a00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f293c0cadc0 0x7f293c0c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.959+0000 7f2942a00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f291406a430 0x7f291406c8f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.959+0000 7f2942a00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f293c0ca6d0 0x7f293c0ac130 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.959+0000 7f2942a00640 1 --2- 172.21.15.39:0/911988265 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f293c06bd40 0x7f293c0b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.959+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 >> 172.21.15.39:0/911988265 conn(0x7f293c06a200 msgr2=0x7f293c15d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.959+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 shutdown_connections 2024-01-24T17:48:54.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:54.959+0000 7f2942a00640 1 -- 172.21.15.39:0/911988265 wait complete. 2024-01-24T17:48:54.980 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:48:54.980 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:48:55.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:48:55.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.266+0000 7f1504600640 1 Processor -- start 2024-01-24T17:48:55.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.266+0000 7f1504600640 1 -- start start 2024-01-24T17:48:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.267+0000 7f1504600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f14fc0b0010 0x7f14fc15bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.267+0000 7f1504600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f14fc15f390 0x7f14fc0c9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.267+0000 7f1504600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14fc070470 con 0x7f14fc070920 2024-01-24T17:48:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.267+0000 7f1504600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14fc15ef00 con 0x7f14fc0b0010 2024-01-24T17:48:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1504600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14fc0710a0 con 0x7f14fc15f390 2024-01-24T17:48:55.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1501200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f14fc070920 0x7f14fc0b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58304/0 (socket says 172.21.15.39:58304) 2024-01-24T17:48:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1501200640 1 -- 172.21.15.39:0/3367905429 learned_addr learned my addr 172.21.15.39:0/3367905429 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1501c00640 1 --2- 172.21.15.39:0/3367905429 >> v2:172.21.15.39:3300/0 conn(0x7f14fc0b0010 0x7f14fc15bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1502600640 1 --2- 172.21.15.39:0/3367905429 >> v2:172.21.15.100:3300/0 conn(0x7f14fc15f390 0x7f14fc0c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1501c00640 1 -- 172.21.15.39:0/3367905429 >> v2:172.21.15.100:3300/0 conn(0x7f14fc15f390 msgr2=0x7f14fc0c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1501c00640 1 --2- 172.21.15.39:0/3367905429 >> v2:172.21.15.100:3300/0 conn(0x7f14fc15f390 0x7f14fc0c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1501c00640 1 -- 172.21.15.39:0/3367905429 >> v1:172.21.15.39:6789/0 conn(0x7f14fc070920 legacy=0x7f14fc0b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.268+0000 7f1501c00640 1 -- 172.21.15.39:0/3367905429 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc15c0d0 con 0x7f14fc0b0010 2024-01-24T17:48:55.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.270+0000 7f1501c00640 1 --2- 172.21.15.39:0/3367905429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc0b0010 0x7f14fc15bb90 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0c55e0 tx=0x7f14ec02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191bc5a3782a272b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:55.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.270+0000 7f14fbe00640 1 -- 172.21.15.39:0/3367905429 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec033640 con 0x7f14fc0b0010 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.270+0000 7f14fbe00640 1 -- 172.21.15.39:0/3367905429 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec0337a0 con 0x7f14fc0b0010 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.270+0000 7f14fbe00640 1 -- 172.21.15.39:0/3367905429 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec033b30 con 0x7f14fc0b0010 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.271+0000 7f1504600640 1 -- 172.21.15.39:0/3367905429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc0b0010 msgr2=0x7f14fc15bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.271+0000 7f1504600640 1 --2- 172.21.15.39:0/3367905429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc0b0010 0x7f14fc15bb90 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0c55e0 tx=0x7f14ec02b7a0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.271+0000 7f1504600640 1 -- 172.21.15.39:0/3367905429 shutdown_connections 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.271+0000 7f1504600640 1 --2- 172.21.15.39:0/3367905429 >> v2:172.21.15.100:3300/0 conn(0x7f14fc15f390 0x7f14fc0c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.271+0000 7f1504600640 1 --2- 172.21.15.39:0/3367905429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc0b0010 0x7f14fc15bb90 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.271+0000 7f1504600640 1 -- 172.21.15.39:0/3367905429 >> 172.21.15.39:0/3367905429 conn(0x7f14fc06bf20 msgr2=0x7f14fc06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.271+0000 7f1504600640 1 -- 172.21.15.39:0/3367905429 shutdown_connections 2024-01-24T17:48:55.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.271+0000 7f1504600640 1 -- 172.21.15.39:0/3367905429 wait complete. 2024-01-24T17:48:55.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.273+0000 7f1504600640 1 Processor -- start 2024-01-24T17:48:55.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.273+0000 7f1504600640 1 -- start start 2024-01-24T17:48:55.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1504600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14fc070920 0x7f14fc14a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1504600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14fc0b0010 0x7f14fc0acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1504600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc15f390 0x7f14fc0ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1504600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc0ae2b0 con 0x7f14fc15f390 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1504600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc0c6970 con 0x7f14fc0b0010 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1504600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc0ad6a0 con 0x7f14fc070920 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1502600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc15f390 0x7f14fc0ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1502600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc15f390 0x7f14fc0ad070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57260/0 (socket says 172.21.15.39:57260) 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1502600640 1 -- 172.21.15.39:0/1149184292 learned_addr learned my addr 172.21.15.39:0/1149184292 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1501200640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14fc0b0010 0x7f14fc0acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1501c00640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14fc070920 0x7f14fc14a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1502600640 1 -- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14fc070920 msgr2=0x7f14fc14a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1502600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14fc070920 0x7f14fc14a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1502600640 1 -- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14fc0b0010 msgr2=0x7f14fc0acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1502600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14fc0b0010 0x7f14fc0acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1502600640 1 -- 172.21.15.39:0/1149184292 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec009400 con 0x7f14fc15f390 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.274+0000 7f1501c00640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14fc070920 0x7f14fc14a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.275+0000 7f1502600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc15f390 0x7f14fc0ad070 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f14f4007e60 tx=0x7f14f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.275+0000 7f14f9600640 1 -- 172.21.15.39:0/1149184292 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4014420 con 0x7f14fc15f390 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.275+0000 7f14f9600640 1 -- 172.21.15.39:0/1149184292 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f40145a0 con 0x7f14fc15f390 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.275+0000 7f14f9600640 1 -- 172.21.15.39:0/1149184292 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4014950 con 0x7f14fc15f390 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.275+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14fc0ad980 con 0x7f14fc15f390 2024-01-24T17:48:55.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.275+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc0adf40 con 0x7f14fc15f390 2024-01-24T17:48:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.276+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14fc0ae420 con 0x7f14fc15f390 2024-01-24T17:48:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.277+0000 7f14f9600640 1 -- 172.21.15.39:0/1149184292 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f14f4014ab0 con 0x7f14fc15f390 2024-01-24T17:48:55.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.278+0000 7f14f9600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14d406a430 0x7f14d406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.278+0000 7f14f9600640 1 -- 172.21.15.39:0/1149184292 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14f40892d0 con 0x7f14fc15f390 2024-01-24T17:48:55.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.279+0000 7f1501c00640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14d406a430 0x7f14d406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.280+0000 7f1501c00640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14d406a430 0x7f14d406c8f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f14ec035000 tx=0x7f14ec02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:55.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.283+0000 7f14f9600640 1 -- 172.21.15.39:0/1149184292 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f14f4054820 con 0x7f14fc15f390 2024-01-24T17:48:55.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.478+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14fc04dff0 con 0x7f14fc15f390 2024-01-24T17:48:55.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.481+0000 7f14f9600640 1 -- 172.21.15.39:0/1149184292 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16042 (secure 0 0 0) 0x7f14f4057e40 con 0x7f14fc15f390 2024-01-24T17:48:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:55.483 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":100,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:48:27.885313+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:48:55.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14d406a430 msgr2=0x7f14d406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14d406a430 0x7f14d406c8f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f14ec035000 tx=0x7f14ec02f000 comp rx=0 tx=0).stop 2024-01-24T17:48:55.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc15f390 msgr2=0x7f14fc0ad070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc15f390 0x7f14fc0ad070 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f14f4007e60 tx=0x7f14f400e4b0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 shutdown_connections 2024-01-24T17:48:55.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f14d406a430 0x7f14d406c8f0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14fc15f390 0x7f14fc0ad070 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14fc0b0010 0x7f14fc0acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 --2- 172.21.15.39:0/1149184292 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f14fc070920 0x7f14fc14a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 >> 172.21.15.39:0/1149184292 conn(0x7f14fc06bf20 msgr2=0x7f14fc15cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:55.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 shutdown_connections 2024-01-24T17:48:55.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.484+0000 7f1504600640 1 -- 172.21.15.39:0/1149184292 wait complete. 2024-01-24T17:48:55.507 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:48:55.508 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:48:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.737+0000 7ff850a00640 1 Processor -- start 2024-01-24T17:48:55.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.737+0000 7ff850a00640 1 -- start start 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.739+0000 7ff850a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 0x7ff84c0b4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.739+0000 7ff850a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff84c15cff0 0x7ff84c0d0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.739+0000 7ff850a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c0b3c30 con 0x7ff84c15cc10 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.739+0000 7ff850a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c0cdf50 con 0x7ff84c15cff0 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.739+0000 7ff850a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0ce230 con 0x7ff84c0c74d0 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.739+0000 7ff84b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff84c15cff0 0x7ff84c0d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.740+0000 7ff84b200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff84c15cff0 0x7ff84c0d0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57266/0 (socket says 172.21.15.39:57266) 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.740+0000 7ff84b200640 1 -- 172.21.15.39:0/88912444 learned_addr learned my addr 172.21.15.39:0/88912444 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.740+0000 7ff841c00640 1 --2- 172.21.15.39:0/88912444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 0x7ff84c0b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.740+0000 7ff84b200640 1 -- 172.21.15.39:0/88912444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 msgr2=0x7ff84c0b4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.740+0000 7ff84b200640 1 --2- 172.21.15.39:0/88912444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 0x7ff84c0b4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.740+0000 7ff84b200640 1 -- 172.21.15.39:0/88912444 >> v1:172.21.15.100:6789/0 conn(0x7ff84c15cc10 legacy=0x7ff84c0b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.740+0000 7ff84b200640 1 -- 172.21.15.39:0/88912444 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c0ce510 con 0x7ff84c15cff0 2024-01-24T17:48:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.742+0000 7ff84b200640 1 --2- 172.21.15.39:0/88912444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cff0 0x7ff84c0d0550 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7ff844009e50 tx=0x7ff84402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607d49f40111acc server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:55.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.742+0000 7ff849e00640 1 -- 172.21.15.39:0/88912444 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84402ca70 con 0x7ff84c15cff0 2024-01-24T17:48:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.742+0000 7ff849e00640 1 -- 172.21.15.39:0/88912444 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84402cbd0 con 0x7ff84c15cff0 2024-01-24T17:48:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.742+0000 7ff849e00640 1 -- 172.21.15.39:0/88912444 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8440343f0 con 0x7ff84c15cff0 2024-01-24T17:48:55.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.743+0000 7ff850a00640 1 -- 172.21.15.39:0/88912444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cff0 msgr2=0x7ff84c0d0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.743+0000 7ff850a00640 1 --2- 172.21.15.39:0/88912444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cff0 0x7ff84c0d0550 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7ff844009e50 tx=0x7ff84402bed0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.743+0000 7ff850a00640 1 -- 172.21.15.39:0/88912444 shutdown_connections 2024-01-24T17:48:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.743+0000 7ff850a00640 1 --2- 172.21.15.39:0/88912444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cff0 0x7ff84c0d0550 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.743+0000 7ff850a00640 1 --2- 172.21.15.39:0/88912444 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 0x7ff84c0b4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.743+0000 7ff850a00640 1 -- 172.21.15.39:0/88912444 >> 172.21.15.39:0/88912444 conn(0x7ff84c0d2800 msgr2=0x7ff84c0d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.743+0000 7ff850a00640 1 -- 172.21.15.39:0/88912444 shutdown_connections 2024-01-24T17:48:55.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.743+0000 7ff850a00640 1 -- 172.21.15.39:0/88912444 wait complete. 2024-01-24T17:48:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.745+0000 7ff850a00640 1 Processor -- start 2024-01-24T17:48:55.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.745+0000 7ff850a00640 1 -- start start 2024-01-24T17:48:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.746+0000 7ff850a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 0x7ff84c1e56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.746+0000 7ff850a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cc10 0x7ff84c1e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.746+0000 7ff850a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff84c15cff0 0x7ff84c0cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.746+0000 7ff850a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0b7a70 con 0x7ff84c15cc10 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.746+0000 7ff850a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c159e30 con 0x7ff84c15cff0 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.746+0000 7ff850a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c1e5bf0 con 0x7ff84c0c74d0 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cc10 0x7ff84c1e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cc10 0x7ff84c1e6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57276/0 (socket says 172.21.15.39:57276) 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 -- 172.21.15.39:0/2877015257 learned_addr learned my addr 172.21.15.39:0/2877015257 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff84b200640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff84c15cff0 0x7ff84c0cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff84a800640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 0x7ff84c1e56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 -- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 msgr2=0x7ff84c1e56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 0x7ff84c1e56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 -- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff84c15cff0 msgr2=0x7ff84c0cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff84c15cff0 0x7ff84c0cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 -- 172.21.15.39:0/2877015257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff844009b30 con 0x7ff84c15cc10 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff84b200640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff84c15cff0 0x7ff84c0cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.747+0000 7ff841c00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cc10 0x7ff84c1e6bc0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7ff83400e8a0 tx=0x7ff83400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.748+0000 7ff843400640 1 -- 172.21.15.39:0/2877015257 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8340094f0 con 0x7ff84c15cc10 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.748+0000 7ff843400640 1 -- 172.21.15.39:0/2877015257 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83400c8e0 con 0x7ff84c15cc10 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.748+0000 7ff843400640 1 -- 172.21.15.39:0/2877015257 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83400cc90 con 0x7ff84c15cc10 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.748+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c0b69d0 con 0x7ff84c15cc10 2024-01-24T17:48:55.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.748+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c0b6f90 con 0x7ff84c15cc10 2024-01-24T17:48:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.749+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c0b5290 con 0x7ff84c15cc10 2024-01-24T17:48:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.750+0000 7ff843400640 1 -- 172.21.15.39:0/2877015257 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff834013470 con 0x7ff84c15cc10 2024-01-24T17:48:55.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.751+0000 7ff843400640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff82406a430 0x7ff82406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.751+0000 7ff843400640 1 -- 172.21.15.39:0/2877015257 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff834088b20 con 0x7ff84c15cc10 2024-01-24T17:48:55.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.752+0000 7ff84a800640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff82406a430 0x7ff82406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:55.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.753+0000 7ff84a800640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff82406a430 0x7ff82406c8f0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff83c000fb0 tx=0x7ff83c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:55.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.756+0000 7ff843400640 1 -- 172.21.15.39:0/2877015257 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff834053fe0 con 0x7ff84c15cc10 2024-01-24T17:48:55.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.949+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff84c04e840 con 0x7ff84c15cc10 2024-01-24T17:48:55.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.950+0000 7ff843400640 1 -- 172.21.15.39:0/2877015257 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff834021020 con 0x7ff84c15cc10 2024-01-24T17:48:55.952 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff82406a430 msgr2=0x7ff82406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff82406a430 0x7ff82406c8f0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff83c000fb0 tx=0x7ff83c009040 comp rx=0 tx=0).stop 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cc10 msgr2=0x7ff84c1e6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cc10 0x7ff84c1e6bc0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7ff83400e8a0 tx=0x7ff83400ed60 comp rx=0 tx=0).stop 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 shutdown_connections 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff84c15cff0 0x7ff84c0cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff82406a430 0x7ff82406c8f0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84c15cc10 0x7ff84c1e6bc0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 --2- 172.21.15.39:0/2877015257 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff84c0c74d0 0x7ff84c1e56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:55.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 >> 172.21.15.39:0/2877015257 conn(0x7ff84c0d2800 msgr2=0x7ff84c15bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:55.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 shutdown_connections 2024-01-24T17:48:55.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:55.953+0000 7ff850a00640 1 -- 172.21.15.39:0/2877015257 wait complete. 2024-01-24T17:48:55.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:48:56.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.199+0000 7f9dab600640 1 Processor -- start 2024-01-24T17:48:56.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.199+0000 7f9dab600640 1 -- start start 2024-01-24T17:48:56.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.200+0000 7f9dab600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9da4159950 0x7f9da40ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.200+0000 7f9dab600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da40cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.200+0000 7f9dab600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da406c000 con 0x7f9da40cb7c0 2024-01-24T17:48:56.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.200+0000 7f9dab600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da40cbc50 con 0x7f9da4159950 2024-01-24T17:48:56.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.200+0000 7f9dab600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da406d010 con 0x7f9da40ca5f0 2024-01-24T17:48:56.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da8c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9da4159950 0x7f9da40ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da8c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9da4159950 0x7f9da40ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57286/0 (socket says 172.21.15.39:57286) 2024-01-24T17:48:56.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da8c00640 1 -- 172.21.15.39:0/2655078234 learned_addr learned my addr 172.21.15.39:0/2655078234 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:56.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da3e00640 1 --2- 172.21.15.39:0/2655078234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da40cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da8c00640 1 -- 172.21.15.39:0/2655078234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 msgr2=0x7f9da40cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:56.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da8c00640 1 --2- 172.21.15.39:0/2655078234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da40cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da8c00640 1 -- 172.21.15.39:0/2655078234 >> v1:172.21.15.100:6789/0 conn(0x7f9da40cb7c0 legacy=0x7f9da415d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:56.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da8c00640 1 -- 172.21.15.39:0/2655078234 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da40cae00 con 0x7f9da4159950 2024-01-24T17:48:56.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.201+0000 7f9da3e00640 1 --2- 172.21.15.39:0/2655078234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da40cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:56.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.203+0000 7f9da8c00640 1 --2- 172.21.15.39:0/2655078234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da4159950 0x7f9da40ad9a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009870 tx=0x7f9d9402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fdc524393eba50 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:56.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.203+0000 7f9da3400640 1 -- 172.21.15.39:0/2655078234 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94030840 con 0x7f9da4159950 2024-01-24T17:48:56.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.203+0000 7f9da3400640 1 -- 172.21.15.39:0/2655078234 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d940309a0 con 0x7f9da4159950 2024-01-24T17:48:56.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.203+0000 7f9da3400640 1 -- 172.21.15.39:0/2655078234 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d94030d30 con 0x7f9da4159950 2024-01-24T17:48:56.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.204+0000 7f9dab600640 1 -- 172.21.15.39:0/2655078234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da4159950 msgr2=0x7f9da40ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.204+0000 7f9dab600640 1 --2- 172.21.15.39:0/2655078234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da4159950 0x7f9da40ad9a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009870 tx=0x7f9d9402b980 comp rx=0 tx=0).stop 2024-01-24T17:48:56.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.204+0000 7f9dab600640 1 -- 172.21.15.39:0/2655078234 shutdown_connections 2024-01-24T17:48:56.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.204+0000 7f9dab600640 1 --2- 172.21.15.39:0/2655078234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da40cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.204+0000 7f9dab600640 1 --2- 172.21.15.39:0/2655078234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da4159950 0x7f9da40ad9a0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.204+0000 7f9dab600640 1 -- 172.21.15.39:0/2655078234 >> 172.21.15.39:0/2655078234 conn(0x7f9da40ce9e0 msgr2=0x7f9da40cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:56.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.204+0000 7f9dab600640 1 -- 172.21.15.39:0/2655078234 shutdown_connections 2024-01-24T17:48:56.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.204+0000 7f9dab600640 1 -- 172.21.15.39:0/2655078234 wait complete. 2024-01-24T17:48:56.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.206+0000 7f9dab600640 1 Processor -- start 2024-01-24T17:48:56.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.206+0000 7f9dab600640 1 -- start start 2024-01-24T17:48:56.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9dab600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da41f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9dab600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da40cb7c0 0x7f9da40ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9dab600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da4159950 0x7f9da40acde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9dab600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da4158c60 con 0x7f9da40cb7c0 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9dab600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da4158050 con 0x7f9da4159950 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9dab600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da4158330 con 0x7f9da40ca5f0 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9da3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da40cb7c0 0x7f9da40ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9da3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da40cb7c0 0x7f9da40ac8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57288/0 (socket says 172.21.15.39:57288) 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9da3e00640 1 -- 172.21.15.39:0/2025891377 learned_addr learned my addr 172.21.15.39:0/2025891377 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9da8c00640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da41f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.208+0000 7f9da9600640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da4159950 0x7f9da40acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da3e00640 1 -- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 msgr2=0x7f9da41f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da3e00640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da41f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da3e00640 1 -- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da4159950 msgr2=0x7f9da40acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da3e00640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da4159950 0x7f9da40acde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da3e00640 1 -- 172.21.15.39:0/2025891377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940094d0 con 0x7f9da40cb7c0 2024-01-24T17:48:56.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da3e00640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da40cb7c0 0x7f9da40ac8a0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00c7f0 tx=0x7f9d8c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:56.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da0c00640 1 -- 172.21.15.39:0/2025891377 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c007820 con 0x7f9da40cb7c0 2024-01-24T17:48:56.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da0c00640 1 -- 172.21.15.39:0/2025891377 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8c007980 con 0x7f9da40cb7c0 2024-01-24T17:48:56.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9da0c00640 1 -- 172.21.15.39:0/2025891377 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c007cf0 con 0x7f9da40cb7c0 2024-01-24T17:48:56.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.209+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da4158610 con 0x7f9da40cb7c0 2024-01-24T17:48:56.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.210+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da4097e20 con 0x7f9da40cb7c0 2024-01-24T17:48:56.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.211+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da404dff0 con 0x7f9da40cb7c0 2024-01-24T17:48:56.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.212+0000 7f9da0c00640 1 -- 172.21.15.39:0/2025891377 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9d8c0113f0 con 0x7f9da40cb7c0 2024-01-24T17:48:56.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.213+0000 7f9da0c00640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9d7c06a430 0x7f9d7c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.213+0000 7f9da0c00640 1 -- 172.21.15.39:0/2025891377 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d8c088c40 con 0x7f9da40cb7c0 2024-01-24T17:48:56.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.214+0000 7f9da8c00640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9d7c06a430 0x7f9d7c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.215+0000 7f9da8c00640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9d7c06a430 0x7f9d7c06c8f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9d94036000 tx=0x7f9d94008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:56.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.218+0000 7f9da0c00640 1 -- 172.21.15.39:0/2025891377 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9d8c054230 con 0x7f9da40cb7c0 2024-01-24T17:48:56.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.410+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9da406dbb0 con 0x7f9da40cb7c0 2024-01-24T17:48:56.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.411+0000 7f9da0c00640 1 -- 172.21.15.39:0/2025891377 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9d8c01f020 con 0x7f9da40cb7c0 2024-01-24T17:48:56.413 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9d7c06a430 msgr2=0x7f9d7c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9d7c06a430 0x7f9d7c06c8f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9d94036000 tx=0x7f9d94008030 comp rx=0 tx=0).stop 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da40cb7c0 msgr2=0x7f9da40ac8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da40cb7c0 0x7f9da40ac8a0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00c7f0 tx=0x7f9d8c00ccb0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 shutdown_connections 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9da4159950 0x7f9da40acde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9d7c06a430 0x7f9d7c06c8f0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9da40cb7c0 0x7f9da40ac8a0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 --2- 172.21.15.39:0/2025891377 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9da40ca5f0 0x7f9da41f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 >> 172.21.15.39:0/2025891377 conn(0x7f9da40ce9e0 msgr2=0x7f9da40ae9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:56.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 shutdown_connections 2024-01-24T17:48:56.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.414+0000 7f9dab600640 1 -- 172.21.15.39:0/2025891377 wait complete. 2024-01-24T17:48:56.437 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:48:56.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.654+0000 7f18fdc00640 1 Processor -- start 2024-01-24T17:48:56.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.655+0000 7f18fdc00640 1 -- start start 2024-01-24T17:48:56.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.656+0000 7f18fdc00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18f806acd0 0x7f18f806c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.656+0000 7f18fdc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f806bde0 con 0x7f18f80b4310 2024-01-24T17:48:56.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.656+0000 7f18fdc00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f806b1d0 con 0x7f18f80ce660 2024-01-24T17:48:56.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.656+0000 7f18fdc00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f80afaf0 con 0x7f18f806acd0 2024-01-24T17:48:56.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.656+0000 7f18fd200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18f806acd0 0x7f18f806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.656+0000 7f18f7e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f18f80b4310 0x7f18f80b46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58308/0 (socket says 172.21.15.39:58308) 2024-01-24T17:48:56.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.657+0000 7f18f7e00640 1 -- 172.21.15.39:0/2372920281 learned_addr learned my addr 172.21.15.39:0/2372920281 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:56.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.656+0000 7f18fd200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f18f806acd0 0x7f18f806c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57290/0 (socket says 172.21.15.39:57290) 2024-01-24T17:48:56.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.657+0000 7f18fd200640 1 -- 172.21.15.39:0/2372920281 >> v1:172.21.15.100:6789/0 conn(0x7f18f80ce660 legacy=0x7f18f80af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:56.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.657+0000 7f18fd200640 1 -- 172.21.15.39:0/2372920281 >> v1:172.21.15.39:6789/0 conn(0x7f18f80b4310 legacy=0x7f18f80b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.657+0000 7f18fd200640 1 -- 172.21.15.39:0/2372920281 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f806b4b0 con 0x7f18f806acd0 2024-01-24T17:48:56.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.658+0000 7f18fd200640 1 --2- 172.21.15.39:0/2372920281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f806c270 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009d90 tx=0x7f18f002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11655f77e728542f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:56.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.658+0000 7f18f6a00640 1 -- 172.21.15.39:0/2372920281 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f002cc60 con 0x7f18f806acd0 2024-01-24T17:48:56.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.658+0000 7f18f6a00640 1 -- 172.21.15.39:0/2372920281 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f002cdc0 con 0x7f18f806acd0 2024-01-24T17:48:56.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.658+0000 7f18f6a00640 1 -- 172.21.15.39:0/2372920281 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f0034660 con 0x7f18f806acd0 2024-01-24T17:48:56.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.659+0000 7f18fdc00640 1 -- 172.21.15.39:0/2372920281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 msgr2=0x7f18f806c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.659+0000 7f18fdc00640 1 --2- 172.21.15.39:0/2372920281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f806c270 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009d90 tx=0x7f18f002beb0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.659+0000 7f18fdc00640 1 -- 172.21.15.39:0/2372920281 shutdown_connections 2024-01-24T17:48:56.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.659+0000 7f18fdc00640 1 --2- 172.21.15.39:0/2372920281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f806c270 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.659+0000 7f18fdc00640 1 -- 172.21.15.39:0/2372920281 >> 172.21.15.39:0/2372920281 conn(0x7f18f80d2890 msgr2=0x7f18f80d2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:56.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.659+0000 7f18fdc00640 1 -- 172.21.15.39:0/2372920281 shutdown_connections 2024-01-24T17:48:56.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.659+0000 7f18fdc00640 1 -- 172.21.15.39:0/2372920281 wait complete. 2024-01-24T17:48:56.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.662+0000 7f18fdc00640 1 Processor -- start 2024-01-24T17:48:56.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.662+0000 7f18fdc00640 1 -- start start 2024-01-24T17:48:56.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.663+0000 7f18fdc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f81f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.663+0000 7f18fdc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18f80b4310 0x7f18f80b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.663+0000 7f18fdc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18f80ce660 0x7f18f80b7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.663+0000 7f18fdc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f80c80f0 con 0x7f18f806acd0 2024-01-24T17:48:56.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.663+0000 7f18fdc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f80c82e0 con 0x7f18f80ce660 2024-01-24T17:48:56.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.663+0000 7f18fdc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f80c85c0 con 0x7f18f80b4310 2024-01-24T17:48:56.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f81f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f81f17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57294/0 (socket says 172.21.15.39:57294) 2024-01-24T17:48:56.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7e00640 1 -- 172.21.15.39:0/2531661028 learned_addr learned my addr 172.21.15.39:0/2531661028 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7e00640 1 -- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18f80b4310 msgr2=0x7f18f80b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18fd200640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18f80ce660 0x7f18f80b7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7400640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18f80b4310 0x7f18f80b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7e00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18f80b4310 0x7f18f80b7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7e00640 1 -- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18f80ce660 msgr2=0x7f18f80b7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7e00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18f80ce660 0x7f18f80b7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18f7e00640 1 -- 172.21.15.39:0/2531661028 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f0009a70 con 0x7f18f806acd0 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.664+0000 7f18fd200640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18f80ce660 0x7f18f80b7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.665+0000 7f18f7e00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f81f17b0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f18e800c480 tx=0x7f18e800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:56.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.665+0000 7f18efe00640 1 -- 172.21.15.39:0/2531661028 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8007480 con 0x7f18f806acd0 2024-01-24T17:48:56.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.665+0000 7f18efe00640 1 -- 172.21.15.39:0/2531661028 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e80075e0 con 0x7f18f806acd0 2024-01-24T17:48:56.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.665+0000 7f18efe00640 1 -- 172.21.15.39:0/2531661028 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e80059a0 con 0x7f18f806acd0 2024-01-24T17:48:56.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.665+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f80c88a0 con 0x7f18f806acd0 2024-01-24T17:48:56.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.665+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f80c8e60 con 0x7f18f806acd0 2024-01-24T17:48:56.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.667+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f804e840 con 0x7f18f806acd0 2024-01-24T17:48:56.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.667+0000 7f18efe00640 1 -- 172.21.15.39:0/2531661028 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f18e8005b00 con 0x7f18f806acd0 2024-01-24T17:48:56.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.669+0000 7f18efe00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f18d006a430 0x7f18d006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:56.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.669+0000 7f18efe00640 1 -- 172.21.15.39:0/2531661028 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18e8087ea0 con 0x7f18f806acd0 2024-01-24T17:48:56.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.669+0000 7f18f7400640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f18d006a430 0x7f18d006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:56.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.671+0000 7f18f7400640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f18d006a430 0x7f18d006c8f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f18f80b1860 tx=0x7f18e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:56.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.673+0000 7f18efe00640 1 -- 172.21.15.39:0/2531661028 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f18e8057070 con 0x7f18f806acd0 2024-01-24T17:48:56.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.860+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18f80d0bc0 con 0x7f18f806acd0 2024-01-24T17:48:56.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.861+0000 7f18efe00640 1 -- 172.21.15.39:0/2531661028 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18e8017020 con 0x7f18f806acd0 2024-01-24T17:48:56.863 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f18d006a430 msgr2=0x7f18d006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f18d006a430 0x7f18d006c8f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f18f80b1860 tx=0x7f18e0008040 comp rx=0 tx=0).stop 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 msgr2=0x7f18f81f17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f81f17b0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f18e800c480 tx=0x7f18e800c940 comp rx=0 tx=0).stop 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 shutdown_connections 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18f80ce660 0x7f18f80b7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f18f80b4310 0x7f18f80b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f18d006a430 0x7f18d006c8f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 --2- 172.21.15.39:0/2531661028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f806acd0 0x7f18f81f17b0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:56.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 >> 172.21.15.39:0/2531661028 conn(0x7f18f80d2890 msgr2=0x7f18f80af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:56.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 shutdown_connections 2024-01-24T17:48:56.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:56.864+0000 7f18fdc00640 1 -- 172.21.15.39:0/2531661028 wait complete. 2024-01-24T17:48:56.883 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:48:57.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.088+0000 7fcccfe00640 1 Processor -- start 2024-01-24T17:48:57.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.088+0000 7fcccfe00640 1 -- start start 2024-01-24T17:48:57.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.089+0000 7fcccfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fccd0159740 0x7fccd015db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:57.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccfe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fccd006bd40 0x7fccd006c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:57.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccfe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd015e0d0 con 0x7fccd00ccbe0 2024-01-24T17:48:57.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccfe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd006c710 con 0x7fccd0159740 2024-01-24T17:48:57.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccfe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd015d140 con 0x7fccd006bd40 2024-01-24T17:48:57.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fccd0159740 0x7fccd015db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:57.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fccd0159740 0x7fccd015db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57306/0 (socket says 172.21.15.39:57306) 2024-01-24T17:48:57.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccea00640 1 -- 172.21.15.39:0/3372718267 learned_addr learned my addr 172.21.15.39:0/3372718267 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:57.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccea00640 1 -- 172.21.15.39:0/3372718267 >> v2:172.21.15.100:3300/0 conn(0x7fccd006bd40 msgr2=0x7fccd006c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:57.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccea00640 1 --2- 172.21.15.39:0/3372718267 >> v2:172.21.15.100:3300/0 conn(0x7fccd006bd40 0x7fccd006c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccea00640 1 -- 172.21.15.39:0/3372718267 >> v1:172.21.15.100:6789/0 conn(0x7fccd00ccbe0 legacy=0x7fccd00c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:57.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.090+0000 7fcccea00640 1 -- 172.21.15.39:0/3372718267 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd015d700 con 0x7fccd0159740 2024-01-24T17:48:57.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.091+0000 7fcccea00640 1 --2- 172.21.15.39:0/3372718267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd015db90 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fccd00c8990 tx=0x7fccc002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b4aa4d81609793 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:57.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.092+0000 7fccce000640 1 -- 172.21.15.39:0/3372718267 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0004230 con 0x7fccd0159740 2024-01-24T17:48:57.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.092+0000 7fccce000640 1 -- 172.21.15.39:0/3372718267 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0004390 con 0x7fccd0159740 2024-01-24T17:48:57.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.092+0000 7fccce000640 1 -- 172.21.15.39:0/3372718267 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0003810 con 0x7fccd0159740 2024-01-24T17:48:57.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.093+0000 7fcccfe00640 1 -- 172.21.15.39:0/3372718267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 msgr2=0x7fccd015db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:57.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.093+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3372718267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd015db90 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fccd00c8990 tx=0x7fccc002b8f0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.093+0000 7fcccfe00640 1 -- 172.21.15.39:0/3372718267 shutdown_connections 2024-01-24T17:48:57.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.093+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3372718267 >> v2:172.21.15.100:3300/0 conn(0x7fccd006bd40 0x7fccd006c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.093+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3372718267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd015db90 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.093+0000 7fcccfe00640 1 -- 172.21.15.39:0/3372718267 >> 172.21.15.39:0/3372718267 conn(0x7fccd006a200 msgr2=0x7fccd00aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:57.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.093+0000 7fcccfe00640 1 -- 172.21.15.39:0/3372718267 shutdown_connections 2024-01-24T17:48:57.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.093+0000 7fcccfe00640 1 -- 172.21.15.39:0/3372718267 wait complete. 2024-01-24T17:48:57.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.095+0000 7fcccfe00640 1 Processor -- start 2024-01-24T17:48:57.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.095+0000 7fcccfe00640 1 -- start start 2024-01-24T17:48:57.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.096+0000 7fcccfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fccd006bd40 0x7fccd00ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:57.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.096+0000 7fcccfe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fccd00ccbe0 0x7fccd0149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:57.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.096+0000 7fcccfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd0159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:57.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.096+0000 7fcccfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd00c6b00 con 0x7fccd0159740 2024-01-24T17:48:57.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.096+0000 7fcccfe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd00c5f30 con 0x7fccd00ccbe0 2024-01-24T17:48:57.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd0149ed0 con 0x7fccd006bd40 2024-01-24T17:48:57.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd0159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:57.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd0159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57308/0 (socket says 172.21.15.39:57308) 2024-01-24T17:48:57.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccf400640 1 -- 172.21.15.39:0/3168674983 learned_addr learned my addr 172.21.15.39:0/3168674983 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fccc7e00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fccd00ccbe0 0x7fccd0149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccea00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fccd006bd40 0x7fccd00ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccf400640 1 -- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fccd006bd40 msgr2=0x7fccd00ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccf400640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fccd006bd40 0x7fccd00ae4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccf400640 1 -- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fccd00ccbe0 msgr2=0x7fccd0149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccf400640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fccd00ccbe0 0x7fccd0149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccf400640 1 -- 172.21.15.39:0/3168674983 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc0009460 con 0x7fccd0159740 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fcccea00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fccd006bd40 0x7fccd00ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.097+0000 7fccc7e00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fccd00ccbe0 0x7fccd0149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.098+0000 7fcccf400640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd0159ea0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fccc800b970 tx=0x7fccc800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.098+0000 7fccc6a00640 1 -- 172.21.15.39:0/3168674983 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc800cb40 con 0x7fccd0159740 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.098+0000 7fccc6a00640 1 -- 172.21.15.39:0/3168674983 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc800ccc0 con 0x7fccd0159740 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.098+0000 7fccc6a00640 1 -- 172.21.15.39:0/3168674983 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8014680 con 0x7fccd0159740 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.098+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccd00c61d0 con 0x7fccd0159740 2024-01-24T17:48:57.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.098+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd00c6790 con 0x7fccd0159740 2024-01-24T17:48:57.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.099+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd00c6cb0 con 0x7fccd0159740 2024-01-24T17:48:57.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.100+0000 7fccc6a00640 1 -- 172.21.15.39:0/3168674983 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fccc801d3b0 con 0x7fccd0159740 2024-01-24T17:48:57.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.101+0000 7fccc6a00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcca806a430 0x7fcca806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:57.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.101+0000 7fccc6a00640 1 -- 172.21.15.39:0/3168674983 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccc8088900 con 0x7fccd0159740 2024-01-24T17:48:57.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.101+0000 7fcccea00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcca806a430 0x7fcca806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:57.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.103+0000 7fcccea00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcca806a430 0x7fcca806c8f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fccc0035000 tx=0x7fccc0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:57.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.105+0000 7fccc6a00640 1 -- 172.21.15.39:0/3168674983 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fccd00c6cb0 con 0x7fccd0159740 2024-01-24T17:48:57.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.312+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fccd004dff0 con 0x7fccd0159740 2024-01-24T17:48:57.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.313+0000 7fccc6a00640 1 -- 172.21.15.39:0/3168674983 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v100) v1 ==== 80+0+0 (secure 0 0 0) 0x7fccc8019020 con 0x7fccd0159740 2024-01-24T17:48:57.314 INFO:teuthology.orchestra.run.smithi039.stderr:listed 0 entries 2024-01-24T17:48:57.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcca806a430 msgr2=0x7fcca806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:57.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcca806a430 0x7fcca806c8f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fccc0035000 tx=0x7fccc0008030 comp rx=0 tx=0).stop 2024-01-24T17:48:57.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 msgr2=0x7fccd0159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:57.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd0159ea0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fccc800b970 tx=0x7fccc800be30 comp rx=0 tx=0).stop 2024-01-24T17:48:57.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 shutdown_connections 2024-01-24T17:48:57.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fcca806a430 0x7fcca806c8f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccd0159740 0x7fccd0159ea0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fccd00ccbe0 0x7fccd0149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 --2- 172.21.15.39:0/3168674983 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fccd006bd40 0x7fccd00ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:57.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.315+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 >> 172.21.15.39:0/3168674983 conn(0x7fccd006a200 msgr2=0x7fccd00c93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:57.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.316+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 shutdown_connections 2024-01-24T17:48:57.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:57.316+0000 7fcccfe00640 1 -- 172.21.15.39:0/3168674983 wait complete. 2024-01-24T17:48:57.335 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:48:57.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:57.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:48:57.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[48667]: fuse finished with error 0 and tester_r 0 2024-01-24T17:48:57.435 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:57.520 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:48:57.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:48:57.520 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:48:57.574 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:48:57.583 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:48:57.749 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-01-24T17:48:57.750 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-01-24T17:48:57.751 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-01-24T17:48:57.752 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-01-24T17:48:57.753 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-01-24T17:48:57.754 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = bitmap 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-01-24T17:48:57.755 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:48:57.756 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:48:57.757 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: mon op complaint time = 120 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:48:57.758 INFO:teuthology.orchestra.run.smithi100.stdout: client mount timeout = 600 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = "" 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout:[mds] 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds = 20 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds balancer = 20 2024-01-24T17:48:57.759 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug frag = True 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug scatterstat = True 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout: mds op complaint time = 180 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout: mds verify scatter = True 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout: mds_sessionmap_keys_per_op = 1 2024-01-24T17:48:57.760 INFO:teuthology.orchestra.run.smithi100.stdout:[client.badguy] 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-01-24T17:48:57.761 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-01-24T17:48:57.762 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.763 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-01-24T17:48:57.764 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:48:57.765 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:48:57.766 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-01-24T17:48:57.767 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.768 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:48:57.769 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-01-24T17:48:57.770 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout: mds_sessionmap_keys_per_op = 1 2024-01-24T17:48:57.771 INFO:teuthology.orchestra.run.smithi039.stdout:[client.badguy] 2024-01-24T17:48:57.810 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-24T17:48:57.810 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:57.811 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-24T17:48:57.811 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:57.811 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-24T17:48:57.812 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:57.812 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-24T17:48:57.812 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:48:57.832 INFO:tasks.ceph.mds.b:Stopped 2024-01-24T17:48:57.832 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-01-24T17:48:57.833 INFO:tasks.ceph.mds.d:Stopped 2024-01-24T17:48:57.833 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-01-24T17:48:57.846 INFO:tasks.ceph.mds.a:Stopped 2024-01-24T17:48:57.846 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-01-24T17:48:57.847 INFO:tasks.ceph.mds.c:Stopped 2024-01-24T17:48:57.847 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-01-24T17:48:58.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.028+0000 7fc90c000640 1 Processor -- start 2024-01-24T17:48:58.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.028+0000 7f6693a00640 1 Processor -- start 2024-01-24T17:48:58.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.028+0000 7fc90c000640 1 -- start start 2024-01-24T17:48:58.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.028+0000 7f6693a00640 1 -- start start 2024-01-24T17:48:58.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7fc90c000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc904144080 0x7fc9040ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.031 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7fc90c000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc904144460 0x7fc9040cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7f6693a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c143e80 0x7f668c0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7fc90c000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90415f590 con 0x7fc9040c3970 2024-01-24T17:48:58.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7fc90c000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9040c44b0 con 0x7fc904144080 2024-01-24T17:48:58.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7fc90c000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90415e7d0 con 0x7fc904144460 2024-01-24T17:48:58.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7f6693a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f668c0cefe0 0x7f668c0cb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7f6693a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668c1616f0 con 0x7f668c0caac0 2024-01-24T17:48:58.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7f6693a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668c0b0ce0 con 0x7f668c0cefe0 2024-01-24T17:48:58.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7f6693a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c161990 con 0x7f668c143e80 2024-01-24T17:48:58.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7fc90a000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc9040c3970 0x7fc9040c3da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58318/0 (socket says 172.21.15.39:58318) 2024-01-24T17:48:58.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7fc90a000640 1 -- 172.21.15.39:0/1680982080 learned_addr learned my addr 172.21.15.39:0/1680982080 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:58.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7f6691a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f668c0caac0 0x7f668c146700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58326/0 (socket says 172.21.15.39:58326) 2024-01-24T17:48:58.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7f6691a00640 1 -- 172.21.15.39:0/1856372003 learned_addr learned my addr 172.21.15.39:0/1856372003 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:58.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.030+0000 7fc908c00640 1 --2- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc904144460 0x7fc9040cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7fc909600640 1 --2- 172.21.15.39:0/1680982080 >> v2:172.21.15.100:3300/0 conn(0x7fc904144080 0x7fc9040ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7f6691000640 1 --2- 172.21.15.39:0/1856372003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c143e80 0x7f668c0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7f668be00640 1 --2- 172.21.15.39:0/1856372003 >> v2:172.21.15.100:3300/0 conn(0x7f668c0cefe0 0x7f668c0cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7fc903e00640 1 -- 172.21.15.39:0/1680982080 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980447928 0 0) 0x7fc90415f590 con 0x7fc9040c3970 2024-01-24T17:48:58.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7fc903e00640 1 -- 172.21.15.39:0/1680982080 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e4003520 con 0x7fc9040c3970 2024-01-24T17:48:58.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7fc909600640 1 -- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc904144460 msgr2=0x7fc9040cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:58.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7fc909600640 1 --2- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc904144460 0x7fc9040cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7fc909600640 1 -- 172.21.15.39:0/1680982080 >> v1:172.21.15.39:6789/0 conn(0x7fc9040c3970 legacy=0x7fc9040c3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7fc909600640 1 -- 172.21.15.39:0/1680982080 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90415eab0 con 0x7fc904144080 2024-01-24T17:48:58.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7fc908c00640 1 --2- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc904144460 0x7fc9040cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:48:58.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7f668b400640 1 -- 172.21.15.39:0/1856372003 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377596745 0 0) 0x7f668c1616f0 con 0x7f668c0caac0 2024-01-24T17:48:58.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.031+0000 7f668b400640 1 -- 172.21.15.39:0/1856372003 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f666c003520 con 0x7f668c0caac0 2024-01-24T17:48:58.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.032+0000 7f668b400640 1 -- 172.21.15.39:0/1856372003 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3743799254 0 0) 0x7f666c003520 con 0x7f668c0caac0 2024-01-24T17:48:58.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.032+0000 7f668be00640 1 -- 172.21.15.39:0/1856372003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c143e80 msgr2=0x7f668c0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:58.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.032+0000 7f668be00640 1 --2- 172.21.15.39:0/1856372003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c143e80 0x7f668c0ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.032+0000 7f668be00640 1 -- 172.21.15.39:0/1856372003 >> v1:172.21.15.39:6789/0 conn(0x7f668c0caac0 legacy=0x7f668c146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.032+0000 7f668be00640 1 -- 172.21.15.39:0/1856372003 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c161c70 con 0x7f668c0cefe0 2024-01-24T17:48:58.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.033+0000 7fc909600640 1 --2- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 0x7fc9040ce6e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40097a0 tx=0x7fc8f402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=712d2b02d412b434 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.033+0000 7fc903e00640 1 -- 172.21.15.39:0/1680982080 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4031780 con 0x7fc904144080 2024-01-24T17:48:58.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.033+0000 7fc903e00640 1 -- 172.21.15.39:0/1680982080 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f40318e0 con 0x7fc904144080 2024-01-24T17:48:58.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.033+0000 7fc903e00640 1 -- 172.21.15.39:0/1680982080 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4031c70 con 0x7fc904144080 2024-01-24T17:48:58.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7f668be00640 1 --2- 172.21.15.39:0/1856372003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c0cefe0 0x7f668c0cb180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f66740097a0 tx=0x7f667402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20351b987c796645 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7fc90c000640 1 -- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 msgr2=0x7fc9040ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7fc90c000640 1 --2- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 0x7fc9040ce6e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40097a0 tx=0x7fc8f402b820 comp rx=0 tx=0).stop 2024-01-24T17:48:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7fc90c000640 1 -- 172.21.15.39:0/1680982080 shutdown_connections 2024-01-24T17:48:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7fc90c000640 1 --2- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc904144460 0x7fc9040cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7fc90c000640 1 --2- 172.21.15.39:0/1680982080 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 0x7fc9040ce6e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7fc90c000640 1 -- 172.21.15.39:0/1680982080 >> 172.21.15.39:0/1680982080 conn(0x7fc904065ff0 msgr2=0x7fc904069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7f668b400640 1 -- 172.21.15.39:0/1856372003 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66840019f0 con 0x7f668c0cefe0 2024-01-24T17:48:58.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7fc90c000640 1 -- 172.21.15.39:0/1680982080 shutdown_connections 2024-01-24T17:48:58.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7fc90c000640 1 -- 172.21.15.39:0/1680982080 wait complete. 2024-01-24T17:48:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7f668b400640 1 -- 172.21.15.39:0/1856372003 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667402e050 con 0x7f668c0cefe0 2024-01-24T17:48:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.034+0000 7f668b400640 1 -- 172.21.15.39:0/1856372003 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6674030710 con 0x7f668c0cefe0 2024-01-24T17:48:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.035+0000 7f6693a00640 1 -- 172.21.15.39:0/1856372003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c0cefe0 msgr2=0x7f668c0cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.035+0000 7f6693a00640 1 --2- 172.21.15.39:0/1856372003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c0cefe0 0x7f668c0cb180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f66740097a0 tx=0x7f667402b790 comp rx=0 tx=0).stop 2024-01-24T17:48:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.036+0000 7f6693a00640 1 -- 172.21.15.39:0/1856372003 shutdown_connections 2024-01-24T17:48:58.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.036+0000 7f6693a00640 1 --2- 172.21.15.39:0/1856372003 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c0cefe0 0x7f668c0cb180 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.036+0000 7f6693a00640 1 --2- 172.21.15.39:0/1856372003 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c143e80 0x7f668c0ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.036+0000 7f6693a00640 1 -- 172.21.15.39:0/1856372003 >> 172.21.15.39:0/1856372003 conn(0x7f668c065ff0 msgr2=0x7f668c069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.036+0000 7f6693a00640 1 -- 172.21.15.39:0/1856372003 shutdown_connections 2024-01-24T17:48:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.036+0000 7f6693a00640 1 -- 172.21.15.39:0/1856372003 wait complete. 2024-01-24T17:48:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.037+0000 7fc90c000640 1 Processor -- start 2024-01-24T17:48:58.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.037+0000 7fc90c000640 1 -- start start 2024-01-24T17:48:58.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.038+0000 7f6693a00640 1 Processor -- start 2024-01-24T17:48:58.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.038+0000 7f6693a00640 1 -- start start 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.038+0000 7fc90c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9040c3970 0x7fc90413de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.038+0000 7fc90c000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 0x7fc90415a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.038+0000 7fc90c000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc90413ee70 0x7fc90413bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.039+0000 7fc90c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc904140570 con 0x7fc9040c3970 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.039+0000 7fc90c000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90413f3a0 con 0x7fc904144080 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.039+0000 7fc90c000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90413f680 con 0x7fc90413ee70 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.039+0000 7fc909600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9040c3970 0x7fc90413de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.039+0000 7fc909600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9040c3970 0x7fc90413de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57322/0 (socket says 172.21.15.39:57322) 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.039+0000 7fc909600640 1 -- 172.21.15.39:0/1350434757 learned_addr learned my addr 172.21.15.39:0/1350434757 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:58.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.039+0000 7fc908c00640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 0x7fc90415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.039+0000 7fc90a000640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc90413ee70 0x7fc90413bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7f6693a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c0caac0 0x7f668c1f9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7fc909600640 1 -- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc90413ee70 msgr2=0x7fc90413bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7fc909600640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc90413ee70 0x7fc90413bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7fc909600640 1 -- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 msgr2=0x7fc90415a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7fc909600640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 0x7fc90415a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7f6693a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f668c143e80 0x7f668c0c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7f6693a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c142ba0 0x7f668c06c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7fc909600640 1 -- 172.21.15.39:0/1350434757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec009400 con 0x7fc9040c3970 2024-01-24T17:48:58.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7f6693a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c0c3ae0 con 0x7f668c143e80 2024-01-24T17:48:58.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7f6693a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c1fa560 con 0x7f668c142ba0 2024-01-24T17:48:58.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7f6693a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c1fa840 con 0x7f668c0caac0 2024-01-24T17:48:58.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7fc909600640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9040c3970 0x7fc90413de80 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4034040 tx=0x7fc8f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7f6691000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c0caac0 0x7f668c1f9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.040+0000 7f668be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f668c143e80 0x7f668c0c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7f6691000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c0caac0 0x7f668c1f9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:51616/0 (socket says 172.21.15.39:51616) 2024-01-24T17:48:58.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7f6691000640 1 -- 172.21.15.39:0/228866687 learned_addr learned my addr 172.21.15.39:0/228866687 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:58.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7f6691a00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c142ba0 0x7f668c06c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7fc901600640 1 -- 172.21.15.39:0/1350434757 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4032820 con 0x7fc9040c3970 2024-01-24T17:48:58.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7fc901600640 1 -- 172.21.15.39:0/1350434757 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4032980 con 0x7fc9040c3970 2024-01-24T17:48:58.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f4009400 con 0x7fc9040c3970 2024-01-24T17:48:58.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7fc901600640 1 -- 172.21.15.39:0/1350434757 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4032d30 con 0x7fc9040c3970 2024-01-24T17:48:58.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90413fc40 con 0x7fc9040c3970 2024-01-24T17:48:58.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7f6691000640 1 -- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c142ba0 msgr2=0x7f668c06c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7f6691000640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c142ba0 0x7f668c06c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.041+0000 7f6691000640 1 -- 172.21.15.39:0/228866687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f668c143e80 msgr2=0x7f668c0c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.042+0000 7f6691000640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f668c143e80 0x7f668c0c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.042+0000 7f6691000640 1 -- 172.21.15.39:0/228866687 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6674009400 con 0x7f668c0caac0 2024-01-24T17:48:58.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.042+0000 7f668be00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f668c143e80 0x7f668c0c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:48:58.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.042+0000 7f6691000640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c0caac0 0x7f668c1f9c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f667c00e870 tx=0x7f667c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.043+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc904073f10 con 0x7fc9040c3970 2024-01-24T17:48:58.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.043+0000 7f6688c00640 1 -- 172.21.15.39:0/228866687 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c0094f0 con 0x7f668c0caac0 2024-01-24T17:48:58.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.043+0000 7f6688c00640 1 -- 172.21.15.39:0/228866687 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c00c940 con 0x7f668c0caac0 2024-01-24T17:48:58.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.043+0000 7f6688c00640 1 -- 172.21.15.39:0/228866687 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c00ccb0 con 0x7f668c0caac0 2024-01-24T17:48:58.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.043+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c1fab20 con 0x7f668c0caac0 2024-01-24T17:48:58.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.043+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668c1fa1c0 con 0x7f668c0caac0 2024-01-24T17:48:58.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.045+0000 7fc901600640 1 -- 172.21.15.39:0/1350434757 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc8f404e440 con 0x7fc9040c3970 2024-01-24T17:48:58.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.045+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668c06ab80 con 0x7f668c0caac0 2024-01-24T17:48:58.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.046+0000 7fc901600640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc8dc06a430 0x7fc8dc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.046+0000 7fc901600640 1 -- 172.21.15.39:0/1350434757 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8f40aac10 con 0x7fc9040c3970 2024-01-24T17:48:58.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.047+0000 7f6688c00640 1 -- 172.21.15.39:0/228866687 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f667c026480 con 0x7f668c0caac0 2024-01-24T17:48:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.047+0000 7fc908c00640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc8dc06a430 0x7fc8dc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.048+0000 7f6688c00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666406a430 0x7f666406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.048+0000 7f6688c00640 1 -- 172.21.15.39:0/228866687 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f667c089190 con 0x7f668c0caac0 2024-01-24T17:48:58.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.048+0000 7fc908c00640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc8dc06a430 0x7fc8dc06c8f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc904069dc0 tx=0x7fc8ec009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.049+0000 7f668be00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666406a430 0x7f666406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.050+0000 7f668be00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666406a430 0x7f666406c8f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6674034000 tx=0x7f667402f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.053+0000 7fc901600640 1 -- 172.21.15.39:0/1350434757 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc8f4076280 con 0x7fc9040c3970 2024-01-24T17:48:58.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.062+0000 7f6688c00640 1 -- 172.21.15.39:0/228866687 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f667c054810 con 0x7f668c0caac0 2024-01-24T17:48:58.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.077+0000 7faff4e00640 1 Processor -- start 2024-01-24T17:48:58.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.078+0000 7faff4e00640 1 -- start start 2024-01-24T17:48:58.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.079+0000 7faff4e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faff0144080 0x7faff00ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.079+0000 7faff4e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff015f590 con 0x7faff0144460 2024-01-24T17:48:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.079+0000 7faff4e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff015e7d0 con 0x7faff00c4080 2024-01-24T17:48:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.079+0000 7faff4e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff015eab0 con 0x7faff0144080 2024-01-24T17:48:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.080+0000 7fafede00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faff0144080 0x7faff00ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.080+0000 7fafede00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faff0144080 0x7faff00ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57336/0 (socket says 172.21.15.39:57336) 2024-01-24T17:48:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.080+0000 7fafede00640 1 -- 172.21.15.39:0/486240847 learned_addr learned my addr 172.21.15.39:0/486240847 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.080+0000 7fafede00640 1 -- 172.21.15.39:0/486240847 >> v1:172.21.15.100:6789/0 conn(0x7faff00c4080 legacy=0x7faff00b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.080+0000 7fafede00640 1 -- 172.21.15.39:0/486240847 >> v1:172.21.15.39:6789/0 conn(0x7faff0144460 legacy=0x7faff00c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.080+0000 7fafede00640 1 -- 172.21.15.39:0/486240847 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff015ed90 con 0x7faff0144080 2024-01-24T17:48:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.081+0000 7ff3d2800640 1 Processor -- start 2024-01-24T17:48:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.081+0000 7ff3d2800640 1 -- start start 2024-01-24T17:48:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.082+0000 7ff3d2800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 0x7ff3cc06ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.082+0000 7ff3d2800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0ce710 con 0x7ff3cc0d35d0 2024-01-24T17:48:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.082+0000 7ff3d2800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0d0650 con 0x7ff3cc0cf120 2024-01-24T17:48:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.082+0000 7ff3d2800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0ce970 con 0x7ff3cc0d1410 2024-01-24T17:48:58.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7ff3d0a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff3cc0d35d0 0x7ff3cc0ce000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58336/0 (socket says 172.21.15.39:58336) 2024-01-24T17:48:58.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7ff3d0a00640 1 -- 172.21.15.39:0/1430994658 learned_addr learned my addr 172.21.15.39:0/1430994658 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:48:58.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7fafede00640 1 --2- 172.21.15.39:0/486240847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff0144080 0x7faff00ce6e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fafe000de50 tx=0x7fafe002fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b51b80d1706511 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7fafeca00640 1 -- 172.21.15.39:0/486240847 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0030a80 con 0x7faff0144080 2024-01-24T17:48:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090695506 0 0) 0x7ff3cc0ce710 con 0x7ff3cc0d35d0 2024-01-24T17:48:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7ff3d1e00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 0x7ff3cc06ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ac003520 con 0x7ff3cc0d35d0 2024-01-24T17:48:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7fafeca00640 1 -- 172.21.15.39:0/486240847 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe0030be0 con 0x7faff0144080 2024-01-24T17:48:58.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.083+0000 7fafeca00640 1 -- 172.21.15.39:0/486240847 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe00383f0 con 0x7faff0144080 2024-01-24T17:48:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271391783 0 0) 0x7ff3ac003520 con 0x7ff3cc0d35d0 2024-01-24T17:48:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 msgr2=0x7ff3cc06ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 0x7ff3cc06ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 >> v1:172.21.15.100:6789/0 conn(0x7ff3cc0cf120 legacy=0x7ff3cc0cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0d0930 con 0x7ff3cc0d35d0 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff3b4001670 con 0x7ff3cc0d35d0 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 >> v1:172.21.15.39:6789/0 conn(0x7ff3cc0d35d0 legacy=0x7ff3cc0ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3ac0039a0 0x7ff3ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 0x7ff3ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac00ba30 con 0x7ff3ac004d00 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac00bba0 con 0x7ff3ac0039a0 2024-01-24T17:48:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac00bd60 con 0x7ff3cc0d1410 2024-01-24T17:48:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7faff4e00640 1 -- 172.21.15.39:0/486240847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff0144080 msgr2=0x7faff00ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7faff4e00640 1 --2- 172.21.15.39:0/486240847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff0144080 0x7faff00ce6e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fafe000de50 tx=0x7fafe002fee0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7ff3d1e00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7faff4e00640 1 -- 172.21.15.39:0/486240847 shutdown_connections 2024-01-24T17:48:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7faff4e00640 1 --2- 172.21.15.39:0/486240847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff0144080 0x7faff00ce6e0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.084+0000 7faff4e00640 1 -- 172.21.15.39:0/486240847 >> 172.21.15.39:0/486240847 conn(0x7faff0065ff0 msgr2=0x7faff0069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7faff4e00640 1 -- 172.21.15.39:0/486240847 shutdown_connections 2024-01-24T17:48:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7ff3d0a00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 0x7ff3ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7ff3d1400640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3ac0039a0 0x7ff3ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7ff3d1e00640 1 -- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 msgr2=0x7ff3ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7ff3d1e00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 0x7ff3ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7ff3d1e00640 1 -- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3ac0039a0 msgr2=0x7ff3ac003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7ff3d1e00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3ac0039a0 0x7ff3ac003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7ff3d1e00640 1 -- 172.21.15.39:0/1430994658 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac003520 con 0x7ff3ac004d00 2024-01-24T17:48:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7ff3d0a00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0d1410 0x7ff3ac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:48:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.085+0000 7faff4e00640 1 -- 172.21.15.39:0/486240847 wait complete. 2024-01-24T17:48:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.087+0000 7faff4e00640 1 Processor -- start 2024-01-24T17:48:58.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.087+0000 7ff3d1e00640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3ac005180 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009f90 tx=0x7ff3ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9570a131d50d9349 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.087+0000 7faff4e00640 1 -- start start 2024-01-24T17:48:58.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.087+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c402cbd0 con 0x7ff3ac004d00 2024-01-24T17:48:58.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.087+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c400f040 con 0x7ff3ac004d00 2024-01-24T17:48:58.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.087+0000 7ff3cbe00640 1 -- 172.21.15.39:0/1430994658 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4034440 con 0x7ff3ac004d00 2024-01-24T17:48:58.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7faff4e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faff00c4080 0x7faff0154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7faff4e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faff0144080 0x7faff01404d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7faff4e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff00cb790 0x7faff00cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7faff4e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff00cc180 con 0x7faff00cb790 2024-01-24T17:48:58.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7faff4e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff01eee60 con 0x7faff0144080 2024-01-24T17:48:58.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7faff4e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff01ee250 con 0x7faff00c4080 2024-01-24T17:48:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7ff3d2800640 1 -- 172.21.15.39:0/1430994658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 msgr2=0x7ff3ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7ff3d2800640 1 --2- 172.21.15.39:0/1430994658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3ac005180 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009f90 tx=0x7ff3ac0057f0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7ff3d1e00640 1 -- 172.21.15.39:0/1430994658 reap_dead start 2024-01-24T17:48:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7fafee800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff00cb790 0x7faff00cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7fafee800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff00cb790 0x7faff00cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57348/0 (socket says 172.21.15.39:57348) 2024-01-24T17:48:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7fafee800640 1 -- 172.21.15.39:0/2327430428 learned_addr learned my addr 172.21.15.39:0/2327430428 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7ff3d2800640 1 -- 172.21.15.39:0/1430994658 shutdown_connections 2024-01-24T17:48:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.088+0000 7ff3d2800640 1 -- 172.21.15.39:0/1430994658 >> 172.21.15.39:0/1430994658 conn(0x7ff3cc09bc10 msgr2=0x7ff3cc0d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7fafed400640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faff0144080 0x7faff01404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7fafede00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faff00c4080 0x7faff0154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7ff3d2800640 1 -- 172.21.15.39:0/1430994658 shutdown_connections 2024-01-24T17:48:58.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7ff3d2800640 1 -- 172.21.15.39:0/1430994658 wait complete. 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7fafee800640 1 -- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faff00c4080 msgr2=0x7faff0154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7fafee800640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faff00c4080 0x7faff0154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7fafee800640 1 -- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faff0144080 msgr2=0x7faff01404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7fafee800640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faff0144080 0x7faff01404d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7fafee800640 1 -- 172.21.15.39:0/2327430428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe000db30 con 0x7faff00cb790 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.089+0000 7fafee800640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff00cb790 0x7faff00cbc40 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fafe800c7b0 tx=0x7fafe800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.090+0000 7fafe6000640 1 -- 172.21.15.39:0/2327430428 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8004b60 con 0x7faff00cb790 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.090+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff01ee530 con 0x7faff00cb790 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.090+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff01eeaf0 con 0x7faff00cb790 2024-01-24T17:48:58.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.090+0000 7fafe6000640 1 -- 172.21.15.39:0/2327430428 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe8004cc0 con 0x7faff00cb790 2024-01-24T17:48:58.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.090+0000 7fafe6000640 1 -- 172.21.15.39:0/2327430428 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8006d50 con 0x7faff00cb790 2024-01-24T17:48:58.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.091+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc0005250 con 0x7faff00cb790 2024-01-24T17:48:58.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.092+0000 7ff3d2800640 1 Processor -- start 2024-01-24T17:48:58.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.092+0000 7ff3d2800640 1 -- start start 2024-01-24T17:48:58.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7fafe6000640 1 -- 172.21.15.39:0/2327430428 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fafe800a850 con 0x7faff00cb790 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.093+0000 7ff3d2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3cc0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.093+0000 7ff3d2800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0cc3d0 0x7ff3cc149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.093+0000 7ff3d2800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3cc149000 0x7ff3cc0d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.093+0000 7ff3d2800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc14c3d0 con 0x7ff3ac004d00 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.093+0000 7ff3d2800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0cc850 con 0x7ff3cc149000 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.093+0000 7ff3d2800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc149eb0 con 0x7ff3cc0cc3d0 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3cc0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3cc0c64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57364/0 (socket says 172.21.15.39:57364) 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d1400640 1 -- 172.21.15.39:0/1350117443 learned_addr learned my addr 172.21.15.39:0/1350117443 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d0a00640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0cc3d0 0x7ff3cc149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d1e00640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3cc149000 0x7ff3cc0d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d1400640 1 -- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0cc3d0 msgr2=0x7ff3cc149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d1400640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0cc3d0 0x7ff3cc149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d1400640 1 -- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3cc149000 msgr2=0x7ff3cc0d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.094+0000 7ff3d1400640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3cc149000 0x7ff3cc0d27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.095+0000 7ff3d1400640 1 -- 172.21.15.39:0/1350117443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc14c5b0 con 0x7ff3ac004d00 2024-01-24T17:48:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.095+0000 7ff3d1400640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3cc0c64f0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00e4d0 tx=0x7ff3bc00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.096+0000 7fafe6000640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fafc806a430 0x7fafc806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.096+0000 7fafe6000640 1 -- 172.21.15.39:0/2327430428 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafe8088cd0 con 0x7faff00cb790 2024-01-24T17:48:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.097+0000 7fafede00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fafc806a430 0x7fafc806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.098+0000 7fafede00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fafc806a430 0x7fafc806c8f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fafe0039000 tx=0x7fafe0034000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.099+0000 7ff3c9600640 1 -- 172.21.15.39:0/1350117443 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc0052e0 con 0x7ff3ac004d00 2024-01-24T17:48:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.099+0000 7ff3c9600640 1 -- 172.21.15.39:0/1350117443 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc005440 con 0x7ff3ac004d00 2024-01-24T17:48:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.099+0000 7ff3c9600640 1 -- 172.21.15.39:0/1350117443 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc003810 con 0x7ff3ac004d00 2024-01-24T17:48:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.099+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc0c7f10 con 0x7ff3ac004d00 2024-01-24T17:48:58.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.099+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc0c84d0 con 0x7ff3ac004d00 2024-01-24T17:48:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.100+0000 7fafe6000640 1 -- 172.21.15.39:0/2327430428 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fafe80542a0 con 0x7faff00cb790 2024-01-24T17:48:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.100+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc0c8690 con 0x7ff3ac004d00 2024-01-24T17:48:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.101+0000 7ff3c9600640 1 -- 172.21.15.39:0/1350117443 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff3bc003970 con 0x7ff3ac004d00 2024-01-24T17:48:58.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.102+0000 7ff3c9600640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3a406a4b0 0x7ff3a406c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.102+0000 7ff3d0a00640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3a406a4b0 0x7ff3a406c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.103+0000 7ff3c9600640 1 -- 172.21.15.39:0/1350117443 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3bc057830 con 0x7ff3ac004d00 2024-01-24T17:48:58.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.104+0000 7ff3d0a00640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3a406a4b0 0x7ff3a406c970 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0c7b60 tx=0x7ff3b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.106+0000 7ff3c9600640 1 -- 172.21.15.39:0/1350117443 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff3bc08d060 con 0x7ff3ac004d00 2024-01-24T17:48:58.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.327+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fc9040b1a10 con 0x7fc9040c3970 2024-01-24T17:48:58.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.375+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7ff3cc0c7d70 con 0x7ff3ac004d00 2024-01-24T17:48:58.378 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:48:58.376+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:48:58.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.390+0000 7fc901600640 1 -- 172.21.15.39:0/1350434757 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5717 v60) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc8f40798a0 con 0x7fc9040c3970 2024-01-24T17:48:58.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.390+0000 7ff3c9600640 1 -- 172.21.15.39:0/1350117443 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7761 v60) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff3cc0c7d70 con 0x7ff3ac004d00 2024-01-24T17:48:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:failed mds gid 5717 2024-01-24T17:48:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:failed mds gid 7761 2024-01-24T17:48:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3a406a4b0 msgr2=0x7ff3a406c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3a406a4b0 0x7ff3a406c970 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0c7b60 tx=0x7ff3b4009040 comp rx=0 tx=0).stop 2024-01-24T17:48:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 msgr2=0x7ff3cc0c64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3cc0c64f0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00e4d0 tx=0x7ff3bc00e990 comp rx=0 tx=0).stop 2024-01-24T17:48:58.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 shutdown_connections 2024-01-24T17:48:58.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3cc149000 0x7ff3cc0d27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3cc0cc3d0 0x7ff3cc149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3a406a4b0 0x7ff3a406c970 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 --2- 172.21.15.39:0/1350117443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3ac004d00 0x7ff3cc0c64f0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 >> 172.21.15.39:0/1350117443 conn(0x7ff3cc09bc10 msgr2=0x7ff3cc0b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc8dc06a430 msgr2=0x7fc8dc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc8dc06a430 0x7fc8dc06c8f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc904069dc0 tx=0x7fc8ec009190 comp rx=0 tx=0).stop 2024-01-24T17:48:58.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9040c3970 msgr2=0x7fc90413de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9040c3970 0x7fc90413de80 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4034040 tx=0x7fc8f40046c0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 shutdown_connections 2024-01-24T17:48:58.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc90413ee70 0x7fc90413bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 shutdown_connections 2024-01-24T17:48:58.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc904144080 0x7fc90415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc8dc06a430 0x7fc8dc06c8f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 --2- 172.21.15.39:0/1350434757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9040c3970 0x7fc90413de80 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 >> 172.21.15.39:0/1350434757 conn(0x7fc904065ff0 msgr2=0x7fc9040c9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7ff3d2800640 1 -- 172.21.15.39:0/1350117443 wait complete. 2024-01-24T17:48:58.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.393+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 shutdown_connections 2024-01-24T17:48:58.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.394+0000 7fc90c000640 1 -- 172.21.15.39:0/1350434757 wait complete. 2024-01-24T17:48:58.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.401+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f668c1fa980 con 0x7f668c0caac0 2024-01-24T17:48:58.405 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:48:58.404+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:48:58.416 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-24T17:48:58.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-24T17:48:58.417 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-24T17:48:58.418 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-24T17:48:58.423 INFO:tasks.ceph.mds.a:Started 2024-01-24T17:48:58.423 DEBUG:teuthology.parallel:result is None 2024-01-24T17:48:58.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.423+0000 7f6688c00640 1 -- 172.21.15.39:0/228866687 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6439 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7f668c1fa980 con 0x7f668c0caac0 2024-01-24T17:48:58.425 INFO:teuthology.orchestra.run.smithi039.stderr:failed mds gid 6439 2024-01-24T17:48:58.426 INFO:tasks.ceph.mds.b:Started 2024-01-24T17:48:58.427 DEBUG:teuthology.parallel:result is None 2024-01-24T17:48:58.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.425+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666406a430 msgr2=0x7f666406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.425+0000 7f6693a00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666406a430 0x7f666406c8f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6674034000 tx=0x7f667402f000 comp rx=0 tx=0).stop 2024-01-24T17:48:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.425+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c0caac0 msgr2=0x7f668c1f9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.425+0000 7f6693a00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c0caac0 0x7f668c1f9c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f667c00e870 tx=0x7f667c00ed30 comp rx=0 tx=0).stop 2024-01-24T17:48:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.426+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 shutdown_connections 2024-01-24T17:48:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.426+0000 7f6693a00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f668c142ba0 0x7f668c06c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.426+0000 7f6693a00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f666406a430 0x7f666406c8f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.426+0000 7f6693a00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f668c143e80 0x7f668c0c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.426+0000 7f6693a00640 1 --2- 172.21.15.39:0/228866687 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f668c0caac0 0x7f668c1f9c80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.426+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 >> 172.21.15.39:0/228866687 conn(0x7f668c065ff0 msgr2=0x7f668c160d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.426+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 shutdown_connections 2024-01-24T17:48:58.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.426+0000 7f6693a00640 1 -- 172.21.15.39:0/228866687 wait complete. 2024-01-24T17:48:58.446 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-24T17:48:58.446 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-24T17:48:58.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.448+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7fafc00057d0 con 0x7faff00cb790 2024-01-24T17:48:58.451 INFO:tasks.ceph.mds.d:Started 2024-01-24T17:48:58.452 DEBUG:teuthology.parallel:result is None 2024-01-24T17:48:58.452 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:48:58.450+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:48:58.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.464+0000 7fafe6000640 1 -- 172.21.15.39:0/2327430428 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6436 v64) v1 ==== 95+0+0 (secure 0 0 0) 0x7fafe80578c0 con 0x7faff00cb790 2024-01-24T17:48:58.465 INFO:teuthology.orchestra.run.smithi039.stderr:failed mds gid 6436 2024-01-24T17:48:58.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fafc806a430 msgr2=0x7fafc806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fafc806a430 0x7fafc806c8f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fafe0039000 tx=0x7fafe0034000 comp rx=0 tx=0).stop 2024-01-24T17:48:58.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff00cb790 msgr2=0x7faff00cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff00cb790 0x7faff00cbc40 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fafe800c7b0 tx=0x7fafe800cc70 comp rx=0 tx=0).stop 2024-01-24T17:48:58.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 shutdown_connections 2024-01-24T17:48:58.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fafc806a430 0x7fafc806c8f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faff00cb790 0x7faff00cbc40 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faff0144080 0x7faff01404d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 --2- 172.21.15.39:0/2327430428 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faff00c4080 0x7faff0154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 >> 172.21.15.39:0/2327430428 conn(0x7faff0065ff0 msgr2=0x7faff0156f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 shutdown_connections 2024-01-24T17:48:58.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.466+0000 7faff4e00640 1 -- 172.21.15.39:0/2327430428 wait complete. 2024-01-24T17:48:58.484 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-24T17:48:58.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-24T17:48:58.531 INFO:tasks.ceph.mds.c:Started 2024-01-24T17:48:58.531 DEBUG:teuthology.parallel:result is None 2024-01-24T17:48:58.532 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:48:58.570 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:48:58.587 INFO:tasks.ceph.mds.b.smithi100.stdout:starting mds.b at 2024-01-24T17:48:58.611 INFO:tasks.ceph.mds.d.smithi100.stdout:starting mds.d at 2024-01-24T17:48:58.676 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:48:58.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.706+0000 7f6e10c00640 1 Processor -- start 2024-01-24T17:48:58.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.706+0000 7f6e10c00640 1 -- start start 2024-01-24T17:48:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e10c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f6e0c15cd90 0x7f6e0c0b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e10c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c0cd320 0x7f6e0c0cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e10c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6e0c0cf110 0x7f6e0c0cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e10c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0c0b6380 con 0x7f6e0c0cf110 2024-01-24T17:48:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e10c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0c0d0cb0 con 0x7f6e0c15cd90 2024-01-24T17:48:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e10c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c0cdd70 con 0x7f6e0c0cd320 2024-01-24T17:48:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e0a800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6e0c0cf110 0x7f6e0c0cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e0a800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f6e0c0cf110 0x7f6e0c0cf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57400/0 (socket says 172.21.15.39:57400) 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e0a800640 1 -- 172.21.15.39:0/3337423972 learned_addr learned my addr 172.21.15.39:0/3337423972 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e09e00640 1 --2- 172.21.15.39:0/3337423972 >> v2:172.21.15.100:3300/0 conn(0x7f6e0c15cd90 0x7f6e0c0b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e03e00640 1 --2- 172.21.15.39:0/3337423972 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c0cd320 0x7f6e0c0cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e0a800640 1 -- 172.21.15.39:0/3337423972 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c0cd320 msgr2=0x7f6e0c0cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e0a800640 1 --2- 172.21.15.39:0/3337423972 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c0cd320 0x7f6e0c0cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e0a800640 1 -- 172.21.15.39:0/3337423972 >> v2:172.21.15.100:3300/0 conn(0x7f6e0c15cd90 msgr2=0x7f6e0c0b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e0a800640 1 --2- 172.21.15.39:0/3337423972 >> v2:172.21.15.100:3300/0 conn(0x7f6e0c15cd90 0x7f6e0c0b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e0a800640 1 -- 172.21.15.39:0/3337423972 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c0ce050 con 0x7f6e0c0cf110 2024-01-24T17:48:58.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.708+0000 7f6e09e00640 1 --2- 172.21.15.39:0/3337423972 >> v2:172.21.15.100:3300/0 conn(0x7f6e0c15cd90 0x7f6e0c0b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.709+0000 7f6e0a800640 1 --2- 172.21.15.39:0/3337423972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c0cf5a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c0b5580 tx=0x7f6e0402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c1c79d1f2bcff5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e09400640 1 -- 172.21.15.39:0/3337423972 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0402c9e0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e09400640 1 -- 172.21.15.39:0/3337423972 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0402cb40 con 0x7f6e0c0cf110 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e09400640 1 -- 172.21.15.39:0/3337423972 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e040343f0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e10c00640 1 -- 172.21.15.39:0/3337423972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 msgr2=0x7f6e0c0cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3337423972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c0cf5a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c0b5580 tx=0x7f6e0402be40 comp rx=0 tx=0).stop 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e10c00640 1 -- 172.21.15.39:0/3337423972 shutdown_connections 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3337423972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c0cf5a0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3337423972 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c0cd320 0x7f6e0c0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3337423972 >> v2:172.21.15.100:3300/0 conn(0x7f6e0c15cd90 0x7f6e0c0b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.710+0000 7f6e10c00640 1 -- 172.21.15.39:0/3337423972 >> 172.21.15.39:0/3337423972 conn(0x7f6e0c06af00 msgr2=0x7f6e0c06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.711+0000 7f6e10c00640 1 -- 172.21.15.39:0/3337423972 shutdown_connections 2024-01-24T17:48:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.711+0000 7f6e10c00640 1 -- 172.21.15.39:0/3337423972 wait complete. 2024-01-24T17:48:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.712+0000 7f6e10c00640 1 Processor -- start 2024-01-24T17:48:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.712+0000 7f6e10c00640 1 -- start start 2024-01-24T17:48:58.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e10c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e0c0cd320 0x7f6e0c0cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e10c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c14f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e10c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c15cd90 0x7f6e0c159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e10c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c0cb090 con 0x7f6e0c0cf110 2024-01-24T17:48:58.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e10c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c15dff0 con 0x7f6e0c0cd320 2024-01-24T17:48:58.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e10c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c15e2d0 con 0x7f6e0c15cd90 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c14f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c14f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57412/0 (socket says 172.21.15.39:57412) 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 -- 172.21.15.39:0/3526310454 learned_addr learned my addr 172.21.15.39:0/3526310454 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 -- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c15cd90 msgr2=0x7f6e0c159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e0a800640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c15cd90 0x7f6e0c159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c15cd90 0x7f6e0c159560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 -- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e0c0cd320 msgr2=0x7f6e0c0cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e0c0cd320 0x7f6e0c0cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 -- 172.21.15.39:0/3526310454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04009b30 con 0x7f6e0c0cf110 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.714+0000 7f6e03e00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c14f220 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c7f0 tx=0x7f6df400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.715+0000 7f6e02000640 1 -- 172.21.15.39:0/3526310454 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4004220 con 0x7f6e0c0cf110 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.715+0000 7f6e02000640 1 -- 172.21.15.39:0/3526310454 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df40043a0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.715+0000 7f6e02000640 1 -- 172.21.15.39:0/3526310454 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4007860 con 0x7f6e0c0cf110 2024-01-24T17:48:58.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.715+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c0b39d0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.715+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c14f760 con 0x7f6e0c0cf110 2024-01-24T17:48:58.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.716+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c0b3ef0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.717+0000 7f6e02000640 1 -- 172.21.15.39:0/3526310454 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6df4007bd0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.717+0000 7f6e02000640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6de406a430 0x7f6de406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:48:58.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.718+0000 7f6e02000640 1 -- 172.21.15.39:0/3526310454 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f6df40889e0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.718+0000 7f6e09e00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6de406a430 0x7f6de406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:48:58.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.719+0000 7f6e09e00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6de406a430 0x7f6de406c8f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc006b10 tx=0x7f6dfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:48:58.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.723+0000 7f6e02000640 1 -- 172.21.15.39:0/3526310454 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f6e0c0b3ef0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.954+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e0c0b3ef0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.955+0000 7f6e02000640 1 -- 172.21.15.39:0/3526310454 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+1702 (secure 0 0 0) 0x7f6e0c0b3ef0 con 0x7f6e0c0cf110 2024-01-24T17:48:58.956 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:48:58.956 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:48:58.451846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:48:58.956 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 64 2024-01-24T17:48:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6de406a430 msgr2=0x7f6de406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6de406a430 0x7f6de406c8f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc006b10 tx=0x7f6dfc008040 comp rx=0 tx=0).stop 2024-01-24T17:48:58.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 msgr2=0x7f6e0c14f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c14f220 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c7f0 tx=0x7f6df400ccb0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 shutdown_connections 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6e0c15cd90 0x7f6e0c159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f6de406a430 0x7f6de406c8f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e0c0cf110 0x7f6e0c14f220 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 --2- 172.21.15.39:0/3526310454 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6e0c0cd320 0x7f6e0c0cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 >> 172.21.15.39:0/3526310454 conn(0x7f6e0c06af00 msgr2=0x7f6e0c15ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 shutdown_connections 2024-01-24T17:48:58.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:48:58.958+0000 7f6e10c00640 1 -- 172.21.15.39:0/3526310454 wait complete. 2024-01-24T17:48:59.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:48:26.880600+0000', 'modified': '2024-01-24T17:48:58.451846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 104, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:48:59.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-01-24T17:49:00.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:49:00.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.603+0000 7f4c68e00640 1 Processor -- start 2024-01-24T17:49:00.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.604+0000 7f4c68e00640 1 -- start start 2024-01-24T17:49:00.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.605+0000 7f4c68e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4c6415e420 0x7f4c64071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.605+0000 7f4c68e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 0x7f4c6415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.605+0000 7f4c68e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c6415be30 con 0x7f4c640c9450 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.605+0000 7f4c68e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c64072220 con 0x7f4c6415e420 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.605+0000 7f4c68e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6415e880 con 0x7f4c6415b460 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.605+0000 7f4c5a800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4c6415e420 0x7f4c64071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.606+0000 7f4c5a800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f4c6415e420 0x7f4c64071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57422/0 (socket says 172.21.15.39:57422) 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.606+0000 7f4c5a800640 1 -- 172.21.15.39:0/868235728 learned_addr learned my addr 172.21.15.39:0/868235728 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.606+0000 7f4c63e00640 1 --2- 172.21.15.39:0/868235728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 0x7f4c6415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.606+0000 7f4c5a800640 1 -- 172.21.15.39:0/868235728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 msgr2=0x7f4c6415b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.606+0000 7f4c5a800640 1 --2- 172.21.15.39:0/868235728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 0x7f4c6415b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.606+0000 7f4c5a800640 1 -- 172.21.15.39:0/868235728 >> v1:172.21.15.100:6789/0 conn(0x7f4c640c9450 legacy=0x7f4c6406fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:00.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.606+0000 7f4c5a800640 1 -- 172.21.15.39:0/868235728 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6415f330 con 0x7f4c6415e420 2024-01-24T17:49:00.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.607+0000 7f4c5a800640 1 --2- 172.21.15.39:0/868235728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c6415e420 0x7f4c64071ce0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009870 tx=0x7f4c4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8989d9be4c47838b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:00.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.607+0000 7f4c62a00640 1 -- 172.21.15.39:0/868235728 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c031640 con 0x7f4c6415e420 2024-01-24T17:49:00.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.607+0000 7f4c62a00640 1 -- 172.21.15.39:0/868235728 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c0317a0 con 0x7f4c6415e420 2024-01-24T17:49:00.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.607+0000 7f4c62a00640 1 -- 172.21.15.39:0/868235728 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c031b30 con 0x7f4c6415e420 2024-01-24T17:49:00.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.608+0000 7f4c68e00640 1 -- 172.21.15.39:0/868235728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c6415e420 msgr2=0x7f4c64071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:00.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.608+0000 7f4c68e00640 1 --2- 172.21.15.39:0/868235728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c6415e420 0x7f4c64071ce0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009870 tx=0x7f4c4c02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.608+0000 7f4c68e00640 1 -- 172.21.15.39:0/868235728 shutdown_connections 2024-01-24T17:49:00.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.608+0000 7f4c68e00640 1 --2- 172.21.15.39:0/868235728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c6415e420 0x7f4c64071ce0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.608+0000 7f4c68e00640 1 --2- 172.21.15.39:0/868235728 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 0x7f4c6415b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.608+0000 7f4c68e00640 1 -- 172.21.15.39:0/868235728 >> 172.21.15.39:0/868235728 conn(0x7f4c6406bf30 msgr2=0x7f4c6406c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.608+0000 7f4c68e00640 1 -- 172.21.15.39:0/868235728 shutdown_connections 2024-01-24T17:49:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.608+0000 7f4c68e00640 1 -- 172.21.15.39:0/868235728 wait complete. 2024-01-24T17:49:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.610+0000 7f4c68e00640 1 Processor -- start 2024-01-24T17:49:00.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.610+0000 7f4c68e00640 1 -- start start 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c68e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c640c9450 0x7f4c640c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c68e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 0x7f4c6414eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c68e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c6415e420 0x7f4c6415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c68e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640c4a50 con 0x7f4c640c9450 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c68e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c641f20f0 con 0x7f4c6415e420 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c68e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c640c39b0 con 0x7f4c6415b460 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c63400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c640c9450 0x7f4c640c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c63400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c640c9450 0x7f4c640c32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57428/0 (socket says 172.21.15.39:57428) 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.611+0000 7f4c63400640 1 -- 172.21.15.39:0/2617036581 learned_addr learned my addr 172.21.15.39:0/2617036581 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c63e00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c6415e420 0x7f4c6415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:00.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c5a800640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 0x7f4c6414eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c63400640 1 -- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 msgr2=0x7f4c6414eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c63400640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 0x7f4c6414eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c63400640 1 -- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c6415e420 msgr2=0x7f4c6415d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c63400640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c6415e420 0x7f4c6415d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c63400640 1 -- 172.21.15.39:0/2617036581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c0094d0 con 0x7f4c640c9450 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c63e00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c6415e420 0x7f4c6415d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c63400640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c640c9450 0x7f4c640c32b0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400c430 tx=0x7f4c5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c5be00640 1 -- 172.21.15.39:0/2617036581 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54011790 con 0x7f4c640c9450 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c5be00640 1 -- 172.21.15.39:0/2617036581 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c540118f0 con 0x7f4c640c9450 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c5be00640 1 -- 172.21.15.39:0/2617036581 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54011ce0 con 0x7f4c640c9450 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c640c3c90 con 0x7f4c640c9450 2024-01-24T17:49:00.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.612+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c640c4250 con 0x7f4c640c9450 2024-01-24T17:49:00.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.613+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6404e840 con 0x7f4c640c9450 2024-01-24T17:49:00.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.614+0000 7f4c5be00640 1 -- 172.21.15.39:0/2617036581 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4c5401c460 con 0x7f4c640c9450 2024-01-24T17:49:00.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.615+0000 7f4c5be00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c3c06a430 0x7f4c3c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:00.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.615+0000 7f4c5be00640 1 -- 172.21.15.39:0/2617036581 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4c54088a80 con 0x7f4c640c9450 2024-01-24T17:49:00.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.616+0000 7f4c5a800640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c3c06a430 0x7f4c3c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:00.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.617+0000 7f4c5a800640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c3c06a430 0x7f4c3c06c8f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c036000 tx=0x7f4c4c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:00.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.620+0000 7f4c5be00640 1 -- 172.21.15.39:0/2617036581 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f4c54053ed0 con 0x7f4c640c9450 2024-01-24T17:49:00.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.840+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c64158780 con 0x7f4c640c9450 2024-01-24T17:49:00.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.842+0000 7f4c5be00640 1 -- 172.21.15.39:0/2617036581 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f4c540574f0 con 0x7f4c640c9450 2024-01-24T17:49:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:00.843 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3903887009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3903887009},{"type":"v1","addr":"172.21.15.39:6833","nonce":3903887009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:49:00.345558+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8118":{"gid":8118,"name":"c","rank":0,"incarnation":65,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.39:6835/1608906778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1608906778},{"type":"v1","addr":"172.21.15.39:6835","nonce":1608906778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:49:00.843 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 66 2024-01-24T17:49:00.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c3c06a430 msgr2=0x7f4c3c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:00.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c3c06a430 0x7f4c3c06c8f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c036000 tx=0x7f4c4c03c040 comp rx=0 tx=0).stop 2024-01-24T17:49:00.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c640c9450 msgr2=0x7f4c640c32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c640c9450 0x7f4c640c32b0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400c430 tx=0x7f4c5400c8f0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 shutdown_connections 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4c6415e420 0x7f4c6415d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f4c6415b460 0x7f4c6414eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4c3c06a430 0x7f4c3c06c8f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 --2- 172.21.15.39:0/2617036581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c640c9450 0x7f4c640c32b0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.844+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 >> 172.21.15.39:0/2617036581 conn(0x7f4c6406bf30 msgr2=0x7f4c64158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.845+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 shutdown_connections 2024-01-24T17:49:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:00.845+0000 7f4c68e00640 1 -- 172.21.15.39:0/2617036581 wait complete. 2024-01-24T17:49:01.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:48:26.880600+0000', 'modified': '2024-01-24T17:49:00.345558+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 104, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8118}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8118': {'gid': 8118, 'name': 'c', 'rank': 0, 'incarnation': 65, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.39:6835/1608906778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6834', 'nonce': 1608906778}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 1608906778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:49:01.268 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8118:up:reconnect 2024-01-24T17:49:02.270 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:49:02.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.497+0000 7f7d06200640 1 Processor -- start 2024-01-24T17:49:02.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.497+0000 7f7d06200640 1 -- start start 2024-01-24T17:49:02.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.498+0000 7f7d06200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f7d0015b5f0 0x7f7d000ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.498+0000 7f7d06200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d00071d40 0x7f7d0015f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7d06200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0015ade0 con 0x7f7d000adb80 2024-01-24T17:49:02.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7d06200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0015bb70 con 0x7f7d0015b5f0 2024-01-24T17:49:02.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7d06200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000b3540 con 0x7f7d00071d40 2024-01-24T17:49:02.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7cfea00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7d000adb80 0x7f7d00071630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58362/0 (socket says 172.21.15.39:58362) 2024-01-24T17:49:02.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7cfea00640 1 -- 172.21.15.39:0/1950234675 learned_addr learned my addr 172.21.15.39:0/1950234675 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:02.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7cffe00640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d00071d40 0x7f7d0015f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:02.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7cff400640 1 --2- 172.21.15.39:0/1950234675 >> v2:172.21.15.100:3300/0 conn(0x7f7d0015b5f0 0x7f7d000ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:02.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123057930 0 0) 0x7f7d0015ade0 con 0x7f7d000adb80 2024-01-24T17:49:02.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.499+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce0003520 con 0x7f7d000adb80 2024-01-24T17:49:02.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789294850 0 0) 0x7f7ce0003520 con 0x7f7d000adb80 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d00071d40 msgr2=0x7f7d0015f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d00071d40 0x7f7d0015f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 >> v2:172.21.15.100:3300/0 conn(0x7f7d0015b5f0 msgr2=0x7f7d000ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 --2- 172.21.15.39:0/1950234675 >> v2:172.21.15.100:3300/0 conn(0x7f7d0015b5f0 0x7f7d000ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0015be10 con 0x7f7d000adb80 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7ce8001670 con 0x7f7d000adb80 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 >> v1:172.21.15.39:6789/0 conn(0x7f7d000adb80 legacy=0x7f7d00071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d00071d40 0x7f7ce0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ce0003ef0 0x7f7ce00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7ce0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0015bb70 con 0x7f7ce0004d00 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000b3540 con 0x7f7ce0003ef0 2024-01-24T17:49:02.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce000b980 con 0x7f7d00071d40 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.500+0000 7f7cffe00640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7ce0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.501+0000 7f7cff400640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d00071d40 0x7f7ce0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.501+0000 7f7cffe00640 1 -- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d00071d40 msgr2=0x7f7ce0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.501+0000 7f7cfea00640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ce0003ef0 0x7f7ce00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.501+0000 7f7cffe00640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d00071d40 0x7f7ce0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.501+0000 7f7cffe00640 1 -- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ce0003ef0 msgr2=0x7f7ce00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.501+0000 7f7cffe00640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ce0003ef0 0x7f7ce00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.501+0000 7f7cffe00640 1 -- 172.21.15.39:0/1950234675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce0003520 con 0x7f7ce0004d00 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.501+0000 7f7cfea00640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ce0003ef0 0x7f7ce00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:02.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.502+0000 7f7cffe00640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7ce0005180 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009fc0 tx=0x7f7ce00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a65e6f8c6a7924d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:02.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.502+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf802cc00 con 0x7f7ce0004d00 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.502+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf800f040 con 0x7f7ce0004d00 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.502+0000 7f7cfe000640 1 -- 172.21.15.39:0/1950234675 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8034440 con 0x7f7ce0004d00 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.503+0000 7f7d06200640 1 -- 172.21.15.39:0/1950234675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 msgr2=0x7f7ce0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.503+0000 7f7d06200640 1 --2- 172.21.15.39:0/1950234675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7ce0005180 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009fc0 tx=0x7f7ce00057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.503+0000 7f7cffe00640 1 -- 172.21.15.39:0/1950234675 reap_dead start 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.503+0000 7f7d06200640 1 -- 172.21.15.39:0/1950234675 shutdown_connections 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.503+0000 7f7d06200640 1 -- 172.21.15.39:0/1950234675 >> 172.21.15.39:0/1950234675 conn(0x7f7d0006a220 msgr2=0x7f7d000ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.503+0000 7f7d06200640 1 -- 172.21.15.39:0/1950234675 shutdown_connections 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.503+0000 7f7d06200640 1 -- 172.21.15.39:0/1950234675 wait complete. 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.505+0000 7f7d06200640 1 Processor -- start 2024-01-24T17:49:02.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.505+0000 7f7d06200640 1 -- start start 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7d06200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7d000c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7d06200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d00159e40 0x7f7d001595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7d06200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d000cc220 0x7f7d000cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7d06200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000ccbe0 con 0x7f7ce0004d00 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7d06200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000ae470 con 0x7f7d00159e40 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7d06200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000ae750 con 0x7f7d000cc220 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7cff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7d000c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7cff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7d000c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57458/0 (socket says 172.21.15.39:57458) 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.506+0000 7f7cff400640 1 -- 172.21.15.39:0/1039909867 learned_addr learned my addr 172.21.15.39:0/1039909867 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cfea00640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d00159e40 0x7f7d001595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:02.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cff400640 1 -- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d000cc220 msgr2=0x7f7d000cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cff400640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d000cc220 0x7f7d000cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cff400640 1 -- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d00159e40 msgr2=0x7f7d001595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cff400640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d00159e40 0x7f7d001595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cff400640 1 -- 172.21.15.39:0/1039909867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d000aea30 con 0x7f7ce0004d00 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cff400640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7d000c8270 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00116d0 tx=0x7f7cf0011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cf7400640 1 -- 172.21.15.39:0/1039909867 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf000f940 con 0x7f7ce0004d00 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cf7400640 1 -- 172.21.15.39:0/1039909867 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf000faa0 con 0x7f7ce0004d00 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d000aed10 con 0x7f7ce0004d00 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.507+0000 7f7cf7400640 1 -- 172.21.15.39:0/1039909867 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf000fe10 con 0x7f7ce0004d00 2024-01-24T17:49:02.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.508+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d00159af0 con 0x7f7ce0004d00 2024-01-24T17:49:02.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.509+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d00073f10 con 0x7f7ce0004d00 2024-01-24T17:49:02.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.510+0000 7f7cf7400640 1 -- 172.21.15.39:0/1039909867 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7cf0015460 con 0x7f7ce0004d00 2024-01-24T17:49:02.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.510+0000 7f7cf7400640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7cd806a430 0x7f7cd806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:02.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.511+0000 7f7cf7400640 1 -- 172.21.15.39:0/1039909867 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7cf00893b0 con 0x7f7ce0004d00 2024-01-24T17:49:02.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.511+0000 7f7cfea00640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7cd806a430 0x7f7cd806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:02.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.513+0000 7f7cfea00640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7cd806a430 0x7f7cd806c8f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80017a0 tx=0x7f7ce8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:02.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.515+0000 7f7cf7400640 1 -- 172.21.15.39:0/1039909867 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7cf0054800 con 0x7f7ce0004d00 2024-01-24T17:49:02.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.744+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d000b2850 con 0x7f7ce0004d00 2024-01-24T17:49:02.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.744+0000 7f7cf7400640 1 -- 172.21.15.39:0/1039909867 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f7cf0057e20 con 0x7f7ce0004d00 2024-01-24T17:49:02.745 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:02.745 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3903887009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3903887009},{"type":"v1","addr":"172.21.15.39:6833","nonce":3903887009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:49:02.371003+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8118":{"gid":8118,"name":"c","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/1608906778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1608906778},{"type":"v1","addr":"172.21.15.39:6835","nonce":1608906778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:49:02.746 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 68 2024-01-24T17:49:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7cd806a430 msgr2=0x7f7cd806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7cd806a430 0x7f7cd806c8f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80017a0 tx=0x7f7ce8009040 comp rx=0 tx=0).stop 2024-01-24T17:49:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 msgr2=0x7f7d000c8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7d000c8270 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00116d0 tx=0x7f7cf0011b90 comp rx=0 tx=0).stop 2024-01-24T17:49:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 shutdown_connections 2024-01-24T17:49:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7d000cc220 0x7f7d000cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7d00159e40 0x7f7d001595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7cd806a430 0x7f7cd806c8f0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 --2- 172.21.15.39:0/1039909867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ce0004d00 0x7f7d000c8270 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 >> 172.21.15.39:0/1039909867 conn(0x7f7d0006a220 msgr2=0x7f7d000cd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 shutdown_connections 2024-01-24T17:49:02.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:02.747+0000 7f7d06200640 1 -- 172.21.15.39:0/1039909867 wait complete. 2024-01-24T17:49:03.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:48:26.880600+0000', 'modified': '2024-01-24T17:49:02.371003+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 104, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8118}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8118': {'gid': 8118, 'name': 'c', 'rank': 0, 'incarnation': 65, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.39:6835/1608906778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6834', 'nonce': 1608906778}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 1608906778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:49:03.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T17:49:03.171 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-24T17:49:03.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.395+0000 7f9f9ca00640 1 Processor -- start 2024-01-24T17:49:03.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.396+0000 7f9f9ca00640 1 -- start start 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.397+0000 7f9f9ca00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9f940b2970 0x7f9f940b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.397+0000 7f9f9ca00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f940ce130 con 0x7f9f941432a0 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.397+0000 7f9f9ca00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f940b32e0 con 0x7f9f940b35f0 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.397+0000 7f9f9ca00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f940cd270 con 0x7f9f940b2970 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.398+0000 7f9f9a000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f9f941432a0 0x7f9f94159e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56854/0 (socket says 172.21.15.39:56854) 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.398+0000 7f9f9a000640 1 -- 172.21.15.39:0/3321272483 learned_addr learned my addr 172.21.15.39:0/3321272483 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.398+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/3321272483 >> v2:172.21.15.39:3300/0 conn(0x7f9f940b2970 0x7f9f940b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.398+0000 7f9f98c00640 1 -- 172.21.15.39:0/3321272483 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503220517 0 0) 0x7f9f940ce130 con 0x7f9f941432a0 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.398+0000 7f9f98c00640 1 -- 172.21.15.39:0/3321272483 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f74003520 con 0x7f9f941432a0 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.398+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3321272483 >> v1:172.21.15.100:6789/0 conn(0x7f9f940b35f0 legacy=0x7f9f940ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.398+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3321272483 >> v1:172.21.15.39:6789/0 conn(0x7f9f941432a0 legacy=0x7f9f94159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.398+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3321272483 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f940cd520 con 0x7f9f940b2970 2024-01-24T17:49:03.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.399+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/3321272483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f940b2da0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009d90 tx=0x7f9f8c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb855c25f2caa18 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f98c00640 1 -- 172.21.15.39:0/3321272483 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c02ccf0 con 0x7f9f940b2970 2024-01-24T17:49:03.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f98c00640 1 -- 172.21.15.39:0/3321272483 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c02ce50 con 0x7f9f940b2970 2024-01-24T17:49:03.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f98c00640 1 -- 172.21.15.39:0/3321272483 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c034660 con 0x7f9f940b2970 2024-01-24T17:49:03.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f9ca00640 1 -- 172.21.15.39:0/3321272483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 msgr2=0x7f9f940b2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f9ca00640 1 --2- 172.21.15.39:0/3321272483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f940b2da0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009d90 tx=0x7f9f8c02bf40 comp rx=0 tx=0).stop 2024-01-24T17:49:03.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f9ca00640 1 -- 172.21.15.39:0/3321272483 shutdown_connections 2024-01-24T17:49:03.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f9ca00640 1 --2- 172.21.15.39:0/3321272483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f940b2da0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f9ca00640 1 -- 172.21.15.39:0/3321272483 >> 172.21.15.39:0/3321272483 conn(0x7f9f94065ff0 msgr2=0x7f9f94069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:03.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.400+0000 7f9f9ca00640 1 -- 172.21.15.39:0/3321272483 shutdown_connections 2024-01-24T17:49:03.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.401+0000 7f9f9ca00640 1 -- 172.21.15.39:0/3321272483 wait complete. 2024-01-24T17:49:03.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.402+0000 7f9f9ca00640 1 Processor -- start 2024-01-24T17:49:03.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.402+0000 7f9f9ca00640 1 -- start start 2024-01-24T17:49:03.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.403+0000 7f9f9ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f941f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.403+0000 7f9f9ca00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f940b35f0 0x7f9f940ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.403+0000 7f9f9ca00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f941432a0 0x7f9f9413c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.403+0000 7f9f9ca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f94141b90 con 0x7f9f940b2970 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.403+0000 7f9f9ca00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9413a210 con 0x7f9f940b35f0 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.403+0000 7f9f9ca00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f941412f0 con 0x7f9f941432a0 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.403+0000 7f9f9a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f941f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f941f2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41798/0 (socket says 172.21.15.39:41798) 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9a000640 1 -- 172.21.15.39:0/2996665885 learned_addr learned my addr 172.21.15.39:0/2996665885 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f99600640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f940b35f0 0x7f9f940ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9a000640 1 -- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f941432a0 msgr2=0x7f9f9413c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f941432a0 0x7f9f9413c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9a000640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f941432a0 0x7f9f9413c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9a000640 1 -- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f940b35f0 msgr2=0x7f9f940ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9a000640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f940b35f0 0x7f9f940ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9a000640 1 -- 172.21.15.39:0/2996665885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c009a70 con 0x7f9f940b2970 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f99600640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f940b35f0 0x7f9f940ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:03.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.404+0000 7f9f9a000640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f941f2430 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400c440 tx=0x7f9f8400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.405+0000 7f9f92000640 1 -- 172.21.15.39:0/2996665885 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84005ce0 con 0x7f9f940b2970 2024-01-24T17:49:03.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.405+0000 7f9f92000640 1 -- 172.21.15.39:0/2996665885 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f84005e40 con 0x7f9f940b2970 2024-01-24T17:49:03.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.405+0000 7f9f92000640 1 -- 172.21.15.39:0/2996665885 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f840072b0 con 0x7f9f940b2970 2024-01-24T17:49:03.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.405+0000 7f9f9ca00640 1 -- 172.21.15.39:0/2996665885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f941415d0 con 0x7f9f940b2970 2024-01-24T17:49:03.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.405+0000 7f9f9ca00640 1 -- 172.21.15.39:0/2996665885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f94139ea0 con 0x7f9f940b2970 2024-01-24T17:49:03.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.407+0000 7f9f92000640 1 -- 172.21.15.39:0/2996665885 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9f84007410 con 0x7f9f940b2970 2024-01-24T17:49:03.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.407+0000 7f9f92000640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f6c06a430 0x7f9f6c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.408+0000 7f9f92000640 1 -- 172.21.15.39:0/2996665885 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9f84088080 con 0x7f9f940b2970 2024-01-24T17:49:03.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.408+0000 7f9f99600640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f6c06a430 0x7f9f6c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.408+0000 7f9f9f774740 1 Processor -- start 2024-01-24T17:49:03.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.409+0000 7f9f9f774740 1 -- start start 2024-01-24T17:49:03.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.409+0000 7f9f99600640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f6c06a430 0x7f9f6c06c8f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9f941f5900 tx=0x7f9f7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.409+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f973010 0x56357f9733f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.409+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f942f90 0x56357f970280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.409+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9707c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.409+0000 7f9f9f774740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357f91e6c0 con 0x56357f942f90 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f9f774740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357f8b3070 con 0x56357f973010 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f9f774740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56357f8b0800 con 0x56357f946b90 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f99600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f942f90 0x56357f970280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f99600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f942f90 0x56357f970280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41814/0 (socket says 172.21.15.39:41814) 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f99600640 1 -- 172.21.15.39:0/1685242628 learned_addr learned my addr 172.21.15.39:0/1685242628 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f9a000640 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f973010 0x56357f9733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f99600640 1 -- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 msgr2=0x56357f9707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f99600640 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9707c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f99600640 1 -- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f973010 msgr2=0x56357f9733f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f99600640 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f973010 0x56357f9733f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f99600640 1 -- 172.21.15.39:0/1685242628 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56357f873780 con 0x56357f942f90 2024-01-24T17:49:03.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:03.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.410+0000 7f9f9a000640 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f973010 0x56357f9733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:03.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.411+0000 7f9f99600640 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f942f90 0x56357f970280 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00d7d0 tx=0x7f9f7c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277ac0853a00613d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.411+0000 7f9f8aa00640 1 -- 172.21.15.39:0/1685242628 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c014ba0 con 0x56357f942f90 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.411+0000 7f9f8aa00640 1 -- 172.21.15.39:0/1685242628 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7c014d00 con 0x56357f942f90 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.411+0000 7f9f8aa00640 1 -- 172.21.15.39:0/1685242628 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c007230 con 0x56357f942f90 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.411+0000 7f9f9f774740 1 -- 172.21.15.39:0/1685242628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f942f90 msgr2=0x56357f970280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.411+0000 7f9f9f774740 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f942f90 0x56357f970280 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00d7d0 tx=0x7f9f7c010000 comp rx=0 tx=0).stop 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 -- 172.21.15.39:0/1685242628 shutdown_connections 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f942f90 0x56357f970280 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 --2- 172.21.15.39:0/1685242628 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f973010 0x56357f9733f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 -- 172.21.15.39:0/1685242628 >> 172.21.15.39:0/1685242628 conn(0x56357f9452a0 msgr2=0x56357f98de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 -- 172.21.15.39:0/1685242628 shutdown_connections 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 -- 172.21.15.39:0/1685242628 wait complete. 2024-01-24T17:49:03.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 -- start start 2024-01-24T17:49:03.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.412+0000 7f9f9f774740 1 Processor -- start 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.413+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f946b90 0x56357f96a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.413+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f973010 0x56357f96af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.413+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f95d4a0 0x56357f973a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.413+0000 7f9f9f774740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357f8ac0f0 con 0x56357f95d4a0 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.413+0000 7f9f9f774740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357f84d8c0 con 0x56357f946b90 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.413+0000 7f9f9f774740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56357f84d560 con 0x56357f973010 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.413+0000 7f9f9f774740 10 client.0 _ll_register_callbacks cb 0x56357f8a5a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f95d4a0 0x56357f973a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f95d4a0 0x56357f973a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41820/0 (socket says 172.21.15.39:41820) 2024-01-24T17:49:03.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3521719771 learned_addr learned my addr 172.21.15.39:0/3521719771 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3521719771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f973010 msgr2=0x56357f96af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9a000640 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f946b90 0x56357f96a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f973010 0x56357f96af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3521719771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f946b90 msgr2=0x56357f96a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f946b90 0x56357f96a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3521719771 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f940b32e0 con 0x56357f95d4a0 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f95d4a0 0x56357f973a50 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c037040 tx=0x7f9f8c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131bee2cd0f43543 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.414+0000 7f9f83400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f83400640 1 -- 172.21.15.39:0/3521719771 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c034ba0 con 0x56357f95d4a0 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f83400640 1 -- 172.21.15.39:0/3521719771 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9f8c034d00 con 0x56357f95d4a0 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 1 -- 172.21.15.39:0/3521719771 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56357f8d6640 con 0x56357f95d4a0 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f83400640 1 -- 172.21.15.39:0/3521719771 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f8c003990 con 0x56357f95d4a0 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 10 client.8151 fetch_fsmap learned FSMap version 68 2024-01-24T17:49:03.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 1 -- 172.21.15.39:0/3521719771 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) v3 -- 0x56357f8aa770 con 0x56357f95d4a0 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f83400640 1 -- 172.21.15.39:0/3521719771 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56357f8d6640 con 0x56357f95d4a0 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f83400640 1 -- 172.21.15.39:0/3521719771 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 68) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9f8c003af0 con 0x56357f95d4a0 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 10 client.8151 fetch_fsmap finished waiting for FSMap version 68 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 10 client.8151 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 20 client.8151 populate_metadata read hostname 'smithi039' 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56357f9a90c0 0x56357f9ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 4 client.8151 mds_command: new command op to 8118 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.415+0000 7f9f9f774740 1 -- 172.21.15.39:0/3521719771 --> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56357f89e940 con 0x56357f9a90c0 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.416+0000 7f9f9a000640 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56357f9a90c0 0x56357f9ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.416+0000 7f9f9a000640 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56357f9a90c0 0x56357f9ab4b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce42ab78bd079e64 server_cookie=9593237907b190f2 in_seq=0 out_seq=0 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.416+0000 7f9f83400640 10 client.8151 ms_handle_connect on v2:172.21.15.39:6834/1608906778 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f83400640 1 -- 172.21.15.39:0/3521719771 <== mds.0 v2:172.21.15.39:6834/1608906778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56357f89e940 con 0x56357f9a90c0 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f83400640 10 client.8151 handle_command_reply: tid=0 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 client.8151 shutdown 2024-01-24T17:49:03.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 -- 172.21.15.39:0/3521719771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f95d4a0 msgr2=0x56357f973a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f95d4a0 0x56357f973a50 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c037040 tx=0x7f9f8c004ab0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 -- 172.21.15.39:0/3521719771 shutdown_connections 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56357f9a90c0 0x56357f9ab4b0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f95d4a0 0x56357f973a50 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f973010 0x56357f96af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3521719771 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f946b90 0x56357f96a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f83400640 0 client.8151 ms_handle_reset on v2:172.21.15.39:6834/1608906778 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 -- 172.21.15.39:0/3521719771 >> 172.21.15.39:0/3521719771 conn(0x56357f942f90 msgr2=0x56357f947550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 -- 172.21.15.39:0/3521719771 shutdown_connections 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.417+0000 7f9f9f774740 1 -- 172.21.15.39:0/3521719771 wait complete. 2024-01-24T17:49:03.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.418+0000 7f9f9f774740 20 client.8151 trim_cache size 0 max 16384 2024-01-24T17:49:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.433+0000 7f9f9f774740 1 Processor -- start 2024-01-24T17:49:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.434+0000 7f9f9f774740 1 -- start start 2024-01-24T17:49:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.434+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f946b90 0x56357f947090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.434+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f95d4a0 0x56357f9475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.434+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f973010 0x56357f942290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.434+0000 7f9f9f774740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357f613f20 con 0x56357f946b90 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.434+0000 7f9f9f774740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357f970b60 con 0x56357f95d4a0 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.434+0000 7f9f9f774740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56357f973a30 con 0x56357f973010 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f946b90 0x56357f947090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f946b90 0x56357f947090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41826/0 (socket says 172.21.15.39:41826) 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 -- 172.21.15.39:0/4156272268 learned_addr learned my addr 172.21.15.39:0/4156272268 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 -- 172.21.15.39:0/4156272268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f973010 msgr2=0x56357f942290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 --2- 172.21.15.39:0/4156272268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f973010 0x56357f942290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 -- 172.21.15.39:0/4156272268 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f95d4a0 msgr2=0x56357f9475d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 --2- 172.21.15.39:0/4156272268 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f95d4a0 0x56357f9475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 -- 172.21.15.39:0/4156272268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56357f8e02f0 con 0x56357f946b90 2024-01-24T17:49:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f9a000640 1 --2- 172.21.15.39:0/4156272268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f946b90 0x56357f947090 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f9f840571e0 tx=0x7f9f8405d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a76ee9c827c7221 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.435+0000 7f9f8b400640 1 -- 172.21.15.39:0/4156272268 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84055450 con 0x56357f946b90 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f8b400640 1 -- 172.21.15.39:0/4156272268 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f84086520 con 0x56357f946b90 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f8b400640 1 -- 172.21.15.39:0/4156272268 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8405d0b0 con 0x56357f946b90 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 -- 172.21.15.39:0/4156272268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f946b90 msgr2=0x56357f947090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 --2- 172.21.15.39:0/4156272268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f946b90 0x56357f947090 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f9f840571e0 tx=0x7f9f8405d040 comp rx=0 tx=0).stop 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 -- 172.21.15.39:0/4156272268 shutdown_connections 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 --2- 172.21.15.39:0/4156272268 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f973010 0x56357f942290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 --2- 172.21.15.39:0/4156272268 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f95d4a0 0x56357f9475d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 --2- 172.21.15.39:0/4156272268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f946b90 0x56357f947090 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 -- 172.21.15.39:0/4156272268 >> 172.21.15.39:0/4156272268 conn(0x56357f942f90 msgr2=0x56357f96ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:03.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 -- 172.21.15.39:0/4156272268 shutdown_connections 2024-01-24T17:49:03.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.436+0000 7f9f9f774740 1 -- 172.21.15.39:0/4156272268 wait complete. 2024-01-24T17:49:03.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.437+0000 7f9f9f774740 1 -- start start 2024-01-24T17:49:03.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.437+0000 7f9f9f774740 1 Processor -- start 2024-01-24T17:49:03.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f95d4a0 0x56357f942290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9f774740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f973010 0x56357f9427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9f774740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357f9abfa0 con 0x56357f973010 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9f774740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x56357f990900 con 0x56357f95d4a0 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9f774740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56357f98e860 con 0x56357f946b90 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9f774740 10 client.0 _ll_register_callbacks cb 0x7f9f8c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f973010 0x56357f9427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f973010 0x56357f9427d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41830/0 (socket says 172.21.15.39:41830) 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.438+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3783789249 learned_addr learned my addr 172.21.15.39:0/3783789249 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f9a000640 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3783789249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 msgr2=0x56357f9f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9f3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3783789249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f95d4a0 msgr2=0x56357f942290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f95d4a0 0x56357f942290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f9aa00640 1 -- 172.21.15.39:0/3783789249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f94139ea0 con 0x56357f973010 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f9aa00640 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f973010 0x56357f9427d0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c002230 tx=0x7f9f8c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e4f49a055bdb81 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f83400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f83400640 1 -- 172.21.15.39:0/3783789249 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c003f10 con 0x56357f973010 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f83400640 1 -- 172.21.15.39:0/3783789249 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9f8c04a7e0 con 0x56357f973010 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.439+0000 7f9f9f774740 1 -- 172.21.15.39:0/3783789249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f7c014d00 con 0x56357f973010 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f83400640 1 -- 172.21.15.39:0/3783789249 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f8c04a9a0 con 0x56357f973010 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9f774740 10 client.8157 fetch_fsmap learned FSMap version 68 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9f774740 1 -- 172.21.15.39:0/3783789249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) v3 -- 0x56357f84cad0 con 0x56357f973010 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f83400640 1 -- 172.21.15.39:0/3783789249 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7c014d00 con 0x56357f973010 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f83400640 1 -- 172.21.15.39:0/3783789249 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 68) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9f8c04ab00 con 0x56357f973010 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9f774740 10 client.8157 fetch_fsmap finished waiting for FSMap version 68 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9f774740 10 client.8157 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9f774740 20 client.8157 populate_metadata read hostname 'smithi039' 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56357f9ab460 0x56357f975ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9f774740 4 client.8157 mds_command: new command op to 8118 tid=0[{"prefix": "status"}] 2024-01-24T17:49:03.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9f774740 1 -- 172.21.15.39:0/3783789249 --> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56357f8a5170 con 0x56357f9ab460 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.440+0000 7f9f9a000640 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56357f9ab460 0x56357f975ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f83400640 10 client.8157 ms_handle_connect on v2:172.21.15.39:6834/1608906778 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9a000640 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56357f9ab460 0x56357f975ac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ecb45836b54c3c0 server_cookie=e49537776e87eba6 in_seq=0 out_seq=0 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f83400640 1 -- 172.21.15.39:0/3783789249 <== mds.0 v2:172.21.15.39:6834/1608906778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x56357f8a5170 con 0x56357f9ab460 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f83400640 10 client.8157 handle_command_reply: tid=0 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 client.8157 shutdown 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 -- 172.21.15.39:0/3783789249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f973010 msgr2=0x56357f9427d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f973010 0x56357f9427d0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c002230 tx=0x7f9f8c009750 comp rx=0 tx=0).stop 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 -- 172.21.15.39:0/3783789249 shutdown_connections 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56357f9ab460 0x56357f975ac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56357f973010 0x56357f9427d0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x56357f95d4a0 0x56357f942290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 --2- 172.21.15.39:0/3783789249 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x56357f946b90 0x56357f9f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f83400640 0 client.8157 ms_handle_reset on v2:172.21.15.39:6834/1608906778 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 -- 172.21.15.39:0/3783789249 >> 172.21.15.39:0/3783789249 conn(0x56357f942f90 msgr2=0x56357f968800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 -- 172.21.15.39:0/3783789249 shutdown_connections 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.441+0000 7f9f9f774740 1 -- 172.21.15.39:0/3783789249 wait complete. 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9f774740 20 client.8157 trim_cache size 0 max 16384 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 8118, 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:49:03.446 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:49:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:49:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 4.0940599940000002, 2024-01-24T17:49:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 68, 2024-01-24T17:49:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 104, 2024-01-24T17:49:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 104, 2024-01-24T17:49:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 4.762231892 2024-01-24T17:49:03.447 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 -- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f6c06a430 msgr2=0x7f9f6c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f6c06a430 0x7f9f6c06c8f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9f941f5900 tx=0x7f9f7c009040 comp rx=0 tx=0).stop 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 -- 172.21.15.39:0/2996665885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 msgr2=0x7f9f941f2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f941f2430 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400c440 tx=0x7f9f8400c900 comp rx=0 tx=0).stop 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 -- 172.21.15.39:0/2996665885 shutdown_connections 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9f941432a0 0x7f9f9413c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9f940b35f0 0x7f9f940ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9f6c06a430 0x7f9f6c06c8f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 --2- 172.21.15.39:0/2996665885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9f940b2970 0x7f9f941f2430 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 -- 172.21.15.39:0/2996665885 >> 172.21.15.39:0/2996665885 conn(0x7f9f94065ff0 msgr2=0x7f9f94070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:03.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 -- 172.21.15.39:0/2996665885 shutdown_connections 2024-01-24T17:49:03.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.442+0000 7f9f9ca00640 1 -- 172.21.15.39:0/2996665885 wait complete. 2024-01-24T17:49:03.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:49:03.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.667+0000 7f5012600640 1 Processor -- start 2024-01-24T17:49:03.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.667+0000 7f5012600640 1 -- start start 2024-01-24T17:49:03.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.669+0000 7f5012600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f500c153930 0x7f500c06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.669+0000 7f5012600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f500c15db30 0x7f500c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.669+0000 7f5012600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0c36a0 con 0x7f500c0ce390 2024-01-24T17:49:03.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.669+0000 7f5012600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c0c3980 con 0x7f500c153930 2024-01-24T17:49:03.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.669+0000 7f5012600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c0c3c60 con 0x7f500c15db30 2024-01-24T17:49:03.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.669+0000 7f5011200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f500c153930 0x7f500c06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.669+0000 7f5011200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f500c153930 0x7f500c06fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41834/0 (socket says 172.21.15.39:41834) 2024-01-24T17:49:03.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.669+0000 7f5011200640 1 -- 172.21.15.39:0/3650993162 learned_addr learned my addr 172.21.15.39:0/3650993162 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:03.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.670+0000 7f5011200640 1 -- 172.21.15.39:0/3650993162 >> v2:172.21.15.100:3300/0 conn(0x7f500c15db30 msgr2=0x7f500c0cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.670+0000 7f5011200640 1 --2- 172.21.15.39:0/3650993162 >> v2:172.21.15.100:3300/0 conn(0x7f500c15db30 0x7f500c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.670+0000 7f5011200640 1 -- 172.21.15.39:0/3650993162 >> v1:172.21.15.100:6789/0 conn(0x7f500c0ce390 legacy=0x7f500c0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:03.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.670+0000 7f5011200640 1 -- 172.21.15.39:0/3650993162 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c0c3f40 con 0x7f500c153930 2024-01-24T17:49:03.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.671+0000 7f5011200640 1 --2- 172.21.15.39:0/3650993162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c153930 0x7f500c06fe90 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f500c0cd480 tx=0x7f4ffc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730a691614ca609b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.671+0000 7f500b400640 1 -- 172.21.15.39:0/3650993162 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc031870 con 0x7f500c153930 2024-01-24T17:49:03.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.671+0000 7f500b400640 1 -- 172.21.15.39:0/3650993162 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ffc0319d0 con 0x7f500c153930 2024-01-24T17:49:03.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.671+0000 7f500b400640 1 -- 172.21.15.39:0/3650993162 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc031d60 con 0x7f500c153930 2024-01-24T17:49:03.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.672+0000 7f5012600640 1 -- 172.21.15.39:0/3650993162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c153930 msgr2=0x7f500c06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.672+0000 7f5012600640 1 --2- 172.21.15.39:0/3650993162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c153930 0x7f500c06fe90 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f500c0cd480 tx=0x7f4ffc02b910 comp rx=0 tx=0).stop 2024-01-24T17:49:03.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.672+0000 7f5012600640 1 -- 172.21.15.39:0/3650993162 shutdown_connections 2024-01-24T17:49:03.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.672+0000 7f5012600640 1 --2- 172.21.15.39:0/3650993162 >> v2:172.21.15.100:3300/0 conn(0x7f500c15db30 0x7f500c0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.672+0000 7f5012600640 1 --2- 172.21.15.39:0/3650993162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c153930 0x7f500c06fe90 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.672+0000 7f5012600640 1 -- 172.21.15.39:0/3650993162 >> 172.21.15.39:0/3650993162 conn(0x7f500c06bf30 msgr2=0x7f500c06c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:03.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.672+0000 7f5012600640 1 -- 172.21.15.39:0/3650993162 shutdown_connections 2024-01-24T17:49:03.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.672+0000 7f5012600640 1 -- 172.21.15.39:0/3650993162 wait complete. 2024-01-24T17:49:03.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.674+0000 7f5012600640 1 Processor -- start 2024-01-24T17:49:03.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.674+0000 7f5012600640 1 -- start start 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.675+0000 7f5012600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500c0ce390 0x7f500c0b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.675+0000 7f5012600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500c153930 0x7f500c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.675+0000 7f5012600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c15db30 0x7f500c0b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.675+0000 7f5012600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0c6320 con 0x7f500c15db30 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.675+0000 7f5012600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0c56a0 con 0x7f500c153930 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.675+0000 7f5012600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0c5980 con 0x7f500c0ce390 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c15db30 0x7f500c0b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c15db30 0x7f500c0b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41842/0 (socket says 172.21.15.39:41842) 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 -- 172.21.15.39:0/3039093075 learned_addr learned my addr 172.21.15.39:0/3039093075 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011200640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500c0ce390 0x7f500c0b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f500be00640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500c153930 0x7f500c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 -- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500c0ce390 msgr2=0x7f500c0b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500c0ce390 0x7f500c0b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 -- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500c153930 msgr2=0x7f500c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500c153930 0x7f500c0b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 -- 172.21.15.39:0/3039093075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc0094d0 con 0x7f500c15db30 2024-01-24T17:49:03.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.676+0000 7f5011c00640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c15db30 0x7f500c0b4080 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f500400b990 tx=0x7f500400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.677+0000 7f5008c00640 1 -- 172.21.15.39:0/3039093075 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500400cba0 con 0x7f500c15db30 2024-01-24T17:49:03.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.677+0000 7f5008c00640 1 -- 172.21.15.39:0/3039093075 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500400cd00 con 0x7f500c15db30 2024-01-24T17:49:03.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.677+0000 7f5008c00640 1 -- 172.21.15.39:0/3039093075 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004014680 con 0x7f500c15db30 2024-01-24T17:49:03.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.677+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c0c5c60 con 0x7f500c15db30 2024-01-24T17:49:03.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.677+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c0b2360 con 0x7f500c15db30 2024-01-24T17:49:03.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.678+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c04e840 con 0x7f500c15db30 2024-01-24T17:49:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.679+0000 7f5008c00640 1 -- 172.21.15.39:0/3039093075 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f500401d3b0 con 0x7f500c15db30 2024-01-24T17:49:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.680+0000 7f5008c00640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4fe406a430 0x7f4fe406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.680+0000 7f5008c00640 1 -- 172.21.15.39:0/3039093075 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5004088960 con 0x7f500c15db30 2024-01-24T17:49:03.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.681+0000 7f5011200640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4fe406a430 0x7f4fe406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:03.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.682+0000 7f5011200640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4fe406a430 0x7f4fe406c8f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc036000 tx=0x7f4ffc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:03.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.684+0000 7f5008c00640 1 -- 172.21.15.39:0/3039093075 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5004053ec0 con 0x7f500c15db30 2024-01-24T17:49:03.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.917+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f500c0c87f0 con 0x7f500c15db30 2024-01-24T17:49:03.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.918+0000 7f5008c00640 1 -- 172.21.15.39:0/3039093075 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v68) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5004019020 con 0x7f500c15db30 2024-01-24T17:49:03.919 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:49:03.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.920+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4fe406a430 msgr2=0x7f4fe406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.920+0000 7f5012600640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4fe406a430 0x7f4fe406c8f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc036000 tx=0x7f4ffc03c040 comp rx=0 tx=0).stop 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.920+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c15db30 msgr2=0x7f500c0b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.920+0000 7f5012600640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c15db30 0x7f500c0b4080 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f500400b990 tx=0x7f500400be50 comp rx=0 tx=0).stop 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.921+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 shutdown_connections 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.921+0000 7f5012600640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f4fe406a430 0x7f4fe406c8f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.921+0000 7f5012600640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f500c15db30 0x7f500c0b4080 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.921+0000 7f5012600640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f500c153930 0x7f500c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.921+0000 7f5012600640 1 --2- 172.21.15.39:0/3039093075 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f500c0ce390 0x7f500c0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.921+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 >> 172.21.15.39:0/3039093075 conn(0x7f500c06bf30 msgr2=0x7f500c070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.921+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 shutdown_connections 2024-01-24T17:49:03.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:03.921+0000 7f5012600640 1 -- 172.21.15.39:0/3039093075 wait complete. 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:49:03.941 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:49:03.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:03.942 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-01-24T17:49:03.996 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:49:03.996 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:49:03.996 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:49:03.996 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:03.996 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42188sec preferred_lft 42188sec 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-01-24T17:49:03.997 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::10c4:acff:fe2a:595b/64 scope link 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether e2:b9:b5:7c:93:51 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::e0b9:b5ff:fe7c:9351/64 scope link 2024-01-24T17:49:03.998 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:03.999 INFO:teuthology.orchestra.run.smithi039.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:49:03.999 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 42:e4:51:49:e5:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:49:03.999 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::40e4:51ff:fe49:e58f/64 scope link 2024-01-24T17:49:03.999 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:04.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:04.000 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:49:04.096 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-01-24T17:49:04.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:04.101 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-01-24T17:49:04.158 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-01-24T17:49:04.158 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:49:04.158 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T17:49:04.158 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T17:49:04.159 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:49:04.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:04.159 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec bash -c ' 2024-01-24T17:49:04.159 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-01-24T17:49:04.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T17:49:04.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T17:49:04.159 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T17:49:04.159 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-01-24T17:49:04.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:04.355 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:49:04.361 INFO:tasks.ceph.mds.c.smithi039.stderr:2024-01-24T17:49:04.358+0000 7fd769c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:49:04.407 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:49:04.407 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:49:04.408 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:49:04.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:04.408 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-01-24T17:49:04.501 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-24T17:49:04.501 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:49:04.502 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-24T17:49:04.557 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-24T17:49:04.558 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:49:04.558 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:49:04.609 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:49:04.699 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:49:04.699 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:49:04.794 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:49:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:49:04.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:04.799 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:49:04.857 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:49:04.858 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-24T17:49:04.865 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-01-24T17:49:04.943 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:49:04.944 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:49:05.015 INFO:teuthology.orchestra.run.smithi039.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:49:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:49:05.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:05.019 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-01-24T17:49:05.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:2024-01-24T17:49:05.051+0000 7fcd50e41480 -1 init, newargv = 0x559eba08af70 newargc=15 2024-01-24T17:49:05.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[72253]: starting ceph client 2024-01-24T17:49:05.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[72253]: starting fuse 2024-01-24T17:49:05.063 INFO:teuthology.orchestra.run.smithi039.stdout:39 2024-01-24T17:49:05.063 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:49:05.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:49:05.063 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-01-24T17:49:05.063 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-01-24T17:49:05.063 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-01-24T17:49:05.063 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-01-24T17:49:05.063 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:49:05.063 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:49:05.064 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-01-24T17:49:05.065 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-01-24T17:49:05.065 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-01-24T17:49:05.065 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:49:05.065 DEBUG:teuthology.orchestra.run.smithi039:> 2024-01-24T17:49:05.065 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.0")) 2024-01-24T17:49:05.065 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-01-24T17:49:05.256 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.0.72253.asok 2024-01-24T17:49:05.437 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72253.asok 2024-01-24T17:49:05.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:05.438 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72253.asok status 2024-01-24T17:49:05.689 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:49:05.689 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-01-24T17:49:05.689 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:49:05.689 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:49:05.689 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:49:05.689 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:49:05.689 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "72253", 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 8169, 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 8169 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:49:05.690 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1791706081 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1791706081 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.8169 192.168.0.1:0/1791706081", 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/1791706081", 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 68, 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 104, 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-01-24T17:49:05.691 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:49:05.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:05.706 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-01-24T17:49:05.706 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-24T17:49:05.760 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-01-24T17:49:05.760 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-24T17:49:05.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:05.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-24T17:49:05.858 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-24T17:49:06.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.086+0000 7f2324e00640 1 Processor -- start 2024-01-24T17:49:06.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.086+0000 7f2324e00640 1 -- start start 2024-01-24T17:49:06.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.088+0000 7f2324e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23200c9450 0x7f232015e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:06.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.088+0000 7f2324e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2320071ce0 0x7f232006fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:06.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.088+0000 7f2324e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232015f330 con 0x7f2320070410 2024-01-24T17:49:06.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.088+0000 7f2324e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2320072140 con 0x7f23200c9450 2024-01-24T17:49:06.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.088+0000 7f2324e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232015ea90 con 0x7f2320071ce0 2024-01-24T17:49:06.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.088+0000 7f231f400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f23200c9450 0x7f232015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.088+0000 7f231fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2320070410 0x7f232015b460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56860/0 (socket says 172.21.15.39:56860) 2024-01-24T17:49:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.088+0000 7f231fe00640 1 -- 172.21.15.39:0/903496812 learned_addr learned my addr 172.21.15.39:0/903496812 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.089+0000 7f231ea00640 1 --2- 172.21.15.39:0/903496812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2320071ce0 0x7f232006fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.089+0000 7f231f400640 1 -- 172.21.15.39:0/903496812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2320071ce0 msgr2=0x7f232006fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:06.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.089+0000 7f231f400640 1 --2- 172.21.15.39:0/903496812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2320071ce0 0x7f232006fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.089+0000 7f231f400640 1 -- 172.21.15.39:0/903496812 >> v1:172.21.15.39:6789/0 conn(0x7f2320070410 legacy=0x7f232015b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:06.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.089+0000 7f231f400640 1 -- 172.21.15.39:0/903496812 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232015ed70 con 0x7f23200c9450 2024-01-24T17:49:06.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.089+0000 7f231e000640 1 -- 172.21.15.39:0/903496812 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834116803 0 0) 0x7f232015f330 con 0x7f2320070410 2024-01-24T17:49:06.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.091+0000 7f231f400640 1 --2- 172.21.15.39:0/903496812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23200c9450 0x7f232015e420 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f23100097a0 tx=0x7f231002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0a50fe4c7063c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:06.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.091+0000 7f231e000640 1 -- 172.21.15.39:0/903496812 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231002e070 con 0x7f23200c9450 2024-01-24T17:49:06.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.091+0000 7f231e000640 1 -- 172.21.15.39:0/903496812 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23100315a0 con 0x7f23200c9450 2024-01-24T17:49:06.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.091+0000 7f231e000640 1 -- 172.21.15.39:0/903496812 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310031930 con 0x7f23200c9450 2024-01-24T17:49:06.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.091+0000 7f2324e00640 1 -- 172.21.15.39:0/903496812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23200c9450 msgr2=0x7f232015e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.091+0000 7f2324e00640 1 --2- 172.21.15.39:0/903496812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23200c9450 0x7f232015e420 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f23100097a0 tx=0x7f231002b840 comp rx=0 tx=0).stop 2024-01-24T17:49:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.092+0000 7f2324e00640 1 -- 172.21.15.39:0/903496812 shutdown_connections 2024-01-24T17:49:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.092+0000 7f2324e00640 1 --2- 172.21.15.39:0/903496812 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2320071ce0 0x7f232006fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.092+0000 7f2324e00640 1 --2- 172.21.15.39:0/903496812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23200c9450 0x7f232015e420 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.092+0000 7f2324e00640 1 -- 172.21.15.39:0/903496812 >> 172.21.15.39:0/903496812 conn(0x7f232006bf30 msgr2=0x7f232006c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.092+0000 7f2324e00640 1 -- 172.21.15.39:0/903496812 shutdown_connections 2024-01-24T17:49:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.092+0000 7f2324e00640 1 -- 172.21.15.39:0/903496812 wait complete. 2024-01-24T17:49:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.093+0000 7f2324e00640 1 Processor -- start 2024-01-24T17:49:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.094+0000 7f2324e00640 1 -- start start 2024-01-24T17:49:06.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.094+0000 7f2324e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2320070410 0x7f23201559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:06.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.094+0000 7f2324e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2320071ce0 0x7f23200ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:06.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f2324e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23200c9450 0x7f2320159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:06.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f2324e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23200c55a0 con 0x7f2320070410 2024-01-24T17:49:06.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f2324e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23200ac2b0 con 0x7f2320071ce0 2024-01-24T17:49:06.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f2324e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23200ac580 con 0x7f23200c9450 2024-01-24T17:49:06.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2320070410 0x7f23201559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:06.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2320070410 0x7f23201559f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41866/0 (socket says 172.21.15.39:41866) 2024-01-24T17:49:06.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231f400640 1 -- 172.21.15.39:0/1955447447 learned_addr learned my addr 172.21.15.39:0/1955447447 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:06.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231fe00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23200c9450 0x7f2320159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:06.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231f400640 1 -- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23200c9450 msgr2=0x7f2320159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231ea00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2320071ce0 0x7f23200ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231f400640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23200c9450 0x7f2320159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231f400640 1 -- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2320071ce0 msgr2=0x7f23200ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231f400640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2320071ce0 0x7f23200ad470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.095+0000 7f231f400640 1 -- 172.21.15.39:0/1955447447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310009400 con 0x7f2320070410 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.096+0000 7f231f400640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2320070410 0x7f23201559f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f2310034040 tx=0x7f23100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.096+0000 7f2317400640 1 -- 172.21.15.39:0/1955447447 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310032840 con 0x7f2320070410 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.096+0000 7f2317400640 1 -- 172.21.15.39:0/1955447447 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23100329a0 con 0x7f2320070410 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.096+0000 7f2317400640 1 -- 172.21.15.39:0/1955447447 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310032d50 con 0x7f2320070410 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.096+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23200c5810 con 0x7f2320070410 2024-01-24T17:49:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.096+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23200ac860 con 0x7f2320070410 2024-01-24T17:49:06.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.097+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23200c5d30 con 0x7f2320070410 2024-01-24T17:49:06.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.098+0000 7f2317400640 1 -- 172.21.15.39:0/1955447447 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f231004e430 con 0x7f2320070410 2024-01-24T17:49:06.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.099+0000 7f2317400640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22f806a480 0x7f22f806c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:06.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.099+0000 7f2317400640 1 -- 172.21.15.39:0/1955447447 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f23100aaa60 con 0x7f2320070410 2024-01-24T17:49:06.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.100+0000 7f231ea00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22f806a480 0x7f22f806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:06.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.101+0000 7f231ea00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22f806a480 0x7f22f806c940 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2320158050 tx=0x7f2308008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:06.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.104+0000 7f2317400640 1 -- 172.21.15.39:0/1955447447 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f23200c5d30 con 0x7f2320070410 2024-01-24T17:49:06.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.337+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f23200c5d30 con 0x7f2320070410 2024-01-24T17:49:06.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.338+0000 7f2317400640 1 -- 172.21.15.39:0/1955447447 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v68) v1 ==== 53+0+73 (secure 0 0 0) 0x7f23200c5d30 con 0x7f2320070410 2024-01-24T17:49:06.339 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-24T17:49:06.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22f806a480 msgr2=0x7f22f806c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:06.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22f806a480 0x7f22f806c940 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2320158050 tx=0x7f2308008040 comp rx=0 tx=0).stop 2024-01-24T17:49:06.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2320070410 msgr2=0x7f23201559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:06.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2320070410 0x7f23201559f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f2310034040 tx=0x7f23100046c0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 shutdown_connections 2024-01-24T17:49:06.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f23200c9450 0x7f2320159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2320071ce0 0x7f23200ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f22f806a480 0x7f22f806c940 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 --2- 172.21.15.39:0/1955447447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2320070410 0x7f23201559f0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:06.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 >> 172.21.15.39:0/1955447447 conn(0x7f232006bf30 msgr2=0x7f232015d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:06.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 shutdown_connections 2024-01-24T17:49:06.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:06.341+0000 7f2324e00640 1 -- 172.21.15.39:0/1955447447 wait complete. 2024-01-24T17:49:06.362 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-24T17:49:06.362 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T17:49:06.362 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-01-24T17:49:06.362 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-24T17:49:06.362 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-24T17:49:06.362 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-24T17:49:06.362 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T17:49:06.363 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T17:49:06.363 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:49:06.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:06.363 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-01-24T17:49:06.422 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T17:49:06.422 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T17:49:06.423 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42183sec preferred_lft 42183sec 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::b854:a4ff:fe8c:b45f/64 scope link 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether f6:04:0c:71:09:be brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::f404:cff:fe71:9be/64 scope link 2024-01-24T17:49:06.424 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:06.425 INFO:teuthology.orchestra.run.smithi100.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-24T17:49:06.425 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether fe:e0:b3:fd:70:d3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:49:06.425 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::fce0:b3ff:fefd:70d3/64 scope link 2024-01-24T17:49:06.425 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-01-24T17:49:06.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:06.426 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T17:49:06.511 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-01-24T17:49:06.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:06.516 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-01-24T17:49:06.569 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-01-24T17:49:06.569 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T17:49:06.569 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-24T17:49:06.569 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-24T17:49:06.569 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T17:49:06.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:06.570 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-01-24T17:49:06.570 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-01-24T17:49:06.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-24T17:49:06.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-24T17:49:06.570 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-24T17:49:06.570 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-01-24T17:49:06.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:06.749 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:49:06.798 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:49:06.798 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:49:06.799 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:49:06.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:06.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-01-24T17:49:06.892 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-24T17:49:06.892 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:49:06.892 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-24T17:49:06.947 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-24T17:49:06.947 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T17:49:06.947 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:49:06.994 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:49:07.134 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:49:07.134 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:49:07.226 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:49:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:49:07.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:07.231 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:49:07.285 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T17:49:07.285 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-24T17:49:07.292 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:49:07.371 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:49:07.371 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:49:07.441 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:49:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:49:07.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:07.444 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:49:07.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:2024-01-24T17:49:07.479+0000 7f8619260480 -1 init, newargv = 0x563ae232c090 newargc=15 2024-01-24T17:49:07.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[49286]: starting ceph client 2024-01-24T17:49:07.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[49286]: starting fuse 2024-01-24T17:49:08.584 DEBUG:teuthology.orchestra.run.smithi100:> sudo modprobe fuse 2024-01-24T17:49:08.682 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T17:49:08.683 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T17:49:08.774 INFO:teuthology.orchestra.run.smithi100.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T17:49:08.778 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T17:49:08.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:08.779 DEBUG:teuthology.orchestra.run.smithi100:> ls /sys/fs/fuse/connections 2024-01-24T17:49:08.830 INFO:teuthology.orchestra.run.smithi100.stdout:39 2024-01-24T17:49:08.830 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> import glob 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> import re 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> import os 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> import subprocess 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> def _find_admin_socket(client_name): 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> files = glob.glob(asok_path) 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:49:08.831 DEBUG:teuthology.orchestra.run.smithi100:> # Given a non-glob path, it better be there 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> if "*" not in asok_path: 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> assert(len(files) == 1) 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> return files[0] 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> for f in files: 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> contents = proc_f.read() 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> if mountpoint in contents: 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> return f 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T17:49:08.832 DEBUG:teuthology.orchestra.run.smithi100:> 2024-01-24T17:49:08.833 DEBUG:teuthology.orchestra.run.smithi100:> print(_find_admin_socket("client.1")) 2024-01-24T17:49:08.833 DEBUG:teuthology.orchestra.run.smithi100:> ' 2024-01-24T17:49:09.005 INFO:teuthology.orchestra.run.smithi100.stdout:/var/run/ceph/ceph-client.1.49286.asok 2024-01-24T17:49:09.185 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.49286.asok 2024-01-24T17:49:09.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:09.186 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.49286.asok status 2024-01-24T17:49:09.422 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-01-24T17:49:09.422 INFO:teuthology.orchestra.run.smithi100.stdout: "metadata": { 2024-01-24T17:49:09.422 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:49:09.422 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:49:09.422 INFO:teuthology.orchestra.run.smithi100.stdout: "entity_id": "1", 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "hostname": "smithi100", 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "pid": "49286", 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "root": "/" 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_count": 0, 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "dentry_pinned_count": 0, 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "id": 5873, 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "inst": { 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "name": { 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "client", 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "num": 5873 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:49:09.423 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1288180873 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": { 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "type": "v1", 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "nonce": 1288180873 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "inst_str": "client.5873 192.168.0.1:0/1288180873", 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "addr_str": "192.168.0.1:0/1288180873", 2024-01-24T17:49:09.424 INFO:teuthology.orchestra.run.smithi100.stdout: "inode_count": 1, 2024-01-24T17:49:09.425 INFO:teuthology.orchestra.run.smithi100.stdout: "mds_epoch": 68, 2024-01-24T17:49:09.425 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch": 104, 2024-01-24T17:49:09.425 INFO:teuthology.orchestra.run.smithi100.stdout: "osd_epoch_barrier": 0, 2024-01-24T17:49:09.425 INFO:teuthology.orchestra.run.smithi100.stdout: "blocklisted": false, 2024-01-24T17:49:09.425 INFO:teuthology.orchestra.run.smithi100.stdout: "fs_name": "cephfs" 2024-01-24T17:49:09.425 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-01-24T17:49:09.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:09.437 DEBUG:teuthology.orchestra.run.smithi100:> stat --file-system '--printf=%T 2024-01-24T17:49:09.437 DEBUG:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-24T17:49:09.486 INFO:teuthology.orchestra.run.smithi100.stdout:fuseblk 2024-01-24T17:49:09.486 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-24T17:49:09.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:09.486 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-24T17:49:09.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-01-24T17:49:09.824 INFO:teuthology.orchestra.run.smithi039.stdout:[{"id":5873,"entity":{"name":{"type":"client","num":5873},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":1288180873}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":2.3229529910000002,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.9946587068705206,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.5873 192.168.0.1:0/1288180873","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000001fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4e3182a65800aed6d4aa2f6b86da3cfd72188134","ceph_version":"ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)","entity_id":"1","hostname":"smithi100","mount_point":"/home/ubuntu/cephtest/mnt.1","pid":"49286","root":"/"}},{"id":8169,"entity":{"name":{"type":"client","num":8169},"addr":{"type":"any","addr":"192.168.0.1:0","nonce":1791706081}},"state":"open","num_leases":0,"num_caps":1,"request_load_avg":0,"uptime":4.7529038159999999,"requests_in_flight":0,"num_completed_requests":0,"num_completed_flushes":0,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":0.98908539304816312,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":1,"last_trim_completed_flushes_tid":0,"delegated_inos":[],"inst":"client.8169 192.168.0.1:0/1791706081","completed_requests":[],"prealloc_inos":[],"client_metadata":{"client_features":{"feature_bits":"0x00000000001fffff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000000ffff"}},"ceph_sha1":"4e3182a65800aed6d4aa2f6b86da3cfd72188134","ceph_version":"ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)","entity_id":"0","hostname":"smithi039","mount_point":"/home/ubuntu/cephtest/mnt.0","pid":"72253","root":"/"}}] 2024-01-24T17:49:09.825 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "client_features": { "feature_bits": "0x00000000001fffff" }, "entity_id": "1", "hostname": "smithi100", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "49286", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 1288180873, "type": "any" }, "name": { "num": 5873, "type": "client" } }, "id": 5873, "inst": "client.5873 192.168.0.1:0/1288180873", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.9946587068705206 }, "state": "open", "uptime": 2.322952991 }, { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "client_features": { "feature_bits": "0x00000000001fffff" }, "entity_id": "0", "hostname": "smithi039", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "72253", "root": "/" }, "completed_requests": [], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 1791706081, "type": "any" }, "name": { "num": 8169, "type": "client" } }, "id": 8169, "inst": "client.8169 192.168.0.1:0/1791706081", "last_trim_completed_flushes_tid": 0, "last_trim_completed_requests_tid": 1, "num_caps": 1, "num_completed_flushes": 0, "num_completed_requests": 0, "num_leases": 0, "prealloc_inos": [], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 0, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 0.9890853930481631 }, "state": "open", "uptime": 4.752903816 } ] 2024-01-24T17:49:09.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-01-24T17:49:10.096 INFO:teuthology.orchestra.run.smithi039.stdout:{"message":"","return_code":0} 2024-01-24T17:49:10.096 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-01-24T17:49:10.096 DEBUG:teuthology.orchestra.run.smithi039:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 0 show session 2024-01-24T17:49:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.202+0000 7f10aae0e100 1 -- start start 2024-01-24T17:49:10.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.202+0000 7f10aae0e100 1 Processor -- start 2024-01-24T17:49:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.203+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560211752c00 0x5602117d8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.203+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560211753800 0x5602117d8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.203+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560211753c00 0x5602117d9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.203+0000 7f10aae0e100 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560211824a00 con 0x560211753c00 2024-01-24T17:49:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.203+0000 7f10aae0e100 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x560211824780 con 0x560211752c00 2024-01-24T17:49:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.203+0000 7f10aae0e100 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x560211824640 con 0x560211753800 2024-01-24T17:49:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560211753c00 0x5602117d9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560211753c00 0x5602117d9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41882/0 (socket says 172.21.15.39:41882) 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 -- 172.21.15.39:0/1075814216 learned_addr learned my addr 172.21.15.39:0/1075814216 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8400640 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560211752c00 0x5602117d8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a7a00640 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560211753800 0x5602117d8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 -- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560211753800 msgr2=0x5602117d8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560211753800 0x5602117d8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 -- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560211752c00 msgr2=0x5602117d8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560211752c00 0x5602117d8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 -- 172.21.15.39:0/1075814216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560211778b60 con 0x560211753c00 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a7a00640 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560211753800 0x5602117d8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.204+0000 7f10a8e00640 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560211753c00 0x5602117d9080 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x56021251ec30 tx=0x560212529080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b8dcf728860646 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10a5200640 1 -- 172.21.15.39:0/1075814216 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56021179b600 con 0x560211753c00 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10a5200640 1 -- 172.21.15.39:0/1075814216 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5602117b0480 con 0x560211753c00 2024-01-24T17:49:10.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10a5200640 1 -- 172.21.15.39:0/1075814216 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56021179b1e0 con 0x560211753c00 2024-01-24T17:49:10.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10aae0e100 1 -- 172.21.15.39:0/1075814216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560211778820 con 0x560211753c00 2024-01-24T17:49:10.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10aae0e100 1 -- 172.21.15.39:0/1075814216 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5602117784e0 con 0x560211753c00 2024-01-24T17:49:10.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10aae0e100 4 waiting for MDS map... 2024-01-24T17:49:10.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10a5200640 1 -- 172.21.15.39:0/1075814216 <== mon.0 v2:172.21.15.39:3300/0 4 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x56021177e380 con 0x560211753c00 2024-01-24T17:49:10.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10a5200640 1 -- 172.21.15.39:0/1075814216 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 68) v1 ==== 3114+0+0 (secure 0 0 0) 0x5602117a4f00 con 0x560211753c00 2024-01-24T17:49:10.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10aae0e100 4 Got MDS map 68 2024-01-24T17:49:10.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.205+0000 7f10aae0e100 10 main: main 2024-01-24T17:49:10.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.206+0000 7f10aae0e100 4 main: connecting to RADOS... 2024-01-24T17:49:10.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.206+0000 7f10aae0e100 1 Processor -- start 2024-01-24T17:49:10.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.206+0000 7f10aae0e100 1 -- start start 2024-01-24T17:49:10.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.207+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611000 0x5602117d9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.207+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611400 0x5602117d9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.207+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611800 0x5602117da100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.207+0000 7f10aae0e100 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56021179de00 con 0x560212611800 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.207+0000 7f10aae0e100 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x560211824780 con 0x560212611000 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.207+0000 7f10aae0e100 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x560211824500 con 0x560212611400 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.207+0000 7f10a7a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611800 0x5602117da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611800 0x5602117da100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41890/0 (socket says 172.21.15.39:41890) 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 -- 172.21.15.39:0/1555213169 learned_addr learned my addr 172.21.15.39:0/1555213169 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a8400640 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611400 0x5602117d9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611000 0x5602117d9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 -- 172.21.15.39:0/1555213169 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611400 msgr2=0x5602117d9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611400 0x5602117d9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 -- 172.21.15.39:0/1555213169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611000 msgr2=0x5602117d9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:10.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611000 0x5602117d9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 -- 172.21.15.39:0/1555213169 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602117781a0 con 0x560212611800 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.208+0000 7f10a7a00640 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611800 0x5602117da100 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x56021251a120 tx=0x560212626d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f22fed3e88486df server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10a2a00640 1 -- 172.21.15.39:0/1555213169 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56021179b600 con 0x560212611800 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10a2a00640 1 -- 172.21.15.39:0/1555213169 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56021253a600 con 0x560212611800 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10a2a00640 1 -- 172.21.15.39:0/1555213169 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56021179b1e0 con 0x560212611800 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 -- 172.21.15.39:0/1555213169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611800 msgr2=0x5602117da100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611800 0x5602117da100 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x56021251a120 tx=0x560212626d20 comp rx=0 tx=0).stop 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 -- 172.21.15.39:0/1555213169 shutdown_connections 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611800 0x5602117da100 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611400 0x5602117d9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 --2- 172.21.15.39:0/1555213169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611000 0x5602117d9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 -- 172.21.15.39:0/1555213169 >> 172.21.15.39:0/1555213169 conn(0x560212610c00 msgr2=0x560211831000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 -- 172.21.15.39:0/1555213169 shutdown_connections 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.209+0000 7f10aae0e100 1 -- 172.21.15.39:0/1555213169 wait complete. 2024-01-24T17:49:10.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.210+0000 7f10aae0e100 1 Processor -- start 2024-01-24T17:49:10.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.210+0000 7f10aae0e100 1 -- start start 2024-01-24T17:49:10.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611000 0x5602117db180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611400 0x5602117dac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10aae0e100 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611800 0x5602117da680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10aae0e100 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602118243c0 con 0x560212611400 2024-01-24T17:49:10.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10aae0e100 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x560211824280 con 0x560212611800 2024-01-24T17:49:10.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10aae0e100 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x560211824140 con 0x560212611000 2024-01-24T17:49:10.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10a8400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611400 0x5602117dac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10a8400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611400 0x5602117dac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41904/0 (socket says 172.21.15.39:41904) 2024-01-24T17:49:10.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.211+0000 7f10a8400640 1 -- 172.21.15.39:0/2813469508 learned_addr learned my addr 172.21.15.39:0/2813469508 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:10.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a7a00640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611800 0x5602117da680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a7a00640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611000 0x5602117db180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a8400640 1 -- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611000 msgr2=0x5602117db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a8400640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560212611000 0x5602117db180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a8400640 1 -- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611800 msgr2=0x5602117da680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a8400640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560212611800 0x5602117da680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a8400640 1 -- 172.21.15.39:0/2813469508 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560211778b60 con 0x560212611400 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a8400640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611400 0x5602117dac00 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x56021251ad20 tx=0x560212528e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a0c00640 1 -- 172.21.15.39:0/2813469508 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56021179b080 con 0x560212611400 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a0c00640 1 -- 172.21.15.39:0/2813469508 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56021253b500 con 0x560212611400 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.212+0000 7f10a0c00640 1 -- 172.21.15.39:0/2813469508 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56021179b600 con 0x560212611400 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.213+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560211778820 con 0x560212611400 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.213+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560211779040 con 0x560212611400 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.213+0000 7f10aae0e100 4 main: resolving pool 14 2024-01-24T17:49:10.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.214+0000 7f10a0c00640 1 -- 172.21.15.39:0/2813469508 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x560211821080 con 0x560212611400 2024-01-24T17:49:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.215+0000 7f10a0c00640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x560212686800 0x5602117db700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.215+0000 7f10a0c00640 1 -- 172.21.15.39:0/2813469508 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x56021177e540 con 0x560212611400 2024-01-24T17:49:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.215+0000 7f10aae0e100 4 main: creating IoCtx.. 2024-01-24T17:49:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.215+0000 7f10aae0e100 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x560212686c00 0x5602117da100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.215+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- osd_op(unknown.0.0:1 14.b 14:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e104) v8 -- 0x560212687000 con 0x560212686c00 2024-01-24T17:49:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.216+0000 7f10a7a00640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x560212686800 0x5602117db700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.216+0000 7f10a7a00640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x560212686c00 0x5602117da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.217+0000 7f10a7a00640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x560212686c00 0x5602117da100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.217+0000 7f10a7a00640 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x560212686800 0x5602117db700 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x56021251a270 tx=0x5602126270e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.218+0000 7f10a7a00640 1 -- 172.21.15.39:0/2813469508 <== osd.6 v2:172.21.15.100:6808/36833 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-header out=14b] v0'0 uv5 ondisk = 0) v8 ==== 159+0+14 (crc 0 0 0) 0x5602124bab40 con 0x560212686c00 2024-01-24T17:49:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.218+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- osd_op(unknown.0.0:2 14.b 14:d0630e4c:::mds0_sessionmap:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e104) v8 -- 0x560212687400 con 0x560212686c00 2024-01-24T17:49:10.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.219+0000 7f10a7a00640 1 -- 172.21.15.39:0/2813469508 <== osd.6 v2:172.21.15.100:6808/36833 2 ==== osd_op_reply(2 mds0_sessionmap [omap-get-vals out=843b] v0'0 uv5 ondisk = 0) v8 ==== 159+0+843 (crc 0 0 0) 0x5602124bab40 con 0x560212686c00 2024-01-24T17:49:10.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.219+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- osd_op(unknown.0.0:3 14.b 14:d0630e4c:::mds0_sessionmap:head [omap-get-vals in=27b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e104) v8 -- 0x560212687800 con 0x560212686c00 2024-01-24T17:49:10.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10a7a00640 1 -- 172.21.15.39:0/2813469508 <== osd.6 v2:172.21.15.100:6808/36833 3 ==== osd_op_reply(3 mds0_sessionmap [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) v8 ==== 159+0+5 (crc 0 0 0) 0x5602124bab40 con 0x560212686c00 2024-01-24T17:49:10.220 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "0": { 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "data": { 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions": [ 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 8169, 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": { 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 8169 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:49:10.221 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "any", 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1791706081 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "open", 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: "num_leases": 0, 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: "num_caps": 0, 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: "request_load_avg": 0, 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 0.00099997900000000002, 2024-01-24T17:49:10.222 INFO:teuthology.orchestra.run.smithi039.stdout: "requests_in_flight": 0, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "num_completed_requests": 0, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "num_completed_flushes": 0, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "reconnecting": false, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "recall_caps": { 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 60 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "release_caps": { 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 60 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "recall_caps_throttle": { 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 1.5 2024-01-24T17:49:10.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "recall_caps_throttle2o": { 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 0.5 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "session_cache_liveness": { 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 300 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_acquisition": { 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 30 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "last_trim_completed_requests_tid": 0, 2024-01-24T17:49:10.224 INFO:teuthology.orchestra.run.smithi039.stdout: "last_trim_completed_flushes_tid": 0, 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "delegated_inos": [], 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": "client.8169 192.168.0.1:0/1791706081", 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "completed_requests": [], 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "prealloc_inos": [], 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "client_metadata": { 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "client_features": { 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_bits": "0x00000000001fffff" 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "metric_spec": { 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "metric_flags": { 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_bits": "0x000000000000ffff" 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:49:10.225 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "0", 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "72253", 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 5873, 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "entity": { 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 5873 2024-01-24T17:49:10.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "any", 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1288180873 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "open", 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "num_leases": 0, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "num_caps": 0, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "request_load_avg": 0, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 0.00099997900000000002, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "requests_in_flight": 0, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "num_completed_requests": 0, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "num_completed_flushes": 0, 2024-01-24T17:49:10.227 INFO:teuthology.orchestra.run.smithi039.stdout: "reconnecting": false, 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "recall_caps": { 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 60 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "release_caps": { 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 60 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "recall_caps_throttle": { 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 1.5 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "recall_caps_throttle2o": { 2024-01-24T17:49:10.228 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 0.5 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: "session_cache_liveness": { 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 300 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_acquisition": { 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: "value": 0, 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: "halflife": 30 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.229 INFO:teuthology.orchestra.run.smithi039.stdout: "last_trim_completed_requests_tid": 0, 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "last_trim_completed_flushes_tid": 0, 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "delegated_inos": [], 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": "client.5873 192.168.0.1:0/1288180873", 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "completed_requests": [], 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "prealloc_inos": [], 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "client_metadata": { 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "client_features": { 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_bits": "0x00000000001fffff" 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "metric_spec": { 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "metric_flags": { 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: "feature_bits": "0x000000000000ffff" 2024-01-24T17:49:10.230 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "1", 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi100", 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "49286", 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: "result": 0 2024-01-24T17:49:10.231 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:10.232 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:49:10.232 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x560212686c00 msgr2=0x5602117da100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x560212686c00 0x5602117da100 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x560212686800 msgr2=0x5602117db700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x560212686800 0x5602117db700 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x56021251a270 tx=0x5602126270e0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611400 msgr2=0x5602117dac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 --2- 172.21.15.39:0/2813469508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560212611400 0x5602117dac00 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x56021251ad20 tx=0x560212528e40 comp rx=0 tx=0).stop 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10a8400640 1 -- 172.21.15.39:0/2813469508 reap_dead start 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 shutdown_connections 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 >> 172.21.15.39:0/2813469508 conn(0x560212610c00 msgr2=0x560211831000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 shutdown_connections 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.220+0000 7f10aae0e100 1 -- 172.21.15.39:0/2813469508 wait complete. 2024-01-24T17:49:10.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 -- 172.21.15.39:0/1075814216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560211753c00 msgr2=0x5602117d9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560211753c00 0x5602117d9080 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x56021251ec30 tx=0x560212529080 comp rx=0 tx=0).stop 2024-01-24T17:49:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 -- 172.21.15.39:0/1075814216 shutdown_connections 2024-01-24T17:49:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560211753c00 0x5602117d9080 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x560211753800 0x5602117d8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 --2- 172.21.15.39:0/1075814216 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x560211752c00 0x5602117d8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 -- 172.21.15.39:0/1075814216 >> 172.21.15.39:0/1075814216 conn(0x560211753000 msgr2=0x560211830800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 -- 172.21.15.39:0/1075814216 shutdown_connections 2024-01-24T17:49:10.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.221+0000 7f10aae0e100 1 -- 172.21.15.39:0/1075814216 wait complete. 2024-01-24T17:49:10.235 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '0', 'show', 'session'] in time 0:00:00.138941, result: { "0": { "data": { "sessions": [ { "id": 8169, "entity": { "name": { "type": "client", "num": 8169 }, "addr": { "type": "any", "addr": "192.168.0.1:0", "nonce": 1791706081 } }, "state": "open", "num_leases": 0, "num_caps": 0, "request_load_avg": 0, "uptime": 0.00099997900000000002, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 0, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 0, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 30 }, "last_trim_completed_requests_tid": 0, "last_trim_completed_flushes_tid": 0, "delegated_inos": [], "inst": "client.8169 192.168.0.1:0/1791706081", "completed_requests": [], "prealloc_inos": [], "client_metadata": { "client_features": { "feature_bits": "0x00000000001fffff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "entity_id": "0", "hostname": "smithi039", "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "72253", "root": "/" } }, { "id": 5873, "entity": { "name": { "type": "client", "num": 5873 }, "addr": { "type": "any", "addr": "192.168.0.1:0", "nonce": 1288180873 } }, "state": "open", "num_leases": 0, "num_caps": 0, "request_load_avg": 0, "uptime": 0.00099997900000000002, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 0, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 0, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 30 }, "last_trim_completed_requests_tid": 0, "last_trim_completed_flushes_tid": 0, "delegated_inos": [], "inst": "client.5873 192.168.0.1:0/1288180873", "completed_requests": [], "prealloc_inos": [], "client_metadata": { "client_features": { "feature_bits": "0x00000000001fffff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "entity_id": "1", "hostname": "smithi100", "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "49286", "root": "/" } } ] }, "result": 0 } } 2024-01-24T17:49:10.236 INFO:tasks.cephfs.test_sessionmap:SessionMap: { "0": { "data": { "sessions": [ { "id": 8169, "entity": { "name": { "type": "client", "num": 8169 }, "addr": { "type": "any", "addr": "192.168.0.1:0", "nonce": 1791706081 } }, "state": "open", "num_leases": 0, "num_caps": 0, "request_load_avg": 0, "uptime": 0.000999979, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 0, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 0, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 30 }, "last_trim_completed_requests_tid": 0, "last_trim_completed_flushes_tid": 0, "delegated_inos": [], "inst": "client.8169 192.168.0.1:0/1791706081", "completed_requests": [], "prealloc_inos": [], "client_metadata": { "client_features": { "feature_bits": "0x00000000001fffff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "entity_id": "0", "hostname": "smithi039", "mount_point": "/home/ubuntu/cephtest/mnt.0", "pid": "72253", "root": "/" } }, { "id": 5873, "entity": { "name": { "type": "client", "num": 5873 }, "addr": { "type": "any", "addr": "192.168.0.1:0", "nonce": 1288180873 } }, "state": "open", "num_leases": 0, "num_caps": 0, "request_load_avg": 0, "uptime": 0.000999979, "requests_in_flight": 0, "num_completed_requests": 0, "num_completed_flushes": 0, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 1.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 0, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 30 }, "last_trim_completed_requests_tid": 0, "last_trim_completed_flushes_tid": 0, "delegated_inos": [], "inst": "client.5873 192.168.0.1:0/1288180873", "completed_requests": [], "prealloc_inos": [], "client_metadata": { "client_features": { "feature_bits": "0x00000000001fffff" }, "metric_spec": { "metric_flags": { "feature_bits": "0x000000000000ffff" } }, "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", "entity_id": "1", "hostname": "smithi100", "mount_point": "/home/ubuntu/cephtest/mnt.1", "pid": "49286", "root": "/" } } ] }, "result": 0 } } 2024-01-24T17:49:10.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:10.236 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir bravo) 2024-01-24T17:49:10.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:10.297 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch bravo/file_a) 2024-01-24T17:49:10.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:10.355 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest/mnt.1 && exec touch bravo/file_b) 2024-01-24T17:49:10.441 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-01-24T17:49:10.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.670+0000 7f2950a00640 1 Processor -- start 2024-01-24T17:49:10.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.671+0000 7f2950a00640 1 -- start start 2024-01-24T17:49:10.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.672+0000 7f2950a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294806c210 0x7f294806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.672+0000 7f2950a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29480c9630 con 0x7f29480cd400 2024-01-24T17:49:10.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.672+0000 7f2950a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294806cb30 con 0x7f29480b25e0 2024-01-24T17:49:10.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.672+0000 7f2950a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29480b4410 con 0x7f294806c210 2024-01-24T17:49:10.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.673+0000 7f294d600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f29480cd400 0x7f29480cd860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56864/0 (socket says 172.21.15.39:56864) 2024-01-24T17:49:10.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.673+0000 7f294d600640 1 -- 172.21.15.39:0/3628875208 learned_addr learned my addr 172.21.15.39:0/3628875208 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:10.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.673+0000 7f294e000640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294806c210 0x7f294806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.673+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762636290 0 0) 0x7f29480c9630 con 0x7f29480cd400 2024-01-24T17:49:10.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.673+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2928003520 con 0x7f29480cd400 2024-01-24T17:49:10.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4232683926 0 0) 0x7f2928003520 con 0x7f29480cd400 2024-01-24T17:49:10.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294806c210 msgr2=0x7f294806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294806c210 0x7f294806c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 >> v1:172.21.15.100:6789/0 conn(0x7f29480b25e0 legacy=0x7f29480b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29480c8d00 con 0x7f29480cd400 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f29300019f0 con 0x7f29480cd400 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 >> v1:172.21.15.39:6789/0 conn(0x7f29480cd400 legacy=0x7f29480cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294806c210 0x7f29280039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f29280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2928004d00 0x7f2928005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29480b4410 con 0x7f2928003f70 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292800bac0 con 0x7f2928004d00 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f292800bc80 con 0x7f294806c210 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.674+0000 7f294d600640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f29280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.675+0000 7f294ea00640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2928004d00 0x7f2928005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.675+0000 7f294d600640 1 -- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294806c210 msgr2=0x7f29280039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:10.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.675+0000 7f294d600640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294806c210 0x7f29280039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.675+0000 7f294d600640 1 -- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2928004d00 msgr2=0x7f2928005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:10.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.675+0000 7f294d600640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2928004d00 0x7f2928005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.675+0000 7f294d600640 1 -- 172.21.15.39:0/3628875208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2928003520 con 0x7f2928003f70 2024-01-24T17:49:10.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.675+0000 7f294ea00640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2928004d00 0x7f2928005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.676+0000 7f294d600640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f29280047c0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f293000a760 tx=0x7f29280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76ed361dc375bbd server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.676+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930034630 con 0x7f2928003f70 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.676+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293002f070 con 0x7f2928003f70 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.676+0000 7f294cc00640 1 -- 172.21.15.39:0/3628875208 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930034a50 con 0x7f2928003f70 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.677+0000 7f2950a00640 1 -- 172.21.15.39:0/3628875208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 msgr2=0x7f29280047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.677+0000 7f2950a00640 1 --2- 172.21.15.39:0/3628875208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f29280047c0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f293000a760 tx=0x7f29280057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.677+0000 7f294ea00640 1 -- 172.21.15.39:0/3628875208 reap_dead start 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.677+0000 7f2950a00640 1 -- 172.21.15.39:0/3628875208 shutdown_connections 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.677+0000 7f2950a00640 1 -- 172.21.15.39:0/3628875208 >> 172.21.15.39:0/3628875208 conn(0x7f294806a5b0 msgr2=0x7f29480ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:10.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.677+0000 7f2950a00640 1 -- 172.21.15.39:0/3628875208 shutdown_connections 2024-01-24T17:49:10.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.677+0000 7f2950a00640 1 -- 172.21.15.39:0/3628875208 wait complete. 2024-01-24T17:49:10.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.679+0000 7f2950a00640 1 Processor -- start 2024-01-24T17:49:10.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.679+0000 7f2950a00640 1 -- start start 2024-01-24T17:49:10.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f2950a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f2948157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f2950a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294815bdf0 0x7f29480adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f2950a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2948157310 0x7f294815ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f2950a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29481581c0 con 0x7f2928003f70 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f2950a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948157790 con 0x7f2948157310 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f2950a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29480aca90 con 0x7f294815bdf0 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f294e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f2948157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f294e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f2948157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41918/0 (socket says 172.21.15.39:41918) 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f294e000640 1 -- 172.21.15.39:0/2051695500 learned_addr learned my addr 172.21.15.39:0/2051695500 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f294e000640 1 -- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294815bdf0 msgr2=0x7f29480adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f294e000640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294815bdf0 0x7f29480adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f294e000640 1 -- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2948157310 msgr2=0x7f294815ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f294e000640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2948157310 0x7f294815ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.680+0000 7f294e000640 1 -- 172.21.15.39:0/2051695500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29480acd70 con 0x7f2928003f70 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.681+0000 7f294e000640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f2948157920 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f293800e7f0 tx=0x7f293800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.681+0000 7f2946000640 1 -- 172.21.15.39:0/2051695500 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293800c8e0 con 0x7f2928003f70 2024-01-24T17:49:10.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.681+0000 7f2946000640 1 -- 172.21.15.39:0/2051695500 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293800ca40 con 0x7f2928003f70 2024-01-24T17:49:10.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.681+0000 7f2946000640 1 -- 172.21.15.39:0/2051695500 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293800cdf0 con 0x7f2928003f70 2024-01-24T17:49:10.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.681+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29480ad050 con 0x7f2928003f70 2024-01-24T17:49:10.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.681+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29480ad610 con 0x7f2928003f70 2024-01-24T17:49:10.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.682+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29480ad7d0 con 0x7f2928003f70 2024-01-24T17:49:10.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.683+0000 7f2946000640 1 -- 172.21.15.39:0/2051695500 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2938013470 con 0x7f2928003f70 2024-01-24T17:49:10.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.684+0000 7f2946000640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f292006a460 0x7f292006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:10.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.684+0000 7f2946000640 1 -- 172.21.15.39:0/2051695500 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2938088b20 con 0x7f2928003f70 2024-01-24T17:49:10.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.685+0000 7f294d600640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f292006a460 0x7f292006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:10.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.686+0000 7f294d600640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f292006a460 0x7f292006c920 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2930034000 tx=0x7f2930009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:10.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.689+0000 7f2946000640 1 -- 172.21.15.39:0/2051695500 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2938054000 con 0x7f2928003f70 2024-01-24T17:49:10.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:10.927+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f2948066440 con 0x7f2928003f70 2024-01-24T17:49:11.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.426+0000 7f2946000640 1 -- 172.21.15.39:0/2051695500 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v69) v1 ==== 105+0+0 (secure 0 0 0) 0x7f29380170b0 con 0x7f2928003f70 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f292006a460 msgr2=0x7f292006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f292006a460 0x7f292006c920 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2930034000 tx=0x7f2930009030 comp rx=0 tx=0).stop 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 msgr2=0x7f2948157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f2948157920 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f293800e7f0 tx=0x7f293800ecb0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 shutdown_connections 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2948157310 0x7f294815ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f294815bdf0 0x7f29480adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f292006a460 0x7f292006c920 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 --2- 172.21.15.39:0/2051695500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2928003f70 0x7f2948157920 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 >> 172.21.15.39:0/2051695500 conn(0x7f294806a5b0 msgr2=0x7f294806ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:11.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 shutdown_connections 2024-01-24T17:49:11.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.430+0000 7f2950a00640 1 -- 172.21.15.39:0/2051695500 wait complete. 2024-01-24T17:49:11.457 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:49:11.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.644+0000 7fa6bf200640 1 Processor -- start 2024-01-24T17:49:11.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.645+0000 7fa6bf200640 1 -- start start 2024-01-24T17:49:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.646+0000 7fa6bf200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6b80c9450 0x7fa6b815e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.646+0000 7fa6bf200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b815f330 con 0x7fa6b8071ce0 2024-01-24T17:49:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.646+0000 7fa6bf200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b8072140 con 0x7fa6b815b460 2024-01-24T17:49:11.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.646+0000 7fa6bf200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b815ea90 con 0x7fa6b80c9450 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.647+0000 7fa6bd400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa6b8071ce0 0x7fa6b806fed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56868/0 (socket says 172.21.15.39:56868) 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.647+0000 7fa6bd400640 1 -- 172.21.15.39:0/1584816743 learned_addr learned my addr 172.21.15.39:0/1584816743 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.647+0000 7fa6bde00640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6b80c9450 0x7fa6b815e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.647+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069223091 0 0) 0x7fa6b815f330 con 0x7fa6b8071ce0 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.647+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698003520 con 0x7fa6b8071ce0 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3371963754 0 0) 0x7fa698003520 con 0x7fa6b8071ce0 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6b80c9450 msgr2=0x7fa6b815e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6b80c9450 0x7fa6b815e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 >> v1:172.21.15.100:6789/0 conn(0x7fa6b815b460 legacy=0x7fa6b80cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b815ed70 con 0x7fa6b8071ce0 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fa6a0001670 con 0x7fa6b8071ce0 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 >> v1:172.21.15.39:6789/0 conn(0x7fa6b8071ce0 legacy=0x7fa6b806fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6b80c9450 0x7fa6980039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa698003ee0 0x7fa6980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa698004d00 0x7fa698005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b815ea90 con 0x7fa698003ee0 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69800ba90 con 0x7fa698004d00 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69800bc50 con 0x7fa6b80c9450 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.648+0000 7fa6bd400640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa698003ee0 0x7fa6980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.649+0000 7fa6bde00640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6b80c9450 0x7fa6980039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.649+0000 7fa6be800640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa698004d00 0x7fa698005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.649+0000 7fa6bd400640 1 -- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6b80c9450 msgr2=0x7fa6980039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.649+0000 7fa6bd400640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa6b80c9450 0x7fa6980039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.649+0000 7fa6bd400640 1 -- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa698004d00 msgr2=0x7fa698005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.649+0000 7fa6bd400640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa698004d00 0x7fa698005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.649+0000 7fa6bd400640 1 -- 172.21.15.39:0/1584816743 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698003520 con 0x7fa698003ee0 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.650+0000 7fa6bd400640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa698003ee0 0x7fa6980047c0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c210 tx=0x7fa6980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142f55668e62797c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.650+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0005ad0 con 0x7fa698003ee0 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.650+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a0005c50 con 0x7fa698003ee0 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.650+0000 7fa6bca00640 1 -- 172.21.15.39:0/1584816743 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a002c8d0 con 0x7fa698003ee0 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.651+0000 7fa6bf200640 1 -- 172.21.15.39:0/1584816743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa698003ee0 msgr2=0x7fa6980047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.651+0000 7fa6bf200640 1 --2- 172.21.15.39:0/1584816743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa698003ee0 0x7fa6980047c0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c210 tx=0x7fa6980057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.651+0000 7fa6be800640 1 -- 172.21.15.39:0/1584816743 reap_dead start 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.651+0000 7fa6bf200640 1 -- 172.21.15.39:0/1584816743 shutdown_connections 2024-01-24T17:49:11.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.651+0000 7fa6bf200640 1 -- 172.21.15.39:0/1584816743 >> 172.21.15.39:0/1584816743 conn(0x7fa6b806bf30 msgr2=0x7fa6b806c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:11.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.651+0000 7fa6bf200640 1 -- 172.21.15.39:0/1584816743 shutdown_connections 2024-01-24T17:49:11.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.651+0000 7fa6bf200640 1 -- 172.21.15.39:0/1584816743 wait complete. 2024-01-24T17:49:11.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.653+0000 7fa6bf200640 1 Processor -- start 2024-01-24T17:49:11.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.653+0000 7fa6bf200640 1 -- start start 2024-01-24T17:49:11.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bf200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa698003ee0 0x7fa6b806c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:11.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bf200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6b8148020 0x7fa6b806c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:11.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bf200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6b814a8e0 0x7fa6b814ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:11.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bf200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81f29f0 con 0x7fa6b8148020 2024-01-24T17:49:11.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bf200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81f2070 con 0x7fa6b814a8e0 2024-01-24T17:49:11.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bf200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b81f2df0 con 0x7fa698003ee0 2024-01-24T17:49:11.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6b8148020 0x7fa6b806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:11.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6b8148020 0x7fa6b806c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41932/0 (socket says 172.21.15.39:41932) 2024-01-24T17:49:11.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.654+0000 7fa6bd400640 1 -- 172.21.15.39:0/436771395 learned_addr learned my addr 172.21.15.39:0/436771395 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:11.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.655+0000 7fa6bde00640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa698003ee0 0x7fa6b806c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:11.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.655+0000 7fa6be800640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6b814a8e0 0x7fa6b814ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:11.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.655+0000 7fa6bd400640 1 -- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa698003ee0 msgr2=0x7fa6b806c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.655+0000 7fa6bd400640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa698003ee0 0x7fa6b806c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.655+0000 7fa6bd400640 1 -- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6b814a8e0 msgr2=0x7fa6b814ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.655+0000 7fa6bd400640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6b814a8e0 0x7fa6b814ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.655+0000 7fa6bd400640 1 -- 172.21.15.39:0/436771395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b81f2330 con 0x7fa6b8148020 2024-01-24T17:49:11.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.655+0000 7fa6bd400640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6b8148020 0x7fa6b806c850 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c210 tx=0x7fa6a0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:11.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.656+0000 7fa6b6000640 1 -- 172.21.15.39:0/436771395 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a003c040 con 0x7fa6b8148020 2024-01-24T17:49:11.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.656+0000 7fa6b6000640 1 -- 172.21.15.39:0/436771395 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a002cce0 con 0x7fa6b8148020 2024-01-24T17:49:11.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.656+0000 7fa6b6000640 1 -- 172.21.15.39:0/436771395 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a00072b0 con 0x7fa6b8148020 2024-01-24T17:49:11.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.656+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b81f2610 con 0x7fa6b8148020 2024-01-24T17:49:11.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.656+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b80c4300 con 0x7fa6b8148020 2024-01-24T17:49:11.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.657+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b804e840 con 0x7fa6b8148020 2024-01-24T17:49:11.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.658+0000 7fa6b6000640 1 -- 172.21.15.39:0/436771395 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa6a0032070 con 0x7fa6b8148020 2024-01-24T17:49:11.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.659+0000 7fa6b6000640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa69006a430 0x7fa69006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:11.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.659+0000 7fa6b6000640 1 -- 172.21.15.39:0/436771395 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa6a00a9960 con 0x7fa6b8148020 2024-01-24T17:49:11.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.660+0000 7fa6bde00640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa69006a430 0x7fa69006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:11.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.661+0000 7fa6bde00640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa69006a430 0x7fa69006c8f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80c5360 tx=0x7fa6a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:11.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.663+0000 7fa6b6000640 1 -- 172.21.15.39:0/436771395 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa6a0074f90 con 0x7fa6b8148020 2024-01-24T17:49:11.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.896+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6b80af9d0 con 0x7fa6b8148020 2024-01-24T17:49:11.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.897+0000 7fa6b6000640 1 -- 172.21.15.39:0/436771395 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4760 (secure 0 0 0) 0x7fa6a00785b0 con 0x7fa6b8148020 2024-01-24T17:49:11.898 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:11.898 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":70,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:49:11.426777+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8118,"mds_1":8100},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8100":{"gid":8100,"name":"a","rank":1,"incarnation":70,"state":"up:creating","state_seq":1,"addr":"172.21.15.39:6833/3903887009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3903887009},{"type":"v1","addr":"172.21.15.39:6833","nonce":3903887009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8118":{"gid":8118,"name":"c","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/1608906778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1608906778},{"type":"v1","addr":"172.21.15.39:6835","nonce":1608906778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:49:11.899 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 70 2024-01-24T17:49:11.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa69006a430 msgr2=0x7fa69006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa69006a430 0x7fa69006c8f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80c5360 tx=0x7fa6a8008040 comp rx=0 tx=0).stop 2024-01-24T17:49:11.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6b8148020 msgr2=0x7fa6b806c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6b8148020 0x7fa6b806c850 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c210 tx=0x7fa6a0001a00 comp rx=0 tx=0).stop 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 shutdown_connections 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa6b814a8e0 0x7fa6b814ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa69006a430 0x7fa69006c8f0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6b8148020 0x7fa6b806c850 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 --2- 172.21.15.39:0/436771395 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa698003ee0 0x7fa6b806c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 >> 172.21.15.39:0/436771395 conn(0x7fa6b806bf30 msgr2=0x7fa6b8072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 shutdown_connections 2024-01-24T17:49:11.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:11.900+0000 7fa6bf200640 1 -- 172.21.15.39:0/436771395 wait complete. 2024-01-24T17:49:12.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:48:26.880600+0000', 'modified': '2024-01-24T17:49:11.426777+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 104, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8118, 'mds_1': 8100}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8100': {'gid': 8100, 'name': 'a', 'rank': 1, 'incarnation': 70, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.39:6833/3903887009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 3903887009}, {'type': 'v1', 'addr': '172.21.15.39:6833', 'nonce': 3903887009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_8118': {'gid': 8118, 'name': 'c', 'rank': 0, 'incarnation': 65, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.39:6835/1608906778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6834', 'nonce': 1608906778}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 1608906778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:49:12.323 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8100:up:creating 2024-01-24T17:49:13.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:49:13.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.551+0000 7fbea1600640 1 Processor -- start 2024-01-24T17:49:13.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.551+0000 7fbea1600640 1 -- start start 2024-01-24T17:49:13.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.553+0000 7fbea1600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe9c071910 0x7fbe9c0c3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:13.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.553+0000 7fbea1600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c071cf0 con 0x7fbe9c0ccd50 2024-01-24T17:49:13.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.553+0000 7fbea1600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c0c4080 con 0x7fbe9c0c3b40 2024-01-24T17:49:13.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.553+0000 7fbea1600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c4360 con 0x7fbe9c071910 2024-01-24T17:49:13.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.553+0000 7fbea0c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fbe9c0ccd50 0x7fbe9c06fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:33620/0 (socket says 172.21.15.39:33620) 2024-01-24T17:49:13.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.553+0000 7fbea0c00640 1 -- 172.21.15.39:0/1411021184 learned_addr learned my addr 172.21.15.39:0/1411021184 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:13.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9be00640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe9c071910 0x7fbe9c0c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:13.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547452883 0 0) 0x7fbe9c071cf0 con 0x7fbe9c0ccd50 2024-01-24T17:49:13.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe7c003520 con 0x7fbe9c0ccd50 2024-01-24T17:49:13.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1454096793 0 0) 0x7fbe7c003520 con 0x7fbe9c0ccd50 2024-01-24T17:49:13.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe9c071910 msgr2=0x7fbe9c0c3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:13.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe9c071910 0x7fbe9c0c3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 >> v1:172.21.15.100:6789/0 conn(0x7fbe9c0c3b40 legacy=0x7fbe9c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:13.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0c4640 con 0x7fbe9c0ccd50 2024-01-24T17:49:13.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fbe940019f0 con 0x7fbe9c0ccd50 2024-01-24T17:49:13.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 >> v1:172.21.15.39:6789/0 conn(0x7fbe9c0ccd50 legacy=0x7fbe9c06fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:13.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe7c0039a0 0x7fbe7c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:13.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe7c0047c0 0x7fbe7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:13.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe9c071910 0x7fbe7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:13.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c4360 con 0x7fbe7c0047c0 2024-01-24T17:49:13.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c00ba30 con 0x7fbe7c0039a0 2024-01-24T17:49:13.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.554+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c00bbf0 con 0x7fbe9c071910 2024-01-24T17:49:13.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbe9b400640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe7c0047c0 0x7fbe7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:13.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbe9be00640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe7c0039a0 0x7fbe7c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:13.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbea0c00640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe9c071910 0x7fbe7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:13.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbe9b400640 1 -- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe9c071910 msgr2=0x7fbe7c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:13.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbe9b400640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe9c071910 0x7fbe7c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbe9b400640 1 -- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe7c0039a0 msgr2=0x7fbe7c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:13.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbe9b400640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe7c0039a0 0x7fbe7c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbe9b400640 1 -- 172.21.15.39:0/1411021184 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c003520 con 0x7fbe7c0047c0 2024-01-24T17:49:13.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.555+0000 7fbe9be00640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe7c0039a0 0x7fbe7c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:49:13.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.556+0000 7fbe9b400640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe7c0047c0 0x7fbe7c004c40 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fbe84001720 tx=0x7fbe7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809df2fe010286e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:13.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe840052d0 con 0x7fbe7c0047c0 2024-01-24T17:49:13.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8402e050 con 0x7fbe7c0047c0 2024-01-24T17:49:13.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbe9aa00640 1 -- 172.21.15.39:0/1411021184 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe840035f0 con 0x7fbe7c0047c0 2024-01-24T17:49:13.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbea1600640 1 -- 172.21.15.39:0/1411021184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe7c0047c0 msgr2=0x7fbe7c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:13.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbea1600640 1 --2- 172.21.15.39:0/1411021184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe7c0047c0 0x7fbe7c004c40 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fbe84001720 tx=0x7fbe7c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbea0c00640 1 -- 172.21.15.39:0/1411021184 reap_dead start 2024-01-24T17:49:13.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbea1600640 1 -- 172.21.15.39:0/1411021184 shutdown_connections 2024-01-24T17:49:13.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbea1600640 1 -- 172.21.15.39:0/1411021184 >> 172.21.15.39:0/1411021184 conn(0x7fbe9c06bf30 msgr2=0x7fbe9c06c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:13.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.557+0000 7fbea1600640 1 -- 172.21.15.39:0/1411021184 shutdown_connections 2024-01-24T17:49:13.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.558+0000 7fbea1600640 1 -- 172.21.15.39:0/1411021184 wait complete. 2024-01-24T17:49:13.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.559+0000 7fbea1600640 1 Processor -- start 2024-01-24T17:49:13.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.559+0000 7fbea1600640 1 -- start start 2024-01-24T17:49:13.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.560+0000 7fbea1600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe7c0047c0 0x7fbe9c1610b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:13.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.560+0000 7fbea1600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe9c162f40 0x7fbe9c158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:13.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.560+0000 7fbea1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe9c0afae0 0x7fbe9c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:13.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.560+0000 7fbea1600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c1595a0 con 0x7fbe9c0afae0 2024-01-24T17:49:13.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.560+0000 7fbea1600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0b04a0 con 0x7fbe9c162f40 2024-01-24T17:49:13.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.560+0000 7fbea1600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c161690 con 0x7fbe7c0047c0 2024-01-24T17:49:13.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbea0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe9c0afae0 0x7fbe9c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:13.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbea0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe9c0afae0 0x7fbe9c0aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35636/0 (socket says 172.21.15.39:35636) 2024-01-24T17:49:13.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbea0c00640 1 -- 172.21.15.39:0/3874670477 learned_addr learned my addr 172.21.15.39:0/3874670477 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:13.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbe9be00640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe7c0047c0 0x7fbe9c1610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:13.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbe9b400640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe9c162f40 0x7fbe9c158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:13.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbea0c00640 1 -- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe7c0047c0 msgr2=0x7fbe9c1610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:13.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbea0c00640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe7c0047c0 0x7fbe9c1610b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbea0c00640 1 -- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe9c162f40 msgr2=0x7fbe9c158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:13.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbea0c00640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe9c162f40 0x7fbe9c158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.561+0000 7fbea0c00640 1 -- 172.21.15.39:0/3874670477 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c161970 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.562+0000 7fbea0c00640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe9c0afae0 0x7fbe9c0aff60 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0cc3e0 tx=0x7fbe9400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.562+0000 7fbe93e00640 1 -- 172.21.15.39:0/3874670477 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9400eb90 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.562+0000 7fbe93e00640 1 -- 172.21.15.39:0/3874670477 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9400ecf0 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.562+0000 7fbe93e00640 1 -- 172.21.15.39:0/3874670477 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94013660 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.562+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c161c50 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.562+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c1633c0 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.563+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c1596e0 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.564+0000 7fbe93e00640 1 -- 172.21.15.39:0/3874670477 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbe940137c0 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.565+0000 7fbe93e00640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbe7406a430 0x7fbe7406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.565+0000 7fbe93e00640 1 -- 172.21.15.39:0/3874670477 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fbe94088e90 con 0x7fbe9c0afae0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.566+0000 7fbe9be00640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbe7406a430 0x7fbe7406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.567+0000 7fbe9be00640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbe7406a430 0x7fbe7406c8f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009a90 tx=0x7fbe8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:13.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.570+0000 7fbe93e00640 1 -- 172.21.15.39:0/3874670477 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbe9c1596e0 con 0x7fbe9c0afae0 2024-01-24T17:49:13.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.804+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe9c163550 con 0x7fbe9c0afae0 2024-01-24T17:49:13.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.805+0000 7fbe93e00640 1 -- 172.21.15.39:0/3874670477 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4758 (secure 0 0 0) 0x7fbe9408a020 con 0x7fbe9c0afae0 2024-01-24T17:49:13.806 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:13.806 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":71,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:49:12.432841+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8118,"mds_1":8100},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8100":{"gid":8100,"name":"a","rank":1,"incarnation":70,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6833/3903887009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3903887009},{"type":"v1","addr":"172.21.15.39:6833","nonce":3903887009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8118":{"gid":8118,"name":"c","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/1608906778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1608906778},{"type":"v1","addr":"172.21.15.39:6835","nonce":1608906778}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:49:13.807 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 71 2024-01-24T17:49:13.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.807+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbe7406a430 msgr2=0x7fbe7406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:13.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.807+0000 7fbea1600640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbe7406a430 0x7fbe7406c8f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009a90 tx=0x7fbe8c008040 comp rx=0 tx=0).stop 2024-01-24T17:49:13.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe9c0afae0 msgr2=0x7fbe9c0aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:13.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe9c0afae0 0x7fbe9c0aff60 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0cc3e0 tx=0x7fbe9400e530 comp rx=0 tx=0).stop 2024-01-24T17:49:13.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 shutdown_connections 2024-01-24T17:49:13.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbe7406a430 0x7fbe7406c8f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe9c0afae0 0x7fbe9c0aff60 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbe9c162f40 0x7fbe9c158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 --2- 172.21.15.39:0/3874670477 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbe7c0047c0 0x7fbe9c1610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:13.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 >> 172.21.15.39:0/3874670477 conn(0x7fbe9c06bf30 msgr2=0x7fbe9c164b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:13.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 shutdown_connections 2024-01-24T17:49:13.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:13.808+0000 7fbea1600640 1 -- 172.21.15.39:0/3874670477 wait complete. 2024-01-24T17:49:14.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T17:48:26.880600+0000', 'modified': '2024-01-24T17:49:12.432841+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 104, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8118, 'mds_1': 8100}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8100': {'gid': 8100, 'name': 'a', 'rank': 1, 'incarnation': 70, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.39:6833/3903887009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6832', 'nonce': 3903887009}, {'type': 'v1', 'addr': '172.21.15.39:6833', 'nonce': 3903887009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_8118': {'gid': 8118, 'name': 'c', 'rank': 0, 'incarnation': 65, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.39:6835/1608906778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6834', 'nonce': 1608906778}, {'type': 'v1', 'addr': '172.21.15.39:6835', 'nonce': 1608906778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-24T17:49:14.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-01-24T17:49:14.232 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-24T17:49:14.359 INFO:tasks.ceph.mds.c.smithi039.stderr:2024-01-24T17:49:14.359+0000 7fd769c00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-01-24T17:49:14.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.467+0000 7f0441600640 1 Processor -- start 2024-01-24T17:49:14.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.467+0000 7f0441600640 1 -- start start 2024-01-24T17:49:14.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.468+0000 7f0441600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f043c0b7840 0x7f043c0d1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.468+0000 7f0441600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c0b60a0 con 0x7f043c0cf920 2024-01-24T17:49:14.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.468+0000 7f0441600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c0d1b10 con 0x7f043c15ce90 2024-01-24T17:49:14.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.468+0000 7f0441600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c0d1df0 con 0x7f043c0b7840 2024-01-24T17:49:14.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.469+0000 7f043a000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f043c0cf920 0x7f043c0d0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:33636/0 (socket says 172.21.15.39:33636) 2024-01-24T17:49:14.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.469+0000 7f043b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f043c0b7840 0x7f043c0d1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.469+0000 7f043a000640 1 -- 172.21.15.39:0/32157761 learned_addr learned my addr 172.21.15.39:0/32157761 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:14.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.469+0000 7f043b400640 1 -- 172.21.15.39:0/32157761 >> v1:172.21.15.100:6789/0 conn(0x7f043c15ce90 legacy=0x7f043c0cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.469+0000 7f043b400640 1 -- 172.21.15.39:0/32157761 >> v1:172.21.15.39:6789/0 conn(0x7f043c0cf920 legacy=0x7f043c0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.469+0000 7f043b400640 1 -- 172.21.15.39:0/32157761 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c0b6310 con 0x7f043c0b7840 2024-01-24T17:49:14.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.470+0000 7f043b400640 1 --2- 172.21.15.39:0/32157761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0b7840 0x7f043c0d1530 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0434009d90 tx=0x7f043402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7f8f6b4346461e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.471+0000 7f0439600640 1 -- 172.21.15.39:0/32157761 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043402cc60 con 0x7f043c0b7840 2024-01-24T17:49:14.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.471+0000 7f0439600640 1 -- 172.21.15.39:0/32157761 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043402cdc0 con 0x7f043c0b7840 2024-01-24T17:49:14.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.471+0000 7f0439600640 1 -- 172.21.15.39:0/32157761 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434034660 con 0x7f043c0b7840 2024-01-24T17:49:14.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.471+0000 7f0441600640 1 -- 172.21.15.39:0/32157761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0b7840 msgr2=0x7f043c0d1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.471+0000 7f0441600640 1 --2- 172.21.15.39:0/32157761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0b7840 0x7f043c0d1530 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0434009d90 tx=0x7f043402beb0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.472+0000 7f0441600640 1 -- 172.21.15.39:0/32157761 shutdown_connections 2024-01-24T17:49:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.472+0000 7f0441600640 1 --2- 172.21.15.39:0/32157761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0b7840 0x7f043c0d1530 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.472+0000 7f0441600640 1 -- 172.21.15.39:0/32157761 >> 172.21.15.39:0/32157761 conn(0x7f043c06af00 msgr2=0x7f043c06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.472+0000 7f0441600640 1 -- 172.21.15.39:0/32157761 shutdown_connections 2024-01-24T17:49:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.472+0000 7f0441600640 1 -- 172.21.15.39:0/32157761 wait complete. 2024-01-24T17:49:14.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.474+0000 7f0441600640 1 Processor -- start 2024-01-24T17:49:14.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.474+0000 7f0441600640 1 -- start start 2024-01-24T17:49:14.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.475+0000 7f0441600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c0b7840 0x7f043c1f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.475+0000 7f0441600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0cf920 0x7f043c1f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.475+0000 7f0441600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c15ce90 0x7f043c15a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.475+0000 7f0441600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c1f62d0 con 0x7f043c0cf920 2024-01-24T17:49:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.475+0000 7f0441600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c1f65b0 con 0x7f043c0b7840 2024-01-24T17:49:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.475+0000 7f0441600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c1f6890 con 0x7f043c15ce90 2024-01-24T17:49:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0cf920 0x7f043c1f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0cf920 0x7f043c1f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35656/0 (socket says 172.21.15.39:35656) 2024-01-24T17:49:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 -- 172.21.15.39:0/3877115625 learned_addr learned my addr 172.21.15.39:0/3877115625 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043aa00640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c0b7840 0x7f043c1f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043b400640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c15ce90 0x7f043c15a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 -- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c15ce90 msgr2=0x7f043c15a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c15ce90 0x7f043c15a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 -- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c0b7840 msgr2=0x7f043c1f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c0b7840 0x7f043c1f5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 -- 172.21.15.39:0/3877115625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434009a70 con 0x7f043c0cf920 2024-01-24T17:49:14.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.476+0000 7f043a000640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0cf920 0x7f043c1f2da0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f042400c4a0 tx=0x7f042400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.477+0000 7f0432a00640 1 -- 172.21.15.39:0/3877115625 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424005ce0 con 0x7f043c0cf920 2024-01-24T17:49:14.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.477+0000 7f0432a00640 1 -- 172.21.15.39:0/3877115625 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0424005e40 con 0x7f043c0cf920 2024-01-24T17:49:14.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.477+0000 7f0432a00640 1 -- 172.21.15.39:0/3877115625 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424007270 con 0x7f043c0cf920 2024-01-24T17:49:14.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.477+0000 7f0441600640 1 -- 172.21.15.39:0/3877115625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c1f25a0 con 0x7f043c0cf920 2024-01-24T17:49:14.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.477+0000 7f0441600640 1 -- 172.21.15.39:0/3877115625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c1f5fc0 con 0x7f043c0cf920 2024-01-24T17:49:14.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.479+0000 7f0432a00640 1 -- 172.21.15.39:0/3877115625 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0424007490 con 0x7f043c0cf920 2024-01-24T17:49:14.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.480+0000 7f0432a00640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.480+0000 7f0432a00640 1 -- 172.21.15.39:0/3877115625 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f0424087e50 con 0x7f043c0cf920 2024-01-24T17:49:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.481+0000 7f043aa00640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.481+0000 7f04443f3740 1 Processor -- start 2024-01-24T17:49:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.481+0000 7f04443f3740 1 -- start start 2024-01-24T17:49:14.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f04443f3740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678de3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f043aa00640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f042c004ea0 tx=0x7f042c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f04443f3740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678adf90 0x5593678db280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f04443f3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678db7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f04443f3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593678896c0 con 0x5593678b1b90 2024-01-24T17:49:14.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f04443f3740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55936781e070 con 0x5593678de010 2024-01-24T17:49:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f04443f3740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55936781b800 con 0x5593678adf90 2024-01-24T17:49:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f043b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678db7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f043b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678db7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35658/0 (socket says 172.21.15.39:35658) 2024-01-24T17:49:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.482+0000 7f043b400640 1 -- 172.21.15.39:0/2552675894 learned_addr learned my addr 172.21.15.39:0/2552675894 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f043aa00640 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678de3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f043a000640 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678adf90 0x5593678db280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f043b400640 1 -- 172.21.15.39:0/2552675894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678adf90 msgr2=0x5593678db280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f043b400640 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678adf90 0x5593678db280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f043b400640 1 -- 172.21.15.39:0/2552675894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 msgr2=0x5593678de3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f043b400640 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678de3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f043b400640 1 -- 172.21.15.39:0/2552675894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593677de780 con 0x5593678b1b90 2024-01-24T17:49:14.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f043b400640 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678db7c0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f0434037040 tx=0x7f043402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb001a211b90bb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f042b400640 1 -- 172.21.15.39:0/2552675894 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043403f680 con 0x5593678b1b90 2024-01-24T17:49:14.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f042b400640 1 -- 172.21.15.39:0/2552675894 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043403f7e0 con 0x5593678b1b90 2024-01-24T17:49:14.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.483+0000 7f042b400640 1 -- 172.21.15.39:0/2552675894 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043403fb50 con 0x5593678b1b90 2024-01-24T17:49:14.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 -- 172.21.15.39:0/2552675894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 msgr2=0x5593678db7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678db7c0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f0434037040 tx=0x7f043402e000 comp rx=0 tx=0).stop 2024-01-24T17:49:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 -- 172.21.15.39:0/2552675894 shutdown_connections 2024-01-24T17:49:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678db7c0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678adf90 0x5593678db280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 --2- 172.21.15.39:0/2552675894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678de3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 -- 172.21.15.39:0/2552675894 >> 172.21.15.39:0/2552675894 conn(0x5593678b02a0 msgr2=0x5593678f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 -- 172.21.15.39:0/2552675894 shutdown_connections 2024-01-24T17:49:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.484+0000 7f04443f3740 1 -- 172.21.15.39:0/2552675894 wait complete. 2024-01-24T17:49:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.485+0000 7f04443f3740 1 -- start start 2024-01-24T17:49:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.485+0000 7f04443f3740 1 Processor -- start 2024-01-24T17:49:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.486+0000 7f04443f3740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678b1b90 0x5593678f9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.486+0000 7f04443f3740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 0x5593678fb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.486+0000 7f04443f3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559367909020 0x5593678fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.486+0000 7f04443f3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593678170f0 con 0x559367909020 2024-01-24T17:49:14.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.486+0000 7f04443f3740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593677b88c0 con 0x5593678b1b90 2024-01-24T17:49:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.486+0000 7f04443f3740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5593677b8560 con 0x5593678de010 2024-01-24T17:49:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.486+0000 7f04443f3740 10 client.0 _ll_register_callbacks cb 0x559367810a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:49:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559367909020 0x5593678fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559367909020 0x5593678fb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35660/0 (socket says 172.21.15.39:35660) 2024-01-24T17:49:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 -- 172.21.15.39:0/1791331747 learned_addr learned my addr 172.21.15.39:0/1791331747 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043aa00640 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678b1b90 0x5593678f9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043a000640 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 0x5593678fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 -- 172.21.15.39:0/1791331747 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 msgr2=0x5593678fb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 0x5593678fb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 -- 172.21.15.39:0/1791331747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678b1b90 msgr2=0x5593678f9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678b1b90 0x5593678f9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 -- 172.21.15.39:0/1791331747 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c0b6310 con 0x559367909020 2024-01-24T17:49:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f043b400640 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559367909020 0x5593678fb8c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f043403fd20 tx=0x7f0434004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e37c63066b03ef server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.487+0000 7f0423e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:49:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f0423e00640 1 -- 172.21.15.39:0/1791331747 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434003da0 con 0x559367909020 2024-01-24T17:49:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f0423e00640 1 -- 172.21.15.39:0/1791331747 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f0434004030 con 0x559367909020 2024-01-24T17:49:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f04443f3740 1 -- 172.21.15.39:0/1791331747 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559367841640 con 0x559367909020 2024-01-24T17:49:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f0423e00640 1 -- 172.21.15.39:0/1791331747 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04340041f0 con 0x559367909020 2024-01-24T17:49:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f04443f3740 10 client.8226 fetch_fsmap learned FSMap version 71 2024-01-24T17:49:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f04443f3740 1 -- 172.21.15.39:0/1791331747 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) v3 -- 0x559367815770 con 0x559367909020 2024-01-24T17:49:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f0423e00640 1 -- 172.21.15.39:0/1791331747 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559367841640 con 0x559367909020 2024-01-24T17:49:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f0423e00640 1 -- 172.21.15.39:0/1791331747 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 71) v1 ==== 3126+0+0 (secure 0 0 0) 0x7f0434047020 con 0x559367909020 2024-01-24T17:49:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f04443f3740 10 client.8226 fetch_fsmap finished waiting for FSMap version 71 2024-01-24T17:49:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f04443f3740 10 client.8226 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-24T17:49:14.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.488+0000 7f04443f3740 20 client.8226 populate_metadata read hostname 'smithi039' 2024-01-24T17:49:14.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.489+0000 7f04443f3740 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] conn(0x5593678d7e20 0x5593678c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.489+0000 7f04443f3740 4 client.8226 mds_command: new command op to 8100 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:49:14.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.489+0000 7f04443f3740 1 -- 172.21.15.39:0/1791331747 --> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559367809940 con 0x5593678d7e20 2024-01-24T17:49:14.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.489+0000 7f043aa00640 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] conn(0x5593678d7e20 0x5593678c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.489+0000 7f0423e00640 10 client.8226 ms_handle_connect on v2:172.21.15.39:6832/3903887009 2024-01-24T17:49:14.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.489+0000 7f043aa00640 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] conn(0x5593678d7e20 0x5593678c3e30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e265891466829624 server_cookie=e1ddcb6cc9390cae in_seq=0 out_seq=0 2024-01-24T17:49:14.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.490+0000 7f0423e00640 1 -- 172.21.15.39:0/1791331747 <== mds.1 v2:172.21.15.39:6832/3903887009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559367809940 con 0x5593678d7e20 2024-01-24T17:49:14.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.490+0000 7f0423e00640 10 client.8226 handle_command_reply: tid=0 2024-01-24T17:49:14.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.490+0000 7f04443f3740 1 client.8226 shutdown 2024-01-24T17:49:14.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 -- 172.21.15.39:0/1791331747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559367909020 msgr2=0x5593678fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559367909020 0x5593678fb8c0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f043403fd20 tx=0x7f0434004ab0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 -- 172.21.15.39:0/1791331747 shutdown_connections 2024-01-24T17:49:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] conn(0x5593678d7e20 0x5593678c3e30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559367909020 0x5593678fb8c0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 0x5593678fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 --2- 172.21.15.39:0/1791331747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678b1b90 0x5593678f9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f0423e00640 0 client.8226 ms_handle_reset on v2:172.21.15.39:6832/3903887009 2024-01-24T17:49:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 -- 172.21.15.39:0/1791331747 >> 172.21.15.39:0/1791331747 conn(0x5593678adf90 msgr2=0x5593678e1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 -- 172.21.15.39:0/1791331747 shutdown_connections 2024-01-24T17:49:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 1 -- 172.21.15.39:0/1791331747 wait complete. 2024-01-24T17:49:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.491+0000 7f04443f3740 20 client.8226 trim_cache size 0 max 16384 2024-01-24T17:49:14.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.508+0000 7f04443f3740 1 Processor -- start 2024-01-24T17:49:14.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.508+0000 7f04443f3740 1 -- start start 2024-01-24T17:49:14.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f04443f3740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678adf90 0x5593678b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f04443f3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678b2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f04443f3740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 0x5593678fb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f04443f3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55936757ef20 con 0x5593678b1b90 2024-01-24T17:49:14.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f04443f3740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593678d8580 con 0x5593678adf90 2024-01-24T17:49:14.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f04443f3740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x559367816290 con 0x5593678de010 2024-01-24T17:49:14.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678b2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35668/0 (socket says 172.21.15.39:35668) 2024-01-24T17:49:14.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043a000640 1 -- 172.21.15.39:0/3605976227 learned_addr learned my addr 172.21.15.39:0/3605976227 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043aa00640 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678adf90 0x5593678b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043b400640 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 0x5593678fb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043a000640 1 -- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 msgr2=0x5593678fb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043a000640 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 0x5593678fb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043a000640 1 -- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678adf90 msgr2=0x5593678b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043a000640 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678adf90 0x5593678b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043a000640 1 -- 172.21.15.39:0/3605976227 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55936784b2f0 con 0x5593678b1b90 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.509+0000 7f043aa00640 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678adf90 0x5593678b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.510+0000 7f043a000640 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678b2280 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f0424086270 tx=0x7f0424056b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941e6bea620694c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.510+0000 7f042be00640 1 -- 172.21.15.39:0/3605976227 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424055440 con 0x5593678b1b90 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.510+0000 7f042be00640 1 -- 172.21.15.39:0/3605976227 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04240537b0 con 0x5593678b1b90 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.510+0000 7f042be00640 1 -- 172.21.15.39:0/3605976227 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04240567d0 con 0x5593678b1b90 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 -- 172.21.15.39:0/3605976227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 msgr2=0x5593678b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678b2280 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f0424086270 tx=0x7f0424056b30 comp rx=0 tx=0).stop 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 -- 172.21.15.39:0/3605976227 shutdown_connections 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678de010 0x5593678fb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678b1b90 0x5593678b2280 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 --2- 172.21.15.39:0/3605976227 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678adf90 0x5593678b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 -- 172.21.15.39:0/3605976227 >> 172.21.15.39:0/3605976227 conn(0x5593678d8880 msgr2=0x5593678ad410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 -- 172.21.15.39:0/3605976227 shutdown_connections 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.511+0000 7f04443f3740 1 -- 172.21.15.39:0/3605976227 wait complete. 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.512+0000 7f04443f3740 1 -- start start 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.512+0000 7f04443f3740 1 Processor -- start 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f04443f3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678adf90 0x5593678e0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f04443f3740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678b1b90 0x5593678f9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f04443f3740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678f9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f04443f3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593678e0a90 con 0x5593678adf90 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f04443f3740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593678c3fb0 con 0x5593678de010 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f04443f3740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5593678d2020 con 0x5593678b1b90 2024-01-24T17:49:14.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f04443f3740 10 client.0 _ll_register_callbacks cb 0x7f0434009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f043aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678adf90 0x5593678e0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f043aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678adf90 0x5593678e0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35682/0 (socket says 172.21.15.39:35682) 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.513+0000 7f043aa00640 1 -- 172.21.15.39:0/2380424243 learned_addr learned my addr 172.21.15.39:0/2380424243 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043a000640 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678b1b90 0x5593678f9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043b400640 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043aa00640 1 -- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678b1b90 msgr2=0x5593678f9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043aa00640 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678b1b90 0x5593678f9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043aa00640 1 -- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 msgr2=0x5593678f9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043aa00640 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678f9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043aa00640 1 -- 172.21.15.39:0/2380424243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c1f5fc0 con 0x5593678adf90 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043b400640 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678f9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f043aa00640 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678adf90 0x5593678e0400 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x5593678d7920 tx=0x7f042c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c110c038e137d05b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f0423e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:49:14.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f0423e00640 1 -- 172.21.15.39:0/2380424243 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c027310 con 0x5593678adf90 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f04443f3740 1 -- 172.21.15.39:0/2380424243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593678d9000 con 0x5593678adf90 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.514+0000 7f0423e00640 1 -- 172.21.15.39:0/2380424243 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f042c027980 con 0x5593678adf90 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f0423e00640 1 -- 172.21.15.39:0/2380424243 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f042c027b40 con 0x5593678adf90 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f04443f3740 10 client.8232 fetch_fsmap learned FSMap version 71 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f04443f3740 1 -- 172.21.15.39:0/2380424243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) v3 -- 0x5593677b7ad0 con 0x5593678adf90 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f0423e00640 1 -- 172.21.15.39:0/2380424243 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5593678d9000 con 0x5593678adf90 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f0423e00640 1 -- 172.21.15.39:0/2380424243 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 71) v1 ==== 3126+0+0 (secure 0 0 0) 0x7f042c027ca0 con 0x5593678adf90 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f04443f3740 10 client.8232 fetch_fsmap finished waiting for FSMap version 71 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f04443f3740 10 client.8232 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f04443f3740 20 client.8232 populate_metadata read hostname 'smithi039' 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f04443f3740 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] conn(0x559367909020 0x5593678d1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f04443f3740 4 client.8232 mds_command: new command op to 8100 tid=0[{"prefix": "status"}] 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f04443f3740 1 -- 172.21.15.39:0/2380424243 --> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f043403f7e0 con 0x559367909020 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.515+0000 7f043a000640 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] conn(0x559367909020 0x5593678d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f043a000640 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] conn(0x559367909020 0x5593678d1310 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=155ac1331d19494c server_cookie=d2da804bc2d362be in_seq=0 out_seq=0 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f0423e00640 10 client.8232 ms_handle_connect on v2:172.21.15.39:6832/3903887009 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f0423e00640 1 -- 172.21.15.39:0/2380424243 <== mds.1 v2:172.21.15.39:6832/3903887009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f043403f7e0 con 0x559367909020 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f0423e00640 10 client.8232 handle_command_reply: tid=0 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f04443f3740 1 client.8232 shutdown 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f04443f3740 1 -- 172.21.15.39:0/2380424243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678adf90 msgr2=0x5593678e0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f04443f3740 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678adf90 0x5593678e0400 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x5593678d7920 tx=0x7f042c0036f0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f04443f3740 1 -- 172.21.15.39:0/2380424243 shutdown_connections 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f04443f3740 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.39:6832/3903887009,v1:172.21.15.39:6833/3903887009] conn(0x559367909020 0x5593678d1310 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.516+0000 7f04443f3740 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5593678de010 0x5593678f9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f04443f3740 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5593678b1b90 0x5593678f9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f04443f3740 1 --2- 172.21.15.39:0/2380424243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5593678adf90 0x5593678e0400 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f04443f3740 1 -- 172.21.15.39:0/2380424243 >> 172.21.15.39:0/2380424243 conn(0x5593678d8880 msgr2=0x5593678b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f0423e00640 0 client.8232 ms_handle_reset on v2:172.21.15.39:6832/3903887009 2024-01-24T17:49:14.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f04443f3740 1 -- 172.21.15.39:0/2380424243 shutdown_connections 2024-01-24T17:49:14.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f04443f3740 1 -- 172.21.15.39:0/2380424243 wait complete. 2024-01-24T17:49:14.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f04443f3740 20 client.8232 trim_cache size 0 max 16384 2024-01-24T17:49:14.534 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:49:14.534 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:49:14.534 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 1, 2024-01-24T17:49:14.534 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 8100, 2024-01-24T17:49:14.534 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:49:14.534 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:49:14.534 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:49:14.534 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 3.0834165339999999, 2024-01-24T17:49:14.535 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 71, 2024-01-24T17:49:14.535 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 104, 2024-01-24T17:49:14.535 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 104, 2024-01-24T17:49:14.535 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 15.943594701 2024-01-24T17:49:14.535 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f0441600640 1 -- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 msgr2=0x7f041406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f0441600640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f042c004ea0 tx=0x7f042c009040 comp rx=0 tx=0).stop 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f0441600640 1 -- 172.21.15.39:0/3877115625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0cf920 msgr2=0x7f043c1f2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.517+0000 7f0441600640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0cf920 0x7f043c1f2da0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f042400c4a0 tx=0x7f042400c960 comp rx=0 tx=0).stop 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.518+0000 7f0441600640 1 -- 172.21.15.39:0/3877115625 shutdown_connections 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.518+0000 7f0441600640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f043c15ce90 0x7f043c15a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.518+0000 7f0441600640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f041406a430 0x7f041406c8f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.518+0000 7f0441600640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f043c0cf920 0x7f043c1f2da0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.518+0000 7f0441600640 1 --2- 172.21.15.39:0/3877115625 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f043c0b7840 0x7f043c1f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.518+0000 7f0441600640 1 -- 172.21.15.39:0/3877115625 >> 172.21.15.39:0/3877115625 conn(0x7f043c06af00 msgr2=0x7f043c06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.518+0000 7f0441600640 1 -- 172.21.15.39:0/3877115625 shutdown_connections 2024-01-24T17:49:14.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.518+0000 7f0441600640 1 -- 172.21.15.39:0/3877115625 wait complete. 2024-01-24T17:49:14.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-24T17:49:14.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.750+0000 7f58ff200640 1 Processor -- start 2024-01-24T17:49:14.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.750+0000 7f58ff200640 1 -- start start 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.751+0000 7f58ff200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f58f81432a0 0x7f58f8159e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.751+0000 7f58ff200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80b35f0 0x7f58f80ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.751+0000 7f58ff200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f80ce130 con 0x7f58f80cd0a0 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.751+0000 7f58ff200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f80cd520 con 0x7f58f81432a0 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.751+0000 7f58ff200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f80cd800 con 0x7f58f80b35f0 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58f7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f58f81432a0 0x7f58f8159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58f7e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f58f81432a0 0x7f58f8159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35698/0 (socket says 172.21.15.39:35698) 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58f7e00640 1 -- 172.21.15.39:0/3222690671 learned_addr learned my addr 172.21.15.39:0/3222690671 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58ef200640 1 --2- 172.21.15.39:0/3222690671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80b35f0 0x7f58f80ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58f7e00640 1 -- 172.21.15.39:0/3222690671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80b35f0 msgr2=0x7f58f80ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58f7e00640 1 --2- 172.21.15.39:0/3222690671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80b35f0 0x7f58f80ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58f7e00640 1 -- 172.21.15.39:0/3222690671 >> v1:172.21.15.39:6789/0 conn(0x7f58f80cd0a0 legacy=0x7f58f80b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58f7e00640 1 -- 172.21.15.39:0/3222690671 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f80cdae0 con 0x7f58f81432a0 2024-01-24T17:49:14.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.752+0000 7f58f7400640 1 -- 172.21.15.39:0/3222690671 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869698848 0 0) 0x7f58f80ce130 con 0x7f58f80cd0a0 2024-01-24T17:49:14.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.753+0000 7f58f7e00640 1 --2- 172.21.15.39:0/3222690671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8159e70 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009870 tx=0x7f58e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac9da4c090e5b19 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58f7400640 1 -- 172.21.15.39:0/3222690671 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8030840 con 0x7f58f81432a0 2024-01-24T17:49:14.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58f7400640 1 -- 172.21.15.39:0/3222690671 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e80309a0 con 0x7f58f81432a0 2024-01-24T17:49:14.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58f7400640 1 -- 172.21.15.39:0/3222690671 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8030d30 con 0x7f58f81432a0 2024-01-24T17:49:14.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58ff200640 1 -- 172.21.15.39:0/3222690671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 msgr2=0x7f58f8159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58ff200640 1 --2- 172.21.15.39:0/3222690671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8159e70 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009870 tx=0x7f58e802b980 comp rx=0 tx=0).stop 2024-01-24T17:49:14.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58ff200640 1 -- 172.21.15.39:0/3222690671 shutdown_connections 2024-01-24T17:49:14.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58ff200640 1 --2- 172.21.15.39:0/3222690671 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80b35f0 0x7f58f80ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58ff200640 1 --2- 172.21.15.39:0/3222690671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8159e70 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58ff200640 1 -- 172.21.15.39:0/3222690671 >> 172.21.15.39:0/3222690671 conn(0x7f58f8065ff0 msgr2=0x7f58f8069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58ff200640 1 -- 172.21.15.39:0/3222690671 shutdown_connections 2024-01-24T17:49:14.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.754+0000 7f58ff200640 1 -- 172.21.15.39:0/3222690671 wait complete. 2024-01-24T17:49:14.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.756+0000 7f58ff200640 1 Processor -- start 2024-01-24T17:49:14.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.756+0000 7f58ff200640 1 -- start start 2024-01-24T17:49:14.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.757+0000 7f58ff200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58f80b35f0 0x7f58f81372e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.757+0000 7f58ff200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80cd0a0 0x7f58f80aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.757+0000 7f58ff200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8131bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.757+0000 7f58ff200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f8132fe0 con 0x7f58f81432a0 2024-01-24T17:49:14.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.757+0000 7f58ff200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f8132130 con 0x7f58f80b35f0 2024-01-24T17:49:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.757+0000 7f58ff200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f81323d0 con 0x7f58f80cd0a0 2024-01-24T17:49:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8131bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8131bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35714/0 (socket says 172.21.15.39:35714) 2024-01-24T17:49:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58fd200640 1 -- 172.21.15.39:0/3187347376 learned_addr learned my addr 172.21.15.39:0/3187347376 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58ef200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80cd0a0 0x7f58f80aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58f7e00640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58f80b35f0 0x7f58f81372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58fd200640 1 -- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80cd0a0 msgr2=0x7f58f80aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58fd200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80cd0a0 0x7f58f80aff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58fd200640 1 -- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58f80b35f0 msgr2=0x7f58f81372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58fd200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58f80b35f0 0x7f58f81372e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.758+0000 7f58fd200640 1 -- 172.21.15.39:0/3187347376 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80094d0 con 0x7f58f81432a0 2024-01-24T17:49:14.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.759+0000 7f58fd200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8131bf0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c430 tx=0x7f58f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.759+0000 7f58f4c00640 1 -- 172.21.15.39:0/3187347376 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f00035f0 con 0x7f58f81432a0 2024-01-24T17:49:14.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.759+0000 7f58f4c00640 1 -- 172.21.15.39:0/3187347376 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f0003770 con 0x7f58f81432a0 2024-01-24T17:49:14.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.759+0000 7f58ff200640 1 -- 172.21.15.39:0/3187347376 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f81326b0 con 0x7f58f81432a0 2024-01-24T17:49:14.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.759+0000 7f58f4c00640 1 -- 172.21.15.39:0/3187347376 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0003b20 con 0x7f58f81432a0 2024-01-24T17:49:14.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.759+0000 7f58ff200640 1 -- 172.21.15.39:0/3187347376 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f8132c70 con 0x7f58f81432a0 2024-01-24T17:49:14.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.761+0000 7f58f4c00640 1 -- 172.21.15.39:0/3187347376 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f58f0003c80 con 0x7f58f81432a0 2024-01-24T17:49:14.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.762+0000 7f58f4c00640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f58d006a430 0x7f58d006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.762+0000 7f58f4c00640 1 -- 172.21.15.39:0/3187347376 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f58f0088100 con 0x7f58f81432a0 2024-01-24T17:49:14.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.763+0000 7f58f7e00640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f58d006a430 0x7f58d006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.763+0000 7f5901ff0740 1 Processor -- start 2024-01-24T17:49:14.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.763+0000 7f5901ff0740 1 -- start start 2024-01-24T17:49:14.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 0x5646418f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418c1f90 0x5646418ef280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 0x5646418ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f5901ff0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56464189d6c0 con 0x5646418c1f90 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f5901ff0740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x564641832070 con 0x5646418f2010 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f5901ff0740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56464182f800 con 0x5646418c5b90 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f58f7e00640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f58d006a430 0x7f58d006c8f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f58e8036000 tx=0x7f58e803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f58ef200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418c1f90 0x5646418ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f58ef200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418c1f90 0x5646418ef280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35724/0 (socket says 172.21.15.39:35724) 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.764+0000 7f58ef200640 1 -- 172.21.15.39:0/2843569147 learned_addr learned my addr 172.21.15.39:0/2843569147 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58f7e00640 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 0x5646418f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58fd200640 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 0x5646418ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58ef200640 1 -- 172.21.15.39:0/2843569147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 msgr2=0x5646418ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58ef200640 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 0x5646418ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58ef200640 1 -- 172.21.15.39:0/2843569147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 msgr2=0x5646418f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58ef200640 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 0x5646418f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58ef200640 1 -- 172.21.15.39:0/2843569147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646417f2780 con 0x5646418c1f90 2024-01-24T17:49:14.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58ef200640 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418c1f90 0x5646418ef280 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f58f81f5830 tx=0x7f58e0009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84edb1a40e9edb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58eca00640 1 -- 172.21.15.39:0/2843569147 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e0014610 con 0x5646418c1f90 2024-01-24T17:49:14.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58eca00640 1 -- 172.21.15.39:0/2843569147 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e000f050 con 0x5646418c1f90 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.765+0000 7f58eca00640 1 -- 172.21.15.39:0/2843569147 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e00149b0 con 0x5646418c1f90 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 -- 172.21.15.39:0/2843569147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418c1f90 msgr2=0x5646418ef280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418c1f90 0x5646418ef280 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f58f81f5830 tx=0x7f58e0009f20 comp rx=0 tx=0).stop 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 -- 172.21.15.39:0/2843569147 shutdown_connections 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 0x5646418ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418c1f90 0x5646418ef280 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 --2- 172.21.15.39:0/2843569147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 0x5646418f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 -- 172.21.15.39:0/2843569147 >> 172.21.15.39:0/2843569147 conn(0x5646418c42a0 msgr2=0x56464190ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 -- 172.21.15.39:0/2843569147 shutdown_connections 2024-01-24T17:49:14.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.766+0000 7f5901ff0740 1 -- 172.21.15.39:0/2843569147 wait complete. 2024-01-24T17:49:14.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.767+0000 7f5901ff0740 1 -- start start 2024-01-24T17:49:14.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.767+0000 7f5901ff0740 1 Processor -- start 2024-01-24T17:49:14.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.768+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 0x5646418f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.768+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 0x5646418e6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.768+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418e7610 0x5646418e97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.768+0000 7f5901ff0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56464182b0f0 con 0x5646418e7610 2024-01-24T17:49:14.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.768+0000 7f5901ff0740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646417cc8c0 con 0x5646418f2010 2024-01-24T17:49:14.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.768+0000 7f5901ff0740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646417cc560 con 0x5646418c5b90 2024-01-24T17:49:14.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.768+0000 7f5901ff0740 10 client.0 _ll_register_callbacks cb 0x564641824a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:49:14.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418e7610 0x5646418e97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418e7610 0x5646418e97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35738/0 (socket says 172.21.15.39:35738) 2024-01-24T17:49:14.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 -- 172.21.15.39:0/3101957817 learned_addr learned my addr 172.21.15.39:0/3101957817 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58ef200640 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 0x5646418e6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58f7e00640 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 0x5646418f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 -- 172.21.15.39:0/3101957817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 msgr2=0x5646418f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 0x5646418f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 -- 172.21.15.39:0/3101957817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 msgr2=0x5646418e6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 0x5646418e6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 -- 172.21.15.39:0/3101957817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e00096f0 con 0x5646418e7610 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58fd200640 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418e7610 0x5646418e97c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f58f00050a0 tx=0x7f58f0057210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bb0287625de3f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.769+0000 7f58e5600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f58e5600640 1 -- 172.21.15.39:0/3101957817 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f005ed60 con 0x5646418e7610 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f58e5600640 1 -- 172.21.15.39:0/3101957817 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f58f005eec0 con 0x5646418e7610 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 1 -- 172.21.15.39:0/3101957817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564641855640 con 0x5646418e7610 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f58e5600640 1 -- 172.21.15.39:0/3101957817 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58f0053000 con 0x5646418e7610 2024-01-24T17:49:14.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 10 client.8247 fetch_fsmap learned FSMap version 71 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 1 -- 172.21.15.39:0/3101957817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) v3 -- 0x564641829770 con 0x5646418e7610 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f58e5600640 1 -- 172.21.15.39:0/3101957817 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564641855640 con 0x5646418e7610 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f58e5600640 1 -- 172.21.15.39:0/3101957817 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 71) v1 ==== 3126+0+0 (secure 0 0 0) 0x7f58f0053160 con 0x5646418e7610 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 10 client.8247 fetch_fsmap finished waiting for FSMap version 71 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 10 client.8247 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 20 client.8247 populate_metadata read hostname 'smithi039' 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x564641927ee0 0x56464192a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 4 client.8247 mds_command: new command op to 8118 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.770+0000 7f5901ff0740 1 -- 172.21.15.39:0/3101957817 --> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56464181d940 con 0x564641927ee0 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.771+0000 7f58f7e00640 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x564641927ee0 0x56464192a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.771+0000 7f58f7e00640 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x564641927ee0 0x56464192a2d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e77adfc6ca1d793 server_cookie=10cead7d1e857006 in_seq=0 out_seq=0 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.771+0000 7f58e5600640 10 client.8247 ms_handle_connect on v2:172.21.15.39:6834/1608906778 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f58e5600640 1 -- 172.21.15.39:0/3101957817 <== mds.0 v2:172.21.15.39:6834/1608906778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56464181d940 con 0x564641927ee0 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f58e5600640 10 client.8247 handle_command_reply: tid=0 2024-01-24T17:49:14.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 client.8247 shutdown 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 -- 172.21.15.39:0/3101957817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418e7610 msgr2=0x5646418e97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418e7610 0x5646418e97c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f58f00050a0 tx=0x7f58f0057210 comp rx=0 tx=0).stop 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 -- 172.21.15.39:0/3101957817 shutdown_connections 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x564641927ee0 0x56464192a2d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418e7610 0x5646418e97c0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418f2010 0x5646418e6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 --2- 172.21.15.39:0/3101957817 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418c5b90 0x5646418f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f5901ff0740 1 -- 172.21.15.39:0/3101957817 >> 172.21.15.39:0/3101957817 conn(0x5646418c1f90 msgr2=0x5646418c6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.772+0000 7f58e5600640 0 client.8247 ms_handle_reset on v2:172.21.15.39:6834/1608906778 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.773+0000 7f5901ff0740 1 -- 172.21.15.39:0/3101957817 shutdown_connections 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.773+0000 7f5901ff0740 1 -- 172.21.15.39:0/3101957817 wait complete. 2024-01-24T17:49:14.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.773+0000 7f5901ff0740 20 client.8247 trim_cache size 0 max 16384 2024-01-24T17:49:14.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.789+0000 7f5901ff0740 1 Processor -- start 2024-01-24T17:49:14.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.789+0000 7f5901ff0740 1 -- start start 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 0x5646418f2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464190e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f5901ff0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564641592f20 con 0x5646418f2010 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f5901ff0740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646418ea1d0 con 0x5646418c5b90 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f5901ff0740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646418be340 con 0x5646418e7610 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f58fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464190e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f58fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464190e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35744/0 (socket says 172.21.15.39:35744) 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.790+0000 7f58fd200640 1 -- 172.21.15.39:0/1561791060 learned_addr learned my addr 172.21.15.39:0/1561791060 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58fd200640 1 -- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 msgr2=0x5646418e5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58ef200640 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58f7e00640 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 0x5646418f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58fd200640 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418e5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58fd200640 1 -- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 msgr2=0x5646418f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58fd200640 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 0x5646418f2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58fd200640 1 -- 172.21.15.39:0/1561791060 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56464185f2f0 con 0x5646418f2010 2024-01-24T17:49:14.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58ef200640 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418e5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:14.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58fd200640 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464190e680 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f58f005ecc0 tx=0x7f58f005d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4cfeac20955700 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58ed400640 1 -- 172.21.15.39:0/1561791060 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0072070 con 0x5646418f2010 2024-01-24T17:49:14.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58ed400640 1 -- 172.21.15.39:0/1561791060 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f000e040 con 0x5646418f2010 2024-01-24T17:49:14.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.791+0000 7f58ed400640 1 -- 172.21.15.39:0/1561791060 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0056b70 con 0x5646418f2010 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 -- 172.21.15.39:0/1561791060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 msgr2=0x56464190e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464190e680 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f58f005ecc0 tx=0x7f58f005d000 comp rx=0 tx=0).stop 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 -- 172.21.15.39:0/1561791060 shutdown_connections 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464190e680 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418e5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1561791060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 0x5646418f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 -- 172.21.15.39:0/1561791060 >> 172.21.15.39:0/1561791060 conn(0x5646418c1f90 msgr2=0x5646418e6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 -- 172.21.15.39:0/1561791060 shutdown_connections 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.792+0000 7f5901ff0740 1 -- 172.21.15.39:0/1561791060 wait complete. 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.793+0000 7f5901ff0740 1 -- start start 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.793+0000 7f5901ff0740 1 Processor -- start 2024-01-24T17:49:14.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 0x56464197f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418c7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f5901ff0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464197c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f5901ff0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56464197c070 con 0x5646418f2010 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f5901ff0740 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x564641965650 con 0x5646418c5b90 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f5901ff0740 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x56464197ff40 con 0x5646418e7610 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f5901ff0740 10 client.0 _ll_register_callbacks cb 0x7f58f0053550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f58fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464197c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f58fd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464197c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35756/0 (socket says 172.21.15.39:35756) 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f58fd200640 1 -- 172.21.15.39:0/1628957915 learned_addr learned my addr 172.21.15.39:0/1628957915 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.794+0000 7f58f7e00640 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 0x56464197f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58ef200640 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58fd200640 1 -- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 msgr2=0x5646418c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58fd200640 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58fd200640 1 -- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 msgr2=0x56464197f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:14.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58fd200640 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 0x56464197f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58fd200640 1 -- 172.21.15.39:0/1628957915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564641829770 con 0x5646418f2010 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58ef200640 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58fd200640 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464197c7a0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f58f0056f10 tx=0x7f58f0014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42651a556db46645 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58e5600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58e5600640 1 -- 172.21.15.39:0/1628957915 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0072030 con 0x5646418f2010 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58e5600640 1 -- 172.21.15.39:0/1628957915 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f58f0073070 con 0x5646418f2010 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f5901ff0740 1 -- 172.21.15.39:0/1628957915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564641824170 con 0x5646418f2010 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f58e5600640 1 -- 172.21.15.39:0/1628957915 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58f000e040 con 0x5646418f2010 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f5901ff0740 10 client.8253 fetch_fsmap learned FSMap version 71 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.795+0000 7f5901ff0740 1 -- 172.21.15.39:0/1628957915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) v3 -- 0x7f58f81340f0 con 0x5646418f2010 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f58e5600640 1 -- 172.21.15.39:0/1628957915 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564641824170 con 0x5646418f2010 2024-01-24T17:49:14.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f58e5600640 1 -- 172.21.15.39:0/1628957915 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 71) v1 ==== 3126+0+0 (secure 0 0 0) 0x7f58f0075b50 con 0x5646418f2010 2024-01-24T17:49:14.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f5901ff0740 10 client.8253 fetch_fsmap finished waiting for FSMap version 71 2024-01-24T17:49:14.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f5901ff0740 10 client.8253 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-24T17:49:14.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f5901ff0740 20 client.8253 populate_metadata read hostname 'smithi039' 2024-01-24T17:49:14.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56464197cce0 0x5646418e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:14.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f5901ff0740 4 client.8253 mds_command: new command op to 8118 tid=0[{"prefix": "status"}] 2024-01-24T17:49:14.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f5901ff0740 1 -- 172.21.15.39:0/1628957915 --> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564641823e90 con 0x56464197cce0 2024-01-24T17:49:14.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.796+0000 7f58f7e00640 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56464197cce0 0x5646418e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:14.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f58f7e00640 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56464197cce0 0x5646418e5b20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8915644b92b0975 server_cookie=3d2b01ac304a8368 in_seq=0 out_seq=0 2024-01-24T17:49:14.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f58e5600640 10 client.8253 ms_handle_connect on v2:172.21.15.39:6834/1608906778 2024-01-24T17:49:14.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f58e5600640 1 -- 172.21.15.39:0/1628957915 <== mds.0 v2:172.21.15.39:6834/1608906778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x564641823e90 con 0x56464197cce0 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f58e5600640 10 client.8253 handle_command_reply: tid=0 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 client.8253 shutdown 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 -- 172.21.15.39:0/1628957915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 msgr2=0x56464197c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464197c7a0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f58f0056f10 tx=0x7f58f0014070 comp rx=0 tx=0).stop 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 -- 172.21.15.39:0/1628957915 shutdown_connections 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.39:6834/1608906778,v1:172.21.15.39:6835/1608906778] conn(0x56464197cce0 0x5646418e5b20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5646418f2010 0x56464197c7a0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x5646418e7610 0x5646418c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f58e5600640 0 client.8253 ms_handle_reset on v2:172.21.15.39:6834/1608906778 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 --2- 172.21.15.39:0/1628957915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x5646418c5b90 0x56464197f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 -- 172.21.15.39:0/1628957915 >> 172.21.15.39:0/1628957915 conn(0x5646418c1f90 msgr2=0x5646418ea0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.797+0000 7f5901ff0740 1 -- 172.21.15.39:0/1628957915 shutdown_connections 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f5901ff0740 1 -- 172.21.15.39:0/1628957915 wait complete. 2024-01-24T17:49:14.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f5901ff0740 20 client.8253 trim_cache size 0 max 16384 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "b13477e8-9bd1-460b-97e1-2729dfdebd46", 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 8118, 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 15.450055271, 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 71, 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 104, 2024-01-24T17:49:14.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 104, 2024-01-24T17:49:14.804 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 16.118231266999999 2024-01-24T17:49:14.804 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:49:14.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 -- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f58d006a430 msgr2=0x7f58d006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f58d006a430 0x7f58d006c8f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f58e8036000 tx=0x7f58e803b040 comp rx=0 tx=0).stop 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 -- 172.21.15.39:0/3187347376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 msgr2=0x7f58f8131bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8131bf0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c430 tx=0x7f58f000c8f0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 -- 172.21.15.39:0/3187347376 shutdown_connections 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f58d006a430 0x7f58d006c8f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58f81432a0 0x7f58f8131bf0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f58f80cd0a0 0x7f58f80aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 --2- 172.21.15.39:0/3187347376 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58f80b35f0 0x7f58f81372e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 -- 172.21.15.39:0/3187347376 >> 172.21.15.39:0/3187347376 conn(0x7f58f8065ff0 msgr2=0x7f58f815b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.798+0000 7f58ff200640 1 -- 172.21.15.39:0/3187347376 shutdown_connections 2024-01-24T17:49:14.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:14.799+0000 7f58ff200640 1 -- 172.21.15.39:0/3187347376 wait complete. 2024-01-24T17:49:14.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-01-24T17:49:15.050 INFO:teuthology.orchestra.run.smithi039.stdout:{"message":"","return_code":0} 2024-01-24T17:49:15.050 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-01-24T17:49:15.050 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump objecter 2024-01-24T17:49:15.289 INFO:teuthology.orchestra.run.smithi039.stdout:{"objecter":{"op_active":0,"op_laggy":0,"op_send":39,"op_send_bytes":11424,"op_resend":27,"op_reply":39,"op_latency":{"avgcount":39,"sum":0.570988439,"avgtime":0.014640729},"op_inflight":0,"oplen_avg":{"avgcount":39,"sum":61},"op":39,"op_r":0,"op_w":39,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":10,"osdop_read":0,"osdop_write":2,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":20,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1}} 2024-01-24T17:49:15.290 DEBUG:tasks.cephfs.filesystem:_json_asok output { "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 39, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 39, "avgtime": 0.014640729, "sum": 0.570988439 }, "op_pg": 0, "op_r": 0, "op_reply": 39, "op_resend": 27, "op_rmw": 0, "op_send": 39, "op_send_bytes": 11424, "op_w": 39, "oplen_avg": { "avgcount": 39, "sum": 61 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 10, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 20, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 2, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 } } 2024-01-24T17:49:15.290 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json export dir /bravo 1 2024-01-24T17:49:15.522 INFO:teuthology.orchestra.run.smithi039.stdout:{"return_code":0} 2024-01-24T17:49:15.523 DEBUG:tasks.cephfs.filesystem:_json_asok output { "return_code": 0 } 2024-01-24T17:49:15.523 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump objecter 2024-01-24T17:49:15.754 INFO:teuthology.orchestra.run.smithi039.stdout:{"objecter":{"op_active":0,"op_laggy":0,"op_send":39,"op_send_bytes":11424,"op_resend":27,"op_reply":39,"op_latency":{"avgcount":39,"sum":0.570988439,"avgtime":0.014640729},"op_inflight":0,"oplen_avg":{"avgcount":39,"sum":61},"op":39,"op_r":0,"op_w":39,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":10,"osdop_read":0,"osdop_write":2,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":20,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1}} 2024-01-24T17:49:15.754 DEBUG:tasks.cephfs.filesystem:_json_asok output { "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 39, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 39, "avgtime": 0.014640729, "sum": 0.570988439 }, "op_pg": 0, "op_r": 0, "op_reply": 39, "op_resend": 27, "op_rmw": 0, "op_send": 39, "op_send_bytes": 11424, "op_w": 39, "oplen_avg": { "avgcount": 39, "sum": 61 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 10, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 20, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 2, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 } } 2024-01-24T17:49:15.755 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-01-24T17:49:20.755 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump objecter 2024-01-24T17:49:20.990 INFO:teuthology.orchestra.run.smithi039.stdout:{"objecter":{"op_active":0,"op_laggy":0,"op_send":41,"op_send_bytes":15036,"op_resend":27,"op_reply":41,"op_latency":{"avgcount":41,"sum":0.580988237,"avgtime":0.014170444},"op_inflight":0,"oplen_avg":{"avgcount":41,"sum":64},"op":41,"op_r":0,"op_w":41,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":10,"osdop_read":0,"osdop_write":3,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":20,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":106,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":17,"omap_rd":0,"omap_del":1}} 2024-01-24T17:49:20.991 DEBUG:tasks.cephfs.filesystem:_json_asok output { "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 106, "map_full": 0, "map_inc": 2, "omap_del": 1, "omap_rd": 0, "omap_wr": 17, "op": 41, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 41, "avgtime": 0.014170444, "sum": 0.580988237 }, "op_pg": 0, "op_r": 0, "op_reply": 41, "op_resend": 27, "op_rmw": 0, "op_send": 41, "op_send_bytes": 15036, "op_w": 41, "oplen_avg": { "avgcount": 41, "sum": 64 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 10, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 20, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 3, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 } } 2024-01-24T17:49:20.991 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-01-24T17:49:21.822 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:49:21.830 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:49:25.993 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump objecter 2024-01-24T17:49:26.241 INFO:teuthology.orchestra.run.smithi039.stdout:{"objecter":{"op_active":0,"op_laggy":0,"op_send":43,"op_send_bytes":15250,"op_resend":27,"op_reply":43,"op_latency":{"avgcount":43,"sum":0.589988055,"avgtime":0.013720652},"op_inflight":0,"oplen_avg":{"avgcount":43,"sum":67},"op":43,"op_r":0,"op_w":43,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":10,"osdop_read":0,"osdop_write":4,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":20,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":106,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":19,"omap_rd":0,"omap_del":1}} 2024-01-24T17:49:26.241 DEBUG:tasks.cephfs.filesystem:_json_asok output { "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 106, "map_full": 0, "map_inc": 2, "omap_del": 1, "omap_rd": 0, "omap_wr": 19, "op": 43, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 43, "avgtime": 0.013720652, "sum": 0.589988055 }, "op_pg": 0, "op_r": 0, "op_reply": 43, "op_resend": 27, "op_rmw": 0, "op_send": 43, "op_send_bytes": 15250, "op_w": 43, "oplen_avg": { "avgcount": 43, "sum": 67 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 10, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 20, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 4, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 } } 2024-01-24T17:49:26.241 DEBUG:tasks.ceph_test_case:wait_until_true: success in 10s and 0 retries 2024-01-24T17:49:26.242 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:49:26.242 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:26.298 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:49:26.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.522+0000 7f1fb6800640 1 Processor -- start 2024-01-24T17:49:26.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.522+0000 7f1fb6800640 1 -- start start 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1fb6800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb015b5f0 0x7f1fb00ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1fb6800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1fb00adb80 0x7f1fb015f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1fb6800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb015ade0 con 0x7f1fb0071630 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1fb6800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb015bb70 con 0x7f1fb00adb80 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1fb6800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb00b3540 con 0x7f1fb015b5f0 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1faea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1fb00adb80 0x7f1fb015f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1faea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f1fb00adb80 0x7f1fb015f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34560/0 (socket says 172.21.15.39:34560) 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1faea00640 1 -- 172.21.15.39:0/3810640796 learned_addr learned my addr 172.21.15.39:0/3810640796 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:26.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.524+0000 7f1faf400640 1 --2- 172.21.15.39:0/3810640796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb015b5f0 0x7f1fb00ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.525+0000 7f1faea00640 1 -- 172.21.15.39:0/3810640796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb015b5f0 msgr2=0x7f1fb00ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:26.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.525+0000 7f1faea00640 1 --2- 172.21.15.39:0/3810640796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb015b5f0 0x7f1fb00ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.525+0000 7f1faea00640 1 -- 172.21.15.39:0/3810640796 >> v1:172.21.15.100:6789/0 conn(0x7f1fb0071630 legacy=0x7f1fb0071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:26.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.525+0000 7f1faea00640 1 -- 172.21.15.39:0/3810640796 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb015be10 con 0x7f1fb00adb80 2024-01-24T17:49:26.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.526+0000 7f1faea00640 1 --2- 172.21.15.39:0/3810640796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb00adb80 0x7f1fb015f020 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1f98009870 tx=0x7f1f9802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d747bb86efb5b756 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:26.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.526+0000 7f1fae000640 1 -- 172.21.15.39:0/3810640796 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f980317c0 con 0x7f1fb00adb80 2024-01-24T17:49:26.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.526+0000 7f1fae000640 1 -- 172.21.15.39:0/3810640796 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f98031920 con 0x7f1fb00adb80 2024-01-24T17:49:26.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.526+0000 7f1fae000640 1 -- 172.21.15.39:0/3810640796 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f98031cb0 con 0x7f1fb00adb80 2024-01-24T17:49:26.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.527+0000 7f1fb6800640 1 -- 172.21.15.39:0/3810640796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb00adb80 msgr2=0x7f1fb015f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.527+0000 7f1fb6800640 1 --2- 172.21.15.39:0/3810640796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb00adb80 0x7f1fb015f020 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1f98009870 tx=0x7f1f9802b860 comp rx=0 tx=0).stop 2024-01-24T17:49:26.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.527+0000 7f1fb6800640 1 -- 172.21.15.39:0/3810640796 shutdown_connections 2024-01-24T17:49:26.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.527+0000 7f1fb6800640 1 --2- 172.21.15.39:0/3810640796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb00adb80 0x7f1fb015f020 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.527+0000 7f1fb6800640 1 --2- 172.21.15.39:0/3810640796 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb015b5f0 0x7f1fb00ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.527+0000 7f1fb6800640 1 -- 172.21.15.39:0/3810640796 >> 172.21.15.39:0/3810640796 conn(0x7f1fb006a220 msgr2=0x7f1fb00ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:26.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.527+0000 7f1fb6800640 1 -- 172.21.15.39:0/3810640796 shutdown_connections 2024-01-24T17:49:26.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.527+0000 7f1fb6800640 1 -- 172.21.15.39:0/3810640796 wait complete. 2024-01-24T17:49:26.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.529+0000 7f1fb6800640 1 Processor -- start 2024-01-24T17:49:26.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.529+0000 7f1fb6800640 1 -- start start 2024-01-24T17:49:26.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fb6800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fb0071630 0x7f1fb0097810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fb6800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb00adb80 0x7f1fb00970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fb6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb015b5f0 0x7f1fb0158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fb6800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb00cccf0 con 0x7f1fb015b5f0 2024-01-24T17:49:26.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fb6800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb0097600 con 0x7f1fb0071630 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fb6800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb00c2180 con 0x7f1fb00adb80 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fafe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb015b5f0 0x7f1fb0158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fafe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb015b5f0 0x7f1fb0158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34572/0 (socket says 172.21.15.39:34572) 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.530+0000 7f1fafe00640 1 -- 172.21.15.39:0/352835683 learned_addr learned my addr 172.21.15.39:0/352835683 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1faf400640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fb0071630 0x7f1fb0097810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1faea00640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb00adb80 0x7f1fb00970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fafe00640 1 -- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb00adb80 msgr2=0x7f1fb00970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fafe00640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb00adb80 0x7f1fb00970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fafe00640 1 -- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fb0071630 msgr2=0x7f1fb0097810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fafe00640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fb0071630 0x7f1fb0097810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fafe00640 1 -- 172.21.15.39:0/352835683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f980094d0 con 0x7f1fb015b5f0 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1faf400640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fb0071630 0x7f1fb0097810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fafe00640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb015b5f0 0x7f1fb0158130 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800b970 tx=0x7f1fa800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:26.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fa7400640 1 -- 172.21.15.39:0/352835683 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa800cb40 con 0x7f1fb015b5f0 2024-01-24T17:49:26.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fa7400640 1 -- 172.21.15.39:0/352835683 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa800cca0 con 0x7f1fb015b5f0 2024-01-24T17:49:26.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fa7400640 1 -- 172.21.15.39:0/352835683 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8014680 con 0x7f1fb015b5f0 2024-01-24T17:49:26.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.531+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb00cc3c0 con 0x7f1fb015b5f0 2024-01-24T17:49:26.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.532+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb00cc980 con 0x7f1fb015b5f0 2024-01-24T17:49:26.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.533+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb0073f10 con 0x7f1fb015b5f0 2024-01-24T17:49:26.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.534+0000 7f1fa7400640 1 -- 172.21.15.39:0/352835683 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1fa801d3b0 con 0x7f1fb015b5f0 2024-01-24T17:49:26.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.535+0000 7f1fa7400640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f8806a430 0x7f1f8806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.535+0000 7f1fa7400640 1 -- 172.21.15.39:0/352835683 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f1fa8088880 con 0x7f1fb015b5f0 2024-01-24T17:49:26.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.535+0000 7f1faf400640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f8806a430 0x7f1f8806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.537+0000 7f1faf400640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f8806a430 0x7f1f8806c8f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0005fd0 tx=0x7f1fa0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:26.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.539+0000 7f1fa7400640 1 -- 172.21.15.39:0/352835683 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1fa8053d30 con 0x7f1fb015b5f0 2024-01-24T17:49:26.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.733+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fb00cce30 con 0x7f1fb015b5f0 2024-01-24T17:49:26.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.735+0000 7f1fa7400640 1 -- 172.21.15.39:0/352835683 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16529 (secure 0 0 0) 0x7f1fa8057350 con 0x7f1fb015b5f0 2024-01-24T17:49:26.736 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:26.736 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":106,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:49:17.472389+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.0","mappings":[{"from":4,"to":0}]},{"pgid":"14.18","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6833/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.39:6832/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.100:6836/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.100:6834/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.39:6835/3495904204":"2024-01-25T17:48:58.377999+0000","172.21.15.39:6834/3495904204":"2024-01-25T17:48:58.377999+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f8806a430 msgr2=0x7f1f8806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f8806a430 0x7f1f8806c8f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0005fd0 tx=0x7f1fa0008040 comp rx=0 tx=0).stop 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb015b5f0 msgr2=0x7f1fb0158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb015b5f0 0x7f1fb0158130 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800b970 tx=0x7f1fa800be30 comp rx=0 tx=0).stop 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 shutdown_connections 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1f8806a430 0x7f1f8806c8f0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb015b5f0 0x7f1fb0158130 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1fb00adb80 0x7f1fb00970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 --2- 172.21.15.39:0/352835683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1fb0071630 0x7f1fb0097810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 >> 172.21.15.39:0/352835683 conn(0x7f1fb006a220 msgr2=0x7f1fb015a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:26.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 shutdown_connections 2024-01-24T17:49:26.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.738+0000 7f1fb6800640 1 -- 172.21.15.39:0/352835683 wait complete. 2024-01-24T17:49:26.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:49:26.757 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:49:26.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.974+0000 7f4378a00640 1 Processor -- start 2024-01-24T17:49:26.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.975+0000 7f4378a00640 1 -- start start 2024-01-24T17:49:26.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.976+0000 7f4378a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f437415cb70 0x7f43740aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.976+0000 7f4378a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 0x7f4374069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.976+0000 7f4378a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43740c9cf0 con 0x7f43740ce920 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.976+0000 7f4378a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437406a130 con 0x7f437415cb70 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.976+0000 7f4378a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437406a3d0 con 0x7f43740cb7e0 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4372800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f437415cb70 0x7f43740aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4372800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f437415cb70 0x7f43740aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34582/0 (socket says 172.21.15.39:34582) 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4373200640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f43740ce920 0x7f43740ceda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47520/0 (socket says 172.21.15.39:47520) 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4372800640 1 -- 172.21.15.39:0/238740718 learned_addr learned my addr 172.21.15.39:0/238740718 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4371e00640 1 --2- 172.21.15.39:0/238740718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 0x7f4374069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4372800640 1 -- 172.21.15.39:0/238740718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 msgr2=0x7f4374069bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4372800640 1 --2- 172.21.15.39:0/238740718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 0x7f4374069bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4372800640 1 -- 172.21.15.39:0/238740718 >> v1:172.21.15.39:6789/0 conn(0x7f43740ce920 legacy=0x7f43740ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.977+0000 7f4372800640 1 -- 172.21.15.39:0/238740718 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43740c9f90 con 0x7f437415cb70 2024-01-24T17:49:26.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.978+0000 7f4372800640 1 --2- 172.21.15.39:0/238740718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f437415cb70 0x7f43740aec10 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f4364009850 tx=0x7f436402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5cbaa0c24e602b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:26.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4371400640 1 -- 172.21.15.39:0/238740718 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364004230 con 0x7f437415cb70 2024-01-24T17:49:26.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4371400640 1 -- 172.21.15.39:0/238740718 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364004390 con 0x7f437415cb70 2024-01-24T17:49:26.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4371400640 1 -- 172.21.15.39:0/238740718 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364003810 con 0x7f437415cb70 2024-01-24T17:49:26.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4378a00640 1 -- 172.21.15.39:0/238740718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f437415cb70 msgr2=0x7f43740aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4378a00640 1 --2- 172.21.15.39:0/238740718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f437415cb70 0x7f43740aec10 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f4364009850 tx=0x7f436402b960 comp rx=0 tx=0).stop 2024-01-24T17:49:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4378a00640 1 -- 172.21.15.39:0/238740718 shutdown_connections 2024-01-24T17:49:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4378a00640 1 --2- 172.21.15.39:0/238740718 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 0x7f4374069bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4378a00640 1 --2- 172.21.15.39:0/238740718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f437415cb70 0x7f43740aec10 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4378a00640 1 -- 172.21.15.39:0/238740718 >> 172.21.15.39:0/238740718 conn(0x7f4374098a60 msgr2=0x7f43740cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4378a00640 1 -- 172.21.15.39:0/238740718 shutdown_connections 2024-01-24T17:49:26.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.979+0000 7f4378a00640 1 -- 172.21.15.39:0/238740718 wait complete. 2024-01-24T17:49:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.981+0000 7f4378a00640 1 Processor -- start 2024-01-24T17:49:26.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.981+0000 7f4378a00640 1 -- start start 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.982+0000 7f4378a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 0x7f43740c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.982+0000 7f4378a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43740ce920 0x7f43740ac250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.982+0000 7f4378a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f437415cb70 0x7f43740ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.982+0000 7f4378a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43740c85c0 con 0x7f43740ce920 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.982+0000 7f4378a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43740accd0 con 0x7f437415cb70 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.982+0000 7f4378a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437415af00 con 0x7f43740cb7e0 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43740ce920 0x7f43740ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43740ce920 0x7f43740ac250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34584/0 (socket says 172.21.15.39:34584) 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 -- 172.21.15.39:0/1313898446 learned_addr learned my addr 172.21.15.39:0/1313898446 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:26.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4373200640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f437415cb70 0x7f43740ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 -- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 msgr2=0x7f43740c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 0x7f43740c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 -- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f437415cb70 msgr2=0x7f43740ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f437415cb70 0x7f43740ac790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 -- 172.21.15.39:0/1313898446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4364009460 con 0x7f43740ce920 2024-01-24T17:49:26.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.983+0000 7f4371e00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43740ce920 0x7f43740ac250 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f435c007e60 tx=0x7f435c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:26.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.984+0000 7f436aa00640 1 -- 172.21.15.39:0/1313898446 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c003b90 con 0x7f43740ce920 2024-01-24T17:49:26.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.984+0000 7f436aa00640 1 -- 172.21.15.39:0/1313898446 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c00ee90 con 0x7f43740ce920 2024-01-24T17:49:26.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.984+0000 7f436aa00640 1 -- 172.21.15.39:0/1313898446 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c004030 con 0x7f43740ce920 2024-01-24T17:49:26.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.984+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43740c79d0 con 0x7f43740ce920 2024-01-24T17:49:26.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.984+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43740c81e0 con 0x7f43740ce920 2024-01-24T17:49:26.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.985+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437404e840 con 0x7f43740ce920 2024-01-24T17:49:26.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.986+0000 7f436aa00640 1 -- 172.21.15.39:0/1313898446 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f435c004190 con 0x7f43740ce920 2024-01-24T17:49:26.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.987+0000 7f436aa00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f434c06a430 0x7f434c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:26.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.987+0000 7f436aa00640 1 -- 172.21.15.39:0/1313898446 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f435c088ca0 con 0x7f43740ce920 2024-01-24T17:49:26.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.988+0000 7f4372800640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f434c06a430 0x7f434c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:26.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.989+0000 7f4372800640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f434c06a430 0x7f434c06c8f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4364035000 tx=0x7f4364030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:26.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:26.992+0000 7f436aa00640 1 -- 172.21.15.39:0/1313898446 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f435c054290 con 0x7f43740ce920 2024-01-24T17:49:27.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.185+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f43740736b0 con 0x7f43740ce920 2024-01-24T17:49:27.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.186+0000 7f436aa00640 1 -- 172.21.15.39:0/1313898446 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f435c01a020 con 0x7f43740ce920 2024-01-24T17:49:27.187 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f434c06a430 msgr2=0x7f434c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f434c06a430 0x7f434c06c8f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4364035000 tx=0x7f4364030000 comp rx=0 tx=0).stop 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43740ce920 msgr2=0x7f43740ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43740ce920 0x7f43740ac250 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f435c007e60 tx=0x7f435c00e4b0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 shutdown_connections 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f437415cb70 0x7f43740ac790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f434c06a430 0x7f434c06c8f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f43740ce920 0x7f43740ac250 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 --2- 172.21.15.39:0/1313898446 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f43740cb7e0 0x7f43740c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 >> 172.21.15.39:0/1313898446 conn(0x7f4374098a60 msgr2=0x7f43740cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 shutdown_connections 2024-01-24T17:49:27.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.189+0000 7f4378a00640 1 -- 172.21.15.39:0/1313898446 wait complete. 2024-01-24T17:49:27.210 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:49:27.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.433+0000 7f80de000640 1 Processor -- start 2024-01-24T17:49:27.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.433+0000 7f80de000640 1 -- start start 2024-01-24T17:49:27.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.435+0000 7f80de000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f80d815a840 0x7f80d815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.435+0000 7f80de000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d80ca770 con 0x7f80d806fe90 2024-01-24T17:49:27.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.435+0000 7f80de000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d815e9a0 con 0x7f80d80cea50 2024-01-24T17:49:27.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.435+0000 7f80de000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d815ec40 con 0x7f80d815a840 2024-01-24T17:49:27.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.435+0000 7f80cfc00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f80d806fe90 0x7f80d80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47526/0 (socket says 172.21.15.39:47526) 2024-01-24T17:49:27.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.435+0000 7f80cfc00640 1 -- 172.21.15.39:0/680028389 learned_addr learned my addr 172.21.15.39:0/680028389 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:27.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.435+0000 7f80dcc00640 1 --2- 172.21.15.39:0/680028389 >> v2:172.21.15.100:3300/0 conn(0x7f80d815a840 0x7f80d815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.435+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17761958 0 0) 0x7f80d80ca770 con 0x7f80d806fe90 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b8003520 con 0x7f80d806fe90 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3442717533 0 0) 0x7f80b8003520 con 0x7f80d806fe90 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 >> v2:172.21.15.100:3300/0 conn(0x7f80d815a840 msgr2=0x7f80d815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 --2- 172.21.15.39:0/680028389 >> v2:172.21.15.100:3300/0 conn(0x7f80d815a840 0x7f80d815e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 >> v1:172.21.15.100:6789/0 conn(0x7f80d80cea50 legacy=0x7f80d80caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d815ef20 con 0x7f80d806fe90 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f80c0001670 con 0x7f80d806fe90 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 >> v1:172.21.15.39:6789/0 conn(0x7f80d806fe90 legacy=0x7f80d80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b80039c0 0x7f80b8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b80047c0 0x7f80b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80b8005180 0x7f80b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d815ec40 con 0x7f80b8005180 2024-01-24T17:49:27.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b800c080 con 0x7f80b80047c0 2024-01-24T17:49:27.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.436+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b800c240 con 0x7f80b80039c0 2024-01-24T17:49:27.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.437+0000 7f80dd600640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80b8005180 0x7f80b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.437+0000 7f80dcc00640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b80039c0 0x7f80b8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.437+0000 7f80dd600640 1 -- 172.21.15.39:0/680028389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b80039c0 msgr2=0x7f80b8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:27.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.437+0000 7f80dd600640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80b80039c0 0x7f80b8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.437+0000 7f80dd600640 1 -- 172.21.15.39:0/680028389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b80047c0 msgr2=0x7f80b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:27.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.437+0000 7f80dd600640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b80047c0 0x7f80b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.437+0000 7f80dd600640 1 -- 172.21.15.39:0/680028389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b8003520 con 0x7f80b8005180 2024-01-24T17:49:27.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.438+0000 7f80dd600640 1 -- 172.21.15.39:0/680028389 reap_dead start 2024-01-24T17:49:27.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.438+0000 7f80dd600640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80b8005180 0x7f80b800b980 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f80d806fe40 tx=0x7f80b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41849d0cccb414ea server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.438+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d0003b70 con 0x7f80b8005180 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.438+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d0003cd0 con 0x7f80b8005180 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.439+0000 7f80d7e00640 1 -- 172.21.15.39:0/680028389 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d0005310 con 0x7f80b8005180 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.439+0000 7f80de000640 1 -- 172.21.15.39:0/680028389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80b8005180 msgr2=0x7f80b800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.439+0000 7f80de000640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80b8005180 0x7f80b800b980 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f80d806fe40 tx=0x7f80b80057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.439+0000 7f80de000640 1 -- 172.21.15.39:0/680028389 shutdown_connections 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.439+0000 7f80de000640 1 --2- 172.21.15.39:0/680028389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80b8005180 0x7f80b800b980 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.439+0000 7f80de000640 1 -- 172.21.15.39:0/680028389 >> 172.21.15.39:0/680028389 conn(0x7f80d806a220 msgr2=0x7f80d806bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.439+0000 7f80de000640 1 -- 172.21.15.39:0/680028389 shutdown_connections 2024-01-24T17:49:27.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.439+0000 7f80de000640 1 -- 172.21.15.39:0/680028389 wait complete. 2024-01-24T17:49:27.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.441+0000 7f80de000640 1 Processor -- start 2024-01-24T17:49:27.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.441+0000 7f80de000640 1 -- start start 2024-01-24T17:49:27.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.442+0000 7f80de000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b8005180 0x7f80d80ab8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.442+0000 7f80de000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80d81f1af0 0x7f80d80c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.442+0000 7f80de000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80d80c3920 0x7f80d815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.442+0000 7f80de000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d80c3da0 con 0x7f80d80c3920 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.442+0000 7f80de000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d80abe20 con 0x7f80b8005180 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.442+0000 7f80de000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d8146410 con 0x7f80d81f1af0 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80d80c3920 0x7f80d815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80d80c3920 0x7f80d815d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34596/0 (socket says 172.21.15.39:34596) 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 -- 172.21.15.39:0/2372999759 learned_addr learned my addr 172.21.15.39:0/2372999759 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80cfc00640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80d81f1af0 0x7f80d80c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dcc00640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b8005180 0x7f80d80ab8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 -- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80d81f1af0 msgr2=0x7f80d80c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80d81f1af0 0x7f80d80c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 -- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b8005180 msgr2=0x7f80d80ab8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b8005180 0x7f80d80ab8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 -- 172.21.15.39:0/2372999759 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d81466f0 con 0x7f80d80c3920 2024-01-24T17:49:27.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.443+0000 7f80dd600640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80d80c3920 0x7f80d815d3d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f80d0001f10 tx=0x7f80d00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:27.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.444+0000 7f80d5600640 1 -- 172.21.15.39:0/2372999759 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d0006d50 con 0x7f80d80c3920 2024-01-24T17:49:27.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.444+0000 7f80d5600640 1 -- 172.21.15.39:0/2372999759 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d0007030 con 0x7f80d80c3920 2024-01-24T17:49:27.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.444+0000 7f80d5600640 1 -- 172.21.15.39:0/2372999759 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d00073c0 con 0x7f80d80c3920 2024-01-24T17:49:27.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.444+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d81469d0 con 0x7f80d80c3920 2024-01-24T17:49:27.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.444+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d81f1f70 con 0x7f80d80c3920 2024-01-24T17:49:27.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.445+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d804e840 con 0x7f80d80c3920 2024-01-24T17:49:27.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.446+0000 7f80d5600640 1 -- 172.21.15.39:0/2372999759 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f80d0007520 con 0x7f80d80c3920 2024-01-24T17:49:27.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.447+0000 7f80d5600640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f80b006a430 0x7f80b006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.447+0000 7f80d5600640 1 -- 172.21.15.39:0/2372999759 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f80d00a99a0 con 0x7f80d80c3920 2024-01-24T17:49:27.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.448+0000 7f80dcc00640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f80b006a430 0x7f80b006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.449+0000 7f80dcc00640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f80b006a430 0x7f80b006c8f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009990 tx=0x7f80c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:27.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.451+0000 7f80d5600640 1 -- 172.21.15.39:0/2372999759 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f80d0074fe0 con 0x7f80d80c3920 2024-01-24T17:49:27.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.647+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80d806cc80 con 0x7f80d80c3920 2024-01-24T17:49:27.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.648+0000 7f80d5600640 1 -- 172.21.15.39:0/2372999759 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f80d0078600 con 0x7f80d80c3920 2024-01-24T17:49:27.649 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:49:27.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.650+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f80b006a430 msgr2=0x7f80b006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.650+0000 7f80de000640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f80b006a430 0x7f80b006c8f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009990 tx=0x7f80c8008040 comp rx=0 tx=0).stop 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.650+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80d80c3920 msgr2=0x7f80d815d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.650+0000 7f80de000640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80d80c3920 0x7f80d815d3d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f80d0001f10 tx=0x7f80d00014a0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.651+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 shutdown_connections 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.651+0000 7f80de000640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f80b006a430 0x7f80b006c8f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.651+0000 7f80de000640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80d80c3920 0x7f80d815d3d0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.651+0000 7f80de000640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f80d81f1af0 0x7f80d80c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.651+0000 7f80de000640 1 --2- 172.21.15.39:0/2372999759 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80b8005180 0x7f80d80ab8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.651+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 >> 172.21.15.39:0/2372999759 conn(0x7f80d806a220 msgr2=0x7f80d8070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.651+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 shutdown_connections 2024-01-24T17:49:27.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.651+0000 7f80de000640 1 -- 172.21.15.39:0/2372999759 wait complete. 2024-01-24T17:49:27.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:49:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.895+0000 7f2453600640 1 Processor -- start 2024-01-24T17:49:27.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.895+0000 7f2453600640 1 -- start start 2024-01-24T17:49:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.896+0000 7f2453600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f244c0cf820 0x7f244c0d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.897+0000 7f2453600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244c0d0cb0 con 0x7f244c0b5e40 2024-01-24T17:49:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.897+0000 7f2453600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244c0cdd70 con 0x7f244c15cd90 2024-01-24T17:49:27.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.897+0000 7f2453600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244c0ce050 con 0x7f244c0cf820 2024-01-24T17:49:27.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.897+0000 7f244be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f244c0b5e40 0x7f244c0cf110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47538/0 (socket says 172.21.15.39:47538) 2024-01-24T17:49:27.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.897+0000 7f244be00640 1 -- 172.21.15.39:0/862597559 learned_addr learned my addr 172.21.15.39:0/862597559 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.897+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460464900 0 0) 0x7f244c0d0cb0 con 0x7f244c0b5e40 2024-01-24T17:49:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.897+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242c003520 con 0x7f244c0b5e40 2024-01-24T17:49:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.897+0000 7f2451600640 1 --2- 172.21.15.39:0/862597559 >> v2:172.21.15.100:3300/0 conn(0x7f244c0cf820 0x7f244c0d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2711212516 0 0) 0x7f242c003520 con 0x7f244c0b5e40 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 >> v2:172.21.15.100:3300/0 conn(0x7f244c0cf820 msgr2=0x7f244c0d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 --2- 172.21.15.39:0/862597559 >> v2:172.21.15.100:3300/0 conn(0x7f244c0cf820 0x7f244c0d1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 >> v1:172.21.15.100:6789/0 conn(0x7f244c15cd90 legacy=0x7f244c0cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c0d0f20 con 0x7f244c0b5e40 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f2434001670 con 0x7f244c0b5e40 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 >> v1:172.21.15.39:6789/0 conn(0x7f244c0b5e40 legacy=0x7f244c0cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f242c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f242c0047c0 0x7f242c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f242c005180 0x7f242c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0ce050 con 0x7f242c003970 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c00c020 con 0x7f242c005180 2024-01-24T17:49:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c00c1e0 con 0x7f242c0047c0 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.898+0000 7f2450c00640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f242c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.899+0000 7f244be00640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f242c0047c0 0x7f242c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.899+0000 7f2451600640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f242c005180 0x7f242c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.899+0000 7f2450c00640 1 -- 172.21.15.39:0/862597559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f242c0047c0 msgr2=0x7f242c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.899+0000 7f2450c00640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f242c0047c0 0x7f242c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.899+0000 7f2450c00640 1 -- 172.21.15.39:0/862597559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f242c005180 msgr2=0x7f242c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.899+0000 7f2450c00640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f242c005180 0x7f242c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.899+0000 7f2451600640 1 -- 172.21.15.39:0/862597559 reap_dead start 2024-01-24T17:49:27.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.899+0000 7f2450c00640 1 -- 172.21.15.39:0/862597559 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c003520 con 0x7f242c003970 2024-01-24T17:49:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.900+0000 7f2450c00640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f242c003d90 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f243c001e50 tx=0x7f242c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef591dcdca311f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c003b70 con 0x7f242c003970 2024-01-24T17:49:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243c003cd0 con 0x7f242c003970 2024-01-24T17:49:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f244b400640 1 -- 172.21.15.39:0/862597559 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c005310 con 0x7f242c003970 2024-01-24T17:49:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f2453600640 1 -- 172.21.15.39:0/862597559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 msgr2=0x7f242c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f2453600640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f242c003d90 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f243c001e50 tx=0x7f242c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f2453600640 1 -- 172.21.15.39:0/862597559 shutdown_connections 2024-01-24T17:49:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f2453600640 1 --2- 172.21.15.39:0/862597559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f242c003d90 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f2453600640 1 -- 172.21.15.39:0/862597559 >> 172.21.15.39:0/862597559 conn(0x7f244c06af00 msgr2=0x7f244c06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f2453600640 1 -- 172.21.15.39:0/862597559 shutdown_connections 2024-01-24T17:49:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.901+0000 7f2453600640 1 -- 172.21.15.39:0/862597559 wait complete. 2024-01-24T17:49:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.903+0000 7f2453600640 1 Processor -- start 2024-01-24T17:49:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.903+0000 7f2453600640 1 -- start start 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.904+0000 7f2453600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f244c14df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.904+0000 7f2453600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f244c15a340 0x7f244c15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.904+0000 7f2453600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f244c0b4660 0x7f244c0b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.904+0000 7f2453600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0b5050 con 0x7f242c003970 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.904+0000 7f2453600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c15ff10 con 0x7f244c0b4660 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.904+0000 7f2453600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c15af00 con 0x7f244c15a340 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.904+0000 7f2450c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f244c14df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2450c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f244c14df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34616/0 (socket says 172.21.15.39:34616) 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2450c00640 1 -- 172.21.15.39:0/2059892632 learned_addr learned my addr 172.21.15.39:0/2059892632 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:27.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2451600640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f244c0b4660 0x7f244c0b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2450c00640 1 -- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f244c15a340 msgr2=0x7f244c15a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f244be00640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f244c15a340 0x7f244c15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2450c00640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f244c15a340 0x7f244c15a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2450c00640 1 -- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f244c0b4660 msgr2=0x7f244c0b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2450c00640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f244c0b4660 0x7f244c0b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2450c00640 1 -- 172.21.15.39:0/2059892632 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c15dc20 con 0x7f242c003970 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2451600640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f244c0b4660 0x7f244c0b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.905+0000 7f2450c00640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f244c14df80 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f243c02bf70 tx=0x7f243c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.906+0000 7f2448c00640 1 -- 172.21.15.39:0/2059892632 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c03e070 con 0x7f242c003970 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.906+0000 7f2448c00640 1 -- 172.21.15.39:0/2059892632 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243c003e40 con 0x7f242c003970 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.906+0000 7f2448c00640 1 -- 172.21.15.39:0/2059892632 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c0057e0 con 0x7f242c003970 2024-01-24T17:49:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.906+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c1580c0 con 0x7f242c003970 2024-01-24T17:49:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.906+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c15d5d0 con 0x7f242c003970 2024-01-24T17:49:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.907+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c073f10 con 0x7f242c003970 2024-01-24T17:49:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.908+0000 7f2448c00640 1 -- 172.21.15.39:0/2059892632 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f243c007030 con 0x7f242c003970 2024-01-24T17:49:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.909+0000 7f2448c00640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f242406a460 0x7f242406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.909+0000 7f2448c00640 1 -- 172.21.15.39:0/2059892632 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f243c0aa680 con 0x7f242c003970 2024-01-24T17:49:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.909+0000 7f244be00640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f242406a460 0x7f242406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.911+0000 7f244be00640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f242406a460 0x7f242406c920 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2434001050 tx=0x7f2434001d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:27.914+0000 7f2448c00640 1 -- 172.21.15.39:0/2059892632 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f243c075bc0 con 0x7f242c003970 2024-01-24T17:49:28.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.108+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f244c0d09f0 con 0x7f242c003970 2024-01-24T17:49:28.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.109+0000 7f2448c00640 1 -- 172.21.15.39:0/2059892632 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7f243c0791e0 con 0x7f242c003970 2024-01-24T17:49:28.110 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f242406a460 msgr2=0x7f242406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f242406a460 0x7f242406c920 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2434001050 tx=0x7f2434001d40 comp rx=0 tx=0).stop 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 msgr2=0x7f244c14df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f244c14df80 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f243c02bf70 tx=0x7f243c0014a0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 shutdown_connections 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f244c0b4660 0x7f244c0b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f244c15a340 0x7f244c15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f242406a460 0x7f242406c920 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 --2- 172.21.15.39:0/2059892632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f242c003970 0x7f244c14df80 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 >> 172.21.15.39:0/2059892632 conn(0x7f244c06af00 msgr2=0x7f244c0cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 shutdown_connections 2024-01-24T17:49:28.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.112+0000 7f2453600640 1 -- 172.21.15.39:0/2059892632 wait complete. 2024-01-24T17:49:28.130 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:49:28.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.351+0000 7f65b6000640 1 Processor -- start 2024-01-24T17:49:28.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.351+0000 7f65b6000640 1 -- start start 2024-01-24T17:49:28.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.353+0000 7f65b6000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65b0144080 0x7f65b00ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.353+0000 7f65b6000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f65b0144460 0x7f65b00cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.353+0000 7f65b6000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b015f590 con 0x7f65b00c3970 2024-01-24T17:49:28.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.353+0000 7f65b6000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b00c4490 con 0x7f65b0144460 2024-01-24T17:49:28.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.353+0000 7f65b6000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b015e7d0 con 0x7f65b0144080 2024-01-24T17:49:28.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.353+0000 7f65afe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f65b00c3970 0x7f65b00c3d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47546/0 (socket says 172.21.15.39:47546) 2024-01-24T17:49:28.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.353+0000 7f65afe00640 1 -- 172.21.15.39:0/3019379799 learned_addr learned my addr 172.21.15.39:0/3019379799 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:28.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.353+0000 7f65af400640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65b0144080 0x7f65b00ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65aea00640 1 --2- 172.21.15.39:0/3019379799 >> v2:172.21.15.100:3300/0 conn(0x7f65b0144460 0x7f65b00cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172537440 0 0) 0x7f65b015f590 con 0x7f65b00c3970 2024-01-24T17:49:28.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6590003520 con 0x7f65b00c3970 2024-01-24T17:49:28.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3426328520 0 0) 0x7f6590003520 con 0x7f65b00c3970 2024-01-24T17:49:28.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65b0144080 msgr2=0x7f65b00ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:28.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65b0144080 0x7f65b00ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 >> v2:172.21.15.100:3300/0 conn(0x7f65b0144460 msgr2=0x7f65b00cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:28.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 --2- 172.21.15.39:0/3019379799 >> v2:172.21.15.100:3300/0 conn(0x7f65b0144460 0x7f65b00cec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b015eab0 con 0x7f65b00c3970 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f65a80019f0 con 0x7f65b00c3970 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 >> v1:172.21.15.39:6789/0 conn(0x7f65b00c3970 legacy=0x7f65b00c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6590003970 0x7f6590003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65b0144080 0x7f65900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6590004d00 0x7f6590005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00c4490 con 0x7f6590003970 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b015e7d0 con 0x7f6590004d00 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.354+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659000b9b0 con 0x7f65b0144080 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.355+0000 7f65af400640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6590003970 0x7f6590003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.355+0000 7f65af400640 1 -- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65b0144080 msgr2=0x7f65900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.355+0000 7f65aea00640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65b0144080 0x7f65900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.355+0000 7f65afe00640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6590004d00 0x7f6590005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.355+0000 7f65af400640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f65b0144080 0x7f65900047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.355+0000 7f65af400640 1 -- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6590004d00 msgr2=0x7f6590005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:28.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.355+0000 7f65af400640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6590004d00 0x7f6590005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.355+0000 7f65af400640 1 -- 172.21.15.39:0/3019379799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6590003520 con 0x7f6590003970 2024-01-24T17:49:28.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.356+0000 7f65af400640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6590003970 0x7f6590003d90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009ac0 tx=0x7f65900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3da5ba09a42e56 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:28.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a002c6b0 con 0x7f6590003970 2024-01-24T17:49:28.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a002c810 con 0x7f6590003970 2024-01-24T17:49:28.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65ae000640 1 -- 172.21.15.39:0/3019379799 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a002cba0 con 0x7f6590003970 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65b6000640 1 -- 172.21.15.39:0/3019379799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6590003970 msgr2=0x7f6590003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65b6000640 1 --2- 172.21.15.39:0/3019379799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6590003970 0x7f6590003d90 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009ac0 tx=0x7f65900057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65afe00640 1 -- 172.21.15.39:0/3019379799 reap_dead start 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65b6000640 1 -- 172.21.15.39:0/3019379799 shutdown_connections 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65b6000640 1 -- 172.21.15.39:0/3019379799 >> 172.21.15.39:0/3019379799 conn(0x7f65b0065ff0 msgr2=0x7f65b0069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65b6000640 1 -- 172.21.15.39:0/3019379799 shutdown_connections 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.357+0000 7f65b6000640 1 -- 172.21.15.39:0/3019379799 wait complete. 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.359+0000 7f65b6000640 1 Processor -- start 2024-01-24T17:49:28.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.359+0000 7f65b6000640 1 -- start start 2024-01-24T17:49:28.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.360+0000 7f65b6000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6590003970 0x7f65b013f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.360+0000 7f65b6000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65b01f2530 0x7f65b013fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.360+0000 7f65b6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65b0142310 0x7f65b0142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.360+0000 7f65b6000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b0142cd0 con 0x7f65b0142310 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.360+0000 7f65b6000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b013fa70 con 0x7f65b01f2530 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.360+0000 7f65b6000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b01406d0 con 0x7f6590003970 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65b0142310 0x7f65b0142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65b0142310 0x7f65b0142790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34628/0 (socket says 172.21.15.39:34628) 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 -- 172.21.15.39:0/1638282018 learned_addr learned my addr 172.21.15.39:0/1638282018 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65af400640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6590003970 0x7f65b013f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65aea00640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65b01f2530 0x7f65b013fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 -- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6590003970 msgr2=0x7f65b013f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6590003970 0x7f65b013f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 -- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65b01f2530 msgr2=0x7f65b013fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65b01f2530 0x7f65b013fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 -- 172.21.15.39:0/1638282018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b01409b0 con 0x7f65b0142310 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65aea00640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65b01f2530 0x7f65b013fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:28.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.361+0000 7f65afe00640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65b0142310 0x7f65b0142790 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f65b00c5680 tx=0x7f65a800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:28.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.362+0000 7f65a7400640 1 -- 172.21.15.39:0/1638282018 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8004d60 con 0x7f65b0142310 2024-01-24T17:49:28.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.362+0000 7f65a7400640 1 -- 172.21.15.39:0/1638282018 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a800bb80 con 0x7f65b0142310 2024-01-24T17:49:28.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.362+0000 7f65a7400640 1 -- 172.21.15.39:0/1638282018 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8007070 con 0x7f65b0142310 2024-01-24T17:49:28.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.362+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b0140c90 con 0x7f65b0142310 2024-01-24T17:49:28.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.362+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b0141250 con 0x7f65b0142310 2024-01-24T17:49:28.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.363+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b0073f10 con 0x7f65b0142310 2024-01-24T17:49:28.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.364+0000 7f65a7400640 1 -- 172.21.15.39:0/1638282018 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f65a80071d0 con 0x7f65b0142310 2024-01-24T17:49:28.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.365+0000 7f65a7400640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f658806a430 0x7f658806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.365+0000 7f65a7400640 1 -- 172.21.15.39:0/1638282018 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f65a8088e60 con 0x7f65b0142310 2024-01-24T17:49:28.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.366+0000 7f65af400640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f658806a430 0x7f658806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.367+0000 7f65af400640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f658806a430 0x7f658806c8f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f65a0036000 tx=0x7f65a0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:28.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.370+0000 7f65a7400640 1 -- 172.21.15.39:0/1638282018 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f65a8054310 con 0x7f65b0142310 2024-01-24T17:49:28.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.579+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f65b00b1a10 con 0x7f65b0142310 2024-01-24T17:49:28.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.580+0000 7f65a7400640 1 -- 172.21.15.39:0/1638282018 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v106) v1 ==== 80+0+366 (secure 0 0 0) 0x7f65a8057930 con 0x7f65b0142310 2024-01-24T17:49:28.581 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6833/872967797 2024-01-25T17:48:58.450631+0000 2024-01-24T17:49:28.581 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6832/872967797 2024-01-25T17:48:58.450631+0000 2024-01-24T17:49:28.581 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.100:6836/3116066709 2024-01-25T17:48:58.404809+0000 2024-01-24T17:49:28.581 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.100:6834/3116066709 2024-01-25T17:48:58.404809+0000 2024-01-24T17:49:28.581 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6835/3495904204 2024-01-25T17:48:58.377999+0000 2024-01-24T17:49:28.581 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6834/3495904204 2024-01-25T17:48:58.377999+0000 2024-01-24T17:49:28.582 INFO:teuthology.orchestra.run.smithi039.stderr:listed 6 entries 2024-01-24T17:49:28.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f658806a430 msgr2=0x7f658806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f658806a430 0x7f658806c8f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f65a0036000 tx=0x7f65a0030000 comp rx=0 tx=0).stop 2024-01-24T17:49:28.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65b0142310 msgr2=0x7f65b0142790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65b0142310 0x7f65b0142790 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f65b00c5680 tx=0x7f65a800b4a0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 shutdown_connections 2024-01-24T17:49:28.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f658806a430 0x7f658806c8f0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f65b0142310 0x7f65b0142790 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f65b01f2530 0x7f65b013fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 --2- 172.21.15.39:0/1638282018 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f6590003970 0x7f65b013f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 >> 172.21.15.39:0/1638282018 conn(0x7f65b0065ff0 msgr2=0x7f65b0071440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:28.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.583+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 shutdown_connections 2024-01-24T17:49:28.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.584+0000 7f65b6000640 1 -- 172.21.15.39:0/1638282018 wait complete. 2024-01-24T17:49:28.603 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:49:28.603 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:28.656 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:49:28.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.876+0000 7fa81b200640 1 Processor -- start 2024-01-24T17:49:28.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.876+0000 7fa81b200640 1 -- start start 2024-01-24T17:49:28.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.878+0000 7fa81b200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fa81c0ca4b0 0x7fa81c06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.878+0000 7fa81b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa81c0cc230 0x7fa81c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.878+0000 7fa81b200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c0703d0 con 0x7fa81c1571f0 2024-01-24T17:49:28.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.878+0000 7fa81b200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa81c0ccf60 con 0x7fa81c0ca4b0 2024-01-24T17:49:28.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.878+0000 7fa81b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0cd780 con 0x7fa81c0cc230 2024-01-24T17:49:28.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.878+0000 7fa819e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa81c1571f0 0x7fa81c0c9da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47554/0 (socket says 172.21.15.39:47554) 2024-01-24T17:49:28.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.878+0000 7fa819e00640 1 -- 172.21.15.39:0/3429955985 learned_addr learned my addr 172.21.15.39:0/3429955985 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:28.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa813e00640 1 --2- 172.21.15.39:0/3429955985 >> v2:172.21.15.100:3300/0 conn(0x7fa81c0ca4b0 0x7fa81c06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa81a800640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa81c0cc230 0x7fa81c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298778975 0 0) 0x7fa81c0703d0 con 0x7fa81c1571f0 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7fc003520 con 0x7fa81c1571f0 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 999313185 0 0) 0x7fa7fc003520 con 0x7fa81c1571f0 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa81c0cc230 msgr2=0x7fa81c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa81c0cc230 0x7fa81c0cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 >> v2:172.21.15.100:3300/0 conn(0x7fa81c0ca4b0 msgr2=0x7fa81c06fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 --2- 172.21.15.39:0/3429955985 >> v2:172.21.15.100:3300/0 conn(0x7fa81c0ca4b0 0x7fa81c06fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0cda60 con 0x7fa81c1571f0 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fa80c001670 con 0x7fa81c1571f0 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 >> v1:172.21.15.39:6789/0 conn(0x7fa81c1571f0 legacy=0x7fa81c0c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7fc003970 0x7fa7fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7fc0047c0 0x7fa7fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa81c0cc230 0x7fa7fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0ccf60 con 0x7fa7fc003970 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0cd780 con 0x7fa7fc0047c0 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.879+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc00b980 con 0x7fa81c0cc230 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.880+0000 7fa819e00640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7fc003970 0x7fa7fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.880+0000 7fa81a800640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa81c0cc230 0x7fa7fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.880+0000 7fa813e00640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7fc0047c0 0x7fa7fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.880+0000 7fa819e00640 1 -- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa81c0cc230 msgr2=0x7fa7fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.880+0000 7fa819e00640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa81c0cc230 0x7fa7fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.880+0000 7fa819e00640 1 -- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7fc0047c0 msgr2=0x7fa7fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.880+0000 7fa819e00640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa7fc0047c0 0x7fa7fc004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.880+0000 7fa819e00640 1 -- 172.21.15.39:0/3429955985 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc003520 con 0x7fa7fc003970 2024-01-24T17:49:28.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.881+0000 7fa819e00640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7fc003970 0x7fa7fc003d90 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00c110 tx=0x7fa7fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec1855eab3401ea server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.881+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c005ad0 con 0x7fa7fc003970 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.881+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80c005c50 con 0x7fa7fc003970 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.881+0000 7fa819400640 1 -- 172.21.15.39:0/3429955985 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80c02c810 con 0x7fa7fc003970 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.882+0000 7fa81b200640 1 -- 172.21.15.39:0/3429955985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7fc003970 msgr2=0x7fa7fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.882+0000 7fa81b200640 1 --2- 172.21.15.39:0/3429955985 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7fc003970 0x7fa7fc003d90 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00c110 tx=0x7fa7fc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.882+0000 7fa81a800640 1 -- 172.21.15.39:0/3429955985 reap_dead start 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.882+0000 7fa81b200640 1 -- 172.21.15.39:0/3429955985 shutdown_connections 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.882+0000 7fa81b200640 1 -- 172.21.15.39:0/3429955985 >> 172.21.15.39:0/3429955985 conn(0x7fa81c06bf30 msgr2=0x7fa81c06c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.882+0000 7fa81b200640 1 -- 172.21.15.39:0/3429955985 shutdown_connections 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.882+0000 7fa81b200640 1 -- 172.21.15.39:0/3429955985 wait complete. 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.884+0000 7fa81b200640 1 Processor -- start 2024-01-24T17:49:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.884+0000 7fa81b200640 1 -- start start 2024-01-24T17:49:28.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.885+0000 7fa81b200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7fc003970 0x7fa81c14a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.885+0000 7fa81b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa81c1531d0 0x7fa81c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.885+0000 7fa81b200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa81c0cae40 0x7fa81c0cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.885+0000 7fa81b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c071800 con 0x7fa81c1531d0 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.885+0000 7fa81b200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0cb800 con 0x7fa81c0cae40 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.885+0000 7fa81b200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1522c0 con 0x7fa7fc003970 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.885+0000 7fa813e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa81c1531d0 0x7fa81c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa813e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa81c1531d0 0x7fa81c0712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34638/0 (socket says 172.21.15.39:34638) 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa813e00640 1 -- 172.21.15.39:0/2094835979 learned_addr learned my addr 172.21.15.39:0/2094835979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa819e00640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7fc003970 0x7fa81c14a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa81a800640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa81c0cae40 0x7fa81c0cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa813e00640 1 -- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7fc003970 msgr2=0x7fa81c14a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa813e00640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7fc003970 0x7fa81c14a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa813e00640 1 -- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa81c0cae40 msgr2=0x7fa81c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:28.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa813e00640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa81c0cae40 0x7fa81c0cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa813e00640 1 -- 172.21.15.39:0/2094835979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0c3a10 con 0x7fa81c1531d0 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa81a800640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa81c0cae40 0x7fa81c0cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.886+0000 7fa813e00640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa81c1531d0 0x7fa81c0712c0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fa804010890 tx=0x7fa804010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.887+0000 7fa812000640 1 -- 172.21.15.39:0/2094835979 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80400eb00 con 0x7fa81c1531d0 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.887+0000 7fa812000640 1 -- 172.21.15.39:0/2094835979 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80400ec60 con 0x7fa81c1531d0 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.887+0000 7fa812000640 1 -- 172.21.15.39:0/2094835979 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80401c400 con 0x7fa81c1531d0 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.887+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c0c3450 con 0x7fa81c1531d0 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.887+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c06fe90 con 0x7fa81c1531d0 2024-01-24T17:49:28.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.888+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c066f70 con 0x7fa81c1531d0 2024-01-24T17:49:28.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.889+0000 7fa812000640 1 -- 172.21.15.39:0/2094835979 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa80401c560 con 0x7fa81c1531d0 2024-01-24T17:49:28.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.890+0000 7fa812000640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa7f406a420 0x7fa7f406c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:28.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.890+0000 7fa812000640 1 -- 172.21.15.39:0/2094835979 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fa804088f30 con 0x7fa81c1531d0 2024-01-24T17:49:28.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.891+0000 7fa819e00640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa7f406a420 0x7fa7f406c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:28.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.892+0000 7fa819e00640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa7f406a420 0x7fa7f406c8e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa80c034000 tx=0x7fa80c007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:28.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:28.894+0000 7fa812000640 1 -- 172.21.15.39:0/2094835979 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa804054340 con 0x7fa81c1531d0 2024-01-24T17:49:29.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.088+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa81c152400 con 0x7fa81c1531d0 2024-01-24T17:49:29.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.091+0000 7fa812000640 1 -- 172.21.15.39:0/2094835979 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16529 (secure 0 0 0) 0x7fa804057960 con 0x7fa81c1531d0 2024-01-24T17:49:29.092 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:29.092 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":106,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:49:17.472389+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.0","mappings":[{"from":4,"to":0}]},{"pgid":"14.18","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6833/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.39:6832/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.100:6836/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.100:6834/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.39:6835/3495904204":"2024-01-25T17:48:58.377999+0000","172.21.15.39:6834/3495904204":"2024-01-25T17:48:58.377999+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:49:29.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa7f406a420 msgr2=0x7fa7f406c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa7f406a420 0x7fa7f406c8e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa80c034000 tx=0x7fa80c007000 comp rx=0 tx=0).stop 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa81c1531d0 msgr2=0x7fa81c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa81c1531d0 0x7fa81c0712c0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fa804010890 tx=0x7fa804010d50 comp rx=0 tx=0).stop 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 shutdown_connections 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa81c0cae40 0x7fa81c0cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa7f406a420 0x7fa7f406c8e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa81c1531d0 0x7fa81c0712c0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 --2- 172.21.15.39:0/2094835979 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa7fc003970 0x7fa81c14a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 >> 172.21.15.39:0/2094835979 conn(0x7fa81c06bf30 msgr2=0x7fa81c06c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 shutdown_connections 2024-01-24T17:49:29.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.094+0000 7fa81b200640 1 -- 172.21.15.39:0/2094835979 wait complete. 2024-01-24T17:49:29.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:49:29.113 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:49:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.336+0000 7ff686400640 1 Processor -- start 2024-01-24T17:49:29.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.336+0000 7ff686400640 1 -- start start 2024-01-24T17:49:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff686400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff68015a840 0x7ff68015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff686400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6800cea50 0x7ff6800caee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff686400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6800ca770 con 0x7ff68006fe90 2024-01-24T17:49:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff686400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68015e9a0 con 0x7ff68015a840 2024-01-24T17:49:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff686400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68015ec40 con 0x7ff6800cea50 2024-01-24T17:49:29.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff67fe00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff68006fe90 0x7ff6800ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:47560/0 (socket says 172.21.15.39:47560) 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff67fe00640 1 -- 172.21.15.39:0/98956293 learned_addr learned my addr 172.21.15.39:0/98956293 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff685000640 1 --2- 172.21.15.39:0/98956293 >> v2:172.21.15.100:3300/0 conn(0x7ff68015a840 0x7ff68015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.338+0000 7ff685a00640 1 --2- 172.21.15.39:0/98956293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6800cea50 0x7ff6800caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.339+0000 7ff67f400640 1 -- 172.21.15.39:0/98956293 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480385386 0 0) 0x7ff6800ca770 con 0x7ff68006fe90 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.339+0000 7ff67f400640 1 -- 172.21.15.39:0/98956293 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff660003520 con 0x7ff68006fe90 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.339+0000 7ff685000640 1 -- 172.21.15.39:0/98956293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6800cea50 msgr2=0x7ff6800caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.339+0000 7ff685000640 1 --2- 172.21.15.39:0/98956293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6800cea50 0x7ff6800caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.339+0000 7ff685000640 1 -- 172.21.15.39:0/98956293 >> v1:172.21.15.39:6789/0 conn(0x7ff68006fe90 legacy=0x7ff6800ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.339+0000 7ff685000640 1 -- 172.21.15.39:0/98956293 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68015ef20 con 0x7ff68015a840 2024-01-24T17:49:29.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.341+0000 7ff685000640 1 --2- 172.21.15.39:0/98956293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff68015a840 0x7ff68015e460 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff6700097a0 tx=0x7ff67002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=169c03725edcef71 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:29.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.341+0000 7ff67f400640 1 -- 172.21.15.39:0/98956293 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67002d040 con 0x7ff68015a840 2024-01-24T17:49:29.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.341+0000 7ff67f400640 1 -- 172.21.15.39:0/98956293 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff670031810 con 0x7ff68015a840 2024-01-24T17:49:29.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.341+0000 7ff67f400640 1 -- 172.21.15.39:0/98956293 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670031b80 con 0x7ff68015a840 2024-01-24T17:49:29.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.342+0000 7ff686400640 1 -- 172.21.15.39:0/98956293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff68015a840 msgr2=0x7ff68015e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.342+0000 7ff686400640 1 --2- 172.21.15.39:0/98956293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff68015a840 0x7ff68015e460 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff6700097a0 tx=0x7ff67002b8b0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.342+0000 7ff686400640 1 -- 172.21.15.39:0/98956293 shutdown_connections 2024-01-24T17:49:29.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.342+0000 7ff686400640 1 --2- 172.21.15.39:0/98956293 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff6800cea50 0x7ff6800caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.342+0000 7ff686400640 1 --2- 172.21.15.39:0/98956293 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff68015a840 0x7ff68015e460 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.342+0000 7ff686400640 1 -- 172.21.15.39:0/98956293 >> 172.21.15.39:0/98956293 conn(0x7ff68006a220 msgr2=0x7ff68006bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:29.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.342+0000 7ff686400640 1 -- 172.21.15.39:0/98956293 shutdown_connections 2024-01-24T17:49:29.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.342+0000 7ff686400640 1 -- 172.21.15.39:0/98956293 wait complete. 2024-01-24T17:49:29.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.344+0000 7ff686400640 1 Processor -- start 2024-01-24T17:49:29.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.344+0000 7ff686400640 1 -- start start 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.345+0000 7ff686400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff68006fe90 0x7ff6800cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.345+0000 7ff686400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6800cea50 0x7ff6800c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.345+0000 7ff686400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff68015a840 0x7ff6801f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.345+0000 7ff686400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801f6320 con 0x7ff68006fe90 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.345+0000 7ff686400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6800ccd40 con 0x7ff6800cea50 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.345+0000 7ff686400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6800ab8e0 con 0x7ff68015a840 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff68006fe90 0x7ff6800cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff68006fe90 0x7ff6800cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34648/0 (socket says 172.21.15.39:34648) 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 -- 172.21.15.39:0/1884340679 learned_addr learned my addr 172.21.15.39:0/1884340679 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff67fe00640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6800cea50 0x7ff6800c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685a00640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff68015a840 0x7ff6801f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 -- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff68015a840 msgr2=0x7ff6801f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff68015a840 0x7ff6801f56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 -- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6800cea50 msgr2=0x7ff6800c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6800cea50 0x7ff6800c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 -- 172.21.15.39:0/1884340679 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff670009400 con 0x7ff68006fe90 2024-01-24T17:49:29.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685a00640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff68015a840 0x7ff6801f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:29.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.346+0000 7ff685000640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff68006fe90 0x7ff6800cc800 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7ff67002b8f0 tx=0x7ff6700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.349+0000 7ff67cc00640 1 -- 172.21.15.39:0/1884340679 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670003820 con 0x7ff68006fe90 2024-01-24T17:49:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.349+0000 7ff67cc00640 1 -- 172.21.15.39:0/1884340679 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67003e070 con 0x7ff68006fe90 2024-01-24T17:49:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.349+0000 7ff67cc00640 1 -- 172.21.15.39:0/1884340679 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670003b80 con 0x7ff68006fe90 2024-01-24T17:49:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.349+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6800ac180 con 0x7ff68006fe90 2024-01-24T17:49:29.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.349+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6800abb30 con 0x7ff68006fe90 2024-01-24T17:49:29.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.350+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68004e840 con 0x7ff68006fe90 2024-01-24T17:49:29.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.351+0000 7ff67cc00640 1 -- 172.21.15.39:0/1884340679 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff670003ce0 con 0x7ff68006fe90 2024-01-24T17:49:29.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.352+0000 7ff67cc00640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff65806a460 0x7ff65806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.352+0000 7ff67cc00640 1 -- 172.21.15.39:0/1884340679 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7ff6700aa870 con 0x7ff68006fe90 2024-01-24T17:49:29.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.352+0000 7ff67fe00640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff65806a460 0x7ff65806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.354+0000 7ff67fe00640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff65806a460 0x7ff65806c920 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff6801f53e0 tx=0x7ff668009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:29.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.357+0000 7ff67cc00640 1 -- 172.21.15.39:0/1884340679 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff670075c80 con 0x7ff68006fe90 2024-01-24T17:49:29.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.551+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff68006cc80 con 0x7ff68006fe90 2024-01-24T17:49:29.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.552+0000 7ff67cc00640 1 -- 172.21.15.39:0/1884340679 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff6700792a0 con 0x7ff68006fe90 2024-01-24T17:49:29.552 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:49:29.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff65806a460 msgr2=0x7ff65806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff65806a460 0x7ff65806c920 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff6801f53e0 tx=0x7ff668009040 comp rx=0 tx=0).stop 2024-01-24T17:49:29.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff68006fe90 msgr2=0x7ff6800cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff68006fe90 0x7ff6800cc800 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7ff67002b8f0 tx=0x7ff6700046c0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 shutdown_connections 2024-01-24T17:49:29.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff68015a840 0x7ff6801f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6800cea50 0x7ff6800c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff65806a460 0x7ff65806c920 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 --2- 172.21.15.39:0/1884340679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff68006fe90 0x7ff6800cc800 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.554+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 >> 172.21.15.39:0/1884340679 conn(0x7ff68006a220 msgr2=0x7ff6801594f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:29.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.555+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 shutdown_connections 2024-01-24T17:49:29.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.555+0000 7ff686400640 1 -- 172.21.15.39:0/1884340679 wait complete. 2024-01-24T17:49:29.573 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:49:29.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.793+0000 7f32a3200640 1 Processor -- start 2024-01-24T17:49:29.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.793+0000 7f32a3200640 1 -- start start 2024-01-24T17:49:29.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.795+0000 7f32a3200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f32a40c1860 0x7f32a41654d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.795+0000 7f32a3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f32a40ce3a0 0x7f32a40ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.795+0000 7f32a3200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a4165a10 con 0x7f32a40cb9c0 2024-01-24T17:49:29.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.795+0000 7f32a3200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a41650b0 con 0x7f32a40ce3a0 2024-01-24T17:49:29.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.795+0000 7f32a3200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a40c9a60 con 0x7f32a40c1860 2024-01-24T17:49:29.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.795+0000 7f32a2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f32a40ce3a0 0x7f32a40ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.795+0000 7f32a2800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f32a40ce3a0 0x7f32a40ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34664/0 (socket says 172.21.15.39:34664) 2024-01-24T17:49:29.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.795+0000 7f32a2800640 1 -- 172.21.15.39:0/570705263 learned_addr learned my addr 172.21.15.39:0/570705263 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:29.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.796+0000 7f32a2800640 1 -- 172.21.15.39:0/570705263 >> v2:172.21.15.100:3300/0 conn(0x7f32a40c1860 msgr2=0x7f32a41654d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:29.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.796+0000 7f32a2800640 1 --2- 172.21.15.39:0/570705263 >> v2:172.21.15.100:3300/0 conn(0x7f32a40c1860 0x7f32a41654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.796+0000 7f32a2800640 1 -- 172.21.15.39:0/570705263 >> v1:172.21.15.100:6789/0 conn(0x7f32a40cb9c0 legacy=0x7f32a40c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:29.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.796+0000 7f32a2800640 1 -- 172.21.15.39:0/570705263 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a40c94a0 con 0x7f32a40ce3a0 2024-01-24T17:49:29.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.797+0000 7f32a2800640 1 --2- 172.21.15.39:0/570705263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40ce810 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f32a40cadf0 tx=0x7f329c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e0970afe38a220 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:29.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.797+0000 7f32a1400640 1 -- 172.21.15.39:0/570705263 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c02c960 con 0x7f32a40ce3a0 2024-01-24T17:49:29.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.797+0000 7f32a1400640 1 -- 172.21.15.39:0/570705263 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c02cac0 con 0x7f32a40ce3a0 2024-01-24T17:49:29.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.797+0000 7f32a1400640 1 -- 172.21.15.39:0/570705263 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c02ce50 con 0x7f32a40ce3a0 2024-01-24T17:49:29.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.798+0000 7f32a3200640 1 -- 172.21.15.39:0/570705263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 msgr2=0x7f32a40ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.798+0000 7f32a3200640 1 --2- 172.21.15.39:0/570705263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40ce810 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f32a40cadf0 tx=0x7f329c02bdc0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.798+0000 7f32a3200640 1 -- 172.21.15.39:0/570705263 shutdown_connections 2024-01-24T17:49:29.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.798+0000 7f32a3200640 1 --2- 172.21.15.39:0/570705263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40ce810 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.798+0000 7f32a3200640 1 --2- 172.21.15.39:0/570705263 >> v2:172.21.15.100:3300/0 conn(0x7f32a40c1860 0x7f32a41654d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.798+0000 7f32a3200640 1 -- 172.21.15.39:0/570705263 >> 172.21.15.39:0/570705263 conn(0x7f32a406a220 msgr2=0x7f32a406bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:29.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.798+0000 7f32a3200640 1 -- 172.21.15.39:0/570705263 shutdown_connections 2024-01-24T17:49:29.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.798+0000 7f32a3200640 1 -- 172.21.15.39:0/570705263 wait complete. 2024-01-24T17:49:29.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.800+0000 7f32a3200640 1 Processor -- start 2024-01-24T17:49:29.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.800+0000 7f32a3200640 1 -- start start 2024-01-24T17:49:29.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a3200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a40c1860 0x7f32a415ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a3200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a40cb9c0 0x7f32a41eb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40b19f0 con 0x7f32a40ce3a0 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a3200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a415ef80 con 0x7f32a40c1860 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a3200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a41ec260 con 0x7f32a40cb9c0 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34666/0 (socket says 172.21.15.39:34666) 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.801+0000 7f32a2800640 1 -- 172.21.15.39:0/900056329 learned_addr learned my addr 172.21.15.39:0/900056329 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f329be00640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a40cb9c0 0x7f32a41eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f32a1e00640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a40c1860 0x7f32a415ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f32a2800640 1 -- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a40cb9c0 msgr2=0x7f32a41eb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f32a2800640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a40cb9c0 0x7f32a41eb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f32a2800640 1 -- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a40c1860 msgr2=0x7f32a415ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f32a2800640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a40c1860 0x7f32a415ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:29.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f32a2800640 1 -- 172.21.15.39:0/900056329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c009b30 con 0x7f32a40ce3a0 2024-01-24T17:49:29.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f32a2800640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40afc90 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f329c036040 tx=0x7f329c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:29.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.802+0000 7f329a000640 1 -- 172.21.15.39:0/900056329 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c03e450 con 0x7f32a40ce3a0 2024-01-24T17:49:29.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.803+0000 7f329a000640 1 -- 172.21.15.39:0/900056329 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c03e5d0 con 0x7f32a40ce3a0 2024-01-24T17:49:29.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.803+0000 7f329a000640 1 -- 172.21.15.39:0/900056329 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c03e980 con 0x7f32a40ce3a0 2024-01-24T17:49:29.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.803+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a41ec540 con 0x7f32a40ce3a0 2024-01-24T17:49:29.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.803+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a40b1c20 con 0x7f32a40ce3a0 2024-01-24T17:49:29.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.804+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a404e840 con 0x7f32a40ce3a0 2024-01-24T17:49:29.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.805+0000 7f329a000640 1 -- 172.21.15.39:0/900056329 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f329c03eae0 con 0x7f32a40ce3a0 2024-01-24T17:49:29.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.806+0000 7f329a000640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f327c06a460 0x7f327c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:29.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.806+0000 7f329a000640 1 -- 172.21.15.39:0/900056329 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f329c0ab4e0 con 0x7f32a40ce3a0 2024-01-24T17:49:29.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.806+0000 7f32a1e00640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f327c06a460 0x7f327c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:29.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.807+0000 7f32a1e00640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f327c06a460 0x7f327c06c920 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f32a40b42c0 tx=0x7f3294008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:29.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.810+0000 7f329a000640 1 -- 172.21.15.39:0/900056329 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f329c076980 con 0x7f32a40ce3a0 2024-01-24T17:49:30.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:29.999+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f32a4165a10 con 0x7f32a40ce3a0 2024-01-24T17:49:30.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.000+0000 7f329a000640 1 -- 172.21.15.39:0/900056329 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f32a4165a10 con 0x7f32a40ce3a0 2024-01-24T17:49:30.001 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:49:30.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.002+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f327c06a460 msgr2=0x7f327c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.002+0000 7f32a3200640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f327c06a460 0x7f327c06c920 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f32a40b42c0 tx=0x7f3294008040 comp rx=0 tx=0).stop 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.002+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 msgr2=0x7f32a40afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.002+0000 7f32a3200640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40afc90 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f329c036040 tx=0x7f329c004a40 comp rx=0 tx=0).stop 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.002+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 shutdown_connections 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.002+0000 7f32a3200640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f327c06a460 0x7f327c06c920 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.003+0000 7f32a3200640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f32a40ce3a0 0x7f32a40afc90 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.003+0000 7f32a3200640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f32a40cb9c0 0x7f32a41eb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.003+0000 7f32a3200640 1 --2- 172.21.15.39:0/900056329 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f32a40c1860 0x7f32a415ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.003+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 >> 172.21.15.39:0/900056329 conn(0x7f32a406a220 msgr2=0x7f32a40c7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.003+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 shutdown_connections 2024-01-24T17:49:30.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.003+0000 7f32a3200640 1 -- 172.21.15.39:0/900056329 wait complete. 2024-01-24T17:49:30.021 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:49:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.234+0000 7f9d0be00640 1 Processor -- start 2024-01-24T17:49:30.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.234+0000 7f9d0be00640 1 -- start start 2024-01-24T17:49:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.235+0000 7f9d0be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9d0c0b35a0 0x7f9d0c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.235+0000 7f9d0be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0d1430 con 0x7f9d0c15ce90 2024-01-24T17:49:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.235+0000 7f9d0be00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0ce830 con 0x7f9d0c15cab0 2024-01-24T17:49:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.235+0000 7f9d0be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0c0d01e0 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.236+0000 7f9d0b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9d0c0b35a0 0x7f9d0c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.236+0000 7f9d0b400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f9d0c0b35a0 0x7f9d0c0ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34678/0 (socket says 172.21.15.39:34678) 2024-01-24T17:49:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.236+0000 7f9d0b400640 1 -- 172.21.15.39:0/2929872828 learned_addr learned my addr 172.21.15.39:0/2929872828 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.236+0000 7f9d0b400640 1 -- 172.21.15.39:0/2929872828 >> v1:172.21.15.100:6789/0 conn(0x7f9d0c15cab0 legacy=0x7f9d0c0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.236+0000 7f9d0b400640 1 -- 172.21.15.39:0/2929872828 >> v1:172.21.15.39:6789/0 conn(0x7f9d0c15ce90 legacy=0x7f9d0c0d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.236+0000 7f9d0b400640 1 -- 172.21.15.39:0/2929872828 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c0ceb10 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.237+0000 7f9d0b400640 1 --2- 172.21.15.39:0/2929872828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0ce130 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009e50 tx=0x7f9d0402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13e730b50ef38e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:30.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.238+0000 7f9d09600640 1 -- 172.21.15.39:0/2929872828 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0402cb10 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.238+0000 7f9d09600640 1 -- 172.21.15.39:0/2929872828 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0402cc70 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.238+0000 7f9d09600640 1 -- 172.21.15.39:0/2929872828 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d040343f0 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.238+0000 7f9d0be00640 1 -- 172.21.15.39:0/2929872828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 msgr2=0x7f9d0c0ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.238+0000 7f9d0be00640 1 --2- 172.21.15.39:0/2929872828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0ce130 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009e50 tx=0x7f9d0402bf70 comp rx=0 tx=0).stop 2024-01-24T17:49:30.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.238+0000 7f9d0be00640 1 -- 172.21.15.39:0/2929872828 shutdown_connections 2024-01-24T17:49:30.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.238+0000 7f9d0be00640 1 --2- 172.21.15.39:0/2929872828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0ce130 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.238+0000 7f9d0be00640 1 -- 172.21.15.39:0/2929872828 >> 172.21.15.39:0/2929872828 conn(0x7f9d0c0d2830 msgr2=0x7f9d0c0d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:30.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.239+0000 7f9d0be00640 1 -- 172.21.15.39:0/2929872828 shutdown_connections 2024-01-24T17:49:30.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.239+0000 7f9d0be00640 1 -- 172.21.15.39:0/2929872828 wait complete. 2024-01-24T17:49:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.241+0000 7f9d0be00640 1 Processor -- start 2024-01-24T17:49:30.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.241+0000 7f9d0be00640 1 -- start start 2024-01-24T17:49:30.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d0c15cab0 0x7f9d0c15d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d0c0b5450 0x7f9d0c15df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c0cbbc0 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c0b59c0 con 0x7f9d0c0b5450 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c0cb500 con 0x7f9d0c15cab0 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34692/0 (socket says 172.21.15.39:34692) 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.242+0000 7f9d0aa00640 1 -- 172.21.15.39:0/1889258684 learned_addr learned my addr 172.21.15.39:0/1889258684 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0a000640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d0c15cab0 0x7f9d0c15d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0aa00640 1 -- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d0c15cab0 msgr2=0x7f9d0c15d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0aa00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d0c15cab0 0x7f9d0c15d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0b400640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d0c0b5450 0x7f9d0c15df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0aa00640 1 -- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d0c0b5450 msgr2=0x7f9d0c15df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0aa00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d0c0b5450 0x7f9d0c15df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0aa00640 1 -- 172.21.15.39:0/1889258684 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d04009b30 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0a000640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d0c15cab0 0x7f9d0c15d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:30.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d0aa00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0cbfe0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc004640 tx=0x7f9cfc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:30.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d03400640 1 -- 172.21.15.39:0/1889258684 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc011790 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.243+0000 7f9d03400640 1 -- 172.21.15.39:0/1889258684 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc0118f0 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.244+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c0cb7e0 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.244+0000 7f9d03400640 1 -- 172.21.15.39:0/1889258684 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc011ca0 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.244+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c15e490 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.245+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c0cb360 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.246+0000 7f9d03400640 1 -- 172.21.15.39:0/1889258684 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9cfc01c460 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.247+0000 7f9d03400640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9ce406a430 0x7f9ce406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.247+0000 7f9d03400640 1 -- 172.21.15.39:0/1889258684 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f9cfc088950 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.248+0000 7f9d0a000640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9ce406a430 0x7f9ce406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.249+0000 7f9d0a000640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9ce406a430 0x7f9ce406c8f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0cd240 tx=0x7f9cf4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:30.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.252+0000 7f9d03400640 1 -- 172.21.15.39:0/1889258684 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f9d0c0cb360 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.446+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9d0c0cb360 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.447+0000 7f9d03400640 1 -- 172.21.15.39:0/1889258684 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9d0c0cb360 con 0x7f9d0c0b35a0 2024-01-24T17:49:30.447 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:49:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.449+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9ce406a430 msgr2=0x7f9ce406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.449+0000 7f9d0be00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9ce406a430 0x7f9ce406c8f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0cd240 tx=0x7f9cf4009040 comp rx=0 tx=0).stop 2024-01-24T17:49:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.449+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 msgr2=0x7f9d0c0cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.449+0000 7f9d0be00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0cbfe0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc004640 tx=0x7f9cfc00c8e0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.450+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 shutdown_connections 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.450+0000 7f9d0be00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9d0c0b5450 0x7f9d0c15df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.450+0000 7f9d0be00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9d0c15cab0 0x7f9d0c15d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.450+0000 7f9d0be00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f9ce406a430 0x7f9ce406c8f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.450+0000 7f9d0be00640 1 --2- 172.21.15.39:0/1889258684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d0c0b35a0 0x7f9d0c0cbfe0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.450+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 >> 172.21.15.39:0/1889258684 conn(0x7f9d0c0d2830 msgr2=0x7f9d0c0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.450+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 shutdown_connections 2024-01-24T17:49:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.450+0000 7f9d0be00640 1 -- 172.21.15.39:0/1889258684 wait complete. 2024-01-24T17:49:30.471 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:49:30.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.694+0000 7f7111c00640 1 Processor -- start 2024-01-24T17:49:30.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.694+0000 7f7111c00640 1 -- start start 2024-01-24T17:49:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.696+0000 7f7111c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f710c144080 0x7f710c0ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.696+0000 7f7111c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c15f590 con 0x7f710c144460 2024-01-24T17:49:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.696+0000 7f7111c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c15e7d0 con 0x7f710c0c4080 2024-01-24T17:49:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.696+0000 7f7111c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f710c15eab0 con 0x7f710c144080 2024-01-24T17:49:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.696+0000 7f710aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f710c144080 0x7f710c0ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.696+0000 7f710aa00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f710c144080 0x7f710c0ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34706/0 (socket says 172.21.15.39:34706) 2024-01-24T17:49:30.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.696+0000 7f710aa00640 1 -- 172.21.15.39:0/1301509286 learned_addr learned my addr 172.21.15.39:0/1301509286 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.697+0000 7f710aa00640 1 -- 172.21.15.39:0/1301509286 >> v1:172.21.15.100:6789/0 conn(0x7f710c0c4080 legacy=0x7f710c0b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.697+0000 7f710aa00640 1 -- 172.21.15.39:0/1301509286 >> v1:172.21.15.39:6789/0 conn(0x7f710c144460 legacy=0x7f710c0c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.697+0000 7f710aa00640 1 -- 172.21.15.39:0/1301509286 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c15ed90 con 0x7f710c144080 2024-01-24T17:49:30.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.698+0000 7f710aa00640 1 --2- 172.21.15.39:0/1301509286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c144080 0x7f710c0ce6e0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009870 tx=0x7f70fc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9a11391b5bd34d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:30.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.698+0000 7f710a000640 1 -- 172.21.15.39:0/1301509286 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc031860 con 0x7f710c144080 2024-01-24T17:49:30.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.698+0000 7f710a000640 1 -- 172.21.15.39:0/1301509286 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc0319c0 con 0x7f710c144080 2024-01-24T17:49:30.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.698+0000 7f710a000640 1 -- 172.21.15.39:0/1301509286 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc031d50 con 0x7f710c144080 2024-01-24T17:49:30.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.699+0000 7f7111c00640 1 -- 172.21.15.39:0/1301509286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c144080 msgr2=0x7f710c0ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.699+0000 7f7111c00640 1 --2- 172.21.15.39:0/1301509286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c144080 0x7f710c0ce6e0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009870 tx=0x7f70fc02b900 comp rx=0 tx=0).stop 2024-01-24T17:49:30.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.699+0000 7f7111c00640 1 -- 172.21.15.39:0/1301509286 shutdown_connections 2024-01-24T17:49:30.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.699+0000 7f7111c00640 1 --2- 172.21.15.39:0/1301509286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c144080 0x7f710c0ce6e0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.699+0000 7f7111c00640 1 -- 172.21.15.39:0/1301509286 >> 172.21.15.39:0/1301509286 conn(0x7f710c065ff0 msgr2=0x7f710c069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:30.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.699+0000 7f7111c00640 1 -- 172.21.15.39:0/1301509286 shutdown_connections 2024-01-24T17:49:30.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.699+0000 7f7111c00640 1 -- 172.21.15.39:0/1301509286 wait complete. 2024-01-24T17:49:30.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.701+0000 7f7111c00640 1 Processor -- start 2024-01-24T17:49:30.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.701+0000 7f7111c00640 1 -- start start 2024-01-24T17:49:30.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.702+0000 7f7111c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f710c0c4080 0x7f710c1f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.702+0000 7f7111c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f710c1f62c0 0x7f710c139840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.702+0000 7f7111c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c15b070 0x7f710c15b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.702+0000 7f7111c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c15a770 con 0x7f710c15b070 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.702+0000 7f7111c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c15ba10 con 0x7f710c0c4080 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.702+0000 7f7111c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c1f3e90 con 0x7f710c1f62c0 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.702+0000 7f710b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c15b070 0x7f710c15b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c15b070 0x7f710c15b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34708/0 (socket says 172.21.15.39:34708) 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710b400640 1 -- 172.21.15.39:0/365508457 learned_addr learned my addr 172.21.15.39:0/365508457 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710aa00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f710c0c4080 0x7f710c1f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f7103e00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f710c1f62c0 0x7f710c139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710b400640 1 -- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f710c1f62c0 msgr2=0x7f710c139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710b400640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f710c1f62c0 0x7f710c139840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710b400640 1 -- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f710c0c4080 msgr2=0x7f710c1f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710b400640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f710c0c4080 0x7f710c1f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710b400640 1 -- 172.21.15.39:0/365508457 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc0094d0 con 0x7f710c15b070 2024-01-24T17:49:30.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.703+0000 7f710b400640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c15b070 0x7f710c15b4d0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f710400c780 tx=0x7f710400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.704+0000 7f7102a00640 1 -- 172.21.15.39:0/365508457 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104004b60 con 0x7f710c15b070 2024-01-24T17:49:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.704+0000 7f7102a00640 1 -- 172.21.15.39:0/365508457 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710400ce90 con 0x7f710c15b070 2024-01-24T17:49:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.704+0000 7f7102a00640 1 -- 172.21.15.39:0/365508457 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710400a850 con 0x7f710c15b070 2024-01-24T17:49:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.704+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c1f3c70 con 0x7f710c15b070 2024-01-24T17:49:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.704+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c1f6ad0 con 0x7f710c15b070 2024-01-24T17:49:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.705+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c1f4000 con 0x7f710c15b070 2024-01-24T17:49:30.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.706+0000 7f7102a00640 1 -- 172.21.15.39:0/365508457 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f710400a9b0 con 0x7f710c15b070 2024-01-24T17:49:30.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.707+0000 7f7102a00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70e406a430 0x7f70e406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:30.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.707+0000 7f7102a00640 1 -- 172.21.15.39:0/365508457 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f7104088ef0 con 0x7f710c15b070 2024-01-24T17:49:30.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.708+0000 7f710aa00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70e406a430 0x7f70e406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:30.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.709+0000 7f710aa00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70e406a430 0x7f70e406c8f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f70fc036000 tx=0x7f70fc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:30.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.711+0000 7f7102a00640 1 -- 172.21.15.39:0/365508457 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f71040543a0 con 0x7f710c15b070 2024-01-24T17:49:30.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.920+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f710c073f10 con 0x7f710c15b070 2024-01-24T17:49:30.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.920+0000 7f7102a00640 1 -- 172.21.15.39:0/365508457 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v106) v1 ==== 80+0+366 (secure 0 0 0) 0x7f71040579c0 con 0x7f710c15b070 2024-01-24T17:49:30.921 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6833/872967797 2024-01-25T17:48:58.450631+0000 2024-01-24T17:49:30.921 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6832/872967797 2024-01-25T17:48:58.450631+0000 2024-01-24T17:49:30.921 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.100:6836/3116066709 2024-01-25T17:48:58.404809+0000 2024-01-24T17:49:30.921 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.100:6834/3116066709 2024-01-25T17:48:58.404809+0000 2024-01-24T17:49:30.922 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6835/3495904204 2024-01-25T17:48:58.377999+0000 2024-01-24T17:49:30.922 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6834/3495904204 2024-01-25T17:48:58.377999+0000 2024-01-24T17:49:30.922 INFO:teuthology.orchestra.run.smithi039.stderr:listed 6 entries 2024-01-24T17:49:30.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.923+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70e406a430 msgr2=0x7f70e406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.923+0000 7f7111c00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70e406a430 0x7f70e406c8f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f70fc036000 tx=0x7f70fc03c040 comp rx=0 tx=0).stop 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.923+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c15b070 msgr2=0x7f710c15b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.923+0000 7f7111c00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c15b070 0x7f710c15b4d0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f710400c780 tx=0x7f710400cc40 comp rx=0 tx=0).stop 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.923+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 shutdown_connections 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.923+0000 7f7111c00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f70e406a430 0x7f70e406c8f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.924+0000 7f7111c00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f710c15b070 0x7f710c15b4d0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.924+0000 7f7111c00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f710c1f62c0 0x7f710c139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.924+0000 7f7111c00640 1 --2- 172.21.15.39:0/365508457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f710c0c4080 0x7f710c1f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.924+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 >> 172.21.15.39:0/365508457 conn(0x7f710c065ff0 msgr2=0x7f710c0707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.924+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 shutdown_connections 2024-01-24T17:49:30.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:30.924+0000 7f7111c00640 1 -- 172.21.15.39:0/365508457 wait complete. 2024-01-24T17:49:30.943 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-01-24T17:49:30.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:30.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-24T17:49:31.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi039.stderr:ceph-fuse[72253]: fuse finished with error 0 and tester_r 0 2024-01-24T17:49:31.039 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:49:37.140 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:49:37.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:37.141 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:49:37.200 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:49:37.200 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:37.258 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:49:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.481+0000 7fd3a2a00640 1 Processor -- start 2024-01-24T17:49:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.481+0000 7fd3a2a00640 1 -- start start 2024-01-24T17:49:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.483+0000 7fd3a2a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd39c15cab0 0x7fd39c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.483+0000 7fd3a2a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd39c0d0d20 0x7fd39c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.483+0000 7fd3a2a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39c0ce830 con 0x7fd39c15ce90 2024-01-24T17:49:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.483+0000 7fd3a2a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39c0d01e0 con 0x7fd39c15cab0 2024-01-24T17:49:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.483+0000 7fd3a2a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39c0ceb10 con 0x7fd39c0d0d20 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd39c15cab0 0x7fd39c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a1600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd39c15cab0 0x7fd39c0ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45346/0 (socket says 172.21.15.39:45346) 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a1600640 1 -- 172.21.15.39:0/664238520 learned_addr learned my addr 172.21.15.39:0/664238520 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a2000640 1 --2- 172.21.15.39:0/664238520 >> v2:172.21.15.100:3300/0 conn(0x7fd39c0d0d20 0x7fd39c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a1600640 1 -- 172.21.15.39:0/664238520 >> v2:172.21.15.100:3300/0 conn(0x7fd39c0d0d20 msgr2=0x7fd39c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a1600640 1 --2- 172.21.15.39:0/664238520 >> v2:172.21.15.100:3300/0 conn(0x7fd39c0d0d20 0x7fd39c0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a1600640 1 -- 172.21.15.39:0/664238520 >> v1:172.21.15.100:6789/0 conn(0x7fd39c15ce90 legacy=0x7fd39c0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a1600640 1 -- 172.21.15.39:0/664238520 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c0cedf0 con 0x7fd39c15cab0 2024-01-24T17:49:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.484+0000 7fd3a2000640 1 --2- 172.21.15.39:0/664238520 >> v2:172.21.15.100:3300/0 conn(0x7fd39c0d0d20 0x7fd39c0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.485+0000 7fd3a1600640 1 --2- 172.21.15.39:0/664238520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c15cab0 0x7fd39c0ce130 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fd39c08d0d0 tx=0x7fd38c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a580e2fecc127b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.485+0000 7fd39be00640 1 -- 172.21.15.39:0/664238520 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c030850 con 0x7fd39c15cab0 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd39be00640 1 -- 172.21.15.39:0/664238520 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38c0309b0 con 0x7fd39c15cab0 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd39be00640 1 -- 172.21.15.39:0/664238520 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c030d40 con 0x7fd39c15cab0 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd3a2a00640 1 -- 172.21.15.39:0/664238520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c15cab0 msgr2=0x7fd39c0ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/664238520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c15cab0 0x7fd39c0ce130 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fd39c08d0d0 tx=0x7fd38c02b990 comp rx=0 tx=0).stop 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd3a2a00640 1 -- 172.21.15.39:0/664238520 shutdown_connections 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/664238520 >> v2:172.21.15.100:3300/0 conn(0x7fd39c0d0d20 0x7fd39c0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/664238520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c15cab0 0x7fd39c0ce130 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd3a2a00640 1 -- 172.21.15.39:0/664238520 >> 172.21.15.39:0/664238520 conn(0x7fd39c0d2830 msgr2=0x7fd39c0d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd3a2a00640 1 -- 172.21.15.39:0/664238520 shutdown_connections 2024-01-24T17:49:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.486+0000 7fd3a2a00640 1 -- 172.21.15.39:0/664238520 wait complete. 2024-01-24T17:49:37.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.488+0000 7fd3a2a00640 1 Processor -- start 2024-01-24T17:49:37.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.488+0000 7fd3a2a00640 1 -- start start 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c0d0d20 0x7fd39c151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a2a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd39c15cab0 0x7fd39c15f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a2a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd39c15ff20 0x7fd39c0cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a2a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c152190 con 0x7fd39c0d0d20 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a2a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c15e940 con 0x7fd39c15ff20 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a2a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c15edd0 con 0x7fd39c15cab0 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c0d0d20 0x7fd39c151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c0d0d20 0x7fd39c151c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45358/0 (socket says 172.21.15.39:45358) 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.489+0000 7fd3a1600640 1 -- 172.21.15.39:0/779319241 learned_addr learned my addr 172.21.15.39:0/779319241 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a0c00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd39c15cab0 0x7fd39c15f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a1600640 1 -- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd39c15cab0 msgr2=0x7fd39c15f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a1600640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd39c15cab0 0x7fd39c15f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a2000640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd39c15ff20 0x7fd39c0cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a1600640 1 -- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd39c15ff20 msgr2=0x7fd39c0cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a1600640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd39c15ff20 0x7fd39c0cb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a1600640 1 -- 172.21.15.39:0/779319241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c009470 con 0x7fd39c0d0d20 2024-01-24T17:49:37.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a0c00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd39c15cab0 0x7fd39c15f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:37.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.490+0000 7fd3a1600640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c0d0d20 0x7fd39c151c50 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd38c000c00 tx=0x7fd38c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:37.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.491+0000 7fd399600640 1 -- 172.21.15.39:0/779319241 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c030820 con 0x7fd39c0d0d20 2024-01-24T17:49:37.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.491+0000 7fd399600640 1 -- 172.21.15.39:0/779319241 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38c043400 con 0x7fd39c0d0d20 2024-01-24T17:49:37.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.491+0000 7fd399600640 1 -- 172.21.15.39:0/779319241 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c0437b0 con 0x7fd39c0d0d20 2024-01-24T17:49:37.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.491+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39c15dd30 con 0x7fd39c0d0d20 2024-01-24T17:49:37.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.491+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39c15e2f0 con 0x7fd39c0d0d20 2024-01-24T17:49:37.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.492+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39c04e840 con 0x7fd39c0d0d20 2024-01-24T17:49:37.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.493+0000 7fd399600640 1 -- 172.21.15.39:0/779319241 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd38c02e070 con 0x7fd39c0d0d20 2024-01-24T17:49:37.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.494+0000 7fd399600640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd37406a430 0x7fd37406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.494+0000 7fd399600640 1 -- 172.21.15.39:0/779319241 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fd38c03e070 con 0x7fd39c0d0d20 2024-01-24T17:49:37.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.494+0000 7fd3a0c00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd37406a430 0x7fd37406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.495+0000 7fd3a0c00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd37406a430 0x7fd37406c8f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0b1420 tx=0x7fd384008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:37.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.498+0000 7fd399600640 1 -- 172.21.15.39:0/779319241 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd38c076fe0 con 0x7fd39c0d0d20 2024-01-24T17:49:37.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.693+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd39c0d2e20 con 0x7fd39c0d0d20 2024-01-24T17:49:37.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.694+0000 7fd399600640 1 -- 172.21.15.39:0/779319241 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16529 (secure 0 0 0) 0x7fd38c034090 con 0x7fd39c0d0d20 2024-01-24T17:49:37.696 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:37.696 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":106,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:49:17.472389+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.0","mappings":[{"from":4,"to":0}]},{"pgid":"14.18","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6833/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.39:6832/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.100:6836/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.100:6834/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.39:6835/3495904204":"2024-01-25T17:48:58.377999+0000","172.21.15.39:6834/3495904204":"2024-01-25T17:48:58.377999+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd37406a430 msgr2=0x7fd37406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd37406a430 0x7fd37406c8f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0b1420 tx=0x7fd384008040 comp rx=0 tx=0).stop 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c0d0d20 msgr2=0x7fd39c151c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c0d0d20 0x7fd39c151c50 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd38c000c00 tx=0x7fd38c004730 comp rx=0 tx=0).stop 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 shutdown_connections 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd39c15ff20 0x7fd39c0cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd39c15cab0 0x7fd39c15f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd37406a430 0x7fd37406c8f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 --2- 172.21.15.39:0/779319241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd39c0d0d20 0x7fd39c151c50 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 >> 172.21.15.39:0/779319241 conn(0x7fd39c0d2830 msgr2=0x7fd39c0cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 shutdown_connections 2024-01-24T17:49:37.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.697+0000 7fd3a2a00640 1 -- 172.21.15.39:0/779319241 wait complete. 2024-01-24T17:49:37.719 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:49:37.720 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:49:37.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.934+0000 7f800ea00640 1 Processor -- start 2024-01-24T17:49:37.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.935+0000 7f800ea00640 1 -- start start 2024-01-24T17:49:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.936+0000 7f800ea00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f80080cfa20 0x7f80080d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.936+0000 7f800ea00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080d1860 con 0x7f800815ca10 2024-01-24T17:49:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.936+0000 7f800ea00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080ce5b0 con 0x7f800815cdf0 2024-01-24T17:49:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.936+0000 7f800ea00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80080d0670 con 0x7f80080cfa20 2024-01-24T17:49:37.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f800d600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f800815ca10 0x7f80080d1150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58064/0 (socket says 172.21.15.39:58064) 2024-01-24T17:49:37.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f800d600640 1 -- 172.21.15.39:0/1198526575 learned_addr learned my addr 172.21.15.39:0/1198526575 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:37.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f800e000640 1 --2- 172.21.15.39:0/1198526575 >> v2:172.21.15.100:3300/0 conn(0x7f80080cfa20 0x7f80080d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158785706 0 0) 0x7f80080d1860 con 0x7f800815ca10 2024-01-24T17:49:37.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8003520 con 0x7f800815ca10 2024-01-24T17:49:37.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157637897 0 0) 0x7f7fe8003520 con 0x7f800815ca10 2024-01-24T17:49:37.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 >> v2:172.21.15.100:3300/0 conn(0x7f80080cfa20 msgr2=0x7f80080d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:37.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f8007e00640 1 --2- 172.21.15.39:0/1198526575 >> v2:172.21.15.100:3300/0 conn(0x7f80080cfa20 0x7f80080d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 >> v1:172.21.15.100:6789/0 conn(0x7f800815cdf0 legacy=0x7f80080cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080d0950 con 0x7f800815ca10 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.937+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7ff8001690 con 0x7f800815ca10 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 >> v1:172.21.15.39:6789/0 conn(0x7f800815ca10 legacy=0x7f80080d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f8007e00640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe80039c0 0x7f7fe8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f8007e00640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fe80047c0 0x7f7fe8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f8007e00640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe8005180 0x7f7fe800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080d0670 con 0x7f7fe80039c0 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe800c080 con 0x7f7fe80047c0 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe800c240 con 0x7f7fe8005180 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f800d600640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe80039c0 0x7f7fe8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f800d600640 1 -- 172.21.15.39:0/1198526575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe8005180 msgr2=0x7f7fe800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:37.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f800d600640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe8005180 0x7f7fe800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f800d600640 1 -- 172.21.15.39:0/1198526575 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fe80047c0 msgr2=0x7f7fe8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:37.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f800d600640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7fe80047c0 0x7f7fe8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f800d600640 1 -- 172.21.15.39:0/1198526575 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8003520 con 0x7f7fe80039c0 2024-01-24T17:49:37.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.938+0000 7f800e000640 1 -- 172.21.15.39:0/1198526575 reap_dead start 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.939+0000 7f800d600640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe80039c0 0x7f7fe8003de0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800c160 tx=0x7f7fe80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb781a874667c61 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.940+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8005ad0 con 0x7f7fe80039c0 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.940+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff8005c30 con 0x7f7fe80039c0 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.940+0000 7f8007e00640 1 -- 172.21.15.39:0/1198526575 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff802c790 con 0x7f7fe80039c0 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.940+0000 7f800ea00640 1 -- 172.21.15.39:0/1198526575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe80039c0 msgr2=0x7f7fe8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.940+0000 7f800ea00640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe80039c0 0x7f7fe8003de0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800c160 tx=0x7f7fe80057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.940+0000 7f800ea00640 1 -- 172.21.15.39:0/1198526575 shutdown_connections 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.940+0000 7f800ea00640 1 --2- 172.21.15.39:0/1198526575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fe80039c0 0x7f7fe8003de0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.940+0000 7f800ea00640 1 -- 172.21.15.39:0/1198526575 >> 172.21.15.39:0/1198526575 conn(0x7f80080d2920 msgr2=0x7f80080d2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:37.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.941+0000 7f800ea00640 1 -- 172.21.15.39:0/1198526575 shutdown_connections 2024-01-24T17:49:37.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.941+0000 7f800ea00640 1 -- 172.21.15.39:0/1198526575 wait complete. 2024-01-24T17:49:37.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.942+0000 7f800ea00640 1 Processor -- start 2024-01-24T17:49:37.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.942+0000 7f800ea00640 1 -- start start 2024-01-24T17:49:37.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.943+0000 7f800ea00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe80039c0 0x7f80081f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.943+0000 7f800ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80080b85c0 0x7f8008150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.943+0000 7f800ea00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80080cbbc0 0x7f80080cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.943+0000 7f800ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080cc590 con 0x7f80080b85c0 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.943+0000 7f800ea00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80081f6560 con 0x7f80080cbbc0 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.943+0000 7f800ea00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80080c6450 con 0x7f7fe80039c0 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80080b85c0 0x7f8008150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80080b85c0 0x7f8008150c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45370/0 (socket says 172.21.15.39:45370) 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800cc00640 1 -- 172.21.15.39:0/3291907544 learned_addr learned my addr 172.21.15.39:0/3291907544 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800d600640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe80039c0 0x7f80081f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800e000640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80080cbbc0 0x7f80080cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800cc00640 1 -- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe80039c0 msgr2=0x7f80081f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800cc00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe80039c0 0x7f80081f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800cc00640 1 -- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80080cbbc0 msgr2=0x7f80080cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800cc00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80080cbbc0 0x7f80080cc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800cc00640 1 -- 172.21.15.39:0/3291907544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080c6730 con 0x7f80080b85c0 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800e000640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80080cbbc0 0x7f80080cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:37.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.944+0000 7f800d600640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe80039c0 0x7f80081f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.945+0000 7f800cc00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80080b85c0 0x7f8008150c50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000c500 tx=0x7f7ff000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.945+0000 7f8005600640 1 -- 172.21.15.39:0/3291907544 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00052d0 con 0x7f80080b85c0 2024-01-24T17:49:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.945+0000 7f8005600640 1 -- 172.21.15.39:0/3291907544 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff0005430 con 0x7f80080b85c0 2024-01-24T17:49:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.945+0000 7f8005600640 1 -- 172.21.15.39:0/3291907544 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0012870 con 0x7f80080b85c0 2024-01-24T17:49:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.945+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80080c6a10 con 0x7f80080b85c0 2024-01-24T17:49:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.945+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80080c6fd0 con 0x7f80080b85c0 2024-01-24T17:49:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.947+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800804e840 con 0x7f80080b85c0 2024-01-24T17:49:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.947+0000 7f8005600640 1 -- 172.21.15.39:0/3291907544 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7ff00129d0 con 0x7f80080b85c0 2024-01-24T17:49:37.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.948+0000 7f8005600640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fe006a430 0x7f7fe006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:37.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.949+0000 7f8005600640 1 -- 172.21.15.39:0/3291907544 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f7ff0088610 con 0x7f80080b85c0 2024-01-24T17:49:37.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.949+0000 7f800d600640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fe006a430 0x7f7fe006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:37.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.950+0000 7f800d600640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fe006a430 0x7f7fe006c8f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8034000 tx=0x7f7ff8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:37.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:37.953+0000 7f8005600640 1 -- 172.21.15.39:0/3291907544 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7ff0053b50 con 0x7f80080b85c0 2024-01-24T17:49:38.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.153+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f800806a6b0 con 0x7f80080b85c0 2024-01-24T17:49:38.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.154+0000 7f8005600640 1 -- 172.21.15.39:0/3291907544 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ff0057170 con 0x7f80080b85c0 2024-01-24T17:49:38.156 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:49:38.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fe006a430 msgr2=0x7f7fe006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fe006a430 0x7f7fe006c8f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8034000 tx=0x7f7ff8009030 comp rx=0 tx=0).stop 2024-01-24T17:49:38.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80080b85c0 msgr2=0x7f8008150c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80080b85c0 0x7f8008150c50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000c500 tx=0x7f7ff000c9c0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 shutdown_connections 2024-01-24T17:49:38.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f80080cbbc0 0x7f80080cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7fe006a430 0x7f7fe006c8f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80080b85c0 0x7f8008150c50 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 --2- 172.21.15.39:0/3291907544 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7fe80039c0 0x7f80081f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.157+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 >> 172.21.15.39:0/3291907544 conn(0x7f80080d2920 msgr2=0x7f800815d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:38.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.158+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 shutdown_connections 2024-01-24T17:49:38.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.158+0000 7f800ea00640 1 -- 172.21.15.39:0/3291907544 wait complete. 2024-01-24T17:49:38.180 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:49:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.409+0000 7f0f47200640 1 Processor -- start 2024-01-24T17:49:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.409+0000 7f0f47200640 1 -- start start 2024-01-24T17:49:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.411+0000 7f0f47200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0f4015cd90 0x7f0f400b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.411+0000 7f0f47200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f400b6380 con 0x7f0f400cd320 2024-01-24T17:49:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.411+0000 7f0f47200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f400d0cb0 con 0x7f0f400cf820 2024-01-24T17:49:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.411+0000 7f0f47200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f400cdd70 con 0x7f0f4015cd90 2024-01-24T17:49:38.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.412+0000 7f0f3fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0f4015cd90 0x7f0f400b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.412+0000 7f0f3fe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0f4015cd90 0x7f0f400b5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45386/0 (socket says 172.21.15.39:45386) 2024-01-24T17:49:38.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.412+0000 7f0f3fe00640 1 -- 172.21.15.39:0/1867595093 learned_addr learned my addr 172.21.15.39:0/1867595093 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:38.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.412+0000 7f0f3fe00640 1 -- 172.21.15.39:0/1867595093 >> v1:172.21.15.100:6789/0 conn(0x7f0f400cf820 legacy=0x7f0f400d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.412+0000 7f0f3fe00640 1 -- 172.21.15.39:0/1867595093 >> v1:172.21.15.39:6789/0 conn(0x7f0f400cd320 legacy=0x7f0f400cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.412+0000 7f0f3fe00640 1 -- 172.21.15.39:0/1867595093 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f400ce050 con 0x7f0f4015cd90 2024-01-24T17:49:38.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.413+0000 7f0f3fe00640 1 --2- 172.21.15.39:0/1867595093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f4015cd90 0x7f0f400b5e40 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009870 tx=0x7f0f3002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe72f0905e6de38 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:38.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.414+0000 7f0f3ea00640 1 -- 172.21.15.39:0/1867595093 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3002f850 con 0x7f0f4015cd90 2024-01-24T17:49:38.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.414+0000 7f0f3ea00640 1 -- 172.21.15.39:0/1867595093 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3002f9b0 con 0x7f0f4015cd90 2024-01-24T17:49:38.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.414+0000 7f0f3ea00640 1 -- 172.21.15.39:0/1867595093 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3002fd40 con 0x7f0f4015cd90 2024-01-24T17:49:38.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.414+0000 7f0f47200640 1 -- 172.21.15.39:0/1867595093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f4015cd90 msgr2=0x7f0f400b5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.414+0000 7f0f47200640 1 --2- 172.21.15.39:0/1867595093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f4015cd90 0x7f0f400b5e40 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009870 tx=0x7f0f3002ba30 comp rx=0 tx=0).stop 2024-01-24T17:49:38.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.414+0000 7f0f47200640 1 -- 172.21.15.39:0/1867595093 shutdown_connections 2024-01-24T17:49:38.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.414+0000 7f0f47200640 1 --2- 172.21.15.39:0/1867595093 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f4015cd90 0x7f0f400b5e40 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.415+0000 7f0f47200640 1 -- 172.21.15.39:0/1867595093 >> 172.21.15.39:0/1867595093 conn(0x7f0f4006af00 msgr2=0x7f0f4006b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:38.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.415+0000 7f0f47200640 1 -- 172.21.15.39:0/1867595093 shutdown_connections 2024-01-24T17:49:38.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.415+0000 7f0f47200640 1 -- 172.21.15.39:0/1867595093 wait complete. 2024-01-24T17:49:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.416+0000 7f0f47200640 1 Processor -- start 2024-01-24T17:49:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.416+0000 7f0f47200640 1 -- start start 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.417+0000 7f0f47200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f400cd320 0x7f0f401f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.417+0000 7f0f47200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f400cf820 0x7f0f4014d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.417+0000 7f0f47200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f4015cd90 0x7f0f400c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.417+0000 7f0f47200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4014d540 con 0x7f0f400cf820 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.417+0000 7f0f47200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4015d200 con 0x7f0f400cd320 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.417+0000 7f0f47200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4015d4a0 con 0x7f0f4015cd90 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f400cf820 0x7f0f4014d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f400cf820 0x7f0f4014d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45400/0 (socket says 172.21.15.39:45400) 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3f400640 1 -- 172.21.15.39:0/226623599 learned_addr learned my addr 172.21.15.39:0/226623599 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3fe00640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f400cd320 0x7f0f401f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3f400640 1 -- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f4015cd90 msgr2=0x7f0f400c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f45200640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f4015cd90 0x7f0f400c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3f400640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f4015cd90 0x7f0f400c8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3f400640 1 -- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f400cd320 msgr2=0x7f0f401f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3f400640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f400cd320 0x7f0f401f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.418+0000 7f0f3f400640 1 -- 172.21.15.39:0/226623599 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f300094d0 con 0x7f0f400cf820 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.419+0000 7f0f3f400640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f400cf820 0x7f0f4014d000 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c450 tx=0x7f0f2800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.419+0000 7f0f37e00640 1 -- 172.21.15.39:0/226623599 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f280052d0 con 0x7f0f400cf820 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.419+0000 7f0f37e00640 1 -- 172.21.15.39:0/226623599 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f28005430 con 0x7f0f400cf820 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.419+0000 7f0f37e00640 1 -- 172.21.15.39:0/226623599 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f28012870 con 0x7f0f400cf820 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.419+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4015d780 con 0x7f0f400cf820 2024-01-24T17:49:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.419+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4015dd40 con 0x7f0f400cf820 2024-01-24T17:49:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.420+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f40073f10 con 0x7f0f400cf820 2024-01-24T17:49:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.421+0000 7f0f37e00640 1 -- 172.21.15.39:0/226623599 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0f280129d0 con 0x7f0f400cf820 2024-01-24T17:49:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.422+0000 7f0f37e00640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f1806a460 0x7f0f1806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.422+0000 7f0f37e00640 1 -- 172.21.15.39:0/226623599 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f0f28088530 con 0x7f0f400cf820 2024-01-24T17:49:38.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.423+0000 7f0f3fe00640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f1806a460 0x7f0f1806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.424+0000 7f0f3fe00640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f1806a460 0x7f0f1806c920 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0f30036000 tx=0x7f0f30008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:38.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.427+0000 7f0f37e00640 1 -- 172.21.15.39:0/226623599 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0f280539d0 con 0x7f0f400cf820 2024-01-24T17:49:38.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.623+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0f400d09f0 con 0x7f0f400cf820 2024-01-24T17:49:38.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.624+0000 7f0f37e00640 1 -- 172.21.15.39:0/226623599 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0f28056ff0 con 0x7f0f400cf820 2024-01-24T17:49:38.626 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:49:38.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f1806a460 msgr2=0x7f0f1806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f1806a460 0x7f0f1806c920 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0f30036000 tx=0x7f0f30008030 comp rx=0 tx=0).stop 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f400cf820 msgr2=0x7f0f4014d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f400cf820 0x7f0f4014d000 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c450 tx=0x7f0f2800c910 comp rx=0 tx=0).stop 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 shutdown_connections 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0f4015cd90 0x7f0f400c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0f1806a460 0x7f0f1806c920 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f400cf820 0x7f0f4014d000 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 --2- 172.21.15.39:0/226623599 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0f400cd320 0x7f0f401f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 >> 172.21.15.39:0/226623599 conn(0x7f0f4006af00 msgr2=0x7f0f400c5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 shutdown_connections 2024-01-24T17:49:38.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.628+0000 7f0f47200640 1 -- 172.21.15.39:0/226623599 wait complete. 2024-01-24T17:49:38.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:49:38.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.870+0000 7ff75d000640 1 Processor -- start 2024-01-24T17:49:38.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.870+0000 7ff75d000640 1 -- start start 2024-01-24T17:49:38.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.871+0000 7ff75d000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff75815a840 0x7ff75815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.871+0000 7ff75d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 0x7ff7580ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.871+0000 7ff75d000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7580b1240 0x7ff7580ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.871+0000 7ff75d000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7580ca770 con 0x7ff7580b1240 2024-01-24T17:49:38.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.871+0000 7ff75d000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff75815e9a0 con 0x7ff75815a840 2024-01-24T17:49:38.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.871+0000 7ff75d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75815ec40 con 0x7ff75806fe90 2024-01-24T17:49:38.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7580b1240 0x7ff7580ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff7580b1240 0x7ff7580ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45412/0 (socket says 172.21.15.39:45412) 2024-01-24T17:49:38.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757e00640 1 -- 172.21.15.39:0/848668194 learned_addr learned my addr 172.21.15.39:0/848668194 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:38.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757e00640 1 -- 172.21.15.39:0/848668194 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 msgr2=0x7ff7580ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:38.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757e00640 1 --2- 172.21.15.39:0/848668194 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 0x7ff7580ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757400640 1 --2- 172.21.15.39:0/848668194 >> v2:172.21.15.100:3300/0 conn(0x7ff75815a840 0x7ff75815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757e00640 1 -- 172.21.15.39:0/848668194 >> v2:172.21.15.100:3300/0 conn(0x7ff75815a840 msgr2=0x7ff75815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:38.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757e00640 1 --2- 172.21.15.39:0/848668194 >> v2:172.21.15.100:3300/0 conn(0x7ff75815a840 0x7ff75815e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.872+0000 7ff757e00640 1 -- 172.21.15.39:0/848668194 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75815ef20 con 0x7ff7580b1240 2024-01-24T17:49:38.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.873+0000 7ff757e00640 1 --2- 172.21.15.39:0/848668194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff7580ce880 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7ff75806c800 tx=0x7ff75002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70512d8081c1a419 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:38.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff756a00640 1 -- 172.21.15.39:0/848668194 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75002ca70 con 0x7ff7580b1240 2024-01-24T17:49:38.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff756a00640 1 -- 172.21.15.39:0/848668194 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75002cbd0 con 0x7ff7580b1240 2024-01-24T17:49:38.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff756a00640 1 -- 172.21.15.39:0/848668194 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7500343f0 con 0x7ff7580b1240 2024-01-24T17:49:38.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff75d000640 1 -- 172.21.15.39:0/848668194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 msgr2=0x7ff7580ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff75d000640 1 --2- 172.21.15.39:0/848668194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff7580ce880 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7ff75806c800 tx=0x7ff75002bed0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff75d000640 1 -- 172.21.15.39:0/848668194 shutdown_connections 2024-01-24T17:49:38.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff75d000640 1 --2- 172.21.15.39:0/848668194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff7580ce880 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff75d000640 1 --2- 172.21.15.39:0/848668194 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 0x7ff7580ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff75d000640 1 --2- 172.21.15.39:0/848668194 >> v2:172.21.15.100:3300/0 conn(0x7ff75815a840 0x7ff75815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.874+0000 7ff75d000640 1 -- 172.21.15.39:0/848668194 >> 172.21.15.39:0/848668194 conn(0x7ff75806a220 msgr2=0x7ff75806bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:38.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.875+0000 7ff75d000640 1 -- 172.21.15.39:0/848668194 shutdown_connections 2024-01-24T17:49:38.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.875+0000 7ff75d000640 1 -- 172.21.15.39:0/848668194 wait complete. 2024-01-24T17:49:38.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.876+0000 7ff75d000640 1 Processor -- start 2024-01-24T17:49:38.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.876+0000 7ff75d000640 1 -- start start 2024-01-24T17:49:38.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.877+0000 7ff75d000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 0x7ff7580cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.877+0000 7ff75d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff75815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.877+0000 7ff75d000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75815a840 0x7ff7580c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.877+0000 7ff75d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75815d240 con 0x7ff7580b1240 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.877+0000 7ff75d000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7580c3790 con 0x7ff75815a840 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.877+0000 7ff75d000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7580c3a70 con 0x7ff75806fe90 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff74fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff75815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff74fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff75815c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45424/0 (socket says 172.21.15.39:45424) 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff74fe00640 1 -- 172.21.15.39:0/4026123195 learned_addr learned my addr 172.21.15.39:0/4026123195 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff757e00640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75815a840 0x7ff7580c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff757400640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 0x7ff7580cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff74fe00640 1 -- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 msgr2=0x7ff7580cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff74fe00640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 0x7ff7580cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff74fe00640 1 -- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75815a840 msgr2=0x7ff7580c4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff74fe00640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75815a840 0x7ff7580c4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff74fe00640 1 -- 172.21.15.39:0/4026123195 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff750009b30 con 0x7ff7580b1240 2024-01-24T17:49:38.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.878+0000 7ff757400640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 0x7ff7580cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.879+0000 7ff74fe00640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff75815c7a0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7ff74000b410 tx=0x7ff74000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.879+0000 7ff74f400640 1 -- 172.21.15.39:0/4026123195 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff740011830 con 0x7ff7580b1240 2024-01-24T17:49:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.879+0000 7ff74f400640 1 -- 172.21.15.39:0/4026123195 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7400119b0 con 0x7ff7580b1240 2024-01-24T17:49:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.879+0000 7ff74f400640 1 -- 172.21.15.39:0/4026123195 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff740011d60 con 0x7ff7580b1240 2024-01-24T17:49:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.879+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7580c3d50 con 0x7ff7580b1240 2024-01-24T17:49:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.879+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7581f6540 con 0x7ff7580b1240 2024-01-24T17:49:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.880+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7580c4270 con 0x7ff7580b1240 2024-01-24T17:49:38.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.881+0000 7ff74f400640 1 -- 172.21.15.39:0/4026123195 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff74001a400 con 0x7ff7580b1240 2024-01-24T17:49:38.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.882+0000 7ff74f400640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff73006a430 0x7ff73006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:38.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.883+0000 7ff74f400640 1 -- 172.21.15.39:0/4026123195 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7ff740088c60 con 0x7ff7580b1240 2024-01-24T17:49:38.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.883+0000 7ff757400640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff73006a430 0x7ff73006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:38.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.884+0000 7ff757400640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff73006a430 0x7ff73006c8f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff748008890 tx=0x7ff748008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:38.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:38.887+0000 7ff74f400640 1 -- 172.21.15.39:0/4026123195 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff7580c4270 con 0x7ff7580b1240 2024-01-24T17:49:39.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.079+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff7580c4270 con 0x7ff7580b1240 2024-01-24T17:49:39.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.079+0000 7ff74f400640 1 -- 172.21.15.39:0/4026123195 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff7580c4270 con 0x7ff7580b1240 2024-01-24T17:49:39.081 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff73006a430 msgr2=0x7ff73006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff73006a430 0x7ff73006c8f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff748008890 tx=0x7ff748008040 comp rx=0 tx=0).stop 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 msgr2=0x7ff75815c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff75815c7a0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7ff74000b410 tx=0x7ff74000b8d0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 shutdown_connections 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff75815a840 0x7ff7580c4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff73006a430 0x7ff73006c8f0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7580b1240 0x7ff75815c7a0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 --2- 172.21.15.39:0/4026123195 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff75806fe90 0x7ff7580cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 >> 172.21.15.39:0/4026123195 conn(0x7ff75806a220 msgr2=0x7ff7580ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:39.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 shutdown_connections 2024-01-24T17:49:39.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.082+0000 7ff75d000640 1 -- 172.21.15.39:0/4026123195 wait complete. 2024-01-24T17:49:39.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:49:39.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.309+0000 7f0bae400640 1 Processor -- start 2024-01-24T17:49:39.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.309+0000 7f0bae400640 1 -- start start 2024-01-24T17:49:39.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.311+0000 7f0bae400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f0ba815cc70 0x7f0ba80cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.311+0000 7f0bae400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f0ba80cff50 0x7f0ba80cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.311+0000 7f0bae400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80d3890 con 0x7f0ba806ad90 2024-01-24T17:49:39.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.311+0000 7f0bae400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba80b1b80 con 0x7f0ba815cc70 2024-01-24T17:49:39.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.311+0000 7f0bae400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba80b2170 con 0x7f0ba80cff50 2024-01-24T17:49:39.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0bada00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0ba806ad90 0x7f0ba806b220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58072/0 (socket says 172.21.15.39:58072) 2024-01-24T17:49:39.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0bada00640 1 -- 172.21.15.39:0/4078152057 learned_addr learned my addr 172.21.15.39:0/4078152057 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:39.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0bad000640 1 --2- 172.21.15.39:0/4078152057 >> v2:172.21.15.39:3300/0 conn(0x7f0ba815cc70 0x7f0ba80cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0b9fe00640 1 --2- 172.21.15.39:0/4078152057 >> v2:172.21.15.100:3300/0 conn(0x7f0ba80cff50 0x7f0ba80cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0ba7e00640 1 -- 172.21.15.39:0/4078152057 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099648293 0 0) 0x7f0ba80d3890 con 0x7f0ba806ad90 2024-01-24T17:49:39.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0ba7e00640 1 -- 172.21.15.39:0/4078152057 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b88003520 con 0x7f0ba806ad90 2024-01-24T17:49:39.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0bad000640 1 -- 172.21.15.39:0/4078152057 >> v2:172.21.15.100:3300/0 conn(0x7f0ba80cff50 msgr2=0x7f0ba80cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:39.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0bad000640 1 --2- 172.21.15.39:0/4078152057 >> v2:172.21.15.100:3300/0 conn(0x7f0ba80cff50 0x7f0ba80cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0bad000640 1 -- 172.21.15.39:0/4078152057 >> v1:172.21.15.39:6789/0 conn(0x7f0ba806ad90 legacy=0x7f0ba806b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0bad000640 1 -- 172.21.15.39:0/4078152057 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80d1870 con 0x7f0ba815cc70 2024-01-24T17:49:39.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.312+0000 7f0b9fe00640 1 --2- 172.21.15.39:0/4078152057 >> v2:172.21.15.100:3300/0 conn(0x7f0ba80cff50 0x7f0ba80cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:49:39.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.314+0000 7f0bad000640 1 --2- 172.21.15.39:0/4078152057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80cf2e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8066840 tx=0x7f0b9802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701452874c1cece1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:39.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.314+0000 7f0ba7e00640 1 -- 172.21.15.39:0/4078152057 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9802e070 con 0x7f0ba815cc70 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.314+0000 7f0ba7e00640 1 -- 172.21.15.39:0/4078152057 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b980315a0 con 0x7f0ba815cc70 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.314+0000 7f0ba7e00640 1 -- 172.21.15.39:0/4078152057 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b98031930 con 0x7f0ba815cc70 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.314+0000 7f0bae400640 1 -- 172.21.15.39:0/4078152057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 msgr2=0x7f0ba80cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.314+0000 7f0bae400640 1 --2- 172.21.15.39:0/4078152057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80cf2e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8066840 tx=0x7f0b9802b840 comp rx=0 tx=0).stop 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.315+0000 7f0bae400640 1 -- 172.21.15.39:0/4078152057 shutdown_connections 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.315+0000 7f0bae400640 1 --2- 172.21.15.39:0/4078152057 >> v2:172.21.15.100:3300/0 conn(0x7f0ba80cff50 0x7f0ba80cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.315+0000 7f0bae400640 1 --2- 172.21.15.39:0/4078152057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80cf2e0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.315+0000 7f0bae400640 1 -- 172.21.15.39:0/4078152057 >> 172.21.15.39:0/4078152057 conn(0x7f0ba80d28f0 msgr2=0x7f0ba80d2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.315+0000 7f0bae400640 1 -- 172.21.15.39:0/4078152057 shutdown_connections 2024-01-24T17:49:39.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.315+0000 7f0bae400640 1 -- 172.21.15.39:0/4078152057 wait complete. 2024-01-24T17:49:39.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.316+0000 7f0bae400640 1 Processor -- start 2024-01-24T17:49:39.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.317+0000 7f0bae400640 1 -- start start 2024-01-24T17:49:39.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.317+0000 7f0bae400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ba806ad90 0x7f0ba81f98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.317+0000 7f0bae400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ba80cff50 0x7f0ba81f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.317+0000 7f0bae400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.317+0000 7f0bae400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80c99c0 con 0x7f0ba815cc70 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.317+0000 7f0bae400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80c9ca0 con 0x7f0ba806ad90 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bae400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba81f9e30 con 0x7f0ba80cff50 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45438/0 (socket says 172.21.15.39:45438) 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 -- 172.21.15.39:0/4080513913 learned_addr learned my addr 172.21.15.39:0/4080513913 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bad000640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ba806ad90 0x7f0ba81f98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 -- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ba80cff50 msgr2=0x7f0ba81f5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ba80cff50 0x7f0ba81f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 -- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ba806ad90 msgr2=0x7f0ba81f98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ba806ad90 0x7f0ba81f98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 -- 172.21.15.39:0/4080513913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98009400 con 0x7f0ba815cc70 2024-01-24T17:49:39.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.318+0000 7f0bada00640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80b06e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f0ba000cb40 tx=0x7f0ba000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:39.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.319+0000 7f0ba5600640 1 -- 172.21.15.39:0/4080513913 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba000aca0 con 0x7f0ba815cc70 2024-01-24T17:49:39.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.319+0000 7f0ba5600640 1 -- 172.21.15.39:0/4080513913 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba000ae00 con 0x7f0ba815cc70 2024-01-24T17:49:39.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.319+0000 7f0ba5600640 1 -- 172.21.15.39:0/4080513913 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0012620 con 0x7f0ba815cc70 2024-01-24T17:49:39.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.319+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba81fa100 con 0x7f0ba815cc70 2024-01-24T17:49:39.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.319+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba81fa6c0 con 0x7f0ba815cc70 2024-01-24T17:49:39.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.320+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba80c9e10 con 0x7f0ba815cc70 2024-01-24T17:49:39.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.321+0000 7f0ba5600640 1 -- 172.21.15.39:0/4080513913 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f0ba0007070 con 0x7f0ba815cc70 2024-01-24T17:49:39.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.322+0000 7f0ba5600640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b8006a480 0x7f0b8006c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.322+0000 7f0ba5600640 1 -- 172.21.15.39:0/4080513913 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f0ba0088a90 con 0x7f0ba815cc70 2024-01-24T17:49:39.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.323+0000 7f0bad000640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b8006a480 0x7f0b8006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.324+0000 7f0bad000640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b8006a480 0x7f0b8006c940 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0b98036000 tx=0x7f0b9803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:39.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.326+0000 7f0ba5600640 1 -- 172.21.15.39:0/4080513913 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f0ba0053fb0 con 0x7f0ba815cc70 2024-01-24T17:49:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.533+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0ba804e840 con 0x7f0ba815cc70 2024-01-24T17:49:39.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.534+0000 7f0ba5600640 1 -- 172.21.15.39:0/4080513913 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v106) v1 ==== 80+0+366 (secure 0 0 0) 0x7f0ba0017080 con 0x7f0ba815cc70 2024-01-24T17:49:39.535 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6833/872967797 2024-01-25T17:48:58.450631+0000 2024-01-24T17:49:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6832/872967797 2024-01-25T17:48:58.450631+0000 2024-01-24T17:49:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.100:6836/3116066709 2024-01-25T17:48:58.404809+0000 2024-01-24T17:49:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.100:6834/3116066709 2024-01-25T17:48:58.404809+0000 2024-01-24T17:49:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6835/3495904204 2024-01-25T17:48:58.377999+0000 2024-01-24T17:49:39.536 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6834/3495904204 2024-01-25T17:48:58.377999+0000 2024-01-24T17:49:39.536 INFO:teuthology.orchestra.run.smithi039.stderr:listed 6 entries 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.536+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b8006a480 msgr2=0x7f0b8006c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.536+0000 7f0bae400640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b8006a480 0x7f0b8006c940 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0b98036000 tx=0x7f0b9803d040 comp rx=0 tx=0).stop 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.536+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 msgr2=0x7f0ba80b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.536+0000 7f0bae400640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80b06e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f0ba000cb40 tx=0x7f0ba000a060 comp rx=0 tx=0).stop 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.537+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 shutdown_connections 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.537+0000 7f0bae400640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f0b8006a480 0x7f0b8006c940 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.537+0000 7f0bae400640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ba815cc70 0x7f0ba80b06e0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.537+0000 7f0bae400640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0ba80cff50 0x7f0ba81f5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.537+0000 7f0bae400640 1 --2- 172.21.15.39:0/4080513913 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0ba806ad90 0x7f0ba81f98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.537+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 >> 172.21.15.39:0/4080513913 conn(0x7f0ba80d28f0 msgr2=0x7f0ba80d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.537+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 shutdown_connections 2024-01-24T17:49:39.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.537+0000 7f0bae400640 1 -- 172.21.15.39:0/4080513913 wait complete. 2024-01-24T17:49:39.556 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:49:39.556 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:39.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:49:39.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.831+0000 7ff424600640 1 Processor -- start 2024-01-24T17:49:39.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.832+0000 7ff424600640 1 -- start start 2024-01-24T17:49:39.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.833+0000 7ff424600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7ff41c071d40 0x7ff41c0ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.833+0000 7ff424600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c0adb80 0x7ff41c15f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.833+0000 7ff424600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c15ade0 con 0x7ff41c15b5f0 2024-01-24T17:49:39.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.833+0000 7ff424600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41c15bb70 con 0x7ff41c071d40 2024-01-24T17:49:39.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.833+0000 7ff424600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b3540 con 0x7ff41c0adb80 2024-01-24T17:49:39.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.833+0000 7ff421c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff41c15b5f0 0x7ff41c071630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58078/0 (socket says 172.21.15.39:58078) 2024-01-24T17:49:39.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.833+0000 7ff421c00640 1 -- 172.21.15.39:0/1348646922 learned_addr learned my addr 172.21.15.39:0/1348646922 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:39.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286248598 0 0) 0x7ff41c15ade0 con 0x7ff41c15b5f0 2024-01-24T17:49:39.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc003520 con 0x7ff41c15b5f0 2024-01-24T17:49:39.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff422600640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c0adb80 0x7ff41c15f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff421200640 1 --2- 172.21.15.39:0/1348646922 >> v2:172.21.15.100:3300/0 conn(0x7ff41c071d40 0x7ff41c0ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4272972261 0 0) 0x7ff3fc003520 con 0x7ff41c15b5f0 2024-01-24T17:49:39.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c0adb80 msgr2=0x7ff41c15f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:39.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c0adb80 0x7ff41c15f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 >> v2:172.21.15.100:3300/0 conn(0x7ff41c071d40 msgr2=0x7ff41c0ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 --2- 172.21.15.39:0/1348646922 >> v2:172.21.15.100:3300/0 conn(0x7ff41c071d40 0x7ff41c0ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c15be10 con 0x7ff41c15b5f0 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7ff40c0017f0 con 0x7ff41c15b5f0 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.834+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 >> v1:172.21.15.39:6789/0 conn(0x7ff41c15b5f0 legacy=0x7ff41c071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff41be00640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc003970 0x7ff3fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff41be00640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3fc0047c0 0x7ff3fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff41be00640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3fc005180 0x7ff3fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c15bb70 con 0x7ff3fc0047c0 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b3540 con 0x7ff3fc003970 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc00bff0 con 0x7ff3fc005180 2024-01-24T17:49:39.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff422600640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c0adb80 0x7ff41c15f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff421200640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3fc0047c0 0x7ff3fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff421c00640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc003970 0x7ff3fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff421200640 1 -- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3fc005180 msgr2=0x7ff3fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff421200640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff3fc005180 0x7ff3fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff421200640 1 -- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc003970 msgr2=0x7ff3fc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff421200640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc003970 0x7ff3fc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff421200640 1 -- 172.21.15.39:0/1348646922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc003520 con 0x7ff3fc0047c0 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.835+0000 7ff421c00640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc003970 0x7ff3fc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.836+0000 7ff421200640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3fc0047c0 0x7ff3fc004c40 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7ff40400ba30 tx=0x7ff3fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c037353b5035e9b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.837+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404032680 con 0x7ff3fc0047c0 2024-01-24T17:49:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.837+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4040327e0 con 0x7ff3fc0047c0 2024-01-24T17:49:39.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.837+0000 7ff41be00640 1 -- 172.21.15.39:0/1348646922 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404032b70 con 0x7ff3fc0047c0 2024-01-24T17:49:39.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.837+0000 7ff424600640 1 -- 172.21.15.39:0/1348646922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3fc0047c0 msgr2=0x7ff3fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.837+0000 7ff424600640 1 --2- 172.21.15.39:0/1348646922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3fc0047c0 0x7ff3fc004c40 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7ff40400ba30 tx=0x7ff3fc0057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.837+0000 7ff422600640 1 -- 172.21.15.39:0/1348646922 reap_dead start 2024-01-24T17:49:39.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.837+0000 7ff424600640 1 -- 172.21.15.39:0/1348646922 shutdown_connections 2024-01-24T17:49:39.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.837+0000 7ff424600640 1 -- 172.21.15.39:0/1348646922 >> 172.21.15.39:0/1348646922 conn(0x7ff41c06a220 msgr2=0x7ff41c0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:39.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.838+0000 7ff424600640 1 -- 172.21.15.39:0/1348646922 shutdown_connections 2024-01-24T17:49:39.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.838+0000 7ff424600640 1 -- 172.21.15.39:0/1348646922 wait complete. 2024-01-24T17:49:39.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.839+0000 7ff424600640 1 Processor -- start 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.839+0000 7ff424600640 1 -- start start 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.840+0000 7ff424600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc0047c0 0x7ff41c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.840+0000 7ff424600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41c0b0af0 0x7ff41c155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.840+0000 7ff424600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c15d1d0 0x7ff41c15d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.840+0000 7ff424600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c15db90 con 0x7ff41c0b0af0 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.840+0000 7ff424600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0c7c00 con 0x7ff3fc0047c0 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.840+0000 7ff424600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0b0f70 con 0x7ff41c15d1d0 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41c0b0af0 0x7ff41c155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41c0b0af0 0x7ff41c155d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45470/0 (socket says 172.21.15.39:45470) 2024-01-24T17:49:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 -- 172.21.15.39:0/434394412 learned_addr learned my addr 172.21.15.39:0/434394412 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421c00640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc0047c0 0x7ff41c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 -- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c15d1d0 msgr2=0x7ff41c15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c15d1d0 0x7ff41c15d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 -- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc0047c0 msgr2=0x7ff41c0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc0047c0 0x7ff41c0c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 -- 172.21.15.39:0/434394412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c0c7ea0 con 0x7ff41c0b0af0 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.841+0000 7ff421200640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41c0b0af0 0x7ff41c155d50 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7ff40402bf00 tx=0x7ff404004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.842+0000 7ff419600640 1 -- 172.21.15.39:0/434394412 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040307e0 con 0x7ff41c0b0af0 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.842+0000 7ff419600640 1 -- 172.21.15.39:0/434394412 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff404030960 con 0x7ff41c0b0af0 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.842+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c0c8180 con 0x7ff41c0b0af0 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.842+0000 7ff419600640 1 -- 172.21.15.39:0/434394412 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404030d10 con 0x7ff41c0b0af0 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.842+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c0c8740 con 0x7ff41c0b0af0 2024-01-24T17:49:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.843+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41c073f10 con 0x7ff41c0b0af0 2024-01-24T17:49:39.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.844+0000 7ff419600640 1 -- 172.21.15.39:0/434394412 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff40403b420 con 0x7ff41c0b0af0 2024-01-24T17:49:39.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.845+0000 7ff419600640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f406a460 0x7ff3f406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:39.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.845+0000 7ff419600640 1 -- 172.21.15.39:0/434394412 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7ff4040aafa0 con 0x7ff41c0b0af0 2024-01-24T17:49:39.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.846+0000 7ff421c00640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f406a460 0x7ff3f406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:39.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.847+0000 7ff421c00640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f406a460 0x7ff3f406c920 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff40c001920 tx=0x7ff40c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:39.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:39.850+0000 7ff419600640 1 -- 172.21.15.39:0/434394412 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7ff404076440 con 0x7ff41c0b0af0 2024-01-24T17:49:40.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.045+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff41c0b2850 con 0x7ff41c0b0af0 2024-01-24T17:49:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.048+0000 7ff419600640 1 -- 172.21.15.39:0/434394412 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16529 (secure 0 0 0) 0x7ff404079a60 con 0x7ff41c0b0af0 2024-01-24T17:49:40.050 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:40.050 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":106,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:49:17.472389+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.0","mappings":[{"from":4,"to":0}]},{"pgid":"14.18","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6833/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.39:6832/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.100:6836/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.100:6834/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.39:6835/3495904204":"2024-01-25T17:48:58.377999+0000","172.21.15.39:6834/3495904204":"2024-01-25T17:48:58.377999+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:49:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.050+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f406a460 msgr2=0x7ff3f406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.050+0000 7ff424600640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f406a460 0x7ff3f406c920 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff40c001920 tx=0x7ff40c009040 comp rx=0 tx=0).stop 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.050+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41c0b0af0 msgr2=0x7ff41c155d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.050+0000 7ff424600640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41c0b0af0 0x7ff41c155d50 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7ff40402bf00 tx=0x7ff404004730 comp rx=0 tx=0).stop 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.050+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 shutdown_connections 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.050+0000 7ff424600640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7ff41c15d1d0 0x7ff41c15d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.050+0000 7ff424600640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7ff3f406a460 0x7ff3f406c920 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.051+0000 7ff424600640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff41c0b0af0 0x7ff41c155d50 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.051+0000 7ff424600640 1 --2- 172.21.15.39:0/434394412 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff3fc0047c0 0x7ff41c0c5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.051+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 >> 172.21.15.39:0/434394412 conn(0x7ff41c06a220 msgr2=0x7ff41c0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.051+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 shutdown_connections 2024-01-24T17:49:40.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.051+0000 7ff424600640 1 -- 172.21.15.39:0/434394412 wait complete. 2024-01-24T17:49:40.073 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:49:40.073 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:49:40.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.295+0000 7f8488e00640 1 Processor -- start 2024-01-24T17:49:40.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.296+0000 7f8488e00640 1 -- start start 2024-01-24T17:49:40.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.297+0000 7f8488e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f848415cb70 0x7f84840aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.297+0000 7f8488e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f84840ce920 0x7f84840cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.297+0000 7f8488e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84840c9cf0 con 0x7f84840cb7e0 2024-01-24T17:49:40.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.297+0000 7f8488e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848406a3d0 con 0x7f84840ce920 2024-01-24T17:49:40.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.297+0000 7f8488e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84840c9f90 con 0x7f848415cb70 2024-01-24T17:49:40.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.298+0000 7f8483e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f84840ce920 0x7f84840cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.298+0000 7f8483e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f84840ce920 0x7f84840cedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45484/0 (socket says 172.21.15.39:45484) 2024-01-24T17:49:40.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.298+0000 7f8483e00640 1 -- 172.21.15.39:0/3615939850 learned_addr learned my addr 172.21.15.39:0/3615939850 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:40.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.298+0000 7f8483400640 1 --2- 172.21.15.39:0/3615939850 >> v2:172.21.15.100:3300/0 conn(0x7f848415cb70 0x7f84840aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.298+0000 7f8483e00640 1 -- 172.21.15.39:0/3615939850 >> v2:172.21.15.100:3300/0 conn(0x7f848415cb70 msgr2=0x7f84840aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.298+0000 7f8483e00640 1 --2- 172.21.15.39:0/3615939850 >> v2:172.21.15.100:3300/0 conn(0x7f848415cb70 0x7f84840aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.298+0000 7f8483e00640 1 -- 172.21.15.39:0/3615939850 >> v1:172.21.15.100:6789/0 conn(0x7f84840cb7e0 legacy=0x7f8484069bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.298+0000 7f8483e00640 1 -- 172.21.15.39:0/3615939850 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84840ca270 con 0x7f84840ce920 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.299+0000 7f8483e00640 1 --2- 172.21.15.39:0/3615939850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f84840cedd0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f84840662c0 tx=0x7f847c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779bd9ed53934a16 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8482000640 1 -- 172.21.15.39:0/3615939850 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c031870 con 0x7f84840ce920 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8482000640 1 -- 172.21.15.39:0/3615939850 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c0319d0 con 0x7f84840ce920 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8482000640 1 -- 172.21.15.39:0/3615939850 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c031d60 con 0x7f84840ce920 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8488e00640 1 -- 172.21.15.39:0/3615939850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 msgr2=0x7f84840cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8488e00640 1 --2- 172.21.15.39:0/3615939850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f84840cedd0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f84840662c0 tx=0x7f847c02b910 comp rx=0 tx=0).stop 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8488e00640 1 -- 172.21.15.39:0/3615939850 shutdown_connections 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8488e00640 1 --2- 172.21.15.39:0/3615939850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f84840cedd0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8488e00640 1 --2- 172.21.15.39:0/3615939850 >> v2:172.21.15.100:3300/0 conn(0x7f848415cb70 0x7f84840aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8488e00640 1 -- 172.21.15.39:0/3615939850 >> 172.21.15.39:0/3615939850 conn(0x7f8484098a60 msgr2=0x7f84840cf600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:40.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.300+0000 7f8488e00640 1 -- 172.21.15.39:0/3615939850 shutdown_connections 2024-01-24T17:49:40.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.301+0000 7f8488e00640 1 -- 172.21.15.39:0/3615939850 wait complete. 2024-01-24T17:49:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.302+0000 7f8488e00640 1 Processor -- start 2024-01-24T17:49:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.302+0000 7f8488e00640 1 -- start start 2024-01-24T17:49:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.303+0000 7f8488e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84840cb7e0 0x7f84840c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.303+0000 7f8488e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f8484154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.303+0000 7f8488e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f848415cb70 0x7f84841f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.303+0000 7f8488e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84841f3880 con 0x7f84840ce920 2024-01-24T17:49:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.303+0000 7f8488e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84841f3b60 con 0x7f84840cb7e0 2024-01-24T17:49:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.303+0000 7f8488e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84841f3e40 con 0x7f848415cb70 2024-01-24T17:49:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8482a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f8484154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8482a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f8484154f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45500/0 (socket says 172.21.15.39:45500) 2024-01-24T17:49:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8482a00640 1 -- 172.21.15.39:0/3953639589 learned_addr learned my addr 172.21.15.39:0/3953639589 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8483e00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f848415cb70 0x7f84841f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8483400640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84840cb7e0 0x7f84840c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8482a00640 1 -- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f848415cb70 msgr2=0x7f84841f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8482a00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f848415cb70 0x7f84841f3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8482a00640 1 -- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84840cb7e0 msgr2=0x7f84840c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8482a00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84840cb7e0 0x7f84840c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8482a00640 1 -- 172.21.15.39:0/3953639589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c0094d0 con 0x7f84840ce920 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.304+0000 7f8483400640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84840cb7e0 0x7f84840c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.305+0000 7f8482a00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f8484154f10 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f846c00b3c0 tx=0x7f846c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.305+0000 7f847b400640 1 -- 172.21.15.39:0/3953639589 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c011830 con 0x7f84840ce920 2024-01-24T17:49:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.305+0000 7f847b400640 1 -- 172.21.15.39:0/3953639589 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c011990 con 0x7f84840ce920 2024-01-24T17:49:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.305+0000 7f847b400640 1 -- 172.21.15.39:0/3953639589 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c011d80 con 0x7f84840ce920 2024-01-24T17:49:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.305+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84841f4120 con 0x7f84840ce920 2024-01-24T17:49:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.305+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84841f3570 con 0x7f84840ce920 2024-01-24T17:49:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.306+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848404e840 con 0x7f84840ce920 2024-01-24T17:49:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.307+0000 7f847b400640 1 -- 172.21.15.39:0/3953639589 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f846c01b460 con 0x7f84840ce920 2024-01-24T17:49:40.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.308+0000 7f847b400640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f845c06a430 0x7f845c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.308+0000 7f847b400640 1 -- 172.21.15.39:0/3953639589 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f846c0885a0 con 0x7f84840ce920 2024-01-24T17:49:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.309+0000 7f8483400640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f845c06a430 0x7f845c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.310+0000 7f8483400640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f845c06a430 0x7f845c06c8f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8474009b70 tx=0x7f84740039d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:40.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.312+0000 7f847b400640 1 -- 172.21.15.39:0/3953639589 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f846c053a40 con 0x7f84840ce920 2024-01-24T17:49:40.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.509+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84840ccc70 con 0x7f84840ce920 2024-01-24T17:49:40.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.509+0000 7f847b400640 1 -- 172.21.15.39:0/3953639589 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f846c057060 con 0x7f84840ce920 2024-01-24T17:49:40.511 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:49:40.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f845c06a430 msgr2=0x7f845c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f845c06a430 0x7f845c06c8f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8474009b70 tx=0x7f84740039d0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 msgr2=0x7f8484154f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f8484154f10 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f846c00b3c0 tx=0x7f846c00b880 comp rx=0 tx=0).stop 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 shutdown_connections 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f848415cb70 0x7f84841f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f845c06a430 0x7f845c06c8f0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84840ce920 0x7f8484154f10 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 --2- 172.21.15.39:0/3953639589 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f84840cb7e0 0x7f84840c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.512+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 >> 172.21.15.39:0/3953639589 conn(0x7f8484098a60 msgr2=0x7f84840ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.513+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 shutdown_connections 2024-01-24T17:49:40.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.513+0000 7f8488e00640 1 -- 172.21.15.39:0/3953639589 wait complete. 2024-01-24T17:49:40.532 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:49:40.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.751+0000 7f871a600640 1 Processor -- start 2024-01-24T17:49:40.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.752+0000 7f871a600640 1 -- start start 2024-01-24T17:49:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.753+0000 7f871a600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f87140cf820 0x7f87140d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.753+0000 7f871a600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87140d0cb0 con 0x7f871415cd90 2024-01-24T17:49:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.753+0000 7f871a600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87140cdd70 con 0x7f87140b5e40 2024-01-24T17:49:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.753+0000 7f871a600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87140ce050 con 0x7f87140cf820 2024-01-24T17:49:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8713400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f871415cd90 0x7f87140cd320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58086/0 (socket says 172.21.15.39:58086) 2024-01-24T17:49:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8713400640 1 -- 172.21.15.39:0/1593652234 learned_addr learned my addr 172.21.15.39:0/1593652234 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8713e00640 1 --2- 172.21.15.39:0/1593652234 >> v2:172.21.15.100:3300/0 conn(0x7f87140cf820 0x7f87140d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468068901 0 0) 0x7f87140d0cb0 con 0x7f871415cd90 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f4003520 con 0x7f871415cd90 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2527829019 0 0) 0x7f86f4003520 con 0x7f871415cd90 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 >> v2:172.21.15.100:3300/0 conn(0x7f87140cf820 msgr2=0x7f87140d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8712a00640 1 --2- 172.21.15.39:0/1593652234 >> v2:172.21.15.100:3300/0 conn(0x7f87140cf820 0x7f87140d1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.754+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 >> v1:172.21.15.100:6789/0 conn(0x7f87140b5e40 legacy=0x7f87140cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87140d0f20 con 0x7f871415cd90 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f8704001670 con 0x7f871415cd90 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 >> v1:172.21.15.39:6789/0 conn(0x7f871415cd90 legacy=0x7f87140cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f40039c0 0x7f86f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f4005180 0x7f86f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87140ce050 con 0x7f86f40039c0 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f400c080 con 0x7f86f40047c0 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f400c240 con 0x7f86f4005180 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8713400640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f40039c0 0x7f86f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8713e00640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f4005180 0x7f86f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f870be00640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8713400640 1 -- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f4005180 msgr2=0x7f86f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8713400640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f4005180 0x7f86f400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8713400640 1 -- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f40047c0 msgr2=0x7f86f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8713400640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8713400640 1 -- 172.21.15.39:0/1593652234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f4003520 con 0x7f86f40039c0 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f870be00640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86f40047c0 0x7f86f4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:40.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.755+0000 7f8713e00640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f4005180 0x7f86f400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:40.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.756+0000 7f8713e00640 1 -- 172.21.15.39:0/1593652234 reap_dead start 2024-01-24T17:49:40.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.757+0000 7f8713400640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f40039c0 0x7f86f4003de0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f870400c0e0 tx=0x7f86f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda3735ab54a5d24 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:40.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.757+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704005ad0 con 0x7f86f40039c0 2024-01-24T17:49:40.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.757+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8704005c50 con 0x7f86f40039c0 2024-01-24T17:49:40.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.757+0000 7f8712a00640 1 -- 172.21.15.39:0/1593652234 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704007070 con 0x7f86f40039c0 2024-01-24T17:49:40.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.757+0000 7f871a600640 1 -- 172.21.15.39:0/1593652234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f40039c0 msgr2=0x7f86f4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.758+0000 7f871a600640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f40039c0 0x7f86f4003de0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f870400c0e0 tx=0x7f86f40057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.758+0000 7f871a600640 1 -- 172.21.15.39:0/1593652234 shutdown_connections 2024-01-24T17:49:40.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.758+0000 7f871a600640 1 --2- 172.21.15.39:0/1593652234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86f40039c0 0x7f86f4003de0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.758+0000 7f871a600640 1 -- 172.21.15.39:0/1593652234 >> 172.21.15.39:0/1593652234 conn(0x7f871406af00 msgr2=0x7f871406b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:40.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.758+0000 7f871a600640 1 -- 172.21.15.39:0/1593652234 shutdown_connections 2024-01-24T17:49:40.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.758+0000 7f871a600640 1 -- 172.21.15.39:0/1593652234 wait complete. 2024-01-24T17:49:40.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.759+0000 7f871a600640 1 Processor -- start 2024-01-24T17:49:40.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.760+0000 7f871a600640 1 -- start start 2024-01-24T17:49:40.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f871a600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f40039c0 0x7f87141530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f871a600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87140b5040 0x7f87141525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f871a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140b3770 0x7f871415b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f871a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8714152f80 con 0x7f87140b3770 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f871a600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87140b3fa0 con 0x7f87140b5040 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f871a600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87140b4280 con 0x7f86f40039c0 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f8713e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140b3770 0x7f871415b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f8713e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140b3770 0x7f871415b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45524/0 (socket says 172.21.15.39:45524) 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f8713e00640 1 -- 172.21.15.39:0/4209879437 learned_addr learned my addr 172.21.15.39:0/4209879437 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f8713400640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f40039c0 0x7f87141530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f870be00640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87140b5040 0x7f87141525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f8713e00640 1 -- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f40039c0 msgr2=0x7f87141530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.761+0000 7f8713e00640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f40039c0 0x7f87141530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.762+0000 7f8713e00640 1 -- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87140b5040 msgr2=0x7f87141525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.762+0000 7f8713e00640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87140b5040 0x7f87141525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.762+0000 7f8713e00640 1 -- 172.21.15.39:0/4209879437 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87140b4560 con 0x7f87140b3770 2024-01-24T17:49:40.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.762+0000 7f8713e00640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140b3770 0x7f871415b4e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f87140b1e50 tx=0x7f870c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:40.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.764+0000 7f870b400640 1 -- 172.21.15.39:0/4209879437 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c016420 con 0x7f87140b3770 2024-01-24T17:49:40.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.764+0000 7f870b400640 1 -- 172.21.15.39:0/4209879437 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c016580 con 0x7f87140b3770 2024-01-24T17:49:40.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.764+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87140b4840 con 0x7f87140b3770 2024-01-24T17:49:40.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.764+0000 7f870b400640 1 -- 172.21.15.39:0/4209879437 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c016910 con 0x7f87140b3770 2024-01-24T17:49:40.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.764+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87140b3bd0 con 0x7f87140b3770 2024-01-24T17:49:40.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.765+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87140c95d0 con 0x7f87140b3770 2024-01-24T17:49:40.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.766+0000 7f870b400640 1 -- 172.21.15.39:0/4209879437 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f870c016a70 con 0x7f87140b3770 2024-01-24T17:49:40.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.767+0000 7f870b400640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86ec06a430 0x7f86ec06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:40.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.767+0000 7f870b400640 1 -- 172.21.15.39:0/4209879437 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f870c089830 con 0x7f87140b3770 2024-01-24T17:49:40.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.768+0000 7f8713400640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86ec06a430 0x7f86ec06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:40.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.769+0000 7f8713400640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86ec06a430 0x7f86ec06c8f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8704034000 tx=0x7f8704009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:40.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.772+0000 7f870b400640 1 -- 172.21.15.39:0/4209879437 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f870c054cd0 con 0x7f87140b3770 2024-01-24T17:49:40.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.967+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f87140b3d60 con 0x7f87140b3770 2024-01-24T17:49:40.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.968+0000 7f870b400640 1 -- 172.21.15.39:0/4209879437 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f87140b3d60 con 0x7f87140b3770 2024-01-24T17:49:40.970 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:49:40.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86ec06a430 msgr2=0x7f86ec06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86ec06a430 0x7f86ec06c8f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8704034000 tx=0x7f8704009030 comp rx=0 tx=0).stop 2024-01-24T17:49:40.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140b3770 msgr2=0x7f871415b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:40.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140b3770 0x7f871415b4e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f87140b1e50 tx=0x7f870c00f520 comp rx=0 tx=0).stop 2024-01-24T17:49:40.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 shutdown_connections 2024-01-24T17:49:40.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f86ec06a430 0x7f86ec06c8f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f87140b3770 0x7f871415b4e0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f87140b5040 0x7f87141525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 --2- 172.21.15.39:0/4209879437 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f86f40039c0 0x7f87141530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:40.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 >> 172.21.15.39:0/4209879437 conn(0x7f871406af00 msgr2=0x7f87140c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:40.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 shutdown_connections 2024-01-24T17:49:40.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:40.971+0000 7f871a600640 1 -- 172.21.15.39:0/4209879437 wait complete. 2024-01-24T17:49:40.991 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:49:41.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.212+0000 7fb102400640 1 Processor -- start 2024-01-24T17:49:41.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.213+0000 7fb102400640 1 -- start start 2024-01-24T17:49:41.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb102400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb0fc0cb9c0 0x7fb0fc1654d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb102400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb0fc0ce3a0 0x7fb0fc0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb102400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0fc165a10 con 0x7fb0fc0c1860 2024-01-24T17:49:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb102400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc1650b0 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb102400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc0c9a60 con 0x7fb0fc0ce3a0 2024-01-24T17:49:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb101000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb0fc0c1860 0x7fb0fc0c8c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58098/0 (socket says 172.21.15.39:58098) 2024-01-24T17:49:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb101000640 1 -- 172.21.15.39:0/630435811 learned_addr learned my addr 172.21.15.39:0/630435811 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:41.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb0f3e00640 1 --2- 172.21.15.39:0/630435811 >> v2:172.21.15.39:3300/0 conn(0x7fb0fc0cb9c0 0x7fb0fc1654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb101a00640 1 --2- 172.21.15.39:0/630435811 >> v2:172.21.15.100:3300/0 conn(0x7fb0fc0ce3a0 0x7fb0fc0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.215+0000 7fb0fbe00640 1 -- 172.21.15.39:0/630435811 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520035999 0 0) 0x7fb0ec001110 con 0x7fb0fc0c1860 2024-01-24T17:49:41.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.216+0000 7fb0f3e00640 1 -- 172.21.15.39:0/630435811 >> v2:172.21.15.100:3300/0 conn(0x7fb0fc0ce3a0 msgr2=0x7fb0fc0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:41.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.216+0000 7fb0f3e00640 1 --2- 172.21.15.39:0/630435811 >> v2:172.21.15.100:3300/0 conn(0x7fb0fc0ce3a0 0x7fb0fc0ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.216+0000 7fb0f3e00640 1 -- 172.21.15.39:0/630435811 >> v1:172.21.15.39:6789/0 conn(0x7fb0fc0c1860 legacy=0x7fb0fc0c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.216+0000 7fb0f3e00640 1 -- 172.21.15.39:0/630435811 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc0c94a0 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.217+0000 7fb0f3e00640 1 --2- 172.21.15.39:0/630435811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1654d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0cc0f0 tx=0x7fb0e402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409ce1b1f4d85aae server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:41.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.217+0000 7fb0fbe00640 1 -- 172.21.15.39:0/630435811 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e402f050 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.217+0000 7fb0fbe00640 1 -- 172.21.15.39:0/630435811 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4031700 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.217+0000 7fb0fbe00640 1 -- 172.21.15.39:0/630435811 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4031a90 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.218+0000 7fb102400640 1 -- 172.21.15.39:0/630435811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 msgr2=0x7fb0fc1654d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.218+0000 7fb102400640 1 --2- 172.21.15.39:0/630435811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1654d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0cc0f0 tx=0x7fb0e402b7a0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.218+0000 7fb102400640 1 -- 172.21.15.39:0/630435811 shutdown_connections 2024-01-24T17:49:41.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.218+0000 7fb102400640 1 --2- 172.21.15.39:0/630435811 >> v2:172.21.15.100:3300/0 conn(0x7fb0fc0ce3a0 0x7fb0fc0ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.218+0000 7fb102400640 1 --2- 172.21.15.39:0/630435811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1654d0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.218+0000 7fb102400640 1 -- 172.21.15.39:0/630435811 >> 172.21.15.39:0/630435811 conn(0x7fb0fc06a220 msgr2=0x7fb0fc06bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:41.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.218+0000 7fb102400640 1 -- 172.21.15.39:0/630435811 shutdown_connections 2024-01-24T17:49:41.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.218+0000 7fb102400640 1 -- 172.21.15.39:0/630435811 wait complete. 2024-01-24T17:49:41.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.220+0000 7fb102400640 1 Processor -- start 2024-01-24T17:49:41.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.220+0000 7fb102400640 1 -- start start 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb102400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0fc0c1860 0x7fb0fc1f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb102400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1f92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb102400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0fc0ce3a0 0x7fb0fc163710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb102400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0be460 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb102400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0be740 con 0x7fb0fc0c1860 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb102400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0c36f0 con 0x7fb0fc0ce3a0 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb0f3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1f92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb0f3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1f92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45540/0 (socket says 172.21.15.39:45540) 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb0f3e00640 1 -- 172.21.15.39:0/3398889867 learned_addr learned my addr 172.21.15.39:0/3398889867 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:41.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.221+0000 7fb101000640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0fc0c1860 0x7fb0fc1f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb101a00640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0fc0ce3a0 0x7fb0fc163710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f3e00640 1 -- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0fc0ce3a0 msgr2=0x7fb0fc163710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f3e00640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0fc0ce3a0 0x7fb0fc163710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f3e00640 1 -- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0fc0c1860 msgr2=0x7fb0fc1f6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f3e00640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0fc0c1860 0x7fb0fc1f6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f3e00640 1 -- 172.21.15.39:0/3398889867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e4009400 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f3e00640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1f92f0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000c00 tx=0x7fb0e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f9600640 1 -- 172.21.15.39:0/3398889867 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4004030 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f9600640 1 -- 172.21.15.39:0/3398889867 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4004190 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb0f9600640 1 -- 172.21.15.39:0/3398889867 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4004520 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.222+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0fc0c39d0 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.223+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc1f9830 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.224+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc04e840 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.225+0000 7fb0f9600640 1 -- 172.21.15.39:0/3398889867 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb0e4033930 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.226+0000 7fb0f9600640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d406a430 0x7fb0d406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.226+0000 7fb0f9600640 1 -- 172.21.15.39:0/3398889867 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fb0e40aab90 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.226+0000 7fb101000640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d406a430 0x7fb0d406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.227+0000 7fb101000640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d406a430 0x7fb0d406c8f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec000f60 tx=0x7fb0ec007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:41.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.230+0000 7fb0f9600640 1 -- 172.21.15.39:0/3398889867 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb0e40760d0 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.426+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0fc0b4390 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.427+0000 7fb0f9600640 1 -- 172.21.15.39:0/3398889867 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb0e4038020 con 0x7fb0fc0cb9c0 2024-01-24T17:49:41.429 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:49:41.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d406a430 msgr2=0x7fb0d406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d406a430 0x7fb0d406c8f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec000f60 tx=0x7fb0ec007c00 comp rx=0 tx=0).stop 2024-01-24T17:49:41.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 msgr2=0x7fb0fc1f92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1f92f0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000c00 tx=0x7fb0e40046c0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 shutdown_connections 2024-01-24T17:49:41.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb0fc0ce3a0 0x7fb0fc163710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb0d406a430 0x7fb0d406c8f0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0fc0cb9c0 0x7fb0fc1f92f0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 --2- 172.21.15.39:0/3398889867 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0fc0c1860 0x7fb0fc1f6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 >> 172.21.15.39:0/3398889867 conn(0x7fb0fc06a220 msgr2=0x7fb0fc0c5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:41.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 shutdown_connections 2024-01-24T17:49:41.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.430+0000 7fb102400640 1 -- 172.21.15.39:0/3398889867 wait complete. 2024-01-24T17:49:41.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T17:49:41.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.663+0000 7fb5c8a00640 1 Processor -- start 2024-01-24T17:49:41.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.664+0000 7fb5c8a00640 1 -- start start 2024-01-24T17:49:41.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.665+0000 7fb5c8a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb5c40cf9f0 0x7fb5c40cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.665+0000 7fb5c8a00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb5c40cff50 0x7fb5c40cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.665+0000 7fb5c8a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c40d3890 con 0x7fb5c415cc70 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.665+0000 7fb5c8a00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c40b1b80 con 0x7fb5c40cff50 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.665+0000 7fb5c8a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c40b2170 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.665+0000 7fb5c3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb5c40cff50 0x7fb5c40cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.665+0000 7fb5c3200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb5c40cff50 0x7fb5c40cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45544/0 (socket says 172.21.15.39:45544) 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.665+0000 7fb5c3200640 1 -- 172.21.15.39:0/3952729994 learned_addr learned my addr 172.21.15.39:0/3952729994 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.666+0000 7fb5c1e00640 1 --2- 172.21.15.39:0/3952729994 >> v2:172.21.15.100:3300/0 conn(0x7fb5c40cf9f0 0x7fb5c40cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.666+0000 7fb5c3200640 1 -- 172.21.15.39:0/3952729994 >> v2:172.21.15.100:3300/0 conn(0x7fb5c40cf9f0 msgr2=0x7fb5c40cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.666+0000 7fb5c3200640 1 --2- 172.21.15.39:0/3952729994 >> v2:172.21.15.100:3300/0 conn(0x7fb5c40cf9f0 0x7fb5c40cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.666+0000 7fb5c3200640 1 -- 172.21.15.39:0/3952729994 >> v1:172.21.15.100:6789/0 conn(0x7fb5c415cc70 legacy=0x7fb5c40cf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:41.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.666+0000 7fb5c3200640 1 -- 172.21.15.39:0/3952729994 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c406bda0 con 0x7fb5c40cff50 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.667+0000 7fb5c3200640 1 --2- 172.21.15.39:0/3952729994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cff50 0x7fb5c40cd8d0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4066840 tx=0x7fb5bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9408835325abd261 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.667+0000 7fb5c1400640 1 -- 172.21.15.39:0/3952729994 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc0317e0 con 0x7fb5c40cff50 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.667+0000 7fb5c1400640 1 -- 172.21.15.39:0/3952729994 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5bc031940 con 0x7fb5c40cff50 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.667+0000 7fb5c1400640 1 -- 172.21.15.39:0/3952729994 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc031cd0 con 0x7fb5c40cff50 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.668+0000 7fb5c8a00640 1 -- 172.21.15.39:0/3952729994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cff50 msgr2=0x7fb5c40cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.668+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/3952729994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cff50 0x7fb5c40cd8d0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4066840 tx=0x7fb5bc02b880 comp rx=0 tx=0).stop 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.668+0000 7fb5c8a00640 1 -- 172.21.15.39:0/3952729994 shutdown_connections 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.668+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/3952729994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cff50 0x7fb5c40cd8d0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.668+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/3952729994 >> v2:172.21.15.100:3300/0 conn(0x7fb5c40cf9f0 0x7fb5c40cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.668+0000 7fb5c8a00640 1 -- 172.21.15.39:0/3952729994 >> 172.21.15.39:0/3952729994 conn(0x7fb5c40d28f0 msgr2=0x7fb5c40d2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.668+0000 7fb5c8a00640 1 -- 172.21.15.39:0/3952729994 shutdown_connections 2024-01-24T17:49:41.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.668+0000 7fb5c8a00640 1 -- 172.21.15.39:0/3952729994 wait complete. 2024-01-24T17:49:41.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.670+0000 7fb5c8a00640 1 Processor -- start 2024-01-24T17:49:41.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.670+0000 7fb5c8a00640 1 -- start start 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c8a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cf9f0 0x7fb5c415ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c8a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5c40cff50 0x7fb5c40b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c8a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c415cc70 0x7fb5c40b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c8a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c4157710 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c8a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c415f2c0 con 0x7fb5c40cff50 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c8a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c415f5a0 con 0x7fb5c415cc70 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cf9f0 0x7fb5c415ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cf9f0 0x7fb5c415ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45554/0 (socket says 172.21.15.39:45554) 2024-01-24T17:49:41.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.671+0000 7fb5c2800640 1 -- 172.21.15.39:0/236842828 learned_addr learned my addr 172.21.15.39:0/236842828 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:41.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c1e00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5c40cff50 0x7fb5c40b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c3200640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c415cc70 0x7fb5c40b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c2800640 1 -- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c415cc70 msgr2=0x7fb5c40b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c2800640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c415cc70 0x7fb5c40b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c2800640 1 -- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5c40cff50 msgr2=0x7fb5c40b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c2800640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5c40cff50 0x7fb5c40b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c2800640 1 -- 172.21.15.39:0/236842828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc0094d0 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c1e00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5c40cff50 0x7fb5c40b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:41.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c3200640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c415cc70 0x7fb5c40b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:41.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c2800640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cf9f0 0x7fb5c415ea20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400b3f0 tx=0x7fb5b400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:41.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5baa00640 1 -- 172.21.15.39:0/236842828 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4011830 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5baa00640 1 -- 172.21.15.39:0/236842828 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b4011990 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5baa00640 1 -- 172.21.15.39:0/236842828 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4011d40 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c415f880 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.672+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c415ef60 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.674+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c40c99a0 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.674+0000 7fb5baa00640 1 -- 172.21.15.39:0/236842828 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb5b401a400 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.675+0000 7fb5baa00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a430 0x7fb59c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:41.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.675+0000 7fb5baa00640 1 -- 172.21.15.39:0/236842828 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fb5b4088850 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.676+0000 7fb5c1e00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a430 0x7fb59c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:41.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.677+0000 7fb5c1e00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a430 0x7fb59c06c8f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb5c40b4e60 tx=0x7fb5ac00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:41.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.680+0000 7fb5baa00640 1 -- 172.21.15.39:0/236842828 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb5b4053d90 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.889+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb5c404e840 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.890+0000 7fb5baa00640 1 -- 172.21.15.39:0/236842828 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v106) v1 ==== 80+0+366 (secure 0 0 0) 0x7fb5b4015080 con 0x7fb5c40cf9f0 2024-01-24T17:49:41.892 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6833/872967797 2024-01-25T17:48:58.450631+0000 2024-01-24T17:49:41.892 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6832/872967797 2024-01-25T17:48:58.450631+0000 2024-01-24T17:49:41.892 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.100:6836/3116066709 2024-01-25T17:48:58.404809+0000 2024-01-24T17:49:41.892 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.100:6834/3116066709 2024-01-25T17:48:58.404809+0000 2024-01-24T17:49:41.892 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6835/3495904204 2024-01-25T17:48:58.377999+0000 2024-01-24T17:49:41.892 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6834/3495904204 2024-01-25T17:48:58.377999+0000 2024-01-24T17:49:41.892 INFO:teuthology.orchestra.run.smithi039.stderr:listed 6 entries 2024-01-24T17:49:41.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.892+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a430 msgr2=0x7fb59c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.892+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a430 0x7fb59c06c8f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb5c40b4e60 tx=0x7fb5ac00b040 comp rx=0 tx=0).stop 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.892+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cf9f0 msgr2=0x7fb5c415ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.892+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cf9f0 0x7fb5c415ea20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400b3f0 tx=0x7fb5b400b8b0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.893+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 shutdown_connections 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.893+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb5c415cc70 0x7fb5c40b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.893+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb5c40cff50 0x7fb5c40b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.893+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb59c06a430 0x7fb59c06c8f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.893+0000 7fb5c8a00640 1 --2- 172.21.15.39:0/236842828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5c40cf9f0 0x7fb5c415ea20 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.893+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 >> 172.21.15.39:0/236842828 conn(0x7fb5c40d28f0 msgr2=0x7fb5c40b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.893+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 shutdown_connections 2024-01-24T17:49:41.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:41.893+0000 7fb5c8a00640 1 -- 172.21.15.39:0/236842828 wait complete. 2024-01-24T17:49:41.912 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi100.front.sepia.ceph.com... 2024-01-24T17:49:41.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:41.912 DEBUG:teuthology.orchestra.run.smithi100:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-24T17:49:42.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi100.stderr:ceph-fuse[49286]: fuse finished with error 0 and tester_r 0 2024-01-24T17:49:42.015 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:49:48.117 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:49:48.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:48.117 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:49:48.176 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-01-24T17:49:48.425 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-01-24T17:49:48.425 DEBUG:tasks.cephfs.filesystem:_json_asok output [] 2024-01-24T17:49:48.425 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-01-24T17:49:48.673 INFO:teuthology.orchestra.run.smithi039.stdout:{"message":"","return_code":0} 2024-01-24T17:49:48.673 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-01-24T17:49:48.674 DEBUG:teuthology.orchestra.run.smithi039:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 0 show session 2024-01-24T17:49:48.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.778+0000 7f9aab839100 1 -- start start 2024-01-24T17:49:48.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.778+0000 7f9aab839100 1 Processor -- start 2024-01-24T17:49:48.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.779+0000 7f9aab839100 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce8606cc00 0x55ce860f2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.779+0000 7f9aab839100 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce8606d800 0x55ce860f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.779+0000 7f9aab839100 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce8606dc00 0x55ce860f3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.779+0000 7f9aab839100 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8613ea00 con 0x55ce8606dc00 2024-01-24T17:49:48.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.779+0000 7f9aab839100 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8613e780 con 0x55ce8606d800 2024-01-24T17:49:48.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.779+0000 7f9aab839100 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce8613e640 con 0x55ce8606cc00 2024-01-24T17:49:48.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.779+0000 7f9aa9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce8606dc00 0x55ce860f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce8606dc00 0x55ce860f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54382/0 (socket says 172.21.15.39:54382) 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9a00640 1 -- 172.21.15.39:0/1452737119 learned_addr learned my addr 172.21.15.39:0/1452737119 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa8600640 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce8606d800 0x55ce860f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9000640 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce8606cc00 0x55ce860f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9a00640 1 -- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce8606cc00 msgr2=0x55ce860f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9a00640 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce8606cc00 0x55ce860f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9a00640 1 -- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce8606d800 msgr2=0x55ce860f2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9a00640 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce8606d800 0x55ce860f2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9a00640 1 -- 172.21.15.39:0/1452737119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce86092b60 con 0x55ce8606dc00 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa8600640 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce8606d800 0x55ce860f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:48.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9000640 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce8606cc00 0x55ce860f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.780+0000 7f9aa9a00640 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce8606dc00 0x55ce860f3080 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x55ce86e90ba0 tx=0x55ce86e3f140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c401d18896c2c480 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aa5e00640 1 -- 172.21.15.39:0/1452737119 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ce860b5600 con 0x55ce8606dc00 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aa5e00640 1 -- 172.21.15.39:0/1452737119 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ce860ca480 con 0x55ce8606dc00 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aa5e00640 1 -- 172.21.15.39:0/1452737119 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ce860b51e0 con 0x55ce8606dc00 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aab839100 1 -- 172.21.15.39:0/1452737119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce86092820 con 0x55ce8606dc00 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aab839100 1 -- 172.21.15.39:0/1452737119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55ce860924e0 con 0x55ce8606dc00 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aab839100 4 waiting for MDS map... 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aa5e00640 1 -- 172.21.15.39:0/1452737119 <== mon.0 v2:172.21.15.39:3300/0 4 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x55ce86098380 con 0x55ce8606dc00 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aa5e00640 1 -- 172.21.15.39:0/1452737119 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 72) v1 ==== 3130+0+0 (secure 0 0 0) 0x55ce860bf200 con 0x55ce8606dc00 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aab839100 4 Got MDS map 72 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aab839100 10 main: main 2024-01-24T17:49:48.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.781+0000 7f9aab839100 4 main: connecting to RADOS... 2024-01-24T17:49:48.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.782+0000 7f9aab839100 1 Processor -- start 2024-01-24T17:49:48.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.783+0000 7f9aab839100 1 -- start start 2024-01-24T17:49:48.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aab839100 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d000 0x55ce860f3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aab839100 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce86f2d400 0x55ce860f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aab839100 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce86f2d800 0x55ce860f4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aab839100 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce860b7e00 con 0x55ce86f2d000 2024-01-24T17:49:48.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aab839100 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8613e780 con 0x55ce86f2d400 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aab839100 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce8613e500 con 0x55ce86f2d800 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aa8600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d000 0x55ce860f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aa8600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d000 0x55ce860f3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54396/0 (socket says 172.21.15.39:54396) 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aa8600640 1 -- 172.21.15.39:0/763363191 learned_addr learned my addr 172.21.15.39:0/763363191 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.784+0000 7f9aa9000640 1 --2- 172.21.15.39:0/763363191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce86f2d400 0x55ce860f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa8600640 1 -- 172.21.15.39:0/763363191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce86f2d800 msgr2=0x55ce860f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa8600640 1 --2- 172.21.15.39:0/763363191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce86f2d800 0x55ce860f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa8600640 1 -- 172.21.15.39:0/763363191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce86f2d400 msgr2=0x55ce860f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa8600640 1 --2- 172.21.15.39:0/763363191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce86f2d400 0x55ce860f3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa8600640 1 -- 172.21.15.39:0/763363191 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce860921a0 con 0x55ce86f2d000 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa8600640 1 --2- 172.21.15.39:0/763363191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d000 0x55ce860f3600 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x55ce86f98090 tx=0x55ce86f48c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21be190387c0ab6c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa3600640 1 -- 172.21.15.39:0/763363191 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ce860b5600 con 0x55ce86f2d000 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa3600640 1 -- 172.21.15.39:0/763363191 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ce86e48c00 con 0x55ce86f2d000 2024-01-24T17:49:48.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.785+0000 7f9aa3600640 1 -- 172.21.15.39:0/763363191 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ce860b51e0 con 0x55ce86f2d000 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 -- 172.21.15.39:0/763363191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d000 msgr2=0x55ce860f3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 --2- 172.21.15.39:0/763363191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d000 0x55ce860f3600 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x55ce86f98090 tx=0x55ce86f48c00 comp rx=0 tx=0).stop 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 -- 172.21.15.39:0/763363191 shutdown_connections 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 --2- 172.21.15.39:0/763363191 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce86f2d800 0x55ce860f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 --2- 172.21.15.39:0/763363191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce86f2d400 0x55ce860f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 --2- 172.21.15.39:0/763363191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d000 0x55ce860f3600 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 -- 172.21.15.39:0/763363191 >> 172.21.15.39:0/763363191 conn(0x55ce86f2cc00 msgr2=0x55ce8614b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 -- 172.21.15.39:0/763363191 shutdown_connections 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.786+0000 7f9aab839100 1 -- 172.21.15.39:0/763363191 wait complete. 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.787+0000 7f9aab839100 1 Processor -- start 2024-01-24T17:49:48.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.787+0000 7f9aab839100 1 -- start start 2024-01-24T17:49:48.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.787+0000 7f9aab839100 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce86f2d000 0x55ce860f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.787+0000 7f9aab839100 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d400 0x55ce860f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.787+0000 7f9aab839100 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce86f2d800 0x55ce860f4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.787+0000 7f9aab839100 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8613e3c0 con 0x55ce86f2d400 2024-01-24T17:49:48.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.787+0000 7f9aab839100 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8613e280 con 0x55ce86f2d000 2024-01-24T17:49:48.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.787+0000 7f9aab839100 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce8613e140 con 0x55ce86f2d800 2024-01-24T17:49:48.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d400 0x55ce860f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d400 0x55ce860f4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54404/0 (socket says 172.21.15.39:54404) 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 -- 172.21.15.39:0/2334176843 learned_addr learned my addr 172.21.15.39:0/2334176843 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 -- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce86f2d800 msgr2=0x55ce860f4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce86f2d800 0x55ce860f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 -- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce86f2d000 msgr2=0x55ce860f5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce86f2d000 0x55ce860f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 -- 172.21.15.39:0/2334176843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce86092b60 con 0x55ce86f2d400 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.788+0000 7f9aa9000640 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d400 0x55ce860f4c00 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x55ce86f98d20 tx=0x55ce86e3ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.789+0000 7f9aa1800640 1 -- 172.21.15.39:0/2334176843 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ce860b5080 con 0x55ce86f2d400 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.789+0000 7f9aa1800640 1 -- 172.21.15.39:0/2334176843 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ce860ca180 con 0x55ce86f2d400 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.789+0000 7f9aa1800640 1 -- 172.21.15.39:0/2334176843 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55ce860b5600 con 0x55ce86f2d400 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.789+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ce86092820 con 0x55ce86f2d400 2024-01-24T17:49:48.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.789+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce860924e0 con 0x55ce86f2d400 2024-01-24T17:49:48.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.789+0000 7f9aab839100 4 main: resolving pool 14 2024-01-24T17:49:48.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.791+0000 7f9aa1800640 1 -- 172.21.15.39:0/2334176843 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x55ce8613b080 con 0x55ce86f2d400 2024-01-24T17:49:48.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.791+0000 7f9aa1800640 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x55ce86f9e800 0x55ce860f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.791+0000 7f9aa1800640 1 -- 172.21.15.39:0/2334176843 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x55ce86098540 con 0x55ce86f2d400 2024-01-24T17:49:48.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.791+0000 7f9aab839100 4 main: creating IoCtx.. 2024-01-24T17:49:48.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.791+0000 7f9aab839100 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x55ce86f9ec00 0x55ce860f4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:48.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.792+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- osd_op(unknown.0.0:1 14.b 14:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x55ce86f9f000 con 0x55ce86f9ec00 2024-01-24T17:49:48.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.792+0000 7f9aa8600640 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x55ce86f9e800 0x55ce860f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:48.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.792+0000 7f9aa8600640 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x55ce86f9ec00 0x55ce860f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:48.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.794+0000 7f9aa8600640 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x55ce86f9ec00 0x55ce860f4100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:48.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.794+0000 7f9aa8600640 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x55ce86f9e800 0x55ce860f5700 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x55ce86f981e0 tx=0x55ce86f48f00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:48.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.794+0000 7f9aa8600640 1 -- 172.21.15.39:0/2334176843 <== osd.6 v2:172.21.15.100:6808/36833 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-header out=14b] v0'0 uv10 ondisk = 0) v8 ==== 159+0+14 (crc 0 0 0) 0x55ce86dd4b40 con 0x55ce86f9ec00 2024-01-24T17:49:48.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.795+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- osd_op(unknown.0.0:2 14.b 14:d0630e4c:::mds0_sessionmap:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x55ce86f9f400 con 0x55ce86f9ec00 2024-01-24T17:49:48.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.795+0000 7f9aa8600640 1 -- 172.21.15.39:0/2334176843 <== osd.6 v2:172.21.15.100:6808/36833 2 ==== osd_op_reply(2 mds0_sessionmap [omap-get-vals out=5b] v0'0 uv10 ondisk = 0) v8 ==== 159+0+5 (crc 0 0 0) 0x55ce86dd4b40 con 0x55ce86f9ec00 2024-01-24T17:49:48.796 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:49:48.796 INFO:teuthology.orchestra.run.smithi039.stdout: "0": { 2024-01-24T17:49:48.796 INFO:teuthology.orchestra.run.smithi039.stdout: "data": { 2024-01-24T17:49:48.797 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions": [] 2024-01-24T17:49:48.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-01-24T17:49:48.797 INFO:teuthology.orchestra.run.smithi039.stdout: "result": 0 2024-01-24T17:49:48.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-01-24T17:49:48.797 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:49:48.797 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:48.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x55ce86f9ec00 msgr2=0x55ce860f4100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:48.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x55ce86f9ec00 0x55ce860f4100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x55ce86f9e800 msgr2=0x55ce860f5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x55ce86f9e800 0x55ce860f5700 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x55ce86f981e0 tx=0x55ce86f48f00 comp rx=0 tx=0).stop 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d400 msgr2=0x55ce860f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 --2- 172.21.15.39:0/2334176843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce86f2d400 0x55ce860f4c00 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x55ce86f98d20 tx=0x55ce86e3ef60 comp rx=0 tx=0).stop 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aa9000640 1 -- 172.21.15.39:0/2334176843 reap_dead start 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 shutdown_connections 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 >> 172.21.15.39:0/2334176843 conn(0x55ce86f2cc00 msgr2=0x55ce8614b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 shutdown_connections 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.796+0000 7f9aab839100 1 -- 172.21.15.39:0/2334176843 wait complete. 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 -- 172.21.15.39:0/1452737119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce8606dc00 msgr2=0x55ce860f3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce8606dc00 0x55ce860f3080 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x55ce86e90ba0 tx=0x55ce86e3f140 comp rx=0 tx=0).stop 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 -- 172.21.15.39:0/1452737119 shutdown_connections 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ce8606dc00 0x55ce860f3080 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x55ce8606d800 0x55ce860f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 --2- 172.21.15.39:0/1452737119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x55ce8606cc00 0x55ce860f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:48.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 -- 172.21.15.39:0/1452737119 >> 172.21.15.39:0/1452737119 conn(0x55ce8606d000 msgr2=0x55ce8614a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:48.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 -- 172.21.15.39:0/1452737119 shutdown_connections 2024-01-24T17:49:48.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:48.797+0000 7f9aab839100 1 -- 172.21.15.39:0/1452737119 wait complete. 2024-01-24T17:49:48.803 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '0', 'show', 'session'] in time 0:00:00.129051, result: { "0": { "data": { "sessions": [] }, "result": 0 } } 2024-01-24T17:49:48.803 INFO:tasks.cephfs.test_sessionmap:SessionMap: { "0": { "data": { "sessions": [] }, "result": 0 } } 2024-01-24T17:49:48.803 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-24T17:49:48.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:49:48.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-24T17:49:49.029 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-24T17:49:49.029 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:49:49.029 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:49.088 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:49:49.088 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:49:49.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:49.088 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-24T17:49:49.146 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-24T17:49:49.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:49:49.147 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:49:49.147 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:49.205 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:49:49.205 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:49:49.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:49.205 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-24T17:49:49.258 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-01-24T17:49:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:49:49.260 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:49:49.260 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:49.319 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:49:49.319 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:49:49.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:49.320 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-24T17:49:49.376 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-24T17:49:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:49:49.378 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:49:49.378 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:49.436 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:49:49.436 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:49:49.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:49:49.436 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-24T17:49:49.490 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-24T17:49:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:49:49.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:49:49.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.722+0000 7fd188e00640 1 Processor -- start 2024-01-24T17:49:49.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.722+0000 7fd188e00640 1 -- start start 2024-01-24T17:49:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.724+0000 7fd188e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd1840adb80 0x7fd1840ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.724+0000 7fd188e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd18415ade0 con 0x7fd184071d40 2024-01-24T17:49:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.724+0000 7fd188e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd18415bb70 con 0x7fd18415b5f0 2024-01-24T17:49:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.724+0000 7fd188e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1840b3540 con 0x7fd1840adb80 2024-01-24T17:49:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.724+0000 7fd17be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd184071d40 0x7fd1840ca490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56078/0 (socket says 172.21.15.39:56078) 2024-01-24T17:49:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.724+0000 7fd17be00640 1 -- 172.21.15.39:0/511376232 learned_addr learned my addr 172.21.15.39:0/511376232 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.724+0000 7fd182800640 1 --2- 172.21.15.39:0/511376232 >> v2:172.21.15.39:3300/0 conn(0x7fd1840adb80 0x7fd1840ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:49.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.725+0000 7fd181400640 1 -- 172.21.15.39:0/511376232 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973728462 0 0) 0x7fd18415ade0 con 0x7fd184071d40 2024-01-24T17:49:49.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.725+0000 7fd181400640 1 -- 172.21.15.39:0/511376232 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd164003520 con 0x7fd184071d40 2024-01-24T17:49:49.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.725+0000 7fd182800640 1 -- 172.21.15.39:0/511376232 >> v1:172.21.15.100:6789/0 conn(0x7fd18415b5f0 legacy=0x7fd184071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:49.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.725+0000 7fd182800640 1 -- 172.21.15.39:0/511376232 >> v1:172.21.15.39:6789/0 conn(0x7fd184071d40 legacy=0x7fd1840ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:49.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.725+0000 7fd182800640 1 -- 172.21.15.39:0/511376232 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18415be10 con 0x7fd1840adb80 2024-01-24T17:49:49.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.726+0000 7fd182800640 1 --2- 172.21.15.39:0/511376232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840adb80 0x7fd1840ad610 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009d90 tx=0x7fd17c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7228be2b5918f86b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.726+0000 7fd181400640 1 -- 172.21.15.39:0/511376232 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c02cd00 con 0x7fd1840adb80 2024-01-24T17:49:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.726+0000 7fd181400640 1 -- 172.21.15.39:0/511376232 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c02ce60 con 0x7fd1840adb80 2024-01-24T17:49:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.726+0000 7fd181400640 1 -- 172.21.15.39:0/511376232 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c034660 con 0x7fd1840adb80 2024-01-24T17:49:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.727+0000 7fd188e00640 1 -- 172.21.15.39:0/511376232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840adb80 msgr2=0x7fd1840ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.727+0000 7fd188e00640 1 --2- 172.21.15.39:0/511376232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840adb80 0x7fd1840ad610 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009d90 tx=0x7fd17c02bf50 comp rx=0 tx=0).stop 2024-01-24T17:49:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.727+0000 7fd188e00640 1 -- 172.21.15.39:0/511376232 shutdown_connections 2024-01-24T17:49:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.727+0000 7fd188e00640 1 --2- 172.21.15.39:0/511376232 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1840adb80 0x7fd1840ad610 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.727+0000 7fd188e00640 1 -- 172.21.15.39:0/511376232 >> 172.21.15.39:0/511376232 conn(0x7fd18406a220 msgr2=0x7fd1840ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.727+0000 7fd188e00640 1 -- 172.21.15.39:0/511376232 shutdown_connections 2024-01-24T17:49:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.727+0000 7fd188e00640 1 -- 172.21.15.39:0/511376232 wait complete. 2024-01-24T17:49:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.729+0000 7fd188e00640 1 Processor -- start 2024-01-24T17:49:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.729+0000 7fd188e00640 1 -- start start 2024-01-24T17:49:49.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd188e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd184071d40 0x7fd1840cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:49.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd188e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1840adb80 0x7fd18415a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:49.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd188e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd18415b5f0 0x7fd1840af0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:49.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd188e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1840cd780 con 0x7fd184071d40 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd188e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1840ccbb0 con 0x7fd18415b5f0 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd188e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1840cce50 con 0x7fd1840adb80 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd181e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd184071d40 0x7fd1840cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd181e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd184071d40 0x7fd1840cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54426/0 (socket says 172.21.15.39:54426) 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.730+0000 7fd181e00640 1 -- 172.21.15.39:0/1643287676 learned_addr learned my addr 172.21.15.39:0/1643287676 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.731+0000 7fd182800640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd18415b5f0 0x7fd1840af0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.731+0000 7fd17be00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1840adb80 0x7fd18415a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.731+0000 7fd181e00640 1 -- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1840adb80 msgr2=0x7fd18415a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.731+0000 7fd181e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1840adb80 0x7fd18415a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.731+0000 7fd181e00640 1 -- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd18415b5f0 msgr2=0x7fd1840af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.731+0000 7fd181e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd18415b5f0 0x7fd1840af0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.731+0000 7fd181e00640 1 -- 172.21.15.39:0/1643287676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c009a70 con 0x7fd184071d40 2024-01-24T17:49:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.731+0000 7fd181e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd184071d40 0x7fd1840cdc10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c450 tx=0x7fd17400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.732+0000 7fd17a000640 1 -- 172.21.15.39:0/1643287676 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1740052d0 con 0x7fd184071d40 2024-01-24T17:49:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.732+0000 7fd17a000640 1 -- 172.21.15.39:0/1643287676 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd174005450 con 0x7fd184071d40 2024-01-24T17:49:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.732+0000 7fd17a000640 1 -- 172.21.15.39:0/1643287676 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174003880 con 0x7fd184071d40 2024-01-24T17:49:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.732+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1840cd130 con 0x7fd184071d40 2024-01-24T17:49:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.732+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1840ce150 con 0x7fd184071d40 2024-01-24T17:49:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.733+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd184073f10 con 0x7fd184071d40 2024-01-24T17:49:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.734+0000 7fd17a000640 1 -- 172.21.15.39:0/1643287676 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd1740039e0 con 0x7fd184071d40 2024-01-24T17:49:49.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.735+0000 7fd17a000640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:49.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.735+0000 7fd17a000640 1 -- 172.21.15.39:0/1643287676 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fd174088620 con 0x7fd184071d40 2024-01-24T17:49:49.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.735+0000 7fd17be00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:49.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.737+0000 7fd17be00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd1840cc9b0 tx=0x7fd16c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:49.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.740+0000 7fd17a000640 1 -- 172.21.15.39:0/1643287676 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd174053ac0 con 0x7fd184071d40 2024-01-24T17:49:49.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.971+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1840c9e00 con 0x7fd184071d40 2024-01-24T17:49:49.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.972+0000 7fd17a000640 1 -- 172.21.15.39:0/1643287676 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4759 (secure 0 0 0) 0x7fd1740570e0 con 0x7fd184071d40 2024-01-24T17:49:49.973 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:49.973 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:49:19.498543+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8118,"mds_1":8100},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8100":{"gid":8100,"name":"a","rank":1,"incarnation":70,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6833/3903887009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3903887009},{"type":"v1","addr":"172.21.15.39:6833","nonce":3903887009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8118":{"gid":8118,"name":"c","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/1608906778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1608906778},{"type":"v1","addr":"172.21.15.39:6835","nonce":1608906778}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:49:49.974 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 72 2024-01-24T17:49:49.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 msgr2=0x7fd15c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:49.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd1840cc9b0 tx=0x7fd16c009040 comp rx=0 tx=0).stop 2024-01-24T17:49:49.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd184071d40 msgr2=0x7fd1840cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:49.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd184071d40 0x7fd1840cdc10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c450 tx=0x7fd17400c910 comp rx=0 tx=0).stop 2024-01-24T17:49:49.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 shutdown_connections 2024-01-24T17:49:49.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd18415b5f0 0x7fd1840af0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:49.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd1840adb80 0x7fd18415a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:49.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd15c06a430 0x7fd15c06c8f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:49.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 --2- 172.21.15.39:0/1643287676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd184071d40 0x7fd1840cdc10 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:49.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 >> 172.21.15.39:0/1643287676 conn(0x7fd18406a220 msgr2=0x7fd1840ca5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:49.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 shutdown_connections 2024-01-24T17:49:49.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:49.975+0000 7fd188e00640 1 -- 172.21.15.39:0/1643287676 wait complete. 2024-01-24T17:49:49.995 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:49:50.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.216+0000 7fe4aae00640 1 Processor -- start 2024-01-24T17:49:50.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.216+0000 7fe4aae00640 1 -- start start 2024-01-24T17:49:50.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.217+0000 7fe4aae00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe4a40c9450 0x7fe4a415e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4aae00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 0x7fe4a406fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4aae00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a415f330 con 0x7fe4a4070410 2024-01-24T17:49:50.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4aae00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a4072140 con 0x7fe4a40c9450 2024-01-24T17:49:50.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4aae00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a415ea90 con 0x7fe4a4071ce0 2024-01-24T17:49:50.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4aa400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe4a4070410 0x7fe4a415b460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56082/0 (socket says 172.21.15.39:56082) 2024-01-24T17:49:50.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4aa400640 1 -- 172.21.15.39:0/3173663774 learned_addr learned my addr 172.21.15.39:0/3173663774 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:50.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4a9a00640 1 --2- 172.21.15.39:0/3173663774 >> v2:172.21.15.100:3300/0 conn(0x7fe4a40c9450 0x7fe4a415e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4a9000640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 0x7fe4a406fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909958631 0 0) 0x7fe4a415f330 con 0x7fe4a4070410 2024-01-24T17:49:50.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.218+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe484003520 con 0x7fe4a4070410 2024-01-24T17:49:50.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 163167416 0 0) 0x7fe484003520 con 0x7fe4a4070410 2024-01-24T17:49:50.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 msgr2=0x7fe4a406fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 0x7fe4a406fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 >> v2:172.21.15.100:3300/0 conn(0x7fe4a40c9450 msgr2=0x7fe4a415e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 --2- 172.21.15.39:0/3173663774 >> v2:172.21.15.100:3300/0 conn(0x7fe4a40c9450 0x7fe4a415e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a415ed70 con 0x7fe4a4070410 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fe49c0019f0 con 0x7fe4a4070410 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 >> v1:172.21.15.39:6789/0 conn(0x7fe4a4070410 legacy=0x7fe4a415b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe484003970 0x7fe484003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4840047c0 0x7fe484004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 0x7fe484005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a4072140 con 0x7fe4840047c0 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a415ea90 con 0x7fe484003970 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48400b980 con 0x7fe4a4071ce0 2024-01-24T17:49:50.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.219+0000 7fe4a9000640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4840047c0 0x7fe484004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.220+0000 7fe4a9a00640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe484003970 0x7fe484003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.220+0000 7fe4aa400640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 0x7fe484005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.220+0000 7fe4a9000640 1 -- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 msgr2=0x7fe484005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:50.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.220+0000 7fe4a9000640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 0x7fe484005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.220+0000 7fe4a9000640 1 -- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe484003970 msgr2=0x7fe484003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:50.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.220+0000 7fe4a9000640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe484003970 0x7fe484003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.220+0000 7fe4a9000640 1 -- 172.21.15.39:0/3173663774 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484003520 con 0x7fe4840047c0 2024-01-24T17:49:50.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.220+0000 7fe4aa400640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4a4071ce0 0x7fe484005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:50.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.221+0000 7fe4a9000640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4840047c0 0x7fe484004c40 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009af0 tx=0x7fe4840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee50155fde2bc7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:50.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.221+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c02c6b0 con 0x7fe4840047c0 2024-01-24T17:49:50.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c02c810 con 0x7fe4840047c0 2024-01-24T17:49:50.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4a3e00640 1 -- 172.21.15.39:0/3173663774 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c02cba0 con 0x7fe4840047c0 2024-01-24T17:49:50.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4aae00640 1 -- 172.21.15.39:0/3173663774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4840047c0 msgr2=0x7fe484004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4aae00640 1 --2- 172.21.15.39:0/3173663774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4840047c0 0x7fe484004c40 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009af0 tx=0x7fe4840057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4aa400640 1 -- 172.21.15.39:0/3173663774 reap_dead start 2024-01-24T17:49:50.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4aae00640 1 -- 172.21.15.39:0/3173663774 shutdown_connections 2024-01-24T17:49:50.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4aae00640 1 -- 172.21.15.39:0/3173663774 >> 172.21.15.39:0/3173663774 conn(0x7fe4a406bf30 msgr2=0x7fe4a406c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:50.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4aae00640 1 -- 172.21.15.39:0/3173663774 shutdown_connections 2024-01-24T17:49:50.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.222+0000 7fe4aae00640 1 -- 172.21.15.39:0/3173663774 wait complete. 2024-01-24T17:49:50.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.224+0000 7fe4aae00640 1 Processor -- start 2024-01-24T17:49:50.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.224+0000 7fe4aae00640 1 -- start start 2024-01-24T17:49:50.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aae00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4840047c0 0x7fe4a40cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aae00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4a40c5500 0x7fe4a41570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4a40af030 0x7fe4a40af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40b3df0 con 0x7fe4a40af030 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aae00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40afa00 con 0x7fe4a40c5500 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aae00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40b35e0 con 0x7fe4840047c0 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aa400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4a40af030 0x7fe4a40af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aa400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4a40af030 0x7fe4a40af4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54432/0 (socket says 172.21.15.39:54432) 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.225+0000 7fe4aa400640 1 -- 172.21.15.39:0/2534710550 learned_addr learned my addr 172.21.15.39:0/2534710550 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4a9a00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4840047c0 0x7fe4a40cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4a9000640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4a40c5500 0x7fe4a41570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4aa400640 1 -- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4840047c0 msgr2=0x7fe4a40cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4aa400640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4840047c0 0x7fe4a40cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4aa400640 1 -- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4a40c5500 msgr2=0x7fe4a41570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4aa400640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4a40c5500 0x7fe4a41570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4aa400640 1 -- 172.21.15.39:0/2534710550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a41f0a80 con 0x7fe4a40af030 2024-01-24T17:49:50.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4a9000640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4a40c5500 0x7fe4a41570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:50.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4aa400640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4a40af030 0x7fe4a40af4c0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40cbfb0 tx=0x7fe49c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:50.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.226+0000 7fe4a1600640 1 -- 172.21.15.39:0/2534710550 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c004d60 con 0x7fe4a40af030 2024-01-24T17:49:50.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.227+0000 7fe4a1600640 1 -- 172.21.15.39:0/2534710550 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c00bb80 con 0x7fe4a40af030 2024-01-24T17:49:50.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.227+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a40b3830 con 0x7fe4a40af030 2024-01-24T17:49:50.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.227+0000 7fe4a1600640 1 -- 172.21.15.39:0/2534710550 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c007110 con 0x7fe4a40af030 2024-01-24T17:49:50.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.227+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a4157600 con 0x7fe4a40af030 2024-01-24T17:49:50.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.228+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a40cd980 con 0x7fe4a40af030 2024-01-24T17:49:50.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.229+0000 7fe4a1600640 1 -- 172.21.15.39:0/2534710550 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe49c007270 con 0x7fe4a40af030 2024-01-24T17:49:50.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.230+0000 7fe4a1600640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe47c06a430 0x7fe47c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.230+0000 7fe4a1600640 1 -- 172.21.15.39:0/2534710550 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fe49c088ea0 con 0x7fe4a40af030 2024-01-24T17:49:50.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.230+0000 7fe4a9a00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe47c06a430 0x7fe47c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.231+0000 7fe4a9a00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe47c06a430 0x7fe47c06c8f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40cd3d0 tx=0x7fe494008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:50.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.234+0000 7fe4a1600640 1 -- 172.21.15.39:0/2534710550 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe49c054340 con 0x7fe4a40af030 2024-01-24T17:49:50.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.428+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4a404e840 con 0x7fe4a40af030 2024-01-24T17:49:50.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.430+0000 7fe4a1600640 1 -- 172.21.15.39:0/2534710550 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16529 (secure 0 0 0) 0x7fe49c057960 con 0x7fe4a40af030 2024-01-24T17:49:50.431 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:50.431 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":106,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:49:17.472389+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.0","mappings":[{"from":4,"to":0}]},{"pgid":"14.18","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6833/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.39:6832/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.100:6836/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.100:6834/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.39:6835/3495904204":"2024-01-25T17:48:58.377999+0000","172.21.15.39:6834/3495904204":"2024-01-25T17:48:58.377999+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe47c06a430 msgr2=0x7fe47c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe47c06a430 0x7fe47c06c8f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40cd3d0 tx=0x7fe494008040 comp rx=0 tx=0).stop 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4a40af030 msgr2=0x7fe4a40af4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4a40af030 0x7fe4a40af4c0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40cbfb0 tx=0x7fe49c00b4a0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 shutdown_connections 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe47c06a430 0x7fe47c06c8f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4a40af030 0x7fe4a40af4c0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe4a40c5500 0x7fe4a41570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.432+0000 7fe4aae00640 1 --2- 172.21.15.39:0/2534710550 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe4840047c0 0x7fe4a40cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.433+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 >> 172.21.15.39:0/2534710550 conn(0x7fe4a406bf30 msgr2=0x7fe4a406c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:50.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.433+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 shutdown_connections 2024-01-24T17:49:50.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.433+0000 7fe4aae00640 1 -- 172.21.15.39:0/2534710550 wait complete. 2024-01-24T17:49:50.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T17:41:22.282091+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T17:48:24.515179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T17:48:25.907386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T17:49:50.455 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T17:49:50.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.684+0000 7fd032000640 1 Processor -- start 2024-01-24T17:49:50.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.684+0000 7fd032000640 1 -- start start 2024-01-24T17:49:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.686+0000 7fd032000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd02c0b35a0 0x7fd02c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.686+0000 7fd032000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0d1430 con 0x7fd02c15cab0 2024-01-24T17:49:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.686+0000 7fd032000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0ce830 con 0x7fd02c15ce90 2024-01-24T17:49:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.686+0000 7fd032000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02c0d01e0 con 0x7fd02c0b35a0 2024-01-24T17:49:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.686+0000 7fd030c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd02c15cab0 0x7fd02c0d0610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56086/0 (socket says 172.21.15.39:56086) 2024-01-24T17:49:50.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.686+0000 7fd030c00640 1 -- 172.21.15.39:0/4096949266 learned_addr learned my addr 172.21.15.39:0/4096949266 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd031600640 1 --2- 172.21.15.39:0/4096949266 >> v2:172.21.15.100:3300/0 conn(0x7fd02c0b35a0 0x7fd02c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292276790 0 0) 0x7fd02c0d1430 con 0x7fd02c15cab0 2024-01-24T17:49:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd00c003520 con 0x7fd02c15cab0 2024-01-24T17:49:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534022906 0 0) 0x7fd00c003520 con 0x7fd02c15cab0 2024-01-24T17:49:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 >> v2:172.21.15.100:3300/0 conn(0x7fd02c0b35a0 msgr2=0x7fd02c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:50.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 --2- 172.21.15.39:0/4096949266 >> v2:172.21.15.100:3300/0 conn(0x7fd02c0b35a0 0x7fd02c0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 >> v1:172.21.15.100:6789/0 conn(0x7fd02c15ce90 legacy=0x7fd02c0d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0ceb10 con 0x7fd02c15cab0 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fd01c0017f0 con 0x7fd02c15cab0 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.687+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 >> v1:172.21.15.39:6789/0 conn(0x7fd02c15cab0 legacy=0x7fd02c0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd02be00640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c003970 0x7fd00c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd02be00640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00c0047c0 0x7fd00c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd02be00640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00c005180 0x7fd00c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c00bff0 con 0x7fd00c003970 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c00c160 con 0x7fd00c005180 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c00c320 con 0x7fd00c0047c0 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd030c00640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c003970 0x7fd00c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd030c00640 1 -- 172.21.15.39:0/4096949266 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00c0047c0 msgr2=0x7fd00c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd030c00640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd00c0047c0 0x7fd00c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd030c00640 1 -- 172.21.15.39:0/4096949266 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00c005180 msgr2=0x7fd00c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:50.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd030c00640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00c005180 0x7fd00c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd030c00640 1 -- 172.21.15.39:0/4096949266 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c003520 con 0x7fd00c003970 2024-01-24T17:49:50.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.688+0000 7fd031600640 1 -- 172.21.15.39:0/4096949266 reap_dead start 2024-01-24T17:49:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.689+0000 7fd030c00640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c003970 0x7fd00c003d90 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00c190 tx=0x7fd00c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968dcea595bed250 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c007270 con 0x7fd00c003970 2024-01-24T17:49:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c0073f0 con 0x7fd00c003970 2024-01-24T17:49:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd02be00640 1 -- 172.21.15.39:0/4096949266 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c02c8b0 con 0x7fd00c003970 2024-01-24T17:49:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd032000640 1 -- 172.21.15.39:0/4096949266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c003970 msgr2=0x7fd00c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd032000640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c003970 0x7fd00c003d90 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00c190 tx=0x7fd00c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd032000640 1 -- 172.21.15.39:0/4096949266 shutdown_connections 2024-01-24T17:49:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd032000640 1 --2- 172.21.15.39:0/4096949266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c003970 0x7fd00c003d90 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd032000640 1 -- 172.21.15.39:0/4096949266 >> 172.21.15.39:0/4096949266 conn(0x7fd02c0d2830 msgr2=0x7fd02c0d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.690+0000 7fd032000640 1 -- 172.21.15.39:0/4096949266 shutdown_connections 2024-01-24T17:49:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.691+0000 7fd032000640 1 -- 172.21.15.39:0/4096949266 wait complete. 2024-01-24T17:49:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.692+0000 7fd032000640 1 Processor -- start 2024-01-24T17:49:50.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.692+0000 7fd032000640 1 -- start start 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.693+0000 7fd032000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00c003970 0x7fd02c159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.693+0000 7fd032000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd02c1598e0 0x7fd02c0d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.693+0000 7fd032000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd02c159e30 0x7fd02c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.693+0000 7fd032000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c15f2e0 con 0x7fd02c1598e0 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.693+0000 7fd032000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0b6170 con 0x7fd00c003970 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.693+0000 7fd032000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0b54f0 con 0x7fd02c159e30 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd023e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd02c1598e0 0x7fd02c0d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd023e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd02c1598e0 0x7fd02c0d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54460/0 (socket says 172.21.15.39:54460) 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd023e00640 1 -- 172.21.15.39:0/4049765300 learned_addr learned my addr 172.21.15.39:0/4049765300 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:50.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd030c00640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00c003970 0x7fd02c159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd031600640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd02c159e30 0x7fd02c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd023e00640 1 -- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd02c159e30 msgr2=0x7fd02c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd023e00640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd02c159e30 0x7fd02c0d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd023e00640 1 -- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00c003970 msgr2=0x7fd02c159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd023e00640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00c003970 0x7fd02c159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.694+0000 7fd023e00640 1 -- 172.21.15.39:0/4049765300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0b57d0 con 0x7fd02c1598e0 2024-01-24T17:49:50.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.695+0000 7fd023e00640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd02c1598e0 0x7fd02c0d0050 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fd01400c3c0 tx=0x7fd01400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.695+0000 7fd029600640 1 -- 172.21.15.39:0/4049765300 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0140052d0 con 0x7fd02c1598e0 2024-01-24T17:49:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.695+0000 7fd029600640 1 -- 172.21.15.39:0/4049765300 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd014005450 con 0x7fd02c1598e0 2024-01-24T17:49:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.695+0000 7fd029600640 1 -- 172.21.15.39:0/4049765300 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd014007260 con 0x7fd02c1598e0 2024-01-24T17:49:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.695+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c0b5ab0 con 0x7fd02c1598e0 2024-01-24T17:49:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.695+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c0cc6e0 con 0x7fd02c1598e0 2024-01-24T17:49:50.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.696+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c04e840 con 0x7fd02c1598e0 2024-01-24T17:49:50.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.697+0000 7fd029600640 1 -- 172.21.15.39:0/4049765300 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd0140073c0 con 0x7fd02c1598e0 2024-01-24T17:49:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.698+0000 7fd029600640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd00406a430 0x7fd00406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.698+0000 7fd029600640 1 -- 172.21.15.39:0/4049765300 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fd014087de0 con 0x7fd02c1598e0 2024-01-24T17:49:50.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.699+0000 7fd030c00640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd00406a430 0x7fd00406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:50.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.700+0000 7fd030c00640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd00406a430 0x7fd00406c8f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd01c034000 tx=0x7fd01c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:50.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.703+0000 7fd029600640 1 -- 172.21.15.39:0/4049765300 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd014053280 con 0x7fd02c1598e0 2024-01-24T17:49:50.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.895+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd02c0af7b0 con 0x7fd02c1598e0 2024-01-24T17:49:50.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.896+0000 7fd029600640 1 -- 172.21.15.39:0/4049765300 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0140568a0 con 0x7fd02c1598e0 2024-01-24T17:49:50.897 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd00406a430 msgr2=0x7fd00406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd00406a430 0x7fd00406c8f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd01c034000 tx=0x7fd01c009030 comp rx=0 tx=0).stop 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd02c1598e0 msgr2=0x7fd02c0d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd02c1598e0 0x7fd02c0d0050 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fd01400c3c0 tx=0x7fd01400c880 comp rx=0 tx=0).stop 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 shutdown_connections 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd02c159e30 0x7fd02c0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd00406a430 0x7fd00406c8f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd02c1598e0 0x7fd02c0d0050 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 --2- 172.21.15.39:0/4049765300 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd00c003970 0x7fd02c159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:50.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 >> 172.21.15.39:0/4049765300 conn(0x7fd02c0d2830 msgr2=0x7fd02c0d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:50.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 shutdown_connections 2024-01-24T17:49:50.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:50.899+0000 7fd032000640 1 -- 172.21.15.39:0/4049765300 wait complete. 2024-01-24T17:49:50.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T17:49:51.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.143+0000 7fd0f1e00640 1 Processor -- start 2024-01-24T17:49:51.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.143+0000 7fd0f1e00640 1 -- start start 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.144+0000 7fd0f1e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd0ec15ca10 0x7fd0ec0d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.144+0000 7fd0f1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0cded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.144+0000 7fd0f1e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd0ec0d1150 0x7fd0ec0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.144+0000 7fd0f1e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec0ce5b0 con 0x7fd0ec0d1150 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.144+0000 7fd0f1e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec0d0670 con 0x7fd0ec15ca10 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.144+0000 7fd0f1e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0d0950 con 0x7fd0ec15cdf0 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd0ec0d1150 0x7fd0ec0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f1400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd0ec0d1150 0x7fd0ec0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54466/0 (socket says 172.21.15.39:54466) 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f1400640 1 -- 172.21.15.39:0/2419851774 learned_addr learned my addr 172.21.15.39:0/2419851774 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:51.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0ebe00640 1 --2- 172.21.15.39:0/2419851774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f0a00640 1 --2- 172.21.15.39:0/2419851774 >> v2:172.21.15.100:3300/0 conn(0x7fd0ec15ca10 0x7fd0ec0d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f1400640 1 -- 172.21.15.39:0/2419851774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 msgr2=0x7fd0ec0cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:51.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f1400640 1 --2- 172.21.15.39:0/2419851774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f1400640 1 -- 172.21.15.39:0/2419851774 >> v2:172.21.15.100:3300/0 conn(0x7fd0ec15ca10 msgr2=0x7fd0ec0d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:51.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f1400640 1 --2- 172.21.15.39:0/2419851774 >> v2:172.21.15.100:3300/0 conn(0x7fd0ec15ca10 0x7fd0ec0d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.145+0000 7fd0f1400640 1 -- 172.21.15.39:0/2419851774 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0d0c30 con 0x7fd0ec0d1150 2024-01-24T17:49:51.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.147+0000 7fd0f1400640 1 --2- 172.21.15.39:0/2419851774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec0cf310 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0cfff0 tx=0x7fd0e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0037e76aad035c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:51.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.147+0000 7fd0eb400640 1 -- 172.21.15.39:0/2419851774 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4031870 con 0x7fd0ec0d1150 2024-01-24T17:49:51.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.147+0000 7fd0eb400640 1 -- 172.21.15.39:0/2419851774 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e40319d0 con 0x7fd0ec0d1150 2024-01-24T17:49:51.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.147+0000 7fd0eb400640 1 -- 172.21.15.39:0/2419851774 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4031d60 con 0x7fd0ec0d1150 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 -- 172.21.15.39:0/2419851774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 msgr2=0x7fd0ec0cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/2419851774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec0cf310 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0cfff0 tx=0x7fd0e402b910 comp rx=0 tx=0).stop 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 -- 172.21.15.39:0/2419851774 shutdown_connections 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/2419851774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec0cf310 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/2419851774 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/2419851774 >> v2:172.21.15.100:3300/0 conn(0x7fd0ec15ca10 0x7fd0ec0d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 -- 172.21.15.39:0/2419851774 >> 172.21.15.39:0/2419851774 conn(0x7fd0ec0d2920 msgr2=0x7fd0ec0d2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 -- 172.21.15.39:0/2419851774 shutdown_connections 2024-01-24T17:49:51.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.148+0000 7fd0f1e00640 1 -- 172.21.15.39:0/2419851774 wait complete. 2024-01-24T17:49:51.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.150+0000 7fd0f1e00640 1 Processor -- start 2024-01-24T17:49:51.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.150+0000 7fd0f1e00640 1 -- start start 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec15a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f1e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0ec15ca10 0x7fd0ec0c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f1e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f1e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0c7f30 con 0x7fd0ec0d1150 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f1e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0c6d50 con 0x7fd0ec15ca10 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f1e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0c7030 con 0x7fd0ec15cdf0 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f0a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec15a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54482/0 (socket says 172.21.15.39:54482) 2024-01-24T17:49:51.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f0a00640 1 -- 172.21.15.39:0/1547213119 learned_addr learned my addr 172.21.15.39:0/1547213119 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0ebe00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0ec15ca10 0x7fd0ec0c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f1400640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f0a00640 1 -- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 msgr2=0x7fd0ec0b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.151+0000 7fd0f0a00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0b0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0f0a00640 1 -- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0ec15ca10 msgr2=0x7fd0ec0c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0f0a00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0ec15ca10 0x7fd0ec0c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0f0a00640 1 -- 172.21.15.39:0/1547213119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e40094d0 con 0x7fd0ec0d1150 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0f1400640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0f0a00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec15a050 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400c7f0 tx=0x7fd0d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0e8c00640 1 -- 172.21.15.39:0/1547213119 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4004220 con 0x7fd0ec0d1150 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0e8c00640 1 -- 172.21.15.39:0/1547213119 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d4004380 con 0x7fd0ec0d1150 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0e8c00640 1 -- 172.21.15.39:0/1547213119 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4007860 con 0x7fd0ec0d1150 2024-01-24T17:49:51.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec0c7310 con 0x7fd0ec0d1150 2024-01-24T17:49:51.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.152+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec15a590 con 0x7fd0ec0d1150 2024-01-24T17:49:51.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.153+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec0c8130 con 0x7fd0ec0d1150 2024-01-24T17:49:51.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.154+0000 7fd0e8c00640 1 -- 172.21.15.39:0/1547213119 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd0d4007bd0 con 0x7fd0ec0d1150 2024-01-24T17:49:51.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.155+0000 7fd0e8c00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd0c406a460 0x7fd0c406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.155+0000 7fd0e8c00640 1 -- 172.21.15.39:0/1547213119 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fd0d4088a00 con 0x7fd0ec0d1150 2024-01-24T17:49:51.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.156+0000 7fd0ebe00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd0c406a460 0x7fd0c406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.157+0000 7fd0ebe00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd0c406a460 0x7fd0c406c920 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0afe20 tx=0x7fd0dc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:51.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.159+0000 7fd0e8c00640 1 -- 172.21.15.39:0/1547213119 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd0ec0c8130 con 0x7fd0ec0d1150 2024-01-24T17:49:51.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.354+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0ec0c8130 con 0x7fd0ec0d1150 2024-01-24T17:49:51.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.355+0000 7fd0e8c00640 1 -- 172.21.15.39:0/1547213119 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd0d401e0a0 con 0x7fd0ec0d1150 2024-01-24T17:49:51.355 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-01-24T17:49:51.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.357+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd0c406a460 msgr2=0x7fd0c406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.357+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd0c406a460 0x7fd0c406c920 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0afe20 tx=0x7fd0dc00b040 comp rx=0 tx=0).stop 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.357+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 msgr2=0x7fd0ec15a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.357+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec15a050 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400c7f0 tx=0x7fd0d400ccb0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.358+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 shutdown_connections 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.358+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd0ec15cdf0 0x7fd0ec0b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.358+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd0ec15ca10 0x7fd0ec0c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.358+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd0c406a460 0x7fd0c406c920 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.358+0000 7fd0f1e00640 1 --2- 172.21.15.39:0/1547213119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0ec0d1150 0x7fd0ec15a050 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.358+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 >> 172.21.15.39:0/1547213119 conn(0x7fd0ec0d2920 msgr2=0x7fd0ec15ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.358+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 shutdown_connections 2024-01-24T17:49:51.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.358+0000 7fd0f1e00640 1 -- 172.21.15.39:0/1547213119 wait complete. 2024-01-24T17:49:51.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T17:49:51.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.597+0000 7f36cfe00640 1 Processor -- start 2024-01-24T17:49:51.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.597+0000 7f36cfe00640 1 -- start start 2024-01-24T17:49:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.598+0000 7f36cfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36d015cab0 0x7f36d00ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.598+0000 7f36cfe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f36d00d0d20 0x7f36d00b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.598+0000 7f36cfe00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00ce830 con 0x7f36d015ce90 2024-01-24T17:49:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.598+0000 7f36cfe00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d00d01e0 con 0x7f36d015cab0 2024-01-24T17:49:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cfe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d00ceb10 con 0x7f36d00d0d20 2024-01-24T17:49:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36d015cab0 0x7f36d00ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cea00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f36d015cab0 0x7f36d00ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54494/0 (socket says 172.21.15.39:54494) 2024-01-24T17:49:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cea00640 1 -- 172.21.15.39:0/1721063028 learned_addr learned my addr 172.21.15.39:0/1721063028 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cf400640 1 --2- 172.21.15.39:0/1721063028 >> v2:172.21.15.100:3300/0 conn(0x7f36d00d0d20 0x7f36d00b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cea00640 1 -- 172.21.15.39:0/1721063028 >> v2:172.21.15.100:3300/0 conn(0x7f36d00d0d20 msgr2=0x7f36d00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cea00640 1 --2- 172.21.15.39:0/1721063028 >> v2:172.21.15.100:3300/0 conn(0x7f36d00d0d20 0x7f36d00b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cea00640 1 -- 172.21.15.39:0/1721063028 >> v1:172.21.15.39:6789/0 conn(0x7f36d015ce90 legacy=0x7f36d00d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cea00640 1 -- 172.21.15.39:0/1721063028 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d00cedf0 con 0x7f36d015cab0 2024-01-24T17:49:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.599+0000 7f36cf400640 1 --2- 172.21.15.39:0/1721063028 >> v2:172.21.15.100:3300/0 conn(0x7f36d00d0d20 0x7f36d00b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.600+0000 7f36cea00640 1 --2- 172.21.15.39:0/1721063028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d015cab0 0x7f36d00ce130 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f36d008d0d0 tx=0x7f36c002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a6a453904599f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.601+0000 7f36cd600640 1 -- 172.21.15.39:0/1721063028 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0004230 con 0x7f36d015cab0 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.601+0000 7f36cd600640 1 -- 172.21.15.39:0/1721063028 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0004390 con 0x7f36d015cab0 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.601+0000 7f36cd600640 1 -- 172.21.15.39:0/1721063028 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c002f830 con 0x7f36d015cab0 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.601+0000 7f36cfe00640 1 -- 172.21.15.39:0/1721063028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d015cab0 msgr2=0x7f36d00ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.601+0000 7f36cfe00640 1 --2- 172.21.15.39:0/1721063028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d015cab0 0x7f36d00ce130 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f36d008d0d0 tx=0x7f36c002ba10 comp rx=0 tx=0).stop 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.602+0000 7f36cfe00640 1 -- 172.21.15.39:0/1721063028 shutdown_connections 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.602+0000 7f36cfe00640 1 --2- 172.21.15.39:0/1721063028 >> v2:172.21.15.100:3300/0 conn(0x7f36d00d0d20 0x7f36d00b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.602+0000 7f36cfe00640 1 --2- 172.21.15.39:0/1721063028 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d015cab0 0x7f36d00ce130 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.602+0000 7f36cfe00640 1 -- 172.21.15.39:0/1721063028 >> 172.21.15.39:0/1721063028 conn(0x7f36d00d2830 msgr2=0x7f36d00d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.602+0000 7f36cfe00640 1 -- 172.21.15.39:0/1721063028 shutdown_connections 2024-01-24T17:49:51.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.602+0000 7f36cfe00640 1 -- 172.21.15.39:0/1721063028 wait complete. 2024-01-24T17:49:51.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.603+0000 7f36cfe00640 1 Processor -- start 2024-01-24T17:49:51.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.604+0000 7f36cfe00640 1 -- start start 2024-01-24T17:49:51.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.604+0000 7f36cfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d00d0d20 0x7f36d0158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.604+0000 7f36cfe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36d015cab0 0x7f36d015f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.604+0000 7f36cfe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36d00b7690 0x7f36d00b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.604+0000 7f36cfe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d00b83c0 con 0x7f36d00d0d20 2024-01-24T17:49:51.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.604+0000 7f36cfe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d0159520 con 0x7f36d015cab0 2024-01-24T17:49:51.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.604+0000 7f36cfe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d00b7b20 con 0x7f36d00b7690 2024-01-24T17:49:51.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d00d0d20 0x7f36d0158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d00d0d20 0x7f36d0158fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54504/0 (socket says 172.21.15.39:54504) 2024-01-24T17:49:51.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cea00640 1 -- 172.21.15.39:0/3126348915 learned_addr learned my addr 172.21.15.39:0/3126348915 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:51.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cf400640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36d00b7690 0x7f36d00b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36ce000640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36d015cab0 0x7f36d015f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cea00640 1 -- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36d00b7690 msgr2=0x7f36d00b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cea00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36d00b7690 0x7f36d00b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cea00640 1 -- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36d015cab0 msgr2=0x7f36d015f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cea00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36d015cab0 0x7f36d015f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36cea00640 1 -- 172.21.15.39:0/3126348915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c0009460 con 0x7f36d00d0d20 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.605+0000 7f36ce000640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36d015cab0 0x7f36d015f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.606+0000 7f36cea00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d00d0d20 0x7f36d0158fe0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009940 tx=0x7f36c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.606+0000 7f36c6a00640 1 -- 172.21.15.39:0/3126348915 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c003d070 con 0x7f36d00d0d20 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.606+0000 7f36c6a00640 1 -- 172.21.15.39:0/3126348915 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0004500 con 0x7f36d00d0d20 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.606+0000 7f36c6a00640 1 -- 172.21.15.39:0/3126348915 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c002fe10 con 0x7f36d00d0d20 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.606+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d00b7e00 con 0x7f36d00d0d20 2024-01-24T17:49:51.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.606+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d015d420 con 0x7f36d00d0d20 2024-01-24T17:49:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.607+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d015d640 con 0x7f36d00d0d20 2024-01-24T17:49:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.608+0000 7f36c6a00640 1 -- 172.21.15.39:0/3126348915 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f36c0003610 con 0x7f36d00d0d20 2024-01-24T17:49:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.609+0000 7f36c6a00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36a806a430 0x7f36a806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:51.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.609+0000 7f36c6a00640 1 -- 172.21.15.39:0/3126348915 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f36c00aac50 con 0x7f36d00d0d20 2024-01-24T17:49:51.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.610+0000 7f36ce000640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36a806a430 0x7f36a806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:51.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.611+0000 7f36ce000640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36a806a430 0x7f36a806c8f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f36d00c90f0 tx=0x7f36b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:51.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.614+0000 7f36c6a00640 1 -- 172.21.15.39:0/3126348915 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f36c00b0020 con 0x7f36d00d0d20 2024-01-24T17:49:51.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.810+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f36d004e840 con 0x7f36d00d0d20 2024-01-24T17:49:51.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.811+0000 7f36c6a00640 1 -- 172.21.15.39:0/3126348915 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7f36c0076220 con 0x7f36d00d0d20 2024-01-24T17:49:51.812 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-01-24T17:49:51.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36a806a430 msgr2=0x7f36a806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36a806a430 0x7f36a806c8f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f36d00c90f0 tx=0x7f36b8009040 comp rx=0 tx=0).stop 2024-01-24T17:49:51.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d00d0d20 msgr2=0x7f36d0158fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:51.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d00d0d20 0x7f36d0158fe0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009940 tx=0x7f36c00046c0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 shutdown_connections 2024-01-24T17:49:51.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f36d00b7690 0x7f36d00b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f36d015cab0 0x7f36d015f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f36a806a430 0x7f36a806c8f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 --2- 172.21.15.39:0/3126348915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36d00d0d20 0x7f36d0158fe0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:51.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 >> 172.21.15.39:0/3126348915 conn(0x7f36d00d2830 msgr2=0x7f36d00cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:51.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 shutdown_connections 2024-01-24T17:49:51.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:51.814+0000 7f36cfe00640 1 -- 172.21.15.39:0/3126348915 wait complete. 2024-01-24T17:49:51.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:49:52.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.058+0000 7f66cb200640 1 Processor -- start 2024-01-24T17:49:52.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.059+0000 7f66cb200640 1 -- start start 2024-01-24T17:49:52.060 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:49:52.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.060+0000 7f66cb200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc15cab0 0x7f66cc0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.060+0000 7f66cb200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f66cc0d0d20 0x7f66cc0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.060+0000 7f66cb200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66cc0ce830 con 0x7f66cc15ce90 2024-01-24T17:49:52.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.060+0000 7f66cb200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66cc0d01e0 con 0x7f66cc0d0d20 2024-01-24T17:49:52.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.060+0000 7f66cb200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0ceb10 con 0x7f66cc15cab0 2024-01-24T17:49:52.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.061+0000 7f66c9400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f66cc15ce90 0x7f66cc0d0610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56110/0 (socket says 172.21.15.39:56110) 2024-01-24T17:49:52.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.061+0000 7f66c9400640 1 -- 172.21.15.39:0/3586312537 learned_addr learned my addr 172.21.15.39:0/3586312537 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:52.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.061+0000 7f66c9e00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc15cab0 0x7f66cc0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.061+0000 7f66ca800640 1 --2- 172.21.15.39:0/3586312537 >> v2:172.21.15.100:3300/0 conn(0x7f66cc0d0d20 0x7f66cc0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.061+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194904017 0 0) 0x7f66cc0ce830 con 0x7f66cc15ce90 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.061+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ac003520 con 0x7f66cc15ce90 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.061+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182370782 0 0) 0x7f66ac003520 con 0x7f66cc15ce90 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc15cab0 msgr2=0x7f66cc0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc15cab0 0x7f66cc0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 >> v2:172.21.15.100:3300/0 conn(0x7f66cc0d0d20 msgr2=0x7f66cc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 --2- 172.21.15.39:0/3586312537 >> v2:172.21.15.100:3300/0 conn(0x7f66cc0d0d20 0x7f66cc0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc0cedf0 con 0x7f66cc15ce90 2024-01-24T17:49:52.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f66b40019f0 con 0x7f66cc15ce90 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 >> v1:172.21.15.39:6789/0 conn(0x7f66cc15ce90 legacy=0x7f66cc0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66ac003970 0x7f66ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc15cab0 0x7f66ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac004d00 0x7f66ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0ceb10 con 0x7f66ac003970 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac00ba10 con 0x7f66ac004d00 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac00bb80 con 0x7f66cc15cab0 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c9e00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66ac003970 0x7f66ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c9400640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc15cab0 0x7f66ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c9e00640 1 -- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc15cab0 msgr2=0x7f66ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c9e00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc15cab0 0x7f66ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66c9e00640 1 -- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac004d00 msgr2=0x7f66ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.062+0000 7f66ca800640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac004d00 0x7f66ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.063+0000 7f66c9e00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac004d00 0x7f66ac005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.063+0000 7f66c9e00640 1 -- 172.21.15.39:0/3586312537 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac003520 con 0x7f66ac003970 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.064+0000 7f66c9e00640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66ac003970 0x7f66ac003d90 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009ab0 tx=0x7f66ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2290ea722693db server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.064+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc005990 con 0x7f66ac003970 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.064+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc005af0 con 0x7f66ac003970 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.064+0000 7f66c8a00640 1 -- 172.21.15.39:0/3586312537 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc005ea0 con 0x7f66ac003970 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.065+0000 7f66cb200640 1 -- 172.21.15.39:0/3586312537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66ac003970 msgr2=0x7f66ac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.065+0000 7f66cb200640 1 --2- 172.21.15.39:0/3586312537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66ac003970 0x7f66ac003d90 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f66bc009ab0 tx=0x7f66ac0057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.065+0000 7f66ca800640 1 -- 172.21.15.39:0/3586312537 reap_dead start 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.065+0000 7f66cb200640 1 -- 172.21.15.39:0/3586312537 shutdown_connections 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.065+0000 7f66cb200640 1 -- 172.21.15.39:0/3586312537 >> 172.21.15.39:0/3586312537 conn(0x7f66cc0d2830 msgr2=0x7f66cc0d2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.065+0000 7f66cb200640 1 -- 172.21.15.39:0/3586312537 shutdown_connections 2024-01-24T17:49:52.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.065+0000 7f66cb200640 1 -- 172.21.15.39:0/3586312537 wait complete. 2024-01-24T17:49:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.067+0000 7f66cb200640 1 Processor -- start 2024-01-24T17:49:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.067+0000 7f66cb200640 1 -- start start 2024-01-24T17:49:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66cb200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac003970 0x7f66cc15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66cb200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc159db0 0x7f66cc0cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66cb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66cc06b7b0 0x7f66cc06bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66cb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc160400 con 0x7f66cc06b7b0 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66cb200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc0ccd60 con 0x7f66ac003970 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66cb200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc06c170 con 0x7f66cc159db0 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66ca800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66cc06b7b0 0x7f66cc06bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66ca800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66cc06b7b0 0x7f66cc06bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54516/0 (socket says 172.21.15.39:54516) 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.068+0000 7f66ca800640 1 -- 172.21.15.39:0/3603246048 learned_addr learned my addr 172.21.15.39:0/3603246048 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66c9e00640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac003970 0x7f66cc15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66c9400640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc159db0 0x7f66cc0cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66ca800640 1 -- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc159db0 msgr2=0x7f66cc0cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66ca800640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc159db0 0x7f66cc0cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66ca800640 1 -- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac003970 msgr2=0x7f66cc15fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66ca800640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac003970 0x7f66cc15fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66ca800640 1 -- 172.21.15.39:0/3603246048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc0c9dc0 con 0x7f66cc06b7b0 2024-01-24T17:49:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66ca800640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66cc06b7b0 0x7f66cc06bc30 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f66cc08d0a0 tx=0x7f66c400e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66c2000640 1 -- 172.21.15.39:0/3603246048 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4018070 con 0x7f66cc06b7b0 2024-01-24T17:49:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66c2000640 1 -- 172.21.15.39:0/3603246048 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c4004020 con 0x7f66cc06b7b0 2024-01-24T17:49:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66c2000640 1 -- 172.21.15.39:0/3603246048 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c400c070 con 0x7f66cc06b7b0 2024-01-24T17:49:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.069+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc15a230 con 0x7f66cc06b7b0 2024-01-24T17:49:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.070+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc1f5bf0 con 0x7f66cc06b7b0 2024-01-24T17:49:52.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.071+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66cc04e840 con 0x7f66cc06b7b0 2024-01-24T17:49:52.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.072+0000 7f66c2000640 1 -- 172.21.15.39:0/3603246048 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f66c4003610 con 0x7f66cc06b7b0 2024-01-24T17:49:52.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.073+0000 7f66c2000640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f66a406a430 0x7f66a406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.073+0000 7f66c2000640 1 -- 172.21.15.39:0/3603246048 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f66c4087b70 con 0x7f66cc06b7b0 2024-01-24T17:49:52.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.073+0000 7f66c9e00640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f66a406a430 0x7f66a406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.075+0000 7f66c9e00640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f66a406a430 0x7f66a406c8f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f66bc035000 tx=0x7f66bc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:52.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.078+0000 7f66c2000640 1 -- 172.21.15.39:0/3603246048 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f66c4052f80 con 0x7f66cc06b7b0 2024-01-24T17:49:52.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.311+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66cc0b2040 con 0x7f66cc06b7b0 2024-01-24T17:49:52.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.312+0000 7f66c2000640 1 -- 172.21.15.39:0/3603246048 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4759 (secure 0 0 0) 0x7f66c40565a0 con 0x7f66cc06b7b0 2024-01-24T17:49:52.313 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:52.313 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:49:19.498543+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8118,"mds_1":8100},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8100":{"gid":8100,"name":"a","rank":1,"incarnation":70,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6833/3903887009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3903887009},{"type":"v1","addr":"172.21.15.39:6833","nonce":3903887009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8118":{"gid":8118,"name":"c","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/1608906778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1608906778},{"type":"v1","addr":"172.21.15.39:6835","nonce":1608906778}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:49:52.313 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 72 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.314+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f66a406a430 msgr2=0x7f66a406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.314+0000 7f66cb200640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f66a406a430 0x7f66a406c8f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f66bc035000 tx=0x7f66bc030000 comp rx=0 tx=0).stop 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.314+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66cc06b7b0 msgr2=0x7f66cc06bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.314+0000 7f66cb200640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66cc06b7b0 0x7f66cc06bc30 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f66cc08d0a0 tx=0x7f66c400e830 comp rx=0 tx=0).stop 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.315+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 shutdown_connections 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.315+0000 7f66cb200640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f66a406a430 0x7f66a406c8f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.315+0000 7f66cb200640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66cc06b7b0 0x7f66cc06bc30 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.315+0000 7f66cb200640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f66cc159db0 0x7f66cc0cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.315+0000 7f66cb200640 1 --2- 172.21.15.39:0/3603246048 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f66ac003970 0x7f66cc15fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.315+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 >> 172.21.15.39:0/3603246048 conn(0x7f66cc0d2830 msgr2=0x7f66cc0b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.315+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 shutdown_connections 2024-01-24T17:49:52.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.315+0000 7f66cb200640 1 -- 172.21.15.39:0/3603246048 wait complete. 2024-01-24T17:49:52.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:49:52.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.554+0000 7f5578200640 1 Processor -- start 2024-01-24T17:49:52.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.554+0000 7f5578200640 1 -- start start 2024-01-24T17:49:52.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.555+0000 7f5578200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f5570071d40 0x7f55700ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.555+0000 7f5578200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f55700adb80 0x7f557015f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.555+0000 7f5578200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557015ade0 con 0x7f557015b5f0 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.555+0000 7f5578200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f557015bb70 con 0x7f55700adb80 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.555+0000 7f5578200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55700b3540 con 0x7f5570071d40 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.555+0000 7f5575800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f557015b5f0 0x7f5570071630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56116/0 (socket says 172.21.15.39:56116) 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.555+0000 7f5575800640 1 -- 172.21.15.39:0/2317402911 learned_addr learned my addr 172.21.15.39:0/2317402911 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.555+0000 7f5576200640 1 --2- 172.21.15.39:0/2317402911 >> v2:172.21.15.39:3300/0 conn(0x7f55700adb80 0x7f557015f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.556+0000 7f5574e00640 1 --2- 172.21.15.39:0/2317402911 >> v2:172.21.15.100:3300/0 conn(0x7f5570071d40 0x7f55700ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.556+0000 7f556fe00640 1 -- 172.21.15.39:0/2317402911 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579717263 0 0) 0x7f557015ade0 con 0x7f557015b5f0 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.556+0000 7f556fe00640 1 -- 172.21.15.39:0/2317402911 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5550003520 con 0x7f557015b5f0 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.556+0000 7f5576200640 1 -- 172.21.15.39:0/2317402911 >> v2:172.21.15.100:3300/0 conn(0x7f5570071d40 msgr2=0x7f55700ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.556+0000 7f5576200640 1 --2- 172.21.15.39:0/2317402911 >> v2:172.21.15.100:3300/0 conn(0x7f5570071d40 0x7f55700ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.556+0000 7f5576200640 1 -- 172.21.15.39:0/2317402911 >> v1:172.21.15.39:6789/0 conn(0x7f557015b5f0 legacy=0x7f5570071630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.556+0000 7f5576200640 1 -- 172.21.15.39:0/2317402911 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557015be10 con 0x7f55700adb80 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.557+0000 7f5576200640 1 --2- 172.21.15.39:0/2317402911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55700adb80 0x7f557015f020 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f55700b00d0 tx=0x7f556802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1f11af5d089ac3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.557+0000 7f556fe00640 1 -- 172.21.15.39:0/2317402911 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556802cc80 con 0x7f55700adb80 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.557+0000 7f556fe00640 1 -- 172.21.15.39:0/2317402911 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556802cde0 con 0x7f55700adb80 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.557+0000 7f556fe00640 1 -- 172.21.15.39:0/2317402911 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568034660 con 0x7f55700adb80 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.558+0000 7f5578200640 1 -- 172.21.15.39:0/2317402911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55700adb80 msgr2=0x7f557015f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.558+0000 7f5578200640 1 --2- 172.21.15.39:0/2317402911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55700adb80 0x7f557015f020 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f55700b00d0 tx=0x7f556802bed0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.558+0000 7f5578200640 1 -- 172.21.15.39:0/2317402911 shutdown_connections 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.558+0000 7f5578200640 1 --2- 172.21.15.39:0/2317402911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55700adb80 0x7f557015f020 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.558+0000 7f5578200640 1 --2- 172.21.15.39:0/2317402911 >> v2:172.21.15.100:3300/0 conn(0x7f5570071d40 0x7f55700ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.558+0000 7f5578200640 1 -- 172.21.15.39:0/2317402911 >> 172.21.15.39:0/2317402911 conn(0x7f557006a220 msgr2=0x7f55700ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:52.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.558+0000 7f5578200640 1 -- 172.21.15.39:0/2317402911 shutdown_connections 2024-01-24T17:49:52.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.558+0000 7f5578200640 1 -- 172.21.15.39:0/2317402911 wait complete. 2024-01-24T17:49:52.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.560+0000 7f5578200640 1 Processor -- start 2024-01-24T17:49:52.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.560+0000 7f5578200640 1 -- start start 2024-01-24T17:49:52.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5578200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5570071d40 0x7f55700cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5578200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55700adb80 0x7f55700ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5578200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557015b5f0 0x7f5570155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5578200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55700cb5d0 con 0x7f557015b5f0 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5578200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55700c9690 con 0x7f5570071d40 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5578200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55700c6840 con 0x7f55700adb80 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5576200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557015b5f0 0x7f5570155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5576200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557015b5f0 0x7f5570155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54536/0 (socket says 172.21.15.39:54536) 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.561+0000 7f5576200640 1 -- 172.21.15.39:0/3625448432 learned_addr learned my addr 172.21.15.39:0/3625448432 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.562+0000 7f5574e00640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55700adb80 0x7f55700ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.562+0000 7f5575800640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5570071d40 0x7f55700cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.562+0000 7f5576200640 1 -- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55700adb80 msgr2=0x7f55700ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.562+0000 7f5576200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55700adb80 0x7f55700ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.562+0000 7f5576200640 1 -- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5570071d40 msgr2=0x7f55700cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.562+0000 7f5576200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5570071d40 0x7f55700cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.562+0000 7f5576200640 1 -- 172.21.15.39:0/3625448432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5568009a70 con 0x7f557015b5f0 2024-01-24T17:49:52.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.562+0000 7f5576200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557015b5f0 0x7f5570155870 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f5568009e80 tx=0x7f5568004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:52.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.563+0000 7f556d600640 1 -- 172.21.15.39:0/3625448432 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568040650 con 0x7f557015b5f0 2024-01-24T17:49:52.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.563+0000 7f556d600640 1 -- 172.21.15.39:0/3625448432 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55680407b0 con 0x7f557015b5f0 2024-01-24T17:49:52.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.563+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55700c6b20 con 0x7f557015b5f0 2024-01-24T17:49:52.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.563+0000 7f556d600640 1 -- 172.21.15.39:0/3625448432 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568040b60 con 0x7f557015b5f0 2024-01-24T17:49:52.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.563+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55700c9bc0 con 0x7f557015b5f0 2024-01-24T17:49:52.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.564+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55700c9db0 con 0x7f557015b5f0 2024-01-24T17:49:52.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.565+0000 7f556d600640 1 -- 172.21.15.39:0/3625448432 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f5568040cc0 con 0x7f557015b5f0 2024-01-24T17:49:52.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.566+0000 7f556d600640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f554806a430 0x7f554806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:52.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.566+0000 7f556d600640 1 -- 172.21.15.39:0/3625448432 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f55680ab740 con 0x7f557015b5f0 2024-01-24T17:49:52.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.567+0000 7f5575800640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f554806a430 0x7f554806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:52.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.568+0000 7f5575800640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f554806a430 0x7f554806c8f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f55600010e0 tx=0x7f5560009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:52.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.570+0000 7f556d600640 1 -- 172.21.15.39:0/3625448432 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f5568076be0 con 0x7f557015b5f0 2024-01-24T17:49:52.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.768+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5570073f10 con 0x7f557015b5f0 2024-01-24T17:49:52.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.770+0000 7f556d600640 1 -- 172.21.15.39:0/3625448432 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16529 (secure 0 0 0) 0x7f5568009450 con 0x7f557015b5f0 2024-01-24T17:49:52.771 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:52.771 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":106,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:49:17.472389+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.0","mappings":[{"from":4,"to":0}]},{"pgid":"14.18","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6833/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.39:6832/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.100:6836/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.100:6834/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.39:6835/3495904204":"2024-01-25T17:48:58.377999+0000","172.21.15.39:6834/3495904204":"2024-01-25T17:48:58.377999+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:49:52.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f554806a430 msgr2=0x7f554806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f554806a430 0x7f554806c8f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f55600010e0 tx=0x7f5560009040 comp rx=0 tx=0).stop 2024-01-24T17:49:52.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557015b5f0 msgr2=0x7f5570155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:52.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557015b5f0 0x7f5570155870 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f5568009e80 tx=0x7f5568004a40 comp rx=0 tx=0).stop 2024-01-24T17:49:52.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 shutdown_connections 2024-01-24T17:49:52.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f554806a430 0x7f554806c8f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557015b5f0 0x7f5570155870 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f55700adb80 0x7f55700ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 --2- 172.21.15.39:0/3625448432 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5570071d40 0x7f55700cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:52.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 >> 172.21.15.39:0/3625448432 conn(0x7f557006a220 msgr2=0x7f55700ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:52.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 shutdown_connections 2024-01-24T17:49:52.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:52.773+0000 7f5578200640 1 -- 172.21.15.39:0/3625448432 wait complete. 2024-01-24T17:49:52.794 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-24T17:49:52.794 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:49:53.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.010+0000 7fc251a00640 1 Processor -- start 2024-01-24T17:49:53.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.010+0000 7fc251a00640 1 -- start start 2024-01-24T17:49:53.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.012+0000 7fc251a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c15e420 0x7fc24c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.012+0000 7fc251a00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fc24c15b460 0x7fc24c15b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.012+0000 7fc251a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c15be30 con 0x7fc24c0c9450 2024-01-24T17:49:53.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.012+0000 7fc251a00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24c072220 con 0x7fc24c15b460 2024-01-24T17:49:53.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.012+0000 7fc251a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c15e880 con 0x7fc24c15e420 2024-01-24T17:49:53.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.012+0000 7fc24be00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc24c0c9450 0x7fc24c06fed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:56128/0 (socket says 172.21.15.39:56128) 2024-01-24T17:49:53.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.012+0000 7fc24be00640 1 -- 172.21.15.39:0/4006014029 learned_addr learned my addr 172.21.15.39:0/4006014029 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:53.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24b400640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c15e420 0x7fc24c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc251000640 1 --2- 172.21.15.39:0/4006014029 >> v2:172.21.15.100:3300/0 conn(0x7fc24c15b460 0x7fc24c15b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025755255 0 0) 0x7fc23c001110 con 0x7fc24c0c9450 2024-01-24T17:49:53.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc22c003520 con 0x7fc24c0c9450 2024-01-24T17:49:53.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1848395088 0 0) 0x7fc22c003520 con 0x7fc24c0c9450 2024-01-24T17:49:53.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c15e420 msgr2=0x7fc24c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:53.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c15e420 0x7fc24c071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 >> v2:172.21.15.100:3300/0 conn(0x7fc24c15b460 msgr2=0x7fc24c15b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 --2- 172.21.15.39:0/4006014029 >> v2:172.21.15.100:3300/0 conn(0x7fc24c15b460 0x7fc24c15b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c15f330 con 0x7fc24c0c9450 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc23c001800 con 0x7fc24c0c9450 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 >> v1:172.21.15.39:6789/0 conn(0x7fc24c0c9450 legacy=0x7fc24c06fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.013+0000 7fc24aa00640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc22c003970 0x7fc22c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc24aa00640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c15e420 0x7fc22c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc24aa00640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c004d00 0x7fc22c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c15e880 con 0x7fc22c004d00 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c00b980 con 0x7fc22c003970 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c00bac0 con 0x7fc24c15e420 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc251000640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c004d00 0x7fc22c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc24b400640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c15e420 0x7fc22c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc251000640 1 -- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c15e420 msgr2=0x7fc22c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:53.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc251000640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c15e420 0x7fc22c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc251000640 1 -- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc22c003970 msgr2=0x7fc22c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:53.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc251000640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc22c003970 0x7fc22c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.014+0000 7fc251000640 1 -- 172.21.15.39:0/4006014029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c003520 con 0x7fc22c004d00 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.015+0000 7fc251000640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c004d00 0x7fc22c005180 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc244009fc0 tx=0x7fc22c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240c2a00a5030deb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24402cc60 con 0x7fc22c004d00 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24400f040 con 0x7fc22c004d00 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc24aa00640 1 -- 172.21.15.39:0/4006014029 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244034440 con 0x7fc22c004d00 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc251a00640 1 -- 172.21.15.39:0/4006014029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c004d00 msgr2=0x7fc22c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc251a00640 1 --2- 172.21.15.39:0/4006014029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c004d00 0x7fc22c005180 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc244009fc0 tx=0x7fc22c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc251000640 1 -- 172.21.15.39:0/4006014029 reap_dead start 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc251a00640 1 -- 172.21.15.39:0/4006014029 shutdown_connections 2024-01-24T17:49:53.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc251a00640 1 -- 172.21.15.39:0/4006014029 >> 172.21.15.39:0/4006014029 conn(0x7fc24c06bf30 msgr2=0x7fc24c06c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:53.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc251a00640 1 -- 172.21.15.39:0/4006014029 shutdown_connections 2024-01-24T17:49:53.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.016+0000 7fc251a00640 1 -- 172.21.15.39:0/4006014029 wait complete. 2024-01-24T17:49:53.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.018+0000 7fc251a00640 1 Processor -- start 2024-01-24T17:49:53.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.018+0000 7fc251a00640 1 -- start start 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.019+0000 7fc251a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc22c004d00 0x7fc24c15da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.019+0000 7fc251a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc24c156640 0x7fc24c151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.019+0000 7fc251a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c0c3ff0 0x7fc24c0c4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.019+0000 7fc251a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c0c49b0 con 0x7fc24c156640 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.019+0000 7fc251a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c151650 con 0x7fc22c004d00 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.019+0000 7fc251a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c15c9d0 con 0x7fc24c0c3ff0 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.019+0000 7fc24b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc24c156640 0x7fc24c151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc24c156640 0x7fc24c151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48432/0 (socket says 172.21.15.39:48432) 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24b400640 1 -- 172.21.15.39:0/3889374060 learned_addr learned my addr 172.21.15.39:0/3889374060 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24b400640 1 -- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c0c3ff0 msgr2=0x7fc24c0c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24be00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc22c004d00 0x7fc24c15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc251000640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c0c3ff0 0x7fc24c0c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24b400640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c0c3ff0 0x7fc24c0c4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24b400640 1 -- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc22c004d00 msgr2=0x7fc24c15da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24b400640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc22c004d00 0x7fc24c15da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24b400640 1 -- 172.21.15.39:0/3889374060 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c15ccb0 con 0x7fc24c156640 2024-01-24T17:49:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc24b400640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc24c156640 0x7fc24c151110 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc234010860 tx=0x7fc234010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.020+0000 7fc243e00640 1 -- 172.21.15.39:0/3889374060 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc234009610 con 0x7fc24c156640 2024-01-24T17:49:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.021+0000 7fc243e00640 1 -- 172.21.15.39:0/3889374060 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23400eb60 con 0x7fc24c156640 2024-01-24T17:49:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.021+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24c15cf90 con 0x7fc24c156640 2024-01-24T17:49:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.021+0000 7fc243e00640 1 -- 172.21.15.39:0/3889374060 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23401c400 con 0x7fc24c156640 2024-01-24T17:49:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.021+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24c15d550 con 0x7fc24c156640 2024-01-24T17:49:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.022+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24c04e840 con 0x7fc24c156640 2024-01-24T17:49:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.023+0000 7fc243e00640 1 -- 172.21.15.39:0/3889374060 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc23401c560 con 0x7fc24c156640 2024-01-24T17:49:53.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.024+0000 7fc243e00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22406a460 0x7fc22406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.024+0000 7fc243e00640 1 -- 172.21.15.39:0/3889374060 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fc234088ff0 con 0x7fc24c156640 2024-01-24T17:49:53.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.024+0000 7fc24be00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22406a460 0x7fc22406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.026+0000 7fc24be00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22406a460 0x7fc22406c920 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0012b0 tx=0x7fc23c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.028+0000 7fc243e00640 1 -- 172.21.15.39:0/3889374060 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc2340bbcb0 con 0x7fc24c156640 2024-01-24T17:49:53.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.260+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc24c0cc3e0 con 0x7fc24c156640 2024-01-24T17:49:53.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.261+0000 7fc243e00640 1 -- 172.21.15.39:0/3889374060 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4759 (secure 0 0 0) 0x7fc234054560 con 0x7fc24c156640 2024-01-24T17:49:53.262 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:53.262 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:49:19.498543+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8118,"mds_1":8100},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8100":{"gid":8100,"name":"a","rank":1,"incarnation":70,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6833/3903887009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3903887009},{"type":"v1","addr":"172.21.15.39:6833","nonce":3903887009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8118":{"gid":8118,"name":"c","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/1608906778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1608906778},{"type":"v1","addr":"172.21.15.39:6835","nonce":1608906778}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:49:53.263 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 72 2024-01-24T17:49:53.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22406a460 msgr2=0x7fc22406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22406a460 0x7fc22406c920 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0012b0 tx=0x7fc23c009040 comp rx=0 tx=0).stop 2024-01-24T17:49:53.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc24c156640 msgr2=0x7fc24c151110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc24c156640 0x7fc24c151110 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc234010860 tx=0x7fc234010d20 comp rx=0 tx=0).stop 2024-01-24T17:49:53.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 shutdown_connections 2024-01-24T17:49:53.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc24c0c3ff0 0x7fc24c0c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc22406a460 0x7fc22406c920 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc24c156640 0x7fc24c151110 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.264+0000 7fc251a00640 1 --2- 172.21.15.39:0/3889374060 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc22c004d00 0x7fc24c15da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.265+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 >> 172.21.15.39:0/3889374060 conn(0x7fc24c06bf30 msgr2=0x7fc24c0cad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:53.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.265+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 shutdown_connections 2024-01-24T17:49:53.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.265+0000 7fc251a00640 1 -- 172.21.15.39:0/3889374060 wait complete. 2024-01-24T17:49:53.283 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:49:53.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.490+0000 7f06b7800640 1 Processor -- start 2024-01-24T17:49:53.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.490+0000 7f06b7800640 1 -- start start 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.491+0000 7f06b7800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f06b015cd90 0x7f06b00b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.491+0000 7f06b7800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f06b00cd320 0x7f06b00cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.491+0000 7f06b7800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b00b6380 con 0x7f06b00cf110 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.491+0000 7f06b7800640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b00d0cb0 con 0x7f06b00cd320 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.491+0000 7f06b7800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b00cdd70 con 0x7f06b015cd90 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06afe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f06b00cd320 0x7f06b00cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06afe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f06b00cd320 0x7f06b00cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48438/0 (socket says 172.21.15.39:48438) 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06afe00640 1 -- 172.21.15.39:0/3584569021 learned_addr learned my addr 172.21.15.39:0/3584569021 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06b4e00640 1 --2- 172.21.15.39:0/3584569021 >> v2:172.21.15.100:3300/0 conn(0x7f06b015cd90 0x7f06b00b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06afe00640 1 -- 172.21.15.39:0/3584569021 >> v2:172.21.15.100:3300/0 conn(0x7f06b015cd90 msgr2=0x7f06b00b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:53.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06afe00640 1 --2- 172.21.15.39:0/3584569021 >> v2:172.21.15.100:3300/0 conn(0x7f06b015cd90 0x7f06b00b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06afe00640 1 -- 172.21.15.39:0/3584569021 >> v1:172.21.15.39:6789/0 conn(0x7f06b00cf110 legacy=0x7f06b00cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06afe00640 1 -- 172.21.15.39:0/3584569021 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00ce050 con 0x7f06b00cd320 2024-01-24T17:49:53.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.492+0000 7f06b4e00640 1 --2- 172.21.15.39:0/3584569021 >> v2:172.21.15.100:3300/0 conn(0x7f06b015cd90 0x7f06b00b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:53.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.493+0000 7f06afe00640 1 --2- 172.21.15.39:0/3584569021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b00cd320 0x7f06b00cd780 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f06b00b5580 tx=0x7f069802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b9c5e95ffb0438 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06af400640 1 -- 172.21.15.39:0/3584569021 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698030870 con 0x7f06b00cd320 2024-01-24T17:49:53.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06af400640 1 -- 172.21.15.39:0/3584569021 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06980309d0 con 0x7f06b00cd320 2024-01-24T17:49:53.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06af400640 1 -- 172.21.15.39:0/3584569021 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698030d60 con 0x7f06b00cd320 2024-01-24T17:49:53.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06b7800640 1 -- 172.21.15.39:0/3584569021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b00cd320 msgr2=0x7f06b00cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06b7800640 1 --2- 172.21.15.39:0/3584569021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b00cd320 0x7f06b00cd780 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f06b00b5580 tx=0x7f069802b9b0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06b7800640 1 -- 172.21.15.39:0/3584569021 shutdown_connections 2024-01-24T17:49:53.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06b7800640 1 --2- 172.21.15.39:0/3584569021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b00cd320 0x7f06b00cd780 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06b7800640 1 --2- 172.21.15.39:0/3584569021 >> v2:172.21.15.100:3300/0 conn(0x7f06b015cd90 0x7f06b00b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.494+0000 7f06b7800640 1 -- 172.21.15.39:0/3584569021 >> 172.21.15.39:0/3584569021 conn(0x7f06b006af00 msgr2=0x7f06b006b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:53.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.495+0000 7f06b7800640 1 -- 172.21.15.39:0/3584569021 shutdown_connections 2024-01-24T17:49:53.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.495+0000 7f06b7800640 1 -- 172.21.15.39:0/3584569021 wait complete. 2024-01-24T17:49:53.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.496+0000 7f06b7800640 1 Processor -- start 2024-01-24T17:49:53.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.496+0000 7f06b7800640 1 -- start start 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.497+0000 7f06b7800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06b00cd320 0x7f06b015fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.497+0000 7f06b7800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06b00cf110 0x7f06b00c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.497+0000 7f06b7800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b015cd90 0x7f06b00b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.497+0000 7f06b7800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b015f6f0 con 0x7f06b015cd90 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.497+0000 7f06b7800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b0160050 con 0x7f06b00cd320 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.497+0000 7f06b7800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00c8f40 con 0x7f06b00cf110 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b015cd90 0x7f06b00b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b015cd90 0x7f06b00b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48454/0 (socket says 172.21.15.39:48454) 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b5800640 1 -- 172.21.15.39:0/1374186468 learned_addr learned my addr 172.21.15.39:0/1374186468 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b4e00640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06b00cd320 0x7f06b015fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06afe00640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06b00cf110 0x7f06b00c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b5800640 1 -- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06b00cf110 msgr2=0x7f06b00c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b5800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06b00cf110 0x7f06b00c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b5800640 1 -- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06b00cd320 msgr2=0x7f06b015fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b5800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06b00cd320 0x7f06b015fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06b5800640 1 -- 172.21.15.39:0/1374186468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980094d0 con 0x7f06b015cd90 2024-01-24T17:49:53.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.498+0000 7f06afe00640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06b00cf110 0x7f06b00c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.499+0000 7f06b5800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b015cd90 0x7f06b00b36a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f06a8001eb0 tx=0x7f06a800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.499+0000 7f06acc00640 1 -- 172.21.15.39:0/1374186468 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80052d0 con 0x7f06b015cd90 2024-01-24T17:49:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.499+0000 7f06acc00640 1 -- 172.21.15.39:0/1374186468 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a8005430 con 0x7f06b015cd90 2024-01-24T17:49:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.499+0000 7f06acc00640 1 -- 172.21.15.39:0/1374186468 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8003810 con 0x7f06b015cd90 2024-01-24T17:49:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.499+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b015ea70 con 0x7f06b015cd90 2024-01-24T17:49:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.499+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b015f030 con 0x7f06b015cd90 2024-01-24T17:49:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.500+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b015f830 con 0x7f06b015cd90 2024-01-24T17:49:53.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.501+0000 7f06acc00640 1 -- 172.21.15.39:0/1374186468 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f06a8003970 con 0x7f06b015cd90 2024-01-24T17:49:53.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.502+0000 7f06acc00640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f068806a430 0x7f068806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.502+0000 7f06acc00640 1 -- 172.21.15.39:0/1374186468 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f06a8087ff0 con 0x7f06b015cd90 2024-01-24T17:49:53.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.503+0000 7f06b4e00640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f068806a430 0x7f068806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.504+0000 7f06b4e00640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f068806a430 0x7f068806c8f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f06a0007510 tx=0x7f06a00074a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.506+0000 7f06acc00640 1 -- 172.21.15.39:0/1374186468 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f06a80bacb0 con 0x7f06b015cd90 2024-01-24T17:49:53.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.739+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06b00d07e0 con 0x7f06b015cd90 2024-01-24T17:49:53.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.740+0000 7f06acc00640 1 -- 172.21.15.39:0/1374186468 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4759 (secure 0 0 0) 0x7f06b015f830 con 0x7f06b015cd90 2024-01-24T17:49:53.741 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:53.741 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":72,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T17:48:26.880600+0000","modified":"2024-01-24T17:49:19.498543+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8118,"mds_1":8100},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8100":{"gid":8100,"name":"a","rank":1,"incarnation":70,"state":"up:active","state_seq":5,"addr":"172.21.15.39:6833/3903887009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3903887009},{"type":"v1","addr":"172.21.15.39:6833","nonce":3903887009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_8118":{"gid":8118,"name":"c","rank":0,"incarnation":65,"state":"up:active","state_seq":4,"addr":"172.21.15.39:6835/1608906778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":1608906778},{"type":"v1","addr":"172.21.15.39:6835","nonce":1608906778}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-24T17:49:53.741 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 72 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f068806a430 msgr2=0x7f068806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f068806a430 0x7f068806c8f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f06a0007510 tx=0x7f06a00074a0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b015cd90 msgr2=0x7f06b00b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b015cd90 0x7f06b00b36a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f06a8001eb0 tx=0x7f06a800c890 comp rx=0 tx=0).stop 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 shutdown_connections 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f068806a430 0x7f068806c8f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06b015cd90 0x7f06b00b36a0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f06b00cf110 0x7f06b00c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 --2- 172.21.15.39:0/1374186468 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f06b00cd320 0x7f06b015fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 >> 172.21.15.39:0/1374186468 conn(0x7f06b006af00 msgr2=0x7f06b015bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:53.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 shutdown_connections 2024-01-24T17:49:53.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.743+0000 7f06b7800640 1 -- 172.21.15.39:0/1374186468 wait complete. 2024-01-24T17:49:53.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:49:53.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.973+0000 7fce8c000640 1 Processor -- start 2024-01-24T17:49:53.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.973+0000 7fce8c000640 1 -- start start 2024-01-24T17:49:53.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.974+0000 7fce8c000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fce840cb770 0x7fce840cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.974+0000 7fce8c000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fce840b22c0 0x7fce840cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.975+0000 7fce8c000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce840b1540 con 0x7fce84160140 2024-01-24T17:49:53.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.975+0000 7fce8c000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce840ce050 con 0x7fce840cb770 2024-01-24T17:49:53.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.975+0000 7fce8c000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce84160520 con 0x7fce840b22c0 2024-01-24T17:49:53.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.975+0000 7fce88c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fce840cb770 0x7fce840cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.975+0000 7fce88c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fce840cb770 0x7fce840cd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48462/0 (socket says 172.21.15.39:48462) 2024-01-24T17:49:53.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.975+0000 7fce88c00640 1 -- 172.21.15.39:0/669942443 learned_addr learned my addr 172.21.15.39:0/669942443 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:53.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.975+0000 7fce88c00640 1 -- 172.21.15.39:0/669942443 >> v2:172.21.15.100:3300/0 conn(0x7fce840b22c0 msgr2=0x7fce840cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:53.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.975+0000 7fce8a000640 1 --2- 172.21.15.39:0/669942443 >> v2:172.21.15.100:3300/0 conn(0x7fce840b22c0 0x7fce840cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.976+0000 7fce88c00640 1 --2- 172.21.15.39:0/669942443 >> v2:172.21.15.100:3300/0 conn(0x7fce840b22c0 0x7fce840cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.976+0000 7fce88c00640 1 -- 172.21.15.39:0/669942443 >> v1:172.21.15.100:6789/0 conn(0x7fce84160140 legacy=0x7fce840cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:53.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.976+0000 7fce88c00640 1 -- 172.21.15.39:0/669942443 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840b17e0 con 0x7fce840cb770 2024-01-24T17:49:53.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.976+0000 7fce8a000640 1 --2- 172.21.15.39:0/669942443 >> v2:172.21.15.100:3300/0 conn(0x7fce840b22c0 0x7fce840cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:53.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.977+0000 7fce88c00640 1 --2- 172.21.15.39:0/669942443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce840cb770 0x7fce840cd5d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fce840c9750 tx=0x7fce6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f0dd527ae1e8e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.977+0000 7fce83e00640 1 -- 172.21.15.39:0/669942443 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c031870 con 0x7fce840cb770 2024-01-24T17:49:53.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.977+0000 7fce83e00640 1 -- 172.21.15.39:0/669942443 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c0319d0 con 0x7fce840cb770 2024-01-24T17:49:53.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.977+0000 7fce83e00640 1 -- 172.21.15.39:0/669942443 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c031d60 con 0x7fce840cb770 2024-01-24T17:49:53.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.978+0000 7fce8c000640 1 -- 172.21.15.39:0/669942443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce840cb770 msgr2=0x7fce840cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.978+0000 7fce8c000640 1 --2- 172.21.15.39:0/669942443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce840cb770 0x7fce840cd5d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fce840c9750 tx=0x7fce6c02b910 comp rx=0 tx=0).stop 2024-01-24T17:49:53.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.978+0000 7fce8c000640 1 -- 172.21.15.39:0/669942443 shutdown_connections 2024-01-24T17:49:53.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.978+0000 7fce8c000640 1 --2- 172.21.15.39:0/669942443 >> v2:172.21.15.100:3300/0 conn(0x7fce840b22c0 0x7fce840cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.978+0000 7fce8c000640 1 --2- 172.21.15.39:0/669942443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce840cb770 0x7fce840cd5d0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.978+0000 7fce8c000640 1 -- 172.21.15.39:0/669942443 >> 172.21.15.39:0/669942443 conn(0x7fce8406a220 msgr2=0x7fce840ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:53.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.978+0000 7fce8c000640 1 -- 172.21.15.39:0/669942443 shutdown_connections 2024-01-24T17:49:53.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.978+0000 7fce8c000640 1 -- 172.21.15.39:0/669942443 wait complete. 2024-01-24T17:49:53.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.980+0000 7fce8c000640 1 Processor -- start 2024-01-24T17:49:53.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.980+0000 7fce8c000640 1 -- start start 2024-01-24T17:49:53.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8c000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce840b22c0 0x7fce840ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8c000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce840cb770 0x7fce840716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce84160140 0x7fce84071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce840af680 con 0x7fce84160140 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8c000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8415b2a0 con 0x7fce840b22c0 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8c000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce840aeb00 con 0x7fce840cb770 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce84160140 0x7fce84071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce84160140 0x7fce84071be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48470/0 (socket says 172.21.15.39:48470) 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.981+0000 7fce8a000640 1 -- 172.21.15.39:0/1588144402 learned_addr learned my addr 172.21.15.39:0/1588144402 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce89600640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce840b22c0 0x7fce840ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce88c00640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce840cb770 0x7fce840716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce8a000640 1 -- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce840cb770 msgr2=0x7fce840716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce8a000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce840cb770 0x7fce840716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce8a000640 1 -- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce840b22c0 msgr2=0x7fce840ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:53.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce8a000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce840b22c0 0x7fce840ae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:53.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce8a000640 1 -- 172.21.15.39:0/1588144402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c0094d0 con 0x7fce84160140 2024-01-24T17:49:53.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce89600640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce840b22c0 0x7fce840ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T17:49:53.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.982+0000 7fce8a000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce84160140 0x7fce84071be0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00ce80 tx=0x7fce7c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.983+0000 7fce81600640 1 -- 172.21.15.39:0/1588144402 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00d840 con 0x7fce84160140 2024-01-24T17:49:53.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.983+0000 7fce81600640 1 -- 172.21.15.39:0/1588144402 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00d9e0 con 0x7fce84160140 2024-01-24T17:49:53.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.983+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce840aede0 con 0x7fce84160140 2024-01-24T17:49:53.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.983+0000 7fce81600640 1 -- 172.21.15.39:0/1588144402 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00dd90 con 0x7fce84160140 2024-01-24T17:49:53.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.983+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce840af3a0 con 0x7fce84160140 2024-01-24T17:49:53.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.984+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8415b460 con 0x7fce84160140 2024-01-24T17:49:53.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.985+0000 7fce81600640 1 -- 172.21.15.39:0/1588144402 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fce7c015440 con 0x7fce84160140 2024-01-24T17:49:53.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.986+0000 7fce81600640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce5c06a430 0x7fce5c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:53.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.986+0000 7fce81600640 1 -- 172.21.15.39:0/1588144402 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fce7c0893c0 con 0x7fce84160140 2024-01-24T17:49:53.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.986+0000 7fce89600640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce5c06a430 0x7fce5c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:53.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.987+0000 7fce89600640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce5c06a430 0x7fce5c06c8f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fce74009890 tx=0x7fce74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:53.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:53.990+0000 7fce81600640 1 -- 172.21.15.39:0/1588144402 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fce7c054910 con 0x7fce84160140 2024-01-24T17:49:54.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.184+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce84073f10 con 0x7fce84160140 2024-01-24T17:49:54.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.187+0000 7fce81600640 1 -- 172.21.15.39:0/1588144402 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16529 (secure 0 0 0) 0x7fce7c057f30 con 0x7fce84160140 2024-01-24T17:49:54.188 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:54.188 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":106,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:49:17.472389+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-24T17:48:24.515179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-24T17:48:25.907386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.0","mappings":[{"from":4,"to":0}]},{"pgid":"14.18","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6833/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.39:6832/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.100:6836/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.100:6834/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.39:6835/3495904204":"2024-01-25T17:48:58.377999+0000","172.21.15.39:6834/3495904204":"2024-01-25T17:48:58.377999+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce5c06a430 msgr2=0x7fce5c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce5c06a430 0x7fce5c06c8f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fce74009890 tx=0x7fce74008040 comp rx=0 tx=0).stop 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce84160140 msgr2=0x7fce84071be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce84160140 0x7fce84071be0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00ce80 tx=0x7fce7c00a710 comp rx=0 tx=0).stop 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 shutdown_connections 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fce5c06a430 0x7fce5c06c8f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce84160140 0x7fce84071be0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fce840cb770 0x7fce840716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 --2- 172.21.15.39:0/1588144402 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fce840b22c0 0x7fce840ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 >> 172.21.15.39:0/1588144402 conn(0x7fce8406a220 msgr2=0x7fce8415f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:54.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 shutdown_connections 2024-01-24T17:49:54.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.190+0000 7fce8c000640 1 -- 172.21.15.39:0/1588144402 wait complete. 2024-01-24T17:49:54.211 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-24T17:49:54.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.434+0000 7f9686400640 1 Processor -- start 2024-01-24T17:49:54.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.435+0000 7f9686400640 1 -- start start 2024-01-24T17:49:54.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.436+0000 7f9686400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f9680160200 0x7f968006bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.436+0000 7f9686400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96800d0ec0 con 0x7f968006b470 2024-01-24T17:49:54.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.436+0000 7f9686400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96800d37c0 con 0x7f968006ca30 2024-01-24T17:49:54.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.436+0000 7f9686400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968006b930 con 0x7f9680160200 2024-01-24T17:49:54.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.436+0000 7f9685a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f968006b470 0x7f96800b0f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60074/0 (socket says 172.21.15.39:60074) 2024-01-24T17:49:54.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.436+0000 7f9685a00640 1 -- 172.21.15.39:0/2681135465 learned_addr learned my addr 172.21.15.39:0/2681135465 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f9685000640 1 --2- 172.21.15.39:0/2681135465 >> v2:172.21.15.100:3300/0 conn(0x7f9680160200 0x7f968006bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694131955 0 0) 0x7f96800d0ec0 con 0x7f968006b470 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660003520 con 0x7f968006b470 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897907909 0 0) 0x7f9660003520 con 0x7f968006b470 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 >> v2:172.21.15.100:3300/0 conn(0x7f9680160200 msgr2=0x7f968006bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 --2- 172.21.15.39:0/2681135465 >> v2:172.21.15.100:3300/0 conn(0x7f9680160200 0x7f968006bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 >> v1:172.21.15.100:6789/0 conn(0x7f968006ca30 legacy=0x7f968006ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968006bc10 con 0x7f968006b470 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f96780019f0 con 0x7f968006b470 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 >> v1:172.21.15.39:6789/0 conn(0x7f968006b470 legacy=0x7f96800b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.437+0000 7f967f400640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96600039c0 0x7f9660003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f967f400640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96600047c0 0x7f9660004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f967f400640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600051d0 0x7f966000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968006b930 con 0x7f96600039c0 2024-01-24T17:49:54.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966000c120 con 0x7f96600051d0 2024-01-24T17:49:54.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966000c2e0 con 0x7f96600047c0 2024-01-24T17:49:54.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f9685000640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96600039c0 0x7f9660003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f9685000640 1 -- 172.21.15.39:0/2681135465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96600047c0 msgr2=0x7f9660004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:54.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f9685000640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f96600047c0 0x7f9660004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f9685000640 1 -- 172.21.15.39:0/2681135465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600051d0 msgr2=0x7f966000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:54.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f9685000640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600051d0 0x7f966000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f9685000640 1 -- 172.21.15.39:0/2681135465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660003520 con 0x7f96600039c0 2024-01-24T17:49:54.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.438+0000 7f9685a00640 1 -- 172.21.15.39:0/2681135465 reap_dead start 2024-01-24T17:49:54.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.439+0000 7f9685000640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96600039c0 0x7f9660003de0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f9670009a90 tx=0x7f96600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a82f032aaf45d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:54.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.440+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967002c6b0 con 0x7f96600039c0 2024-01-24T17:49:54.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.440+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967002c810 con 0x7f96600039c0 2024-01-24T17:49:54.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.440+0000 7f967f400640 1 -- 172.21.15.39:0/2681135465 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967002cba0 con 0x7f96600039c0 2024-01-24T17:49:54.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.440+0000 7f9686400640 1 -- 172.21.15.39:0/2681135465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96600039c0 msgr2=0x7f9660003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.440+0000 7f9686400640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96600039c0 0x7f9660003de0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f9670009a90 tx=0x7f96600057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.440+0000 7f9686400640 1 -- 172.21.15.39:0/2681135465 shutdown_connections 2024-01-24T17:49:54.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.440+0000 7f9686400640 1 --2- 172.21.15.39:0/2681135465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96600039c0 0x7f9660003de0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.440+0000 7f9686400640 1 -- 172.21.15.39:0/2681135465 >> 172.21.15.39:0/2681135465 conn(0x7f96800d28e0 msgr2=0x7f96800d2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:54.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.441+0000 7f9686400640 1 -- 172.21.15.39:0/2681135465 shutdown_connections 2024-01-24T17:49:54.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.441+0000 7f9686400640 1 -- 172.21.15.39:0/2681135465 wait complete. 2024-01-24T17:49:54.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.442+0000 7f9686400640 1 Processor -- start 2024-01-24T17:49:54.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.443+0000 7f9686400640 1 -- start start 2024-01-24T17:49:54.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.443+0000 7f9686400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600039c0 0x7f96800b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.443+0000 7f9686400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9680156740 0x7f96800cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.443+0000 7f9686400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96800ca810 0x7f96800cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.443+0000 7f9686400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800ca3f0 con 0x7f96800ca810 2024-01-24T17:49:54.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.443+0000 7f9686400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800cc7a0 con 0x7f96600039c0 2024-01-24T17:49:54.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.443+0000 7f9686400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96800c9a50 con 0x7f9680156740 2024-01-24T17:49:54.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96800ca810 0x7f96800cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96800ca810 0x7f96800cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48492/0 (socket says 172.21.15.39:48492) 2024-01-24T17:49:54.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685a00640 1 -- 172.21.15.39:0/1220193106 learned_addr learned my addr 172.21.15.39:0/1220193106 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:54.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f967fe00640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9680156740 0x7f96800cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685000640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600039c0 0x7f96800b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685a00640 1 -- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9680156740 msgr2=0x7f96800cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685a00640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9680156740 0x7f96800cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685a00640 1 -- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600039c0 msgr2=0x7f96800b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685a00640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600039c0 0x7f96800b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685a00640 1 -- 172.21.15.39:0/1220193106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96800c9d30 con 0x7f96800ca810 2024-01-24T17:49:54.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.444+0000 7f9685000640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600039c0 0x7f96800b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:54.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.445+0000 7f9685a00640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96800ca810 0x7f96800cac90 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9678001b20 tx=0x7f967800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:54.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.445+0000 7f967cc00640 1 -- 172.21.15.39:0/1220193106 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800ce70 con 0x7f96800ca810 2024-01-24T17:49:54.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.445+0000 7f967cc00640 1 -- 172.21.15.39:0/1220193106 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967800a610 con 0x7f96800ca810 2024-01-24T17:49:54.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.445+0000 7f967cc00640 1 -- 172.21.15.39:0/1220193106 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800a9c0 con 0x7f96800ca810 2024-01-24T17:49:54.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.445+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96800ca010 con 0x7f96800ca810 2024-01-24T17:49:54.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.445+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9680156bc0 con 0x7f96800ca810 2024-01-24T17:49:54.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.446+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96800ca530 con 0x7f96800ca810 2024-01-24T17:49:54.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.447+0000 7f967cc00640 1 -- 172.21.15.39:0/1220193106 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f967800ab20 con 0x7f96800ca810 2024-01-24T17:49:54.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.448+0000 7f967cc00640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f965806a460 0x7f965806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.448+0000 7f967cc00640 1 -- 172.21.15.39:0/1220193106 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f9678088250 con 0x7f96800ca810 2024-01-24T17:49:54.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.449+0000 7f9685000640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f965806a460 0x7f965806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.450+0000 7f9685000640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f965806a460 0x7f965806c920 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9670036000 tx=0x7f9670030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:54.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.453+0000 7f967cc00640 1 -- 172.21.15.39:0/1220193106 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f96800ca530 con 0x7f96800ca810 2024-01-24T17:49:54.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.685+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f96800ca530 con 0x7f96800ca810 2024-01-24T17:49:54.688 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-01-24T17:49:54.687+0000 7fc2ae600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-24T17:49:54.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.707+0000 7f967cc00640 1 -- 172.21.15.39:0/1220193106 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) v1 ==== 161+0+0 (secure 0 0 0) 0x7f96800ca530 con 0x7f96800ca810 2024-01-24T17:49:54.708 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-24T17:49:54.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f965806a460 msgr2=0x7f965806c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f965806a460 0x7f965806c920 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9670036000 tx=0x7f9670030000 comp rx=0 tx=0).stop 2024-01-24T17:49:54.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96800ca810 msgr2=0x7f96800cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96800ca810 0x7f96800cac90 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9678001b20 tx=0x7f967800cc20 comp rx=0 tx=0).stop 2024-01-24T17:49:54.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 shutdown_connections 2024-01-24T17:49:54.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f965806a460 0x7f965806c920 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96800ca810 0x7f96800cac90 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f9680156740 0x7f96800cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 --2- 172.21.15.39:0/1220193106 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f96600039c0 0x7f96800b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 >> 172.21.15.39:0/1220193106 conn(0x7f96800d28e0 msgr2=0x7f968015f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:54.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 shutdown_connections 2024-01-24T17:49:54.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.710+0000 7f9686400640 1 -- 172.21.15.39:0/1220193106 wait complete. 2024-01-24T17:49:54.720 INFO:tasks.ceph.mds.c.smithi039.stderr: -6280> 2024-01-24T17:49:04.358+0000 7fd769c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T17:49:54.724 INFO:tasks.ceph.mds.c.smithi039.stderr: -3948> 2024-01-24T17:49:14.359+0000 7fd769c00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-01-24T17:49:54.731 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-24T17:49:54.777 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2024-01-24T17:49:54.779 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2024-01-24T17:49:54.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.924+0000 7fb880400640 1 Processor -- start 2024-01-24T17:49:54.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.924+0000 7fb880400640 1 -- start start 2024-01-24T17:49:54.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb880400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780cab40 0x7fb878070090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb880400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8780b21b0 0x7fb87813fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb880400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878142080 con 0x7fb8780caf20 2024-01-24T17:49:54.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb880400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb878071340 con 0x7fb8780b21b0 2024-01-24T17:49:54.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb880400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780b2640 con 0x7fb8780cab40 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb87e400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8780b21b0 0x7fb87813fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb87e400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb8780b21b0 0x7fb87813fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48526/0 (socket says 172.21.15.39:48526) 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb87e400640 1 -- 172.21.15.39:0/2368557613 learned_addr learned my addr 172.21.15.39:0/2368557613 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb87d000640 1 --1- 172.21.15.39:0/2368557613 >> v1:172.21.15.39:6789/0 conn(0x7fb8780caf20 0x7fb8780717d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60094/0 (socket says 172.21.15.39:60094) 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.926+0000 7fb87da00640 1 --2- 172.21.15.39:0/2368557613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780cab40 0x7fb878070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.927+0000 7fb87e400640 1 -- 172.21.15.39:0/2368557613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780cab40 msgr2=0x7fb878070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.927+0000 7fb87e400640 1 --2- 172.21.15.39:0/2368557613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780cab40 0x7fb878070090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.927+0000 7fb87e400640 1 -- 172.21.15.39:0/2368557613 >> v1:172.21.15.39:6789/0 conn(0x7fb8780caf20 legacy=0x7fb8780717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.927+0000 7fb877e00640 1 -- 172.21.15.39:0/2368557613 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476581970 0 0) 0x7fb860001110 con 0x7fb8780caf20 2024-01-24T17:49:54.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.927+0000 7fb87e400640 1 -- 172.21.15.39:0/2368557613 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb878070730 con 0x7fb8780b21b0 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.928+0000 7fb87e400640 1 --2- 172.21.15.39:0/2368557613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780b21b0 0x7fb87813fc20 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fb870009e50 tx=0x7fb87002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ed967038ab8f0c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.928+0000 7fb877e00640 1 -- 172.21.15.39:0/2368557613 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87002cb00 con 0x7fb8780b21b0 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.928+0000 7fb877e00640 1 -- 172.21.15.39:0/2368557613 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87002cc60 con 0x7fb8780b21b0 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.928+0000 7fb877e00640 1 -- 172.21.15.39:0/2368557613 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8700343f0 con 0x7fb8780b21b0 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.929+0000 7fb880400640 1 -- 172.21.15.39:0/2368557613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780b21b0 msgr2=0x7fb87813fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.929+0000 7fb880400640 1 --2- 172.21.15.39:0/2368557613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780b21b0 0x7fb87813fc20 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fb870009e50 tx=0x7fb87002bf60 comp rx=0 tx=0).stop 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.929+0000 7fb880400640 1 -- 172.21.15.39:0/2368557613 shutdown_connections 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.929+0000 7fb880400640 1 --2- 172.21.15.39:0/2368557613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780b21b0 0x7fb87813fc20 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.929+0000 7fb880400640 1 --2- 172.21.15.39:0/2368557613 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780cab40 0x7fb878070090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.929+0000 7fb880400640 1 -- 172.21.15.39:0/2368557613 >> 172.21.15.39:0/2368557613 conn(0x7fb878066020 msgr2=0x7fb878069bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:54.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.929+0000 7fb880400640 1 -- 172.21.15.39:0/2368557613 shutdown_connections 2024-01-24T17:49:54.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.929+0000 7fb880400640 1 -- 172.21.15.39:0/2368557613 wait complete. 2024-01-24T17:49:54.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.931+0000 7fb880400640 1 Processor -- start 2024-01-24T17:49:54.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.931+0000 7fb880400640 1 -- start start 2024-01-24T17:49:54.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb880400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780b21b0 0x7fb8780ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb880400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb8780cab40 0x7fb878139120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb880400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780caf20 0x7fb8781488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb880400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8780cdbe0 con 0x7fb8780caf20 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb880400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87815afe0 con 0x7fb8780cab40 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb880400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87815b560 con 0x7fb8780b21b0 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb87e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780caf20 0x7fb8781488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb87e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780caf20 0x7fb8781488d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48538/0 (socket says 172.21.15.39:48538) 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.932+0000 7fb87e400640 1 -- 172.21.15.39:0/4239742594 learned_addr learned my addr 172.21.15.39:0/4239742594 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb87da00640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780b21b0 0x7fb8780ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb87d000640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb8780cab40 0x7fb878139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb87e400640 1 -- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780b21b0 msgr2=0x7fb8780ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb87e400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780b21b0 0x7fb8780ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb87e400640 1 -- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb8780cab40 msgr2=0x7fb878139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb87e400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb8780cab40 0x7fb878139120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb87e400640 1 -- 172.21.15.39:0/4239742594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb870009b30 con 0x7fb8780caf20 2024-01-24T17:49:54.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb87e400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780caf20 0x7fb8781488d0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fb870036040 tx=0x7fb870004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:54.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.933+0000 7fb875600640 1 -- 172.21.15.39:0/4239742594 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb870009510 con 0x7fb8780caf20 2024-01-24T17:49:54.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.934+0000 7fb875600640 1 -- 172.21.15.39:0/4239742594 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87003e450 con 0x7fb8780caf20 2024-01-24T17:49:54.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.934+0000 7fb875600640 1 -- 172.21.15.39:0/4239742594 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87003e7e0 con 0x7fb8780caf20 2024-01-24T17:49:54.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.934+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb878157d70 con 0x7fb8780caf20 2024-01-24T17:49:54.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.934+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8780cd900 con 0x7fb8780caf20 2024-01-24T17:49:54.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.935+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb878073f10 con 0x7fb8780caf20 2024-01-24T17:49:54.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.936+0000 7fb875600640 1 -- 172.21.15.39:0/4239742594 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb87003e940 con 0x7fb8780caf20 2024-01-24T17:49:54.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.937+0000 7fb875600640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb85006a430 0x7fb85006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:54.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.937+0000 7fb875600640 1 -- 172.21.15.39:0/4239742594 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6197+0+0 (secure 0 0 0) 0x7fb8700ab990 con 0x7fb8780caf20 2024-01-24T17:49:54.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.937+0000 7fb87da00640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb85006a430 0x7fb85006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:54.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.938+0000 7fb87da00640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb85006a430 0x7fb85006c8f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b9b0 tx=0x7fb868008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:54.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:54.941+0000 7fb875600640 1 -- 172.21.15.39:0/4239742594 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb870076cf0 con 0x7fb8780caf20 2024-01-24T17:49:55.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.171+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb87815b6a0 con 0x7fb8780caf20 2024-01-24T17:49:55.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.711+0000 7fb875600640 1 -- 172.21.15.39:0/4239742594 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb87007a310 con 0x7fb8780caf20 2024-01-24T17:49:55.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb85006a430 msgr2=0x7fb85006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:55.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb85006a430 0x7fb85006c8f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b9b0 tx=0x7fb868008040 comp rx=0 tx=0).stop 2024-01-24T17:49:55.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780caf20 msgr2=0x7fb8781488d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:55.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780caf20 0x7fb8781488d0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fb870036040 tx=0x7fb870004a40 comp rx=0 tx=0).stop 2024-01-24T17:49:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 shutdown_connections 2024-01-24T17:49:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb85006a430 0x7fb85006c8f0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8780caf20 0x7fb8781488d0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb8780cab40 0x7fb878139120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 --2- 172.21.15.39:0/4239742594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb8780b21b0 0x7fb8780ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.715+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 >> 172.21.15.39:0/4239742594 conn(0x7fb878066020 msgr2=0x7fb878158160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.716+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 shutdown_connections 2024-01-24T17:49:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.716+0000 7fb880400640 1 -- 172.21.15.39:0/4239742594 wait complete. 2024-01-24T17:49:55.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-24T17:49:55.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.957+0000 7f3a1fe00640 1 Processor -- start 2024-01-24T17:49:55.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.958+0000 7f3a1fe00640 1 -- start start 2024-01-24T17:49:55.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.959+0000 7f3a1fe00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f3a2015ffc0 0x7f3a200d3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:55.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.959+0000 7f3a1fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200cf870 0x7f3a200b2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:55.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.959+0000 7f3a1fe00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a200ce8e0 con 0x7f3a200d04f0 2024-01-24T17:49:55.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.959+0000 7f3a1fe00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a200ce040 con 0x7f3a2015ffc0 2024-01-24T17:49:55.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.959+0000 7f3a1fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200ce320 con 0x7f3a200cf870 2024-01-24T17:49:55.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.960+0000 7f3a1f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200cf870 0x7f3a200b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:55.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.960+0000 7f3a1f400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200cf870 0x7f3a200b2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:37794/0 (socket says 172.21.15.39:37794) 2024-01-24T17:49:55.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.960+0000 7f3a1f400640 1 -- 172.21.15.39:0/1184917488 learned_addr learned my addr 172.21.15.39:0/1184917488 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:55.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.960+0000 7f3a1ea00640 1 --2- 172.21.15.39:0/1184917488 >> v2:172.21.15.100:3300/0 conn(0x7f3a2015ffc0 0x7f3a200d3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:55.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.961+0000 7f3a1e000640 1 -- 172.21.15.39:0/1184917488 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370624570 0 0) 0x7f3a200ce8e0 con 0x7f3a200d04f0 2024-01-24T17:49:55.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.961+0000 7f3a1e000640 1 -- 172.21.15.39:0/1184917488 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a00003520 con 0x7f3a200d04f0 2024-01-24T17:49:55.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.961+0000 7f3a1f400640 1 -- 172.21.15.39:0/1184917488 >> v2:172.21.15.100:3300/0 conn(0x7f3a2015ffc0 msgr2=0x7f3a200d3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:55.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.961+0000 7f3a1f400640 1 --2- 172.21.15.39:0/1184917488 >> v2:172.21.15.100:3300/0 conn(0x7f3a2015ffc0 0x7f3a200d3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.961+0000 7f3a1f400640 1 -- 172.21.15.39:0/1184917488 >> v1:172.21.15.100:6789/0 conn(0x7f3a200d04f0 legacy=0x7f3a200cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:55.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.961+0000 7f3a1f400640 1 -- 172.21.15.39:0/1184917488 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a200ce600 con 0x7f3a200cf870 2024-01-24T17:49:55.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.961+0000 7f3a1ea00640 1 --2- 172.21.15.39:0/1184917488 >> v2:172.21.15.100:3300/0 conn(0x7f3a2015ffc0 0x7f3a200d3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:49:55.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.963+0000 7f3a1f400640 1 --2- 172.21.15.39:0/1184917488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200cf870 0x7f3a200b2580 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009870 tx=0x7f3a1802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=204f984a91f0c97a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:55.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.963+0000 7f3a1e000640 1 -- 172.21.15.39:0/1184917488 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18030840 con 0x7f3a200cf870 2024-01-24T17:49:55.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.963+0000 7f3a1e000640 1 -- 172.21.15.39:0/1184917488 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a180309a0 con 0x7f3a200cf870 2024-01-24T17:49:55.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.963+0000 7f3a1e000640 1 -- 172.21.15.39:0/1184917488 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18030d30 con 0x7f3a200cf870 2024-01-24T17:49:55.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.964+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1184917488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200cf870 msgr2=0x7f3a200b2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:55.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.964+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1184917488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200cf870 0x7f3a200b2580 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009870 tx=0x7f3a1802b980 comp rx=0 tx=0).stop 2024-01-24T17:49:55.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.964+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1184917488 shutdown_connections 2024-01-24T17:49:55.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.964+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1184917488 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200cf870 0x7f3a200b2580 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.964+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1184917488 >> v2:172.21.15.100:3300/0 conn(0x7f3a2015ffc0 0x7f3a200d3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.964+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1184917488 >> 172.21.15.39:0/1184917488 conn(0x7f3a2009bc30 msgr2=0x7f3a200d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:55.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.964+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1184917488 shutdown_connections 2024-01-24T17:49:55.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.964+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1184917488 wait complete. 2024-01-24T17:49:55.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.966+0000 7f3a1fe00640 1 Processor -- start 2024-01-24T17:49:55.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.966+0000 7f3a1fe00640 1 -- start start 2024-01-24T17:49:55.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.967+0000 7f3a1fe00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a200cf870 0x7f3a201588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:55.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.967+0000 7f3a1fe00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200d04f0 0x7f3a201536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:55.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.967+0000 7f3a1fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2015ffc0 0x7f3a200b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:55.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.967+0000 7f3a1fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200c68f0 con 0x7f3a2015ffc0 2024-01-24T17:49:55.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.967+0000 7f3a1fe00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200c6b40 con 0x7f3a200cf870 2024-01-24T17:49:55.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.967+0000 7f3a1fe00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200c6e20 con 0x7f3a200d04f0 2024-01-24T17:49:55.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.967+0000 7f3a1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2015ffc0 0x7f3a200b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:55.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2015ffc0 0x7f3a200b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48544/0 (socket says 172.21.15.39:48544) 2024-01-24T17:49:55.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a1f400640 1 -- 172.21.15.39:0/1540074536 learned_addr learned my addr 172.21.15.39:0/1540074536 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a15e00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200d04f0 0x7f3a201536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a1ea00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a200cf870 0x7f3a201588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a1f400640 1 -- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200d04f0 msgr2=0x7f3a201536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a1f400640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200d04f0 0x7f3a201536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a1f400640 1 -- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a200cf870 msgr2=0x7f3a201588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a1f400640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a200cf870 0x7f3a201588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.968+0000 7f3a1f400640 1 -- 172.21.15.39:0/1540074536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a10009a70 con 0x7f3a2015ffc0 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.969+0000 7f3a1f400640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2015ffc0 0x7f3a200b04e0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009870 tx=0x7f3a18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.969+0000 7f3a17400640 1 -- 172.21.15.39:0/1540074536 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18039c90 con 0x7f3a2015ffc0 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.969+0000 7f3a17400640 1 -- 172.21.15.39:0/1540074536 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a18039e10 con 0x7f3a2015ffc0 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.969+0000 7f3a17400640 1 -- 172.21.15.39:0/1540074536 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18043640 con 0x7f3a2015ffc0 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.969+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a180094d0 con 0x7f3a2015ffc0 2024-01-24T17:49:55.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.969+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a200c73e0 con 0x7f3a2015ffc0 2024-01-24T17:49:55.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.970+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a200d3890 con 0x7f3a2015ffc0 2024-01-24T17:49:55.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.971+0000 7f3a17400640 1 -- 172.21.15.39:0/1540074536 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3a180437a0 con 0x7f3a2015ffc0 2024-01-24T17:49:55.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.972+0000 7f3a17400640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f39f806a430 0x7f39f806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:55.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.972+0000 7f3a17400640 1 -- 172.21.15.39:0/1540074536 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6197+0+0 (secure 0 0 0) 0x7f3a1803e070 con 0x7f3a2015ffc0 2024-01-24T17:49:55.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.973+0000 7f3a1ea00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f39f806a430 0x7f39f806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:55.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.974+0000 7f3a1ea00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f39f806a430 0x7f39f806c8f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3a100075c0 tx=0x7f3a1000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:55.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:55.977+0000 7f3a17400640 1 -- 172.21.15.39:0/1540074536 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f3a18076e40 con 0x7f3a2015ffc0 2024-01-24T17:49:56.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.176+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3a2004e840 con 0x7f3a2015ffc0 2024-01-24T17:49:56.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.716+0000 7f3a17400640 1 -- 172.21.15.39:0/1540074536 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v108) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3a180340d0 con 0x7f3a2015ffc0 2024-01-24T17:49:56.718 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-01-24T17:49:56.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f39f806a430 msgr2=0x7f39f806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:56.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f39f806a430 0x7f39f806c8f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3a100075c0 tx=0x7f3a1000b040 comp rx=0 tx=0).stop 2024-01-24T17:49:56.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2015ffc0 msgr2=0x7f3a200b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:56.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2015ffc0 0x7f3a200b04e0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f3a18009870 tx=0x7f3a18004730 comp rx=0 tx=0).stop 2024-01-24T17:49:56.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 shutdown_connections 2024-01-24T17:49:56.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f39f806a430 0x7f39f806c8f0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2015ffc0 0x7f3a200b04e0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3a200d04f0 0x7f3a201536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 --2- 172.21.15.39:0/1540074536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3a200cf870 0x7f3a201588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.722+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 >> 172.21.15.39:0/1540074536 conn(0x7f3a2009bc30 msgr2=0x7f3a200b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:56.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.723+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 shutdown_connections 2024-01-24T17:49:56.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.723+0000 7f3a1fe00640 1 -- 172.21.15.39:0/1540074536 wait complete. 2024-01-24T17:49:56.755 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-24T17:49:56.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.976+0000 7faa0e400640 1 Processor -- start 2024-01-24T17:49:56.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.976+0000 7faa0e400640 1 -- start start 2024-01-24T17:49:56.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.978+0000 7faa0e400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7faa080b5e40 0x7faa080cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:56.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.978+0000 7faa0e400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080d0cb0 con 0x7faa0815cd90 2024-01-24T17:49:56.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.978+0000 7faa0e400640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080cdd70 con 0x7faa080cf650 2024-01-24T17:49:56.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.978+0000 7faa0e400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa080ce050 con 0x7faa080b5e40 2024-01-24T17:49:56.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.978+0000 7faa07400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7faa0815cd90 0x7faa080cd320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60100/0 (socket says 172.21.15.39:60100) 2024-01-24T17:49:56.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.978+0000 7faa07400640 1 -- 172.21.15.39:0/601639076 learned_addr learned my addr 172.21.15.39:0/601639076 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:56.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.978+0000 7faa06a00640 1 --2- 172.21.15.39:0/601639076 >> v2:172.21.15.100:3300/0 conn(0x7faa080b5e40 0x7faa080cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:56.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670625342 0 0) 0x7faa080d0cb0 con 0x7faa0815cd90 2024-01-24T17:49:56.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e8003520 con 0x7faa0815cd90 2024-01-24T17:49:56.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 165084425 0 0) 0x7fa9e8003520 con 0x7faa0815cd90 2024-01-24T17:49:56.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 >> v2:172.21.15.100:3300/0 conn(0x7faa080b5e40 msgr2=0x7faa080cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:56.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 --2- 172.21.15.39:0/601639076 >> v2:172.21.15.100:3300/0 conn(0x7faa080b5e40 0x7faa080cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 >> v1:172.21.15.100:6789/0 conn(0x7faa080cf650 legacy=0x7faa080d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:56.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa080d0f20 con 0x7faa0815cd90 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fa9f8001670 con 0x7faa0815cd90 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06a00640 1 --2- 172.21.15.39:0/601639076 >> v2:172.21.15.100:3300/0 conn(0x7faa080b5e40 0x7faa080cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 >> v1:172.21.15.39:6789/0 conn(0x7faa0815cd90 legacy=0x7faa080cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e8003970 0x7fa9e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e80047c0 0x7fa9e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9e8005180 0x7fa9e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa080ce050 con 0x7fa9e8003970 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e800c120 con 0x7fa9e8005180 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.979+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e800c2e0 con 0x7fa9e80047c0 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07400640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e8003970 0x7fa9e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa06a00640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e80047c0 0x7fa9e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07e00640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9e8005180 0x7fa9e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07400640 1 -- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e80047c0 msgr2=0x7fa9e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07400640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e80047c0 0x7fa9e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07400640 1 -- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9e8005180 msgr2=0x7fa9e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:56.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07400640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9e8005180 0x7fa9e800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07400640 1 -- 172.21.15.39:0/601639076 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8003520 con 0x7fa9e8003970 2024-01-24T17:49:56.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa06a00640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e80047c0 0x7fa9e8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:56.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07e00640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa9e8005180 0x7fa9e800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:56.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.980+0000 7faa07e00640 1 -- 172.21.15.39:0/601639076 reap_dead start 2024-01-24T17:49:56.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.981+0000 7faa07400640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e8003970 0x7fa9e8003d90 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800c240 tx=0x7fa9e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a7bf4e7165dfbc server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:56.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.982+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8005ad0 con 0x7fa9e8003970 2024-01-24T17:49:56.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.982+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f8005c30 con 0x7fa9e8003970 2024-01-24T17:49:56.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.982+0000 7faa06000640 1 -- 172.21.15.39:0/601639076 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f802caf0 con 0x7fa9e8003970 2024-01-24T17:49:56.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.982+0000 7faa0e400640 1 -- 172.21.15.39:0/601639076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e8003970 msgr2=0x7fa9e8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:56.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.982+0000 7faa0e400640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e8003970 0x7fa9e8003d90 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800c240 tx=0x7fa9e80057f0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.982+0000 7faa0e400640 1 -- 172.21.15.39:0/601639076 shutdown_connections 2024-01-24T17:49:56.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.983+0000 7faa0e400640 1 --2- 172.21.15.39:0/601639076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9e8003970 0x7fa9e8003d90 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.983+0000 7faa0e400640 1 -- 172.21.15.39:0/601639076 >> 172.21.15.39:0/601639076 conn(0x7faa0806af00 msgr2=0x7faa0806b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:56.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.983+0000 7faa0e400640 1 -- 172.21.15.39:0/601639076 shutdown_connections 2024-01-24T17:49:56.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.983+0000 7faa0e400640 1 -- 172.21.15.39:0/601639076 wait complete. 2024-01-24T17:49:56.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.984+0000 7faa0e400640 1 Processor -- start 2024-01-24T17:49:56.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.985+0000 7faa0e400640 1 -- start start 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa0e400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e8003970 0x7faa08148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa0e400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa080b4680 0x7faa080b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa0e400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa0815ab10 0x7faa0815af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa0e400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa080b5040 con 0x7faa080b4680 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa0e400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0815b4d0 con 0x7faa0815ab10 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa0e400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa080b2ea0 con 0x7fa9e8003970 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa06a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa080b4680 0x7faa080b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa06a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa080b4680 0x7faa080b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48564/0 (socket says 172.21.15.39:48564) 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa06a00640 1 -- 172.21.15.39:0/713974214 learned_addr learned my addr 172.21.15.39:0/713974214 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa07400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e8003970 0x7faa08148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa06a00640 1 -- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e8003970 msgr2=0x7faa08148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:56.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa07e00640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa0815ab10 0x7faa0815af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:56.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa06a00640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e8003970 0x7faa08148dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa06a00640 1 -- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa0815ab10 msgr2=0x7faa0815af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:56.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa06a00640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa0815ab10 0x7faa0815af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:56.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.986+0000 7faa06a00640 1 -- 172.21.15.39:0/713974214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa080b3180 con 0x7faa080b4680 2024-01-24T17:49:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.987+0000 7faa06a00640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa080b4680 0x7faa080b4b00 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0010900 tx=0x7fa9f0010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.987+0000 7fa9ff400640 1 -- 172.21.15.39:0/713974214 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000eb60 con 0x7faa080b4680 2024-01-24T17:49:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.987+0000 7fa9ff400640 1 -- 172.21.15.39:0/713974214 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f000ecc0 con 0x7faa080b4680 2024-01-24T17:49:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.987+0000 7fa9ff400640 1 -- 172.21.15.39:0/713974214 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f001c400 con 0x7faa080b4680 2024-01-24T17:49:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.987+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa080b3460 con 0x7faa080b4680 2024-01-24T17:49:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.987+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa080b3a20 con 0x7faa080b4680 2024-01-24T17:49:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.988+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa08066f70 con 0x7faa080b4680 2024-01-24T17:49:56.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.989+0000 7fa9ff400640 1 -- 172.21.15.39:0/713974214 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fa9f001c560 con 0x7faa080b4680 2024-01-24T17:49:56.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.990+0000 7fa9ff400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa9e006a460 0x7fa9e006c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:56.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.990+0000 7fa9ff400640 1 -- 172.21.15.39:0/713974214 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5701+0+0 (secure 0 0 0) 0x7fa9f0088de0 con 0x7faa080b4680 2024-01-24T17:49:56.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.991+0000 7faa07400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa9e006a460 0x7fa9e006c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:56.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.992+0000 7faa07400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa9e006a460 0x7fa9e006c920 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8034000 tx=0x7fa9f8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:56.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:56.995+0000 7fa9ff400640 1 -- 172.21.15.39:0/713974214 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fa9f0054470 con 0x7faa080b4680 2024-01-24T17:49:57.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.188+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faa08067760 con 0x7faa080b4680 2024-01-24T17:49:57.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.730+0000 7fa9ff400640 1 -- 172.21.15.39:0/713974214 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v109) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa9f0004640 con 0x7faa080b4680 2024-01-24T17:49:57.731 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-01-24T17:49:57.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.735+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa9e006a460 msgr2=0x7fa9e006c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.735+0000 7faa0e400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa9e006a460 0x7fa9e006c920 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8034000 tx=0x7fa9f8009030 comp rx=0 tx=0).stop 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.735+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa080b4680 msgr2=0x7faa080b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.735+0000 7faa0e400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa080b4680 0x7faa080b4b00 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0010900 tx=0x7fa9f0010dc0 comp rx=0 tx=0).stop 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.736+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 shutdown_connections 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.736+0000 7faa0e400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faa0815ab10 0x7faa0815af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.736+0000 7faa0e400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fa9e006a460 0x7fa9e006c920 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.736+0000 7faa0e400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa080b4680 0x7faa080b4b00 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.736+0000 7faa0e400640 1 --2- 172.21.15.39:0/713974214 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fa9e8003970 0x7faa08148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.736+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 >> 172.21.15.39:0/713974214 conn(0x7faa0806af00 msgr2=0x7faa080b82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.736+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 shutdown_connections 2024-01-24T17:49:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:57.736+0000 7faa0e400640 1 -- 172.21.15.39:0/713974214 wait complete. 2024-01-24T17:49:57.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:49:57.768 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-24T17:49:57.922 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-01-24T17:49:57.922 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-01-24T17:49:57.922 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:49:57.922 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.923 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.924 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-01-24T17:49:57.925 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-01-24T17:49:57.926 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = bitmap 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-01-24T17:49:57.927 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-01-24T17:49:57.928 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.929 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: mon op complaint time = 120 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:49:57.930 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout: client mount timeout = 600 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout: debug client = 20 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = "" 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.a] 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.c] 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.b] 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.d] 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout:[mds] 2024-01-24T17:49:57.931 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds = 20 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: debug mds balancer = 20 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug frag = True 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: mds debug scatterstat = True 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: mds op complaint time = 180 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: mds verify scatter = True 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: osd op complaint time = 180 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: rados mon op timeout = 900 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout: rados osd op timeout = 900 2024-01-24T17:49:57.932 INFO:teuthology.orchestra.run.smithi100.stdout:[client.badguy] 2024-01-24T17:49:57.939 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-01-24T17:49:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-01-24T17:49:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-24T17:49:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-01-24T17:49:57.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-24T17:49:57.940 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.941 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,172.21.15.100,[v2:172.21.15.100:3301,v1:172.21.15.100:6790]" 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = b13477e8-9bd1-460b-97e1-2729dfdebd46 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-01-24T17:49:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-01-24T17:49:57.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:49:57.944 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:49:57.945 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-01-24T17:49:57.946 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-24T17:49:57.947 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2024-01-24T17:49:57.948 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 20 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2024-01-24T17:49:57.949 INFO:teuthology.orchestra.run.smithi039.stdout:[client.badguy] 2024-01-24T17:49:57.984 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-24T17:49:58.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.179+0000 7fe13ac00640 1 Processor -- start 2024-01-24T17:49:58.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.180+0000 7fe13ac00640 1 -- start start 2024-01-24T17:49:58.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.181+0000 7fe13ac00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe134159400 0x7fe1340cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.181+0000 7fe13ac00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe1340cdbf0 0x7fe1340ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.181+0000 7fe13ac00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe134070db0 con 0x7fe1340c8a70 2024-01-24T17:49:58.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.181+0000 7fe13ac00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1340c9bb0 con 0x7fe1340cdbf0 2024-01-24T17:49:58.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.181+0000 7fe13ac00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340c9030 con 0x7fe134159400 2024-01-24T17:49:58.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.182+0000 7fe139800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe134159400 0x7fe1340cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.182+0000 7fe139800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe134159400 0x7fe1340cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:37832/0 (socket says 172.21.15.39:37832) 2024-01-24T17:49:58.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.182+0000 7fe139800640 1 -- 172.21.15.39:0/2497850341 learned_addr learned my addr 172.21.15.39:0/2497850341 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:58.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.182+0000 7fe138e00640 1 --2- 172.21.15.39:0/2497850341 >> v2:172.21.15.100:3300/0 conn(0x7fe1340cdbf0 0x7fe1340ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.183+0000 7fe139800640 1 -- 172.21.15.39:0/2497850341 >> v2:172.21.15.100:3300/0 conn(0x7fe1340cdbf0 msgr2=0x7fe1340ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:49:58.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.183+0000 7fe139800640 1 --2- 172.21.15.39:0/2497850341 >> v2:172.21.15.100:3300/0 conn(0x7fe1340cdbf0 0x7fe1340ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.183+0000 7fe139800640 1 -- 172.21.15.39:0/2497850341 >> v1:172.21.15.100:6789/0 conn(0x7fe1340c8a70 legacy=0x7fe1340cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.183+0000 7fe139800640 1 -- 172.21.15.39:0/2497850341 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1340c9310 con 0x7fe134159400 2024-01-24T17:49:58.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.184+0000 7fe139800640 1 --2- 172.21.15.39:0/2497850341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe134159400 0x7fe1340cbff0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe124009870 tx=0x7fe12402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=375348801c2d310c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:58.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.185+0000 7fe133e00640 1 -- 172.21.15.39:0/2497850341 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe124031640 con 0x7fe134159400 2024-01-24T17:49:58.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.185+0000 7fe133e00640 1 -- 172.21.15.39:0/2497850341 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1240317a0 con 0x7fe134159400 2024-01-24T17:49:58.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.185+0000 7fe133e00640 1 -- 172.21.15.39:0/2497850341 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe124031b30 con 0x7fe134159400 2024-01-24T17:49:58.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.186+0000 7fe13ac00640 1 -- 172.21.15.39:0/2497850341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe134159400 msgr2=0x7fe1340cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.186+0000 7fe13ac00640 1 --2- 172.21.15.39:0/2497850341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe134159400 0x7fe1340cbff0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe124009870 tx=0x7fe12402b8f0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.186+0000 7fe13ac00640 1 -- 172.21.15.39:0/2497850341 shutdown_connections 2024-01-24T17:49:58.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.186+0000 7fe13ac00640 1 --2- 172.21.15.39:0/2497850341 >> v2:172.21.15.100:3300/0 conn(0x7fe1340cdbf0 0x7fe1340ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.186+0000 7fe13ac00640 1 --2- 172.21.15.39:0/2497850341 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe134159400 0x7fe1340cbff0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.186+0000 7fe13ac00640 1 -- 172.21.15.39:0/2497850341 >> 172.21.15.39:0/2497850341 conn(0x7fe13406a200 msgr2=0x7fe1340affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:58.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.186+0000 7fe13ac00640 1 -- 172.21.15.39:0/2497850341 shutdown_connections 2024-01-24T17:49:58.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.186+0000 7fe13ac00640 1 -- 172.21.15.39:0/2497850341 wait complete. 2024-01-24T17:49:58.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.188+0000 7fe13ac00640 1 Processor -- start 2024-01-24T17:49:58.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.188+0000 7fe13ac00640 1 -- start start 2024-01-24T17:49:58.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe13ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1340c8a70 0x7fe134152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe13ac00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1340cdbf0 0x7fe1340ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe13ac00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe134159400 0x7fe1340ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe13ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340c5f30 con 0x7fe1340c8a70 2024-01-24T17:49:58.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe13ac00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340c7ea0 con 0x7fe134159400 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe13ac00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340c72f0 con 0x7fe1340cdbf0 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe139800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1340c8a70 0x7fe134152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe139800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1340c8a70 0x7fe134152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48574/0 (socket says 172.21.15.39:48574) 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.189+0000 7fe139800640 1 -- 172.21.15.39:0/895589509 learned_addr learned my addr 172.21.15.39:0/895589509 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe138e00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1340cdbf0 0x7fe1340ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe13a200640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe134159400 0x7fe1340ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe139800640 1 -- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1340cdbf0 msgr2=0x7fe1340ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe139800640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1340cdbf0 0x7fe1340ce650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe139800640 1 -- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe134159400 msgr2=0x7fe1340ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe139800640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe134159400 0x7fe1340ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe139800640 1 -- 172.21.15.39:0/895589509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1240094d0 con 0x7fe1340c8a70 2024-01-24T17:49:58.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe13a200640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe134159400 0x7fe1340ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe139800640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1340c8a70 0x7fe134152310 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe1240090b0 tx=0x7fe1240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe131600640 1 -- 172.21.15.39:0/895589509 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12403b940 con 0x7fe1340c8a70 2024-01-24T17:49:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe131600640 1 -- 172.21.15.39:0/895589509 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12403bac0 con 0x7fe1340c8a70 2024-01-24T17:49:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe131600640 1 -- 172.21.15.39:0/895589509 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12403be70 con 0x7fe1340c8a70 2024-01-24T17:49:58.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.190+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1340c8330 con 0x7fe1340c8a70 2024-01-24T17:49:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.191+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1340c7850 con 0x7fe1340c8a70 2024-01-24T17:49:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.192+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe13404dff0 con 0x7fe1340c8a70 2024-01-24T17:49:58.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.193+0000 7fe131600640 1 -- 172.21.15.39:0/895589509 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fe12402f050 con 0x7fe1340c8a70 2024-01-24T17:49:58.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.194+0000 7fe131600640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe10c06a430 0x7fe10c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.194+0000 7fe131600640 1 -- 172.21.15.39:0/895589509 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fe1240aa660 con 0x7fe1340c8a70 2024-01-24T17:49:58.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.194+0000 7fe138e00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe10c06a430 0x7fe10c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.195+0000 7fe138e00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe10c06a430 0x7fe10c06c8f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe1340b0880 tx=0x7fe11c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:58.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.198+0000 7fe131600640 1 -- 172.21.15.39:0/895589509 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fe124075d60 con 0x7fe1340c8a70 2024-01-24T17:49:58.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.392+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe134071ab0 con 0x7fe1340c8a70 2024-01-24T17:49:58.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.401+0000 7fe131600640 1 -- 172.21.15.39:0/895589509 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe124079380 con 0x7fe1340c8a70 2024-01-24T17:49:58.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe10c06a430 msgr2=0x7fe10c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe10c06a430 0x7fe10c06c8f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe1340b0880 tx=0x7fe11c008040 comp rx=0 tx=0).stop 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1340c8a70 msgr2=0x7fe134152310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1340c8a70 0x7fe134152310 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe1240090b0 tx=0x7fe1240046c0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 shutdown_connections 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe134159400 0x7fe1340ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe1340cdbf0 0x7fe1340ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fe10c06a430 0x7fe10c06c8f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 --2- 172.21.15.39:0/895589509 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1340c8a70 0x7fe134152310 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 >> 172.21.15.39:0/895589509 conn(0x7fe13406a200 msgr2=0x7fe1340cd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 shutdown_connections 2024-01-24T17:49:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.404+0000 7fe13ac00640 1 -- 172.21.15.39:0/895589509 wait complete. 2024-01-24T17:49:58.423 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_version_splitting' 2024-01-24T17:49:58.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.633+0000 7fddada00640 1 Processor -- start 2024-01-24T17:49:58.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.634+0000 7fddada00640 1 -- start start 2024-01-24T17:49:58.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.635+0000 7fddada00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fdda80b0010 0x7fdda815bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.635+0000 7fddada00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8070470 con 0x7fdda815f390 2024-01-24T17:49:58.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.635+0000 7fddada00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda815ef00 con 0x7fdda8070920 2024-01-24T17:49:58.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.635+0000 7fddada00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda80710a0 con 0x7fdda80b0010 2024-01-24T17:49:58.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.636+0000 7fdda7400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fdda815f390 0x7fdda80c9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60108/0 (socket says 172.21.15.39:60108) 2024-01-24T17:49:58.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.636+0000 7fdda7400640 1 -- 172.21.15.39:0/554963329 learned_addr learned my addr 172.21.15.39:0/554963329 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:49:58.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.636+0000 7fdda6a00640 1 --2- 172.21.15.39:0/554963329 >> v2:172.21.15.100:3300/0 conn(0x7fdda80b0010 0x7fdda815bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.636+0000 7fdda6000640 1 -- 172.21.15.39:0/554963329 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488129294 0 0) 0x7fdda8070470 con 0x7fdda815f390 2024-01-24T17:49:58.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.636+0000 7fdda6000640 1 -- 172.21.15.39:0/554963329 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd88003520 con 0x7fdda815f390 2024-01-24T17:49:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.637+0000 7fdda6000640 1 -- 172.21.15.39:0/554963329 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1911553767 0 0) 0x7fdd88003520 con 0x7fdda815f390 2024-01-24T17:49:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.637+0000 7fdda6a00640 1 -- 172.21.15.39:0/554963329 >> v1:172.21.15.100:6789/0 conn(0x7fdda8070920 legacy=0x7fdda80b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.637+0000 7fdda6a00640 1 -- 172.21.15.39:0/554963329 >> v1:172.21.15.39:6789/0 conn(0x7fdda815f390 legacy=0x7fdda80c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.637+0000 7fdda6a00640 1 -- 172.21.15.39:0/554963329 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda815c0d0 con 0x7fdda80b0010 2024-01-24T17:49:58.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.638+0000 7fdda6a00640 1 --2- 172.21.15.39:0/554963329 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda80b0010 0x7fdda815bb90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdd980097a0 tx=0x7fdd9802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=233f415d8ebc64fb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:58.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fdda6000640 1 -- 172.21.15.39:0/554963329 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda00019f0 con 0x7fdda80b0010 2024-01-24T17:49:58.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fdda6000640 1 -- 172.21.15.39:0/554963329 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd98031590 con 0x7fdda80b0010 2024-01-24T17:49:58.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fdda6000640 1 -- 172.21.15.39:0/554963329 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd98031900 con 0x7fdda80b0010 2024-01-24T17:49:58.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fddada00640 1 -- 172.21.15.39:0/554963329 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda80b0010 msgr2=0x7fdda815bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fddada00640 1 --2- 172.21.15.39:0/554963329 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda80b0010 0x7fdda815bb90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdd980097a0 tx=0x7fdd9802b8b0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fddada00640 1 -- 172.21.15.39:0/554963329 shutdown_connections 2024-01-24T17:49:58.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fddada00640 1 --2- 172.21.15.39:0/554963329 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda80b0010 0x7fdda815bb90 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fddada00640 1 -- 172.21.15.39:0/554963329 >> 172.21.15.39:0/554963329 conn(0x7fdda806bf20 msgr2=0x7fdda806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:58.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.639+0000 7fddada00640 1 -- 172.21.15.39:0/554963329 shutdown_connections 2024-01-24T17:49:58.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.640+0000 7fddada00640 1 -- 172.21.15.39:0/554963329 wait complete. 2024-01-24T17:49:58.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.641+0000 7fddada00640 1 Processor -- start 2024-01-24T17:49:58.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.641+0000 7fddada00640 1 -- start start 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.642+0000 7fddada00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdda8070920 0x7fdda81419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.642+0000 7fddada00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdda80b0010 0x7fdda8144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.642+0000 7fddada00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda815f390 0x7fdda80c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.642+0000 7fddada00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda8145150 con 0x7fdda8070920 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.642+0000 7fddada00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda80c7620 con 0x7fdda815f390 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.642+0000 7fddada00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda80c7870 con 0x7fdda80b0010 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdda8070920 0x7fdda81419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdda8070920 0x7fdda81419d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48586/0 (socket says 172.21.15.39:48586) 2024-01-24T17:49:58.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 -- 172.21.15.39:0/461763384 learned_addr learned my addr 172.21.15.39:0/461763384 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda7400640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda815f390 0x7fdda80c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdd9fe00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdda80b0010 0x7fdda8144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 -- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdda80b0010 msgr2=0x7fdda8144c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdda80b0010 0x7fdda8144c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 -- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda815f390 msgr2=0x7fdda80c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda815f390 0x7fdda80c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 -- 172.21.15.39:0/461763384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd98009400 con 0x7fdda8070920 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda7400640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda815f390 0x7fdda80c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.643+0000 7fdda6a00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdda8070920 0x7fdda81419d0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fdd9802b8f0 tx=0x7fdd980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.644+0000 7fdd9ea00640 1 -- 172.21.15.39:0/461763384 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd980044c0 con 0x7fdda8070920 2024-01-24T17:49:58.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.644+0000 7fdd9ea00640 1 -- 172.21.15.39:0/461763384 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd98040070 con 0x7fdda8070920 2024-01-24T17:49:58.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.644+0000 7fdd9ea00640 1 -- 172.21.15.39:0/461763384 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd98003830 con 0x7fdda8070920 2024-01-24T17:49:58.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.644+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda80c7b50 con 0x7fdda8070920 2024-01-24T17:49:58.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.644+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda80c8110 con 0x7fdda8070920 2024-01-24T17:49:58.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.645+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda804dff0 con 0x7fdda8070920 2024-01-24T17:49:58.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.646+0000 7fdd9ea00640 1 -- 172.21.15.39:0/461763384 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdd98003990 con 0x7fdda8070920 2024-01-24T17:49:58.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.647+0000 7fdd9ea00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd8006a430 0x7fdd8006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:49:58.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.647+0000 7fdd9ea00640 1 -- 172.21.15.39:0/461763384 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fdd98042070 con 0x7fdda8070920 2024-01-24T17:49:58.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.648+0000 7fdd9fe00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd8006a430 0x7fdd8006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:49:58.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.649+0000 7fdd9fe00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd8006a430 0x7fdd8006c8f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdda80af600 tx=0x7fdd90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:49:58.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.651+0000 7fdd9ea00640 1 -- 172.21.15.39:0/461763384 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fdd98076500 con 0x7fdda8070920 2024-01-24T17:49:58.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:58.917+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_version_splitting"]} v 0) v1 -- 0x7fdda8070700 con 0x7fdda8070920 2024-01-24T17:49:59.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.752+0000 7fdd9ea00640 1 -- 172.21.15.39:0/461763384 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_version_splitting"]}]=0 v436) v1 ==== 145+0+0 (secure 0 0 0) 0x7fdd9804c0d0 con 0x7fdda8070920 2024-01-24T17:49:59.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd8006a430 msgr2=0x7fdd8006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd8006a430 0x7fdd8006c8f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdda80af600 tx=0x7fdd90009040 comp rx=0 tx=0).stop 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdda8070920 msgr2=0x7fdda81419d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdda8070920 0x7fdda81419d0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fdd9802b8f0 tx=0x7fdd980046c0 comp rx=0 tx=0).stop 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 shutdown_connections 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdda815f390 0x7fdda80c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fdda80b0010 0x7fdda8144c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fdd8006a430 0x7fdd8006c8f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 --2- 172.21.15.39:0/461763384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdda8070920 0x7fdda81419d0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.755+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 >> 172.21.15.39:0/461763384 conn(0x7fdda806bf20 msgr2=0x7fdda80ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.756+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 shutdown_connections 2024-01-24T17:49:59.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:49:59.756+0000 7fddada00640 1 -- 172.21.15.39:0/461763384 wait complete. 2024-01-24T17:49:59.781 INFO:tasks.cephfs_test_runner:That when many sessions are updated, they are correctly ... ok 2024-01-24T17:49:59.781 INFO:tasks.cephfs_test_runner: 2024-01-24T17:49:59.781 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-24T17:49:59.781 INFO:tasks.cephfs_test_runner:Ran 6 tests in 461.852s 2024-01-24T17:49:59.781 INFO:tasks.cephfs_test_runner: 2024-01-24T17:49:59.781 INFO:tasks.cephfs_test_runner:OK 2024-01-24T17:49:59.782 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-24T17:49:59.791 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-01-24T17:49:59.813 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-01-24T17:49:59.813 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:49:59.813 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:59.873 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:49:59.873 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-01-24T17:49:59.873 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:59.929 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053862,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498820k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-24T17:49:59.930 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:49:59.930 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:49:59.987 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:49:59.987 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-01-24T17:49:59.987 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T17:50:00.040 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053818,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249372k,nr_inodes=812343,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498748k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-24T17:50:00.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:50:00.041 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-01-24T17:50:00.102 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-24T17:50:00.103 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-24T17:50:00.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:50:00.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-24T17:50:00.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:50:00.205 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-24T17:50:00.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:50:00.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-01-24T17:50:00.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:50:00.422 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-01-24T17:50:00.480 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-24T17:50:00.480 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-24T17:50:00.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:50:00.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-24T17:50:00.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:50:00.579 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-24T17:50:00.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T17:50:00.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link delete ceph-brx 2024-01-24T17:50:00.803 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-24T17:50:00.812 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-24T17:50:00.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:50:01.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.045+0000 7faeb8e00640 1 Processor -- start 2024-01-24T17:50:01.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.045+0000 7faeb8e00640 1 -- start start 2024-01-24T17:50:01.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.047+0000 7faeb8e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7faeb40cdaf0 0x7faeb406bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.047+0000 7faeb8e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb41540f0 con 0x7faeb40ca320 2024-01-24T17:50:01.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.047+0000 7faeb8e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb415da00 con 0x7faeb40ca700 2024-01-24T17:50:01.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.047+0000 7faeb8e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb4159fd0 con 0x7faeb40cdaf0 2024-01-24T17:50:01.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.047+0000 7faeb3400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7faeb40ca320 0x7faeb415d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60110/0 (socket says 172.21.15.39:60110) 2024-01-24T17:50:01.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.047+0000 7faeb3400640 1 -- 172.21.15.39:0/3791794843 learned_addr learned my addr 172.21.15.39:0/3791794843 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:01.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb3e00640 1 --2- 172.21.15.39:0/3791794843 >> v2:172.21.15.100:3300/0 conn(0x7faeb40cdaf0 0x7faeb406bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094564560 0 0) 0x7faeb41540f0 con 0x7faeb40ca320 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae94003520 con 0x7faeb40ca320 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1473491132 0 0) 0x7fae94003520 con 0x7faeb40ca320 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 >> v2:172.21.15.100:3300/0 conn(0x7faeb40cdaf0 msgr2=0x7faeb406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 --2- 172.21.15.39:0/3791794843 >> v2:172.21.15.100:3300/0 conn(0x7faeb40cdaf0 0x7faeb406bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 >> v1:172.21.15.100:6789/0 conn(0x7faeb40ca700 legacy=0x7faeb415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb415dce0 con 0x7faeb40ca320 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7faea40017f0 con 0x7faeb40ca320 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 >> v1:172.21.15.39:6789/0 conn(0x7faeb40ca320 legacy=0x7faeb415d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae94003970 0x7fae94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae940047c0 0x7fae94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7fae9400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4159fd0 con 0x7fae94005380 2024-01-24T17:50:01.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9400c210 con 0x7fae94003970 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.048+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9400c3d0 con 0x7fae940047c0 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.049+0000 7faeb3e00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7fae9400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.049+0000 7faeb3400640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae94003970 0x7fae94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.049+0000 7faeabe00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae940047c0 0x7fae94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.049+0000 7faeb3e00640 1 -- 172.21.15.39:0/3791794843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae940047c0 msgr2=0x7fae94004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.049+0000 7faeb3e00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fae940047c0 0x7fae94004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.049+0000 7faeb3e00640 1 -- 172.21.15.39:0/3791794843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae94003970 msgr2=0x7fae94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.049+0000 7faeb3e00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fae94003970 0x7fae94003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.049+0000 7faeb3e00640 1 -- 172.21.15.39:0/3791794843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94003520 con 0x7fae94005380 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.050+0000 7faeb3e00640 1 -- 172.21.15.39:0/3791794843 reap_dead start 2024-01-24T17:50:01.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.050+0000 7faeb3e00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7fae9400b980 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7faeac007330 tx=0x7fae940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea080c9fc4cd06cc server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac02ccd0 con 0x7fae94005380 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac02ce30 con 0x7fae94005380 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb2a00640 1 -- 172.21.15.39:0/3791794843 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac034660 con 0x7fae94005380 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb8e00640 1 -- 172.21.15.39:0/3791794843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 msgr2=0x7fae9400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb8e00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7fae9400b980 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7faeac007330 tx=0x7fae940057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb8e00640 1 -- 172.21.15.39:0/3791794843 shutdown_connections 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb8e00640 1 --2- 172.21.15.39:0/3791794843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7fae9400b980 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb8e00640 1 -- 172.21.15.39:0/3791794843 >> 172.21.15.39:0/3791794843 conn(0x7faeb406a340 msgr2=0x7faeb40cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb8e00640 1 -- 172.21.15.39:0/3791794843 shutdown_connections 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.051+0000 7faeb8e00640 1 -- 172.21.15.39:0/3791794843 wait complete. 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.053+0000 7faeb8e00640 1 Processor -- start 2024-01-24T17:50:01.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.053+0000 7faeb8e00640 1 -- start start 2024-01-24T17:50:01.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7faeb415afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb8e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeb415b510 0x7faeb4140be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb8e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeb4141120 0x7faeb41415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb8e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40c1560 con 0x7fae94005380 2024-01-24T17:50:01.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb8e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb414efc0 con 0x7faeb415b510 2024-01-24T17:50:01.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb8e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4156ca0 con 0x7faeb4141120 2024-01-24T17:50:01.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7faeb415afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7faeb415afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48612/0 (socket says 172.21.15.39:48612) 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.054+0000 7faeb3400640 1 -- 172.21.15.39:0/1749220182 learned_addr learned my addr 172.21.15.39:0/1749220182 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeb3400640 1 -- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeb4141120 msgr2=0x7faeb41415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeb3400640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeb4141120 0x7faeb41415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeb3400640 1 -- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeb415b510 msgr2=0x7faeb4140be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeb3400640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeb415b510 0x7faeb4140be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeb3400640 1 -- 172.21.15.39:0/1749220182 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb415b990 con 0x7fae94005380 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeb3400640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7faeb415afd0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7faea400e780 tx=0x7faea400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeab400640 1 -- 172.21.15.39:0/1749220182 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea40070d0 con 0x7fae94005380 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeab400640 1 -- 172.21.15.39:0/1749220182 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea4007250 con 0x7fae94005380 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.055+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb415bc20 con 0x7fae94005380 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.056+0000 7faeab400640 1 -- 172.21.15.39:0/1749220182 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4007600 con 0x7fae94005380 2024-01-24T17:50:01.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.056+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb415c1e0 con 0x7fae94005380 2024-01-24T17:50:01.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.057+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb4156a50 con 0x7fae94005380 2024-01-24T17:50:01.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.058+0000 7faeab400640 1 -- 172.21.15.39:0/1749220182 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7faea40057a0 con 0x7fae94005380 2024-01-24T17:50:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.058+0000 7faeab400640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8c06a430 0x7fae8c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.059+0000 7faeab400640 1 -- 172.21.15.39:0/1749220182 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7faea40887d0 con 0x7fae94005380 2024-01-24T17:50:01.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.059+0000 7faeabe00640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8c06a430 0x7fae8c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.060+0000 7faeabe00640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8c06a430 0x7fae8c06c8f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fae9c000f60 tx=0x7fae9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.063+0000 7faeab400640 1 -- 172.21.15.39:0/1749220182 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7faea4053ec0 con 0x7fae94005380 2024-01-24T17:50:01.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.253+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faeb4159bc0 con 0x7fae8c06a430 2024-01-24T17:50:01.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.259+0000 7faeab400640 1 -- 172.21.15.39:0/1749220182 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48729 (secure 0 0 0) 0x7fae9c001290 con 0x7fae8c06a430 2024-01-24T17:50:01.261 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:50:01.261 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:50:01.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8c06a430 msgr2=0x7fae8c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8c06a430 0x7fae8c06c8f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fae9c000f60 tx=0x7fae9c009040 comp rx=0 tx=0).stop 2024-01-24T17:50:01.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 msgr2=0x7faeb415afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7faeb415afd0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7faea400e780 tx=0x7faea400eb80 comp rx=0 tx=0).stop 2024-01-24T17:50:01.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 shutdown_connections 2024-01-24T17:50:01.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7faeb4141120 0x7faeb41415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faeb415b510 0x7faeb4140be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fae8c06a430 0x7fae8c06c8f0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 --2- 172.21.15.39:0/1749220182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae94005380 0x7faeb415afd0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.262+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 >> 172.21.15.39:0/1749220182 conn(0x7faeb406a340 msgr2=0x7faeb40c28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:01.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.263+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 shutdown_connections 2024-01-24T17:50:01.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.263+0000 7faeb8e00640 1 -- 172.21.15.39:0/1749220182 wait complete. 2024-01-24T17:50:01.282 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-01-24T17:50:00.393912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260532,"kb_used_data":11604,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754714188,"statfs":{"total":773094113280,"available":772827328512,"internally_reserved":0,"allocated":11882496,"data_stored":5820749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27738,"num_objects":-42,"num_object_clones":0,"num_object_copies":-84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-42,"num_whiteouts":0,"num_read":-95,"num_read_kb":-89,"num_write":-167,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002067"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":243,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-01-24T17:49:55.821460+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:49:55.821460+0000","last_peered":"2024-01-24T17:49:55.821460+0000","last_clean":"2024-01-24T17:49:55.821460+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:49:55.821460+0000","last_undegraded":"2024-01-24T17:49:55.821460+0000","last_fullsized":"2024-01-24T17:49:55.821460+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1331200,"data_stored":576702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0720000000000001,"15min":1.0449999999999999},"min":{"1min":0.73299999999999998,"5min":0.64900000000000002,"15min":0.61899999999999999},"max":{"1min":1.653,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.1359999999999999,"15min":1.0700000000000001},"min":{"1min":0.63400000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.411,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.753}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.68200000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.472,"5min":1.6479999999999999,"15min":2.04},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2070000000000001,"15min":1.196},"min":{"1min":0.75700000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.252}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1779999999999999,"15min":1.167},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.59099999999999997},"max":{"1min":1.484,"5min":1.992,"15min":2.1400000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.109,"5min":1.1910000000000001,"15min":1.2030000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.577,"5min":2.056,"15min":2.484},"last":1.0609999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2350000000000001,"15min":1.244},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.843,"5min":2.0179999999999998,"15min":2.2930000000000001},"last":1.232},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.3100000000000001,"15min":1.3049999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5740000000000001,"5min":1.9059999999999999,"15min":2.6269999999999998},"last":1.5640000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2729999999999999,"15min":1.3009999999999999},"min":{"1min":0.72999999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5029999999999999,"5min":2.2549999999999999,"15min":2.6179999999999999},"last":1.524},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3879999999999999,"15min":1.4179999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":2.6459999999999999},"last":1.4710000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.3320000000000001},"min":{"1min":0.58799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.885,"5min":1.9810000000000001,"15min":2.8460000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3740000000000001,"15min":1.4079999999999999},"min":{"1min":0.68300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6359999999999999,"5min":2.2480000000000002,"15min":2.3719999999999999},"last":1.6279999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.47,"15min":1.5069999999999999},"min":{"1min":0.83399999999999996,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.7190000000000001,"5min":2.4129999999999998,"15min":2.6960000000000002},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4219999999999999,"15min":1.4630000000000001},"min":{"1min":0.66600000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.946,"5min":2.0790000000000002,"15min":2.4830000000000001},"last":1.1479999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574953,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1339392,"data_stored":582411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0740000000000001,"15min":1.042},"min":{"1min":0.69699999999999995,"5min":0.69599999999999995,"15min":0.61199999999999999},"max":{"1min":1.4390000000000001,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0820000000000001,"15min":1.044},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.66300000000000003},"max":{"1min":1.5760000000000001,"5min":1.744,"15min":1.744},"last":1.0129999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.169,"15min":1.056},"min":{"1min":0.88500000000000001,"5min":0.71199999999999997,"15min":0.64200000000000002},"max":{"1min":2.9790000000000001,"5min":2.9790000000000001,"15min":2.9790000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.1379999999999999,"15min":1.1279999999999999},"min":{"1min":0.71599999999999997,"5min":0.67800000000000005,"15min":0.56399999999999995},"max":{"1min":1.425,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":1.0680000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.1970000000000001,"15min":1.155},"min":{"1min":0.85499999999999998,"5min":0.71899999999999997,"15min":0.63600000000000001},"max":{"1min":1.675,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.22,"15min":1.1850000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.56599999999999995},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.2050000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2709999999999999,"15min":1.258},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73399999999999999},"max":{"1min":3.21,"5min":3.21,"15min":3.21},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.234,"15min":1.2430000000000001},"min":{"1min":0.91100000000000003,"5min":0.84999999999999998,"15min":0.69799999999999995},"max":{"1min":1.554,"5min":2.0489999999999999,"15min":2.46},"last":0.93799999999999994}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3080000000000001,"15min":1.264},"min":{"1min":0.59199999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.3240000000000001,"15min":1.3089999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.44600000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.4339999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3380000000000001,"15min":1.3120000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.46800000000000003},"max":{"1min":3.3759999999999999,"5min":3.3759999999999999,"15min":3.3759999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.462,"15min":1.4419999999999999},"min":{"1min":0.67100000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.4849999999999999},"last":1.399}]},{"osd":7,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.361,"15min":1.363},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.51400000000000001},"max":{"1min":1.786,"5min":1.921,"15min":2.2050000000000001},"last":1.345},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4710000000000001,"15min":1.464},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.52300000000000002},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.6020000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574955,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1933312,"data_stored":1172779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0680000000000001,"15min":1.083},"min":{"1min":0.72099999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5389999999999999,"5min":3.609,"15min":3.609},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.0680000000000001,"15min":1.0429999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.391,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.629}]},{"osd":1,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.119,"15min":1.089},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.622},"max":{"1min":1.5249999999999999,"5min":3.5470000000000002,"15min":3.5470000000000002},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.153,"15min":1.22},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.544,"5min":1.859,"15min":2.1579999999999999},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.224,"15min":1.226},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.59199999999999997},"max":{"1min":1.6399999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3500000000000001,"15min":1.319},"min":{"1min":0.96499999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.663,"5min":8.1509999999999998,"15min":8.1509999999999998},"last":0.93600000000000005}]},{"osd":3,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2569999999999999,"15min":1.3049999999999999},"min":{"1min":0.99099999999999999,"5min":0.76400000000000001,"15min":0.70799999999999996},"max":{"1min":1.466,"5min":2.7559999999999998,"15min":3.036},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.28,"15min":1.3109999999999999},"min":{"1min":0.70499999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.474,"5min":4.1230000000000002,"15min":4.1230000000000002},"last":1.3420000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.355,"15min":1.417},"min":{"1min":0.68600000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6259999999999999,"5min":1.9690000000000001,"15min":2.069},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.254,"15min":1.319},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.77,"5min":2.1139999999999999,"15min":3.2810000000000001},"last":1.6220000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.304,"15min":1.3360000000000001},"min":{"1min":0.871,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.754,"5min":2.028,"15min":2.504},"last":1.9419999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.405,"15min":1.484},"min":{"1min":0.63800000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.575,"5min":2.1059999999999999,"15min":2.2290000000000001},"last":1.881}]},{"osd":7,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3280000000000001,"15min":1.4319999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.6759999999999999,"5min":2.2010000000000001,"15min":3.161},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.5129999999999999,"15min":1.5860000000000001},"min":{"1min":1.042,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8420000000000001,"5min":2.262,"15min":3.2410000000000001},"last":1.8169999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574956,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1331200,"data_stored":576702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.0589999999999999},"min":{"1min":0.73299999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3540000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.099,"15min":1.097},"min":{"1min":0.874,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6359999999999999,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.86799999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1739999999999999,"15min":1.097},"min":{"1min":0.78900000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5980000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1679999999999999,"15min":1.196},"min":{"1min":0.73099999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.698,"5min":1.712,"15min":1.849},"last":1.054}]},{"osd":2,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2430000000000001,"15min":1.2210000000000001},"min":{"1min":0.97299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.788,"5min":1.9850000000000001,"15min":2.105},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3080000000000001,"15min":1.3200000000000001},"min":{"1min":0.99399999999999999,"5min":0.69599999999999995,"15min":0.64500000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.986},"last":1.181}]},{"osd":3,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1819999999999999,"15min":1.1870000000000001},"min":{"1min":0.76800000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":1.637,"15min":1.7769999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.296,"15min":1.331},"min":{"1min":0.998,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.9409999999999998},"last":1.4159999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.3169999999999999,"15min":1.3500000000000001},"min":{"1min":1.044,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0979999999999999},"last":1.758},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":0.91700000000000004,"5min":0.65600000000000003,"15min":0.55200000000000005},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.9039999999999999},"last":1.9419999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.351,"15min":1.429},"min":{"1min":0.95599999999999996,"5min":0.67800000000000005,"15min":0.64400000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":3.0840000000000001},"last":1.901},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4550000000000001,"15min":1.468},"min":{"1min":1.1499999999999999,"5min":0.67700000000000005,"15min":0.66500000000000004},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.8759999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":1.0129999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.4910000000000001,"15min":1.5740000000000001},"min":{"1min":0.98399999999999999,"5min":0.79300000000000004,"15min":0.60399999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":4.3250000000000002},"last":1.6319999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1339392,"data_stored":582411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.86599999999999999},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.4590000000000001,"5min":1.5580000000000001,"15min":5.1929999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.88800000000000001,"15min":0.90100000000000002},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.4279999999999999,"5min":1.6739999999999999,"15min":8.6989999999999998},"last":1.1990000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.91700000000000004},"min":{"1min":0.46600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.4099999999999999,"5min":1.5229999999999999,"15min":1.8720000000000001},"last":1.282},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0389999999999999,"15min":1.0580000000000001},"min":{"1min":0.69499999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.679,"5min":1.679,"15min":3.7240000000000002},"last":0.82799999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0760000000000001,"15min":1.0980000000000001},"min":{"1min":0.499,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.581,"5min":1.7170000000000001,"15min":3.4630000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1279999999999999,"15min":1.155},"min":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":4.2919999999999998},"last":1.141}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.208,"15min":1.2689999999999999},"min":{"1min":0.76700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6060000000000001,"5min":2.0049999999999999,"15min":6.7469999999999999},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.2789999999999999,"15min":1.3280000000000001},"min":{"1min":0.873,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.6950000000000001,"5min":2.0390000000000001,"15min":3.4049999999999998},"last":1.448}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3129999999999999,"15min":1.395},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7050000000000001,"5min":2.1269999999999998,"15min":4.3789999999999996},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.363,"15min":1.4019999999999999},"min":{"1min":0.78300000000000003,"5min":0.746,"15min":0.746},"max":{"1min":2.1859999999999999,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":2.2669999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.411,"15min":1.4770000000000001},"min":{"1min":0.82199999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.331,"5min":2.331,"15min":9.5440000000000005},"last":1.851},{"interface":"front","average":{"1min":1.284,"5min":1.327,"15min":1.419},"min":{"1min":0.78300000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.659,"5min":2.109,"15min":4.8170000000000002},"last":1.3640000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.373,"15min":1.472},"min":{"1min":0.90000000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.736,"5min":2.3410000000000002,"15min":6.6970000000000001},"last":2.0350000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.431,"15min":1.4970000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.889,"5min":2.0990000000000002,"15min":6.6529999999999996},"last":1.6279999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1347584,"data_stored":580845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.78700000000000003,"15min":0.79300000000000004},"min":{"1min":0.42099999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.6180000000000001,"5min":1.667,"15min":1.667},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90000000000000002,"15min":0.92600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.472,"5min":1.7070000000000001,"15min":1.7989999999999999},"last":0.96499999999999997}]},{"osd":1,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.018,"15min":1.04},"min":{"1min":0.52100000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.026},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1080000000000001,"15min":1.1719999999999999},"min":{"1min":0.56000000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5960000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.242}]},{"osd":3,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95799999999999996,"15min":0.98099999999999998},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.7130000000000001,"15min":2.2120000000000002},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.1459999999999999,"15min":1.1919999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.996},"last":1.6919999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.244,"15min":1.2969999999999999},"min":{"1min":0.79500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.748,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.8939999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.256,"15min":1.349},"min":{"1min":0.71899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7250000000000001,"5min":1.889,"15min":2.3839999999999999},"last":1.677}]},{"osd":5,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.371},"min":{"1min":0.99299999999999999,"5min":0.81899999999999995,"15min":0.73899999999999999},"max":{"1min":1.6879999999999999,"5min":1.851,"15min":2.3140000000000001},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.2230000000000001,"15min":1.2889999999999999},"min":{"1min":0.95599999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.804,"5min":1.889,"15min":2.0720000000000001},"last":1.365}]},{"osd":6,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.294,"15min":1.3600000000000001},"min":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.623},"max":{"1min":2.056,"5min":2.056,"15min":2.2679999999999998},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.335,"15min":1.4259999999999999},"min":{"1min":0.92100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":3.7410000000000001},"last":2.0019999999999998}]},{"osd":7,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.383,"15min":1.508},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6879999999999999,"5min":2.1499999999999999,"15min":3.343},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.375,"15min":1.458},"min":{"1min":0.76100000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.1600000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.478}]}]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1933312,"data_stored":1172779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.85599999999999998,"15min":0.86099999999999999},"min":{"1min":0.42999999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.54,"5min":1.722,"15min":2.0339999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.95999999999999996,"15min":0.95199999999999996},"min":{"1min":0.53700000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.5089999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.70999999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88100000000000001,"15min":0.94499999999999995},"min":{"1min":0.51300000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3580000000000001,"5min":1.3959999999999999,"15min":1.6379999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.069,"15min":1.0900000000000001},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.41799999999999998},"max":{"1min":1.556,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.177}]},{"osd":3,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.171,"15min":1.163},"min":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.47699999999999998},"max":{"1min":1.623,"5min":1.8440000000000001,"15min":1.9610000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.0760000000000001,"15min":1.1799999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.41499999999999998},"max":{"1min":1.446,"5min":1.6910000000000001,"15min":3.149},"last":0.85999999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.355},"min":{"1min":0.79000000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.7250000000000001,"5min":1.962,"15min":2.2410000000000001},"last":1.373},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2969999999999999},"min":{"1min":0.871,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.673,"5min":1.734,"15min":2.3849999999999998},"last":1.202}]},{"osd":5,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.248,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.581,"5min":1.964,"15min":2.577},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.323,"5min":1.373,"15min":1.371},"min":{"1min":0.90400000000000003,"5min":0.89800000000000002,"15min":0.81299999999999994},"max":{"1min":1.802,"5min":2.2770000000000001,"15min":2.9809999999999999},"last":1.345}]},{"osd":6,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3560000000000001,"15min":1.4079999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.73099999999999998},"max":{"1min":1.7809999999999999,"5min":2.008,"15min":3.2850000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4590000000000001},"min":{"1min":0.94699999999999995,"5min":0.72599999999999998,"15min":0.70199999999999996},"max":{"1min":1.6950000000000001,"5min":2.032,"15min":3.5859999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.4219999999999999,"15min":1.454},"min":{"1min":0.91400000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.97,"5min":2.5880000000000001,"15min":2.8439999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4419999999999999,"15min":1.5089999999999999},"min":{"1min":0.83499999999999996,"5min":0.82399999999999995,"15min":0.74099999999999999},"max":{"1min":1.891,"5min":2.1400000000000001,"15min":3.4350000000000001},"last":1.268}]}]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1327104,"data_stored":576120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81499999999999995,"15min":0.81399999999999995},"min":{"1min":0.42499999999999999,"5min":0.40999999999999998,"15min":0.29099999999999998},"max":{"1min":1.266,"5min":1.4750000000000001,"15min":1.7290000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.98899999999999999,"15min":0.96899999999999997},"min":{"1min":0.65700000000000003,"5min":0.433,"15min":0.376},"max":{"1min":1.7569999999999999,"5min":1.778,"15min":1.778},"last":0.94499999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86099999999999999,"15min":0.84399999999999997},"min":{"1min":0.73499999999999999,"5min":0.42999999999999999,"15min":0.34699999999999998},"max":{"1min":1.1100000000000001,"5min":1.837,"15min":1.837},"last":1.075},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0249999999999999,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.50600000000000001,"15min":0.42399999999999999},"max":{"1min":1.4359999999999999,"5min":1.583,"15min":2.0499999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.143,"15min":1.165},"min":{"1min":0.497,"5min":0.497,"15min":0.35799999999999998},"max":{"1min":1.5900000000000001,"5min":1.8320000000000001,"15min":1.9670000000000001},"last":1.331},{"interface":"front","average":{"1min":1.248,"5min":1.212,"15min":1.1919999999999999},"min":{"1min":0.66200000000000003,"5min":0.49299999999999999,"15min":0.46800000000000003},"max":{"1min":1.849,"5min":1.849,"15min":2.407},"last":1.264}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.298,"15min":1.367},"min":{"1min":0.94199999999999995,"5min":0.90000000000000002,"15min":0.70799999999999996},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":2.2589999999999999},"last":1.383},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.345,"15min":1.4279999999999999},"min":{"1min":1.069,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":3.3740000000000001},"last":1.679}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3049999999999999,"15min":1.3560000000000001},"min":{"1min":0.98299999999999998,"5min":0.83699999999999997,"15min":0.73499999999999999},"max":{"1min":2.101,"5min":2.101,"15min":2.9300000000000002},"last":1.552},{"interface":"front","average":{"1min":1.331,"5min":1.262,"15min":1.262},"min":{"1min":0.86599999999999999,"5min":0.77000000000000002,"15min":0.65600000000000003},"max":{"1min":1.794,"5min":1.8520000000000001,"15min":2.3799999999999999},"last":1.512}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.353,"15min":1.3540000000000001},"min":{"1min":0.89200000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.855,"5min":1.954,"15min":2.335},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.413,"15min":1.4370000000000001},"min":{"1min":1.0740000000000001,"5min":0.90600000000000003,"15min":0.70999999999999996},"max":{"1min":1.873,"5min":1.931,"15min":2.2389999999999999},"last":1.603}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4319999999999999,"15min":1.488},"min":{"1min":1.153,"5min":0.89700000000000002,"15min":0.76100000000000001},"max":{"1min":1.917,"5min":2.0019999999999998,"15min":3.0230000000000001},"last":1.752},{"interface":"front","average":{"1min":1.538,"5min":1.4750000000000001,"15min":1.55},"min":{"1min":0.91600000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.962,"5min":2.1659999999999999,"15min":3.1600000000000001},"last":1.798}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:50:01.284 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:50:01.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.493+0000 7fbcf3e00640 1 Processor -- start 2024-01-24T17:50:01.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.493+0000 7fbcf3e00640 1 -- start start 2024-01-24T17:50:01.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.494+0000 7fbcf3e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fbcf4070540 0x7fbcf415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.494+0000 7fbcf3e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbcf40b3e00 0x7fbcf40c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40c8f70 0x7fbcf415e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf3e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf41551b0 con 0x7fbcf40b3e00 2024-01-24T17:50:01.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf3e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf415e410 con 0x7fbcf4070540 2024-01-24T17:50:01.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf3e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf415b1b0 con 0x7fbcf40c8f70 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf2000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbcf40b3e00 0x7fbcf40c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf2000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fbcf40b3e00 0x7fbcf40c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48626/0 (socket says 172.21.15.39:48626) 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf2000640 1 -- 172.21.15.39:0/2486047199 learned_addr learned my addr 172.21.15.39:0/2486047199 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf3400640 1 --2- 172.21.15.39:0/2486047199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40c8f70 0x7fbcf415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf2a00640 1 --2- 172.21.15.39:0/2486047199 >> v2:172.21.15.100:3300/0 conn(0x7fbcf4070540 0x7fbcf415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf2000640 1 -- 172.21.15.39:0/2486047199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40c8f70 msgr2=0x7fbcf415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf2000640 1 --2- 172.21.15.39:0/2486047199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40c8f70 0x7fbcf415e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.495+0000 7fbcf2000640 1 -- 172.21.15.39:0/2486047199 >> v2:172.21.15.100:3300/0 conn(0x7fbcf4070540 msgr2=0x7fbcf415ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.496+0000 7fbcf2000640 1 --2- 172.21.15.39:0/2486047199 >> v2:172.21.15.100:3300/0 conn(0x7fbcf4070540 0x7fbcf415ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.496+0000 7fbcf2000640 1 -- 172.21.15.39:0/2486047199 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf415b480 con 0x7fbcf40b3e00 2024-01-24T17:50:01.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.496+0000 7fbcf2a00640 1 --2- 172.21.15.39:0/2486047199 >> v2:172.21.15.100:3300/0 conn(0x7fbcf4070540 0x7fbcf415ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:01.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.497+0000 7fbcf2000640 1 --2- 172.21.15.39:0/2486047199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40b3e00 0x7fbcf40c8a30 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009870 tx=0x7fbcdc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138079a1616122d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.497+0000 7fbcf1600640 1 -- 172.21.15.39:0/2486047199 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc031640 con 0x7fbcf40b3e00 2024-01-24T17:50:01.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.497+0000 7fbcf1600640 1 -- 172.21.15.39:0/2486047199 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcdc0317a0 con 0x7fbcf40b3e00 2024-01-24T17:50:01.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.497+0000 7fbcf1600640 1 -- 172.21.15.39:0/2486047199 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc031b30 con 0x7fbcf40b3e00 2024-01-24T17:50:01.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 -- 172.21.15.39:0/2486047199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40b3e00 msgr2=0x7fbcf40c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/2486047199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40b3e00 0x7fbcf40c8a30 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009870 tx=0x7fbcdc02b8f0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 -- 172.21.15.39:0/2486047199 shutdown_connections 2024-01-24T17:50:01.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/2486047199 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40c8f70 0x7fbcf415e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/2486047199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40b3e00 0x7fbcf40c8a30 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/2486047199 >> v2:172.21.15.100:3300/0 conn(0x7fbcf4070540 0x7fbcf415ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 -- 172.21.15.39:0/2486047199 >> 172.21.15.39:0/2486047199 conn(0x7fbcf406a340 msgr2=0x7fbcf40cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:01.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 -- 172.21.15.39:0/2486047199 shutdown_connections 2024-01-24T17:50:01.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.498+0000 7fbcf3e00640 1 -- 172.21.15.39:0/2486047199 wait complete. 2024-01-24T17:50:01.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.500+0000 7fbcf3e00640 1 Processor -- start 2024-01-24T17:50:01.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.500+0000 7fbcf3e00640 1 -- start start 2024-01-24T17:50:01.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbcf4070540 0x7fbcf414c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40b3e00 0x7fbcf40b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40c8f70 0x7fbcf40afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf40c32c0 con 0x7fbcf40c8f70 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf40c2ef0 con 0x7fbcf4070540 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf414cd30 con 0x7fbcf40b3e00 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40c8f70 0x7fbcf40afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40c8f70 0x7fbcf40afd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48634/0 (socket says 172.21.15.39:48634) 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.501+0000 7fbcf3400640 1 -- 172.21.15.39:0/3179985937 learned_addr learned my addr 172.21.15.39:0/3179985937 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.502+0000 7fbcf2000640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40b3e00 0x7fbcf40b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.502+0000 7fbcf2a00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbcf4070540 0x7fbcf414c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.502+0000 7fbcf3400640 1 -- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40b3e00 msgr2=0x7fbcf40b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.502+0000 7fbcf3400640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40b3e00 0x7fbcf40b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.502+0000 7fbcf3400640 1 -- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbcf4070540 msgr2=0x7fbcf414c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.502+0000 7fbcf3400640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbcf4070540 0x7fbcf414c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.502+0000 7fbcf3400640 1 -- 172.21.15.39:0/3179985937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc0094d0 con 0x7fbcf40c8f70 2024-01-24T17:50:01.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.502+0000 7fbcf3400640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40c8f70 0x7fbcf40afd10 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fbcec007e30 tx=0x7fbcec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.503+0000 7fbceaa00640 1 -- 172.21.15.39:0/3179985937 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec014620 con 0x7fbcf40c8f70 2024-01-24T17:50:01.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.503+0000 7fbceaa00640 1 -- 172.21.15.39:0/3179985937 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec00ee90 con 0x7fbcf40c8f70 2024-01-24T17:50:01.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.503+0000 7fbceaa00640 1 -- 172.21.15.39:0/3179985937 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0149c0 con 0x7fbcf40c8f70 2024-01-24T17:50:01.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.503+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf40c34d0 con 0x7fbcf40c8f70 2024-01-24T17:50:01.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.503+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf40c3a90 con 0x7fbcf40c8f70 2024-01-24T17:50:01.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.504+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf404dff0 con 0x7fbcf40c8f70 2024-01-24T17:50:01.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.505+0000 7fbceaa00640 1 -- 172.21.15.39:0/3179985937 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fbcec014b20 con 0x7fbcf40c8f70 2024-01-24T17:50:01.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.506+0000 7fbceaa00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbccc06a430 0x7fbccc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.506+0000 7fbceaa00640 1 -- 172.21.15.39:0/3179985937 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fbcec0890d0 con 0x7fbcf40c8f70 2024-01-24T17:50:01.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.506+0000 7fbcf2a00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbccc06a430 0x7fbccc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.508+0000 7fbcf2a00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbccc06a430 0x7fbccc06c8f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fbce4006fd0 tx=0x7fbce4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.510+0000 7fbceaa00640 1 -- 172.21.15.39:0/3179985937 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fbcec054850 con 0x7fbcf40c8f70 2024-01-24T17:50:01.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.701+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbcf415ba40 con 0x7fbccc06a430 2024-01-24T17:50:01.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.707+0000 7fbceaa00640 1 -- 172.21.15.39:0/3179985937 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48729 (secure 0 0 0) 0x7fbce400c050 con 0x7fbccc06a430 2024-01-24T17:50:01.708 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:50:01.709 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:50:01.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbccc06a430 msgr2=0x7fbccc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbccc06a430 0x7fbccc06c8f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fbce4006fd0 tx=0x7fbce4008040 comp rx=0 tx=0).stop 2024-01-24T17:50:01.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40c8f70 msgr2=0x7fbcf40afd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40c8f70 0x7fbcf40afd10 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fbcec007e30 tx=0x7fbcec00e4b0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 shutdown_connections 2024-01-24T17:50:01.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fbccc06a430 0x7fbccc06c8f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcf40c8f70 0x7fbcf40afd10 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fbcf40b3e00 0x7fbcf40b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 --2- 172.21.15.39:0/3179985937 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbcf4070540 0x7fbcf414c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 >> 172.21.15.39:0/3179985937 conn(0x7fbcf406a340 msgr2=0x7fbcf40af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:01.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 shutdown_connections 2024-01-24T17:50:01.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.710+0000 7fbcf3e00640 1 -- 172.21.15.39:0/3179985937 wait complete. 2024-01-24T17:50:01.731 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-01-24T17:50:00.393912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260532,"kb_used_data":11604,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754714188,"statfs":{"total":773094113280,"available":772827328512,"internally_reserved":0,"allocated":11882496,"data_stored":5820749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27738,"num_objects":-42,"num_object_clones":0,"num_object_copies":-84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-42,"num_whiteouts":0,"num_read":-95,"num_read_kb":-89,"num_write":-167,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002067"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":243,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-01-24T17:49:55.821460+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:49:55.821460+0000","last_peered":"2024-01-24T17:49:55.821460+0000","last_clean":"2024-01-24T17:49:55.821460+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:49:55.821460+0000","last_undegraded":"2024-01-24T17:49:55.821460+0000","last_fullsized":"2024-01-24T17:49:55.821460+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1331200,"data_stored":576702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0720000000000001,"15min":1.0449999999999999},"min":{"1min":0.73299999999999998,"5min":0.64900000000000002,"15min":0.61899999999999999},"max":{"1min":1.653,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.1359999999999999,"15min":1.0700000000000001},"min":{"1min":0.63400000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.411,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.753}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.68200000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.472,"5min":1.6479999999999999,"15min":2.04},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2070000000000001,"15min":1.196},"min":{"1min":0.75700000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.252}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1779999999999999,"15min":1.167},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.59099999999999997},"max":{"1min":1.484,"5min":1.992,"15min":2.1400000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.109,"5min":1.1910000000000001,"15min":1.2030000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.577,"5min":2.056,"15min":2.484},"last":1.0609999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2350000000000001,"15min":1.244},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.843,"5min":2.0179999999999998,"15min":2.2930000000000001},"last":1.232},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.3100000000000001,"15min":1.3049999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5740000000000001,"5min":1.9059999999999999,"15min":2.6269999999999998},"last":1.5640000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2729999999999999,"15min":1.3009999999999999},"min":{"1min":0.72999999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5029999999999999,"5min":2.2549999999999999,"15min":2.6179999999999999},"last":1.524},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3879999999999999,"15min":1.4179999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":2.6459999999999999},"last":1.4710000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.3320000000000001},"min":{"1min":0.58799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.885,"5min":1.9810000000000001,"15min":2.8460000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3740000000000001,"15min":1.4079999999999999},"min":{"1min":0.68300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6359999999999999,"5min":2.2480000000000002,"15min":2.3719999999999999},"last":1.6279999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.47,"15min":1.5069999999999999},"min":{"1min":0.83399999999999996,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.7190000000000001,"5min":2.4129999999999998,"15min":2.6960000000000002},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4219999999999999,"15min":1.4630000000000001},"min":{"1min":0.66600000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.946,"5min":2.0790000000000002,"15min":2.4830000000000001},"last":1.1479999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574953,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1339392,"data_stored":582411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0740000000000001,"15min":1.042},"min":{"1min":0.69699999999999995,"5min":0.69599999999999995,"15min":0.61199999999999999},"max":{"1min":1.4390000000000001,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0820000000000001,"15min":1.044},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.66300000000000003},"max":{"1min":1.5760000000000001,"5min":1.744,"15min":1.744},"last":1.0129999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.169,"15min":1.056},"min":{"1min":0.88500000000000001,"5min":0.71199999999999997,"15min":0.64200000000000002},"max":{"1min":2.9790000000000001,"5min":2.9790000000000001,"15min":2.9790000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.1379999999999999,"15min":1.1279999999999999},"min":{"1min":0.71599999999999997,"5min":0.67800000000000005,"15min":0.56399999999999995},"max":{"1min":1.425,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":1.0680000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.1970000000000001,"15min":1.155},"min":{"1min":0.85499999999999998,"5min":0.71899999999999997,"15min":0.63600000000000001},"max":{"1min":1.675,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.22,"15min":1.1850000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.56599999999999995},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.2050000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2709999999999999,"15min":1.258},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73399999999999999},"max":{"1min":3.21,"5min":3.21,"15min":3.21},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.234,"15min":1.2430000000000001},"min":{"1min":0.91100000000000003,"5min":0.84999999999999998,"15min":0.69799999999999995},"max":{"1min":1.554,"5min":2.0489999999999999,"15min":2.46},"last":0.93799999999999994}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3080000000000001,"15min":1.264},"min":{"1min":0.59199999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.3240000000000001,"15min":1.3089999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.44600000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.4339999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3380000000000001,"15min":1.3120000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.46800000000000003},"max":{"1min":3.3759999999999999,"5min":3.3759999999999999,"15min":3.3759999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.462,"15min":1.4419999999999999},"min":{"1min":0.67100000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.4849999999999999},"last":1.399}]},{"osd":7,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.361,"15min":1.363},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.51400000000000001},"max":{"1min":1.786,"5min":1.921,"15min":2.2050000000000001},"last":1.345},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4710000000000001,"15min":1.464},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.52300000000000002},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.6020000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574955,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1933312,"data_stored":1172779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0680000000000001,"15min":1.083},"min":{"1min":0.72099999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5389999999999999,"5min":3.609,"15min":3.609},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.0680000000000001,"15min":1.0429999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.391,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.629}]},{"osd":1,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.119,"15min":1.089},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.622},"max":{"1min":1.5249999999999999,"5min":3.5470000000000002,"15min":3.5470000000000002},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.153,"15min":1.22},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.544,"5min":1.859,"15min":2.1579999999999999},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.224,"15min":1.226},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.59199999999999997},"max":{"1min":1.6399999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3500000000000001,"15min":1.319},"min":{"1min":0.96499999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.663,"5min":8.1509999999999998,"15min":8.1509999999999998},"last":0.93600000000000005}]},{"osd":3,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2569999999999999,"15min":1.3049999999999999},"min":{"1min":0.99099999999999999,"5min":0.76400000000000001,"15min":0.70799999999999996},"max":{"1min":1.466,"5min":2.7559999999999998,"15min":3.036},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.28,"15min":1.3109999999999999},"min":{"1min":0.70499999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.474,"5min":4.1230000000000002,"15min":4.1230000000000002},"last":1.3420000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.355,"15min":1.417},"min":{"1min":0.68600000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6259999999999999,"5min":1.9690000000000001,"15min":2.069},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.254,"15min":1.319},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.77,"5min":2.1139999999999999,"15min":3.2810000000000001},"last":1.6220000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.304,"15min":1.3360000000000001},"min":{"1min":0.871,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.754,"5min":2.028,"15min":2.504},"last":1.9419999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.405,"15min":1.484},"min":{"1min":0.63800000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.575,"5min":2.1059999999999999,"15min":2.2290000000000001},"last":1.881}]},{"osd":7,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3280000000000001,"15min":1.4319999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.6759999999999999,"5min":2.2010000000000001,"15min":3.161},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.5129999999999999,"15min":1.5860000000000001},"min":{"1min":1.042,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8420000000000001,"5min":2.262,"15min":3.2410000000000001},"last":1.8169999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574956,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1331200,"data_stored":576702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.0589999999999999},"min":{"1min":0.73299999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3540000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.099,"15min":1.097},"min":{"1min":0.874,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6359999999999999,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.86799999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1739999999999999,"15min":1.097},"min":{"1min":0.78900000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5980000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1679999999999999,"15min":1.196},"min":{"1min":0.73099999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.698,"5min":1.712,"15min":1.849},"last":1.054}]},{"osd":2,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2430000000000001,"15min":1.2210000000000001},"min":{"1min":0.97299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.788,"5min":1.9850000000000001,"15min":2.105},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3080000000000001,"15min":1.3200000000000001},"min":{"1min":0.99399999999999999,"5min":0.69599999999999995,"15min":0.64500000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.986},"last":1.181}]},{"osd":3,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1819999999999999,"15min":1.1870000000000001},"min":{"1min":0.76800000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":1.637,"15min":1.7769999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.296,"15min":1.331},"min":{"1min":0.998,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.9409999999999998},"last":1.4159999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.3169999999999999,"15min":1.3500000000000001},"min":{"1min":1.044,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0979999999999999},"last":1.758},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":0.91700000000000004,"5min":0.65600000000000003,"15min":0.55200000000000005},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.9039999999999999},"last":1.9419999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.351,"15min":1.429},"min":{"1min":0.95599999999999996,"5min":0.67800000000000005,"15min":0.64400000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":3.0840000000000001},"last":1.901},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4550000000000001,"15min":1.468},"min":{"1min":1.1499999999999999,"5min":0.67700000000000005,"15min":0.66500000000000004},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.8759999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":1.0129999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.4910000000000001,"15min":1.5740000000000001},"min":{"1min":0.98399999999999999,"5min":0.79300000000000004,"15min":0.60399999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":4.3250000000000002},"last":1.6319999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1339392,"data_stored":582411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.86599999999999999},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.4590000000000001,"5min":1.5580000000000001,"15min":5.1929999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.88800000000000001,"15min":0.90100000000000002},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.4279999999999999,"5min":1.6739999999999999,"15min":8.6989999999999998},"last":1.1990000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.91700000000000004},"min":{"1min":0.46600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.4099999999999999,"5min":1.5229999999999999,"15min":1.8720000000000001},"last":1.282},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0389999999999999,"15min":1.0580000000000001},"min":{"1min":0.69499999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.679,"5min":1.679,"15min":3.7240000000000002},"last":0.82799999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0760000000000001,"15min":1.0980000000000001},"min":{"1min":0.499,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.581,"5min":1.7170000000000001,"15min":3.4630000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1279999999999999,"15min":1.155},"min":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":4.2919999999999998},"last":1.141}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.208,"15min":1.2689999999999999},"min":{"1min":0.76700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6060000000000001,"5min":2.0049999999999999,"15min":6.7469999999999999},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.2789999999999999,"15min":1.3280000000000001},"min":{"1min":0.873,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.6950000000000001,"5min":2.0390000000000001,"15min":3.4049999999999998},"last":1.448}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3129999999999999,"15min":1.395},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7050000000000001,"5min":2.1269999999999998,"15min":4.3789999999999996},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.363,"15min":1.4019999999999999},"min":{"1min":0.78300000000000003,"5min":0.746,"15min":0.746},"max":{"1min":2.1859999999999999,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":2.2669999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.411,"15min":1.4770000000000001},"min":{"1min":0.82199999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.331,"5min":2.331,"15min":9.5440000000000005},"last":1.851},{"interface":"front","average":{"1min":1.284,"5min":1.327,"15min":1.419},"min":{"1min":0.78300000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.659,"5min":2.109,"15min":4.8170000000000002},"last":1.3640000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.373,"15min":1.472},"min":{"1min":0.90000000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.736,"5min":2.3410000000000002,"15min":6.6970000000000001},"last":2.0350000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.431,"15min":1.4970000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.889,"5min":2.0990000000000002,"15min":6.6529999999999996},"last":1.6279999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1347584,"data_stored":580845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.78700000000000003,"15min":0.79300000000000004},"min":{"1min":0.42099999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.6180000000000001,"5min":1.667,"15min":1.667},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90000000000000002,"15min":0.92600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.472,"5min":1.7070000000000001,"15min":1.7989999999999999},"last":0.96499999999999997}]},{"osd":1,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.018,"15min":1.04},"min":{"1min":0.52100000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.026},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1080000000000001,"15min":1.1719999999999999},"min":{"1min":0.56000000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5960000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.242}]},{"osd":3,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95799999999999996,"15min":0.98099999999999998},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.7130000000000001,"15min":2.2120000000000002},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.1459999999999999,"15min":1.1919999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.996},"last":1.6919999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.244,"15min":1.2969999999999999},"min":{"1min":0.79500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.748,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.8939999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.256,"15min":1.349},"min":{"1min":0.71899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7250000000000001,"5min":1.889,"15min":2.3839999999999999},"last":1.677}]},{"osd":5,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.371},"min":{"1min":0.99299999999999999,"5min":0.81899999999999995,"15min":0.73899999999999999},"max":{"1min":1.6879999999999999,"5min":1.851,"15min":2.3140000000000001},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.2230000000000001,"15min":1.2889999999999999},"min":{"1min":0.95599999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.804,"5min":1.889,"15min":2.0720000000000001},"last":1.365}]},{"osd":6,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.294,"15min":1.3600000000000001},"min":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.623},"max":{"1min":2.056,"5min":2.056,"15min":2.2679999999999998},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.335,"15min":1.4259999999999999},"min":{"1min":0.92100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":3.7410000000000001},"last":2.0019999999999998}]},{"osd":7,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.383,"15min":1.508},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6879999999999999,"5min":2.1499999999999999,"15min":3.343},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.375,"15min":1.458},"min":{"1min":0.76100000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.1600000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.478}]}]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1933312,"data_stored":1172779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.85599999999999998,"15min":0.86099999999999999},"min":{"1min":0.42999999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.54,"5min":1.722,"15min":2.0339999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.95999999999999996,"15min":0.95199999999999996},"min":{"1min":0.53700000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.5089999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.70999999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88100000000000001,"15min":0.94499999999999995},"min":{"1min":0.51300000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3580000000000001,"5min":1.3959999999999999,"15min":1.6379999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.069,"15min":1.0900000000000001},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.41799999999999998},"max":{"1min":1.556,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.177}]},{"osd":3,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.171,"15min":1.163},"min":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.47699999999999998},"max":{"1min":1.623,"5min":1.8440000000000001,"15min":1.9610000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.0760000000000001,"15min":1.1799999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.41499999999999998},"max":{"1min":1.446,"5min":1.6910000000000001,"15min":3.149},"last":0.85999999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.355},"min":{"1min":0.79000000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.7250000000000001,"5min":1.962,"15min":2.2410000000000001},"last":1.373},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2969999999999999},"min":{"1min":0.871,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.673,"5min":1.734,"15min":2.3849999999999998},"last":1.202}]},{"osd":5,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.248,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.581,"5min":1.964,"15min":2.577},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.323,"5min":1.373,"15min":1.371},"min":{"1min":0.90400000000000003,"5min":0.89800000000000002,"15min":0.81299999999999994},"max":{"1min":1.802,"5min":2.2770000000000001,"15min":2.9809999999999999},"last":1.345}]},{"osd":6,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3560000000000001,"15min":1.4079999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.73099999999999998},"max":{"1min":1.7809999999999999,"5min":2.008,"15min":3.2850000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4590000000000001},"min":{"1min":0.94699999999999995,"5min":0.72599999999999998,"15min":0.70199999999999996},"max":{"1min":1.6950000000000001,"5min":2.032,"15min":3.5859999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.4219999999999999,"15min":1.454},"min":{"1min":0.91400000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.97,"5min":2.5880000000000001,"15min":2.8439999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4419999999999999,"15min":1.5089999999999999},"min":{"1min":0.83499999999999996,"5min":0.82399999999999995,"15min":0.74099999999999999},"max":{"1min":1.891,"5min":2.1400000000000001,"15min":3.4350000000000001},"last":1.268}]}]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1327104,"data_stored":576120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81499999999999995,"15min":0.81399999999999995},"min":{"1min":0.42499999999999999,"5min":0.40999999999999998,"15min":0.29099999999999998},"max":{"1min":1.266,"5min":1.4750000000000001,"15min":1.7290000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.98899999999999999,"15min":0.96899999999999997},"min":{"1min":0.65700000000000003,"5min":0.433,"15min":0.376},"max":{"1min":1.7569999999999999,"5min":1.778,"15min":1.778},"last":0.94499999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86099999999999999,"15min":0.84399999999999997},"min":{"1min":0.73499999999999999,"5min":0.42999999999999999,"15min":0.34699999999999998},"max":{"1min":1.1100000000000001,"5min":1.837,"15min":1.837},"last":1.075},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0249999999999999,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.50600000000000001,"15min":0.42399999999999999},"max":{"1min":1.4359999999999999,"5min":1.583,"15min":2.0499999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.143,"15min":1.165},"min":{"1min":0.497,"5min":0.497,"15min":0.35799999999999998},"max":{"1min":1.5900000000000001,"5min":1.8320000000000001,"15min":1.9670000000000001},"last":1.331},{"interface":"front","average":{"1min":1.248,"5min":1.212,"15min":1.1919999999999999},"min":{"1min":0.66200000000000003,"5min":0.49299999999999999,"15min":0.46800000000000003},"max":{"1min":1.849,"5min":1.849,"15min":2.407},"last":1.264}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.298,"15min":1.367},"min":{"1min":0.94199999999999995,"5min":0.90000000000000002,"15min":0.70799999999999996},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":2.2589999999999999},"last":1.383},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.345,"15min":1.4279999999999999},"min":{"1min":1.069,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":3.3740000000000001},"last":1.679}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3049999999999999,"15min":1.3560000000000001},"min":{"1min":0.98299999999999998,"5min":0.83699999999999997,"15min":0.73499999999999999},"max":{"1min":2.101,"5min":2.101,"15min":2.9300000000000002},"last":1.552},{"interface":"front","average":{"1min":1.331,"5min":1.262,"15min":1.262},"min":{"1min":0.86599999999999999,"5min":0.77000000000000002,"15min":0.65600000000000003},"max":{"1min":1.794,"5min":1.8520000000000001,"15min":2.3799999999999999},"last":1.512}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.353,"15min":1.3540000000000001},"min":{"1min":0.89200000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.855,"5min":1.954,"15min":2.335},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.413,"15min":1.4370000000000001},"min":{"1min":1.0740000000000001,"5min":0.90600000000000003,"15min":0.70999999999999996},"max":{"1min":1.873,"5min":1.931,"15min":2.2389999999999999},"last":1.603}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4319999999999999,"15min":1.488},"min":{"1min":1.153,"5min":0.89700000000000002,"15min":0.76100000000000001},"max":{"1min":1.917,"5min":2.0019999999999998,"15min":3.0230000000000001},"last":1.752},{"interface":"front","average":{"1min":1.538,"5min":1.4750000000000001,"15min":1.55},"min":{"1min":0.91600000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.962,"5min":2.1659999999999999,"15min":3.1600000000000001},"last":1.798}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:50:01.732 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-24T17:50:01.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:50:01.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.946+0000 7f6072200640 1 Processor -- start 2024-01-24T17:50:01.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.946+0000 7f6072200640 1 -- start start 2024-01-24T17:50:01.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f6072200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f606c0ca700 0x7f606c15a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f6072200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f606c15a910 0x7f606c06bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f6072200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c1540f0 con 0x7f606c0ca320 2024-01-24T17:50:01.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f6072200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c159fd0 con 0x7f606c15a910 2024-01-24T17:50:01.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f6072200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c15da00 con 0x7f606c0ca700 2024-01-24T17:50:01.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f6071800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f606c15a910 0x7f606c06bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f6071800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f606c15a910 0x7f606c06bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48636/0 (socket says 172.21.15.39:48636) 2024-01-24T17:50:01.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f6071800640 1 -- 172.21.15.39:0/1913126415 learned_addr learned my addr 172.21.15.39:0/1913126415 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:01.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.948+0000 7f606be00640 1 --2- 172.21.15.39:0/1913126415 >> v2:172.21.15.100:3300/0 conn(0x7f606c0ca700 0x7f606c15a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.949+0000 7f6071800640 1 -- 172.21.15.39:0/1913126415 >> v2:172.21.15.100:3300/0 conn(0x7f606c0ca700 msgr2=0x7f606c15a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:01.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.949+0000 7f6071800640 1 --2- 172.21.15.39:0/1913126415 >> v2:172.21.15.100:3300/0 conn(0x7f606c0ca700 0x7f606c15a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.949+0000 7f6071800640 1 -- 172.21.15.39:0/1913126415 >> v1:172.21.15.100:6789/0 conn(0x7f606c0ca320 legacy=0x7f606c15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:01.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.949+0000 7f6071800640 1 -- 172.21.15.39:0/1913126415 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c15dce0 con 0x7f606c15a910 2024-01-24T17:50:01.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.950+0000 7f6071800640 1 --2- 172.21.15.39:0/1913126415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c06bda0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f606c070600 tx=0x7f606402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590d8f33ae3569a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.950+0000 7f606b400640 1 -- 172.21.15.39:0/1913126415 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606402c960 con 0x7f606c15a910 2024-01-24T17:50:01.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.950+0000 7f606b400640 1 -- 172.21.15.39:0/1913126415 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606402cac0 con 0x7f606c15a910 2024-01-24T17:50:01.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.950+0000 7f606b400640 1 -- 172.21.15.39:0/1913126415 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606402ce50 con 0x7f606c15a910 2024-01-24T17:50:01.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.951+0000 7f6072200640 1 -- 172.21.15.39:0/1913126415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 msgr2=0x7f606c06bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.951+0000 7f6072200640 1 --2- 172.21.15.39:0/1913126415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c06bda0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f606c070600 tx=0x7f606402bdc0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.951+0000 7f6072200640 1 -- 172.21.15.39:0/1913126415 shutdown_connections 2024-01-24T17:50:01.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.951+0000 7f6072200640 1 --2- 172.21.15.39:0/1913126415 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c06bda0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.951+0000 7f6072200640 1 --2- 172.21.15.39:0/1913126415 >> v2:172.21.15.100:3300/0 conn(0x7f606c0ca700 0x7f606c15a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.951+0000 7f6072200640 1 -- 172.21.15.39:0/1913126415 >> 172.21.15.39:0/1913126415 conn(0x7f606c06a340 msgr2=0x7f606c0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:01.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.951+0000 7f6072200640 1 -- 172.21.15.39:0/1913126415 shutdown_connections 2024-01-24T17:50:01.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.952+0000 7f6072200640 1 -- 172.21.15.39:0/1913126415 wait complete. 2024-01-24T17:50:01.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.953+0000 7f6072200640 1 Processor -- start 2024-01-24T17:50:01.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.953+0000 7f6072200640 1 -- start start 2024-01-24T17:50:01.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.954+0000 7f6072200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606c0ca320 0x7f606c14eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.954+0000 7f6072200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606c0ca700 0x7f606c156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.954+0000 7f6072200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.954+0000 7f6072200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c0c58c0 con 0x7f606c15a910 2024-01-24T17:50:01.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.954+0000 7f6072200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c14f1e0 con 0x7f606c0ca320 2024-01-24T17:50:01.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.954+0000 7f6072200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c157680 con 0x7f606c0ca700 2024-01-24T17:50:01.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6071800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6071800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48640/0 (socket says 172.21.15.39:48640) 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6071800640 1 -- 172.21.15.39:0/4183917960 learned_addr learned my addr 172.21.15.39:0/4183917960 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f606be00640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606c0ca700 0x7f606c156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6070e00640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606c0ca320 0x7f606c14eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6071800640 1 -- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606c0ca700 msgr2=0x7f606c156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6071800640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606c0ca700 0x7f606c156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6071800640 1 -- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606c0ca320 msgr2=0x7f606c14eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6071800640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606c0ca320 0x7f606c14eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.955+0000 7f6071800640 1 -- 172.21.15.39:0/4183917960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6064009b30 con 0x7f606c15a910 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.956+0000 7f6071800640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c158330 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f6064009f40 tx=0x7f6064004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.956+0000 7f6068c00640 1 -- 172.21.15.39:0/4183917960 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064009510 con 0x7f606c15a910 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.956+0000 7f6068c00640 1 -- 172.21.15.39:0/4183917960 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606403e450 con 0x7f606c15a910 2024-01-24T17:50:01.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.956+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c157960 con 0x7f606c15a910 2024-01-24T17:50:01.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.956+0000 7f6068c00640 1 -- 172.21.15.39:0/4183917960 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606403e7e0 con 0x7f606c15a910 2024-01-24T17:50:01.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.956+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c0c5550 con 0x7f606c15a910 2024-01-24T17:50:01.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.957+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c0c5a70 con 0x7f606c15a910 2024-01-24T17:50:01.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.958+0000 7f6068c00640 1 -- 172.21.15.39:0/4183917960 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f606403e940 con 0x7f606c15a910 2024-01-24T17:50:01.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.960+0000 7f6068c00640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f604406a460 0x7f604406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:01.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.960+0000 7f6068c00640 1 -- 172.21.15.39:0/4183917960 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f60640ab250 con 0x7f606c15a910 2024-01-24T17:50:01.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.960+0000 7f6070e00640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f604406a460 0x7f604406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:01.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.962+0000 7f6070e00640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f604406a460 0x7f604406c920 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f606c06c030 tx=0x7f6054009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:01.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:01.964+0000 7f6068c00640 1 -- 172.21.15.39:0/4183917960 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f606c0c5a70 con 0x7f606c15a910 2024-01-24T17:50:02.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.154+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f606c15dfc0 con 0x7f604406a460 2024-01-24T17:50:02.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.161+0000 7f6068c00640 1 -- 172.21.15.39:0/4183917960 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48729 (secure 0 0 0) 0x7f605400d4f0 con 0x7f604406a460 2024-01-24T17:50:02.162 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:50:02.163 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:50:02.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.163+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f604406a460 msgr2=0x7f604406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.163+0000 7f6072200640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f604406a460 0x7f604406c920 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f606c06c030 tx=0x7f6054009040 comp rx=0 tx=0).stop 2024-01-24T17:50:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.163+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 msgr2=0x7f606c158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.163+0000 7f6072200640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c158330 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f6064009f40 tx=0x7f6064004a40 comp rx=0 tx=0).stop 2024-01-24T17:50:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.164+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 shutdown_connections 2024-01-24T17:50:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.164+0000 7f6072200640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f604406a460 0x7f604406c920 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.164+0000 7f6072200640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f606c15a910 0x7f606c158330 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.164+0000 7f6072200640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f606c0ca700 0x7f606c156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.164+0000 7f6072200640 1 --2- 172.21.15.39:0/4183917960 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f606c0ca320 0x7f606c14eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.164+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 >> 172.21.15.39:0/4183917960 conn(0x7f606c06a340 msgr2=0x7f606c0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:02.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.164+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 shutdown_connections 2024-01-24T17:50:02.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.164+0000 7f6072200640 1 -- 172.21.15.39:0/4183917960 wait complete. 2024-01-24T17:50:02.183 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-01-24T17:50:00.393912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260532,"kb_used_data":11604,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754714188,"statfs":{"total":773094113280,"available":772827328512,"internally_reserved":0,"allocated":11882496,"data_stored":5820749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27738,"num_objects":-42,"num_object_clones":0,"num_object_copies":-84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-42,"num_whiteouts":0,"num_read":-95,"num_read_kb":-89,"num_write":-167,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002067"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":243,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-01-24T17:49:55.821460+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:49:55.821460+0000","last_peered":"2024-01-24T17:49:55.821460+0000","last_clean":"2024-01-24T17:49:55.821460+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:49:55.821460+0000","last_undegraded":"2024-01-24T17:49:55.821460+0000","last_fullsized":"2024-01-24T17:49:55.821460+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T22:58:32.902685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1331200,"data_stored":576702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0720000000000001,"15min":1.0449999999999999},"min":{"1min":0.73299999999999998,"5min":0.64900000000000002,"15min":0.61899999999999999},"max":{"1min":1.653,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.1359999999999999,"15min":1.0700000000000001},"min":{"1min":0.63400000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.411,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.753}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.68200000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.472,"5min":1.6479999999999999,"15min":2.04},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2070000000000001,"15min":1.196},"min":{"1min":0.75700000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.252}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1779999999999999,"15min":1.167},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.59099999999999997},"max":{"1min":1.484,"5min":1.992,"15min":2.1400000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.109,"5min":1.1910000000000001,"15min":1.2030000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.577,"5min":2.056,"15min":2.484},"last":1.0609999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2350000000000001,"15min":1.244},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.843,"5min":2.0179999999999998,"15min":2.2930000000000001},"last":1.232},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.3100000000000001,"15min":1.3049999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5740000000000001,"5min":1.9059999999999999,"15min":2.6269999999999998},"last":1.5640000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2729999999999999,"15min":1.3009999999999999},"min":{"1min":0.72999999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5029999999999999,"5min":2.2549999999999999,"15min":2.6179999999999999},"last":1.524},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3879999999999999,"15min":1.4179999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":2.6459999999999999},"last":1.4710000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.3320000000000001},"min":{"1min":0.58799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.885,"5min":1.9810000000000001,"15min":2.8460000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3740000000000001,"15min":1.4079999999999999},"min":{"1min":0.68300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6359999999999999,"5min":2.2480000000000002,"15min":2.3719999999999999},"last":1.6279999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.47,"15min":1.5069999999999999},"min":{"1min":0.83399999999999996,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.7190000000000001,"5min":2.4129999999999998,"15min":2.6960000000000002},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4219999999999999,"15min":1.4630000000000001},"min":{"1min":0.66600000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.946,"5min":2.0790000000000002,"15min":2.4830000000000001},"last":1.1479999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574953,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1339392,"data_stored":582411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0740000000000001,"15min":1.042},"min":{"1min":0.69699999999999995,"5min":0.69599999999999995,"15min":0.61199999999999999},"max":{"1min":1.4390000000000001,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0820000000000001,"15min":1.044},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.66300000000000003},"max":{"1min":1.5760000000000001,"5min":1.744,"15min":1.744},"last":1.0129999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.169,"15min":1.056},"min":{"1min":0.88500000000000001,"5min":0.71199999999999997,"15min":0.64200000000000002},"max":{"1min":2.9790000000000001,"5min":2.9790000000000001,"15min":2.9790000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.1379999999999999,"15min":1.1279999999999999},"min":{"1min":0.71599999999999997,"5min":0.67800000000000005,"15min":0.56399999999999995},"max":{"1min":1.425,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":1.0680000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.1970000000000001,"15min":1.155},"min":{"1min":0.85499999999999998,"5min":0.71899999999999997,"15min":0.63600000000000001},"max":{"1min":1.675,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.22,"15min":1.1850000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.56599999999999995},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.2050000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2709999999999999,"15min":1.258},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73399999999999999},"max":{"1min":3.21,"5min":3.21,"15min":3.21},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.234,"15min":1.2430000000000001},"min":{"1min":0.91100000000000003,"5min":0.84999999999999998,"15min":0.69799999999999995},"max":{"1min":1.554,"5min":2.0489999999999999,"15min":2.46},"last":0.93799999999999994}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3080000000000001,"15min":1.264},"min":{"1min":0.59199999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.3240000000000001,"15min":1.3089999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.44600000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.4339999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3380000000000001,"15min":1.3120000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.46800000000000003},"max":{"1min":3.3759999999999999,"5min":3.3759999999999999,"15min":3.3759999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.462,"15min":1.4419999999999999},"min":{"1min":0.67100000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.4849999999999999},"last":1.399}]},{"osd":7,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.361,"15min":1.363},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.51400000000000001},"max":{"1min":1.786,"5min":1.921,"15min":2.2050000000000001},"last":1.345},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4710000000000001,"15min":1.464},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.52300000000000002},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.6020000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574955,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1933312,"data_stored":1172779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0680000000000001,"15min":1.083},"min":{"1min":0.72099999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5389999999999999,"5min":3.609,"15min":3.609},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.0680000000000001,"15min":1.0429999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.391,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.629}]},{"osd":1,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.119,"15min":1.089},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.622},"max":{"1min":1.5249999999999999,"5min":3.5470000000000002,"15min":3.5470000000000002},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.153,"15min":1.22},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.544,"5min":1.859,"15min":2.1579999999999999},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.224,"15min":1.226},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.59199999999999997},"max":{"1min":1.6399999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3500000000000001,"15min":1.319},"min":{"1min":0.96499999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.663,"5min":8.1509999999999998,"15min":8.1509999999999998},"last":0.93600000000000005}]},{"osd":3,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2569999999999999,"15min":1.3049999999999999},"min":{"1min":0.99099999999999999,"5min":0.76400000000000001,"15min":0.70799999999999996},"max":{"1min":1.466,"5min":2.7559999999999998,"15min":3.036},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.28,"15min":1.3109999999999999},"min":{"1min":0.70499999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.474,"5min":4.1230000000000002,"15min":4.1230000000000002},"last":1.3420000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.355,"15min":1.417},"min":{"1min":0.68600000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6259999999999999,"5min":1.9690000000000001,"15min":2.069},"last":1.9850000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.254,"15min":1.319},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.77,"5min":2.1139999999999999,"15min":3.2810000000000001},"last":1.6220000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.304,"15min":1.3360000000000001},"min":{"1min":0.871,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.754,"5min":2.028,"15min":2.504},"last":1.9419999999999999},{"interface":"front","average":{"1min":1.331,"5min":1.405,"15min":1.484},"min":{"1min":0.63800000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.575,"5min":2.1059999999999999,"15min":2.2290000000000001},"last":1.881}]},{"osd":7,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3280000000000001,"15min":1.4319999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.6759999999999999,"5min":2.2010000000000001,"15min":3.161},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.5129999999999999,"15min":1.5860000000000001},"min":{"1min":1.042,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8420000000000001,"5min":2.262,"15min":3.2410000000000001},"last":1.8169999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574956,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1331200,"data_stored":576702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.0589999999999999},"min":{"1min":0.73299999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3540000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.099,"15min":1.097},"min":{"1min":0.874,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6359999999999999,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.86799999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1739999999999999,"15min":1.097},"min":{"1min":0.78900000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5980000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1679999999999999,"15min":1.196},"min":{"1min":0.73099999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.698,"5min":1.712,"15min":1.849},"last":1.054}]},{"osd":2,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2430000000000001,"15min":1.2210000000000001},"min":{"1min":0.97299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.788,"5min":1.9850000000000001,"15min":2.105},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.347,"5min":1.3080000000000001,"15min":1.3200000000000001},"min":{"1min":0.99399999999999999,"5min":0.69599999999999995,"15min":0.64500000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.986},"last":1.181}]},{"osd":3,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1819999999999999,"15min":1.1870000000000001},"min":{"1min":0.76800000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":1.637,"15min":1.7769999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.296,"15min":1.331},"min":{"1min":0.998,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.9409999999999998},"last":1.4159999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.3169999999999999,"15min":1.3500000000000001},"min":{"1min":1.044,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0979999999999999},"last":1.758},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":0.91700000000000004,"5min":0.65600000000000003,"15min":0.55200000000000005},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.9039999999999999},"last":1.9419999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.351,"15min":1.429},"min":{"1min":0.95599999999999996,"5min":0.67800000000000005,"15min":0.64400000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":3.0840000000000001},"last":1.901},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4550000000000001,"15min":1.468},"min":{"1min":1.1499999999999999,"5min":0.67700000000000005,"15min":0.66500000000000004},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.8759999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":1.0129999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.4910000000000001,"15min":1.5740000000000001},"min":{"1min":0.98399999999999999,"5min":0.79300000000000004,"15min":0.60399999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":4.3250000000000002},"last":1.6319999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1339392,"data_stored":582411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.86599999999999999},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.4590000000000001,"5min":1.5580000000000001,"15min":5.1929999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.88800000000000001,"15min":0.90100000000000002},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.4279999999999999,"5min":1.6739999999999999,"15min":8.6989999999999998},"last":1.1990000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.91700000000000004},"min":{"1min":0.46600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.4099999999999999,"5min":1.5229999999999999,"15min":1.8720000000000001},"last":1.282},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0389999999999999,"15min":1.0580000000000001},"min":{"1min":0.69499999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.679,"5min":1.679,"15min":3.7240000000000002},"last":0.82799999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0760000000000001,"15min":1.0980000000000001},"min":{"1min":0.499,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.581,"5min":1.7170000000000001,"15min":3.4630000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1279999999999999,"15min":1.155},"min":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":4.2919999999999998},"last":1.141}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.208,"15min":1.2689999999999999},"min":{"1min":0.76700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6060000000000001,"5min":2.0049999999999999,"15min":6.7469999999999999},"last":1.5549999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.2789999999999999,"15min":1.3280000000000001},"min":{"1min":0.873,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.6950000000000001,"5min":2.0390000000000001,"15min":3.4049999999999998},"last":1.448}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3129999999999999,"15min":1.395},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7050000000000001,"5min":2.1269999999999998,"15min":4.3789999999999996},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.363,"15min":1.4019999999999999},"min":{"1min":0.78300000000000003,"5min":0.746,"15min":0.746},"max":{"1min":2.1859999999999999,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":2.2669999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.411,"15min":1.4770000000000001},"min":{"1min":0.82199999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.331,"5min":2.331,"15min":9.5440000000000005},"last":1.851},{"interface":"front","average":{"1min":1.284,"5min":1.327,"15min":1.419},"min":{"1min":0.78300000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.659,"5min":2.109,"15min":4.8170000000000002},"last":1.3640000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.373,"15min":1.472},"min":{"1min":0.90000000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.736,"5min":2.3410000000000002,"15min":6.6970000000000001},"last":2.0350000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.431,"15min":1.4970000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.889,"5min":2.0990000000000002,"15min":6.6529999999999996},"last":1.6279999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1347584,"data_stored":580845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.78700000000000003,"15min":0.79300000000000004},"min":{"1min":0.42099999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.6180000000000001,"5min":1.667,"15min":1.667},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90000000000000002,"15min":0.92600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.472,"5min":1.7070000000000001,"15min":1.7989999999999999},"last":0.96499999999999997}]},{"osd":1,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.018,"15min":1.04},"min":{"1min":0.52100000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.026},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1080000000000001,"15min":1.1719999999999999},"min":{"1min":0.56000000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5960000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.242}]},{"osd":3,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95799999999999996,"15min":0.98099999999999998},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.7130000000000001,"15min":2.2120000000000002},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.1459999999999999,"15min":1.1919999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.996},"last":1.6919999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.244,"15min":1.2969999999999999},"min":{"1min":0.79500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.748,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.8939999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.256,"15min":1.349},"min":{"1min":0.71899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7250000000000001,"5min":1.889,"15min":2.3839999999999999},"last":1.677}]},{"osd":5,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.371},"min":{"1min":0.99299999999999999,"5min":0.81899999999999995,"15min":0.73899999999999999},"max":{"1min":1.6879999999999999,"5min":1.851,"15min":2.3140000000000001},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.2230000000000001,"15min":1.2889999999999999},"min":{"1min":0.95599999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.804,"5min":1.889,"15min":2.0720000000000001},"last":1.365}]},{"osd":6,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.294,"15min":1.3600000000000001},"min":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.623},"max":{"1min":2.056,"5min":2.056,"15min":2.2679999999999998},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.335,"15min":1.4259999999999999},"min":{"1min":0.92100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":3.7410000000000001},"last":2.0019999999999998}]},{"osd":7,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.383,"15min":1.508},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6879999999999999,"5min":2.1499999999999999,"15min":3.343},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.375,"15min":1.458},"min":{"1min":0.76100000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.1600000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.478}]}]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1933312,"data_stored":1172779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.85599999999999998,"15min":0.86099999999999999},"min":{"1min":0.42999999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.54,"5min":1.722,"15min":2.0339999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.95999999999999996,"15min":0.95199999999999996},"min":{"1min":0.53700000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.5089999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.70999999999999996}]},{"osd":2,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88100000000000001,"15min":0.94499999999999995},"min":{"1min":0.51300000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3580000000000001,"5min":1.3959999999999999,"15min":1.6379999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.069,"15min":1.0900000000000001},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.41799999999999998},"max":{"1min":1.556,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.177}]},{"osd":3,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.171,"15min":1.163},"min":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.47699999999999998},"max":{"1min":1.623,"5min":1.8440000000000001,"15min":1.9610000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.0760000000000001,"15min":1.1799999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.41499999999999998},"max":{"1min":1.446,"5min":1.6910000000000001,"15min":3.149},"last":0.85999999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.355},"min":{"1min":0.79000000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.7250000000000001,"5min":1.962,"15min":2.2410000000000001},"last":1.373},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2969999999999999},"min":{"1min":0.871,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.673,"5min":1.734,"15min":2.3849999999999998},"last":1.202}]},{"osd":5,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.248,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.581,"5min":1.964,"15min":2.577},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.323,"5min":1.373,"15min":1.371},"min":{"1min":0.90400000000000003,"5min":0.89800000000000002,"15min":0.81299999999999994},"max":{"1min":1.802,"5min":2.2770000000000001,"15min":2.9809999999999999},"last":1.345}]},{"osd":6,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3560000000000001,"15min":1.4079999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.73099999999999998},"max":{"1min":1.7809999999999999,"5min":2.008,"15min":3.2850000000000001},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4590000000000001},"min":{"1min":0.94699999999999995,"5min":0.72599999999999998,"15min":0.70199999999999996},"max":{"1min":1.6950000000000001,"5min":2.032,"15min":3.5859999999999999},"last":1.3180000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.4219999999999999,"15min":1.454},"min":{"1min":0.91400000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.97,"5min":2.5880000000000001,"15min":2.8439999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4419999999999999,"15min":1.5089999999999999},"min":{"1min":0.83499999999999996,"5min":0.82399999999999995,"15min":0.74099999999999999},"max":{"1min":1.891,"5min":2.1400000000000001,"15min":3.4350000000000001},"last":1.268}]}]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1327104,"data_stored":576120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81499999999999995,"15min":0.81399999999999995},"min":{"1min":0.42499999999999999,"5min":0.40999999999999998,"15min":0.29099999999999998},"max":{"1min":1.266,"5min":1.4750000000000001,"15min":1.7290000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.98899999999999999,"15min":0.96899999999999997},"min":{"1min":0.65700000000000003,"5min":0.433,"15min":0.376},"max":{"1min":1.7569999999999999,"5min":1.778,"15min":1.778},"last":0.94499999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86099999999999999,"15min":0.84399999999999997},"min":{"1min":0.73499999999999999,"5min":0.42999999999999999,"15min":0.34699999999999998},"max":{"1min":1.1100000000000001,"5min":1.837,"15min":1.837},"last":1.075},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0249999999999999,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.50600000000000001,"15min":0.42399999999999999},"max":{"1min":1.4359999999999999,"5min":1.583,"15min":2.0499999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.143,"15min":1.165},"min":{"1min":0.497,"5min":0.497,"15min":0.35799999999999998},"max":{"1min":1.5900000000000001,"5min":1.8320000000000001,"15min":1.9670000000000001},"last":1.331},{"interface":"front","average":{"1min":1.248,"5min":1.212,"15min":1.1919999999999999},"min":{"1min":0.66200000000000003,"5min":0.49299999999999999,"15min":0.46800000000000003},"max":{"1min":1.849,"5min":1.849,"15min":2.407},"last":1.264}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.298,"15min":1.367},"min":{"1min":0.94199999999999995,"5min":0.90000000000000002,"15min":0.70799999999999996},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":2.2589999999999999},"last":1.383},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.345,"15min":1.4279999999999999},"min":{"1min":1.069,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":3.3740000000000001},"last":1.679}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3049999999999999,"15min":1.3560000000000001},"min":{"1min":0.98299999999999998,"5min":0.83699999999999997,"15min":0.73499999999999999},"max":{"1min":2.101,"5min":2.101,"15min":2.9300000000000002},"last":1.552},{"interface":"front","average":{"1min":1.331,"5min":1.262,"15min":1.262},"min":{"1min":0.86599999999999999,"5min":0.77000000000000002,"15min":0.65600000000000003},"max":{"1min":1.794,"5min":1.8520000000000001,"15min":2.3799999999999999},"last":1.512}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.353,"15min":1.3540000000000001},"min":{"1min":0.89200000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.855,"5min":1.954,"15min":2.335},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.413,"15min":1.4370000000000001},"min":{"1min":1.0740000000000001,"5min":0.90600000000000003,"15min":0.70999999999999996},"max":{"1min":1.873,"5min":1.931,"15min":2.2389999999999999},"last":1.603}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4319999999999999,"15min":1.488},"min":{"1min":1.153,"5min":0.89700000000000002,"15min":0.76100000000000001},"max":{"1min":1.917,"5min":2.0019999999999998,"15min":3.0230000000000001},"last":1.752},{"interface":"front","average":{"1min":1.538,"5min":1.4750000000000001,"15min":1.55},"min":{"1min":0.91600000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.962,"5min":2.1659999999999999,"15min":3.1600000000000001},"last":1.798}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:50:02.185 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T17:50:02.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.402+0000 7fc1a0c00640 1 Processor -- start 2024-01-24T17:50:02.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.402+0000 7fc1a0c00640 1 -- start start 2024-01-24T17:50:02.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.404+0000 7fc1a0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 0x7fc19c0d12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:02.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.404+0000 7fc1a0c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0d17f0 con 0x7fc19c164b40 2024-01-24T17:50:02.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.404+0000 7fc1a0c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0b8820 con 0x7fc19c15cbe0 2024-01-24T17:50:02.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.404+0000 7fc1a0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0b5790 con 0x7fc19c0d0e90 2024-01-24T17:50:02.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.404+0000 7fc19a800640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc19c164b40 0x7fc19c166f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:60126/0 (socket says 172.21.15.39:60126) 2024-01-24T17:50:02.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.404+0000 7fc19a800640 1 -- 172.21.15.39:0/2013853723 learned_addr learned my addr 172.21.15.39:0/2013853723 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc193e00640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 0x7fc19c0d12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781304404 0 0) 0x7fc19c0d17f0 con 0x7fc19c164b40 2024-01-24T17:50:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc17c003520 con 0x7fc19c164b40 2024-01-24T17:50:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1189855520 0 0) 0x7fc17c003520 con 0x7fc19c164b40 2024-01-24T17:50:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 msgr2=0x7fc19c0d12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 0x7fc19c0d12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 >> v1:172.21.15.100:6789/0 conn(0x7fc19c15cbe0 legacy=0x7fc19c0cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0b5a50 con 0x7fc19c164b40 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fc1940019f0 con 0x7fc19c164b40 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 >> v1:172.21.15.39:6789/0 conn(0x7fc19c164b40 legacy=0x7fc19c166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc17c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 0x7fc17c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c004d00 0x7fc17c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0b5790 con 0x7fc17c0039a0 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c00ba90 con 0x7fc17c004d00 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.405+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c00bc50 con 0x7fc19c0d0e90 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc199e00640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc17c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc19a800640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c004d00 0x7fc17c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc193e00640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 0x7fc17c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:02.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc199e00640 1 -- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 msgr2=0x7fc17c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:02.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc199e00640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 0x7fc17c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc199e00640 1 -- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c004d00 msgr2=0x7fc17c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:02.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc199e00640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c004d00 0x7fc17c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc199e00640 1 -- 172.21.15.39:0/2013853723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c003520 con 0x7fc17c0039a0 2024-01-24T17:50:02.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc19a800640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c004d00 0x7fc17c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:02.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.406+0000 7fc193e00640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0d0e90 0x7fc17c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:02.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.408+0000 7fc199e00640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc17c003dc0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fc18c007fd0 tx=0x7fc17c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf340fcb0924d6b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:02.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.408+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c0052d0 con 0x7fc17c0039a0 2024-01-24T17:50:02.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.408+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c02e050 con 0x7fc17c0039a0 2024-01-24T17:50:02.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.408+0000 7fc199400640 1 -- 172.21.15.39:0/2013853723 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c003640 con 0x7fc17c0039a0 2024-01-24T17:50:02.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.409+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2013853723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 msgr2=0x7fc17c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.409+0000 7fc1a0c00640 1 --2- 172.21.15.39:0/2013853723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc17c003dc0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fc18c007fd0 tx=0x7fc17c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.409+0000 7fc19a800640 1 -- 172.21.15.39:0/2013853723 reap_dead start 2024-01-24T17:50:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.409+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2013853723 shutdown_connections 2024-01-24T17:50:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.409+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2013853723 >> 172.21.15.39:0/2013853723 conn(0x7fc19c09bc80 msgr2=0x7fc19c0d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.409+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2013853723 shutdown_connections 2024-01-24T17:50:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.409+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2013853723 wait complete. 2024-01-24T17:50:02.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.411+0000 7fc1a0c00640 1 Processor -- start 2024-01-24T17:50:02.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.411+0000 7fc1a0c00640 1 -- start start 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.412+0000 7fc1a0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc19c1f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.412+0000 7fc1a0c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0ca6e0 0x7fc19c0cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.412+0000 7fc1a0c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc19c1f5520 0x7fc19c15f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.412+0000 7fc1a0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c1593e0 con 0x7fc17c0039a0 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.412+0000 7fc1a0c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c158ab0 con 0x7fc19c1f5520 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.412+0000 7fc1a0c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c158d90 con 0x7fc19c0ca6e0 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc199e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc19c1f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc199e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc19c1f39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48664/0 (socket says 172.21.15.39:48664) 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc199e00640 1 -- 172.21.15.39:0/2001121780 learned_addr learned my addr 172.21.15.39:0/2001121780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc19a800640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc19c1f5520 0x7fc19c15f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc193e00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0ca6e0 0x7fc19c0cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc199e00640 1 -- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0ca6e0 msgr2=0x7fc19c0cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc199e00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0ca6e0 0x7fc19c0cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc199e00640 1 -- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc19c1f5520 msgr2=0x7fc19c15f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc199e00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc19c1f5520 0x7fc19c15f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc199e00640 1 -- 172.21.15.39:0/2001121780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c159070 con 0x7fc17c0039a0 2024-01-24T17:50:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.413+0000 7fc19a800640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc19c1f5520 0x7fc19c15f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.414+0000 7fc199e00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc19c1f39c0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fc18c02bbc0 tx=0x7fc18c001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.414+0000 7fc192000640 1 -- 172.21.15.39:0/2001121780 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c006ea0 con 0x7fc17c0039a0 2024-01-24T17:50:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.414+0000 7fc192000640 1 -- 172.21.15.39:0/2001121780 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c007a80 con 0x7fc17c0039a0 2024-01-24T17:50:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.414+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c0cb810 con 0x7fc17c0039a0 2024-01-24T17:50:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.414+0000 7fc192000640 1 -- 172.21.15.39:0/2001121780 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c007450 con 0x7fc17c0039a0 2024-01-24T17:50:02.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.414+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c1f5b80 con 0x7fc17c0039a0 2024-01-24T17:50:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.416+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c04dff0 con 0x7fc17c0039a0 2024-01-24T17:50:02.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.417+0000 7fc192000640 1 -- 172.21.15.39:0/2001121780 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc18c0075b0 con 0x7fc17c0039a0 2024-01-24T17:50:02.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.418+0000 7fc192000640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc17406a460 0x7fc17406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:02.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.418+0000 7fc192000640 1 -- 172.21.15.39:0/2001121780 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fc18c0aa1b0 con 0x7fc17c0039a0 2024-01-24T17:50:02.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.418+0000 7fc193e00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc17406a460 0x7fc17406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:02.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.419+0000 7fc193e00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc17406a460 0x7fc17406c920 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc19c15bef0 tx=0x7fc184008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:02.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.422+0000 7fc192000640 1 -- 172.21.15.39:0/2001121780 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc18c075930 con 0x7fc17c0039a0 2024-01-24T17:50:02.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.623+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc19c0d0c70 con 0x7fc17c0039a0 2024-01-24T17:50:02.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.625+0000 7fc192000640 1 -- 172.21.15.39:0/2001121780 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+13041 (secure 0 0 0) 0x7fc18c078f50 con 0x7fc17c0039a0 2024-01-24T17:50:02.626 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:50:02.626 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":109,"fsid":"b13477e8-9bd1-460b-97e1-2729dfdebd46","created":"2024-01-24T17:41:09.136495+0000","modified":"2024-01-24T17:49:57.722825+0000","last_up_change":"2024-01-24T17:41:21.628378+0000","last_in_change":"2024-01-24T17:41:12.654149+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T17:41:22.282091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c62a7326-0fe4-4a6b-93ec-6ac162fe8358","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":36720},{"type":"v1","addr":"172.21.15.39:6801","nonce":36720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":36720},{"type":"v1","addr":"172.21.15.39:6803","nonce":36720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":36720},{"type":"v1","addr":"172.21.15.39:6807","nonce":36720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":36720},{"type":"v1","addr":"172.21.15.39:6805","nonce":36720}]},"public_addr":"172.21.15.39:6801/36720","cluster_addr":"172.21.15.39:6803/36720","heartbeat_back_addr":"172.21.15.39:6807/36720","heartbeat_front_addr":"172.21.15.39:6805/36720","state":["exists","up"]},{"osd":1,"uuid":"25fca6a0-197d-4518-bdaf-9bf4eec56bc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":36723},{"type":"v1","addr":"172.21.15.39:6822","nonce":36723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":36723},{"type":"v1","addr":"172.21.15.39:6826","nonce":36723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":36723},{"type":"v1","addr":"172.21.15.39:6831","nonce":36723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":36723},{"type":"v1","addr":"172.21.15.39:6829","nonce":36723}]},"public_addr":"172.21.15.39:6822/36723","cluster_addr":"172.21.15.39:6826/36723","heartbeat_back_addr":"172.21.15.39:6831/36723","heartbeat_front_addr":"172.21.15.39:6829/36723","state":["exists","up"]},{"osd":2,"uuid":"cd1f3734-5bd2-4183-a0e5-1c5f519b42d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":36726},{"type":"v1","addr":"172.21.15.39:6809","nonce":36726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":36726},{"type":"v1","addr":"172.21.15.39:6811","nonce":36726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":36726},{"type":"v1","addr":"172.21.15.39:6815","nonce":36726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":36726},{"type":"v1","addr":"172.21.15.39:6813","nonce":36726}]},"public_addr":"172.21.15.39:6809/36726","cluster_addr":"172.21.15.39:6811/36726","heartbeat_back_addr":"172.21.15.39:6815/36726","heartbeat_front_addr":"172.21.15.39:6813/36726","state":["exists","up"]},{"osd":3,"uuid":"840d0d11-bf09-4c4a-aff4-23345809b0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":36725},{"type":"v1","addr":"172.21.15.39:6817","nonce":36725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":36725},{"type":"v1","addr":"172.21.15.39:6819","nonce":36725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6825","nonce":36725},{"type":"v1","addr":"172.21.15.39:6827","nonce":36725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6821","nonce":36725},{"type":"v1","addr":"172.21.15.39:6823","nonce":36725}]},"public_addr":"172.21.15.39:6817/36725","cluster_addr":"172.21.15.39:6819/36725","heartbeat_back_addr":"172.21.15.39:6827/36725","heartbeat_front_addr":"172.21.15.39:6823/36725","state":["exists","up"]},{"osd":4,"uuid":"98c2f848-b979-4d9f-87c7-9a510fa337a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":36832},{"type":"v1","addr":"172.21.15.100:6801","nonce":36832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":36832},{"type":"v1","addr":"172.21.15.100:6803","nonce":36832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":36832},{"type":"v1","addr":"172.21.15.100:6807","nonce":36832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":36832},{"type":"v1","addr":"172.21.15.100:6805","nonce":36832}]},"public_addr":"172.21.15.100:6801/36832","cluster_addr":"172.21.15.100:6803/36832","heartbeat_back_addr":"172.21.15.100:6807/36832","heartbeat_front_addr":"172.21.15.100:6805/36832","state":["exists","up"]},{"osd":5,"uuid":"f131d23e-1a25-48e6-8fc9-071732abd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":36844},{"type":"v1","addr":"172.21.15.100:6817","nonce":36844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":36844},{"type":"v1","addr":"172.21.15.100:6819","nonce":36844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":36844},{"type":"v1","addr":"172.21.15.100:6827","nonce":36844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":36844},{"type":"v1","addr":"172.21.15.100:6823","nonce":36844}]},"public_addr":"172.21.15.100:6817/36844","cluster_addr":"172.21.15.100:6819/36844","heartbeat_back_addr":"172.21.15.100:6827/36844","heartbeat_front_addr":"172.21.15.100:6823/36844","state":["exists","up"]},{"osd":6,"uuid":"3b0f4907-1db1-4d56-92b9-9dedceff5179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":36833},{"type":"v1","addr":"172.21.15.100:6809","nonce":36833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":36833},{"type":"v1","addr":"172.21.15.100:6811","nonce":36833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":36833},{"type":"v1","addr":"172.21.15.100:6815","nonce":36833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":36833},{"type":"v1","addr":"172.21.15.100:6813","nonce":36833}]},"public_addr":"172.21.15.100:6809/36833","cluster_addr":"172.21.15.100:6811/36833","heartbeat_back_addr":"172.21.15.100:6815/36833","heartbeat_front_addr":"172.21.15.100:6813/36833","state":["exists","up"]},{"osd":7,"uuid":"2ad6299c-d6c2-4a16-a09a-4d3fd3a3c563","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":36857},{"type":"v1","addr":"172.21.15.100:6822","nonce":36857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":36857},{"type":"v1","addr":"172.21.15.100:6826","nonce":36857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6830","nonce":36857},{"type":"v1","addr":"172.21.15.100:6831","nonce":36857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":36857},{"type":"v1","addr":"172.21.15.100:6829","nonce":36857}]},"public_addr":"172.21.15.100:6822/36857","cluster_addr":"172.21.15.100:6826/36857","heartbeat_back_addr":"172.21.15.100:6831/36857","heartbeat_front_addr":"172.21.15.100:6829/36857","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.132886+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.013733+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.039188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:18.002727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.966313+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.997246+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.926441+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T17:41:17.957868+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:6833/3903887009":"2024-01-25T17:49:54.687017+0000","172.21.15.39:6834/1608906778":"2024-01-25T17:49:54.687049+0000","172.21.15.39:6832/3903887009":"2024-01-25T17:49:54.687017+0000","172.21.15.39:6833/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.39:6832/872967797":"2024-01-25T17:48:58.450631+0000","172.21.15.100:6836/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.100:6834/3116066709":"2024-01-25T17:48:58.404809+0000","172.21.15.39:6835/3495904204":"2024-01-25T17:48:58.377999+0000","172.21.15.39:6835/1608906778":"2024-01-25T17:49:54.687049+0000","172.21.15.39:6834/3495904204":"2024-01-25T17:48:58.377999+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc17406a460 msgr2=0x7fc17406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc17406a460 0x7fc17406c920 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc19c15bef0 tx=0x7fc184008040 comp rx=0 tx=0).stop 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 msgr2=0x7fc19c1f39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc19c1f39c0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fc18c02bbc0 tx=0x7fc18c001170 comp rx=0 tx=0).stop 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 shutdown_connections 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc19c1f5520 0x7fc19c15f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc19c0ca6e0 0x7fc19c0cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc17406a460 0x7fc17406c920 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 --2- 172.21.15.39:0/2001121780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc17c0039a0 0x7fc19c1f39c0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:02.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 >> 172.21.15.39:0/2001121780 conn(0x7fc19c09bc80 msgr2=0x7fc19c0d1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:02.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 shutdown_connections 2024-01-24T17:50:02.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:02.628+0000 7fc1a0c00640 1 -- 172.21.15.39:0/2001121780 wait complete. 2024-01-24T17:50:03.650 INFO:tasks.ceph:Scrubbing osd.0 2024-01-24T17:50:03.650 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-24T17:50:03.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.865+0000 7f0259600640 1 Processor -- start 2024-01-24T17:50:03.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.866+0000 7f0259600640 1 -- start start 2024-01-24T17:50:03.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0259600640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f025406bdb0 0x7f02540c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0259600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 0x7f02540cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0259600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02540ccf40 con 0x7f025406c190 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0259600640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02540ca830 con 0x7f025406bdb0 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0259600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02540ca270 con 0x7f0254159830 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0253e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f025406bdb0 0x7f02540c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0258c00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f025406c190 0x7f02540b11c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58186/0 (socket says 172.21.15.39:58186) 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0258c00640 1 -- 172.21.15.39:0/3394362629 learned_addr learned my addr 172.21.15.39:0/3394362629 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.867+0000 7f0253e00640 1 --2- 172.21.15.39:0/3394362629 >> v2:172.21.15.39:3300/0 conn(0x7f025406bdb0 0x7f02540c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56802/0 (socket says 172.21.15.39:56802) 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.868+0000 7f024be00640 1 --2- 172.21.15.39:0/3394362629 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 0x7f02540cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.868+0000 7f0253e00640 1 -- 172.21.15.39:0/3394362629 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 msgr2=0x7f02540cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.868+0000 7f0253e00640 1 --2- 172.21.15.39:0/3394362629 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 0x7f02540cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.868+0000 7f0253e00640 1 -- 172.21.15.39:0/3394362629 >> v1:172.21.15.39:6789/0 conn(0x7f025406c190 legacy=0x7f02540b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.868+0000 7f0253e00640 1 -- 172.21.15.39:0/3394362629 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02540ca550 con 0x7f025406bdb0 2024-01-24T17:50:03.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.868+0000 7f024be00640 1 --2- 172.21.15.39:0/3394362629 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 0x7f02540cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.869+0000 7f0253e00640 1 --2- 172.21.15.39:0/3394362629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406bdb0 0x7f02540c9be0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f024400ba40 tx=0x7f024402e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72497b1ca33ee936 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.869+0000 7f0253400640 1 -- 172.21.15.39:0/3394362629 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024402ee30 con 0x7f025406bdb0 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.869+0000 7f0253400640 1 -- 172.21.15.39:0/3394362629 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0244036440 con 0x7f025406bdb0 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.869+0000 7f0253400640 1 -- 172.21.15.39:0/3394362629 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02440367d0 con 0x7f025406bdb0 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.870+0000 7f0259600640 1 -- 172.21.15.39:0/3394362629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406bdb0 msgr2=0x7f02540c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.870+0000 7f0259600640 1 --2- 172.21.15.39:0/3394362629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406bdb0 0x7f02540c9be0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f024400ba40 tx=0x7f024402e080 comp rx=0 tx=0).stop 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.870+0000 7f0259600640 1 -- 172.21.15.39:0/3394362629 shutdown_connections 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.870+0000 7f0259600640 1 --2- 172.21.15.39:0/3394362629 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 0x7f02540cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.870+0000 7f0259600640 1 --2- 172.21.15.39:0/3394362629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406bdb0 0x7f02540c9be0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:03.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.870+0000 7f0259600640 1 -- 172.21.15.39:0/3394362629 >> 172.21.15.39:0/3394362629 conn(0x7f025406a310 msgr2=0x7f02540cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:03.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.870+0000 7f0259600640 1 -- 172.21.15.39:0/3394362629 shutdown_connections 2024-01-24T17:50:03.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.870+0000 7f0259600640 1 -- 172.21.15.39:0/3394362629 wait complete. 2024-01-24T17:50:03.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.872+0000 7f0259600640 1 Processor -- start 2024-01-24T17:50:03.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.872+0000 7f0259600640 1 -- start start 2024-01-24T17:50:03.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f0259600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f025406bdb0 0x7f025414f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f0259600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406c190 0x7f02540aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f0259600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 0x7f02541f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f0259600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02541f5450 con 0x7f025406c190 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f0259600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02541f5730 con 0x7f025406bdb0 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f0259600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02541541e0 con 0x7f0254159830 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f024be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406c190 0x7f02540aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f024be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406c190 0x7f02540aee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56818/0 (socket says 172.21.15.39:56818) 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f024be00640 1 -- 172.21.15.39:0/1284721792 learned_addr learned my addr 172.21.15.39:0/1284721792 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f024be00640 1 -- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 msgr2=0x7f02541f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f024be00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f0254159830 0x7f02541f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f024be00640 1 -- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f025406bdb0 msgr2=0x7f025414f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f024be00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f025406bdb0 0x7f025414f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:03.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.873+0000 7f024be00640 1 -- 172.21.15.39:0/1284721792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024400b6a0 con 0x7f025406c190 2024-01-24T17:50:03.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.874+0000 7f024be00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406c190 0x7f02540aee00 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c820 tx=0x7f023c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:03.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.874+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c007a30 con 0x7f025406c190 2024-01-24T17:50:03.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.874+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023c007b90 con 0x7f025406c190 2024-01-24T17:50:03.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.874+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c00ce10 con 0x7f025406c190 2024-01-24T17:50:03.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.874+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025414fc60 con 0x7f025406c190 2024-01-24T17:50:03.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.874+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0254150200 con 0x7f025406c190 2024-01-24T17:50:03.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.875+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0254071160 con 0x7f025406c190 2024-01-24T17:50:03.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.876+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f023c0113f0 con 0x7f025406c190 2024-01-24T17:50:03.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.877+0000 7f0250c00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f022c06a480 0x7f022c06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:03.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.877+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f023c088850 con 0x7f025406c190 2024-01-24T17:50:03.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.877+0000 7f0250c00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f022c070560 0x7f022c072980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:03.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.877+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 --> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f022c073030 con 0x7f022c070560 2024-01-24T17:50:03.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.877+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f023c08ae30 con 0x7f025406c190 2024-01-24T17:50:03.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.877+0000 7f0258c00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f022c070560 0x7f022c072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:03.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.877+0000 7f0253e00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f022c06a480 0x7f022c06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:03.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.878+0000 7f0258c00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f022c070560 0x7f022c072980 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:03.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.879+0000 7f0253e00640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f022c06a480 0x7f022c06c940 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0244038000 tx=0x7f0244030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:03.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.879+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 <== osd.0 v2:172.21.15.39:6800/36720 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f022c073030 con 0x7f022c070560 2024-01-24T17:50:03.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.892+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 --> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0254070e80 con 0x7f022c070560 2024-01-24T17:50:03.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0250c00640 1 -- 172.21.15.39:0/1284721792 <== osd.0 v2:172.21.15.39:6800/36720 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f024c001c10 con 0x7f022c070560 2024-01-24T17:50:03.893 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:50:03.893 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-24T17:50:03.893 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:50:03.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f022c070560 msgr2=0x7f022c072980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:03.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.39:6800/36720,v1:172.21.15.39:6801/36720] conn(0x7f022c070560 0x7f022c072980 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f022c06a480 msgr2=0x7f022c06c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f022c06a480 0x7f022c06c940 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0244038000 tx=0x7f0244030000 comp rx=0 tx=0).stop 2024-01-24T17:50:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406c190 msgr2=0x7f02540aee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 --2- 172.21.15.39:0/1284721792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f025406c190 0x7f02540aee00 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c820 tx=0x7f023c00cce0 comp rx=0 tx=0).stop 2024-01-24T17:50:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0258c00640 1 -- 172.21.15.39:0/1284721792 reap_dead start 2024-01-24T17:50:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 shutdown_connections 2024-01-24T17:50:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 >> 172.21.15.39:0/1284721792 conn(0x7f025406a310 msgr2=0x7f0254157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:03.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 shutdown_connections 2024-01-24T17:50:03.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:03.893+0000 7f0259600640 1 -- 172.21.15.39:0/1284721792 wait complete. 2024-01-24T17:50:03.910 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-24T17:50:04.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.109+0000 7fd4e8400640 1 Processor -- start 2024-01-24T17:50:04.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.109+0000 7fd4e8400640 1 -- start start 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.110+0000 7fd4e8400640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd4e015cf10 0x7fd4e0070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.110+0000 7fd4e8400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd4e00b0720 0x7fd4e00cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.110+0000 7fd4e8400640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e00ccf90 con 0x7fd4e0070980 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.110+0000 7fd4e8400640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e00c9bd0 con 0x7fd4e00b0720 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.110+0000 7fd4e8400640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e00c9eb0 con 0x7fd4e015cf10 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e6400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd4e00b0720 0x7fd4e00cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e6400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd4e00b0720 0x7fd4e00cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56834/0 (socket says 172.21.15.39:56834) 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e5000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd4e0070980 0x7fd4e00b0010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58200/0 (socket says 172.21.15.39:58200) 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e6400640 1 -- 172.21.15.39:0/1104576443 learned_addr learned my addr 172.21.15.39:0/1104576443 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e5a00640 1 --2- 172.21.15.39:0/1104576443 >> v2:172.21.15.100:3300/0 conn(0x7fd4e015cf10 0x7fd4e0070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e6400640 1 -- 172.21.15.39:0/1104576443 >> v2:172.21.15.100:3300/0 conn(0x7fd4e015cf10 msgr2=0x7fd4e0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:04.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e6400640 1 --2- 172.21.15.39:0/1104576443 >> v2:172.21.15.100:3300/0 conn(0x7fd4e015cf10 0x7fd4e0070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e6400640 1 -- 172.21.15.39:0/1104576443 >> v1:172.21.15.39:6789/0 conn(0x7fd4e0070980 legacy=0x7fd4e00b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.111+0000 7fd4e6400640 1 -- 172.21.15.39:0/1104576443 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e00cc940 con 0x7fd4e00b0720 2024-01-24T17:50:04.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.112+0000 7fd4e6400640 1 --2- 172.21.15.39:0/1104576443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e00cd420 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd4e006e420 tx=0x7fd4d802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a758102c725fd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4dfe00640 1 -- 172.21.15.39:0/1104576443 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d802ca80 con 0x7fd4e00b0720 2024-01-24T17:50:04.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4dfe00640 1 -- 172.21.15.39:0/1104576443 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d802cbe0 con 0x7fd4e00b0720 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4dfe00640 1 -- 172.21.15.39:0/1104576443 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80343f0 con 0x7fd4e00b0720 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4e8400640 1 -- 172.21.15.39:0/1104576443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 msgr2=0x7fd4e00cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4e8400640 1 --2- 172.21.15.39:0/1104576443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e00cd420 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd4e006e420 tx=0x7fd4d802bee0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4e8400640 1 -- 172.21.15.39:0/1104576443 shutdown_connections 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4e8400640 1 --2- 172.21.15.39:0/1104576443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e00cd420 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4e8400640 1 --2- 172.21.15.39:0/1104576443 >> v2:172.21.15.100:3300/0 conn(0x7fd4e015cf10 0x7fd4e0070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4e8400640 1 -- 172.21.15.39:0/1104576443 >> 172.21.15.39:0/1104576443 conn(0x7fd4e006bf20 msgr2=0x7fd4e006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4e8400640 1 -- 172.21.15.39:0/1104576443 shutdown_connections 2024-01-24T17:50:04.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.113+0000 7fd4e8400640 1 -- 172.21.15.39:0/1104576443 wait complete. 2024-01-24T17:50:04.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.115+0000 7fd4e8400640 1 Processor -- start 2024-01-24T17:50:04.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.116+0000 7fd4e8400640 1 -- start start 2024-01-24T17:50:04.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e8400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4e0070980 0x7fd4e01f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e8400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e8400640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4e015cf10 0x7fd4e0145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e8400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01f5ce0 con 0x7fd4e00b0720 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e8400640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e014ba70 con 0x7fd4e015cf10 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e8400640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e0147d20 con 0x7fd4e0070980 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e015bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56842/0 (socket says 172.21.15.39:56842) 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5000640 1 -- 172.21.15.39:0/2747038685 learned_addr learned my addr 172.21.15.39:0/2747038685 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e6400640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4e015cf10 0x7fd4e0145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5a00640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4e0070980 0x7fd4e01f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5000640 1 -- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4e0070980 msgr2=0x7fd4e01f21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5000640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4e0070980 0x7fd4e01f21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5000640 1 -- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4e015cf10 msgr2=0x7fd4e0145720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5000640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4e015cf10 0x7fd4e0145720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.117+0000 7fd4e5000640 1 -- 172.21.15.39:0/2747038685 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d8009b30 con 0x7fd4e00b0720 2024-01-24T17:50:04.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.118+0000 7fd4e5000640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e015bc60 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800c4d0 tx=0x7fd4c800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.118+0000 7fd4dd600640 1 -- 172.21.15.39:0/2747038685 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8007480 con 0x7fd4e00b0720 2024-01-24T17:50:04.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.118+0000 7fd4dd600640 1 -- 172.21.15.39:0/2747038685 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c80075e0 con 0x7fd4e00b0720 2024-01-24T17:50:04.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.118+0000 7fd4dd600640 1 -- 172.21.15.39:0/2747038685 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c80049e0 con 0x7fd4e00b0720 2024-01-24T17:50:04.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.118+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e0148000 con 0x7fd4e00b0720 2024-01-24T17:50:04.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.118+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e01485c0 con 0x7fd4e00b0720 2024-01-24T17:50:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.119+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e00cbd30 con 0x7fd4e00b0720 2024-01-24T17:50:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.120+0000 7fd4dd600640 1 -- 172.21.15.39:0/2747038685 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd4c8004b40 con 0x7fd4e00b0720 2024-01-24T17:50:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.121+0000 7fd4dd600640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd4b806a430 0x7fd4b806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.121+0000 7fd4dd600640 1 -- 172.21.15.39:0/2747038685 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fd4c80877a0 con 0x7fd4e00b0720 2024-01-24T17:50:04.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.122+0000 7fd4e5a00640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd4b806a430 0x7fd4b806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.123+0000 7fd4e5a00640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd4b806a430 0x7fd4b806c8f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0006fd0 tx=0x7fd4d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.125+0000 7fd4dd600640 1 -- 172.21.15.39:0/2747038685 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fd4c8053070 con 0x7fd4e00b0720 2024-01-24T17:50:04.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.316+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4e00ccc20 con 0x7fd4b806a430 2024-01-24T17:50:04.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.317+0000 7fd4dd600640 1 -- 172.21.15.39:0/2747038685 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd4d000e4b0 con 0x7fd4b806a430 2024-01-24T17:50:04.319 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 0 to deep-scrub 2024-01-24T17:50:04.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd4b806a430 msgr2=0x7fd4b806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd4b806a430 0x7fd4b806c8f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0006fd0 tx=0x7fd4d0008040 comp rx=0 tx=0).stop 2024-01-24T17:50:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 msgr2=0x7fd4e015bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e015bc60 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd4c800c4d0 tx=0x7fd4c800c990 comp rx=0 tx=0).stop 2024-01-24T17:50:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 shutdown_connections 2024-01-24T17:50:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd4e015cf10 0x7fd4e0145720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd4b806a430 0x7fd4b806c8f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4e00b0720 0x7fd4e015bc60 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 --2- 172.21.15.39:0/2747038685 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd4e0070980 0x7fd4e01f21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 >> 172.21.15.39:0/2747038685 conn(0x7fd4e006bf20 msgr2=0x7fd4e00cb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 shutdown_connections 2024-01-24T17:50:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.320+0000 7fd4e8400640 1 -- 172.21.15.39:0/2747038685 wait complete. 2024-01-24T17:50:04.340 INFO:tasks.ceph:Scrubbing osd.1 2024-01-24T17:50:04.340 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-24T17:50:04.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.549+0000 7f9829600640 1 Processor -- start 2024-01-24T17:50:04.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.550+0000 7f9829600640 1 -- start start 2024-01-24T17:50:04.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.551+0000 7f9829600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f982406b6e0 0x7f982406bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.551+0000 7f9829600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f982406b180 0x7f9824160d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.551+0000 7f9829600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982406c000 con 0x7f982406acf0 2024-01-24T17:50:04.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.551+0000 7f9829600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98240b42d0 con 0x7f982406b6e0 2024-01-24T17:50:04.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.551+0000 7f9829600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98240b45b0 con 0x7f982406b180 2024-01-24T17:50:04.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.551+0000 7f9823400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f982406acf0 0x7f9824160630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58202/0 (socket says 172.21.15.39:58202) 2024-01-24T17:50:04.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.551+0000 7f9823400640 1 -- 172.21.15.39:0/3287767100 learned_addr learned my addr 172.21.15.39:0/3287767100 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:04.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9823e00640 1 --2- 172.21.15.39:0/3287767100 >> v2:172.21.15.100:3300/0 conn(0x7f982406b6e0 0x7f982406bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9828c00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f982406b180 0x7f9824160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647867645 0 0) 0x7f982406c000 con 0x7f982406acf0 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9804003520 con 0x7f982406acf0 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915240229 0 0) 0x7f9804003520 con 0x7f982406acf0 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f982406b180 msgr2=0x7f9824160d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f982406b180 0x7f9824160d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 >> v2:172.21.15.100:3300/0 conn(0x7f982406b6e0 msgr2=0x7f982406bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 --2- 172.21.15.39:0/3287767100 >> v2:172.21.15.100:3300/0 conn(0x7f982406b6e0 0x7f982406bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98240b4890 con 0x7f982406acf0 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f980c001670 con 0x7f982406acf0 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 >> v1:172.21.15.39:6789/0 conn(0x7f982406acf0 legacy=0x7f9824160630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f9804003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.552+0000 7f9822a00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98040047c0 0x7f9804004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9822a00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f982406b180 0x7f9804005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98240b42d0 con 0x7f9804003970 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98240b45b0 con 0x7f98040047c0 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980400b980 con 0x7f982406b180 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9823e00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f9804003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9828c00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f982406b180 0x7f9804005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9823e00640 1 -- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f982406b180 msgr2=0x7f9804005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9823e00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f982406b180 0x7f9804005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9823e00640 1 -- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98040047c0 msgr2=0x7f9804004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9823e00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98040047c0 0x7f9804004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.553+0000 7f9823e00640 1 -- 172.21.15.39:0/3287767100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9804003520 con 0x7f9804003970 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.554+0000 7f9823e00640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f9804003d90 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f981400e170 tx=0x7f98040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0845a9e00cf03b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.554+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981402cca0 con 0x7f9804003970 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981400f040 con 0x7f9804003970 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9822a00640 1 -- 172.21.15.39:0/3287767100 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814034440 con 0x7f9804003970 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9829600640 1 -- 172.21.15.39:0/3287767100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 msgr2=0x7f9804003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9829600640 1 --2- 172.21.15.39:0/3287767100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f9804003d90 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f981400e170 tx=0x7f98040057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9828c00640 1 -- 172.21.15.39:0/3287767100 reap_dead start 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9829600640 1 -- 172.21.15.39:0/3287767100 shutdown_connections 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9829600640 1 -- 172.21.15.39:0/3287767100 >> 172.21.15.39:0/3287767100 conn(0x7f98240d2850 msgr2=0x7f98240d2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9829600640 1 -- 172.21.15.39:0/3287767100 shutdown_connections 2024-01-24T17:50:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.555+0000 7f9829600640 1 -- 172.21.15.39:0/3287767100 wait complete. 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.557+0000 7f9829600640 1 Processor -- start 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.557+0000 7f9829600640 1 -- start start 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9829600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f98241601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9829600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98241f2580 0x7f98241f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9829600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98240b2a30 0x7f98240cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9829600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98240cbfc0 con 0x7f9804003970 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9829600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9824160730 con 0x7f98240b2a30 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9829600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98241f2b20 con 0x7f98241f2580 2024-01-24T17:50:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9823e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f98241601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9823e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f98241601f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56852/0 (socket says 172.21.15.39:56852) 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.558+0000 7f9823e00640 1 -- 172.21.15.39:0/1783412515 learned_addr learned my addr 172.21.15.39:0/1783412515 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f9823e00640 1 -- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98241f2580 msgr2=0x7f98241f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f9823e00640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f98241f2580 0x7f98241f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f9823e00640 1 -- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98240b2a30 msgr2=0x7f98240cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f9823e00640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f98240b2a30 0x7f98240cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f9823e00640 1 -- 172.21.15.39:0/1783412515 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98241f2e00 con 0x7f9804003970 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f9823e00640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f98241601f0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f981400e170 tx=0x7f9814009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814034d90 con 0x7f9804003970 2024-01-24T17:50:04.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9814040450 con 0x7f9804003970 2024-01-24T17:50:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98140407e0 con 0x7f9804003970 2024-01-24T17:50:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98241f30e0 con 0x7f9804003970 2024-01-24T17:50:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.559+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98240ca8e0 con 0x7f9804003970 2024-01-24T17:50:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.560+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f98240b4e50 con 0x7f9804003970 2024-01-24T17:50:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.561+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9814040940 con 0x7f9804003970 2024-01-24T17:50:04.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.562+0000 7f981be00640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f97fc06a430 0x7f97fc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.562+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f98140ab170 con 0x7f9804003970 2024-01-24T17:50:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.563+0000 7f981be00640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f97fc070510 0x7f97fc072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.563+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 --> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97fc072fe0 con 0x7f97fc070510 2024-01-24T17:50:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.563+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98140ad6f0 con 0x7f9804003970 2024-01-24T17:50:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.563+0000 7f9828c00640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f97fc070510 0x7f97fc072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.563+0000 7f9823400640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f97fc06a430 0x7f97fc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.563+0000 7f9828c00640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f97fc070510 0x7f97fc072930 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.564+0000 7f9823400640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f97fc06a430 0x7f97fc06c8f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f980c0017e0 tx=0x7f980c001450 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.564+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 <== osd.1 v2:172.21.15.39:6820/36723 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f97fc072fe0 con 0x7f97fc070510 2024-01-24T17:50:04.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.578+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 --> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f98240b4b70 con 0x7f97fc070510 2024-01-24T17:50:04.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.578+0000 7f981be00640 1 -- 172.21.15.39:0/1783412515 <== osd.1 v2:172.21.15.39:6820/36723 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f981c0095c0 con 0x7f97fc070510 2024-01-24T17:50:04.579 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:50:04.579 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-24T17:50:04.579 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:50:04.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f97fc070510 msgr2=0x7f97fc072930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.39:6820/36723,v1:172.21.15.39:6822/36723] conn(0x7f97fc070510 0x7f97fc072930 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f97fc06a430 msgr2=0x7f97fc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f97fc06a430 0x7f97fc06c8f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f980c0017e0 tx=0x7f980c001450 comp rx=0 tx=0).stop 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 msgr2=0x7f98241601f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 --2- 172.21.15.39:0/1783412515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9804003970 0x7f98241601f0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f981400e170 tx=0x7f9814009750 comp rx=0 tx=0).stop 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9828c00640 1 -- 172.21.15.39:0/1783412515 reap_dead start 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 shutdown_connections 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 >> 172.21.15.39:0/1783412515 conn(0x7f98240d2850 msgr2=0x7f98240d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 shutdown_connections 2024-01-24T17:50:04.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.579+0000 7f9829600640 1 -- 172.21.15.39:0/1783412515 wait complete. 2024-01-24T17:50:04.598 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-24T17:50:04.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.808+0000 7fb043e00640 1 Processor -- start 2024-01-24T17:50:04.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.808+0000 7fb043e00640 1 -- start start 2024-01-24T17:50:04.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.810+0000 7fb043e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb04406acb0 0x7fb0440d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.810+0000 7fb043e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0440ce010 con 0x7fb04415cbf0 2024-01-24T17:50:04.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.810+0000 7fb043e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0440ce2b0 con 0x7fb0440cf8a0 2024-01-24T17:50:04.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.810+0000 7fb043e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0440ce870 con 0x7fb04406acb0 2024-01-24T17:50:04.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.810+0000 7fb042a00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb04415cbf0 0x7fb0440cf190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58210/0 (socket says 172.21.15.39:58210) 2024-01-24T17:50:04.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.810+0000 7fb042a00640 1 -- 172.21.15.39:0/2024467592 learned_addr learned my addr 172.21.15.39:0/2024467592 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:04.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.810+0000 7fb043400640 1 --2- 172.21.15.39:0/2024467592 >> v2:172.21.15.39:3300/0 conn(0x7fb04406acb0 0x7fb0440d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.811+0000 7fb041600640 1 -- 172.21.15.39:0/2024467592 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109089993 0 0) 0x7fb0440ce010 con 0x7fb04415cbf0 2024-01-24T17:50:04.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.811+0000 7fb041600640 1 -- 172.21.15.39:0/2024467592 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb024003520 con 0x7fb04415cbf0 2024-01-24T17:50:04.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.811+0000 7fb043400640 1 -- 172.21.15.39:0/2024467592 >> v1:172.21.15.100:6789/0 conn(0x7fb0440cf8a0 legacy=0x7fb0440d0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:04.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.811+0000 7fb043400640 1 -- 172.21.15.39:0/2024467592 >> v1:172.21.15.39:6789/0 conn(0x7fb04415cbf0 legacy=0x7fb0440cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.811+0000 7fb043400640 1 -- 172.21.15.39:0/2024467592 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0440ce590 con 0x7fb04406acb0 2024-01-24T17:50:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.812+0000 7fb043400640 1 --2- 172.21.15.39:0/2024467592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb04406acb0 0x7fb0440d11a0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009d90 tx=0x7fb03c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d76b23181b9cf6 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.812+0000 7fb041600640 1 -- 172.21.15.39:0/2024467592 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c02cc60 con 0x7fb04406acb0 2024-01-24T17:50:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.812+0000 7fb041600640 1 -- 172.21.15.39:0/2024467592 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c02cdc0 con 0x7fb04406acb0 2024-01-24T17:50:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.812+0000 7fb041600640 1 -- 172.21.15.39:0/2024467592 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c034660 con 0x7fb04406acb0 2024-01-24T17:50:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.813+0000 7fb043e00640 1 -- 172.21.15.39:0/2024467592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb04406acb0 msgr2=0x7fb0440d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.813+0000 7fb043e00640 1 --2- 172.21.15.39:0/2024467592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb04406acb0 0x7fb0440d11a0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009d90 tx=0x7fb03c02beb0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.813+0000 7fb043e00640 1 -- 172.21.15.39:0/2024467592 shutdown_connections 2024-01-24T17:50:04.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.813+0000 7fb043e00640 1 --2- 172.21.15.39:0/2024467592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb04406acb0 0x7fb0440d11a0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.813+0000 7fb043e00640 1 -- 172.21.15.39:0/2024467592 >> 172.21.15.39:0/2024467592 conn(0x7fb04409bad0 msgr2=0x7fb0440d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:04.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.813+0000 7fb043e00640 1 -- 172.21.15.39:0/2024467592 shutdown_connections 2024-01-24T17:50:04.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.813+0000 7fb043e00640 1 -- 172.21.15.39:0/2024467592 wait complete. 2024-01-24T17:50:04.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.815+0000 7fb043e00640 1 Processor -- start 2024-01-24T17:50:04.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.815+0000 7fb043e00640 1 -- start start 2024-01-24T17:50:04.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb043e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04406acb0 0x7fb044159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb043e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0440cf8a0 0x7fb0440c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb043e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb04415cbf0 0x7fb0440b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb043e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440c8e40 con 0x7fb0440cf8a0 2024-01-24T17:50:04.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb043e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04415eac0 con 0x7fb04406acb0 2024-01-24T17:50:04.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb043e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440ccbc0 con 0x7fb04415cbf0 2024-01-24T17:50:04.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb042000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0440cf8a0 0x7fb0440c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb042000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0440cf8a0 0x7fb0440c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56860/0 (socket says 172.21.15.39:56860) 2024-01-24T17:50:04.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.816+0000 7fb042000640 1 -- 172.21.15.39:0/1234787147 learned_addr learned my addr 172.21.15.39:0/1234787147 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:04.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb042a00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04406acb0 0x7fb044159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb043400640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb04415cbf0 0x7fb0440b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb042000640 1 -- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb04415cbf0 msgr2=0x7fb0440b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb042000640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb04415cbf0 0x7fb0440b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb042000640 1 -- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04406acb0 msgr2=0x7fb044159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb042000640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04406acb0 0x7fb044159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb042000640 1 -- 172.21.15.39:0/1234787147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c009a70 con 0x7fb0440cf8a0 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb043400640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb04415cbf0 0x7fb0440b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb042a00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04406acb0 0x7fb044159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb042000640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0440cf8a0 0x7fb0440c91c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00c400 tx=0x7fb02c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb03aa00640 1 -- 172.21.15.39:0/1234787147 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0052d0 con 0x7fb0440cf8a0 2024-01-24T17:50:04.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb03aa00640 1 -- 172.21.15.39:0/1234787147 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c005450 con 0x7fb0440cf8a0 2024-01-24T17:50:04.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.817+0000 7fb03aa00640 1 -- 172.21.15.39:0/1234787147 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c007260 con 0x7fb0440cf8a0 2024-01-24T17:50:04.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.818+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0440c9700 con 0x7fb0440cf8a0 2024-01-24T17:50:04.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.818+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04415e470 con 0x7fb0440cf8a0 2024-01-24T17:50:04.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.819+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04415cfd0 con 0x7fb0440cf8a0 2024-01-24T17:50:04.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.820+0000 7fb03aa00640 1 -- 172.21.15.39:0/1234787147 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb02c0073c0 con 0x7fb0440cf8a0 2024-01-24T17:50:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.820+0000 7fb03aa00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb01c06a430 0x7fb01c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.821+0000 7fb03aa00640 1 -- 172.21.15.39:0/1234787147 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fb02c087c50 con 0x7fb0440cf8a0 2024-01-24T17:50:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.821+0000 7fb042a00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb01c06a430 0x7fb01c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:04.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.822+0000 7fb042a00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb01c06a430 0x7fb01c06c8f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb0340010e0 tx=0x7fb034009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:04.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:04.825+0000 7fb03aa00640 1 -- 172.21.15.39:0/1234787147 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb04415cfd0 con 0x7fb0440cf8a0 2024-01-24T17:50:05.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.020+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0440cfd30 con 0x7fb01c06a430 2024-01-24T17:50:05.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.021+0000 7fb03aa00640 1 -- 172.21.15.39:0/1234787147 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb034001c20 con 0x7fb01c06a430 2024-01-24T17:50:05.021 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 1 to deep-scrub 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb01c06a430 msgr2=0x7fb01c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb01c06a430 0x7fb01c06c8f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb0340010e0 tx=0x7fb034009040 comp rx=0 tx=0).stop 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0440cf8a0 msgr2=0x7fb0440c91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0440cf8a0 0x7fb0440c91c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00c400 tx=0x7fb02c00c8c0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 shutdown_connections 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb04415cbf0 0x7fb0440b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb01c06a430 0x7fb01c06c8f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0440cf8a0 0x7fb0440c91c0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.023+0000 7fb043e00640 1 --2- 172.21.15.39:0/1234787147 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb04406acb0 0x7fb044159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.024+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 >> 172.21.15.39:0/1234787147 conn(0x7fb04409bad0 msgr2=0x7fb0440d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:05.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.024+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 shutdown_connections 2024-01-24T17:50:05.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.024+0000 7fb043e00640 1 -- 172.21.15.39:0/1234787147 wait complete. 2024-01-24T17:50:05.042 INFO:tasks.ceph:Scrubbing osd.2 2024-01-24T17:50:05.042 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-24T17:50:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.249+0000 7fb853800640 1 Processor -- start 2024-01-24T17:50:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.249+0000 7fb853800640 1 -- start start 2024-01-24T17:50:05.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.251+0000 7fb853800640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb84c15f390 0x7fb84c165ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.251+0000 7fb853800640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84c070470 con 0x7fb84c070920 2024-01-24T17:50:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.251+0000 7fb853800640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84c15ef00 con 0x7fb84c0b0010 2024-01-24T17:50:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.251+0000 7fb853800640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84c0710a0 con 0x7fb84c15f390 2024-01-24T17:50:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.251+0000 7fb843e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb84c070920 0x7fb84c0b03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58222/0 (socket says 172.21.15.39:58222) 2024-01-24T17:50:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.251+0000 7fb843e00640 1 -- 172.21.15.39:0/1927998375 learned_addr learned my addr 172.21.15.39:0/1927998375 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.251+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299851417 0 0) 0x7fb84c070470 con 0x7fb84c070920 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.251+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c003520 con 0x7fb84c070920 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb851800640 1 --2- 172.21.15.39:0/1927998375 >> v2:172.21.15.100:3300/0 conn(0x7fb84c15f390 0x7fb84c165ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339938801 0 0) 0x7fb82c003520 con 0x7fb84c070920 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 >> v2:172.21.15.100:3300/0 conn(0x7fb84c15f390 msgr2=0x7fb84c165ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 --2- 172.21.15.39:0/1927998375 >> v2:172.21.15.100:3300/0 conn(0x7fb84c15f390 0x7fb84c165ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 >> v1:172.21.15.100:6789/0 conn(0x7fb84c0b0010 legacy=0x7fb84c15bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c070dc0 con 0x7fb84c070920 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fb834001670 con 0x7fb84c070920 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 >> v1:172.21.15.39:6789/0 conn(0x7fb84c070920 legacy=0x7fb84c0b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c003970 0x7fb82c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb82c0047c0 0x7fb82c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb82c005180 0x7fb82c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c0710a0 con 0x7fb82c0047c0 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c00c0b0 con 0x7fb82c003970 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c00c270 con 0x7fb82c005180 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.252+0000 7fb843e00640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb82c0047c0 0x7fb82c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.253+0000 7fb850e00640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c003970 0x7fb82c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.253+0000 7fb851800640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb82c005180 0x7fb82c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.253+0000 7fb843e00640 1 -- 172.21.15.39:0/1927998375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb82c005180 msgr2=0x7fb82c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.253+0000 7fb843e00640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb82c005180 0x7fb82c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.253+0000 7fb843e00640 1 -- 172.21.15.39:0/1927998375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c003970 msgr2=0x7fb82c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.253+0000 7fb843e00640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c003970 0x7fb82c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.253+0000 7fb843e00640 1 -- 172.21.15.39:0/1927998375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c003520 con 0x7fb82c0047c0 2024-01-24T17:50:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.253+0000 7fb851800640 1 -- 172.21.15.39:0/1927998375 reap_dead start 2024-01-24T17:50:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.254+0000 7fb843e00640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb82c0047c0 0x7fb82c004c40 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fb83400c190 tx=0x7fb82c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e401a616c08db8c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.254+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834005ad0 con 0x7fb82c0047c0 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.254+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb834005c30 con 0x7fb82c0047c0 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.254+0000 7fb84be00640 1 -- 172.21.15.39:0/1927998375 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83402cab0 con 0x7fb82c0047c0 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.255+0000 7fb853800640 1 -- 172.21.15.39:0/1927998375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb82c0047c0 msgr2=0x7fb82c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.255+0000 7fb853800640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb82c0047c0 0x7fb82c004c40 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fb83400c190 tx=0x7fb82c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.255+0000 7fb853800640 1 -- 172.21.15.39:0/1927998375 shutdown_connections 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.255+0000 7fb853800640 1 --2- 172.21.15.39:0/1927998375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb82c0047c0 0x7fb82c004c40 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.255+0000 7fb853800640 1 -- 172.21.15.39:0/1927998375 >> 172.21.15.39:0/1927998375 conn(0x7fb84c06bf20 msgr2=0x7fb84c06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.255+0000 7fb853800640 1 -- 172.21.15.39:0/1927998375 shutdown_connections 2024-01-24T17:50:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.255+0000 7fb853800640 1 -- 172.21.15.39:0/1927998375 wait complete. 2024-01-24T17:50:05.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.257+0000 7fb853800640 1 Processor -- start 2024-01-24T17:50:05.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.257+0000 7fb853800640 1 -- start start 2024-01-24T17:50:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.258+0000 7fb853800640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c0047c0 0x7fb84c1efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.258+0000 7fb853800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb84c0c8620 0x7fb84c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.258+0000 7fb853800640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84c15d9a0 0x7fb84c159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.258+0000 7fb853800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c15de20 con 0x7fb84c0c8620 2024-01-24T17:50:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.258+0000 7fb853800640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c15ad30 con 0x7fb82c0047c0 2024-01-24T17:50:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.258+0000 7fb853800640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c0c7860 con 0x7fb84c15d9a0 2024-01-24T17:50:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.258+0000 7fb843e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb84c0c8620 0x7fb84c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.258+0000 7fb843e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb84c0c8620 0x7fb84c15d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56882/0 (socket says 172.21.15.39:56882) 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb843e00640 1 -- 172.21.15.39:0/3768621169 learned_addr learned my addr 172.21.15.39:0/3768621169 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb851800640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84c15d9a0 0x7fb84c159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb850e00640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c0047c0 0x7fb84c1efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb843e00640 1 -- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84c15d9a0 msgr2=0x7fb84c159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb843e00640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb84c15d9a0 0x7fb84c159f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb843e00640 1 -- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c0047c0 msgr2=0x7fb84c1efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb843e00640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c0047c0 0x7fb84c1efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb843e00640 1 -- 172.21.15.39:0/3768621169 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c0c7b40 con 0x7fb84c0c8620 2024-01-24T17:50:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb850e00640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb82c0047c0 0x7fb84c1efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.259+0000 7fb843e00640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb84c0c8620 0x7fb84c15d460 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fb834001930 tx=0x7fb83402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.260+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83403c040 con 0x7fb84c0c8620 2024-01-24T17:50:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.260+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83402cc80 con 0x7fb84c0c8620 2024-01-24T17:50:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.260+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834007070 con 0x7fb84c0c8620 2024-01-24T17:50:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.260+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c0c7e20 con 0x7fb84c0c8620 2024-01-24T17:50:05.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.260+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c1f0240 con 0x7fb84c0c8620 2024-01-24T17:50:05.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.261+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb84c15eb90 con 0x7fb84c0c8620 2024-01-24T17:50:05.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.262+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb834032070 con 0x7fb84c0c8620 2024-01-24T17:50:05.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.263+0000 7fb849600640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb82406a430 0x7fb82406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.263+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fb8340a9c00 con 0x7fb84c0c8620 2024-01-24T17:50:05.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.264+0000 7fb849600640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fb824070510 0x7fb824072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.264+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 --> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb824072fe0 con 0x7fb824070510 2024-01-24T17:50:05.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.264+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8340ae2a0 con 0x7fb84c0c8620 2024-01-24T17:50:05.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.264+0000 7fb851800640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fb824070510 0x7fb824072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.264+0000 7fb850e00640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb82406a430 0x7fb82406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.265+0000 7fb851800640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fb824070510 0x7fb824072930 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.265+0000 7fb850e00640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb82406a430 0x7fb82406c8f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb84c072a80 tx=0x7fb83c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.266+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 <== osd.2 v2:172.21.15.39:6808/36726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb824072fe0 con 0x7fb824070510 2024-01-24T17:50:05.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.280+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 --> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb84c15e8b0 con 0x7fb824070510 2024-01-24T17:50:05.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.280+0000 7fb849600640 1 -- 172.21.15.39:0/3768621169 <== osd.2 v2:172.21.15.39:6808/36726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb844003ed0 con 0x7fb824070510 2024-01-24T17:50:05.281 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:50:05.281 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-24T17:50:05.281 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:50:05.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.280+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fb824070510 msgr2=0x7fb824072930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.39:6808/36726,v1:172.21.15.39:6809/36726] conn(0x7fb824070510 0x7fb824072930 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb82406a430 msgr2=0x7fb82406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb82406a430 0x7fb82406c8f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb84c072a80 tx=0x7fb83c009040 comp rx=0 tx=0).stop 2024-01-24T17:50:05.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb84c0c8620 msgr2=0x7fb84c15d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 --2- 172.21.15.39:0/3768621169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb84c0c8620 0x7fb84c15d460 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fb834001930 tx=0x7fb83402e070 comp rx=0 tx=0).stop 2024-01-24T17:50:05.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb851800640 1 -- 172.21.15.39:0/3768621169 reap_dead start 2024-01-24T17:50:05.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 shutdown_connections 2024-01-24T17:50:05.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 >> 172.21.15.39:0/3768621169 conn(0x7fb84c06bf20 msgr2=0x7fb84c0cb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:05.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 shutdown_connections 2024-01-24T17:50:05.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.281+0000 7fb853800640 1 -- 172.21.15.39:0/3768621169 wait complete. 2024-01-24T17:50:05.297 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-24T17:50:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.510+0000 7f7c63000640 1 Processor -- start 2024-01-24T17:50:05.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.510+0000 7f7c63000640 1 -- start start 2024-01-24T17:50:05.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.512+0000 7f7c63000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c5c070920 0x7f7c5c0b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.512+0000 7f7c63000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c070470 con 0x7f7c5c15f390 2024-01-24T17:50:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.512+0000 7f7c63000640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c15ef00 con 0x7f7c5c0b0010 2024-01-24T17:50:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.512+0000 7f7c63000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0710a0 con 0x7f7c5c070920 2024-01-24T17:50:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.512+0000 7f7c61000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f7c5c15f390 0x7f7c5c0c9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58228/0 (socket says 172.21.15.39:58228) 2024-01-24T17:50:05.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.512+0000 7f7c61000640 1 -- 172.21.15.39:0/2515920320 learned_addr learned my addr 172.21.15.39:0/2515920320 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.512+0000 7f7c5b400640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c5c070920 0x7f7c5c0b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846253710 0 0) 0x7f7c5c070470 con 0x7f7c5c15f390 2024-01-24T17:50:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c3c003520 con 0x7f7c5c15f390 2024-01-24T17:50:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946139782 0 0) 0x7f7c3c003520 con 0x7f7c5c15f390 2024-01-24T17:50:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c5c070920 msgr2=0x7f7c5c0b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c5c070920 0x7f7c5c0b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 >> v1:172.21.15.100:6789/0 conn(0x7f7c5c0b0010 legacy=0x7f7c5c15bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c0b0930 con 0x7f7c5c15f390 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f7c540019f0 con 0x7f7c5c15f390 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 >> v1:172.21.15.39:6789/0 conn(0x7f7c5c15f390 legacy=0x7f7c5c0c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c0039a0 0x7f7c3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c3c0047c0 0x7f7c3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c5c070920 0x7f7c3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0710a0 con 0x7f7c3c0039a0 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c00bac0 con 0x7f7c3c0047c0 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3c00bc80 con 0x7f7c5c070920 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.513+0000 7f7c5be00640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c0039a0 0x7f7c3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.514+0000 7f7c5b400640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c3c0047c0 0x7f7c3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.514+0000 7f7c61000640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c5c070920 0x7f7c3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.514+0000 7f7c5be00640 1 -- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c5c070920 msgr2=0x7f7c3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.514+0000 7f7c5be00640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c5c070920 0x7f7c3c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.514+0000 7f7c5be00640 1 -- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c3c0047c0 msgr2=0x7f7c3c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.514+0000 7f7c5be00640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c3c0047c0 0x7f7c3c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.514+0000 7f7c5be00640 1 -- 172.21.15.39:0/2515920320 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c003520 con 0x7f7c3c0039a0 2024-01-24T17:50:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.515+0000 7f7c5be00640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c0039a0 0x7f7c3c003dc0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c001720 tx=0x7f7c3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494748dea8e63d94 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.515+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0052d0 con 0x7f7c3c0039a0 2024-01-24T17:50:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.515+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c02e050 con 0x7f7c3c0039a0 2024-01-24T17:50:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.515+0000 7f7c5aa00640 1 -- 172.21.15.39:0/2515920320 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c0035f0 con 0x7f7c3c0039a0 2024-01-24T17:50:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.516+0000 7f7c63000640 1 -- 172.21.15.39:0/2515920320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c0039a0 msgr2=0x7f7c3c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.516+0000 7f7c63000640 1 --2- 172.21.15.39:0/2515920320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c3c0039a0 0x7f7c3c003dc0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c001720 tx=0x7f7c3c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.516+0000 7f7c61000640 1 -- 172.21.15.39:0/2515920320 reap_dead start 2024-01-24T17:50:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.516+0000 7f7c63000640 1 -- 172.21.15.39:0/2515920320 shutdown_connections 2024-01-24T17:50:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.516+0000 7f7c63000640 1 -- 172.21.15.39:0/2515920320 >> 172.21.15.39:0/2515920320 conn(0x7f7c5c06bf20 msgr2=0x7f7c5c06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.516+0000 7f7c63000640 1 -- 172.21.15.39:0/2515920320 shutdown_connections 2024-01-24T17:50:05.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.516+0000 7f7c63000640 1 -- 172.21.15.39:0/2515920320 wait complete. 2024-01-24T17:50:05.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.518+0000 7f7c63000640 1 Processor -- start 2024-01-24T17:50:05.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.518+0000 7f7c63000640 1 -- start start 2024-01-24T17:50:05.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c63000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c3c0039a0 0x7f7c5c153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c63000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c5c157490 0x7f7c5c1f0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c63000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c5c1f1650 0x7f7c5c14e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c63000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c1501e0 con 0x7f7c5c157490 2024-01-24T17:50:05.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c63000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c1f1470 con 0x7f7c5c1f1650 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c63000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c153540 con 0x7f7c3c0039a0 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c5b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c5c157490 0x7f7c5c1f0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c5b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c5c157490 0x7f7c5c1f0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56908/0 (socket says 172.21.15.39:56908) 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.519+0000 7f7c5b400640 1 -- 172.21.15.39:0/3432450742 learned_addr learned my addr 172.21.15.39:0/3432450742 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c5be00640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c3c0039a0 0x7f7c5c153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c61000640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c5c1f1650 0x7f7c5c14e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c5b400640 1 -- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c3c0039a0 msgr2=0x7f7c5c153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c5b400640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c3c0039a0 0x7f7c5c153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c5b400640 1 -- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c5c1f1650 msgr2=0x7f7c5c14e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c5b400640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c5c1f1650 0x7f7c5c14e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c5b400640 1 -- 172.21.15.39:0/3432450742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c1f0eb0 con 0x7f7c5c157490 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c5be00640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c3c0039a0 0x7f7c5c153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c61000640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c5c1f1650 0x7f7c5c14e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:05.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c5b400640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c5c157490 0x7f7c5c1f0560 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7c44011630 tx=0x7f7c44011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c53e00640 1 -- 172.21.15.39:0/3432450742 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400f8c0 con 0x7f7c5c157490 2024-01-24T17:50:05.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c53e00640 1 -- 172.21.15.39:0/3432450742 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4400fa20 con 0x7f7c5c157490 2024-01-24T17:50:05.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c53e00640 1 -- 172.21.15.39:0/3432450742 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400fd90 con 0x7f7c5c157490 2024-01-24T17:50:05.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.520+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c1f1190 con 0x7f7c5c157490 2024-01-24T17:50:05.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.521+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c1506d0 con 0x7f7c5c157490 2024-01-24T17:50:05.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.522+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c04dff0 con 0x7f7c5c157490 2024-01-24T17:50:05.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.523+0000 7f7c53e00640 1 -- 172.21.15.39:0/3432450742 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7c44015460 con 0x7f7c5c157490 2024-01-24T17:50:05.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.524+0000 7f7c53e00640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7c3406a460 0x7f7c3406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.524+0000 7f7c53e00640 1 -- 172.21.15.39:0/3432450742 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f7c44088ba0 con 0x7f7c5c157490 2024-01-24T17:50:05.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.524+0000 7f7c5be00640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7c3406a460 0x7f7c3406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.525+0000 7f7c5be00640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7c3406a460 0x7f7c3406c920 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c033000 tx=0x7f7c4c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.528+0000 7f7c53e00640 1 -- 172.21.15.39:0/3432450742 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7c44054290 con 0x7f7c5c157490 2024-01-24T17:50:05.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.720+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c5c15e8b0 con 0x7f7c3406a460 2024-01-24T17:50:05.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.721+0000 7f7c53e00640 1 -- 172.21.15.39:0/3432450742 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7c4c0054e0 con 0x7f7c3406a460 2024-01-24T17:50:05.722 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 2 to deep-scrub 2024-01-24T17:50:05.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7c3406a460 msgr2=0x7f7c3406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7c3406a460 0x7f7c3406c920 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c033000 tx=0x7f7c4c009030 comp rx=0 tx=0).stop 2024-01-24T17:50:05.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c5c157490 msgr2=0x7f7c5c1f0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c5c157490 0x7f7c5c1f0560 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7c44011630 tx=0x7f7c44011af0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 shutdown_connections 2024-01-24T17:50:05.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7c5c1f1650 0x7f7c5c14e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7c3406a460 0x7f7c3406c920 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c5c157490 0x7f7c5c1f0560 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 --2- 172.21.15.39:0/3432450742 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7c3c0039a0 0x7f7c5c153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.723+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 >> 172.21.15.39:0/3432450742 conn(0x7f7c5c06bf20 msgr2=0x7f7c5c0aeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:05.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.724+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 shutdown_connections 2024-01-24T17:50:05.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.724+0000 7f7c63000640 1 -- 172.21.15.39:0/3432450742 wait complete. 2024-01-24T17:50:05.744 INFO:tasks.ceph:Scrubbing osd.3 2024-01-24T17:50:05.744 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-24T17:50:05.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.972+0000 7fd957e00640 1 Processor -- start 2024-01-24T17:50:05.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.972+0000 7fd957e00640 1 -- start start 2024-01-24T17:50:05.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.974+0000 7fd957e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fd9500b0010 0x7fd9500b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.974+0000 7fd957e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9500cd420 0x7fd9500cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.974+0000 7fd957e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9500cddf0 con 0x7fd95015cf10 2024-01-24T17:50:05.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.974+0000 7fd957e00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9500b09d0 con 0x7fd9500b0010 2024-01-24T17:50:05.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.974+0000 7fd957e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500ccf90 con 0x7fd9500cd420 2024-01-24T17:50:05.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.975+0000 7fd955400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd95015cf10 0x7fd950070440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58232/0 (socket says 172.21.15.39:58232) 2024-01-24T17:50:05.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.975+0000 7fd955400640 1 -- 172.21.15.39:0/695312723 learned_addr learned my addr 172.21.15.39:0/695312723 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:05.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.975+0000 7fd955e00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9500cd420 0x7fd9500cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.975+0000 7fd954a00640 1 --2- 172.21.15.39:0/695312723 >> v2:172.21.15.100:3300/0 conn(0x7fd9500b0010 0x7fd9500b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.975+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867604219 0 0) 0x7fd9500cddf0 con 0x7fd95015cf10 2024-01-24T17:50:05.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.975+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd930003520 con 0x7fd95015cf10 2024-01-24T17:50:05.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020508094 0 0) 0x7fd930003520 con 0x7fd95015cf10 2024-01-24T17:50:05.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9500cd420 msgr2=0x7fd9500cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9500cd420 0x7fd9500cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 >> v2:172.21.15.100:3300/0 conn(0x7fd9500b0010 msgr2=0x7fd9500b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 --2- 172.21.15.39:0/695312723 >> v2:172.21.15.100:3300/0 conn(0x7fd9500b0010 0x7fd9500b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9500c9bd0 con 0x7fd95015cf10 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fd940001670 con 0x7fd95015cf10 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 >> v1:172.21.15.39:6789/0 conn(0x7fd95015cf10 legacy=0x7fd950070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9500cd420 0x7fd930003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd930003f40 0x7fd9300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd930005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500b09d0 con 0x7fd930004d00 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500ccf90 con 0x7fd930003f40 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93000b980 con 0x7fd9500cd420 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.976+0000 7fd955e00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd930005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.977+0000 7fd955400640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9500cd420 0x7fd930003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.977+0000 7fd954a00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd930003f40 0x7fd9300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.977+0000 7fd955e00640 1 -- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9500cd420 msgr2=0x7fd930003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.977+0000 7fd955e00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9500cd420 0x7fd930003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.977+0000 7fd955e00640 1 -- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd930003f40 msgr2=0x7fd9300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.977+0000 7fd955e00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd930003f40 0x7fd9300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.977+0000 7fd955e00640 1 -- 172.21.15.39:0/695312723 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd930003520 con 0x7fd930004d00 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.977+0000 7fd954a00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd930003f40 0x7fd9300047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.978+0000 7fd955e00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd930005180 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fd94800e170 tx=0x7fd9300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f334cdb2529a024 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.978+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94802cbf0 con 0x7fd930004d00 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.978+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94800f040 con 0x7fd930004d00 2024-01-24T17:50:05.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.978+0000 7fd94fe00640 1 -- 172.21.15.39:0/695312723 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948034440 con 0x7fd930004d00 2024-01-24T17:50:05.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.979+0000 7fd957e00640 1 -- 172.21.15.39:0/695312723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 msgr2=0x7fd930005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.979+0000 7fd957e00640 1 --2- 172.21.15.39:0/695312723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd930005180 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fd94800e170 tx=0x7fd9300057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.979+0000 7fd955e00640 1 -- 172.21.15.39:0/695312723 reap_dead start 2024-01-24T17:50:05.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.979+0000 7fd957e00640 1 -- 172.21.15.39:0/695312723 shutdown_connections 2024-01-24T17:50:05.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.979+0000 7fd957e00640 1 -- 172.21.15.39:0/695312723 >> 172.21.15.39:0/695312723 conn(0x7fd95006bf20 msgr2=0x7fd95006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:05.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.979+0000 7fd957e00640 1 -- 172.21.15.39:0/695312723 shutdown_connections 2024-01-24T17:50:05.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.979+0000 7fd957e00640 1 -- 172.21.15.39:0/695312723 wait complete. 2024-01-24T17:50:05.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.981+0000 7fd957e00640 1 Processor -- start 2024-01-24T17:50:05.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.981+0000 7fd957e00640 1 -- start start 2024-01-24T17:50:05.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd957e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd95015ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd957e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9501f4ba0 0x7fd9501f5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd957e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9500c6700 0x7fd9500c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd957e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9501f5540 con 0x7fd930004d00 2024-01-24T17:50:05.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd957e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950160200 con 0x7fd9500c6700 2024-01-24T17:50:05.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd957e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500c5660 con 0x7fd9501f4ba0 2024-01-24T17:50:05.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd955400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd95015ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd955400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd95015ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56926/0 (socket says 172.21.15.39:56926) 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd955400640 1 -- 172.21.15.39:0/1240193811 learned_addr learned my addr 172.21.15.39:0/1240193811 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd955400640 1 -- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9501f4ba0 msgr2=0x7fd9501f5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd955400640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fd9501f4ba0 0x7fd9501f5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd955400640 1 -- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9500c6700 msgr2=0x7fd9500c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd955400640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd9500c6700 0x7fd9500c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.982+0000 7fd955400640 1 -- 172.21.15.39:0/1240193811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9500c5940 con 0x7fd930004d00 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.983+0000 7fd955400640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd95015ece0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fd94000e7b0 tx=0x7fd94000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.983+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000c850 con 0x7fd930004d00 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.983+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94000c9d0 con 0x7fd930004d00 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.983+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000cd80 con 0x7fd930004d00 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.983+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9500c5c20 con 0x7fd930004d00 2024-01-24T17:50:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.983+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9500c61e0 con 0x7fd930004d00 2024-01-24T17:50:05.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.984+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd9500cc940 con 0x7fd930004d00 2024-01-24T17:50:05.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.985+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fd94001d460 con 0x7fd930004d00 2024-01-24T17:50:05.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.986+0000 7fd94d600640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd92806a430 0x7fd92806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.986+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fd940089280 con 0x7fd930004d00 2024-01-24T17:50:05.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.987+0000 7fd94d600640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fd928070510 0x7fd928072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:05.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.987+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 --> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd928072fe0 con 0x7fd928070510 2024-01-24T17:50:05.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.987+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9400895f0 con 0x7fd930004d00 2024-01-24T17:50:05.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.987+0000 7fd955e00640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fd928070510 0x7fd928072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.987+0000 7fd954a00640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd92806a430 0x7fd92806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:05.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.987+0000 7fd955e00640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fd928070510 0x7fd928072930 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.988+0000 7fd954a00640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd92806a430 0x7fd92806c8f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd9500c19b0 tx=0x7fd938008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:05.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:05.988+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 <== osd.3 v2:172.21.15.39:6816/36725 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd928072fe0 con 0x7fd928070510 2024-01-24T17:50:06.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.003+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 --> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd9500c9eb0 con 0x7fd928070510 2024-01-24T17:50:06.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.004+0000 7fd94d600640 1 -- 172.21.15.39:0/1240193811 <== osd.3 v2:172.21.15.39:6816/36725 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd94802da80 con 0x7fd928070510 2024-01-24T17:50:06.005 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:50:06.005 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-24T17:50:06.005 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.004+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fd928070510 msgr2=0x7fd928072930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.004+0000 7fd957e00640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.39:6816/36725,v1:172.21.15.39:6817/36725] conn(0x7fd928070510 0x7fd928072930 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.004+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd92806a430 msgr2=0x7fd92806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.004+0000 7fd957e00640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fd92806a430 0x7fd92806c8f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd9500c19b0 tx=0x7fd938008040 comp rx=0 tx=0).stop 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.004+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 msgr2=0x7fd95015ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.004+0000 7fd957e00640 1 --2- 172.21.15.39:0/1240193811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd930004d00 0x7fd95015ece0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fd94000e7b0 tx=0x7fd94000ec70 comp rx=0 tx=0).stop 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.005+0000 7fd955e00640 1 -- 172.21.15.39:0/1240193811 reap_dead start 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.005+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 shutdown_connections 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.005+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 >> 172.21.15.39:0/1240193811 conn(0x7fd95006bf20 msgr2=0x7fd950071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.005+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 shutdown_connections 2024-01-24T17:50:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.005+0000 7fd957e00640 1 -- 172.21.15.39:0/1240193811 wait complete. 2024-01-24T17:50:06.025 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-24T17:50:06.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.251+0000 7f2e51200640 1 Processor -- start 2024-01-24T17:50:06.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.252+0000 7f2e51200640 1 -- start start 2024-01-24T17:50:06.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.253+0000 7f2e51200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c15cbf0 0x7f2e4c0cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.253+0000 7f2e51200640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f2e4c0b1570 0x7f2e4c0cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.253+0000 7f2e51200640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4c0ce010 con 0x7f2e4c06acb0 2024-01-24T17:50:06.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.253+0000 7f2e51200640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e4c0ce870 con 0x7f2e4c0b1570 2024-01-24T17:50:06.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.253+0000 7f2e51200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c0ce2b0 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.254+0000 7f2e4b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c15cbf0 0x7f2e4c0cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.254+0000 7f2e4b400640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c15cbf0 0x7f2e4c0cf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3301/0 says I am v2:172.21.15.39:35572/0 (socket says 172.21.15.39:35572) 2024-01-24T17:50:06.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.254+0000 7f2e4b400640 1 -- 172.21.15.39:0/4209123307 learned_addr learned my addr 172.21.15.39:0/4209123307 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:06.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.254+0000 7f2e4be00640 1 --2- 172.21.15.39:0/4209123307 >> v2:172.21.15.100:3300/0 conn(0x7f2e4c0b1570 0x7f2e4c0cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.255+0000 7f2e4b400640 1 -- 172.21.15.39:0/4209123307 >> v2:172.21.15.100:3300/0 conn(0x7f2e4c0b1570 msgr2=0x7f2e4c0cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:06.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.255+0000 7f2e4b400640 1 --2- 172.21.15.39:0/4209123307 >> v2:172.21.15.100:3300/0 conn(0x7f2e4c0b1570 0x7f2e4c0cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.255+0000 7f2e4b400640 1 -- 172.21.15.39:0/4209123307 >> v1:172.21.15.100:6789/0 conn(0x7f2e4c06acb0 legacy=0x7f2e4c1605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.255+0000 7f2e4b400640 1 -- 172.21.15.39:0/4209123307 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c0ce590 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.255+0000 7f2e4be00640 1 --2- 172.21.15.39:0/4209123307 >> v2:172.21.15.100:3300/0 conn(0x7f2e4c0b1570 0x7f2e4c0cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T17:50:06.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.255+0000 7f2e4a000640 1 -- 172.21.15.39:0/4209123307 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101617530 0 0) 0x7f2e4c0ce010 con 0x7f2e4c06acb0 2024-01-24T17:50:06.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.257+0000 7f2e4b400640 1 --2- 172.21.15.39:0/4209123307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c15cbf0 0x7f2e4c0cf190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009850 tx=0x7f2e3402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef6bceaadfef0627 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:06.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.257+0000 7f2e4a000640 1 -- 172.21.15.39:0/4209123307 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34004230 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.257+0000 7f2e4a000640 1 -- 172.21.15.39:0/4209123307 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e34004390 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.258+0000 7f2e4a000640 1 -- 172.21.15.39:0/4209123307 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e34031830 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.258+0000 7f2e51200640 1 -- 172.21.15.39:0/4209123307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c15cbf0 msgr2=0x7f2e4c0cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.258+0000 7f2e51200640 1 --2- 172.21.15.39:0/4209123307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c15cbf0 0x7f2e4c0cf190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009850 tx=0x7f2e3402b8d0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.258+0000 7f2e51200640 1 -- 172.21.15.39:0/4209123307 shutdown_connections 2024-01-24T17:50:06.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.258+0000 7f2e51200640 1 --2- 172.21.15.39:0/4209123307 >> v2:172.21.15.100:3300/0 conn(0x7f2e4c0b1570 0x7f2e4c0cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.258+0000 7f2e51200640 1 --2- 172.21.15.39:0/4209123307 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c15cbf0 0x7f2e4c0cf190 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.258+0000 7f2e51200640 1 -- 172.21.15.39:0/4209123307 >> 172.21.15.39:0/4209123307 conn(0x7f2e4c09bad0 msgr2=0x7f2e4c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:06.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.258+0000 7f2e51200640 1 -- 172.21.15.39:0/4209123307 shutdown_connections 2024-01-24T17:50:06.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.259+0000 7f2e51200640 1 -- 172.21.15.39:0/4209123307 wait complete. 2024-01-24T17:50:06.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.261+0000 7f2e51200640 1 Processor -- start 2024-01-24T17:50:06.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.261+0000 7f2e51200640 1 -- start start 2024-01-24T17:50:06.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.262+0000 7f2e51200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c06acb0 0x7f2e4c14d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.262+0000 7f2e51200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e4c0b1570 0x7f2e4c09cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.262+0000 7f2e51200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e4c15cbf0 0x7f2e4c0b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.262+0000 7f2e51200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c1508f0 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.262+0000 7f2e51200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c14fa20 con 0x7f2e4c0b1570 2024-01-24T17:50:06.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.262+0000 7f2e51200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4c14d6b0 con 0x7f2e4c06acb0 2024-01-24T17:50:06.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e4c15cbf0 0x7f2e4c0b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e4c15cbf0 0x7f2e4c0b2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56940/0 (socket says 172.21.15.39:56940) 2024-01-24T17:50:06.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4be00640 1 -- 172.21.15.39:0/2599054208 learned_addr learned my addr 172.21.15.39:0/2599054208 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:06.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4b400640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c06acb0 0x7f2e4c14d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4aa00640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e4c0b1570 0x7f2e4c09cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4be00640 1 -- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c06acb0 msgr2=0x7f2e4c14d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4be00640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c06acb0 0x7f2e4c14d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4be00640 1 -- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e4c0b1570 msgr2=0x7f2e4c09cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4be00640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e4c0b1570 0x7f2e4c09cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.263+0000 7f2e4be00640 1 -- 172.21.15.39:0/2599054208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e44009400 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.264+0000 7f2e4be00640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e4c15cbf0 0x7f2e4c0b2010 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400e930 tx=0x7f2e4400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:06.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.264+0000 7f2e43400640 1 -- 172.21.15.39:0/2599054208 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e44009700 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.264+0000 7f2e43400640 1 -- 172.21.15.39:0/2599054208 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4400ca00 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.264+0000 7f2e43400640 1 -- 172.21.15.39:0/2599054208 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400cdb0 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.264+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e34009460 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.264+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4c14ff50 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.265+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e4c04dff0 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.267+0000 7f2e43400640 1 -- 172.21.15.39:0/2599054208 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2e44013470 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.267+0000 7f2e43400640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2e2406a430 0x7f2e2406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.268+0000 7f2e43400640 1 -- 172.21.15.39:0/2599054208 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f2e44088a30 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.268+0000 7f2e4b400640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2e2406a430 0x7f2e2406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.269+0000 7f2e4b400640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2e2406a430 0x7f2e2406c8f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2e34035000 tx=0x7f2e34008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:06.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.272+0000 7f2e43400640 1 -- 172.21.15.39:0/2599054208 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2e440541b0 con 0x7f2e4c15cbf0 2024-01-24T17:50:06.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.468+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e4c0cfd30 con 0x7f2e2406a430 2024-01-24T17:50:06.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.469+0000 7f2e43400640 1 -- 172.21.15.39:0/2599054208 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2e34040070 con 0x7f2e2406a430 2024-01-24T17:50:06.469 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 3 to deep-scrub 2024-01-24T17:50:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.471+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2e2406a430 msgr2=0x7f2e2406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.471+0000 7f2e51200640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2e2406a430 0x7f2e2406c8f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2e34035000 tx=0x7f2e34008030 comp rx=0 tx=0).stop 2024-01-24T17:50:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.471+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e4c15cbf0 msgr2=0x7f2e4c0b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.471+0000 7f2e51200640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e4c15cbf0 0x7f2e4c0b2010 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400e930 tx=0x7f2e4400edf0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.472+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 shutdown_connections 2024-01-24T17:50:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.472+0000 7f2e51200640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2e2406a430 0x7f2e2406c8f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.472+0000 7f2e51200640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e4c15cbf0 0x7f2e4c0b2010 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.472+0000 7f2e51200640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e4c0b1570 0x7f2e4c09cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.472+0000 7f2e51200640 1 --2- 172.21.15.39:0/2599054208 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e4c06acb0 0x7f2e4c14d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.472+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 >> 172.21.15.39:0/2599054208 conn(0x7f2e4c09bad0 msgr2=0x7f2e4c06b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.472+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 shutdown_connections 2024-01-24T17:50:06.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.472+0000 7f2e51200640 1 -- 172.21.15.39:0/2599054208 wait complete. 2024-01-24T17:50:06.491 INFO:tasks.ceph:Scrubbing osd.4 2024-01-24T17:50:06.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-24T17:50:06.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.708+0000 7f3c6be00640 1 Processor -- start 2024-01-24T17:50:06.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.708+0000 7f3c6be00640 1 -- start start 2024-01-24T17:50:06.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.710+0000 7f3c6be00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3c6c06b6e0 0x7f3c6c06bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.710+0000 7f3c6be00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c161450 con 0x7f3c6c06acf0 2024-01-24T17:50:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.710+0000 7f3c6be00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c06c000 con 0x7f3c6c06b180 2024-01-24T17:50:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.710+0000 7f3c6be00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c0b42d0 con 0x7f3c6c06b6e0 2024-01-24T17:50:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.711+0000 7f3c63e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3c6c06acf0 0x7f3c6c160630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58248/0 (socket says 172.21.15.39:58248) 2024-01-24T17:50:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.711+0000 7f3c63e00640 1 -- 172.21.15.39:0/1419611989 learned_addr learned my addr 172.21.15.39:0/1419611989 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:06.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.711+0000 7f3c6aa00640 1 --2- 172.21.15.39:0/1419611989 >> v2:172.21.15.39:3300/0 conn(0x7f3c6c06b6e0 0x7f3c6c06bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.711+0000 7f3c6aa00640 1 -- 172.21.15.39:0/1419611989 >> v1:172.21.15.100:6789/0 conn(0x7f3c6c06b180 legacy=0x7f3c6c160d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.711+0000 7f3c6aa00640 1 -- 172.21.15.39:0/1419611989 >> v1:172.21.15.39:6789/0 conn(0x7f3c6c06acf0 legacy=0x7f3c6c160630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.711+0000 7f3c6aa00640 1 -- 172.21.15.39:0/1419611989 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c0b45b0 con 0x7f3c6c06b6e0 2024-01-24T17:50:06.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.712+0000 7f3c6aa00640 1 --2- 172.21.15.39:0/1419611989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b6e0 0x7f3c6c06bac0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009790 tx=0x7f3c5c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702d7a0cb29b686c server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:06.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6a000640 1 -- 172.21.15.39:0/1419611989 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c030800 con 0x7f3c6c06b6e0 2024-01-24T17:50:06.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6a000640 1 -- 172.21.15.39:0/1419611989 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c030960 con 0x7f3c6c06b6e0 2024-01-24T17:50:06.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6a000640 1 -- 172.21.15.39:0/1419611989 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c030cf0 con 0x7f3c6c06b6e0 2024-01-24T17:50:06.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6be00640 1 -- 172.21.15.39:0/1419611989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b6e0 msgr2=0x7f3c6c06bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6be00640 1 --2- 172.21.15.39:0/1419611989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b6e0 0x7f3c6c06bac0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009790 tx=0x7f3c5c02b940 comp rx=0 tx=0).stop 2024-01-24T17:50:06.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6be00640 1 -- 172.21.15.39:0/1419611989 shutdown_connections 2024-01-24T17:50:06.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6be00640 1 --2- 172.21.15.39:0/1419611989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b6e0 0x7f3c6c06bac0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6be00640 1 -- 172.21.15.39:0/1419611989 >> 172.21.15.39:0/1419611989 conn(0x7f3c6c0d2850 msgr2=0x7f3c6c0d2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:06.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6be00640 1 -- 172.21.15.39:0/1419611989 shutdown_connections 2024-01-24T17:50:06.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.713+0000 7f3c6be00640 1 -- 172.21.15.39:0/1419611989 wait complete. 2024-01-24T17:50:06.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.715+0000 7f3c6be00640 1 Processor -- start 2024-01-24T17:50:06.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.715+0000 7f3c6be00640 1 -- start start 2024-01-24T17:50:06.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.716+0000 7f3c6be00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c6c06acf0 0x7f3c6c155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.716+0000 7f3c6be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b180 0x7f3c6c148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.716+0000 7f3c6be00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c6c06b6e0 0x7f3c6c0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.716+0000 7f3c6be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0c9da0 con 0x7f3c6c06b180 2024-01-24T17:50:06.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.716+0000 7f3c6be00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0ca770 con 0x7f3c6c06b6e0 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.716+0000 7f3c6be00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c15c820 con 0x7f3c6c06acf0 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.716+0000 7f3c63e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b180 0x7f3c6c148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c63e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b180 0x7f3c6c148d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56948/0 (socket says 172.21.15.39:56948) 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c63e00640 1 -- 172.21.15.39:0/1678944858 learned_addr learned my addr 172.21.15.39:0/1678944858 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c6b400640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c6c06b6e0 0x7f3c6c0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c63e00640 1 -- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c6c06acf0 msgr2=0x7f3c6c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c6aa00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c6c06acf0 0x7f3c6c155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c63e00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c6c06acf0 0x7f3c6c155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c63e00640 1 -- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c6c06b6e0 msgr2=0x7f3c6c0ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c63e00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c6c06b6e0 0x7f3c6c0ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c63e00640 1 -- 172.21.15.39:0/1678944858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0093a0 con 0x7f3c6c06b180 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c6b400640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c6c06b6e0 0x7f3c6c0ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:06.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c6aa00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f3c6c06acf0 0x7f3c6c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:06.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c63e00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b180 0x7f3c6c148d70 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c4a0 tx=0x7f3c5400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:06.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.717+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54005ce0 con 0x7f3c6c06b180 2024-01-24T17:50:06.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.718+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c54005e40 con 0x7f3c6c06b180 2024-01-24T17:50:06.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.718+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54007270 con 0x7f3c6c06b180 2024-01-24T17:50:06.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.718+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c15cb00 con 0x7f3c6c06b180 2024-01-24T17:50:06.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.718+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c15bfd0 con 0x7f3c6c06b180 2024-01-24T17:50:06.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.719+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3c6c0b4b70 con 0x7f3c6c06b180 2024-01-24T17:50:06.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.720+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3c54007490 con 0x7f3c6c06b180 2024-01-24T17:50:06.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.721+0000 7f3c62a00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c4406a430 0x7f3c4406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.721+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f3c54087b80 con 0x7f3c6c06b180 2024-01-24T17:50:06.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.721+0000 7f3c62a00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f3c44070510 0x7f3c44072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.721+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 --> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c44072fe0 con 0x7f3c44070510 2024-01-24T17:50:06.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.721+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c5408c2c0 con 0x7f3c6c06b180 2024-01-24T17:50:06.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.722+0000 7f3c6aa00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c4406a430 0x7f3c4406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.722+0000 7f3c6b400640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f3c44070510 0x7f3c44072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.723+0000 7f3c6aa00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c4406a430 0x7f3c4406c8f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c036000 tx=0x7f3c5c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:06.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.723+0000 7f3c6b400640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f3c44070510 0x7f3c44072930 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:06.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.725+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 <== osd.4 v2:172.21.15.100:6800/36832 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3c44072fe0 con 0x7f3c44070510 2024-01-24T17:50:06.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.739+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 --> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3c6c0b4890 con 0x7f3c44070510 2024-01-24T17:50:06.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.740+0000 7f3c62a00640 1 -- 172.21.15.39:0/1678944858 <== osd.4 v2:172.21.15.100:6800/36832 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f3c64001ea0 con 0x7f3c44070510 2024-01-24T17:50:06.741 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:50:06.741 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-24T17:50:06.741 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:50:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f3c44070510 msgr2=0x7f3c44072930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6800/36832,v1:172.21.15.100:6801/36832] conn(0x7f3c44070510 0x7f3c44072930 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c4406a430 msgr2=0x7f3c4406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f3c4406a430 0x7f3c4406c8f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c036000 tx=0x7f3c5c03b040 comp rx=0 tx=0).stop 2024-01-24T17:50:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b180 msgr2=0x7f3c6c148d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 --2- 172.21.15.39:0/1678944858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c6c06b180 0x7f3c6c148d70 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c4a0 tx=0x7f3c5400c960 comp rx=0 tx=0).stop 2024-01-24T17:50:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6b400640 1 -- 172.21.15.39:0/1678944858 reap_dead start 2024-01-24T17:50:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 shutdown_connections 2024-01-24T17:50:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 >> 172.21.15.39:0/1678944858 conn(0x7f3c6c0d2850 msgr2=0x7f3c6c15d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 shutdown_connections 2024-01-24T17:50:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.741+0000 7f3c6be00640 1 -- 172.21.15.39:0/1678944858 wait complete. 2024-01-24T17:50:06.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-24T17:50:06.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.988+0000 7f1a5fc00640 1 Processor -- start 2024-01-24T17:50:06.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.988+0000 7f1a5fc00640 1 -- start start 2024-01-24T17:50:06.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.989+0000 7f1a5fc00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f1a580b22b0 0x7f1a5806c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.989+0000 7f1a5fc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58161850 0x7f1a580b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.989+0000 7f1a5fc00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5806c650 con 0x7f1a580cefb0 2024-01-24T17:50:06.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.989+0000 7f1a5fc00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5806bcf0 con 0x7f1a580b22b0 2024-01-24T17:50:06.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.989+0000 7f1a5fc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580b2bd0 con 0x7f1a58161850 2024-01-24T17:50:06.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.990+0000 7f1a5dc00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f1a580cefb0 0x7f1a580cb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58262/0 (socket says 172.21.15.39:58262) 2024-01-24T17:50:06.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.990+0000 7f1a5dc00640 1 -- 172.21.15.39:0/3261312418 learned_addr learned my addr 172.21.15.39:0/3261312418 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.990+0000 7f1a5d200640 1 --2- 172.21.15.39:0/3261312418 >> v2:172.21.15.100:3300/0 conn(0x7f1a580b22b0 0x7f1a5806c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.990+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270753944 0 0) 0x7f1a5806c650 con 0x7f1a580cefb0 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.990+0000 7f1a57e00640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58161850 0x7f1a580b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.990+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a38003520 con 0x7f1a580cefb0 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 533803603 0 0) 0x7f1a38003520 con 0x7f1a580cefb0 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58161850 msgr2=0x7f1a580b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58161850 0x7f1a580b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 >> v2:172.21.15.100:3300/0 conn(0x7f1a580b22b0 msgr2=0x7f1a5806c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 --2- 172.21.15.39:0/3261312418 >> v2:172.21.15.100:3300/0 conn(0x7f1a580b22b0 0x7f1a5806c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5806b070 con 0x7f1a580cefb0 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1a500019f0 con 0x7f1a580cefb0 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 >> v1:172.21.15.39:6789/0 conn(0x7f1a580cefb0 legacy=0x7f1a580cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a380047c0 0x7f1a38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58161850 0x7f1a38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5806bcf0 con 0x7f1a38003970 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580b2bd0 con 0x7f1a380047c0 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3800b9d0 con 0x7f1a58161850 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.991+0000 7f1a5d200640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.992+0000 7f1a5dc00640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58161850 0x7f1a38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.992+0000 7f1a57e00640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a380047c0 0x7f1a38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.992+0000 7f1a5d200640 1 -- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58161850 msgr2=0x7f1a38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:06.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.992+0000 7f1a5d200640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58161850 0x7f1a38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.992+0000 7f1a5d200640 1 -- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a380047c0 msgr2=0x7f1a38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:06.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.992+0000 7f1a5d200640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a380047c0 0x7f1a38004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.992+0000 7f1a5d200640 1 -- 172.21.15.39:0/3261312418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a38003520 con 0x7f1a38003970 2024-01-24T17:50:06.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.992+0000 7f1a57e00640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a380047c0 0x7f1a38004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:06.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.993+0000 7f1a5d200640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a38003d90 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009af0 tx=0x7f1a380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65aa6cff190baf3d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:06.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.993+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4802c710 con 0x7f1a38003970 2024-01-24T17:50:06.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.993+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4802c870 con 0x7f1a38003970 2024-01-24T17:50:06.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.993+0000 7f1a57400640 1 -- 172.21.15.39:0/3261312418 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4802cc00 con 0x7f1a38003970 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.994+0000 7f1a5fc00640 1 -- 172.21.15.39:0/3261312418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 msgr2=0x7f1a38003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.994+0000 7f1a5fc00640 1 --2- 172.21.15.39:0/3261312418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a38003d90 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009af0 tx=0x7f1a380057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.994+0000 7f1a5dc00640 1 -- 172.21.15.39:0/3261312418 reap_dead start 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.994+0000 7f1a5fc00640 1 -- 172.21.15.39:0/3261312418 shutdown_connections 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.994+0000 7f1a5fc00640 1 -- 172.21.15.39:0/3261312418 >> 172.21.15.39:0/3261312418 conn(0x7f1a580cd2b0 msgr2=0x7f1a580cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.994+0000 7f1a5fc00640 1 -- 172.21.15.39:0/3261312418 shutdown_connections 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.994+0000 7f1a5fc00640 1 -- 172.21.15.39:0/3261312418 wait complete. 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.996+0000 7f1a5fc00640 1 Processor -- start 2024-01-24T17:50:06.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.997+0000 7f1a5fc00640 1 -- start start 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5fc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a580ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5fc00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58152c80 0x7f1a580adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5fc00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b5800 0x7f1a580b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5fc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580acf00 con 0x7f1a38003970 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5fc00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a581f3a30 con 0x7f1a580b5800 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5fc00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580b61c0 con 0x7f1a58152c80 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a580ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a580ac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56962/0 (socket says 172.21.15.39:56962) 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5d200640 1 -- 172.21.15.39:0/246395651 learned_addr learned my addr 172.21.15.39:0/246395651 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a57e00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58152c80 0x7f1a580adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:06.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5dc00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b5800 0x7f1a580b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5d200640 1 -- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58152c80 msgr2=0x7f1a580adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5d200640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58152c80 0x7f1a580adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5d200640 1 -- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b5800 msgr2=0x7f1a580b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5d200640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b5800 0x7f1a580b5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.998+0000 7f1a5d200640 1 -- 172.21.15.39:0/246395651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a580ac940 con 0x7f1a38003970 2024-01-24T17:50:07.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.999+0000 7f1a5d200640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a580ac2d0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f1a48037040 tx=0x7f1a48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.999+0000 7f1a54c00640 1 -- 172.21.15.39:0/246395651 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48003810 con 0x7f1a38003970 2024-01-24T17:50:07.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.999+0000 7f1a54c00640 1 -- 172.21.15.39:0/246395651 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a48003970 con 0x7f1a38003970 2024-01-24T17:50:07.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.999+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a580acc20 con 0x7f1a38003970 2024-01-24T17:50:07.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.999+0000 7f1a54c00640 1 -- 172.21.15.39:0/246395651 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a48003d20 con 0x7f1a38003970 2024-01-24T17:50:07.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:06.999+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a580b38d0 con 0x7f1a38003970 2024-01-24T17:50:07.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.000+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5806cd60 con 0x7f1a38003970 2024-01-24T17:50:07.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.001+0000 7f1a54c00640 1 -- 172.21.15.39:0/246395651 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f1a48040450 con 0x7f1a38003970 2024-01-24T17:50:07.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.002+0000 7f1a54c00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1a3006a430 0x7f1a3006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.002+0000 7f1a54c00640 1 -- 172.21.15.39:0/246395651 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f1a480aa710 con 0x7f1a38003970 2024-01-24T17:50:07.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.003+0000 7f1a57e00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1a3006a430 0x7f1a3006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.004+0000 7f1a57e00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1a3006a430 0x7f1a3006c8f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1a580c4240 tx=0x7f1a40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.006+0000 7f1a54c00640 1 -- 172.21.15.39:0/246395651 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f1a48075e90 con 0x7f1a38003970 2024-01-24T17:50:07.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.201+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a5806b630 con 0x7f1a3006a430 2024-01-24T17:50:07.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.202+0000 7f1a54c00640 1 -- 172.21.15.39:0/246395651 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1a400118b0 con 0x7f1a3006a430 2024-01-24T17:50:07.202 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 4 to deep-scrub 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1a3006a430 msgr2=0x7f1a3006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1a3006a430 0x7f1a3006c8f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1a580c4240 tx=0x7f1a40008040 comp rx=0 tx=0).stop 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 msgr2=0x7f1a580ac2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a580ac2d0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f1a48037040 tx=0x7f1a48004730 comp rx=0 tx=0).stop 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 shutdown_connections 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1a580b5800 0x7f1a580b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f1a58152c80 0x7f1a580adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f1a3006a430 0x7f1a3006c8f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 --2- 172.21.15.39:0/246395651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a38003970 0x7f1a580ac2d0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 >> 172.21.15.39:0/246395651 conn(0x7f1a580cd2b0 msgr2=0x7f1a580c6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 shutdown_connections 2024-01-24T17:50:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.205+0000 7f1a5fc00640 1 -- 172.21.15.39:0/246395651 wait complete. 2024-01-24T17:50:07.223 INFO:tasks.ceph:Scrubbing osd.5 2024-01-24T17:50:07.223 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-24T17:50:07.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.449+0000 7fec9c000640 1 Processor -- start 2024-01-24T17:50:07.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.450+0000 7fec9c000640 1 -- start start 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.451+0000 7fec9c000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec9406b6f0 0x7fec9406bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.451+0000 7fec9c000640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fec940d14c0 0x7fec940cf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.451+0000 7fec9c000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec941606e0 0x7fec94160b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.451+0000 7fec9c000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940cff70 con 0x7fec941606e0 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9c000640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9415d060 con 0x7fec940d14c0 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9c000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec940b0c10 con 0x7fec9406b6f0 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec941606e0 0x7fec94160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fec941606e0 0x7fec94160b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56966/0 (socket says 172.21.15.39:56966) 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9a000640 1 -- 172.21.15.39:0/2692358010 learned_addr learned my addr 172.21.15.39:0/2692358010 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec99600640 1 --2- 172.21.15.39:0/2692358010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec9406b6f0 0x7fec9406bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec98c00640 1 --2- 172.21.15.39:0/2692358010 >> v2:172.21.15.100:3300/0 conn(0x7fec940d14c0 0x7fec940cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9a000640 1 -- 172.21.15.39:0/2692358010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec9406b6f0 msgr2=0x7fec9406bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:07.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9a000640 1 --2- 172.21.15.39:0/2692358010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec9406b6f0 0x7fec9406bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9a000640 1 -- 172.21.15.39:0/2692358010 >> v2:172.21.15.100:3300/0 conn(0x7fec940d14c0 msgr2=0x7fec940cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:07.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9a000640 1 --2- 172.21.15.39:0/2692358010 >> v2:172.21.15.100:3300/0 conn(0x7fec940d14c0 0x7fec940cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.452+0000 7fec9a000640 1 -- 172.21.15.39:0/2692358010 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9406c010 con 0x7fec941606e0 2024-01-24T17:50:07.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.454+0000 7fec9a000640 1 --2- 172.21.15.39:0/2692358010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 0x7fec94160b70 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fec94071ca0 tx=0x7fec8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5abe6dc22d37a567 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.454+0000 7fec93e00640 1 -- 172.21.15.39:0/2692358010 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c02ca70 con 0x7fec941606e0 2024-01-24T17:50:07.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.454+0000 7fec93e00640 1 -- 172.21.15.39:0/2692358010 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c02cbd0 con 0x7fec941606e0 2024-01-24T17:50:07.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.454+0000 7fec93e00640 1 -- 172.21.15.39:0/2692358010 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c0343f0 con 0x7fec941606e0 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 -- 172.21.15.39:0/2692358010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 msgr2=0x7fec94160b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 --2- 172.21.15.39:0/2692358010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 0x7fec94160b70 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fec94071ca0 tx=0x7fec8c02bed0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 -- 172.21.15.39:0/2692358010 shutdown_connections 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 --2- 172.21.15.39:0/2692358010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 0x7fec94160b70 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 --2- 172.21.15.39:0/2692358010 >> v2:172.21.15.100:3300/0 conn(0x7fec940d14c0 0x7fec940cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 --2- 172.21.15.39:0/2692358010 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec9406b6f0 0x7fec9406bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 -- 172.21.15.39:0/2692358010 >> 172.21.15.39:0/2692358010 conn(0x7fec94099b10 msgr2=0x7fec940d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 -- 172.21.15.39:0/2692358010 shutdown_connections 2024-01-24T17:50:07.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.455+0000 7fec9c000640 1 -- 172.21.15.39:0/2692358010 wait complete. 2024-01-24T17:50:07.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.458+0000 7fec9c000640 1 Processor -- start 2024-01-24T17:50:07.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.458+0000 7fec9c000640 1 -- start start 2024-01-24T17:50:07.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9c000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec9406b6f0 0x7fec940cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9c000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec940d14c0 0x7fec9415d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 0x7fec940c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec94160170 con 0x7fec941606e0 2024-01-24T17:50:07.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9c000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec94160450 con 0x7fec9406b6f0 2024-01-24T17:50:07.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9c000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec940c5400 con 0x7fec940d14c0 2024-01-24T17:50:07.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 0x7fec940c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 0x7fec940c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56972/0 (socket says 172.21.15.39:56972) 2024-01-24T17:50:07.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.459+0000 7fec9a000640 1 -- 172.21.15.39:0/2837236580 learned_addr learned my addr 172.21.15.39:0/2837236580 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:07.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec99600640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec9406b6f0 0x7fec940cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec98c00640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec940d14c0 0x7fec9415d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec9a000640 1 -- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec940d14c0 msgr2=0x7fec9415d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec9a000640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec940d14c0 0x7fec9415d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec9a000640 1 -- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec9406b6f0 msgr2=0x7fec940cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec9a000640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fec9406b6f0 0x7fec940cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec9a000640 1 -- 172.21.15.39:0/2837236580 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c009b30 con 0x7fec941606e0 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec98c00640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fec940d14c0 0x7fec9415d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.460+0000 7fec9a000640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 0x7fec940c5ee0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fec8c036040 tx=0x7fec8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.461+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c009510 con 0x7fec941606e0 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.461+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8c03e450 con 0x7fec941606e0 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.461+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8c03e800 con 0x7fec941606e0 2024-01-24T17:50:07.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.461+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec940c56e0 con 0x7fec941606e0 2024-01-24T17:50:07.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.461+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec940cb720 con 0x7fec941606e0 2024-01-24T17:50:07.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.462+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec940b05c0 con 0x7fec941606e0 2024-01-24T17:50:07.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.463+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fec8c03e960 con 0x7fec941606e0 2024-01-24T17:50:07.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.464+0000 7fec91600640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec6c06a460 0x7fec6c06c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.464+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fec8c0aba00 con 0x7fec941606e0 2024-01-24T17:50:07.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.464+0000 7fec91600640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7fec6c070540 0x7fec6c072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.464+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 --> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec6c073010 con 0x7fec6c070540 2024-01-24T17:50:07.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.464+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec8c0abe50 con 0x7fec941606e0 2024-01-24T17:50:07.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.465+0000 7fec99600640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec6c06a460 0x7fec6c06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.465+0000 7fec98c00640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7fec6c070540 0x7fec6c072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.466+0000 7fec98c00640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7fec6c070540 0x7fec6c072960 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.466+0000 7fec99600640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec6c06a460 0x7fec6c06c920 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fec84006fd0 tx=0x7fec84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.468+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 <== osd.5 v2:172.21.15.100:6816/36844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fec6c073010 con 0x7fec6c070540 2024-01-24T17:50:07.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.482+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 --> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fec940b02e0 con 0x7fec6c070540 2024-01-24T17:50:07.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec91600640 1 -- 172.21.15.39:0/2837236580 <== osd.5 v2:172.21.15.100:6816/36844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fec7c009550 con 0x7fec6c070540 2024-01-24T17:50:07.483 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:50:07.484 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-24T17:50:07.484 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7fec6c070540 msgr2=0x7fec6c072960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9c000640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6816/36844,v1:172.21.15.100:6817/36844] conn(0x7fec6c070540 0x7fec6c072960 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec6c06a460 msgr2=0x7fec6c06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9c000640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fec6c06a460 0x7fec6c06c920 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fec84006fd0 tx=0x7fec84008040 comp rx=0 tx=0).stop 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 msgr2=0x7fec940c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9c000640 1 --2- 172.21.15.39:0/2837236580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec941606e0 0x7fec940c5ee0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fec8c036040 tx=0x7fec8c004a40 comp rx=0 tx=0).stop 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9a000640 1 -- 172.21.15.39:0/2837236580 reap_dead start 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 shutdown_connections 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.483+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 >> 172.21.15.39:0/2837236580 conn(0x7fec94099b10 msgr2=0x7fec94160af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.484+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 shutdown_connections 2024-01-24T17:50:07.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.484+0000 7fec9c000640 1 -- 172.21.15.39:0/2837236580 wait complete. 2024-01-24T17:50:07.503 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-24T17:50:07.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.718+0000 7f13d9a00640 1 Processor -- start 2024-01-24T17:50:07.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.718+0000 7f13d9a00640 1 -- start start 2024-01-24T17:50:07.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.719+0000 7f13d9a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d40b0010 0x7f13d415bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.720+0000 7f13d9a00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d4070470 con 0x7f13d415f390 2024-01-24T17:50:07.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.720+0000 7f13d9a00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d415ef00 con 0x7f13d4070920 2024-01-24T17:50:07.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.720+0000 7f13d9a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40710a0 con 0x7f13d40b0010 2024-01-24T17:50:07.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.720+0000 7f13d3400640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f13d415f390 0x7f13d40c9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58264/0 (socket says 172.21.15.39:58264) 2024-01-24T17:50:07.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.720+0000 7f13d3400640 1 -- 172.21.15.39:0/1497905594 learned_addr learned my addr 172.21.15.39:0/1497905594 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:07.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.720+0000 7f13d2a00640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d40b0010 0x7f13d415bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.720+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543190362 0 0) 0x7f13d4070470 con 0x7f13d415f390 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b4003520 con 0x7f13d415f390 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659698592 0 0) 0x7f13b4003520 con 0x7f13d415f390 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d40b0010 msgr2=0x7f13d415bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d40b0010 0x7f13d415bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 >> v1:172.21.15.100:6789/0 conn(0x7f13d4070920 legacy=0x7f13d40b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d415c0d0 con 0x7f13d415f390 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f13cc0019f0 con 0x7f13d415f390 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 >> v1:172.21.15.39:6789/0 conn(0x7f13d415f390 legacy=0x7f13d40c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b40039a0 0x7f13b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d40b0010 0x7f13b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b4004d00 0x7f13b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40710a0 con 0x7f13b40039a0 2024-01-24T17:50:07.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b400baf0 con 0x7f13b4004d00 2024-01-24T17:50:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b400bcb0 con 0x7f13d40b0010 2024-01-24T17:50:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.721+0000 7f13d2a00640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b40039a0 0x7f13b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.722+0000 7f13cbe00640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d40b0010 0x7f13b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.722+0000 7f13d2a00640 1 -- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d40b0010 msgr2=0x7f13b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.722+0000 7f13d2a00640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d40b0010 0x7f13b40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.722+0000 7f13d2a00640 1 -- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b4004d00 msgr2=0x7f13b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.722+0000 7f13d2a00640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b4004d00 0x7f13b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.722+0000 7f13d2a00640 1 -- 172.21.15.39:0/1497905594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4003520 con 0x7f13b40039a0 2024-01-24T17:50:07.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.723+0000 7f13d2a00640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b40039a0 0x7f13b4003dc0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009a90 tx=0x7f13b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d2043be8e24f7d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.723+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c402c680 con 0x7f13b40039a0 2024-01-24T17:50:07.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.723+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c402c7e0 con 0x7f13b40039a0 2024-01-24T17:50:07.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.723+0000 7f13d2000640 1 -- 172.21.15.39:0/1497905594 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c402cb70 con 0x7f13b40039a0 2024-01-24T17:50:07.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.724+0000 7f13d9a00640 1 -- 172.21.15.39:0/1497905594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b40039a0 msgr2=0x7f13b4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.724+0000 7f13d9a00640 1 --2- 172.21.15.39:0/1497905594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13b40039a0 0x7f13b4003dc0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009a90 tx=0x7f13b40057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.724+0000 7f13d3400640 1 -- 172.21.15.39:0/1497905594 reap_dead start 2024-01-24T17:50:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.724+0000 7f13d9a00640 1 -- 172.21.15.39:0/1497905594 shutdown_connections 2024-01-24T17:50:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.724+0000 7f13d9a00640 1 -- 172.21.15.39:0/1497905594 >> 172.21.15.39:0/1497905594 conn(0x7f13d406bf20 msgr2=0x7f13d406c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.724+0000 7f13d9a00640 1 -- 172.21.15.39:0/1497905594 shutdown_connections 2024-01-24T17:50:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.724+0000 7f13d9a00640 1 -- 172.21.15.39:0/1497905594 wait complete. 2024-01-24T17:50:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.726+0000 7f13d9a00640 1 Processor -- start 2024-01-24T17:50:07.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.726+0000 7f13d9a00640 1 -- start start 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13d9a00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b40039a0 0x7f13d41f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13d9a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d40c46e0 0x7f13d414c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13d9a00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d4159d30 0x7f13d415a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13d9a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d41f63d0 con 0x7f13d40c46e0 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13d9a00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d415a6d0 con 0x7f13b40039a0 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13d9a00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40c4f60 con 0x7f13d4159d30 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d40c46e0 0x7f13d414c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13cbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d40c46e0 0x7f13d414c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56992/0 (socket says 172.21.15.39:56992) 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.727+0000 7f13cbe00640 1 -- 172.21.15.39:0/3089113785 learned_addr learned my addr 172.21.15.39:0/3089113785 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:07.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13d3400640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d4159d30 0x7f13d415a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13cbe00640 1 -- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d4159d30 msgr2=0x7f13d415a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13cbe00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d4159d30 0x7f13d415a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13d2a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b40039a0 0x7f13d41f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13cbe00640 1 -- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b40039a0 msgr2=0x7f13d41f5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13cbe00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b40039a0 0x7f13d41f5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13cbe00640 1 -- 172.21.15.39:0/3089113785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d40cc370 con 0x7f13d40c46e0 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13cbe00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d40c46e0 0x7f13d414c980 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00d5a0 tx=0x7f13bc00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13caa00640 1 -- 172.21.15.39:0/3089113785 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc005480 con 0x7f13d40c46e0 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13caa00640 1 -- 172.21.15.39:0/3089113785 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13bc003610 con 0x7f13d40c46e0 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13caa00640 1 -- 172.21.15.39:0/3089113785 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13bc003980 con 0x7f13d40c46e0 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.728+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d41f6600 con 0x7f13d40c46e0 2024-01-24T17:50:07.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.729+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d41f6bc0 con 0x7f13d40c46e0 2024-01-24T17:50:07.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.730+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d40cc500 con 0x7f13d40c46e0 2024-01-24T17:50:07.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.731+0000 7f13caa00640 1 -- 172.21.15.39:0/3089113785 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f13bc003ae0 con 0x7f13d40c46e0 2024-01-24T17:50:07.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.732+0000 7f13caa00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13ac06a430 0x7f13ac06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:07.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.732+0000 7f13caa00640 1 -- 172.21.15.39:0/3089113785 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f13bc088630 con 0x7f13d40c46e0 2024-01-24T17:50:07.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.732+0000 7f13d2a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13ac06a430 0x7f13ac06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:07.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.734+0000 7f13d2a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13ac06a430 0x7f13ac06c8f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f13c4036000 tx=0x7f13c4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.736+0000 7f13caa00640 1 -- 172.21.15.39:0/3089113785 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f13bc053db0 con 0x7f13d40c46e0 2024-01-24T17:50:07.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.930+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f13d415e8b0 con 0x7f13ac06a430 2024-01-24T17:50:07.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.931+0000 7f13caa00640 1 -- 172.21.15.39:0/3089113785 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f13c4004020 con 0x7f13ac06a430 2024-01-24T17:50:07.932 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 5 to deep-scrub 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13ac06a430 msgr2=0x7f13ac06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13ac06a430 0x7f13ac06c8f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f13c4036000 tx=0x7f13c4030000 comp rx=0 tx=0).stop 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d40c46e0 msgr2=0x7f13d414c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d40c46e0 0x7f13d414c980 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00d5a0 tx=0x7f13bc00da60 comp rx=0 tx=0).stop 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 shutdown_connections 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f13d4159d30 0x7f13d415a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f13ac06a430 0x7f13ac06c8f0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13d40c46e0 0x7f13d414c980 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 --2- 172.21.15.39:0/3089113785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f13b40039a0 0x7f13d41f5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:07.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 >> 172.21.15.39:0/3089113785 conn(0x7f13d406bf20 msgr2=0x7f13d40ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 shutdown_connections 2024-01-24T17:50:07.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:07.934+0000 7f13d9a00640 1 -- 172.21.15.39:0/3089113785 wait complete. 2024-01-24T17:50:07.953 INFO:tasks.ceph:Scrubbing osd.6 2024-01-24T17:50:07.953 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-24T17:50:08.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.172+0000 7f1742c00640 1 Processor -- start 2024-01-24T17:50:08.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.172+0000 7f1742c00640 1 -- start start 2024-01-24T17:50:08.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.174+0000 7f1742c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f173c06bdb0 0x7f173c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.174+0000 7f1742c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f173c06c190 0x7f173c0b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.174+0000 7f1742c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173c0ca830 con 0x7f173c159830 2024-01-24T17:50:08.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.174+0000 7f1742c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173c0ca270 con 0x7f173c06c190 2024-01-24T17:50:08.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.174+0000 7f1742c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0ca550 con 0x7f173c06bdb0 2024-01-24T17:50:08.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.174+0000 7f1740e00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f173c159830 0x7f173c0cca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58280/0 (socket says 172.21.15.39:58280) 2024-01-24T17:50:08.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f1740e00640 1 -- 172.21.15.39:0/2825224081 learned_addr learned my addr 172.21.15.39:0/2825224081 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:08.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f1741800640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f173c06bdb0 0x7f173c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f1742200640 1 --2- 172.21.15.39:0/2825224081 >> v2:172.21.15.100:3300/0 conn(0x7f173c06c190 0x7f173c0b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160210787 0 0) 0x7f173c0ca830 con 0x7f173c159830 2024-01-24T17:50:08.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f171c003520 con 0x7f173c159830 2024-01-24T17:50:08.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009379616 0 0) 0x7f171c003520 con 0x7f173c159830 2024-01-24T17:50:08.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f173c06bdb0 msgr2=0x7f173c0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:08.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f173c06bdb0 0x7f173c0c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 >> v2:172.21.15.100:3300/0 conn(0x7f173c06c190 msgr2=0x7f173c0b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 --2- 172.21.15.39:0/2825224081 >> v2:172.21.15.100:3300/0 conn(0x7f173c06c190 0x7f173c0b11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c070e80 con 0x7f173c159830 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.175+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7f1724001670 con 0x7f173c159830 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 >> v1:172.21.15.39:6789/0 conn(0x7f173c159830 legacy=0x7f173c0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f173be00640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f173c06bdb0 0x7f171c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f173be00640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171c003ef0 0x7f171c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f173be00640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171c004d00 0x7f171c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0ca270 con 0x7f171c004d00 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0ca550 con 0x7f171c003ef0 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c00b980 con 0x7f173c06bdb0 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f1742200640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171c004d00 0x7f171c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f1741800640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f173c06bdb0 0x7f171c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f1740e00640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171c003ef0 0x7f171c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f1742200640 1 -- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f173c06bdb0 msgr2=0x7f171c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:08.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f1742200640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f173c06bdb0 0x7f171c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f1742200640 1 -- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171c003ef0 msgr2=0x7f171c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:08.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f1742200640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171c003ef0 0x7f171c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.176+0000 7f1742200640 1 -- 172.21.15.39:0/2825224081 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c003520 con 0x7f171c004d00 2024-01-24T17:50:08.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.177+0000 7f1740e00640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f171c003ef0 0x7f171c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.178+0000 7f1742200640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171c004d00 0x7f171c005180 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f1734009fc0 tx=0x7f171c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39939b2e2d14654d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.178+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173402ee50 con 0x7f171c004d00 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.178+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173400f040 con 0x7f171c004d00 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.178+0000 7f173be00640 1 -- 172.21.15.39:0/2825224081 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734036680 con 0x7f171c004d00 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.179+0000 7f1742c00640 1 -- 172.21.15.39:0/2825224081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171c004d00 msgr2=0x7f171c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.179+0000 7f1742c00640 1 --2- 172.21.15.39:0/2825224081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f171c004d00 0x7f171c005180 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f1734009fc0 tx=0x7f171c0057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.179+0000 7f1742200640 1 -- 172.21.15.39:0/2825224081 reap_dead start 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.179+0000 7f1742c00640 1 -- 172.21.15.39:0/2825224081 shutdown_connections 2024-01-24T17:50:08.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.179+0000 7f1742c00640 1 -- 172.21.15.39:0/2825224081 >> 172.21.15.39:0/2825224081 conn(0x7f173c06a310 msgr2=0x7f173c0cadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:08.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.179+0000 7f1742c00640 1 -- 172.21.15.39:0/2825224081 shutdown_connections 2024-01-24T17:50:08.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.179+0000 7f1742c00640 1 -- 172.21.15.39:0/2825224081 wait complete. 2024-01-24T17:50:08.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.181+0000 7f1742c00640 1 Processor -- start 2024-01-24T17:50:08.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.181+0000 7f1742c00640 1 -- start start 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f171c004d00 0x7f173c14d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f173c0c3f70 0x7f173c15b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f173c15b890 0x7f173c0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c0c3b50 con 0x7f173c15b890 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c15bd10 con 0x7f173c0c3f70 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173c14dd20 con 0x7f171c004d00 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f173c15b890 0x7f173c0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f173c15b890 0x7f173c0c5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57004/0 (socket says 172.21.15.39:57004) 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742200640 1 -- 172.21.15.39:0/1472128196 learned_addr learned my addr 172.21.15.39:0/1472128196 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1741800640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f171c004d00 0x7f173c14d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1740e00640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f173c0c3f70 0x7f173c15b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742200640 1 -- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f171c004d00 msgr2=0x7f173c14d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.182+0000 7f1742200640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f171c004d00 0x7f173c14d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1742200640 1 -- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f173c0c3f70 msgr2=0x7f173c15b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1742200640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f173c0c3f70 0x7f173c15b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1742200640 1 -- 172.21.15.39:0/1472128196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c0c2ed0 con 0x7f173c15b890 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1741800640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f171c004d00 0x7f173c14d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1742200640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f173c15b890 0x7f173c0c5720 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f1734039040 tx=0x7f1734009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734043070 con 0x7f173c15b890 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1734036b50 con 0x7f173c15b890 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734003990 con 0x7f173c15b890 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173c0c31b0 con 0x7f173c15b890 2024-01-24T17:50:08.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.183+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173c0c3770 con 0x7f173c15b890 2024-01-24T17:50:08.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.184+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f173c06c600 con 0x7f173c15b890 2024-01-24T17:50:08.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.185+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f173403d070 con 0x7f173c15b890 2024-01-24T17:50:08.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.186+0000 7f1739600640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f171406a460 0x7f171406c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.186+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f17340a9ef0 con 0x7f173c15b890 2024-01-24T17:50:08.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.186+0000 7f1739600640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f1714070540 0x7f1714072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.187+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1714073010 con 0x7f1714070540 2024-01-24T17:50:08.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.187+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17340ac5b0 con 0x7f173c15b890 2024-01-24T17:50:08.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.187+0000 7f1741800640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f171406a460 0x7f171406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.187+0000 7f1740e00640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f1714070540 0x7f1714072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.188+0000 7f1741800640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f171406a460 0x7f171406c920 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f172c009a00 tx=0x7f172c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.189+0000 7f1740e00640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f1714070540 0x7f1714072960 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.191+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 <== osd.6 v2:172.21.15.100:6808/36833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1714073010 con 0x7f1714070540 2024-01-24T17:50:08.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.205+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 --> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f173c071160 con 0x7f1714070540 2024-01-24T17:50:08.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.206+0000 7f1739600640 1 -- 172.21.15.39:0/1472128196 <== osd.6 v2:172.21.15.100:6808/36833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f172400d040 con 0x7f1714070540 2024-01-24T17:50:08.206 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:50:08.207 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-24T17:50:08.207 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.206+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f1714070540 msgr2=0x7f1714072960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.206+0000 7f1742c00640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6808/36833,v1:172.21.15.100:6809/36833] conn(0x7f1714070540 0x7f1714072960 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f171406a460 msgr2=0x7f171406c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742c00640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f171406a460 0x7f171406c920 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f172c009a00 tx=0x7f172c008040 comp rx=0 tx=0).stop 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f173c15b890 msgr2=0x7f173c0c5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742c00640 1 --2- 172.21.15.39:0/1472128196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f173c15b890 0x7f173c0c5720 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f1734039040 tx=0x7f1734009750 comp rx=0 tx=0).stop 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742200640 1 -- 172.21.15.39:0/1472128196 reap_dead start 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 shutdown_connections 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 >> 172.21.15.39:0/1472128196 conn(0x7f173c06a310 msgr2=0x7f173c0cc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 shutdown_connections 2024-01-24T17:50:08.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.207+0000 7f1742c00640 1 -- 172.21.15.39:0/1472128196 wait complete. 2024-01-24T17:50:08.228 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-24T17:50:08.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.439+0000 7f2e0d200640 1 Processor -- start 2024-01-24T17:50:08.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.439+0000 7f2e0d200640 1 -- start start 2024-01-24T17:50:08.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.440+0000 7f2e0d200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2e08071c00 0x7f2e080cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.440+0000 7f2e0d200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e080b3f20 0x7f2e080ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.440+0000 7f2e0d200640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e080ccd10 con 0x7f2e08159d00 2024-01-24T17:50:08.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.440+0000 7f2e0d200640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e080ce230 con 0x7f2e08071c00 2024-01-24T17:50:08.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.440+0000 7f2e0d200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080ccff0 con 0x7f2e080b3f20 2024-01-24T17:50:08.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e05400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2e08071c00 0x7f2e080cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e05400640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f2e08071c00 0x7f2e080cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57020/0 (socket says 172.21.15.39:57020) 2024-01-24T17:50:08.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e05400640 1 -- 172.21.15.39:0/1206317585 learned_addr learned my addr 172.21.15.39:0/1206317585 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e06800640 1 --2- 172.21.15.39:0/1206317585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e080b3f20 0x7f2e080ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e05400640 1 -- 172.21.15.39:0/1206317585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e080b3f20 msgr2=0x7f2e080ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e05400640 1 --2- 172.21.15.39:0/1206317585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e080b3f20 0x7f2e080ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e05400640 1 -- 172.21.15.39:0/1206317585 >> v1:172.21.15.39:6789/0 conn(0x7f2e08159d00 legacy=0x7f2e080714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e05400640 1 -- 172.21.15.39:0/1206317585 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e080ad720 con 0x7f2e08071c00 2024-01-24T17:50:08.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.441+0000 7f2e04a00640 1 -- 172.21.15.39:0/1206317585 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133051348 0 0) 0x7f2e080ccd10 con 0x7f2e08159d00 2024-01-24T17:50:08.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.442+0000 7f2e05400640 1 --2- 172.21.15.39:0/1206317585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e08071c00 0x7f2e080cdcf0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009870 tx=0x7f2df002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2faefdf5548abdad server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e04a00640 1 -- 172.21.15.39:0/1206317585 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0030840 con 0x7f2e08071c00 2024-01-24T17:50:08.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e04a00640 1 -- 172.21.15.39:0/1206317585 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df00309a0 con 0x7f2e08071c00 2024-01-24T17:50:08.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e04a00640 1 -- 172.21.15.39:0/1206317585 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0030d30 con 0x7f2e08071c00 2024-01-24T17:50:08.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e0d200640 1 -- 172.21.15.39:0/1206317585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e08071c00 msgr2=0x7f2e080cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e0d200640 1 --2- 172.21.15.39:0/1206317585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e08071c00 0x7f2e080cdcf0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009870 tx=0x7f2df002b980 comp rx=0 tx=0).stop 2024-01-24T17:50:08.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e0d200640 1 -- 172.21.15.39:0/1206317585 shutdown_connections 2024-01-24T17:50:08.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e0d200640 1 --2- 172.21.15.39:0/1206317585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e08071c00 0x7f2e080cdcf0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e0d200640 1 --2- 172.21.15.39:0/1206317585 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e080b3f20 0x7f2e080ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e0d200640 1 -- 172.21.15.39:0/1206317585 >> 172.21.15.39:0/1206317585 conn(0x7f2e0806bf20 msgr2=0x7f2e0806c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:08.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e0d200640 1 -- 172.21.15.39:0/1206317585 shutdown_connections 2024-01-24T17:50:08.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.443+0000 7f2e0d200640 1 -- 172.21.15.39:0/1206317585 wait complete. 2024-01-24T17:50:08.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.445+0000 7f2e0d200640 1 Processor -- start 2024-01-24T17:50:08.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.445+0000 7f2e0d200640 1 -- start start 2024-01-24T17:50:08.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.446+0000 7f2e0d200640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e08071c00 0x7f2e08155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.446+0000 7f2e0d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b3f20 0x7f2e0815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.446+0000 7f2e0d200640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e08159d00 0x7f2e0815c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.446+0000 7f2e0d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e08155eb0 con 0x7f2e080b3f20 2024-01-24T17:50:08.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.446+0000 7f2e0d200640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080c7860 con 0x7f2e08159d00 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.446+0000 7f2e0d200640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e080c7b40 con 0x7f2e08071c00 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.446+0000 7f2e05400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b3f20 0x7f2e0815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b3f20 0x7f2e0815c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57032/0 (socket says 172.21.15.39:57032) 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05400640 1 -- 172.21.15.39:0/4139073213 learned_addr learned my addr 172.21.15.39:0/4139073213 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05e00640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e08071c00 0x7f2e08155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05400640 1 -- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e08071c00 msgr2=0x7f2e08155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05400640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e08071c00 0x7f2e08155970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05400640 1 -- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e08159d00 msgr2=0x7f2e0815c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05400640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e08159d00 0x7f2e0815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05400640 1 -- 172.21.15.39:0/4139073213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df00094d0 con 0x7f2e080b3f20 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.447+0000 7f2e05400640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b3f20 0x7f2e0815c120 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f2df0000c00 tx=0x7f2df0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.448+0000 7f2dfe000640 1 -- 172.21.15.39:0/4139073213 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0039c90 con 0x7f2e080b3f20 2024-01-24T17:50:08.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.448+0000 7f2dfe000640 1 -- 172.21.15.39:0/4139073213 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df0039df0 con 0x7f2e080b3f20 2024-01-24T17:50:08.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.448+0000 7f2dfe000640 1 -- 172.21.15.39:0/4139073213 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0043640 con 0x7f2e080b3f20 2024-01-24T17:50:08.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.448+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e080c7e20 con 0x7f2e080b3f20 2024-01-24T17:50:08.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.448+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e080c83e0 con 0x7f2e080b3f20 2024-01-24T17:50:08.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.449+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0806fff0 con 0x7f2e080b3f20 2024-01-24T17:50:08.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.450+0000 7f2dfe000640 1 -- 172.21.15.39:0/4139073213 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f2df00437a0 con 0x7f2e080b3f20 2024-01-24T17:50:08.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.451+0000 7f2dfe000640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2de006a430 0x7f2de006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.451+0000 7f2dfe000640 1 -- 172.21.15.39:0/4139073213 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f2df003e070 con 0x7f2e080b3f20 2024-01-24T17:50:08.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.452+0000 7f2e05e00640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2de006a430 0x7f2de006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.453+0000 7f2e05e00640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2de006a430 0x7f2de006c8f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2df8000f60 tx=0x7f2df8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.456+0000 7f2dfe000640 1 -- 172.21.15.39:0/4139073213 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f2df0076ff0 con 0x7f2e080b3f20 2024-01-24T17:50:08.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.648+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e080adce0 con 0x7f2de006a430 2024-01-24T17:50:08.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.649+0000 7f2dfe000640 1 -- 172.21.15.39:0/4139073213 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2df80050d0 con 0x7f2de006a430 2024-01-24T17:50:08.650 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 6 to deep-scrub 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2de006a430 msgr2=0x7f2de006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2de006a430 0x7f2de006c8f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2df8000f60 tx=0x7f2df8009040 comp rx=0 tx=0).stop 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b3f20 msgr2=0x7f2e0815c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b3f20 0x7f2e0815c120 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f2df0000c00 tx=0x7f2df0004730 comp rx=0 tx=0).stop 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 shutdown_connections 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e08159d00 0x7f2e0815c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f2de006a430 0x7f2de006c8f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b3f20 0x7f2e0815c120 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 --2- 172.21.15.39:0/4139073213 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f2e08071c00 0x7f2e08155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 >> 172.21.15.39:0/4139073213 conn(0x7f2e0806bf20 msgr2=0x7f2e08157ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:08.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 shutdown_connections 2024-01-24T17:50:08.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.652+0000 7f2e0d200640 1 -- 172.21.15.39:0/4139073213 wait complete. 2024-01-24T17:50:08.672 INFO:tasks.ceph:Scrubbing osd.7 2024-01-24T17:50:08.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-24T17:50:08.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.902+0000 7f49d5000640 1 Processor -- start 2024-01-24T17:50:08.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.902+0000 7f49d5000640 1 -- start start 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.903+0000 7f49d5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0071c00 0x7f49d00cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.903+0000 7f49d5000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49d00b3f20 0x7f49d00ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.903+0000 7f49d5000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d00ccd10 con 0x7f49d0159d00 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.903+0000 7f49d5000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d00ce230 con 0x7f49d00b3f20 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.903+0000 7f49d5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00ccff0 con 0x7f49d0071c00 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49ce800640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f49d00b3f20 0x7f49d00ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49cde00640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f49d0159d00 0x7f49d00714f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58292/0 (socket says 172.21.15.39:58292) 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49cde00640 1 -- 172.21.15.39:0/1420775365 learned_addr learned my addr 172.21.15.39:0/1420775365 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:08.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49c5200640 1 --2- 172.21.15.39:0/1420775365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0071c00 0x7f49d00cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49ce800640 1 -- 172.21.15.39:0/1420775365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0071c00 msgr2=0x7f49d00cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:08.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49ce800640 1 --2- 172.21.15.39:0/1420775365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0071c00 0x7f49d00cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49ce800640 1 -- 172.21.15.39:0/1420775365 >> v1:172.21.15.39:6789/0 conn(0x7f49d0159d00 legacy=0x7f49d00714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49ce800640 1 -- 172.21.15.39:0/1420775365 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d00ad720 con 0x7f49d00b3f20 2024-01-24T17:50:08.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.904+0000 7f49c5200640 1 --2- 172.21.15.39:0/1420775365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0071c00 0x7f49d00cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:08.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.906+0000 7f49ce800640 1 --2- 172.21.15.39:0/1420775365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 0x7f49d00ca2f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009d90 tx=0x7f49c802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f245af8630fa1961 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.906+0000 7f49cd400640 1 -- 172.21.15.39:0/1420775365 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c802cc50 con 0x7f49d00b3f20 2024-01-24T17:50:08.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.906+0000 7f49cd400640 1 -- 172.21.15.39:0/1420775365 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c802cdb0 con 0x7f49d00b3f20 2024-01-24T17:50:08.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.906+0000 7f49cd400640 1 -- 172.21.15.39:0/1420775365 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8034660 con 0x7f49d00b3f20 2024-01-24T17:50:08.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.906+0000 7f49d5000640 1 -- 172.21.15.39:0/1420775365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 msgr2=0x7f49d00ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.906+0000 7f49d5000640 1 --2- 172.21.15.39:0/1420775365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 0x7f49d00ca2f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009d90 tx=0x7f49c802bea0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.907+0000 7f49d5000640 1 -- 172.21.15.39:0/1420775365 shutdown_connections 2024-01-24T17:50:08.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.907+0000 7f49d5000640 1 --2- 172.21.15.39:0/1420775365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 0x7f49d00ca2f0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.907+0000 7f49d5000640 1 --2- 172.21.15.39:0/1420775365 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0071c00 0x7f49d00cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.907+0000 7f49d5000640 1 -- 172.21.15.39:0/1420775365 >> 172.21.15.39:0/1420775365 conn(0x7f49d006bf20 msgr2=0x7f49d006c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:08.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.907+0000 7f49d5000640 1 -- 172.21.15.39:0/1420775365 shutdown_connections 2024-01-24T17:50:08.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.907+0000 7f49d5000640 1 -- 172.21.15.39:0/1420775365 wait complete. 2024-01-24T17:50:08.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.909+0000 7f49d5000640 1 Processor -- start 2024-01-24T17:50:08.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.909+0000 7f49d5000640 1 -- start start 2024-01-24T17:50:08.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49d5000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49d0071c00 0x7f49d015bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49d5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 0x7f49d0152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49d5000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0159d00 0x7f49d00c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49d5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d00c59e0 con 0x7f49d00b3f20 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49d5000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d015cfd0 con 0x7f49d0071c00 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49d5000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d015a3a0 con 0x7f49d0159d00 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49c5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 0x7f49d0152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49c5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 0x7f49d0152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57054/0 (socket says 172.21.15.39:57054) 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49c5200640 1 -- 172.21.15.39:0/4040571473 learned_addr learned my addr 172.21.15.39:0/4040571473 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49cde00640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49d0071c00 0x7f49d015bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.910+0000 7f49ce800640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0159d00 0x7f49d00c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c5200640 1 -- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0159d00 msgr2=0x7f49d00c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c5200640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f49d0159d00 0x7f49d00c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c5200640 1 -- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49d0071c00 msgr2=0x7f49d015bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c5200640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f49d0071c00 0x7f49d015bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c5200640 1 -- 172.21.15.39:0/4040571473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8009a70 con 0x7f49d00b3f20 2024-01-24T17:50:08.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c5200640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 0x7f49d0152760 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f49b800c7f0 tx=0x7f49b800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8007a30 con 0x7f49d00b3f20 2024-01-24T17:50:08.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b8007b90 con 0x7f49d00b3f20 2024-01-24T17:50:08.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d015c3c0 con 0x7f49d00b3f20 2024-01-24T17:50:08.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b800ce40 con 0x7f49d00b3f20 2024-01-24T17:50:08.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.911+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d015c980 con 0x7f49d00b3f20 2024-01-24T17:50:08.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.912+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f49d00adce0 con 0x7f49d00b3f20 2024-01-24T17:50:08.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.913+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f49b80113f0 con 0x7f49d00b3f20 2024-01-24T17:50:08.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.914+0000 7f49c6a00640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49a806a430 0x7f49a806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.914+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f49b80888b0 con 0x7f49d00b3f20 2024-01-24T17:50:08.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.915+0000 7f49c6a00640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7f49a8070510 0x7f49a8072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:08.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.915+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 --> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49a8072fe0 con 0x7f49a8070510 2024-01-24T17:50:08.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.915+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49b808d2c0 con 0x7f49d00b3f20 2024-01-24T17:50:08.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.915+0000 7f49cde00640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49a806a430 0x7f49a806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.916+0000 7f49ce800640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7f49a8070510 0x7f49a8072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:08.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.916+0000 7f49cde00640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49a806a430 0x7f49a806c8f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f49c0005880 tx=0x7f49c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.917+0000 7f49ce800640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7f49a8070510 0x7f49a8072930 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:08.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.919+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 <== osd.7 v2:172.21.15.100:6820/36857 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f49a8072fe0 con 0x7f49a8070510 2024-01-24T17:50:08.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.933+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 --> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f49d00ada00 con 0x7f49a8070510 2024-01-24T17:50:08.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49c6a00640 1 -- 172.21.15.39:0/4040571473 <== osd.7 v2:172.21.15.100:6820/36857 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f49c8009450 con 0x7f49a8070510 2024-01-24T17:50:08.934 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-01-24T17:50:08.935 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-01-24T17:50:08.935 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7f49a8070510 msgr2=0x7f49a8072930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49d5000640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6820/36857,v1:172.21.15.100:6822/36857] conn(0x7f49a8070510 0x7f49a8072930 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49a806a430 msgr2=0x7f49a806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49d5000640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f49a806a430 0x7f49a806c8f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f49c0005880 tx=0x7f49c0009040 comp rx=0 tx=0).stop 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 msgr2=0x7f49d0152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49d5000640 1 --2- 172.21.15.39:0/4040571473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49d00b3f20 0x7f49d0152760 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f49b800c7f0 tx=0x7f49b800cbf0 comp rx=0 tx=0).stop 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49ce800640 1 -- 172.21.15.39:0/4040571473 reap_dead start 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 shutdown_connections 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.934+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 >> 172.21.15.39:0/4040571473 conn(0x7f49d006bf20 msgr2=0x7f49d00c20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:08.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.935+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 shutdown_connections 2024-01-24T17:50:08.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:08.935+0000 7f49d5000640 1 -- 172.21.15.39:0/4040571473 wait complete. 2024-01-24T17:50:08.951 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-24T17:50:09.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.168+0000 7fb995600640 1 Processor -- start 2024-01-24T17:50:09.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.169+0000 7fb995600640 1 -- start start 2024-01-24T17:50:09.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.170+0000 7fb995600640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fb9900d12e0 0x7fb990168270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.170+0000 7fb995600640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9900b8820 con 0x7fb9900d0e90 2024-01-24T17:50:09.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.170+0000 7fb995600640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9900b5790 con 0x7fb99015cbe0 2024-01-24T17:50:09.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.170+0000 7fb995600640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9900b5a50 con 0x7fb9900d12e0 2024-01-24T17:50:09.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.170+0000 7fb98e000640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fb9900d0e90 0x7fb990164b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:58296/0 (socket says 172.21.15.39:58296) 2024-01-24T17:50:09.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.170+0000 7fb98e000640 1 -- 172.21.15.39:0/4270492139 learned_addr learned my addr 172.21.15.39:0/4270492139 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:09.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.171+0000 7fb98f400640 1 --2- 172.21.15.39:0/4270492139 >> v2:172.21.15.100:3300/0 conn(0x7fb9900d12e0 0x7fb990168270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.171+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762308027 0 0) 0x7fb9900b8820 con 0x7fb9900d0e90 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.171+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb970003520 con 0x7fb9900d0e90 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.171+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 <== mon.1 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206542989 0 0) 0x7fb9900b5790 con 0x7fb99015cbe0 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.171+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9900b8820 con 0x7fb99015cbe0 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.171+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1026924130 0 0) 0x7fb970003520 con 0x7fb9900d0e90 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.171+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 >> v2:172.21.15.100:3300/0 conn(0x7fb9900d12e0 msgr2=0x7fb990168270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 --2- 172.21.15.39:0/4270492139 >> v2:172.21.15.100:3300/0 conn(0x7fb9900d12e0 0x7fb990168270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 >> v1:172.21.15.100:6789/0 conn(0x7fb99015cbe0 legacy=0x7fb9900cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9900b5d30 con 0x7fb9900d0e90 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 77223776 0 0) 0x7fb9780015f0 con 0x7fb9900d0e90 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 >> v1:172.21.15.39:6789/0 conn(0x7fb9900d0e90 legacy=0x7fb990164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb970003b50 0x7fb970003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9700047c0 0x7fb970004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970005180 0x7fb97000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900b5a50 con 0x7fb970003b50 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97000c050 con 0x7fb9700047c0 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97000c210 con 0x7fb970005180 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98ea00640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb970003b50 0x7fb970003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98e000640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9700047c0 0x7fb970004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98f400640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970005180 0x7fb97000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98ea00640 1 -- 172.21.15.39:0/4270492139 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970005180 msgr2=0x7fb97000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98ea00640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970005180 0x7fb97000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98ea00640 1 -- 172.21.15.39:0/4270492139 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9700047c0 msgr2=0x7fb970004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98ea00640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb9700047c0 0x7fb970004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.172+0000 7fb98ea00640 1 -- 172.21.15.39:0/4270492139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970003520 con 0x7fb970003b50 2024-01-24T17:50:09.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.173+0000 7fb98f400640 1 -- 172.21.15.39:0/4270492139 reap_dead start 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.174+0000 7fb98ea00640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb970003b50 0x7fb970003f50 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fb980002010 tx=0x7fb9700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5734b5632e0fb950 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.174+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb980003bc0 con 0x7fb970003b50 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.174+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb980003d40 con 0x7fb970003b50 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.174+0000 7fb98d600640 1 -- 172.21.15.39:0/4270492139 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb980005310 con 0x7fb970003b50 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.175+0000 7fb995600640 1 -- 172.21.15.39:0/4270492139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb970003b50 msgr2=0x7fb970003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.175+0000 7fb995600640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb970003b50 0x7fb970003f50 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fb980002010 tx=0x7fb9700057f0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.175+0000 7fb995600640 1 -- 172.21.15.39:0/4270492139 shutdown_connections 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.175+0000 7fb995600640 1 --2- 172.21.15.39:0/4270492139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb970003b50 0x7fb970003f50 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.175+0000 7fb995600640 1 -- 172.21.15.39:0/4270492139 >> 172.21.15.39:0/4270492139 conn(0x7fb99009bc80 msgr2=0x7fb9900d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.175+0000 7fb995600640 1 -- 172.21.15.39:0/4270492139 shutdown_connections 2024-01-24T17:50:09.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.175+0000 7fb995600640 1 -- 172.21.15.39:0/4270492139 wait complete. 2024-01-24T17:50:09.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.177+0000 7fb995600640 1 Processor -- start 2024-01-24T17:50:09.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.177+0000 7fb995600640 1 -- start start 2024-01-24T17:50:09.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb995600640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970003b50 0x7fb990152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb995600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99006ae70 0x7fb99006b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb995600640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb99009f4e0 0x7fb99009f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb995600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900cb4e0 con 0x7fb99006ae70 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb995600640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99006b830 con 0x7fb99009f4e0 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb995600640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99009fea0 con 0x7fb970003b50 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb98e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99006ae70 0x7fb99006b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb98e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99006ae70 0x7fb99006b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57074/0 (socket says 172.21.15.39:57074) 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.178+0000 7fb98e000640 1 -- 172.21.15.39:0/2858761092 learned_addr learned my addr 172.21.15.39:0/2858761092 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98ea00640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970003b50 0x7fb990152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98f400640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb99009f4e0 0x7fb99009f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98e000640 1 -- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970003b50 msgr2=0x7fb990152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98e000640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970003b50 0x7fb990152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98e000640 1 -- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb99009f4e0 msgr2=0x7fb99009f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98e000640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb99009f4e0 0x7fb99009f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98e000640 1 -- 172.21.15.39:0/2858761092 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9900a0160 con 0x7fb99006ae70 2024-01-24T17:50:09.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98f400640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb99009f4e0 0x7fb99009f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:09.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb98e000640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99006ae70 0x7fb99006b2f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fb97800e780 tx=0x7fb97800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:09.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb986a00640 1 -- 172.21.15.39:0/2858761092 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97800c660 con 0x7fb99006ae70 2024-01-24T17:50:09.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb986a00640 1 -- 172.21.15.39:0/2858761092 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97800c7c0 con 0x7fb99006ae70 2024-01-24T17:50:09.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb986a00640 1 -- 172.21.15.39:0/2858761092 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97800cb30 con 0x7fb99006ae70 2024-01-24T17:50:09.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9900a0440 con 0x7fb99006ae70 2024-01-24T17:50:09.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.179+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9900cb100 con 0x7fb99006ae70 2024-01-24T17:50:09.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.181+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9900cb620 con 0x7fb99006ae70 2024-01-24T17:50:09.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.181+0000 7fb986a00640 1 -- 172.21.15.39:0/2858761092 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb97800cc90 con 0x7fb99006ae70 2024-01-24T17:50:09.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.182+0000 7fb986a00640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb96806a430 0x7fb96806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.182+0000 7fb986a00640 1 -- 172.21.15.39:0/2858761092 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fb978088760 con 0x7fb99006ae70 2024-01-24T17:50:09.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.183+0000 7fb98ea00640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb96806a430 0x7fb96806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.184+0000 7fb98ea00640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb96806a430 0x7fb96806c8f0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb980034000 tx=0x7fb980009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:09.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.187+0000 7fb986a00640 1 -- 172.21.15.39:0/2858761092 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb9900cb620 con 0x7fb99006ae70 2024-01-24T17:50:09.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.381+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9900b62f0 con 0x7fb96806a430 2024-01-24T17:50:09.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.382+0000 7fb986a00640 1 -- 172.21.15.39:0/2858761092 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb9800050d0 con 0x7fb96806a430 2024-01-24T17:50:09.383 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 7 to deep-scrub 2024-01-24T17:50:09.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.385+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb96806a430 msgr2=0x7fb96806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.385+0000 7fb995600640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb96806a430 0x7fb96806c8f0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb980034000 tx=0x7fb980009030 comp rx=0 tx=0).stop 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.385+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99006ae70 msgr2=0x7fb99006b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.385+0000 7fb995600640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99006ae70 0x7fb99006b2f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fb97800e780 tx=0x7fb97800ec40 comp rx=0 tx=0).stop 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.386+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 shutdown_connections 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.386+0000 7fb995600640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb99009f4e0 0x7fb99009f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.386+0000 7fb995600640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb96806a430 0x7fb96806c8f0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.386+0000 7fb995600640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb99006ae70 0x7fb99006b2f0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.386+0000 7fb995600640 1 --2- 172.21.15.39:0/2858761092 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb970003b50 0x7fb990152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.386+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 >> 172.21.15.39:0/2858761092 conn(0x7fb99009bc80 msgr2=0x7fb9900cdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.386+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 shutdown_connections 2024-01-24T17:50:09.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.386+0000 7fb995600640 1 -- 172.21.15.39:0/2858761092 wait complete. 2024-01-24T17:50:09.406 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:50:09.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.622+0000 7f7699c00640 1 Processor -- start 2024-01-24T17:50:09.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.623+0000 7f7699c00640 1 -- start start 2024-01-24T17:50:09.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.624+0000 7f7699c00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f76940cf200 0x7f76940d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.624+0000 7f7699c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940b1bf0 0x7f76940b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.624+0000 7f7699c00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76940ce2f0 con 0x7f76940d05d0 2024-01-24T17:50:09.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.624+0000 7f7699c00640 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76940cd440 con 0x7f76940cf200 2024-01-24T17:50:09.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.624+0000 7f7699c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76940ce780 con 0x7f76940b1bf0 2024-01-24T17:50:09.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.625+0000 7f7693e00640 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7f76940cf200 0x7f76940d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.625+0000 7f7693400640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f76940d05d0 0x7f76940cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.39:48002/0 (socket says 172.21.15.39:48002) 2024-01-24T17:50:09.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.625+0000 7f7693400640 1 -- 172.21.15.39:0/401415319 learned_addr learned my addr 172.21.15.39:0/401415319 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:09.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.625+0000 7f7699200640 1 --2- 172.21.15.39:0/401415319 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940b1bf0 0x7f76940b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.626+0000 7f7693e00640 1 -- 172.21.15.39:0/401415319 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940b1bf0 msgr2=0x7f76940b2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:09.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.626+0000 7f7693e00640 1 --2- 172.21.15.39:0/401415319 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940b1bf0 0x7f76940b2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.626+0000 7f7693e00640 1 -- 172.21.15.39:0/401415319 >> v1:172.21.15.100:6789/0 conn(0x7f76940d05d0 legacy=0x7f76940cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.626+0000 7f7693e00640 1 -- 172.21.15.39:0/401415319 --> v2:172.21.15.100:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76940cd6e0 con 0x7f76940cf200 2024-01-24T17:50:09.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.627+0000 7f7693e00640 1 --2- 172.21.15.39:0/401415319 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940cf200 0x7f76940d3310 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7684009790 tx=0x7f768402b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=890676c3eaa6f8c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:09.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.628+0000 7f7692a00640 1 -- 172.21.15.39:0/401415319 <== mon.1 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684031890 con 0x7f76940cf200 2024-01-24T17:50:09.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.628+0000 7f7692a00640 1 -- 172.21.15.39:0/401415319 <== mon.1 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76840319f0 con 0x7f76940cf200 2024-01-24T17:50:09.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.628+0000 7f7692a00640 1 -- 172.21.15.39:0/401415319 <== mon.1 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684031d80 con 0x7f76940cf200 2024-01-24T17:50:09.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.629+0000 7f7699c00640 1 -- 172.21.15.39:0/401415319 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940cf200 msgr2=0x7f76940d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.629+0000 7f7699c00640 1 --2- 172.21.15.39:0/401415319 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940cf200 0x7f76940d3310 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7684009790 tx=0x7f768402b930 comp rx=0 tx=0).stop 2024-01-24T17:50:09.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.629+0000 7f7699c00640 1 -- 172.21.15.39:0/401415319 shutdown_connections 2024-01-24T17:50:09.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.629+0000 7f7699c00640 1 --2- 172.21.15.39:0/401415319 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940b1bf0 0x7f76940b2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.629+0000 7f7699c00640 1 --2- 172.21.15.39:0/401415319 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940cf200 0x7f76940d3310 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.629+0000 7f7699c00640 1 -- 172.21.15.39:0/401415319 >> 172.21.15.39:0/401415319 conn(0x7f76940d27f0 msgr2=0x7f76940d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:09.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.629+0000 7f7699c00640 1 -- 172.21.15.39:0/401415319 shutdown_connections 2024-01-24T17:50:09.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.629+0000 7f7699c00640 1 -- 172.21.15.39:0/401415319 wait complete. 2024-01-24T17:50:09.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.631+0000 7f7699c00640 1 Processor -- start 2024-01-24T17:50:09.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.631+0000 7f7699c00640 1 -- start start 2024-01-24T17:50:09.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.632+0000 7f7699c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940b1bf0 0x7f76941f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.632+0000 7f7699c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76940cf200 0x7f7694156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.632+0000 7f7699c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940d05d0 0x7f7694156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.632+0000 7f7699c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76940b7570 con 0x7f76940cf200 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.632+0000 7f7699c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76940ccbe0 con 0x7f76940b1bf0 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.632+0000 7f7699c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76940b6f60 con 0x7f76940d05d0 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.632+0000 7f7693400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76940cf200 0x7f7694156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76940cf200 0x7f7694156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:57084/0 (socket says 172.21.15.39:57084) 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693400640 1 -- 172.21.15.39:0/1865180900 learned_addr learned my addr 172.21.15.39:0/1865180900 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693e00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940b1bf0 0x7f76941f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7699200640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940d05d0 0x7f7694156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693400640 1 -- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940d05d0 msgr2=0x7f7694156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693400640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940d05d0 0x7f7694156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693400640 1 -- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940b1bf0 msgr2=0x7f76941f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693400640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940b1bf0 0x7f76941f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693400640 1 -- 172.21.15.39:0/1865180900 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840093a0 con 0x7f76940cf200 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693e00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940b1bf0 0x7f76941f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.633+0000 7f7693400640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76940cf200 0x7f7694156070 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c480 tx=0x7f767c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.634+0000 7f768be00640 1 -- 172.21.15.39:0/1865180900 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c005ce0 con 0x7f76940cf200 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.634+0000 7f768be00640 1 -- 172.21.15.39:0/1865180900 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c005e40 con 0x7f76940cf200 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.634+0000 7f768be00640 1 -- 172.21.15.39:0/1865180900 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c0072f0 con 0x7f76940cf200 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.634+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76940cc900 con 0x7f76940cf200 2024-01-24T17:50:09.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.634+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76941f2b10 con 0x7f76940cf200 2024-01-24T17:50:09.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.635+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769404dff0 con 0x7f76940cf200 2024-01-24T17:50:09.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.636+0000 7f768be00640 1 -- 172.21.15.39:0/1865180900 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f767c007510 con 0x7f76940cf200 2024-01-24T17:50:09.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.637+0000 7f768be00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f766c06a430 0x7f766c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:09.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.637+0000 7f768be00640 1 -- 172.21.15.39:0/1865180900 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f767c087c90 con 0x7f76940cf200 2024-01-24T17:50:09.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.637+0000 7f7693e00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f766c06a430 0x7f766c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:09.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.639+0000 7f7693e00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f766c06a430 0x7f766c06c8f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7684036000 tx=0x7f7684008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:09.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.641+0000 7f768be00640 1 -- 172.21.15.39:0/1865180900 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f767c0534c0 con 0x7f76940cf200 2024-01-24T17:50:09.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.833+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76940cdca0 con 0x7f766c06a430 2024-01-24T17:50:09.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.840+0000 7f768be00640 1 -- 172.21.15.39:0/1865180900 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48523 (secure 0 0 0) 0x7f7684045400 con 0x7f766c06a430 2024-01-24T17:50:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:50:09.842 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.842+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f766c06a430 msgr2=0x7f766c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.842+0000 7f7699c00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f766c06a430 0x7f766c06c8f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7684036000 tx=0x7f7684008030 comp rx=0 tx=0).stop 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76940cf200 msgr2=0x7f7694156070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76940cf200 0x7f7694156070 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f767c00c480 tx=0x7f767c00c940 comp rx=0 tx=0).stop 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 shutdown_connections 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f76940d05d0 0x7f7694156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f766c06a430 0x7f766c06c8f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76940cf200 0x7f7694156070 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 --2- 172.21.15.39:0/1865180900 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76940b1bf0 0x7f76941f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 >> 172.21.15.39:0/1865180900 conn(0x7f76940d27f0 msgr2=0x7f76940b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 shutdown_connections 2024-01-24T17:50:09.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:09.843+0000 7f7699c00640 1 -- 172.21.15.39:0/1865180900 wait complete. 2024-01-24T17:50:09.864 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-01-24T17:50:08.396141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260708,"kb_used_data":11752,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754714012,"statfs":{"total":773094113280,"available":772827148288,"internally_reserved":0,"allocated":12034048,"data_stored":5887296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.667434"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":245,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-24T17:49:57.733468+0000","last_change":"2024-01-24T17:41:23.668992+0000","last_active":"2024-01-24T17:49:57.733468+0000","last_peered":"2024-01-24T17:49:57.733468+0000","last_clean":"2024-01-24T17:49:57.733468+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:49:57.733468+0000","last_undegraded":"2024-01-24T17:49:57.733468+0000","last_fullsized":"2024-01-24T17:49:57.733468+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-24T17:41:22.633799+0000","last_clean_scrub_stamp":"2024-01-24T17:41:22.633799+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32440,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0720000000000001,"15min":1.0449999999999999},"min":{"1min":0.73299999999999998,"5min":0.64900000000000002,"15min":0.61899999999999999},"max":{"1min":1.653,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.077,"5min":1.1359999999999999,"15min":1.0700000000000001},"min":{"1min":0.63400000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.411,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.99399999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.68200000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.472,"5min":1.6479999999999999,"15min":2.04},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2070000000000001,"15min":1.196},"min":{"1min":0.75700000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.381}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1779999999999999,"15min":1.167},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.59099999999999997},"max":{"1min":1.484,"5min":1.992,"15min":2.1400000000000001},"last":1.268},{"interface":"front","average":{"1min":1.109,"5min":1.1910000000000001,"15min":1.2030000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.577,"5min":2.056,"15min":2.484},"last":1.091}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2350000000000001,"15min":1.244},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.843,"5min":2.0179999999999998,"15min":2.2930000000000001},"last":1.129},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.3100000000000001,"15min":1.3049999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5740000000000001,"5min":1.9059999999999999,"15min":2.6269999999999998},"last":1.1639999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2729999999999999,"15min":1.3009999999999999},"min":{"1min":0.72999999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5029999999999999,"5min":2.2549999999999999,"15min":2.6179999999999999},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3879999999999999,"15min":1.4179999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":2.6459999999999999},"last":1.464}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.3320000000000001},"min":{"1min":0.58799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.885,"5min":1.9810000000000001,"15min":2.8460000000000001},"last":1.488},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3740000000000001,"15min":1.4079999999999999},"min":{"1min":0.68300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6359999999999999,"5min":2.2480000000000002,"15min":2.3719999999999999},"last":1.7370000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.47,"15min":1.5069999999999999},"min":{"1min":0.83399999999999996,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.7190000000000001,"5min":2.4129999999999998,"15min":2.6960000000000002},"last":1.869},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4219999999999999,"15min":1.4630000000000001},"min":{"1min":0.66600000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.946,"5min":2.0790000000000002,"15min":2.4830000000000001},"last":1.5860000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0740000000000001,"15min":1.042},"min":{"1min":0.69699999999999995,"5min":0.69599999999999995,"15min":0.61199999999999999},"max":{"1min":1.4390000000000001,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0820000000000001,"15min":1.044},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.66300000000000003},"max":{"1min":1.5760000000000001,"5min":1.744,"15min":1.744},"last":1.2070000000000001}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.169,"15min":1.056},"min":{"1min":0.88500000000000001,"5min":0.71199999999999997,"15min":0.64200000000000002},"max":{"1min":2.9790000000000001,"5min":2.9790000000000001,"15min":2.9790000000000001},"last":1.004},{"interface":"front","average":{"1min":1.079,"5min":1.1379999999999999,"15min":1.1279999999999999},"min":{"1min":0.71599999999999997,"5min":0.67800000000000005,"15min":0.56399999999999995},"max":{"1min":1.425,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":1.1439999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.1970000000000001,"15min":1.155},"min":{"1min":0.85499999999999998,"5min":0.71899999999999997,"15min":0.63600000000000001},"max":{"1min":1.675,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":1.056},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.22,"15min":1.1850000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.56599999999999995},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.409}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2709999999999999,"15min":1.258},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73399999999999999},"max":{"1min":3.21,"5min":3.21,"15min":3.21},"last":1.29},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.234,"15min":1.2430000000000001},"min":{"1min":0.91100000000000003,"5min":0.84999999999999998,"15min":0.69799999999999995},"max":{"1min":1.554,"5min":2.0489999999999999,"15min":2.46},"last":1.3520000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3080000000000001,"15min":1.264},"min":{"1min":0.59199999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.268},{"interface":"front","average":{"1min":1.379,"5min":1.3240000000000001,"15min":1.3089999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.44600000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.619}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3380000000000001,"15min":1.3120000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.46800000000000003},"max":{"1min":3.3759999999999999,"5min":3.3759999999999999,"15min":3.3759999999999999},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.462,"15min":1.4419999999999999},"min":{"1min":0.67100000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.4849999999999999},"last":1.4430000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.361,"15min":1.363},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.51400000000000001},"max":{"1min":1.786,"5min":1.921,"15min":2.2050000000000001},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4710000000000001,"15min":1.464},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.52300000000000002},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.6870000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33020,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338820,"statfs":{"total":96636764160,"available":96602951680,"internally_reserved":0,"allocated":1949696,"data_stored":1178688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0680000000000001,"15min":1.083},"min":{"1min":0.72099999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5389999999999999,"5min":3.609,"15min":3.609},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.032,"5min":1.0680000000000001,"15min":1.0429999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.391,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":0.92500000000000004}]},{"osd":1,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.119,"15min":1.089},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.622},"max":{"1min":1.5249999999999999,"5min":3.5470000000000002,"15min":3.5470000000000002},"last":1.165},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.153,"15min":1.22},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.544,"5min":1.859,"15min":2.1579999999999999},"last":1.1970000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.224,"15min":1.226},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.59199999999999997},"max":{"1min":1.6399999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":1.036},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3500000000000001,"15min":1.319},"min":{"1min":0.96499999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.663,"5min":8.1509999999999998,"15min":8.1509999999999998},"last":0.86899999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2569999999999999,"15min":1.3049999999999999},"min":{"1min":0.99099999999999999,"5min":0.76400000000000001,"15min":0.70799999999999996},"max":{"1min":1.466,"5min":2.7559999999999998,"15min":3.036},"last":1.101},{"interface":"front","average":{"1min":1.2,"5min":1.28,"15min":1.3109999999999999},"min":{"1min":0.70499999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.474,"5min":4.1230000000000002,"15min":4.1230000000000002},"last":1.304}]},{"osd":4,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.355,"15min":1.417},"min":{"1min":0.68600000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6259999999999999,"5min":1.9690000000000001,"15min":2.069},"last":1.381},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.254,"15min":1.319},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.77,"5min":2.1139999999999999,"15min":3.2810000000000001},"last":0.82899999999999996}]},{"osd":6,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.304,"15min":1.3360000000000001},"min":{"1min":0.871,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.754,"5min":2.028,"15min":2.504},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.331,"5min":1.405,"15min":1.484},"min":{"1min":0.63800000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.575,"5min":2.1059999999999999,"15min":2.2290000000000001},"last":1.4590000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3280000000000001,"15min":1.4319999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.6759999999999999,"5min":2.2010000000000001,"15min":3.161},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.486,"5min":1.5129999999999999,"15min":1.5860000000000001},"min":{"1min":1.042,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8420000000000001,"5min":2.262,"15min":3.2410000000000001},"last":1.2390000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.0589999999999999},"min":{"1min":0.73299999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3540000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.099,"15min":1.097},"min":{"1min":0.874,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6359999999999999,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":1.0329999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1739999999999999,"15min":1.097},"min":{"1min":0.78900000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5980000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.639},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1679999999999999,"15min":1.196},"min":{"1min":0.73099999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.698,"5min":1.712,"15min":1.849},"last":1.125}]},{"osd":2,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2430000000000001,"15min":1.2210000000000001},"min":{"1min":0.97299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.788,"5min":1.9850000000000001,"15min":2.105},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.347,"5min":1.3080000000000001,"15min":1.3200000000000001},"min":{"1min":0.99399999999999999,"5min":0.69599999999999995,"15min":0.64500000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.986},"last":1.1870000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1819999999999999,"15min":1.1870000000000001},"min":{"1min":0.76800000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":1.637,"15min":1.7769999999999999},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.296,"15min":1.331},"min":{"1min":0.998,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.9409999999999998},"last":1.1000000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.3169999999999999,"15min":1.3500000000000001},"min":{"1min":1.044,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0979999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":0.91700000000000004,"5min":0.65600000000000003,"15min":0.55200000000000005},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.9039999999999999},"last":1.593}]},{"osd":6,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.351,"15min":1.429},"min":{"1min":0.95599999999999996,"5min":0.67800000000000005,"15min":0.64400000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":3.0840000000000001},"last":1.786},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4550000000000001,"15min":1.468},"min":{"1min":1.1499999999999999,"5min":0.67700000000000005,"15min":0.66500000000000004},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.5369999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":1.0129999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.756},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.4910000000000001,"15min":1.5740000000000001},"min":{"1min":0.98399999999999999,"5min":0.79300000000000004,"15min":0.60399999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":4.3250000000000002},"last":1.8460000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.86599999999999999},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.4590000000000001,"5min":1.5580000000000001,"15min":5.1929999999999996},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.88800000000000001,"15min":0.90100000000000002},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.4279999999999999,"5min":1.6739999999999999,"15min":8.6989999999999998},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.91700000000000004},"min":{"1min":0.46600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.4099999999999999,"5min":1.5229999999999999,"15min":1.8720000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0389999999999999,"15min":1.0580000000000001},"min":{"1min":0.69499999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.679,"5min":1.679,"15min":3.7240000000000002},"last":1.641}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0760000000000001,"15min":1.0980000000000001},"min":{"1min":0.499,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.581,"5min":1.7170000000000001,"15min":3.4630000000000001},"last":0.88},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1279999999999999,"15min":1.155},"min":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":4.2919999999999998},"last":1.2989999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.208,"15min":1.2689999999999999},"min":{"1min":0.76700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6060000000000001,"5min":2.0049999999999999,"15min":6.7469999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.2789999999999999,"15min":1.3280000000000001},"min":{"1min":0.873,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.6950000000000001,"5min":2.0390000000000001,"15min":3.4049999999999998},"last":1.018}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3129999999999999,"15min":1.395},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7050000000000001,"5min":2.1269999999999998,"15min":4.3789999999999996},"last":1.089},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.363,"15min":1.4019999999999999},"min":{"1min":0.78300000000000003,"5min":0.746,"15min":0.746},"max":{"1min":2.1859999999999999,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.444}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.411,"15min":1.4770000000000001},"min":{"1min":0.82199999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.331,"5min":2.331,"15min":9.5440000000000005},"last":1.552},{"interface":"front","average":{"1min":1.284,"5min":1.327,"15min":1.419},"min":{"1min":0.78300000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.659,"5min":2.109,"15min":4.8170000000000002},"last":1.3440000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.373,"15min":1.472},"min":{"1min":0.90000000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.736,"5min":2.3410000000000002,"15min":6.6970000000000001},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.431,"15min":1.4970000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.889,"5min":2.0990000000000002,"15min":6.6529999999999996},"last":1.2709999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.78700000000000003,"15min":0.79300000000000004},"min":{"1min":0.42099999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.6180000000000001,"5min":1.667,"15min":1.667},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90000000000000002,"15min":0.92600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.472,"5min":1.7070000000000001,"15min":1.7989999999999999},"last":0.81200000000000006}]},{"osd":1,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.018,"15min":1.04},"min":{"1min":0.52100000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1080000000000001,"15min":1.1719999999999999},"min":{"1min":0.56000000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5960000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.755}]},{"osd":3,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95799999999999996,"15min":0.98099999999999998},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.7130000000000001,"15min":2.2120000000000002},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.276,"5min":1.1459999999999999,"15min":1.1919999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.996},"last":1.3049999999999999}]},{"osd":4,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.244,"15min":1.2969999999999999},"min":{"1min":0.79500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.748,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.077},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.256,"15min":1.349},"min":{"1min":0.71899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7250000000000001,"5min":1.889,"15min":2.3839999999999999},"last":1.3340000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.371},"min":{"1min":0.99299999999999999,"5min":0.81899999999999995,"15min":0.73899999999999999},"max":{"1min":1.6879999999999999,"5min":1.851,"15min":2.3140000000000001},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.2230000000000001,"15min":1.2889999999999999},"min":{"1min":0.95599999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.804,"5min":1.889,"15min":2.0720000000000001},"last":1.2490000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.294,"15min":1.3600000000000001},"min":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.623},"max":{"1min":2.056,"5min":2.056,"15min":2.2679999999999998},"last":1.411},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.335,"15min":1.4259999999999999},"min":{"1min":0.92100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":3.7410000000000001},"last":1.2130000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.383,"15min":1.508},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6879999999999999,"5min":2.1499999999999999,"15min":3.343},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.375,"15min":1.458},"min":{"1min":0.76100000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.1600000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.5029999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1949696,"data_stored":1178688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.85599999999999998,"15min":0.86099999999999999},"min":{"1min":0.42999999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.54,"5min":1.722,"15min":2.0339999999999998},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.95999999999999996,"15min":0.95199999999999996},"min":{"1min":0.53700000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.5089999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.69399999999999995}]},{"osd":2,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88100000000000001,"15min":0.94499999999999995},"min":{"1min":0.51300000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3580000000000001,"5min":1.3959999999999999,"15min":1.6379999999999999},"last":1.115},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.069,"15min":1.0900000000000001},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.41799999999999998},"max":{"1min":1.556,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.97499999999999998}]},{"osd":3,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.171,"15min":1.163},"min":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.47699999999999998},"max":{"1min":1.623,"5min":1.8440000000000001,"15min":1.9610000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.0760000000000001,"15min":1.1799999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.41499999999999998},"max":{"1min":1.446,"5min":1.6910000000000001,"15min":3.149},"last":0.58099999999999996}]},{"osd":4,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.355},"min":{"1min":0.79000000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.7250000000000001,"5min":1.962,"15min":2.2410000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2969999999999999},"min":{"1min":0.871,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.673,"5min":1.734,"15min":2.3849999999999998},"last":1.0680000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.248,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.581,"5min":1.964,"15min":2.577},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.373,"15min":1.371},"min":{"1min":0.90400000000000003,"5min":0.89800000000000002,"15min":0.81299999999999994},"max":{"1min":1.802,"5min":2.2770000000000001,"15min":2.9809999999999999},"last":1.446}]},{"osd":6,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3560000000000001,"15min":1.4079999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.73099999999999998},"max":{"1min":1.7809999999999999,"5min":2.008,"15min":3.2850000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4590000000000001},"min":{"1min":0.94699999999999995,"5min":0.72599999999999998,"15min":0.70199999999999996},"max":{"1min":1.6950000000000001,"5min":2.032,"15min":3.5859999999999999},"last":1.163}]},{"osd":7,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.4219999999999999,"15min":1.454},"min":{"1min":0.91400000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.97,"5min":2.5880000000000001,"15min":2.8439999999999999},"last":1.371},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4419999999999999,"15min":1.5089999999999999},"min":{"1min":0.83499999999999996,"5min":0.82399999999999995,"15min":0.74099999999999999},"max":{"1min":1.891,"5min":2.1400000000000001,"15min":3.4350000000000001},"last":1.1319999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81499999999999995,"15min":0.81399999999999995},"min":{"1min":0.42499999999999999,"5min":0.40999999999999998,"15min":0.29099999999999998},"max":{"1min":1.266,"5min":1.4750000000000001,"15min":1.7290000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.98899999999999999,"15min":0.96899999999999997},"min":{"1min":0.65700000000000003,"5min":0.433,"15min":0.376},"max":{"1min":1.7569999999999999,"5min":1.778,"15min":1.778},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86099999999999999,"15min":0.84399999999999997},"min":{"1min":0.73499999999999999,"5min":0.42999999999999999,"15min":0.34699999999999998},"max":{"1min":1.1100000000000001,"5min":1.837,"15min":1.837},"last":0.63700000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0249999999999999,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.50600000000000001,"15min":0.42399999999999999},"max":{"1min":1.4359999999999999,"5min":1.583,"15min":2.0499999999999998},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.143,"15min":1.165},"min":{"1min":0.497,"5min":0.497,"15min":0.35799999999999998},"max":{"1min":1.5900000000000001,"5min":1.8320000000000001,"15min":1.9670000000000001},"last":0.755},{"interface":"front","average":{"1min":1.248,"5min":1.212,"15min":1.1919999999999999},"min":{"1min":0.66200000000000003,"5min":0.49299999999999999,"15min":0.46800000000000003},"max":{"1min":1.849,"5min":1.849,"15min":2.407},"last":0.67500000000000004}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.298,"15min":1.367},"min":{"1min":0.94199999999999995,"5min":0.90000000000000002,"15min":0.70799999999999996},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":2.2589999999999999},"last":1.014},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.345,"15min":1.4279999999999999},"min":{"1min":1.069,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":3.3740000000000001},"last":0.95699999999999996}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3049999999999999,"15min":1.3560000000000001},"min":{"1min":0.98299999999999998,"5min":0.83699999999999997,"15min":0.73499999999999999},"max":{"1min":2.101,"5min":2.101,"15min":2.9300000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.331,"5min":1.262,"15min":1.262},"min":{"1min":0.86599999999999999,"5min":0.77000000000000002,"15min":0.65600000000000003},"max":{"1min":1.794,"5min":1.8520000000000001,"15min":2.3799999999999999},"last":1.1100000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.353,"15min":1.3540000000000001},"min":{"1min":0.89200000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.855,"5min":1.954,"15min":2.335},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.413,"15min":1.4370000000000001},"min":{"1min":1.0740000000000001,"5min":0.90600000000000003,"15min":0.70999999999999996},"max":{"1min":1.873,"5min":1.931,"15min":2.2389999999999999},"last":1.0900000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4319999999999999,"15min":1.488},"min":{"1min":1.153,"5min":0.89700000000000002,"15min":0.76100000000000001},"max":{"1min":1.917,"5min":2.0019999999999998,"15min":3.0230000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.538,"5min":1.4750000000000001,"15min":1.55},"min":{"1min":0.91600000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.962,"5min":2.1659999999999999,"15min":3.1600000000000001},"last":1.0309999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:50:09.868 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-24T17:41:22.633799+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=24, tm_hour=17, tm_min=41, tm_sec=22, tm_wday=2, tm_yday=24, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=24, tm_hour=17, tm_min=50, tm_sec=2, tm_wday=2, tm_yday=24, tm_isdst=0) 2024-01-24T17:50:09.868 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-24T17:50:22.199 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:50:22.207 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:50:29.869 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-24T17:50:30.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.088+0000 7fb4d5c00640 1 Processor -- start 2024-01-24T17:50:30.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.088+0000 7fb4d5c00640 1 -- start start 2024-01-24T17:50:30.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4d5c00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb4d00b3e00 0x7fb4d00c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4d5c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00c8f70 0x7fb4d015e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4d5c00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d01551b0 con 0x7fb4d0070540 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4d5c00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d015e410 con 0x7fb4d00b3e00 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4d5c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d015b480 con 0x7fb4d00c8f70 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4c7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb4d00b3e00 0x7fb4d00c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4c7200640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb4d00b3e00 0x7fb4d00c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58664/0 (socket says 172.21.15.39:58664) 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4c7200640 1 -- 172.21.15.39:0/309017413 learned_addr learned my addr 172.21.15.39:0/309017413 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.090+0000 7fb4d5200640 1 --2- 172.21.15.39:0/309017413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00c8f70 0x7fb4d015e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.091+0000 7fb4c7200640 1 -- 172.21.15.39:0/309017413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00c8f70 msgr2=0x7fb4d015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.091+0000 7fb4c7200640 1 --2- 172.21.15.39:0/309017413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00c8f70 0x7fb4d015e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.091+0000 7fb4c7200640 1 -- 172.21.15.39:0/309017413 >> v1:172.21.15.39:6789/0 conn(0x7fb4d0070540 legacy=0x7fb4d015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.091+0000 7fb4c7200640 1 -- 172.21.15.39:0/309017413 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d015b760 con 0x7fb4d00b3e00 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.091+0000 7fb4cf400640 1 -- 172.21.15.39:0/309017413 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333325784 0 0) 0x7fb4d01551b0 con 0x7fb4d0070540 2024-01-24T17:50:30.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.091+0000 7fb4d5200640 1 --2- 172.21.15.39:0/309017413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00c8f70 0x7fb4d015e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:30.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.092+0000 7fb4c7200640 1 --2- 172.21.15.39:0/309017413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00b3e00 0x7fb4d00c8a30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80098a0 tx=0x7fb4b802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e295da6e2e1e7dd server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:30.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.092+0000 7fb4cf400640 1 -- 172.21.15.39:0/309017413 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8030840 con 0x7fb4d00b3e00 2024-01-24T17:50:30.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.092+0000 7fb4cf400640 1 -- 172.21.15.39:0/309017413 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b80309a0 con 0x7fb4d00b3e00 2024-01-24T17:50:30.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.092+0000 7fb4cf400640 1 -- 172.21.15.39:0/309017413 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8030d30 con 0x7fb4d00b3e00 2024-01-24T17:50:30.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.093+0000 7fb4d5c00640 1 -- 172.21.15.39:0/309017413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00b3e00 msgr2=0x7fb4d00c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.093+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/309017413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00b3e00 0x7fb4d00c8a30 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80098a0 tx=0x7fb4b802b980 comp rx=0 tx=0).stop 2024-01-24T17:50:30.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.093+0000 7fb4d5c00640 1 -- 172.21.15.39:0/309017413 shutdown_connections 2024-01-24T17:50:30.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.093+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/309017413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00b3e00 0x7fb4d00c8a30 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.093+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/309017413 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00c8f70 0x7fb4d015e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.093+0000 7fb4d5c00640 1 -- 172.21.15.39:0/309017413 >> 172.21.15.39:0/309017413 conn(0x7fb4d006a340 msgr2=0x7fb4d00cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:30.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.093+0000 7fb4d5c00640 1 -- 172.21.15.39:0/309017413 shutdown_connections 2024-01-24T17:50:30.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.093+0000 7fb4d5c00640 1 -- 172.21.15.39:0/309017413 wait complete. 2024-01-24T17:50:30.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.095+0000 7fb4d5c00640 1 Processor -- start 2024-01-24T17:50:30.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.095+0000 7fb4d5c00640 1 -- start start 2024-01-24T17:50:30.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5c00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4d0070540 0x7fb4d014b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5c00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00b3e00 0x7fb4d014b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00c8f70 0x7fb4d015d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00c3100 con 0x7fb4d00c8f70 2024-01-24T17:50:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5c00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d014c370 con 0x7fb4d0070540 2024-01-24T17:50:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5c00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d014c650 con 0x7fb4d00b3e00 2024-01-24T17:50:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00c8f70 0x7fb4d015d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00c8f70 0x7fb4d015d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58672/0 (socket says 172.21.15.39:58672) 2024-01-24T17:50:30.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5200640 1 -- 172.21.15.39:0/2487076683 learned_addr learned my addr 172.21.15.39:0/2487076683 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4cfe00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4d0070540 0x7fb4d014b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4c7200640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00b3e00 0x7fb4d014b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.096+0000 7fb4d5200640 1 -- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00b3e00 msgr2=0x7fb4d014b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4d5200640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00b3e00 0x7fb4d014b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4d5200640 1 -- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4d0070540 msgr2=0x7fb4d014b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4d5200640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4d0070540 0x7fb4d014b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4d5200640 1 -- 172.21.15.39:0/2487076683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b80094d0 con 0x7fb4d00c8f70 2024-01-24T17:50:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4cfe00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4d0070540 0x7fb4d014b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T17:50:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4d5200640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00c8f70 0x7fb4d015d210 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800c5c0 tx=0x7fb4c800ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4ccc00640 1 -- 172.21.15.39:0/2487076683 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8011800 con 0x7fb4d00c8f70 2024-01-24T17:50:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4ccc00640 1 -- 172.21.15.39:0/2487076683 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c8011980 con 0x7fb4d00c8f70 2024-01-24T17:50:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4ccc00640 1 -- 172.21.15.39:0/2487076683 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8011d30 con 0x7fb4d00c8f70 2024-01-24T17:50:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d014c930 con 0x7fb4d00c8f70 2024-01-24T17:50:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.097+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d014b660 con 0x7fb4d00c8f70 2024-01-24T17:50:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.099+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d00c3240 con 0x7fb4d00c8f70 2024-01-24T17:50:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.099+0000 7fb4ccc00640 1 -- 172.21.15.39:0/2487076683 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb4c8022350 con 0x7fb4d00c8f70 2024-01-24T17:50:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.100+0000 7fb4ccc00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a806a430 0x7fb4a806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.100+0000 7fb4ccc00640 1 -- 172.21.15.39:0/2487076683 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fb4c80888d0 con 0x7fb4d00c8f70 2024-01-24T17:50:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.101+0000 7fb4cfe00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a806a430 0x7fb4a806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.102+0000 7fb4cfe00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a806a430 0x7fb4a806c8f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0001350 tx=0x7fb4c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.105+0000 7fb4ccc00640 1 -- 172.21.15.39:0/2487076683 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fb4d00c3240 con 0x7fb4d00c8f70 2024-01-24T17:50:30.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.298+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 --> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4d015ba40 con 0x7fb4a806a430 2024-01-24T17:50:30.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.305+0000 7fb4ccc00640 1 -- 172.21.15.39:0/2487076683 <== mgr.4104 v2:172.21.15.100:6832/36263 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48437 (secure 0 0 0) 0x7fb4c0003970 con 0x7fb4a806a430 2024-01-24T17:50:30.307 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:50:30.307 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-01-24T17:50:30.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a806a430 msgr2=0x7fb4a806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a806a430 0x7fb4a806c8f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0001350 tx=0x7fb4c0009040 comp rx=0 tx=0).stop 2024-01-24T17:50:30.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00c8f70 msgr2=0x7fb4d015d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00c8f70 0x7fb4d015d210 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800c5c0 tx=0x7fb4c800ca80 comp rx=0 tx=0).stop 2024-01-24T17:50:30.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 shutdown_connections 2024-01-24T17:50:30.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fb4a806a430 0x7fb4a806c8f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4d00c8f70 0x7fb4d015d210 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fb4d00b3e00 0x7fb4d014b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 --2- 172.21.15.39:0/2487076683 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb4d0070540 0x7fb4d014b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.307+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 >> 172.21.15.39:0/2487076683 conn(0x7fb4d006a340 msgr2=0x7fb4d015bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:30.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.308+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 shutdown_connections 2024-01-24T17:50:30.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.308+0000 7fb4d5c00640 1 -- 172.21.15.39:0/2487076683 wait complete. 2024-01-24T17:50:30.329 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-01-24T17:50:28.401739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260692,"kb_used_data":11752,"kb_used_omap":13,"kb_used_meta":248818,"kb_avail":754714028,"statfs":{"total":773094113280,"available":772827164672,"internally_reserved":0,"allocated":12034048,"data_stored":5887296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":254789720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003336"},"pg_stats":[{"pgid":"1.0","version":"109'119","reported_seq":258,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-24T17:50:10.896008+0000","last_change":"2024-01-24T17:50:10.896248+0000","last_active":"2024-01-24T17:50:10.896008+0000","last_peered":"2024-01-24T17:50:10.896008+0000","last_clean":"2024-01-24T17:50:10.896008+0000","last_became_active":"2024-01-24T17:41:23.667964+0000","last_became_peered":"2024-01-24T17:41:23.667964+0000","last_unstale":"2024-01-24T17:50:10.896008+0000","last_undegraded":"2024-01-24T17:50:10.896008+0000","last_fullsized":"2024-01-24T17:50:10.896008+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-01-24T17:50:10.895952+0000","last_deep_scrub":"109'119","last_deep_scrub_stamp":"2024-01-24T17:50:10.895952+0000","last_clean_scrub_stamp":"2024-01-24T17:50:10.895952+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-25T23:28:25.322857+0000","scrub_duration":0.013637949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32440,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0720000000000001,"15min":1.0449999999999999},"min":{"1min":0.73299999999999998,"5min":0.64900000000000002,"15min":0.61899999999999999},"max":{"1min":1.653,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.077,"5min":1.1359999999999999,"15min":1.0700000000000001},"min":{"1min":0.63400000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.411,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":1.05}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.68200000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.472,"5min":1.6479999999999999,"15min":2.04},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.2070000000000001,"15min":1.196},"min":{"1min":0.75700000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.554,"5min":1.9910000000000001,"15min":1.9910000000000001},"last":1.1679999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1779999999999999,"15min":1.167},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.59099999999999997},"max":{"1min":1.484,"5min":1.992,"15min":2.1400000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.1910000000000001,"15min":1.2030000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.577,"5min":2.056,"15min":2.484},"last":1.117}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2350000000000001,"15min":1.244},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.843,"5min":2.0179999999999998,"15min":2.2930000000000001},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.3100000000000001,"15min":1.3049999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5740000000000001,"5min":1.9059999999999999,"15min":2.6269999999999998},"last":1.373}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2729999999999999,"15min":1.3009999999999999},"min":{"1min":0.72999999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5029999999999999,"5min":2.2549999999999999,"15min":2.6179999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3879999999999999,"15min":1.4179999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":2.6459999999999999},"last":1.6779999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3089999999999999,"15min":1.3320000000000001},"min":{"1min":0.58799999999999997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.885,"5min":1.9810000000000001,"15min":2.8460000000000001},"last":1.262},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3740000000000001,"15min":1.4079999999999999},"min":{"1min":0.68300000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6359999999999999,"5min":2.2480000000000002,"15min":2.3719999999999999},"last":1.6539999999999999}]},{"osd":6,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.47,"15min":1.5069999999999999},"min":{"1min":0.83399999999999996,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.7190000000000001,"5min":2.4129999999999998,"15min":2.6960000000000002},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4219999999999999,"15min":1.4630000000000001},"min":{"1min":0.66600000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.946,"5min":2.0790000000000002,"15min":2.4830000000000001},"last":1.6200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0740000000000001,"15min":1.042},"min":{"1min":0.69699999999999995,"5min":0.69599999999999995,"15min":0.61199999999999999},"max":{"1min":1.4390000000000001,"5min":1.7929999999999999,"15min":1.8460000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0820000000000001,"15min":1.044},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.66300000000000003},"max":{"1min":1.5760000000000001,"5min":1.744,"15min":1.744},"last":1.028}]},{"osd":1,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.169,"15min":1.056},"min":{"1min":0.88500000000000001,"5min":0.71199999999999997,"15min":0.64200000000000002},"max":{"1min":2.9790000000000001,"5min":2.9790000000000001,"15min":2.9790000000000001},"last":1.466},{"interface":"front","average":{"1min":1.079,"5min":1.1379999999999999,"15min":1.1279999999999999},"min":{"1min":0.71599999999999997,"5min":0.67800000000000005,"15min":0.56399999999999995},"max":{"1min":1.425,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":1.268}]},{"osd":2,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.1970000000000001,"15min":1.155},"min":{"1min":0.85499999999999998,"5min":0.71899999999999997,"15min":0.63600000000000001},"max":{"1min":1.675,"5min":1.6919999999999999,"15min":2.0049999999999999},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.22,"15min":1.1850000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.56599999999999995},"max":{"1min":3.1259999999999999,"5min":3.1259999999999999,"15min":3.1259999999999999},"last":1.552}]},{"osd":3,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.2709999999999999,"15min":1.258},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73399999999999999},"max":{"1min":3.21,"5min":3.21,"15min":3.21},"last":1.23},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.234,"15min":1.2430000000000001},"min":{"1min":0.91100000000000003,"5min":0.84999999999999998,"15min":0.69799999999999995},"max":{"1min":1.554,"5min":2.0489999999999999,"15min":2.46},"last":1.0980000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.3080000000000001,"15min":1.264},"min":{"1min":0.59199999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":1.198},{"interface":"front","average":{"1min":1.379,"5min":1.3240000000000001,"15min":1.3089999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.44600000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.3859999999999999}]},{"osd":5,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.3380000000000001,"15min":1.3120000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.46800000000000003},"max":{"1min":3.3759999999999999,"5min":3.3759999999999999,"15min":3.3759999999999999},"last":1.361},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.462,"15min":1.4419999999999999},"min":{"1min":0.67100000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":2.4849999999999999},"last":1.319}]},{"osd":7,"last update":"Wed Jan 24 17:49:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.361,"15min":1.363},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.51400000000000001},"max":{"1min":1.786,"5min":1.921,"15min":2.2050000000000001},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4710000000000001,"15min":1.464},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.52300000000000002},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":3.4740000000000002},"last":1.6120000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1949696,"data_stored":1178688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0680000000000001,"15min":1.083},"min":{"1min":0.72099999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5389999999999999,"5min":3.609,"15min":3.609},"last":1.008},{"interface":"front","average":{"1min":1.032,"5min":1.0680000000000001,"15min":1.0429999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.391,"5min":3.7240000000000002,"15min":3.7240000000000002},"last":1.411}]},{"osd":1,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.119,"15min":1.089},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.622},"max":{"1min":1.5249999999999999,"5min":3.5470000000000002,"15min":3.5470000000000002},"last":1.119},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.153,"15min":1.22},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.544,"5min":1.859,"15min":2.1579999999999999},"last":1.2330000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.224,"15min":1.226},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.59199999999999997},"max":{"1min":1.6399999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":1.609},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3500000000000001,"15min":1.319},"min":{"1min":0.96499999999999997,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.663,"5min":8.1509999999999998,"15min":8.1509999999999998},"last":1.716}]},{"osd":3,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2569999999999999,"15min":1.3049999999999999},"min":{"1min":0.99099999999999999,"5min":0.76400000000000001,"15min":0.70799999999999996},"max":{"1min":1.466,"5min":2.7559999999999998,"15min":3.036},"last":1.298},{"interface":"front","average":{"1min":1.2,"5min":1.28,"15min":1.3109999999999999},"min":{"1min":0.70499999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.474,"5min":4.1230000000000002,"15min":4.1230000000000002},"last":1.3480000000000001}]},{"osd":4,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.355,"15min":1.417},"min":{"1min":0.68600000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6259999999999999,"5min":1.9690000000000001,"15min":2.069},"last":1.821},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.254,"15min":1.319},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.77,"5min":2.1139999999999999,"15min":3.2810000000000001},"last":1.663}]},{"osd":6,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.304,"15min":1.3360000000000001},"min":{"1min":0.871,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.754,"5min":2.028,"15min":2.504},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.405,"15min":1.484},"min":{"1min":0.63800000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.575,"5min":2.1059999999999999,"15min":2.2290000000000001},"last":1.7609999999999999}]},{"osd":7,"last update":"Wed Jan 24 17:49:42 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.3280000000000001,"15min":1.4319999999999999},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.6759999999999999,"5min":2.2010000000000001,"15min":3.161},"last":1.516},{"interface":"front","average":{"1min":1.486,"5min":1.5129999999999999,"15min":1.5860000000000001},"min":{"1min":1.042,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.8420000000000001,"5min":2.262,"15min":3.2410000000000001},"last":1.8360000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.0589999999999999},"min":{"1min":0.73299999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3540000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.099,"15min":1.097},"min":{"1min":0.874,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6359999999999999,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.84999999999999998}]},{"osd":1,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.1739999999999999,"15min":1.097},"min":{"1min":0.78900000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5980000000000001,"5min":2.1480000000000001,"15min":2.1480000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1679999999999999,"15min":1.196},"min":{"1min":0.73099999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.698,"5min":1.712,"15min":1.849},"last":1.7450000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.2430000000000001,"15min":1.2210000000000001},"min":{"1min":0.97299999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.788,"5min":1.9850000000000001,"15min":2.105},"last":1.286},{"interface":"front","average":{"1min":1.347,"5min":1.3080000000000001,"15min":1.3200000000000001},"min":{"1min":0.99399999999999999,"5min":0.69599999999999995,"15min":0.64500000000000002},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.986},"last":1.4470000000000001}]},{"osd":3,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.1819999999999999,"15min":1.1870000000000001},"min":{"1min":0.76800000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":1.637,"15min":1.7769999999999999},"last":1.081},{"interface":"front","average":{"1min":1.421,"5min":1.296,"15min":1.331},"min":{"1min":0.998,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.9409999999999998},"last":1.4710000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.3169999999999999,"15min":1.3500000000000001},"min":{"1min":1.044,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.0569999999999999,"5min":2.0569999999999999,"15min":2.0979999999999999},"last":1.532},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.278,"15min":1.2949999999999999},"min":{"1min":0.91700000000000004,"5min":0.65600000000000003,"15min":0.55200000000000005},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":1.9039999999999999},"last":1.2050000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.351,"15min":1.429},"min":{"1min":0.95599999999999996,"5min":0.67800000000000005,"15min":0.64400000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":3.0840000000000001},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4550000000000001,"15min":1.468},"min":{"1min":1.1499999999999999,"5min":0.67700000000000005,"15min":0.66500000000000004},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.661}]},{"osd":7,"last update":"Wed Jan 24 17:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.399,"15min":1.4179999999999999},"min":{"1min":1.0129999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.4910000000000001,"15min":1.5740000000000001},"min":{"1min":0.98399999999999999,"5min":0.79300000000000004,"15min":0.60399999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":4.3250000000000002},"last":1.8879999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.83299999999999996,"15min":0.86599999999999999},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.4590000000000001,"5min":1.5580000000000001,"15min":5.1929999999999996},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.88800000000000001,"15min":0.90100000000000002},"min":{"1min":0.40999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.4279999999999999,"5min":1.6739999999999999,"15min":8.6989999999999998},"last":1.504}]},{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.85599999999999998,"15min":0.91700000000000004},"min":{"1min":0.46600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.4099999999999999,"5min":1.5229999999999999,"15min":1.8720000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.0389999999999999,"15min":1.0580000000000001},"min":{"1min":0.69499999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.679,"5min":1.679,"15min":3.7240000000000002},"last":1.0069999999999999}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0760000000000001,"15min":1.0980000000000001},"min":{"1min":0.499,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.581,"5min":1.7170000000000001,"15min":3.4630000000000001},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1279999999999999,"15min":1.155},"min":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":4.2919999999999998},"last":1.3}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.208,"15min":1.2689999999999999},"min":{"1min":0.76700000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6060000000000001,"5min":2.0049999999999999,"15min":6.7469999999999999},"last":1.353},{"interface":"front","average":{"1min":1.292,"5min":1.2789999999999999,"15min":1.3280000000000001},"min":{"1min":0.873,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.6950000000000001,"5min":2.0390000000000001,"15min":3.4049999999999998},"last":1.375}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3129999999999999,"15min":1.395},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.7050000000000001,"5min":2.1269999999999998,"15min":4.3789999999999996},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.363,"15min":1.4019999999999999},"min":{"1min":0.78300000000000003,"5min":0.746,"15min":0.746},"max":{"1min":2.1859999999999999,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.55}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.411,"15min":1.4770000000000001},"min":{"1min":0.82199999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.331,"5min":2.331,"15min":9.5440000000000005},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.327,"15min":1.419},"min":{"1min":0.78300000000000003,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.659,"5min":2.109,"15min":4.8170000000000002},"last":1.2450000000000001}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.373,"15min":1.472},"min":{"1min":0.90000000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.736,"5min":2.3410000000000002,"15min":6.6970000000000001},"last":1.413},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.431,"15min":1.4970000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.889,"5min":2.0990000000000002,"15min":6.6529999999999996},"last":1.609}]}]},{"osd":2,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.78700000000000003,"15min":0.79300000000000004},"min":{"1min":0.42099999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.6180000000000001,"5min":1.667,"15min":1.667},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.90000000000000002,"15min":0.92600000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.472,"5min":1.7070000000000001,"15min":1.7989999999999999},"last":0.78400000000000003}]},{"osd":1,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.018,"15min":1.04},"min":{"1min":0.52100000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1080000000000001,"15min":1.1719999999999999},"min":{"1min":0.56000000000000005,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5960000000000001,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.391}]},{"osd":3,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95799999999999996,"15min":0.98099999999999998},"min":{"1min":0.47699999999999998,"5min":0.46200000000000002,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.7130000000000001,"15min":2.2120000000000002},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.276,"5min":1.1459999999999999,"15min":1.1919999999999999},"min":{"1min":0.54900000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":2.996},"last":1.583}]},{"osd":4,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.244,"15min":1.2969999999999999},"min":{"1min":0.79500000000000004,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.748,"5min":2.0619999999999998,"15min":2.7120000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.256,"15min":1.349},"min":{"1min":0.71899999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7250000000000001,"5min":1.889,"15min":2.3839999999999999},"last":1.851}]},{"osd":5,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.371},"min":{"1min":0.99299999999999999,"5min":0.81899999999999995,"15min":0.73899999999999999},"max":{"1min":1.6879999999999999,"5min":1.851,"15min":2.3140000000000001},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.2230000000000001,"15min":1.2889999999999999},"min":{"1min":0.95599999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.804,"5min":1.889,"15min":2.0720000000000001},"last":1.52}]},{"osd":6,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.294,"15min":1.3600000000000001},"min":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.623},"max":{"1min":2.056,"5min":2.056,"15min":2.2679999999999998},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.335,"15min":1.4259999999999999},"min":{"1min":0.92100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.2589999999999999,"5min":2.2589999999999999,"15min":3.7410000000000001},"last":1.72}]},{"osd":7,"last update":"Wed Jan 24 17:49:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.383,"15min":1.508},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.6879999999999999,"5min":2.1499999999999999,"15min":3.343},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.375,"15min":1.458},"min":{"1min":0.76100000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.1600000000000001,"5min":2.173,"15min":2.3130000000000002},"last":1.623}]}]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1949696,"data_stored":1178688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.85599999999999998,"15min":0.86099999999999999},"min":{"1min":0.42999999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.54,"5min":1.722,"15min":2.0339999999999998},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.95999999999999996,"15min":0.95199999999999996},"min":{"1min":0.53700000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.5089999999999999,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.0840000000000001}]},{"osd":2,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88100000000000001,"15min":0.94499999999999995},"min":{"1min":0.51300000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3580000000000001,"5min":1.3959999999999999,"15min":1.6379999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.069,"15min":1.0900000000000001},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.41799999999999998},"max":{"1min":1.556,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":1.224}]},{"osd":3,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.171,"15min":1.163},"min":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.47699999999999998},"max":{"1min":1.623,"5min":1.8440000000000001,"15min":1.9610000000000001},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.052,"5min":1.0760000000000001,"15min":1.1799999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.41499999999999998},"max":{"1min":1.446,"5min":1.6910000000000001,"15min":3.149},"last":0.97499999999999998}]},{"osd":4,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.355},"min":{"1min":0.79000000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.7250000000000001,"5min":1.962,"15min":2.2410000000000001},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2969999999999999},"min":{"1min":0.871,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.673,"5min":1.734,"15min":2.3849999999999998},"last":1.514}]},{"osd":5,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.248,"15min":1.3149999999999999},"min":{"1min":0.876,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.581,"5min":1.964,"15min":2.577},"last":1.397},{"interface":"front","average":{"1min":1.323,"5min":1.373,"15min":1.371},"min":{"1min":0.90400000000000003,"5min":0.89800000000000002,"15min":0.81299999999999994},"max":{"1min":1.802,"5min":2.2770000000000001,"15min":2.9809999999999999},"last":1.5860000000000001}]},{"osd":6,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3560000000000001,"15min":1.4079999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.73099999999999998},"max":{"1min":1.7809999999999999,"5min":2.008,"15min":3.2850000000000001},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.351,"15min":1.4590000000000001},"min":{"1min":0.94699999999999995,"5min":0.72599999999999998,"15min":0.70199999999999996},"max":{"1min":1.6950000000000001,"5min":2.032,"15min":3.5859999999999999},"last":1.159}]},{"osd":7,"last update":"Wed Jan 24 17:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.4219999999999999,"15min":1.454},"min":{"1min":0.91400000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.97,"5min":2.5880000000000001,"15min":2.8439999999999999},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4419999999999999,"15min":1.5089999999999999},"min":{"1min":0.83499999999999996,"5min":0.82399999999999995,"15min":0.74099999999999999},"max":{"1min":1.891,"5min":2.1400000000000001,"15min":3.4350000000000001},"last":1.45}]}]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32444,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":1355776,"data_stored":588320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":31848715},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81499999999999995,"15min":0.81399999999999995},"min":{"1min":0.42499999999999999,"5min":0.40999999999999998,"15min":0.29099999999999998},"max":{"1min":1.266,"5min":1.4750000000000001,"15min":1.7290000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.98899999999999999,"15min":0.96899999999999997},"min":{"1min":0.65700000000000003,"5min":0.433,"15min":0.376},"max":{"1min":1.7569999999999999,"5min":1.778,"15min":1.778},"last":0.872}]},{"osd":2,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.86099999999999999,"15min":0.84399999999999997},"min":{"1min":0.73499999999999999,"5min":0.42999999999999999,"15min":0.34699999999999998},"max":{"1min":1.1100000000000001,"5min":1.837,"15min":1.837},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0249999999999999,"15min":1.0660000000000001},"min":{"1min":0.69299999999999995,"5min":0.50600000000000001,"15min":0.42399999999999999},"max":{"1min":1.4359999999999999,"5min":1.583,"15min":2.0499999999999998},"last":1.1579999999999999}]},{"osd":3,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.143,"15min":1.165},"min":{"1min":0.497,"5min":0.497,"15min":0.35799999999999998},"max":{"1min":1.5900000000000001,"5min":1.8320000000000001,"15min":1.9670000000000001},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.212,"15min":1.1919999999999999},"min":{"1min":0.66200000000000003,"5min":0.49299999999999999,"15min":0.46800000000000003},"max":{"1min":1.849,"5min":1.849,"15min":2.407},"last":1.417}]},{"osd":4,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.298,"15min":1.367},"min":{"1min":0.94199999999999995,"5min":0.90000000000000002,"15min":0.70799999999999996},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":2.2589999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.345,"15min":1.4279999999999999},"min":{"1min":1.069,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":3.3740000000000001},"last":1.3560000000000001}]},{"osd":5,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3049999999999999,"15min":1.3560000000000001},"min":{"1min":0.98299999999999998,"5min":0.83699999999999997,"15min":0.73499999999999999},"max":{"1min":2.101,"5min":2.101,"15min":2.9300000000000002},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.262,"15min":1.262},"min":{"1min":0.86599999999999999,"5min":0.77000000000000002,"15min":0.65600000000000003},"max":{"1min":1.794,"5min":1.8520000000000001,"15min":2.3799999999999999},"last":1.014}]},{"osd":6,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.353,"15min":1.3540000000000001},"min":{"1min":0.89200000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.855,"5min":1.954,"15min":2.335},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.413,"15min":1.4370000000000001},"min":{"1min":1.0740000000000001,"5min":0.90600000000000003,"15min":0.70999999999999996},"max":{"1min":1.873,"5min":1.931,"15min":2.2389999999999999},"last":1.494}]},{"osd":7,"last update":"Wed Jan 24 17:49:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4319999999999999,"15min":1.488},"min":{"1min":1.153,"5min":0.89700000000000002,"15min":0.76100000000000001},"max":{"1min":1.917,"5min":2.0019999999999998,"15min":3.0230000000000001},"last":1.212},{"interface":"front","average":{"1min":1.538,"5min":1.4750000000000001,"15min":1.55},"min":{"1min":0.91600000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.962,"5min":2.1659999999999999,"15min":3.1600000000000001},"last":1.5509999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-24T17:50:30.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-24T17:50:30.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.544+0000 7f7ac5e00640 1 Processor -- start 2024-01-24T17:50:30.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.544+0000 7f7ac5e00640 1 -- start start 2024-01-24T17:50:30.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.546+0000 7f7ac5e00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7ac00cf8a0 0x7f7ac00d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.546+0000 7f7ac5e00640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00ce010 con 0x7f7ac015cbf0 2024-01-24T17:50:30.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.546+0000 7f7ac5e00640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00ce2b0 con 0x7f7ac006acb0 2024-01-24T17:50:30.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.546+0000 7f7ac5e00640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac00ce870 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.546+0000 7f7abfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7ac00cf8a0 0x7f7ac00d0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.546+0000 7f7abfe00640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7ac00cf8a0 0x7f7ac00d0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58684/0 (socket says 172.21.15.39:58684) 2024-01-24T17:50:30.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.546+0000 7f7abfe00640 1 -- 172.21.15.39:0/2476805475 learned_addr learned my addr 172.21.15.39:0/2476805475 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:30.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.547+0000 7f7abfe00640 1 -- 172.21.15.39:0/2476805475 >> v1:172.21.15.100:6789/0 conn(0x7f7ac006acb0 legacy=0x7f7ac00d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:30.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.547+0000 7f7abfe00640 1 -- 172.21.15.39:0/2476805475 >> v1:172.21.15.39:6789/0 conn(0x7f7ac015cbf0 legacy=0x7f7ac00cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.547+0000 7f7abfe00640 1 -- 172.21.15.39:0/2476805475 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac00ce590 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.548+0000 7f7abfe00640 1 --2- 172.21.15.39:0/2476805475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac00d0a90 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009870 tx=0x7f7aa802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25325df806a5bb24 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:30.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.548+0000 7f7abf400640 1 -- 172.21.15.39:0/2476805475 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8031870 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.548+0000 7f7abf400640 1 -- 172.21.15.39:0/2476805475 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa80319d0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.548+0000 7f7abf400640 1 -- 172.21.15.39:0/2476805475 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8031d60 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.549+0000 7f7ac5e00640 1 -- 172.21.15.39:0/2476805475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 msgr2=0x7f7ac00d0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.549+0000 7f7ac5e00640 1 --2- 172.21.15.39:0/2476805475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac00d0a90 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009870 tx=0x7f7aa802b910 comp rx=0 tx=0).stop 2024-01-24T17:50:30.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.549+0000 7f7ac5e00640 1 -- 172.21.15.39:0/2476805475 shutdown_connections 2024-01-24T17:50:30.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.549+0000 7f7ac5e00640 1 --2- 172.21.15.39:0/2476805475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac00d0a90 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.549+0000 7f7ac5e00640 1 -- 172.21.15.39:0/2476805475 >> 172.21.15.39:0/2476805475 conn(0x7f7ac009bad0 msgr2=0x7f7ac00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:30.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.549+0000 7f7ac5e00640 1 -- 172.21.15.39:0/2476805475 shutdown_connections 2024-01-24T17:50:30.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.549+0000 7f7ac5e00640 1 -- 172.21.15.39:0/2476805475 wait complete. 2024-01-24T17:50:30.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.552+0000 7f7ac5e00640 1 Processor -- start 2024-01-24T17:50:30.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.552+0000 7f7ac5e00640 1 -- start start 2024-01-24T17:50:30.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.553+0000 7f7ac5e00640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7ac006acb0 0x7f7ac00cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.553+0000 7f7ac5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac01f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.553+0000 7f7ac5e00640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ac015cbf0 0x7f7ac01f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.553+0000 7f7ac5e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00c7cd0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.553+0000 7f7ac5e00640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00cc2d0 con 0x7f7ac015cbf0 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.553+0000 7f7ac5e00640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00cbd10 con 0x7f7ac006acb0 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac01f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac01f5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58696/0 (socket says 172.21.15.39:58696) 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7abfe00640 1 -- 172.21.15.39:0/3618469894 learned_addr learned my addr 172.21.15.39:0/3618469894 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7ac4a00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7ac006acb0 0x7f7ac00cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7ac5400640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ac015cbf0 0x7f7ac01f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7abfe00640 1 -- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7ac006acb0 msgr2=0x7f7ac00cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7abfe00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7ac006acb0 0x7f7ac00cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7abfe00640 1 -- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ac015cbf0 msgr2=0x7f7ac01f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7abfe00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ac015cbf0 0x7f7ac01f5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.554+0000 7f7abfe00640 1 -- 172.21.15.39:0/3618469894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80094d0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.555+0000 7f7abfe00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac01f5660 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009870 tx=0x7f7aa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:30.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.555+0000 7f7abcc00640 1 -- 172.21.15.39:0/3618469894 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8031820 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.555+0000 7f7abcc00640 1 -- 172.21.15.39:0/3618469894 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa80443f0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.555+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac00cbff0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.555+0000 7f7abcc00640 1 -- 172.21.15.39:0/3618469894 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa80447a0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.555+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac00c81e0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.557+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac004dff0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.558+0000 7f7abcc00640 1 -- 172.21.15.39:0/3618469894 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f7aa804d450 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.558+0000 7f7abcc00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a9806a430 0x7f7a9806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:30.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.559+0000 7f7abcc00640 1 -- 172.21.15.39:0/3618469894 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f7aa802d040 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.559+0000 7f7ac4a00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a9806a430 0x7f7a9806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:30.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.560+0000 7f7ac4a00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a9806a430 0x7f7a9806c8f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0000fd0 tx=0x7f7ab0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:30.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.563+0000 7f7abcc00640 1 -- 172.21.15.39:0/3618469894 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7f7aa80766e0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.756+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f7ac00b86f0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.765+0000 7f7abcc00640 1 -- 172.21.15.39:0/3618469894 <== mon.0 v2:172.21.15.39:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f7aa80766e0 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.765+0000 7f7abcc00640 1 -- 172.21.15.39:0/3618469894 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v26)=0 v26) v1 ==== 123+0+0 (secure 0 0 0) 0x7f7aa8079d00 con 0x7f7ac00cf8a0 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.768+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a9806a430 msgr2=0x7f7a9806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.768+0000 7f7ac5e00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a9806a430 0x7f7a9806c8f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0000fd0 tx=0x7f7ab0009040 comp rx=0 tx=0).stop 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.768+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 msgr2=0x7f7ac01f5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.768+0000 7f7ac5e00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac01f5660 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009870 tx=0x7f7aa80046c0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.769+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 shutdown_connections 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.769+0000 7f7ac5e00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ac015cbf0 0x7f7ac01f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.769+0000 7f7ac5e00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7f7a9806a430 0x7f7a9806c8f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.769+0000 7f7ac5e00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac00cf8a0 0x7f7ac01f5660 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.769+0000 7f7ac5e00640 1 --2- 172.21.15.39:0/3618469894 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7f7ac006acb0 0x7f7ac00cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.769+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 >> 172.21.15.39:0/3618469894 conn(0x7f7ac009bad0 msgr2=0x7f7ac00b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.769+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 shutdown_connections 2024-01-24T17:50:30.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:30.769+0000 7f7ac5e00640 1 -- 172.21.15.39:0/3618469894 wait complete. 2024-01-24T17:50:30.792 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-24T17:50:30.792 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T17:50:31.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.006+0000 7fc15a000640 1 Processor -- start 2024-01-24T17:50:31.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.007+0000 7fc15a000640 1 -- start start 2024-01-24T17:50:31.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.008+0000 7fc15a000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc154071e10 0x7fc15406fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:31.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.008+0000 7fc15a000640 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fc1540c1500 0x7fc1540afe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:31.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.008+0000 7fc15a000640 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1541658b0 con 0x7fc1540c19e0 2024-01-24T17:50:31.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.008+0000 7fc15a000640 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc154161250 con 0x7fc1540c1500 2024-01-24T17:50:31.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.008+0000 7fc15a000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc154164ca0 con 0x7fc154071e10 2024-01-24T17:50:31.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc159600640 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fc1540c19e0 0x7fc1540cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54702/0 (socket says 172.21.15.39:54702) 2024-01-24T17:50:31.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc159600640 1 -- 172.21.15.39:0/675622481 learned_addr learned my addr 172.21.15.39:0/675622481 (peer_addr_for_me v1:172.21.15.39:0/0) 2024-01-24T17:50:31.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc14be00640 1 --2- 172.21.15.39:0/675622481 >> v2:172.21.15.39:3300/0 conn(0x7fc1540c1500 0x7fc1540afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:31.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc158c00640 1 --2- 172.21.15.39:0/675622481 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc154071e10 0x7fc15406fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:31.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc153e00640 1 -- 172.21.15.39:0/675622481 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034891938 0 0) 0x7fc1541658b0 con 0x7fc1540c19e0 2024-01-24T17:50:31.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc153e00640 1 -- 172.21.15.39:0/675622481 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc134003520 con 0x7fc1540c19e0 2024-01-24T17:50:31.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc14be00640 1 -- 172.21.15.39:0/675622481 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc154071e10 msgr2=0x7fc15406fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T17:50:31.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc14be00640 1 --2- 172.21.15.39:0/675622481 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc154071e10 0x7fc15406fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc14be00640 1 -- 172.21.15.39:0/675622481 >> v1:172.21.15.39:6789/0 conn(0x7fc1540c19e0 legacy=0x7fc1540cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:31.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.009+0000 7fc14be00640 1 -- 172.21.15.39:0/675622481 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc154164f80 con 0x7fc1540c1500 2024-01-24T17:50:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.010+0000 7fc14be00640 1 --2- 172.21.15.39:0/675622481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1540c1500 0x7fc1540afe60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0097d0 tx=0x7fc13c02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcacd762bc315e77 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc153e00640 1 -- 172.21.15.39:0/675622481 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c031770 con 0x7fc1540c1500 2024-01-24T17:50:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc153e00640 1 -- 172.21.15.39:0/675622481 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc13c0318d0 con 0x7fc1540c1500 2024-01-24T17:50:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc153e00640 1 -- 172.21.15.39:0/675622481 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13c031c40 con 0x7fc1540c1500 2024-01-24T17:50:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc15a000640 1 -- 172.21.15.39:0/675622481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1540c1500 msgr2=0x7fc1540afe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc15a000640 1 --2- 172.21.15.39:0/675622481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1540c1500 0x7fc1540afe60 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fc13c0097d0 tx=0x7fc13c02b810 comp rx=0 tx=0).stop 2024-01-24T17:50:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc15a000640 1 -- 172.21.15.39:0/675622481 shutdown_connections 2024-01-24T17:50:31.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc15a000640 1 --2- 172.21.15.39:0/675622481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1540c1500 0x7fc1540afe60 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc15a000640 1 --2- 172.21.15.39:0/675622481 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc154071e10 0x7fc15406fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.011+0000 7fc15a000640 1 -- 172.21.15.39:0/675622481 >> 172.21.15.39:0/675622481 conn(0x7fc15406a340 msgr2=0x7fc1540cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:31.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.012+0000 7fc15a000640 1 -- 172.21.15.39:0/675622481 shutdown_connections 2024-01-24T17:50:31.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.012+0000 7fc15a000640 1 -- 172.21.15.39:0/675622481 wait complete. 2024-01-24T17:50:31.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.013+0000 7fc15a000640 1 Processor -- start 2024-01-24T17:50:31.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.013+0000 7fc15a000640 1 -- start start 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.014+0000 7fc15a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154071e10 0x7fc1540c3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.014+0000 7fc15a000640 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1540c1500 0x7fc154154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.014+0000 7fc15a000640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1540c19e0 0x7fc1540b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.014+0000 7fc15a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540b1460 con 0x7fc154071e10 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.014+0000 7fc15a000640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540b16c0 con 0x7fc1540c19e0 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.014+0000 7fc15a000640 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540b19a0 con 0x7fc1540c1500 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc158c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154071e10 0x7fc1540c3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc158c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154071e10 0x7fc1540c3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58726/0 (socket says 172.21.15.39:58726) 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc158c00640 1 -- 172.21.15.39:0/3993757389 learned_addr learned my addr 172.21.15.39:0/3993757389 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc14be00640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1540c1500 0x7fc154154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc158c00640 1 -- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1540c1500 msgr2=0x7fc154154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:31.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc158c00640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1540c1500 0x7fc154154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc158c00640 1 -- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1540c19e0 msgr2=0x7fc1540b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:31.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc159600640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1540c19e0 0x7fc1540b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:31.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc158c00640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1540c19e0 0x7fc1540b0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.015+0000 7fc158c00640 1 -- 172.21.15.39:0/3993757389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c009400 con 0x7fc154071e10 2024-01-24T17:50:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.016+0000 7fc158c00640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154071e10 0x7fc1540c3010 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fc14400d7d0 tx=0x7fc14400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.016+0000 7fc151600640 1 -- 172.21.15.39:0/3993757389 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1440096c0 con 0x7fc154071e10 2024-01-24T17:50:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.016+0000 7fc151600640 1 -- 172.21.15.39:0/3993757389 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fc144009820 con 0x7fc154071e10 2024-01-24T17:50:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.016+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1540b1c80 con 0x7fc154071e10 2024-01-24T17:50:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.016+0000 7fc151600640 1 -- 172.21.15.39:0/3993757389 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144009bf0 con 0x7fc154071e10 2024-01-24T17:50:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.016+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1540b2240 con 0x7fc154071e10 2024-01-24T17:50:31.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.017+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15404dff0 con 0x7fc154071e10 2024-01-24T17:50:31.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.019+0000 7fc151600640 1 -- 172.21.15.39:0/3993757389 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fc144009d50 con 0x7fc154071e10 2024-01-24T17:50:31.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.019+0000 7fc151600640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc12c06a430 0x7fc12c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T17:50:31.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.020+0000 7fc151600640 1 -- 172.21.15.39:0/3993757389 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fc144088f70 con 0x7fc154071e10 2024-01-24T17:50:31.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.020+0000 7fc14be00640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc12c06a430 0x7fc12c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T17:50:31.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.021+0000 7fc14be00640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc12c06a430 0x7fc12c06c8f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc13c036000 tx=0x7fc13c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T17:50:31.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.024+0000 7fc151600640 1 -- 172.21.15.39:0/3993757389 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179360 (secure 0 0 0) 0x7fc1440bbcb0 con 0x7fc154071e10 2024-01-24T17:50:31.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.264+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1541645f0 con 0x7fc154071e10 2024-01-24T17:50:31.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.264+0000 7fc151600640 1 -- 172.21.15.39:0/3993757389 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fc144058190 con 0x7fc154071e10 2024-01-24T17:50:31.266 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-01-24T17:50:31.266 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5846,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6835/2422700998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6834","nonce":2422700998},{"type":"v1","addr":"172.21.15.100:6835","nonce":2422700998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":6643,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.100:6837/1453932633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6836","nonce":1453932633},{"type":"v1","addr":"172.21.15.100:6837","nonce":1453932633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65},{"gid":8499,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6834/2220905980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":2220905980},{"type":"v1","addr":"172.21.15.39:6834","nonce":2220905980}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74},{"gid":8502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/3774148591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6833","nonce":3774148591},{"type":"v1","addr":"172.21.15.39:6835","nonce":3774148591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":74}],"filesystems":[]} 2024-01-24T17:50:31.267 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 74 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc12c06a430 msgr2=0x7fc12c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc12c06a430 0x7fc12c06c8f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc13c036000 tx=0x7fc13c03d040 comp rx=0 tx=0).stop 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154071e10 msgr2=0x7fc1540c3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154071e10 0x7fc1540c3010 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fc14400d7d0 tx=0x7fc14400dc90 comp rx=0 tx=0).stop 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 shutdown_connections 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc1540c19e0 0x7fc1540b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fc1540c1500 0x7fc154154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.100:6832/36263,v1:172.21.15.100:6833/36263] conn(0x7fc12c06a430 0x7fc12c06c8f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 --2- 172.21.15.39:0/3993757389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154071e10 0x7fc1540c3010 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T17:50:31.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 >> 172.21.15.39:0/3993757389 conn(0x7fc15406a340 msgr2=0x7fc154161a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T17:50:31.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.267+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 shutdown_connections 2024-01-24T17:50:31.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-01-24T17:50:31.268+0000 7fc15a000640 1 -- 172.21.15.39:0/3993757389 wait complete. 2024-01-24T17:50:31.288 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-24T17:50:31.288 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-24T17:50:31.288 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-24T17:50:31.288 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.308 INFO:tasks.ceph.mds.a:Stopped 2024-01-24T17:50:31.308 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-24T17:50:31.308 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.370 INFO:tasks.ceph.mds.c:Stopped 2024-01-24T17:50:31.370 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-24T17:50:31.370 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.398 INFO:tasks.ceph.mds.b:Stopped 2024-01-24T17:50:31.398 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-24T17:50:31.398 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.465 INFO:tasks.ceph.mds.d:Stopped 2024-01-24T17:50:31.466 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-24T17:50:31.466 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-24T17:50:31.466 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.514 INFO:tasks.ceph.osd.0:Stopped 2024-01-24T17:50:31.514 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-24T17:50:31.514 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.575 INFO:tasks.ceph.osd.1:Stopped 2024-01-24T17:50:31.575 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-24T17:50:31.576 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.651 INFO:tasks.ceph.osd.2:Stopped 2024-01-24T17:50:31.651 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-24T17:50:31.651 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.732 INFO:tasks.ceph.osd.3:Stopped 2024-01-24T17:50:31.732 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-24T17:50:31.732 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.799 INFO:tasks.ceph.osd.4:Stopped 2024-01-24T17:50:31.799 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-24T17:50:31.799 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.866 INFO:tasks.ceph.osd.5:Stopped 2024-01-24T17:50:31.866 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-24T17:50:31.867 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.924 INFO:tasks.ceph.osd.6:Stopped 2024-01-24T17:50:31.924 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-24T17:50:31.924 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:31.978 INFO:tasks.ceph.osd.7:Stopped 2024-01-24T17:50:31.978 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-24T17:50:31.978 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-24T17:50:31.978 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:35.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-01-24T17:50:38.080 INFO:tasks.ceph.mgr.y:Stopped 2024-01-24T17:50:38.080 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-24T17:50:38.080 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:38.180 INFO:tasks.ceph.mgr.x:Stopped 2024-01-24T17:50:38.180 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-24T17:50:38.181 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-24T17:50:38.181 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:38.233 INFO:tasks.ceph.mon.a:Stopped 2024-01-24T17:50:38.234 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-24T17:50:38.234 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:38.276 INFO:tasks.ceph.mon.b:Stopped 2024-01-24T17:50:38.277 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-24T17:50:38.277 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T17:50:38.313 INFO:tasks.ceph.mon.c:Stopped 2024-01-24T17:50:38.313 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-24T17:50:38.313 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'client session with non-allowable root' | head -n 1 2024-01-24T17:50:38.420 INFO:teuthology.orchestra.run.smithi039.stdout:2024-01-24T17:46:18.401026+0000 mon.a (mon.0) 671 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-01-24T17:50:38.421 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-01-24T17:50:38.421 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'client session with non-allowable root' | head -n 1 2024-01-24T17:50:38.508 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'client session with non-allowable root' | head -n 1 2024-01-24T17:50:38.590 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'client session with non-allowable root' | head -n 1 2024-01-24T17:50:38.673 INFO:teuthology.orchestra.run.smithi039.stdout:2024-01-24T17:46:18.401026+0000 mon.a (mon.0) 671 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-01-24T17:50:38.674 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:50:38.674 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-24T17:50:38.936 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:50:38.937 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-24T17:50:39.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:50:39.139 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-24T17:50:39.301 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:50:39.301 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-24T17:50:39.491 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:50:39.492 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-24T17:50:39.657 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:50:39.657 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-24T17:50:39.799 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:50:39.799 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-24T17:50:39.967 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:50:39.968 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-24T17:50:40.136 INFO:tasks.ceph:Archiving mon data... 2024-01-24T17:50:40.137 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/data/mon.a.tgz 2024-01-24T17:50:40.138 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-01-24T17:50:40.198 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.6ONETS73P3 2024-01-24T17:50:40.199 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6ONETS73P3 2024-01-24T17:50:40.477 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.6ONETS73P3 2024-01-24T17:50:40.640 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.6ONETS73P3 is 344KB 2024-01-24T17:50:40.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-01-24T17:50:40.713 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.6ONETS73P3 2024-01-24T17:50:40.769 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/data/mon.b.tgz 2024-01-24T17:50:40.770 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-01-24T17:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.llGUeUHqBf 2024-01-24T17:50:40.827 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.llGUeUHqBf 2024-01-24T17:50:41.014 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.llGUeUHqBf 2024-01-24T17:50:41.167 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.llGUeUHqBf is 348KB 2024-01-24T17:50:41.272 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.llGUeUHqBf 2024-01-24T17:50:41.329 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/data/mon.c.tgz 2024-01-24T17:50:41.329 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-01-24T17:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.ANUtqiwwNd 2024-01-24T17:50:41.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ANUtqiwwNd 2024-01-24T17:50:41.538 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.ANUtqiwwNd 2024-01-24T17:50:41.685 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.ANUtqiwwNd is 348KB 2024-01-24T17:50:41.793 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.ANUtqiwwNd 2024-01-24T17:50:41.850 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-24T17:50:41.850 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-24T17:50:41.858 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-24T17:50:41.954 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-01-24T17:50:41.954 DEBUG:teuthology.orchestra.run.smithi039:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-24T17:50:41.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-24T17:50:42.058 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-24T17:50:42.063 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-24T17:50:42.074 INFO:tasks.ceph:Archiving crash dumps... 2024-01-24T17:50:42.076 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/remote/smithi039/crash 2024-01-24T17:50:42.077 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-24T17:50:42.159 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/remote/smithi100/crash 2024-01-24T17:50:42.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-24T17:50:42.253 INFO:tasks.ceph:Shutting down logrotate 2024-01-24T17:50:42.253 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:50:42.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-24T17:50:42.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-24T17:50:42.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-24T17:50:42.587 INFO:tasks.ceph:Compressing logs... 2024-01-24T17:50:42.587 DEBUG:teuthology.orchestra.run.smithi039:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T17:50:42.595 DEBUG:teuthology.orchestra.run.smithi100:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T17:50:45.727 INFO:tasks.ceph:Archiving logs... 2024-01-24T17:50:45.727 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/remote/smithi039/log 2024-01-24T17:50:45.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-24T17:50:46.669 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/remote/smithi100/log 2024-01-24T17:50:46.670 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-24T17:50:47.090 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-24T17:50:47.103 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-24T17:50:47.103 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-24T17:50:47.112 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-24T17:50:47.267 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-24T17:50:47.277 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-24T17:50:47.277 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T17:50:47.286 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T17:50:47.336 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-01-24T17:50:47.341 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-01-24T17:50:47.347 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T17:50:47.347 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-01-24T17:50:47.348 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -90us[ -90us] +/- 74ms 2024-01-24T17:50:47.348 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -238us[ -228us] +/- 58ms 2024-01-24T17:50:47.348 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +454us[ +454us] +/- 78ms 2024-01-24T17:50:47.348 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:50:47.349 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T17:50:47.350 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-01-24T17:50:47.350 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -88us[ -88us] +/- 74ms 2024-01-24T17:50:47.350 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -234us[ -198us] +/- 58ms 2024-01-24T17:50:47.350 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +437us[ +437us] +/- 78ms 2024-01-24T17:50:47.350 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T17:50:47.351 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-24T17:50:47.363 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-24T17:50:47.364 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-24T17:50:47.381 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-24T17:50:47.642 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-24T17:50:47.868 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T17:50:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:50:47.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T17:50:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:50:48.082 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-24T17:50:48.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-24T17:50:48.122 INFO:teuthology.task.internal:Duration was 1099.644725 seconds 2024-01-24T17:50:48.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-24T17:50:48.155 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-24T17:50:48.156 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-24T17:50:48.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-24T17:50:48.331 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T17:50:48.335 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T17:50:48.903 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-24T17:50:48.903 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-01-24T17:50:48.903 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-24T17:50:48.985 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2024-01-24T17:50:48.985 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-24T17:50:49.058 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-24T17:50:49.059 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T17:50:49.067 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T17:50:49.162 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-24T17:50:49.162 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-24T17:50:49.168 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-24T17:50:49.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-24T17:50:49.390 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-24T17:50:49.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-24T17:50:49.491 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-24T17:50:49.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-24T17:50:49.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-24T17:50:49.610 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-24T17:50:49.717 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-01-24T17:50:49.721 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-01-24T17:50:49.776 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-24T17:50:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:50:49.826 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-24T17:50:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T17:50:49.883 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-24T17:50:49.899 INFO:teuthology.task.internal:Transferring archived files... 2024-01-24T17:50:49.900 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/remote/smithi039 2024-01-24T17:50:49.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-24T17:50:50.015 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972/remote/smithi100 2024-01-24T17:50:50.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-24T17:50:50.120 INFO:teuthology.task.internal:Removing archive directory... 2024-01-24T17:50:50.121 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-24T17:50:50.129 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-24T17:50:50.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-24T17:50:50.195 INFO:teuthology.task.internal:Not uploading archives. 2024-01-24T17:50:50.195 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-24T17:50:50.224 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-24T17:50:50.224 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-24T17:50:50.232 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-24T17:50:50.284 INFO:teuthology.orchestra.run.smithi039.stdout: 393936 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 17:50 /home/ubuntu/cephtest 2024-01-24T17:50:50.287 INFO:teuthology.orchestra.run.smithi100.stdout: 393945 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 17:50 /home/ubuntu/cephtest 2024-01-24T17:50:50.290 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-24T17:50:50.304 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-24T17:50:50.367 INFO:teuthology.nuke:Checking targets against current locks 2024-01-24T17:50:50.393 DEBUG:teuthology.nuke:shortname: smithi039 2024-01-24T17:50:50.394 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T17:50:50.418 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 17:22:05.655884', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKIzEkcl8WMnQW1kTCFzMXo6NA5rOiCIVmprT9OVTY5g4soVADD3sd9U5Sy2pPHYpjgZjbhWpvpAbWGKYxXKpMg='} 2024-01-24T17:50:50.419 DEBUG:teuthology.nuke:shortname: smithi100 2024-01-24T17:50:50.419 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T17:50:50.440 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 17:22:05.657644', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6sBC4h2uBb88W06XRSHJhI8OMNG9o8ip/9uHl7OoVnW/PfJCu9Q38OVllMnilIDdnnCzFgqRJ8AZ8nKvxECl4='} 2024-01-24T17:50:50.467 INFO:teuthology.orchestra.console.smithi039:Power off 2024-01-24T17:50:50.467 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-24T17:50:50.492 DEBUG:teuthology.orchestra.console.smithi039:power off output: Chassis Power Control: Down/Off 2024-01-24T17:50:50.492 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T17:50:50.507 INFO:teuthology.orchestra.console.smithi100:Power off 2024-01-24T17:50:50.508 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-24T17:50:50.525 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2024-01-24T17:50:50.531 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off 2024-01-24T17:50:50.532 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T17:50:50.554 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2024-01-24T17:50:54.526 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T17:50:54.556 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T17:50:54.637 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2024-01-24T17:50:54.667 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2024-01-24T17:50:58.640 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T17:50:58.667 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T17:50:58.751 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2024-01-24T17:50:58.777 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off 2024-01-24T17:50:58.878 INFO:teuthology.orchestra.console.smithi100:Power off completed 2024-01-24T17:51:02.754 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T17:51:02.864 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is off 2024-01-24T17:51:02.964 INFO:teuthology.orchestra.console.smithi039:Power off completed 2024-01-24T17:51:03.069 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} duration: 1099.6447253227234 failure_reason: '"2024-01-24T17:46:18.401026+0000 mon.a (mon.0) 671 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log' flavor: default mon.a-kernel-sha1: f2d3460408951339ac0933340ad449487c9efddc mon.b-kernel-sha1: f2d3460408951339ac0933340ad449487c9efddc owner: scheduled_vshankar@teuthology success: false 2024-01-24T17:51:03.070 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-24T17:51:03.152 INFO:teuthology.run:FAIL